Tests that run on every change and block what actually breaks
We build the suite and wire it into your pipeline. Then comes the harder part: keeping the results believable. A suite people rerun until it turns green is worse than no suite at all.
In short: We build an automated test suite and connect it to your build pipeline, so it runs on every change and stops broken code from merging. Two things decide whether it lasts a year: how hard we work on flakiness, and whether your own developers can read the test code.
Running on every change, not on request
A suite someone remembers to run before a release is a checklist. A suite that runs automatically on every push is a gate. We connect the tests to your build system so every branch is checked before it merges, the result posts back onto the pull request, and a failing run stops the merge instead of starting a conversation. If there is no pipeline to plug into yet, we build one first, which is CI/CD work rather than testing work, and we say so before the estimate.
Run time matters more than it looks. When a suite takes long enough that developers stop waiting for it, they start merging on hope. So we split it. Fast unit and contract checks run on every push and finish quickly. The slow end-to-end set runs on merge to the main branch and on a schedule overnight. The split is not elegant, but it keeps the feedback close enough to the mistake that someone still remembers writing it.
Timing dependence
The test waits a fixed period for something that usually finishes sooner. On a loaded build agent it does not. The fix is waiting on a condition, never on a clock.
Order dependence
One test leaves a record behind and the next one passes because of it. Shuffle the order and the second fails. Each test has to create what it needs and clean up after itself.
Shared data in parallel runs
Two tests running at the same time claim the same account or the same row. One of them loses. We give every test its own fixture, and lock the few cases that genuinely cannot be isolated.
Reaching a real service
A test that calls a live payment sandbox fails whenever that sandbox is slow. Your code was fine. We stub at the boundary and check the real integration in a separate, smaller run.
Flakiness is what kills suites
Here is how it goes. A test fails. Somebody reruns it. It passes. Nothing in the code changed between those two runs. Do that a handful of times and the team has learned something genuinely dangerous, which is that a red build means nothing and the correct response is to press the button again. After that the suite costs money every month and catches almost nothing, because the one time it was right, it got rerun too.
So we treat it as a measurement problem rather than a mood. Every failure is recorded against its commit, along with whether a rerun of that identical commit went green. That gives a rerun-pass rate for each individual test. Then the tests get ranked by it, worst first, and the top of that list is the work queue. Some get fixed at the cause. Some get deleted, and deleting is a real option we will argue for, because a test nobody believes is worse than a gap everybody knows about.
Four causes account for most of it, and they are listed above. None of them are exotic. What makes flakiness hard is not diagnosis, it is that fixing it is unglamorous work with no new feature at the end, so it gets postponed until the suite is already dead.
The suite has to belong to your developers
A test suite only your vendor can extend is not an asset. It is a subscription. We write in the language your team already works in, using the runner already in the repository, so a developer who has never opened the test folder can still guess how it works. Helpers get named after things in your product rather than things in the framework. Setup lives in one obvious place. Adding a case for a new screen should mean copying a neighboring file and changing a few lines.
There is a simple test for whether we got this right. When a developer breaks something and the suite goes red, can that person read the failing test and work out what it was protecting, without asking us? If not, the next thing they will do is delete the test. We would rather write something plainer that survives. This applies just as much when the suite sits alongside custom software we built for you, since the handover matters more, not less.
What we automate, and in what order
Not everything, and not evenly. The first tests cover the paths where a break costs the most, which usually means signup, checkout, and anything that moves money or writes customer data. Next come the bugs you have already fixed once, each one turned into a test so it cannot come back quietly. After that, the checks that are tedious for a person and instant for a machine: form validation, permission matrices, API response shapes.
Things still being redesigned stay out of the suite. A screen that changes every week produces a broken test every week, and the team learns to blame the tests rather than read them. Those areas belong in manual and exploratory testing until the design settles, and then we automate them.
What we won’t do
We won’t chase a coverage percentage
Coverage is easy to raise and tells you very little. You can execute a line without checking anything about what it did. If you need a number for a board slide, we will give you the number, but we will not let it choose what gets tested.
We won’t automate a screen that changes every week
Writing tests against a moving design produces maintenance work and nothing else. We will say the area is not ready and suggest covering it with a person instead, even when automating it would be the easier thing to invoice for.
We won’t quarantine failing tests to keep the build green
Muting a test makes the dashboard look better and the suite mean less. A quarantined test gets a deadline and an owner, and if it is still unreliable at the end of it, we delete it and tell you what coverage you just lost.
Does your team still believe the test results?
Give us access to the repository and your pipeline config. We will come back with which tests are unreliable, what is causing each one, and what a suite worth trusting would take.
The paths where a failure costs the most, and the bugs you have already fixed once. Those give the most protection per test written. Broad coverage of rarely used screens comes much later, if it comes at all.
What counts as a flaky test?
A test that passes and fails on the same code. We track it as a rerun-pass rate per test: how often a failure turns green on a rerun with nothing changed. Anything above a low threshold goes into the fix-or-delete queue.
Can our own developers maintain the suite afterwards?
That is the point of building it this way. We use the language and test runner already in your repository, keep the structure flat enough to read, and leave a short guide for adding a case. If your developers cannot extend it without us, we built the wrong thing.
Should failing tests block a merge?
Yes, once they are reliable enough to be believed. A suite that runs but never blocks anything gets ignored quickly. We usually start with a small blocking set and grow it as the flakiness numbers come down.
