Taming Flaky Tests: A Practical Triage Guide
A flaky test is one that passes and fails on the same code. Left alone, flaky tests teach a team to ignore red builds, and that is far more expensive than the test itself.
Step 1: Prove it is flaky
Rerun the failing test in isolation 50 to 100 times (pytest —count=100, go test -count=100, or a simple shell loop). If it never fails alone but fails in the full suite, the problem is shared state, not the test logic.
Step 2: Classify the cause
Most flakiness falls into a few buckets:
- Time: tests that depend on now(), sleeps, or timeouts tuned on a fast laptop. Inject a clock instead of sleeping.
- Ordering: tests that pass only when another test runs first and leaves data behind. Randomize test order to expose them.
- Concurrency: races between a background worker and the assertion. Wait on a condition, never on a fixed sleep.
- External services: real network calls, DNS, or third-party sandboxes. Replace them with fakes at the boundary.
- Unordered collections: asserting on the order of a set, map, or SQL query without ORDER BY.
Step 3: Quarantine, do not delete
Move the test to a quarantine job that still runs but does not block merges, and open a ticket with an owner and a deadline. Quarantine without an owner is just deletion with extra steps.
Step 4: Measure
Track the flake rate per test (failures that pass on retry divided by total runs). A weekly list of the top five offenders keeps the problem visible and small.
Interview angle
“How would you deal with a flaky CI pipeline?” is a common senior-engineer question. A strong answer walks through reproduce, classify, quarantine with ownership, and measure. If you want to rehearse answers like this out loud before a real interview, Linkjob AI is an interview practice assistant I work on (disclosure: I'm on the team).