Taming Flaky Tests: A Practical Triage Guide

A flaky test is one that passes and fails on the same code. Left alone, flaky tests teach a team to ignore red builds, and that is far more expensive than the test itself.

Step 1: Prove it is flaky

Rerun the failing test in isolation 50 to 100 times (pytest —count=100, go test -count=100, or a simple shell loop). If it never fails alone but fails in the full suite, the problem is shared state, not the test logic.

Step 2: Classify the cause

Most flakiness falls into a few buckets:

Step 3: Quarantine, do not delete

Move the test to a quarantine job that still runs but does not block merges, and open a ticket with an owner and a deadline. Quarantine without an owner is just deletion with extra steps.

Step 4: Measure

Track the flake rate per test (failures that pass on retry divided by total runs). A weekly list of the top five offenders keeps the problem visible and small.

Interview angle

“How would you deal with a flaky CI pipeline?” is a common senior-engineer question. A strong answer walks through reproduce, classify, quarantine with ownership, and measure. If you want to rehearse answers like this out loud before a real interview, Linkjob AI is an interview practice assistant I work on (disclosure: I'm on the team).