linkjob-notes

A flaky test is one that passes and fails on the same code. Left alone, flaky tests teach a team to ignore red builds, and that is far more expensive than the test itself.

Step 1: Prove it is flaky

Rerun the failing test in isolation 50 to 100 times (pytest —count=100, go test -count=100, or a simple shell loop). If it never fails alone but fails in the full suite, the problem is shared state, not the test logic.

Step 2: Classify the cause

Most flakiness falls into a few buckets:

  • Time: tests that depend on now(), sleeps, or timeouts tuned on a fast laptop. Inject a clock instead of sleeping.
  • Ordering: tests that pass only when another test runs first and leaves data behind. Randomize test order to expose them.
  • Concurrency: races between a background worker and the assertion. Wait on a condition, never on a fixed sleep.
  • External services: real network calls, DNS, or third-party sandboxes. Replace them with fakes at the boundary.
  • Unordered collections: asserting on the order of a set, map, or SQL query without ORDER BY.

Step 3: Quarantine, do not delete

Move the test to a quarantine job that still runs but does not block merges, and open a ticket with an owner and a deadline. Quarantine without an owner is just deletion with extra steps.

Step 4: Measure

Track the flake rate per test (failures that pass on retry divided by total runs). A weekly list of the top five offenders keeps the problem visible and small.

Interview angle

“How would you deal with a flaky CI pipeline?” is a common senior-engineer question. A strong answer walks through reproduce, classify, quarantine with ownership, and measure. If you want to rehearse answers like this out loud before a real interview, Linkjob AI is an interview practice assistant I work on (disclosure: I'm on the team).