Flaky tests: same commit, sometimes green, sometimes red Basic

The problem

CI goes red on the test “token expires after one second”. Rerun it and it is green, with no code change. The team adds three retries so the pipeline goes red less often.

A month later the test goes red exactly when there is a real bug in expiry handling, but everyone ignores it because “that test is flaky”. The biggest loss is not rerun time; it is that the CI signal loses its value.

A flaky test is a test that sometimes passes and sometimes fails without any deliberate change to code, input or expectations. A green rerun does not prove the code is correct; it only shows that the failing condition did not recur.

Classifying the causes

Martin Fowler lists five root groups: lack of isolation, asynchronous behavior, remote services, time, and resource leaks. Below is a more practical table:

Cause	Symptoms / how to detect	Fix
Real clock (Date.now, timezone)	Fails near second/day boundaries; fails only on CI in another timezone	Inject a Clock or use fake timers; measure durations with a monotonic clock (performance.now()), not Date.now()
Order dependency	Passes alone, fails when the whole suite runs	Shuffle the order to expose it (for example Jest --randomize --seed, shuffling test order within each file and printing the seed); each test builds its own data
Shared state (fixed IDs, shared DB, global variables)	Fails only in parallel or when parallelism changes	Per-test fixtures, isolated schema/DB, randomly generated IDs
Waiting for async work with sleep	Fails when CI is under heavy load	Poll for a specific condition with a deadline, or use a callback/event if available
Network, external services	Fails by time of day or region; connection errors/timeouts	Stubs/sandboxes in regular tests, keep a few controlled contract tests
Resource leaks (ports, files, connections, timers)	Later tests fail with “address in use” or run out of connections	Close in finally/teardown hooks, port 0, check for open handles
Unguaranteed ordering (query without ORDER BY, Map/Set iteration)	Correct results in the wrong order, randomly	Sort explicitly in code or in the assertion

Do not guess: collect evidence (commit, seed, runtime version, timezone, parallelism, logs) before picking a row of the table. Raising a timeout from 1 to 10 seconds when the cause is shared state only makes CI slower.

Sleep and an injected clock

An example TTL test: create a token that expires after 1000 ms, then check it has expired. Two ways of writing it have very different reliability:

A sleep-based test and an injected-clock test both checking a 1-second TTL

◀ Prev▶ PlayNext ▶↺ RestartStep 1/4

Both tests check a 1-second TTL. The top test uses the real clock; the bottom test sets now = 1000, which it controls itself.

The second approach is dependency injection at its simplest: the logic receives a Clock with now(). Production passes Date.now(); tests pass a fake clock and set its value themselves. The three points to check are before the deadline, at the deadline and after it, without sleeping a single second.

type Clock = { now(): number };
const expiresAt = (c: Clock, ttlMs: number) => c.now() + ttlMs;
const isExpired = (c: Clock, expiry: number) => c.now() >= expiry;

The Node test runner also has mock.timers built in: enable mock.timers.enable({ apis: ['Date'], now: 100 }) and then mock.timers.tick(200) to advance time. Note that this API does not support destructured imports such as import { setTimeout } from 'node:timers', and when you mock both Date and setTimeout they share one fake clock.

A fake clock only checks the expiry logic. Whether a real Redis TTL or setTimeout behaves correctly still needs a separate integration test, with a justified tolerance rather than millisecond-tight assertions.

Flaky test or product bug?

Not every intermittent failure is the test’s fault. A race in the product, a query without ORDER BY, a timeout set too low or a resource leak all make tests fluctuate precisely because the product fluctuates.

Loading diagram…

Quarantine and flake policy

Fowler recommends moving unstable tests into a quarantine area and fixing them quickly, so they do not destroy the main pipeline’s signal. A minimal policy:

1. Measure the flake rate: the proportion of runs on the same commit whose result changes (red, then green on rerun) out of total runs, per test.
2. Conditional quarantine: past the threshold the team sets, move the test to a separate suite that still runs but does not block merges.
3. Each quarantined test has an owner and a fix deadline (for example a few working days); past the deadline, delete or rewrite it rather than leaving it overdue indefinitely.
4. No blind retries in the main pipeline. If you must retry, record each retry as a flake event so it is still measured.

When to use / not use

Technique	Use when	Do not use when
Fake clock / fake timers	Time-dependent logic: TTL, expiry, backoff, scheduler	You need to prove an external system’s timer/TTL (Redis) works
Poll with a deadline	The contract really is eventual (email, queue, projection)	The contract promises a synchronous response: fix the product
Quarantine	The test is adding noise to the pipeline and needs investigation time	Used as a permanent graveyard for tests
Test retry	Temporary, measured, with a deadline	To hide a real race

Real-world example

Order confirmation email

An integration test calls the create-order API, then sleep(200) and checks a fake mailbox. When CI is under heavy load, the email arrives after more than 200 ms and the test goes red.

The team confirms the contract is asynchronous processing, so they replace sleep with polling the mailbox by orderId with a 2-second deadline.

Each test uses its own address so they do not collide, and on timeout the job log is printed. The email content mapping is tested separately, fast and deterministic.

If the worker really fails, the test still goes red after the deadline with concrete data; it is not “fixed” by infinite retries. (The 200 ms and 2-second numbers are illustrative.)

Run the clock demo:

npx --prefix examples ts-node examples/src/flaky-tests/clock-demo.ts

The real clock exact timestamp varies line changes every run because it uses Date.now(). The other two lines are stable: before boundary: false and at boundary: true.

Do not copy a timestamp from the output into an expected value, because that is exactly how a test becomes dependent on the current time.

To verify a fixed test, run it repeatedly, both alone and as part of the full suite in parallel, and track the flake rate before and after. One green run is not enough.

Trade-offs

Controlling dependencies

• A fake clock and per-test data make tests fast, repeatable and clearly bounded.
• Logs/seeds on failure support investigation instead of just rerunning.
• A more trustworthy CI: every red is an actionable signal.

Things to watch

• Too much mocking can miss real integration bugs.
• Isolated fixtures and integration environments add operational cost.
• Polling with a deadline is still environment-sensitive if the deadline is not based on the contract.

Common mistakes

Common pitfall

Retrying until green and calling it fixed. Retries hide real races. Record the failure frequency and find the source of nondeterminism.

Common pitfall

A fixed sleep to wait for async work. Fowler: never use bare sleeps; use callbacks or polling. If the contract is synchronous, fix the product.

Common pitfall

Using Date.now() to measure durations. The wall clock can jump (NTP sync, manual adjustment). Measure durations with performance.now().

Common pitfall

Mocking everything and declaring the system fine. A fake clock does not test Redis TTL or runtime timers. Keep integration tests at the important boundaries.

Production checklist

1. Every source of time, randomness and IDs goes through an injectable dependency.
2. Each test creates and cleans up its own data; no fixed IDs or shared DB between tests.
3. No fixed sleep for waiting; only polling with a deadline, or callbacks.
4. CI periodically runs with shuffled order and prints the seed on failure.
5. Track the flake rate per test; have a dashboard or a weekly report.
6. Have a quarantine suite with an owner and a fix deadline for each test.
7. Every failure stores logs, seed, timezone and parallelism as artifacts.

Summary

Key takeaways

• A flaky test changes its result without any deliberate change; it destroys the value of the CI signal.
• Look up the cause table (clock, ordering, shared state, async, network, resources) and use evidence, not guesses.
• For time logic, inject a clock and check the exact boundaries instead of sleeping and hoping.
• An intermittent failure can be a product bug; quarantine with an owner and a deadline instead of blind retries.

Self-check: (1) Why is sleep(1000) not a good way to test a one-second TTL? (2) A test that fails only in parallel points to which source? (3) When does polling match the contract, and when does it hide a bug? (4) Why use performance.now() instead of Date.now() to measure durations?

References

• Eradicating Non-Determinism in Tests — Martin Fowler
• Mocking timers — Node.js test runner
• Jest CLI: –randomize, –seed