React

Finding flaky tests before CI does

A flaky test shows up with shuffled order: the one that passes on the second attempt has a cause. A retry hides it instead of removing it.

Key points

  • A retry turns the pipeline green and leaves the bug: use it to measure, not to fix.
  • Shuffled order exposes shared state between files, the most common cause.
  • A flaky test that never reproduces locally depends on time, network or execution order.

A flaky test shows up when you repeat the suite in shuffled order: the one that passes on the second attempt has a cause, usually shared state, wall-clock time or the network.

The base rule

Vitest, under the MIT licence, has two options for this job: --sequence.shuffle changes file order and --retry repeats a failed test. The first exposes the problem, the second measures it.

The command that exposes them

npx vitest run --sequence.shuffle --retry=2 --reporter=verbose

When the suite is green only with retries, the attempt count is the size of the problem: on main the acceptable value is zero, because every retry is a test that sometimes lies.

The five causes

  • Shared state: an imported module that keeps values between files, visible only when the order changes.
  • Time: new Date() or Date.now() read inside the test or the component.
  • Randomness: Math.random(), crypto.randomUUID(), ids generated on mount.
  • Network and filesystem: a real call that answers sometimes.
  • Animation: transition and requestAnimationFrame that never finish in CI.

Repeating the suite

for i in 1 2 3 4 5; do
  npx vitest run --sequence.shuffle > "run-$i.log" 2>&1 || echo "fail $i"
done

Five runs with different seeds are enough to surface an order dependency. The seed belongs in the log: without it the failure never reproduces and the diagnosis starts over.

Live records in the catalogue

Three records that depend on time or animation: image-auto-slider, sonner and spinner. To browse by category start from 111,985 components.

The limitation

Shuffled order never catches tests that depend on the time of day or on a slow external service, because those fail only inside certain windows. A retry in CI with no report also hides the count: the flake survives and nobody sees it.

Where to continue

The test model is in Testing Library: the five tests; the double mount that confuses tests is in StrictMode; the render profile is in React Profiler; the measured library comparison is in React libraries 2026.

Frequently asked questions

Does a retry in CI fix the problem?

No, it hides it. It measures how often the failure happens; it does not remove the cause.

How is a flaky test reproduced?

With the same order seed and many repeats: without the seed the failure never returns and the diagnosis stalls.

Why does it fail only in CI?

Because CI is slower and sits in another time zone, so fixed waits and local dates give way there and never locally.

The test depends on order: what do I check?

Module-level state: an array, a cache or a counter living outside the test function.

Delete the test or fix it?

Fix it. Deleting it removes the coverage and leaves the bug the test had found.