Making a Flaky Test Suite Trustworthy Again

The first sprint carried only the runner, not any interpretation of flakiness. Its brief asked for suite selection, repeated execution, concurrency settings, focused filters, fallback runner behavior, and raw result persistence. That was a bounded command-line and fixture problem a fresh worker could test without running expensive real stress loops. The gate did not send it back, so the later report layer received a stable input format.

This sprint was sequenced after the runner because it needed raw run records to aggregate. The brief kept it to deterministic report generation over representative output, including both rich and fallback runner shapes. It did not need to diagnose a real flake, so the worker could focus on offender identity, representative failure text, snapshot naming, and non-clobbering writes. The cut held without repair.

This was the bridge from tools to decisions. The sprint had to run enough current samples to separate confirmed targets from historical leads, compare synced and non-synced checkouts, and write the limits into the repository. It passed its own gate, but the milestone review narrowed the claim: the full-suite evidence was smoke-level, while the driver-sprint class was confirmed. That boundary prevented the next milestone from treating a thin ranking as a repair queue.

The first remediation sprint deliberately changed no production behavior. Its brief asked for a larger baseline so the milestone would not chase plausible but unconfirmed timing or ordering issues. That made it a self-contained measurement task on top of the runner and report generator. It stayed green and gave the following fix sprints a ranked, current target list.

This sprint took the strongest confirmed offender and investigated one class rather than the whole suite. The brief required a root cause and regression without broad serialization, ignored tests, or weaker assertions. The run facts show this sprint started more than once, but each recorded stage outcome was green. Its output was therefore useful but incomplete: it fixed one prompt-matching bug while later evidence showed a second fixture-isolation cause remained.

This sprint was a guided sweep, not a static audit of every suspicious call. The brief limited it to shared process-state hazards confirmed or implicated by the driver-sprint work. Its gate errored once before the sprint finished, locating stress at the boundary where process-global state and test isolation meet. The bounded scope kept the worker from serializing the suite as a blanket answer.

This sprint took the confirmed overlay timing class after the shared-state work had narrowed the list. Its brief asked for deterministic readiness or causal ordering, not looser assertions. The boundary was small enough for one worker because it centered on the overlay recording tests and their process-log expectations. It finished without repair, so the later evidence sprint could evaluate it as a specific fixed class.

This sprint tested whether the fixes actually held. Its brief was explicitly about comparison and bounded claims, not more repair. The gate failed, a fix ran, and review forced a re-plan after the report showed the driver-sprint class was still confirmed. That is the job's central boundary event: evidence closed the first remediation pass by changing the remaining work.

This sprint was added by the re-plan, and its brief sharply limited the work. It was not asked to invent a new root cause unless a real residual reproduced; it had to prove and lock the same-tag isolation fix. The sprint started more than once but its gates stayed green. The boundary was verification plus a deterministic regression, not another broad investigation.

This was a propagation sprint over a proven pattern. The brief made it bounded: document and test the canonical helpers, then migrate fixtures already implicated by the milestone, without sweeping every temporary path in the repository. That made the work mechanical enough for a fresh worker and kept legacy safe patterns from becoming false blockers. It passed without a repair stage.

This sprint converted the new fixture vocabulary into a normal verification check. The brief kept the guardrail heuristic and representative: catch known bad shapes and explain the accepted helper, without claiming to prove every possible flake. That boundary let the worker test good and bad examples instead of auditing the whole tree. The sprint stayed green.

This sprint closed the expanded remediation milestone by documenting the path a future contributor should follow. Its brief required bounded evidence language: resolved-and-guarded for the driver-sprint class, not a claim that flakes are permanently absent. It was a low-risk docs unit after the helper and guardrail code existed. The top-level milestone note then recorded that the planned prevention milestone had to reconsider its scope because this milestone had already built much of it.

This sprint was an audit against the current tree, not a rebuild. Its brief asked whether the guardrail, representative coverage, and reliability guide still agreed after the previous milestone's re-plan. That made the scope small and checkable for a fresh worker. It passed cleanly and tightened the guardrail around repeated re-enumeration log tags.

This sprint restored the job's data-before-edits rule after the audit. It sampled the guarded driver-sprint class, the overlay timing area, and the driver-loop finalization lead before any new fix claim. The sprint itself stayed green, but it ran a re-plan because the evidence identified one current residual signature. That re-plan narrowed the next sprint to the overlay compositor-fallback readiness failure.

The re-plan gave this sprint a single repair target: the overlay compositor-fallback recorder had to survive long enough to complete the ghostty cell-size probe. The brief ruled out ignoring the test, broad serialization, or longer timeouts as a disguise. Because the residual evidence already named the failure signature, the worker could focus on the readiness mechanism and regression. The sprint finished without a repair pass.

The last sprint was an evidence alignment task. It had to make the guide, analysis notes, and final classifications agree without overclaiming passing samples. That made it a documentation closeout over already-built code and measured snapshots. It stayed green, so the job ended with the measurement, remediation, and prevention story in one place.