skip to content
Replays

Recovering a Worktree Left Dirty by an Interrupted Chunk

When gantry dies mid-chunk, resume discards the unverified work and rebuilds from the last committed green state.

Gantry sprints

0 milestones 3 sprints

When gantry dies mid-chunk, resume discards the unverified work and rebuilds from the last committed green state.

40m 35s total 12m 40s per sprint

16.5M tokens in 127.4k tokens out

$18.23 nominal cost

Gantry treats committed chunks as finished work and uncommitted edits as work still under test. This job handled the case where the owner process died in the middle of a chunk, leaving half-written project files in the run worktree while the last completed chunk was already committed.

The change gives resume a run-level cleanup path for that dirty state. It discards unverified project edits, lets the pending chunk be rebuilt from the last committed baseline, and writes recovery records so the reset appears in the run history instead of disappearing into a silent retry.

How this walkthrough is structured

Feature

What did the run build and ship?

Resume can now clean an interrupted commit-per-chunk worktree before the baseline gate, preserve run bookkeeping, and replay the recovery as durable history.

Build

How did Gantry structure the work?

The cut separated the git operation, the resume decision, and the audit records, with a review re-plan after the wiring clarified that observability had to handle the journal state around the reset.

Worktree Reset Primitive

0 agents 2m 20s wall time 8.9k tokens out $1.76 nominal cost

The worktree module still exposes the explicit whole-worktree reset primitive. It refuses to run beside an existing stash, runs the hard reset and clean, returns the head commit name, and surfaces git errors instead of swallowing them. Today the active resume path uses a related project-only reset that preserves Gantry bookkeeping, so this original primitive survives as an available operation rather than the one resume calls.

This first piece carried only the git mechanic: turn a dirty run worktree back into its committed head and report the short commit name. That was a bounded assignment for a fresh agent because it stayed inside the worktree module and could be proven with a throwaway repository dirtied through tracked edits, staged additions, and stray files. Its gate and review completed without repair, giving the next sprint a tested primitive instead of asking it to reason about git cleanup and resume timing at once.

Reset On Resume

0 agents 12m 40s wall time 43.6k tokens out $5.81 nominal cost

The build engine still has a run-level resume check that asks the selected driver whether interrupted-worktree reset applies. Commit-per-chunk drivers opt in, while map opts out because its uncommitted unit artifacts are resume state rather than disposable scratch. When the check fires, current code resets project files to head while preserving the orchestration directory, then lets normal resume rediscover the interrupted chunk as pending.

This sprint moved from mechanism to policy: a resumed run checks for dirtiness before any driver baseline gate runs. The brief kept the trigger simple, using worktree dirtiness rather than ledger forensics, and scoped the behavior to preexisting worktrees so fresh builds were left alone. The run stayed green here, but review added a re-plan before the final sprint, which marks this boundary as the place where making recovery happen exposed the need to account for what the reset would do to the run history.

Recovery Observability

0 agents 14m 47s wall time 51.5k tokens out $7.50 nominal cost

The history layer can still report stages that started without a matching finish, and the build engine closes those stages as interrupted during resume recovery. The reset itself is recorded through the existing support-stage event shape, naming the commit it reset to and replaying through the same journal path as other run events. Current tests cover the recovery journal, the reset record, clean resumes, fresh worktrees, and map resumes that must remain untouched.

The last sprint carried the audit side of the feature after the resume behavior was already in place. Its hard part was ordering: the orphaned stage had to be read from the journal before cleanup, while the closing record and reset record had to be written after cleanup so they became durable history. It passed without a repair loop, and the brief also bounded the map case explicitly so this recovery record was tied only to drivers where reset actually runs.