skip to content
Replays

A Fast Parallel Resume Scan for Map Runs

Classify every already-done unit at once, up front, instead of two slow sequential sweeps.

Gantry milestones

0 milestones 3 sprints

Classify every already-done unit at once, up front, instead of two slow sequential sweeps.

37m 45s total 7m 45s per sprint

12M tokens in 115.6k tokens out

$16.15 nominal cost

Resuming a large map run has to re-check the filesystem rather than trust a stored flag: a unit is already done only when its artifact is present and its skip-test still passes. This job changed that resume path from repeated sequential scans into one parallel classification pass that runs before the worker pool and reports already-done units without briefly marking them as active work.

The digest split the work at the main dependency boundaries. The gate module first learned to classify a roster on its own, the driver then consumed that result while preserving checkpoint accounting, and the front-end pass checked that the changed event ordering rendered correctly.

How this walkthrough is structured

Feature

What did the run build and ship?

Map resume now derives done units through a host-width parallel scan, feeds only unfinished units to the pool, and renders direct Skipped events without in-flight flicker.

Build

How did Gantry structure the work?

The run isolated the pure classifier before the driver rewrite and left rendering verification for last, so each fresh agent received one boundary with an existing oracle or a narrow event-shape claim.

Parallel Classifier Design

0 agents 7m 45s wall time 28.8k tokens out $3.03 nominal cost

The tree still exposes `classify` in `src/engine/map_gate.rs`. It fans skip-test decisions over a bounded thread pool sized from host parallelism rather than build concurrency, returns a per-unit result, and records cancellation separately from a real verdict. `count_done_on_disk` now counts Skip results from that same classifier, so the partial-merge summary path no longer owns a separate sequential implementation. The module tests still cover mixed outcomes and parity with the old single-unit decision.

This sprint carried the foundation in the map gate module, away from the driver and the display surfaces. Its brief was self-contained: call the existing per-unit decision path in parallel, keep results in enumerated order, and preserve error and cancellation semantics. A fresh agent could prove the change against the sequential behavior because the old predicate still existed as an oracle. The sprint stayed green, which fits a cut where the hard part was making a pure replacement match its predecessor exactly.

Single-Pass Driver Integration

0 agents 14m 28s wall time 51.7k tokens out $6.53 nominal cost

`src/engine/build/driver_map.rs` now runs the pre-scan before worker admission, partitions Skip, work, and classification-error outcomes, and sends Skip units straight to their terminal finish path. The checkpoint state is seeded from the Skip set and each skipped artifact is recorded during the burst, preserving the cadence behavior that used to come from pool-streamed skips. Driver tests still assert that a resumed run starts only unfinished units, merges the final result, and records an unspawnable skip-test as a per-unit failure rather than a silent skip or whole-scan abort.

This was the load-bearing integration piece: it had to consume the classifier once, seed checkpoint state from that scan, emit already-done units, and admit only the unfinished remainder to the pool. The brief called out the subtle checkpoint boundary explicitly, because skipped units had previously flowed through pool recording and advanced completion windows there. Sequencing it after the classifier meant the agent did not have to prove parallel decision semantics and driver accounting at the same time. The gate and review stayed green, so no retry or re-plan changed the decomposition.

Clean Skipped Rendering

0 agents 3m 33s wall time 10.1k tokens out $2.50 nominal cost

The direct-to-Skipped behavior is now locked into the output surfaces. Headless prose tests assert the terminal Skipped lines appear without synthetic start output, JSONL tests show consumers can count skipped records from finished events alone, and the TUI state test keeps the in-flight count at zero while roster rows move straight to Skipped. In the current tree this contribution is mostly test coverage around existing render paths, which is exactly the scope the brief assigned.

The last sprint was intentionally a verification pass over the event consumers after the driver had changed ordering. Its unit of work was narrow: accept a terminal Skipped event for a known map unit without requiring a preceding start event. Because the roster already announces map units, the front-end agent could focus on folding one event shape through each surface rather than touching the driver again. The run facts show no red gate here, so the expected tolerance was either already present or made testable without broader repair.