Map: Fanning Gantry's Pipeline Over a List
A job type that runs one fresh worker per enumerated item — one at a time, or several at once.
2 milestones 11 sprints 31 agents
216k peak context
175k median execute
3h 50m total 13m 10s per sprint
25m 36s plan 2h 16m execute 26m 41s review
140 edits 707 commands
claude harness
81.9M tokens in 709.8k tokens out
$86.47 nominal cost
This job gave Gantry a Map job type: a way to run the same checked task over an enumerated list, with each item handled by a fresh worker and judged by a per-item gate. The run first built the sequential engine so the generic spec, enumerator, gate, ledger, and resume contract could be proved without concurrent state. It then widened that path into a worker pool, added sampled checkpoint QA, and made the fan-out visible in both front ends.
How this walkthrough is structured
Feature
What did the run build and ship?
Gantry now has a Map driver that turns a spec and enumerator output into checked per-item work, with skip-heal resume, bounded concurrency, sampled QA, and fan-out reporting still present in the current tree.
Build
How did Gantry structure the work?
The decomposition put the generic, single-worker Map path under test before adding shared-state concurrency, so the later pool and checkpoint work could be measured against an already-working per-unit contract.
Milestone 1
Sequential Map Core
7 sprints 0 agents 1h 26m wall time 212k peak context $38.42 nominal cost
The sequential core still stands as the typed Map substrate. The current tree keeps `map_spec`, `map_unit`, `map_gate`, and the sequential smoke coverage, while the original build-driver edits have since been folded into the split build module. A Map spec names the workflow, units are derived from enumerator output, gates are process verdicts, and resume is grounded in artifact state rather than the old ledger alone.
This milestone carried the complete Map job without parallelism. That made the boundary small enough for each worker: define the spec, produce ordered units, gate an artifact, and only then assemble the driver that repeats that path. The ledger shows one retry in the enumeration sprint after a session-limit stop, and later replans adjusted remaining work without changing the milestone's scope. Its closing note also found a prompt-delivery constraint that the concurrency milestone had to handle for checkpoint QA.
Map Spec Format
0 agents 6m 53s wall time 78k peak context 23.9k tokens out $2.82 nominal cost
`src/engine/map_spec.rs` is still the Map spec parser. It reads the small config format, validates the required workflow fields, carries the gate command and skip-test variant, and preserves an optional checkpoint prompt with cadence. Later code now also uses the same model for prose-generated Map specs.
This sprint was asked to give the rest of the job a typed input model. It was a clean first cut for a fresh agent because it stopped at parsing and validation, with the checkpoint field preserved but not run. The run record shows no repair or retry here, which matches a boundary that was mostly format, error reporting, and tests.
Map Job Variant
0 agents 13m 5s wall time 134k peak context 42.5k tokens out $5.74 nominal cost
Map is now a selectable job route rather than an internal helper. The command layer still exposes Map-related verbs and the build job model carries Map payloads into `driver_map`. The exact surface has grown since this sprint, but the first-class route it introduced remains the path current Map runs use.
This unit connected the new spec to Gantry's public command surface. The brief kept the work narrow: add the job variant, select it from the CLI, parse the pool-width flag, and pass opaque slice arguments through without interpreting them. A replan ran after review, but the sprint itself stayed green and left the driver body for later units.
Enumeration And Fields
0 agents 3m 31s wall time 58k peak context 9.5k tokens out $1.90 nominal cost
`src/engine/map_unit.rs` still owns enumeration and template filling. It supports JSON rows and tab-delimited rows with a header, keeps the emitted order, resolves the artifact placeholder from the artifact template, and reports missing or malformed placeholders as errors. That module is still the pure input-to-unit layer used by the Map driver.
This was the deterministic center of the sequential plan: run the enumerator, parse rows, preserve order, and substitute fields into templates. It was also where the first milestone came under visible stress, with the initial worker stopped by a session limit and the sprint retried before landing. Because the work was isolated from agents and gates, the retry did not spill into later driver behavior.
Gate And Skip-Heal Logic
0 agents 6m 43s wall time 82k peak context 22.3k tokens out $2.58 nominal cost
`src/engine/map_gate.rs` still contains the per-unit gate and skip-heal decision. It treats exit status as the verdict, ignores gate chatter, distinguishes build gates from skip tests, and classifies existing artifacts before a worker is admitted. Current code has extended that module with parallel classification, but the original skip-heal contract is still the basis for resume.
This sprint took the unit model and added Gantry's ground-truth checks. Its brief was deliberately below the driver: invoke a substituted gate, invoke its skip-test form, and decide whether a unit should be built, skipped, or healed. The run facts show it passed without repair, which left the next sprint with a small decision API instead of raw process handling.
Sequential Driver Loop
0 agents 21m 50s wall time 212k peak context 80.7k tokens out $10.20 nominal cost
The sequential loop no longer lives in the old monolithic build file, but its behavior is visible in `src/engine/build/driver_map.rs`. A Map run enumerates units, announces a roster, renders worker preambles, builds or heals each item, records terminal status, and finalizes the run. The current pool code still preserves the single-worker path as the default behavior.
This was the assembly sprint for the sequential engine. The worker had to combine the already-built spec, enumerator, gate, repair, ledger, and event pieces while changing the failure boundary from build-style halt to Map-style log and continue. The run stayed green, which is useful evidence that the earlier cuts left this unit with orchestration rather than fresh parsing or gate semantics.
Systemic Failure Halt
0 agents 12m 12s wall time 122k peak context 45.4k tokens out $5.04 nominal cost
Systemic monitoring is still part of the Map driver path. The current driver lets local per-unit failure finish as a recorded item while a systemic fault quiesces the run instead of admitting the rest of the list. That distinction is now used inside the concurrent pool as well as the single-worker path.
The previous sprint made ordinary red units local, so this one restored the run-level stop condition. Its brief focused on classification: a single artifact failure continues, while auth faults, repeated timeouts, and growing blockers halt the run. A replan ran after the sprint, but no gate repair was needed, so the boundary appears to have held at the decision logic.
End-to-End Demonstration
0 agents 13m 10s wall time 175k peak context 46.7k tokens out $7.04 nominal cost
`src/engine/map_smoke.rs` still carries the money-free Map demonstration. Its tests drive a scratch Map run through enumeration, worker execution, gates, repair, skipped resume, healed artifacts, and systemic halt. Later concurrency tests were added to the same file, so the original sequential proof became the base fixture rather than a discarded test.
This capstone did not add a new feature surface so much as prove the sequential one as a whole. It was sized for one agent because the fixture could use a fake harness, stub enumerator, real gate scripts, and a scratch repository instead of live model calls. The run record shows a clean finish after the preceding systemic-halt replan, so the milestone closed with executable acceptance rather than prose assurance.
Milestone 2
Concurrent Fan-Out Monitoring
4 sprints 0 agents 1h 55m wall time 216k peak context $46.28 nominal cost
The current Map implementation is concurrent. `driver_map` runs a bounded pool, `driver_map/checkpoint.rs` owns sampled QA cadence, `headless.rs` and the TUI state project fan-out status, and `map_smoke.rs` covers concurrent parity and resume. The standalone checkpoint file named in the git facts no longer exists because checkpoint logic was folded under the split driver module.
This milestone was intentionally a delta over the sequential core. It widened the driver into a worker pool, activated checkpoint QA, projected fan-out state into the front ends, and then extended the same smoke fixture for concurrent acceptance. The worker-pool sprint is where the gate found a real defect and a fix ran, which fits the milestone's stated risk: shared state under multiple in-flight units.
Concurrency Worker Pool
0 agents 27m 3s wall time 176k peak context 93.7k tokens out $11.09 nominal cost
`src/engine/build/driver_map.rs` now maps units through a bounded worker pool. It clones the event sender for interleaved unit progress, relies on serialized ledger and journal writes, stops admitting work on abort or systemic halt, and keeps the single-worker path as the default. The pool consumes the same unit pipeline that the sequential milestone built.
This was the largest engine change in the job: keep the per-unit pipeline unchanged while letting several units run at once. The brief named shared-state correctness as the hard part, and the run record confirms it: the gate went red, a fix agent ran, and review triggered a replan for the remaining concurrency work. The failure sits exactly at the boundary this sprint owned, not in the Map spec or gate primitives it inherited.
Checkpoint QA Agent
0 agents 28m 30s wall time 216k peak context 112.4k tokens out $11.85 nominal cost
Checkpoint cadence and sampling live under the Map driver today. The code derives due windows from completed units, samples recent artifacts, marks windows as fired, and treats the end of a run as a boundary. The driver writes a separate checkpoint system prompt, so QA can run beside workers without changing worker preambles or recorded unit outcomes.
This unit used the optional checkpoint field that the spec parser already carried. Its separate challenge came from the previous milestone's note: workers shared one system-prompt file, while a concurrent checkpoint agent needed different prompt delivery. The sprint stayed green and was followed by a replan, so the remaining work could assume checkpoint events existed without making checkpoint QA a unit gate.
Concurrent Progress Monitor
0 agents 24m 14s wall time 208k peak context 79.2k tokens out $12.04 nominal cost
The fan-out view is still visible in both front ends. `RunEvent::MapStarted` tells consumers to switch projections, the TUI state counts in-flight, done, failed, and skipped units, and headless output tags interleaved unit records while keeping checkpoint review events distinct. Non-Map runs keep the older single-card projection.
The engine could already run interleaved units by this point, so this sprint stayed on the projection boundary. Its brief warned that checkpoint events are not worker units, which made the task about classifying the event stream correctly rather than adding more driver logic. The run stayed green and a replan followed, leaving the final demonstration to assert behavior on the real stream.
Parity Demo and Resume
0 agents 26m 33s wall time 204k peak context 89.6k tokens out $8.44 nominal cost
The smoke coverage now exercises concurrent Map behavior directly. It compares worker-pool outcomes with the single-worker path, observes simultaneous in-flight units, checks that checkpoint QA runs in the background, verifies systemic halt under concurrency, and tests kill-and-resume behavior. That test bed remains in the current tree as the executable claim that Map fan-out matches the intended conductor shape.
The final sprint was proof work over the whole concurrent design. It extended the existing money-free fixture instead of creating a separate harness, with assertions written around per-unit verdict sets and stream-derived skip status because concurrent completion order is not stable. The run facts show no repair here, so the prior pool, checkpoint, and projection cuts were sufficient for the capstone.