skip to content

The map pattern

Map fan-out: enumerate a work list and run a per-task pipeline concurrently across a worker pool, with per-task repair, systemic thresholds, and checkpoint QA. Use for embarrassingly-parallel batch work over a spec-defined roster.

map — the fan-out built-in

A bounded write-spec repair loop, then a fan-out: a classify pre-scan, a per-task worker→gate body with per-task repair, a systemic rule block that quiesces on consecutive faults, and an advisory checkpoint side loop.

The careful.toml / steady.toml siblings are this pattern with a single systemic-threshold line changed — the acid test that the control-flow constants are data, not code.

map 
[header]
name = "map"
version = "1.1"
description = """
Map fan-out: enumerate a work list and run a per-task pipeline concurrently across a worker pool,
with per-task repair, systemic thresholds, and checkpoint QA. Use for embarrassingly-parallel batch
work over a spec-defined roster.
"""
entry = "main"

Completed fan-out artifacts stay uncommitted until finalization or partial merge, so map opts out of the resume-time dirty-worktree reset. Ledger DONE rows remain the resume ground truth; preserving the dirty tree keeps already-completed artifacts available to land.

map 
keeps_uncommitted_worktree_state = true

[[header.levels]]
label = "row"

The parameter interface

The roster source, plus two operator passthroughs the enumerator and the pool read.

map 
[[header.parameters]]
name = "map_spec"
kind = "file"
declared_plan_source = true
description = "The map spec or prose source used to produce the fan-out roster."

[[header.parameters]]
name = "worker_pool_width"
kind = "string"
default = ""
description = "Optional operator override for the number of worker slots the fan-out may drain concurrently."

[[header.parameters]]
name = "slice_args"
kind = "string"
default = ""
description = "Opaque arguments forwarded to the map roster enumerator."

The systemic fault streak: consecutive faulted units — a red gate or a terminal worker error, all folded to task-failed. It advances in completion order and resets on a clean unit, so an isolated failure between successes never trips the threshold.

map 
[[counters]]
name = "consecutive_faults"
reset = "consecutive"
increment_on = "task-failed"

main

Author and validate the spec, bounded by attempts, then fan out over the roster.

map 
[[blocks.main]]
type = "loop"
body = "write_spec"
rules = "spec_repair"

The fan-out: a classify pre-scan, a per-task worker→gate body with its own repair, a systemic rule block, three declared finalizers, and a cadence-driven advisory checkpoint side loop.

map 
[[blocks.main]]
type = "fan-out"
input = { kind = "generated-spec", spec = "$orchestration/map-spec.map", slice_args = "slice_args" }
classify = "classify"
body = "process"
systemic_rules = "systemic"
finalizers = ["abort", "systemic-quiesce", "partial-merge"]
side_loop = { body = "checkpoint", cadence = 5, sample_size = 1 }
pool_width = 4

write_spec

Write the spec from the declared source, then run the engine's preflight. A source that already is a spec is copied; prose is authored by a bounded agent fed the previous attempt's rejection diagnostic, so each re-authoring knows why the last spec failed.

map 
[[blocks.write_spec]]
type = "write-map-spec"
input = "map_spec"
produces = "$orchestration/map-spec.map"

[[blocks.write_spec]]
type = "validate-map-spec"
input = "map_spec"
spec = "$orchestration/map-spec.map"
slice_args = "slice_args"
if = "green"
then = "complete"

classify

Partition the roster into skip / heal / build and write that partition as the run's classification product. Engine work, not an agent's opinion: the ledger says what is already done, and the roster's declared per-unit health probe says which existing artifacts are stale.

map 
[[blocks.classify]]
type = "classify-units"

process

Per-task body: the worker — fall-through, since its artifact and the gate render the verdict — then the gate. The worker declares no static produces: map's artifact is per-unit, and completion is the run ledger's ground truth rather than a fixed file-contract path.

map 
[[blocks.process]]
type = "agent"
role = "build"
prompt = "execute"
generated_spec_worker = true
failure = "fall-through"

[[blocks.process]]
type = "gate"
if = "red"
then = "repair"

One repair attempt per task: the same worker again, then the final gate whose verdict feeds the fault streak.

map 
[[blocks.repair]]
type = "agent"
role = "build"
prompt = "execute"
generated_spec_worker = true
failure = "fall-through"

[[blocks.repair]]
type = "gate"

checkpoint

The side-loop body: a review whose failed-review default is declared, so a truncated review does not silently pass work.

map 
[[blocks.checkpoint]]
type = "agent"
role = "review"
prompt = "review"
failure = "degrade"
degrade_default = "retry"
produces = ["checkpoint.md"]

The rule blocks

Systemic: a streak of consecutive faults quiesces the fan-out into its wind-down. This threshold is the one line careful.toml and steady.toml change.

map 
[[rule_blocks.systemic]]
condition = { kind = "counter-threshold", counter = "consecutive_faults", threshold = 2 }
outcome = { kind = "quiesce" }

Spec-repair, first-match-wins: a validated spec completes the loop and falls through to the fan-out from the preflight's own green verdict; otherwise the third attempt gives up, and any earlier attempt keeps looping to re-author the spec with the last rejection's diagnostic.

map 
[[rule_blocks.spec_repair]]
condition = { kind = "attempt", number = 3 }
outcome = "failure"