skip to content
Replays

Riding Out a Transient API Outage

Gantry tells a momentary upstream blip from a real failure and waits it out instead of halting the run.

Gantry milestones

0 milestones 3 sprints

Gantry tells a momentary upstream blip from a real failure and waits it out instead of halting the run.

39m 37s total 9m 7s per sprint

20.8M tokens in 127k tokens out

$20.26 nominal cost

A Gantry build can run for hours, and every agent it spawns reaches the model through a single runner. This job gave that runner a way to distinguish a retryable upstream outage from an ordinary agent failure, wait on a bounded backoff, and tell the operator why the run is paused.

How this walkthrough is structured

Feature

What did the run build and ship?

Gantry now has a transient outcome class, a configurable retry loop in the agent runner, and event-stream visibility in both front-ends.

Build

How did Gantry structure the work?

The run split a cross-harness reliability change at the runner boundary: classify first, retry second, and only then publish the waiting state to operators.

Transient Detection

0 agents 14m wall time 49.3k tokens out $9.06 nominal cost

The current tree still has the transient outcome in the runner and a shared harness helper for retryable signatures. The matcher is allow-list based, case-insensitive, and seeded from a captured overload transcript; it also rejects ordinary error text and preserves ceiling detection ahead of transient detection. Claude, Codex, Gemini, and OpenCode still classify through that harness layer, although later work expanded the helper to cover usage-limit text and shared oversized matching as well.

This first cut carried the classification surface without changing the run's behavior. A fresh agent could hold it because the brief kept the boundary at parsing: add the outcome class, feed captured stderr into every harness parser, and collapse the new result back to the old failure class before any caller could observe it. That made the risky part the text classifier, not the control flow above it. The sprint stayed green, including an extra gate pass after review, so the next unit inherited a proven parser contract rather than a speculative one.

Retry Loop

0 agents 9m 7s wall time 36.9k tokens out $4.70 nominal cost

The runner still wraps a single spawn in a retry driver that consumes transient outcomes and returns only ordinary success or failure to callers. Static overload backoffs use the configured delay list, while later usage-limit waits reuse the same machinery with a separate hinted-wait budget. The policy remains configurable through the retry setting and the legacy environment variable, with empty configuration disabling retry and malformed values falling back with a warning. The removed build module shows some call-site code moved on, but the retry loop itself is still central.

With classification already landed, this sprint could be a contained swap inside the single agent runner. Its brief asked for the former collapse shim to become a retry loop with per-attempt timeout, abort-aware sleeping, injectable waits for tests, and a config schedule, while keeping the transient class invisible above the runner. That is a narrow control-flow job for a clean-context agent because the callers were explicitly out of scope. The sprint did not need a fix or retry pass, so the stress the brief named was absorbed by the runner tests and config parsing.

Retry Visibility

0 agents 8m 21s wall time 24.9k tokens out $4.25 nominal cost

The domain event for agent retrying still exists and is serialized through history and attach streams. Headless output renders an explicit retrying line, and the TUI folds the same event into retry and wait notes on active cards and recent-event views. Later work extended the event fields for usage-limit reset waits and absolute wake times, but the original overload backoff path remains the same event-stream concern rather than a ledger status. Public documentation still names the retry delay setting.

The last sprint was intentionally thinner: the behavior already existed, so the unit only had to make waiting observable. The brief used the event channel as the boundary, adding one retry event and wiring the build context to emit it while leaving utility one-shots quiet. Because the front-ends already consumed the same run event stream, this was a propagation task across history, headless output, TUI state, and docs rather than another change to retry semantics. It also stayed green, matching that lower-risk shape.