Development record

When an agent or sub-agent loses its network connection mid-task, the work stalls or fails instead of waiting the outage out, and this keeps happening on the product thinker's machine. An agent that notices a lost connection should pause and retry on a widening schedule: wait one minute and try again, then five minutes, then ten, then an hour. Sibling agents should also learn of the outage without depending on each other over the network, since they may have lost the connection too: a channel on the machine itself, such as the run state and run log abcd already shares between sessions, would let one agent record the outage and the others hold off instead of each failing on its own.

issueresolvedmajoriss-2610080620372731

The product thinker's decisions (interview of 2026-10-09)

What it covers: every kind of lost connection, because which one strikes first is not yet known: the host's own model calls stalling, a sub-agent coming back failed mid-task, and a tool's network call (git push, gh, the merge-queue call, a download) failing. The first lab run records which kinds actually occur.
After the hour: retries continue hourly for up to 8 hours after the hourly stage begins, so an outage at 23:00 is still retried in the morning; then the run stops cleanly and reports what was done and what is left.
Who waits: lanes carry on with offline work (editing, tests); a lane that reaches a network step waits on one shared probe recorded in the run state and run log, instead of each lane retrying on its own.
A lane whose agent died: when the network returns, a fresh agent starts from the lane's last commit; uncommitted edits are saved aside for review, never built on.
Telling the product thinker: a notification only when the 8-hour limit is reached and the run stops; shorter outages appear in the end-of-run report with how long each lasted and what was retried.

Further decisions (2026-10-09, while building)

One outage, two services: the model service (Claude) being down is the same issue as the network being down. Each probe tests only what failed: the network by the git remote, the model service by a tiny canary agent the lead starts on the quickest, cheapest tier and reports with implement outage probe --model ok|fail.
A usage limit is not an outage: a usage or rate limit (a 429, "usage limit") is reported as it is and never waited out as a lost connection.
Q1, an agent cut off mid-task: it does not commit its edits. It stops and hands back with a final NETWORK: <cmd> line and no receipt; its uncommitted edits are saved aside for review through implement step --restart <lane> --yielded "NETWORK: <cmd>", and the fresh agent starts from the last commit.
Q2, a lead whose own turn died: every run arms a standing wake-up every 20 minutes for its whole duration, so a lead whose turn died to an outage still follows the schedule.
Q3, proving the model service back: only the canary agent proves it, never the lead's own turn running. The canary runs only when implement outage shows a probe is due (read next_probe_at first), never once per lane.

Proof belongs in a lab that cuts the network during a live run and shows each of the three kinds waited out, the shared probe holding the other lanes, a dead lane restarted from its last commit, and the give-up path ending in one notification.

Grounds

pursued: a run that loses its connection now waits on one shared probe instead of failing lane by lane; the live lab in iss-2610091937218987 would show it wrong if a lane fails, retries alone, or a give-up sends more than one notification

Frontmatter

id	iss-2610080620372731
type	issue
lifecycle	resolved
severity	major

.abcd/work/issues/resolved/iss-2610080620372731-when-an-agent-or-sub-agent-loses-its.md
open on GitHub ↗ · commit history ↗

mentions4

dcn-26101000000000022026-10-10 — Four points spc-2609301921521360 left open, settled by lane-1 of run-2610100550297632 in building it (itd-2609201925079472 criteria 7 and 8). (1) A runner's quota is counted in agent runs (`runner.Quota.Remaining`, reported through the optional `runner.QuotaReporter`), and the estimate from the spec's size is the least the run starts: per step to build, one implementer and one round of the two reviewers, and the fidelity audit once for an intent; fix rounds and syncs are not foreseen. Neither shipped runner reports a quota (the claude CLI and opencode expose none a launch can read before it starts), so today every run names its runners and skips the check out loud; the comparison is reached by a runner that reports one. (2) The rate-limit response is read from the claude CLI's own events, a `rate_limit_event` whose `rate_limit_info.status` is `rejected` or an assistant message whose `error` is `rate_limit`, on a run that did not finish; opencode's error events carry no rate-limit shape this build has verified, so an opencode limit stays a refusal and falls back as before. A rate limit is never fallen back on: every route spends the budget the window paces. (3) "Checkpointed to its branch" reads as the restart of 2026-10-09 reads a gone agent (iss-2610080620372731): the limited agent's uncommitted work and partial receipt are saved aside for review, never built on, its worktree reset to the branch's last commit, and its await dropped so the first step after the pause hands the same work to a fresh agent; a limited validator has only its partial return saved aside, since it edits nothing and may share the worktree with its round. Every other lane with work in flight is checkpointed at its branch's head in the record and left running, its agents' receipts still taken inside the pause, because their agents are not this process's to stop. (4) The pause after a rate limit is the run's own pause minutes, not the reset time a harness reports (decision 2 of the intent), and a pause already running is kept.
iss-2610080621450495Consider running abcd's autonomous runs as a collection of agents, a shape seen in an outside open-source agent tool: the person talks to one lead agent; the lead hands each task to a worker running as its own terminal session in its own git worktree; and a plain shell watcher, which costs no model tokens, sleeps on the workers and wakes the lead only when one finishes, fails or needs a decision. Today abcd's lanes run as sub-agents inside the host session, and the lead keeps checking on them itself (re-arming watchers on pull requests, polling queue state), which spends its context on waiting. Independent worker sessions would also contain a lost connection or a crash to one worker instead of the whole run, and state kept on disk lets every worker resume after a restart. The same tool can run workers on another machine over SSH, which is worth weighing later.
iss-2610091937218987The lost-connection fix of iss-2610080620372731 is proven by unit tests with an injected clock and prober, not yet by a live run: the record asks for a lab that cuts the network during a real autonomous run and shows each kind of loss (the host's own turn, a sub-agent, a tool's network call, and the model service) waited out on the one shared probe, the other lanes held only at network steps, a dead or yielded lane restarted from its last commit with its edits kept for review, and the 8-hour give-up ending in one notification. The product thinker ruled on 2026-10-09 that this lab runs after the next patch release is cut, since it switches off the machine's network for a few minutes.
itd-2611041549461012A session that takes over a run steps on from what abcd wrote down