Development record
On the v0.13.1 release PR's merge-queue run, the macOS race lane failed TestHookSubagentStopConcurrentCompletionsAllStage: 12 simultaneous subagent-stop hooks all exited 0 but only 10 transcripts were staged (internal/surface/cli/hook_subagent_stop_test.go:336). The same suite passed on the PR's own run and on every queue run earlier that day, so the loss appears only under a slow, race-instrumented host, but it is a silent loss: the hook reports success and the transcript never reaches the store. It is the defect iss-2609090828371674 resolved (the shared lock's backoff ceiling capping staging throughput), shown not to hold under load.
On the v0.13.1 release PR's merge-queue run, the macOS race lane failed TestHookSubagentStopConcurrentCompletionsAllStage: 12 simultaneous subagent-stop hooks all exited 0 but only 10 transcripts were staged (internal/surface/cli/hook_subagent_stop_test.go:336). The same suite passed on the PR's own run and on every queue run earlier that day, so the loss appears only under a slow, race-instrumented host, but it is a silent loss: the hook reports success and the transcript never reaches the store. It is the defect iss-2609090828371674 resolved (the shared lock's backoff ceiling capping staging throughput), shown not to hold under load.
Deferral 2026-10-05
Deferred past v0.13.0: Found during the v0.13.1 cut in code that release does not touch (the staging path shipped before v0.13.0); v0.13.1 is a patch of unrelated fixes, and a correct fix needs the staging lock redesigned and proved under load, which is a change of its own for the next cycle.
Deferral 2026-10-07
Deferred past v0.13.1: Second deferral, made by the 2026-10-07 autonomous run cutting v0.13.2: that patch ships the hook-message fix and the change-guard hardening and touches no staging code. The loss has appeared only on a slow, race-instrumented macOS runner, so its cause is not yet established, and the remedy needs the staging lock reworked and proved under a deliberately slowed lock. That is an investigation of its own, owed before the next minor release; a third deferral should go to the product thinker.
Evidence 2026-10-08
Recurred in the merge queue: PR #869 (records only) was dropped at 06:59Z when its merge group's macOS check failed TestHookSubagentStopConcurrentCompletionsAllStage, staging 11 of 12 transcripts from 12 simultaneous completions (run 37736585477). Re-enqueued; it merged. The deferral past v0.13.1 lapsed at v0.13.2, and the v0.13.3 cut's findings gate did not trip on it because it predates that anchor.
Cause established 2026-10-18
Reproduced deterministically, so the cause no longer rests on the slow macOS runner alone. The staging lock is not it: since iss-2609090828371674 each agent stages behind its own lock, so the twelve completions of the burst never contend for one, and 25 rounds of the burst under -race here staged every transcript. Slowing all seventeen git calls the hook makes (gittest.SlowPipeGit) still stages, only slowly.
What drops a transcript is a git call that fails once rather than slowly. gitutil.RootCommit reads any git failure other than a timeout as "no root commit", so resolveSubagentStore takes the payload's cwd for a directory that is not a repository and falls back to the session note. A burst's first completions run before any sibling has written that note, so the hook prints "cannot resolve the repository for this sub-agent … staging nothing" on stderr, exits 0, and the transcript is gone. A git shim that fails the root-commit lookup once reproduces it: the hook exits 0 and nothing is staged. A loaded runner produces the same failure when a fork or a git process fails, and the burst test never printed the hooks' stderr, which is why the cause went unseen. The other path to the same silent loss is a stage that still fails, such as a lock held past its timeout, with the store known.
Grounds
- pursued: a burst of sub-agent completions on a loaded machine stages every transcript or names each it lost; a transcript missing from the store with no marker counting it would show this wrong
Frontmatter
| id | iss-2610051434156079 |
|---|---|
| type | issue |
| lifecycle | resolved |
| severity | major |
.abcd/work/issues/resolved/iss-2610051434156079-a-burst-of-subagent-stops-still-loses.md
open on GitHub ↗ · commit history ↗
mentions3
- dcn-26101816062230442026-10-18 — A sub-agent stage retries a transient failure and marks a final one, but queues no unresolved transcript (facilitator-on-delegation, iss-2610051434156079): the hook retries resolving the repository when the payload's cwd carries a .git entry but git gave no root commit, re-reading the session note between tries, since burst siblings write it as they resolve; it retries a stage refused by lock contention, waiting longer as the record's remedy asks; both within a bound well inside the host's hook timeout, saying each retry on stderr. A stage that still fails for a known store leaves a counted marker that abcd history staged reports, so the loss is visible to the person and not only in a debug log. No machine-level queue is added for a transcript whose repository cannot be resolved at all, because it would keep unredacted transcripts in a new location outside any store, and the retries remove the transient case the record names (grounds: the cause established on the record on 2026-10-18; gitutil.RepoShaped; the existing sub-agent gap marker's pattern; principles/loud-staging.md)
- iss-2609090828371674The shared file lock's backoff cap limits staging to roughly ten writers a second, so a burst of simultaneous sub-agent completions loses transcripts. The lock helper backs off exponentially to a hundred-millisecond ceiling, so the rate a contended lock admits is set by that ceiling and not by how short the critical section is. Measured here, sixteen simultaneous stages take 2.12 seconds, consistently across three runs. Extrapolating the same rate, a burst past roughly forty to fifty simultaneous completions begins exceeding the five-second staging lock timeout, and a stage that times out is refused: the transcript it carried is not written anywhere, which is the loss the capture work exists to prevent. Sessions that fan out widely are exactly the sessions whose delegated reasoning is most worth keeping, so the ceiling bites hardest where the value is highest. The condition is pre-existing in the locking helper rather than introduced by sub-agent capture, but nothing reached the lock concurrently before, so it was unreachable in practice until now. The remedies are design-shaped and should be chosen rather than assumed: raise the timeout, lower the backoff ceiling, sharded locks keyed per agent, or a lock-free append with reconciliation at drain.
- itd-2610110940216682Every change is checked in full once, on exactly what lands