Development record
The preflight load gate can starve every waiter on a busy machine and holds its only slot for a hung command. It opens only while the one-minute load is below twice the online cores (20 on 10 cores), but the preflight stages it does not gate (lint, record gates, evals, build, vet) run before it, so seven sessions' preflights on 2026-10-24 kept the load near 25 and the gate never opened: one drain lane's preflight ended 'load gate still shut after 2h0m0s: load 20.9 is not below 20 (2 x 10 online cores); 3 make preflights are running'. Later the same day, at 14:04 UTC, the load was 8 on 10 cores and about nine preflights were still queued, their logs unchanged for 45 minutes, because the one open slot (~/.abcd.noindex/load-wait.json, form command, opened 13:19 UTC, held to 14:19) belonged to a package test run that had used about 11% CPU for 45 minutes; load.preflights was 2, yet only one slot was open, possibly because waiters started before the setting was written read one slot when they began. Nothing reclaims a slot whose holder is idle, and nothing lets the gate open when the load it waits on comes from the ungated stages of its own queued runs. Adjacent records on an unmerged branch cover the ungated stages running unbounded and the unmeasured wait, which this issue does not restate.
issueresolvedmajoriss-2610241414124940
The preflight load gate can starve every waiter on a busy machine and holds its only slot for a hung command. It opens only while the one-minute load is below twice the online cores (20 on 10 cores), but the preflight stages it does not gate (lint, record gates, evals, build, vet) run before it, so seven sessions' preflights on 2026-10-24 kept the load near 25 and the gate never opened: one drain lane's preflight ended 'load gate still shut after 2h0m0s: load 20.9 is not below 20 (2 x 10 online cores); 3 make preflights are running'. Later the same day, at 14:04 UTC, the load was 8 on 10 cores and about nine preflights were still queued, their logs unchanged for 45 minutes, because the one open slot (~/.abcd.noindex/load-wait.json, form command, opened 13:19 UTC, held to 14:19) belonged to a package test run that had used about 11% CPU for 45 minutes; load.preflights was 2, yet only one slot was open, possibly because waiters started before the setting was written read one slot when they began. Nothing reclaims a slot whose holder is idle, and nothing lets the gate open when the load it waits on comes from the ungated stages of its own queued runs. Adjacent records on an unmerged branch cover the ungated stages running unbounded and the unmeasured wait, which this issue does not restate.
Further evidence 2026-10-24
- A records-only preflight timed out at two hours with the load at 19, below the threshold, so load was not the cause: the gate refused because another make preflight was running and six waiters were queued ahead. Under this load the test stages take 25 to 75 minutes, and only one preflight's test stages pass at a time even with load.preflights at 2, so with six or more waiters no run can meet a two-hour timeout.
- The test stage's
go test -p 4 -timeout 20m ./...bounds each test binary, not the stage, so one stage ran 73 minutes and more with no overall bound. - With several preflights' test stages admitted together at a load near 20, six packages hit their 20-minute per-binary timeouts in one run: admitting more than one at a time made all of them fail. The sessions worked around it by running whole preflights one at a time through the gate's command form (
abcd implement load --wait --timeout 8h -- make preflight).
Further evidence 2026-10-25
- A whole preflight run alone through the command form, admitted at a load of 8.5 on 10 cores after a 2h12m wait, still timed out three packages at 20 minutes:
internal/core/claim(43 s in an earlier passing run),internal/core/implement/loopandinternal/surface/cli(16 s earlier). The goroutine dumps show the tests waiting on child processes (git), not on a lock or a deadlock. - In the same window, the machine's endpoint security scanner used 100 to 210 per cent CPU, sampled every five seconds, while every test binary together used 9 to 38 per cent. The tests make many throwaway repositories under the temporary directory, and an on-access scanner inspects each write and exec. Load average does not show this cost, so the gate opens on a machine whose test throughput is a small fraction of normal.
- So the gate's signal (the one-minute load) is the wrong measure on a machine with on-access scanning. A remedy grounded in that: measure throughput, not load, for example by timing a fixed probe (one throwaway
git initand commit) before admitting a run, and say so when it is many times slower than the baseline; and ask the machine's administrator to exclude the Go build cache, the temporary directory and~/.abcd.noindex/from on-access scanning.
Frontmatter
| id | iss-2610241414124940 |
|---|---|
| type | issue |
| lifecycle | resolved |
| severity | major |
.abcd/work/issues/resolved/iss-2610241414124940-the-preflight-load-gate-can-starve-every.md
open on GitHub ↗ · commit history ↗
mentions5
- dcn-26102719225024802026-10-27 — Majors ruling for the next cut, product thinker, 27 October 2026, on abcd-78's recommendations for the 17 hand-captured majors since v0.15.1. Fix before the cut: the eight recommended fixes (iss-2610251555094324, iss-2610241537197997, iss-2610241414124940, iss-2610241911052191, iss-2610241240133009, iss-2610241133284600, iss-2610241239492159, iss-2610241519088123) and five recommended for deferral that the product thinker chose to fix instead: the short install command in the docs (iss-2610240827243654), a worktree per session (iss-2610240859590333), clearing lapsed holds everywhere (iss-2610241107191205), the reflect criteria (iss-2610241431221889) and the measured home-folder receipt (iss-2610241551259104). Deferred one cycle: iss-2610241228359748, iss-2610240932215918, iss-2610241613191748. Amended and closed: iss-2610241427397332 (itd-3's obsolete criteria). The 69 consistency-pass majors are ruled on branch decisions/interview-2026-10-27.
- dcn-26102907410943042026-10-29 — Declined part of iss-2610241414124940's remedy: once a run holds a slot, the gate does not switch to judging capacity by the slot count alone and keeps its load limit. Grounds, the timeout evidence of the two records: on 2026-10-24, with several preflights' test stages admitted together at a load near 20 on 10 cores, six packages hit their 20-minute per-binary timeouts in one run, so admitting more than one at a time made all of them fail (iss-2610241414124940); and the gate opened a whole run at load 17.8 on 10 cores, under its limit, after which three packages (claim, implement/loop, surface/cli) hit the 20-minute timeout because the load stayed near the threshold for the whole run (iss-2610241911052191). Admitting by slots past the load would admit more runs into exactly that load. The starvation that part answered came from queued runs' ungated stages feeding the load they waited on, which whole-run admission removes (itd-2610262127138979): a queued run now does no work. The idle-slot part of the same remedy is carried by iss-2610262155232329. The same record's 2026-10-25 evidence points the other way from slot-only admission too: a run admitted alone at load 8.5 on 10 cores still timed out three packages while the machine's on-access scanner used one to two cores, a cost the load average does not show, so the load is if anything an optimistic signal there; admission on measured headroom is the draft itd-2610262141001890.
- dcn-26103006183713212026-10-30 — itd-2610262127138979: when a look reads more than one slot, a first-gate command hold written by a run admitted at one slot is judged as a settle hold (in force only until its open plus the settle time), so raising load.preflights reaches a run already queued behind a held command, as iss-2610241414124940's resolution says; the running preflights are still counted against the slots from the process table. A disk refusal at the open look whose floor rose because the slot count rose says so, naming the new count, rather than saying the disk fell.
- iss-2610262155232329A load-gate slot held by a run that has stopped using the machine is never released: a holder that hangs or idles keeps its slot until its own timeout, and every other waiter waits behind it (part of the remedy of iss-2610241414124940, split out of itd-2610262127138979 at its planning on 2026-10-26, because whole-run admission does not decide it). Admitting a run on the headroom it will need rather than the load at one instant (part of iss-2610241911052191) moves with it to the adaptive count, itd-2610262141001890.
- itd-2610262127138979A preflight waits for its turn as one whole run