Development record
2026-10-16 — Refines dcn-2610162239308903, the load gate's queue (iss-2610161634043273), on the gap it accepted and on how far its no-cycle claim reaches. The gap is larger than that record says. Two waiters whose start and pid disagree, as after the process ids wrap, wait on neither, and across accounts no hold keeps them apart, so re-checks that fall in one moment, before either shows in the process table as running its command, both open, and their commands overlap. Under `--wait -- make preflight` the overlap becomes a stall: each preflight's own gate, the inner waiter that takes its running ancestor's place, still counts every other waiter running its command as ahead, so each waits on the other's outer waiter and both stand until their two-hour timeouts. It needs a pid wrap, two accounts and coinciding re-checks. The mitigation uses no shared file, which iss-2610140638074587 still rules out. The command form shows itself in the table as running its command from the moment it opens, by a short-lived child of the waiter (a /bin/sleep, its mark, stopped once the command has started), waits two seconds (WaitOpenPause) and reads the table once more before it starts its command. If another waiter is then running its command, opened within that pause and the start tolerance, and has the lower pid, this waiter stands back: it stops its mark, gives up its hold, re-draws its jitter and waits again, while the lower pid keeps going. Because each opener is marked from the moment it opens, two that open at the same instant each see the other, and exactly one runs. A waiter stands back at most three times (WaitBackoffCap) and then opens without the look, so no wait goes round for ever. The plain form starts no command and neither marks nor looks. Within one account nothing changes: the hold already admits one. What stays open: a mark that cannot be started, a table that cannot be read at the look, the cap, and a process start slower than the pause. Rejected: having only the lower pid look, since an opener cannot tell that it is the lower of a disagreeing pair; and closing the stall in the inner gate alone, which leaves two plain race runs overlapping. The earlier record's 'whatever mix of code' covers this code and the pid-only waiters built before the start-order change, and nothing else. A waiter still running the branch's first version, which ordered the queue by start alone, waits on a higher pid that started earlier and can deadlock against pid-only waiters, and no change here can break that, since that waiter never runs it. So when this lands, any waiter running that start-only build is stopped and requeued.
2026-10-16 — Refines dcn-2610162239308903, the load gate's queue (iss-2610161634043273), on the gap it accepted and on how far its no-cycle claim reaches. The gap is larger than that record says. Two waiters whose start and pid disagree, as after the process ids wrap, wait on neither, and across accounts no hold keeps them apart, so re-checks that fall in one moment, before either shows in the process table as running its command, both open, and their commands overlap. Under --wait -- make preflight the overlap becomes a stall: each preflight's own gate, the inner waiter that takes its running ancestor's place, still counts every other waiter running its command as ahead, so each waits on the other's outer waiter and both stand until their two-hour timeouts. It needs a pid wrap, two accounts and coinciding re-checks. The mitigation uses no shared file, which iss-2610140638074587 still rules out. The command form shows itself in the table as running its command from the moment it opens, by a short-lived child of the waiter (a /bin/sleep, its mark, stopped once the command has started), waits two seconds (WaitOpenPause) and reads the table once more before it starts its command. If another waiter is then running its command, opened within that pause and the start tolerance, and has the lower pid, this waiter stands back: it stops its mark, gives up its hold, re-draws its jitter and waits again, while the lower pid keeps going. Because each opener is marked from the moment it opens, two that open at the same instant each see the other, and exactly one runs. A waiter stands back at most three times (WaitBackoffCap) and then opens without the look, so no wait goes round for ever. The plain form starts no command and neither marks nor looks. Within one account nothing changes: the hold already admits one. What stays open: a mark that cannot be started, a table that cannot be read at the look, the cap, and a process start slower than the pause. Rejected: having only the lower pid look, since an opener cannot tell that it is the lower of a disagreeing pair; and closing the stall in the inner gate alone, which leaves two plain race runs overlapping. The earlier record's 'whatever mix of code' covers this code and the pid-only waiters built before the start-order change, and nothing else. A waiter still running the branch's first version, which ordered the queue by start alone, waits on a higher pid that started earlier and can deadlock against pid-only waiters, and no change here can break that, since that waiter never runs it. So when this lands, any waiter running that start-only build is stopped and requeued.
A further case stays open, named by the re-check review: the window the look uses (the pause plus the start tolerance) is measured from the look, not from the waiter's own re-check read. A waiter whose path from its re-check to its mark is slow under load can therefore discard a concurrent opener it did see, and both run; it is captured as iss-2610162306239284, with measuring the window from the re-check read as the fix.
Frontmatter
| id | dcn-2610162257477571 |
|---|---|
| type | decision |
| date | 2026-10-16 |
.abcd/work/decisions/dcn-2610162257477571-refines-dcn-2610162239308903-the-load.md
open on GitHub ↗ · commit history ↗
mentions5
- dcn-26101622393089032026-10-16 — The load gate's queue (iss-2610161634043273) counts a queued waiter as ahead only when it is ahead by start time and also has the lower pid. Rejected alternatives: start order alone, as the branch had it, deadlocks against waiters built from code that predates the fix, which wait on every lower pid; on the night of 2026-10-16 a young low-pid waiter and two older high-pid ones each saw the other ahead for hours. Reverting to pid order alone also meets the no-mutual-wait property, but brings back the starvation after a pid wrap that the issue fixed. Telling old waiters from new ones in the process table is not possible, since both run the same arguments, so any safe rule may only wait on lower pids. Within that limit, start order AND lower pid keeps the most of the fix, and it is the smallest change: queueOrder and its chained-tie grouping are unchanged, and the pid check is one condition on top. Every wait from either code now points at a lower pid, so no cycle of any length can form, whatever mix of code and whatever each waiter's own read of the table. The grouping no longer prevents cycles; it only fixes the order within a chain of ties. After a wrap, an early high-pid waiter is held only behind waiters that started before it. Two waiters whose start and pid disagree wait on neither. Within one account the flock-serialised hold keeps them from opening together. Across accounts there is no shared hold: only the settle jitter of 15 to 60 seconds and the re-check keep them apart, and a re-check that reads the table in the moment between the other's open and its command showing in the table can still open. That gap is narrow and rare, since it needs a pid wrap, two accounts and coinciding re-checks. A pair in pid order has no such gap. It is accepted rather than closed with a cross-account file, which iss-2610140638074587 rules out.
- iss-2610140638074587The load waiter's one-waiter-at-a-time hold, ~/.abcd.noindex/load-wait.json, lives in the caller's own home, so it is per account, not per machine as the help and the LOAD rule call it. A second macOS account on the same machine (a colleague joining this machine) runs its own waiter against its own home and never sees the first account's hold, so two accounts' gated preflights are admitted together whenever the load and race-run conditions are open at once, which is exactly the moment between a preflight's start and its race step. Sharing the hold file across accounts is not a fix: a file another account can write would let one account hold every other account's gate shut, which the security rule forbids.
- iss-2610161634043273The load gate's queue orders waiters by pid, so a waiter with a high pid starves once the machine's process ids wrap. machineload.QueueAhead counts another waiter as ahead when its pid is lower than this one's. After a reboot on 2026-10-16 the ids wrapped: waiters started between 16:07 and 17:26, with pids 50460 to 71218, all queued ahead of one started at 16:04 with pid 96274. That waiter could only wait out its timeout while lower pids kept arriving. The docs and the LOAD rule promise one waiter at a time in queue order.
- iss-2610162306239284The load gate's stand-back window (the open pause plus the start tolerance) is measured from the look, not from the waiter's own re-check read, so a waiter whose path from that read to starting its mark is slow under load (about 2.5 s) discards a concurrent lower-pid opener it did see, and both run their commands, which can bring back two preflights each waiting on the other's running waiter; no test pins the window's bound.
- itd-2609151138388536Every Decision Is Its Own Record, and the Log Stops Conflicting