Development record
The check job's race passes are set by their total work: the macOS pass keeps a steady 2.80 to 2.85 of its runner's three test slots busy and the ubuntu one about 3.4 of four, so no reordering shortens them, and their slowest packages (internal/core/implement/loop, internal/surface/cli, internal/core/claim, about 46% of the macOS pass) spend most of it in git processes. A first cut, a default-branch name per claim operation and a batched view (#1099, #1100), took one drain test from 1,107 git processes to 830 but the whole of the loop and claim packages only from 126,334 to 119,327 (5.5%), which CI could not tell from the difference between runners; a cut measured on one test overstates the package.
The check job's race passes are set by their total work: the macOS pass keeps a steady 2.80 to 2.85 of its runner's three test slots busy and the ubuntu one about 3.4 of four, so no reordering shortens them, and their slowest packages (internal/core/implement/loop, internal/surface/cli, internal/core/claim, about 46% of the macOS pass) spend most of it in git processes. A first cut, a default-branch name per claim operation and a batched view (#1099, #1100), took one drain test from 1,107 git processes to 830 but the whole of the loop and claim packages only from 126,334 to 119,327 (5.5%), which CI could not tell from the difference between runners; a cut measured on one test overstates the package.
On 2026-11-01 the macOS pass ran into the job's cap in a merge group and dropped two pull requests (#1326, #1322) with no test failing. The watchdog stopped it at its 1h20m deadline (the 85-minute cap less its 5-minute margin), run 41317526495. In that run internal/surface/cli took 1203s, internal/core/implement/loop 1511s and internal/adapter/scanner 1133s under the race detector, and internal/core/memory, internal/core/release and internal/core/repolint were still running when it was stopped. Since the pass already keeps its three slots busy, splitting it across separate jobs is the only change that shortens one job, besides cutting the work this record measures. #1326 merged after one re-arm. #1322 was re-queued too, then left the queue again over an unrelated conflict with main.
2026-11-03: in #1347's merge-group run 41534357202 the macOS race job reached the ci-watchdog's 1h20m deadline, which sent SIGQUIT to three test processes, and a timing-bound test failed under the load. #1347, a records-only change, left the queue four times that night (a cancelled job, then three timing failures), each attempt costing about 80 minutes.
2026-11-03: after #1372's speed cuts merged (11:34), #1355's merge-group macOS job still reached the ci-watchdog's 1h20m deadline (run 41664455317, started 15:37) and left the queue; #1364 and #1370 had done the same earlier that day. The race pass is still at the cap: more work must come out of the slowest packages (internal/core/implement/loop, internal/surface/cli), or the macOS pass split across jobs, a ruleset change for the person.
2026-11-03, measured across merge-group runs with the same work and the same restored cache: run 41735871600 took 34 minutes (concurrency 4.7, implement/loop 1132s), runs 41706118055 and 41693198677 reached the 1h20m deadline (concurrency 2.8, implement/loop 2232s and 1891s). Hosted macOS runners vary about 2x, so the cap cannot be met reliably by cutting tests alone. The queue's check_response_timeout_minutes of 90 sets the cap.
Frontmatter
| id | iss-2610181757540736 |
|---|---|
| type | issue |
| lifecycle | open |
| severity | minor |
.abcd/work/issues/open/iss-2610181757540736-the-check-job-s-race-passes-are-set-by.md
open on GitHub ↗ · commit history ↗
mentions4
- dcn-26102820362195032026-10-28 — facilitator-on-delegation: the coordinating session (abcd-f5), on delegation from the technical facilitator, raises the check job's macOS leg ceiling in ci.yml from 70 to 85 minutes and moves its warning line from 60 to 75, keeping it ten minutes ahead; the ubuntu leg stays at 45. On 2026-10-28 merge-group run 40752008734 (PR #1196) had check (macos-latest) cancelled after about 71 minutes, from 19:11:30Z to 20:22:05Z, and the queue dropped the group with failed_checks, while that day's passing merge-group runs took roughly 46 to 68 minutes. 85 stays below the merge queue's 90-minute check response timeout. The lasting remedy is still a faster suite: iss-2610181711260494 named it and was resolved by the raise to 70, and the faster race pass stays open as iss-2610181757540736; this resolves iss-2610282035340363.
- iss-2610180545134593The implement loop package's tests are bound by git subprocesses: the slowest, TestPollingWithoutAnAnswerNeverHoldsTheIssuePastOneLease, runs Drain 120 times (a 4-minute poll over four 2-hour leases, kept inside the 5-minute take-over margin on purpose) and took 67s plain on a loaded machine, 31s on a quieter one with 3.84s of its own CPU; the next three take about 20s each, and the package is the slowest in both of the ubuntu leg's passes (320-410s plain, 470-650s race on 2026-10-18), with none of its 500 tests parallel.
- iss-2610282035340363The macOS leg of the check job in ci.yml cancels merge-group runs at its 70-minute ceiling: on 2026-10-28 run 40752008734 (PR #1196's merge group) had check (macos-latest) cancelled after about 71 minutes, from 19:11:30Z to 20:22:05Z, and the queue dropped the group with failed_checks. That day's passing merge-group runs took roughly 46 to 68 minutes, so the leg runs near its cap and each cancel rebuilds every group queued behind it. The lasting remedy is a faster suite (iss-2610181711260494, resolved by the raise to 70, which named a faster race pass, iss-2610181757540736, as the longer lever).
- iss-2611040337103775Re-reserving a record with a new footprint replaces the old footprint, with only a warning: on 2026-11-04 adding .github/workflows/ci.yml to iss-2610181757540736 dropped three directories other live work relied on, until a coordinator noticed the warning and reserved all four again.