Development record
junctionProbe's compile fallback turns stolenJunctions from a bounded candidate walk into a per-byte one, a latent cost cliff. junctionProbe (internal/adapter/scanner/scanner.go) builds one combined alternation over every pattern's boundary-free body; if that alternation fails to compile it falls back to regexp.MustCompile(`(?s).`), which matches at EVERY byte offset. The fallback was chosen so the candidate generator can over-produce but never under-produce — correct for detection, but it changes stolenJunctions' cost class: instead of one wholeMatch validation per real candidate junction in the backtrack window, wholeMatch runs once per byte offset in the window, and each of those validations is itself proportional to the match's own length (it re-runs the anchored probe over line[m.start:cut]). Cost per match goes from O(window) validations to O(window x match length) work. A reviewer measured roughly a 500x slowdown substituting the fallback directly — 4.21s against 8.25ms on one 200KB match. Not reachable with the current bundled pattern set: every bundled body alternates cleanly, and triggering the compile failure needs a pathological custom .abcd/config/pii.json override whose added pattern makes the joined alternation invalid (the per-pattern regexes are each compiled and validated on merge, so this is hard but not provably impossible to reach). Latent rather than urgent. Options if it is ever worth closing: cap the fallback's candidate count the way the window caps the search, drop patterns from the alternation one at a time until it compiles rather than abandoning the whole set, or mark the scanner unavailable (fail-closed) when the combined probe cannot be built — the last is consistent with how the package already treats a config fault.
issueresolvedminoriss-191
junctionProbe's compile fallback turns stolenJunctions from a bounded candidate walk into a per-byte one, a latent cost cliff. junctionProbe (internal/adapter/scanner/scanner.go) builds one combined alternation over every pattern's boundary-free body; if that alternation fails to compile it falls back to regexp.MustCompile((?s).), which matches at EVERY byte offset. The fallback was chosen so the candidate generator can over-produce but never under-produce — correct for detection, but it changes stolenJunctions' cost class: instead of one wholeMatch validation per real candidate junction in the backtrack window, wholeMatch runs once per byte offset in the window, and each of those validations is itself proportional to the match's own length (it re-runs the anchored probe over line[m.start:cut]). Cost per match goes from O(window) validations to O(window x match length) work. A reviewer measured roughly a 500x slowdown substituting the fallback directly — 4.21s against 8.25ms on one 200KB match. Not reachable with the current bundled pattern set: every bundled body alternates cleanly, and triggering the compile failure needs a pathological custom .abcd/config/pii.json override whose added pattern makes the joined alternation invalid (the per-pattern regexes are each compiled and validated on merge, so this is hard but not provably impossible to reach). Latent rather than urgent. Options if it is ever worth closing: cap the fallback's candidate count the way the window caps the search, drop patterns from the alternation one at a time until it compiles rather than abandoning the whole set, or mark the scanner unavailable (fail-closed) when the combined probe cannot be built — the last is consistent with how the package already treats a config fault.
Grounds
- pursued: a custom pii.json pattern that breaks the alternation no longer turns the junction search into a per-byte walk, and a token such a scan recovered before is still recovered; a swallowed custom token going unrecovered, or the validation count climbing with the window again, would show it wrong
Frontmatter
| id | iss-191 |
|---|---|
| type | issue |
| lifecycle | resolved |
| severity | minor |
.abcd/work/issues/resolved/iss-191-junctionprobe-s-compile-fallback-turns.md
open on GitHub ↗ · commit history ↗
mentions4
- dcn-26080500000000112026-08-05 — iss-188 follow-up (two independent adversarial reviews of the fix above, both landing on the same defect): the bounded backward search in `internal/adapter/scanner/scanner.go` (`stolenJunctions`) could step OVER the junction it was looking for. `junctionProbe` is unanchored, so one of its hits can SPAN a real junction — begin before it and end after it — without beginning at it; the loop nevertheless resumed at that hit's END (`off += loc[1]`) even when the hit's own offset had just been REJECTED by `wholeMatch`, skipping every byte in between, the true junction among them. Nothing revisits a skipped range, so the search returned fewer cuts than it should and iss-188's own failure mode reopened through a narrower trigger: `"ghp_"+32×a+"AIza"+4×b+"ghp_"+36×c` reported one finding and left the second token's tail raw, and `"ghp_"+10×a+"AIza"+22×z+"sk-proj-"+40×i` — where the skipped-over decoy is itself a syntactically valid secret — left 48 raw bytes of an OpenAI project key through `Redact` with the fail-closed residual re-scan reporting the output clean. Fixed by always resuming at `cut + 1`, dropping the jump-to-hit-end fast path entirely: a rejected candidate can hide a real junction one byte later, and an accepted one is not worth a special case. This stays bounded for the reason the whole feature exists: the loop is capped at `maxAdjacencyBacktrack` (512) iterations per match regardless of how long the match is, and each iteration's search is capped at `maxAdjacencyBacktrack + maxAdjacencyProbeWindow` bytes by RE2's linear guarantee — so the per-match cost is a constant factor independent of the REST of the line, and the whole scan stays linear in line length rather than quadratic. Measured: doubling and quadrupling an adversarial 247KB line doubled and quadrupled the time (2.3s / 4.6s / 9.2s), which is the property that matters. Repro: `internal/adapter/scanner/adjacency_test.go`, `TestStolenJunctionSearchDoesNotSkipPastRejectedCandidate` and `TestStolenJunctionSearchSkipsPastValidDecoySecret`, both watched failing on pre-fix code for the claimed reason; cost guard `TestJunctionBacktrackIsBounded`, extended with `dense_rejected_candidates_in_backtrack_window` — every match's window packed with candidates that all fail validation, the worst case for advancing one byte at a time. Two further gaps the same reviews found were CAPTURED, not folded in: iss-190 (a recovered match longer than `maxAdjacencyProbeWindow` is truncated, and the misaligned artificial end breaks the chain that would recover a THIRD abutting token — the pre-existing iss-185 window trade-off reached through a new path, whose real fix is an adaptive window that risks reintroducing the unbounded per-match cost) and iss-191 (`junctionProbe`'s `(?s).` compile fallback would validate once per byte instead of once per candidate, unreachable with the bundled pattern set). Two comment claims were also corrected as inaccurate: cost is bounded independently of the rest of the line but IS proportional to the match's own length, since `wholeMatch` re-runs the probe over the prefix; and a window-exceeding recovery can miss a token ENTIRELY, not merely truncate it, when the pattern's required structural markers (`jwt_shaped`'s two `.` separators) both fall outside the window.
- dcn-26080700000000022026-08-07 — iss-191 re-scoped by maintainer ruling: the round-3 merge-gate
- dcn-26080800000000022026-08-08 — bug-hunt loop round 9 (state issue #197): a multi-angle sweep (5
- iss-195The scanner's rigid/open-ended heuristic sends every IPv4 and IPv6 match through stolenJunctions' backward search, a measurable cost regression on ordinary log content. stolenJunctions (internal/adapter/scanner/scanner.go) skips its bounded backward search only for patterns whose match cannot be one byte shorter; net_ipv4 and net_ipv6 (internal/adapter/scanner/network.go) are variable-length, so every address they match in ordinary content enters the search — no custom config, no compile-failure fallback needed (that latter, narrower mechanism remains iss-191's scope and still does not trigger on the bundled set, whose combined alternation compiles cleanly). Measured on identical content, current main (bddb6fa) vs the tree immediately before the iss-188 machinery (eb61acd), 100-line ~12KB documents, all identifiers reserved documentation values (RFC 3849/5737/7042): neutral content 1.0x; one MAC per line 3.6x slower (net_mac itself classifies rigid, but ipv6Re also matches MAC-shaped colon-hex and that overlapping match is open-ended); one compressed IPv6 per line 1.9x; one MD5-style SSH fingerprint per line 23x; one dense colon-hex known_hosts-style line 27x. Because Skip/SkipAt reserved-range filtering runs AFTER scanAllPatterns, reserved addresses that are never reported still pay the full search cost. The iss-188 review also demonstrated a correctness angle on this same path: duplicate hard_fail findings manufactured for a single compressed IPv6, reproduced on this repo's own tracked fixtures (internal/core/memory/memory_test.go, internal/urlguard/urlguard_test.go). Candidate directions: exempt network-class patterns from the backward search where a stolen-junction leak is not plausible for them, or short-circuit when the only patterns whose bodies can match in the backtrack window are network-class; whether the residual cost is then a documented trade-off is a maintainer call. Split out of iss-191 per maintainer ruling (2026-08-07); benchmark method and full table recorded here because the scratch artefacts under .abcd/.work.local/ are ephemeral.