Development record
implement the galloping/exponential-doubling probe (trueMatchEnd) that replaces the fixed 512-byte adjacency window in scanAllPatterns/probeAt: the window grows only while a match keeps running into its edge and stops the moment the match ends short of the edge or reaches the real end of line, so a match end is never a truncation artifact. This is the structural fix designed in the 2026-08-08 decision [dcn-2608080000000001](../../../work/decisions/dcn-2608080000000001-iss-189-iss-190-shared-root-cause.md) after bug-hunt rounds 6-8 each BLOCKed on local patches for the same root cause (a truncated window-edge view driving a discard/skip decision reaching further than the ambiguity). It dissolves iss-189 (trailing boundary satisfied by the artificial window edge, spurious over-redaction) and iss-190 (window-capped recovery truncating a token and breaking the recovery chain) outright, without reintroducing the round-6 cost regression: the window grows only for matches genuinely still growing, at the amortized cost class the top-level unbounded match already pays. iss-189 and iss-190 (both superseded by this capture) are its acceptance corpus: their repro shapes must pass without a boundary classifier or a clipped-so-skip special case.
Press Release
Seeded by promotion from iss-229. Expand into the full press-release narrative before planning.
Why This Matters
Graduated from iss-229: implement the galloping/exponential-doubling probe (trueMatchEnd) that replaces the fixed 512-byte adjacency window in scanAllPatterns/probeAt: the window grows only while a match keeps running into its edge and stops the moment the match ends short of the edge or reaches the real end of line, so a match end is never a truncation artifact. This is the structural fix designed in the 2026-08-08 decision dcn-2608080000000001 after bug-hunt rounds 6-8 each BLOCKed on local patches for the same root cause (a truncated window-edge view driving a discard/skip decision reaching further than the ambiguity). It dissolves iss-189 (trailing boundary satisfied by the artificial window edge, spurious over-redaction) and iss-190 (window-capped recovery truncating a token and breaking the recovery chain) outright, without reintroducing the round-6 cost regression: the window grows only for matches genuinely still growing, at the amortized cost class the top-level unbounded match already pays. iss-189 and iss-190 (both superseded by this capture) are its acceptance corpus: their repro shapes must pass without a boundary classifier or a clipped-so-skip special case.. Read that issue record for the source observation.
Scope Conditions
None stated.
Acceptance Criteria
- Given a redactable match whose extent runs past the old fixed 512-byte adjacency window, when the scanner probes
trueMatchEnd, then the whole match is captured and its end is never a truncation artifact of the window edge. - Given a short match that ends well before 512 bytes, when the scanner runs, then its result and its cost are unchanged from the fixed-window behaviour.
- Given the iss-189 repro shape (a trailing boundary satisfied only by the artificial window edge), when the scanner runs, then it produces no spurious over-redaction, and does so without a boundary classifier.
- Given the iss-190 repro shape (a window-capped recovery that truncated a token), when the scanner runs, then the recovery chain stays intact, and does so without a clipped-so-skip special case.
- Given a match that is genuinely still growing at the window edge, when the probe extends, then the window doubles only while the match keeps running into its edge, staying within the amortized cost class the top-level unbounded match already pays.
Open Questions
None recorded yet.
Audit Notes
Fidelity review — receipt rcp-f317a716a8e4 (verifier intent-auditor claude-fable-5-1).
Provenance: intent-auditor@claude-fable-5-1 · rubric_hash sha256:effa65b3e9e88ff29433b443ec2be159522a8b0b71cf1434526514aa61edb13e · prompt_hash sha256:3a57d743083c8f2f18760747e08a2430559e49e4bd0cc334bb055e983c625285 Input attestations: diff:328a6755^1..328a6755 (PR #555), judged against the tree at bad1c73e@-;
Acceptance rollup: MET 4 · MET_WITH_CONCERNS 1 · NOT_MET 0 · INCONCLUSIVE 0
Per-criterion verdicts:
- ac-1 — MET_WITH_CONCERNS: gallopingFind doubles the window while the match runs into its edge and a 4096-byte recovered token is captured whole in the test; concern: a per-line growth budget (4len(line)+4096) exists, and once it is spent the probe keeps the fixed window it has, so a truncation artefact remains possible on a line engineered to grow the window at many junctions — a single long token always fits evidence: internal/adapter/scanner/scanner.go:572 — "func gallopingFind(re matcher, line string, at, base int, budget int) []int {" evidence: internal/adapter/scanner/scanner.go:584 — "keeps the fixed window it already has. See gallopBudget." evidence: internal/adapter/scanner/scanner.go:611 — "return 4len(line) + 8maxAdjacencyProbeWindow" evidence: internal/adapter/scanner/adjacency_test.go:458 — "func TestAdjacencyRecoveryCapturesALongTokenWhole"
- ac-2 — MET: the first attempt uses exactly the old 512-byte window, so a match ending before it returns on the first probe with the same result and cost; the pre-existing short-match and linearity tests pass at BASE evidence: internal/adapter/scanner/scanner.go:573 — "for w := maxAdjacencyProbeWindow; ; w *= 2 {" evidence: internal/adapter/scanner/scanner.go:524 — "const maxAdjacencyProbeWindow = 512" evidence: internal/adapter/scanner/adjacency_test.go:18 — "func TestConcatenatedSecretsBothDetected" evidence: internal/adapter/scanner/adjacency_test.go:260 — "func TestAdjacencyProbeStaysLinearOnLongLines"
- ac-3 — MET: the iss-189 repro places
.localexactly at the old window edge with the token continuing, and asserts no LAN-host finding; the stop conditions evaluate the boundary against the real line end and no classifier symbol exists in the package evidence: internal/adapter/scanner/adjacency_test.go:405 — "func TestAdjacencyProbeWindowEdgeIsNotAWordBoundary" evidence: internal/adapter/scanner/scanner.go:578 — "if hi == len(line) || loc == nil || at+loc[1] < hi {" - ac-4 — MET: the iss-190 repro (three abutting PATs with a 600-byte middle) asserts all three are found and redacted; stolenJunctions' forward reach is the same galloping probe with no clipped-so-skip branch evidence: internal/adapter/scanner/adjacency_test.go:425 — "func TestAdjacencyRecoveryChainSurvivesALongToken" evidence: internal/adapter/scanner/scanner.go:725 — "loc := gallopingFind(junctions, line, off, m.end, budget)"
- ac-5 — MET: the loop doubles only when the match reaches the edge with line remaining, a failing attempt never grows, and the cost test asserts the schedule is logarithmic in match length and the scan linear in line length evidence: internal/adapter/scanner/scanner.go:562 — "a FAILING attempt is what two bundled patterns'" evidence: internal/adapter/scanner/adjacency_test.go:532 — "t.Run("long_match_doubles_logarithmically", func(t *testing.T) {" evidence: internal/adapter/scanner/adjacency_test.go:631 — "func TestGallopingProbeCostIsLinearInLineLength"
Gap audit:
- honoured:
- one mechanism replaces the fixed window in both probeAt and stolenJunctions; no boundary classifier and no clipped-so-skip branch were introduced evidence: internal/adapter/scanner/scanner.go:652 — "m := gallopingFind(probes[j], line, at, at, &budget)" evidence: internal/adapter/scanner/scanner.go:725 — "loc := gallopingFind(junctions, line, off, m.end, budget)"
- the round-6 cost regression is guarded by tests that assert the doubling schedule and linearity directly evidence: internal/adapter/scanner/adjacency_test.go:503 — "func TestGallopingProbeCostClass" evidence: internal/adapter/scanner/adjacency_test.go:570 — "func TestGallopingProbeStaysBoundedOnLongLines"
- diverged:
- the promise 'a match end is never a truncation artefact' holds under a per-line growth budget; when it is exhausted the probe reverts to the fixed 512-byte window, so on an adversarial many-junction line the old truncation shape can recur (a deliberate resource-exhaustion trade-off recorded in code, not in the intent) evidence: internal/adapter/scanner/scanner.go:606 — "reverts to exactly the fixed-window behaviour, which is bounded and was never" evidence: internal/adapter/scanner/scanner.go:583 — "if *budget < hi-at {"
- missing: (none)
Frontmatter
| id | itd-155 |
|---|---|
| type | intent |
| lifecycle | shipped |
| kind | standalone |
| severity | minor |
.abcd/development/intents/shipped/itd-155-scanner-adjacency-galloping-probe.md
open on GitHub ↗ · commit history ↗
Referenced by2
- iss-229implement the galloping/exponential-doubling probe (trueMatchEnd) that replaces the fixed 512-byte adjacency window in scanAllPatterns/probeAt: the window grows only while a match keeps running into its edge and stops the moment the match ends short of the edge or reaches the real end of line, so a match end is never a truncation artifact. This is the structural fix designed in the 2026-08-08 decision [dcn-2608080000000001](../../decisions/dcn-2608080000000001-iss-189-iss-190-shared-root-cause.md) after bug-hunt rounds 6-8 each BLOCKed on local patches for the same root cause (a truncated window-edge view driving a discard/skip decision reaching further than the ambiguity). It dissolves iss-189 (trailing boundary satisfied by the artificial window edge, spurious over-redaction) and iss-190 (window-capped recovery truncating a token and breaking the recovery chain) outright, without reintroducing the round-6 cost regression: the window grows only for matches genuinely still growing, at the amortized cost class the top-level unbounded match already pays. iss-189 and iss-190 (both superseded by this capture) are its acceptance corpus: their repro shapes must pass without a boundary classifier or a clipped-so-skip special case.related
- spc-48scanner-adjacency-galloping-probe-structural-fiximplemented by
mentions4
- dcn-26080800000000012026-08-08 — iss-189/iss-190 shared root cause identified, fix designed but not
- iss-189adjacency probe's trailing \b can be satisfied by the window edge instead of real content, causing a spurious over-redaction. scanAllPatterns (scanner.go) slices the line to a bounded window (line[end:min(end+512,len(line))]) before running each pattern's adjacencyProbe, to keep one probe attempt's cost constant regardless of what follows (iss-185's fix). Some probes retain an internal trailing \b of their own (e.g. net_lan_hostname's `\b` at the very end of its alternation) which normally requires a real non-word character (or true end of string) after the match. But when the probe is truncated to the window, "end of window" is treated by the regex engine exactly like "end of string" — so if the window happens to end immediately after what LOOKS like a complete hostname/token, the trailing \b is satisfied by the artificial window boundary rather than by real surrounding content, even though the FULL, untruncated line would not have matched there (the real next byte continues the token). Concrete repro: a line built as `"AKIA"+16*"Q"` (a confirmed match, establishing the probe's junction) + 506 filler bytes + ".local" + "zzz" — the window (512 bytes from the junction) ends exactly after "local", so net_lan_hostname's probe matches "b...local" (satisfied by end-of-window) even though the real line continues with "zzz" right after ".local", meaning the FULL line's own un-windowed net_lan_hostname pattern would NOT match there at all (no real word boundary follows in the actual text). Impact: a spurious net:lan_hostname finding (warn severity only, not hard_fail) causes Redact to needlessly rewrite up to 512 bytes of legitimate content as fingerprinted/starred output — not a security leak (over-redaction, not under-redaction) but a correctness/usability defect: legitimate file content gets mangled on a contrived but reachable input shape. Suggested fix (from the reviewer that found this): when a probe match's end lands exactly at the window's limit AND limit < len(line) (i.e. the window was actually truncated, not naturally at the line's end), re-run that one probe against the full, untruncated line[end:] before accepting the match — paying the larger cost only in this rare edge case rather than on every probe. Found during a merge-gate correctness review of iss-185's PR (bugfix/iss-185-scanner-adjacent-secret-boundary-bypass); non-blocking (contrived input, warn-severity-only impact), so not fixed inline — captured for its own verify-and-fix cycle.
- iss-190scanner adjacency recovery is capped by maxAdjacencyProbeWindow, so a recovered token longer than the window is truncated — and the truncation can break the recovery CHAIN, hiding a third abutting token entirely. probeAt (scanAllPatterns, internal/adapter/scanner/scanner.go) runs its anchored probe against line[at:at+maxAdjacencyProbeWindow] (512 bytes), so a recovered open-ended match longer than that is recorded with an artificial end at the window edge. Two manifestations. (a) Known and accepted since iss-185: the recovered token's far tail past the window stays raw in Redact's output; the token's prefix is still masked, so this is a partial-redaction trade, not a total miss. (b) NEW, found by two independent adversarial reviews of iss-188's fix: because the artificial end does not align with the token's real end, the next iteration's probeAt(m.end) and stolenJunctions both start from a byte offset that is inside the token rather than at its junction, so the chain that would otherwise unwind a run of three or more abutting tokens breaks — the third token becomes entirely invisible, not merely truncated. Repro: line := "ghp_" + strings.Repeat("a",36) + "ghp_" + strings.Repeat("m",600) + "ghp_" + strings.Repeat("z",36) yields exactly 2 token:github_pat findings where an unbounded search would find 3; the third token's 36 raw bytes survive Redact and the fail-closed residual re-scan reports the output clean. A sharper variant of (a) affects jwt_shaped specifically: its regex requires two literal '.' separators, so a JWT-shaped token whose header and payload together exceed the window has NEITHER separator inside the probe's slice and the anchored probe finds no match at all — the token is missed entirely rather than truncated, e.g. a fixed-length match immediately followed by a JWT whose first two segments total more than ~502 bytes. This is NOT a defect introduced by iss-188's backward search; it is the same pre-existing maxAdjacencyProbeWindow trade-off from iss-185, now reachable through a second code path because the backward search feeds more candidates into probeAt. Fixing it properly needs a different mechanism (a growing or adaptive window, or re-probing chained from a recovered match's real end) that must not reintroduce the unbounded per-match cost the window exists to prevent — the exact risk iss-188's own ledger entry warned against. Deliberately not folded into iss-188's fix pass to keep that diff scoped to the confirmed junction-search defect.
- iss-2609230949465250itd-155 ac-1 narrowed: the galloping adjacency probe (internal/adapter/scanner/scanner.go gallopingFind) grows under a per-line budget of 4*len(line)+4096 bytes and, once that budget is spent, keeps the fixed 512-byte window it already has — so on a line engineered to grow the window at many junctions a match end can again be a truncation artefact of the window edge, the shape iss-189/iss-190 were about. A single long token always fits the budget. The trade-off (a measured quadratic cliff without the cap) is recorded in code only; the intent promises 'never a truncation artifact' unconditionally, so the promise or the record should say where the guarantee ends