Development record
The PEM block consumer redacts the common renderings of a key body but not every one, and one armour form is not detected at all. scanner.pemBodyLineRe accepts an optional gutter then a single base64 run, so a body rendered as log-prefixed lines (a timestamp/level prefix), as CSV cells, as one element per XML tag, with a trailing hash comment, as a source-string concatenation, as a JSON array on one line, or with double-escaped backslash-n separators is not body-shaped, the block ends at that line and the rest is stored verbatim; the tail of a block longer than maxPEMBodyLines survives by the same route. The RFC4716 armour (four dashes, BEGIN SSH2 ENCRYPTED PRIVATE KEY) does not match the bundled pem_private_key pattern at all, so no block is ever opened for it. Evidence: scanner.pemBodyLineRe and maxPEMBodyLines in internal/adapter/scanner/pem.go, DefaultPatterns pem_private_key in patterns.go. A fix must either widen the body-shape rule over these renderings or arm an entropy rule for a headerless base64 run (iss-96), and add the RFC4716 armour to the header pattern.
The PEM block consumer redacts the common renderings of a key body but not every one, and one armour form is not detected at all. scanner.pemBodyLineRe accepts an optional gutter then a single base64 run, so a body rendered as log-prefixed lines (a timestamp/level prefix), as CSV cells, as one element per XML tag, with a trailing hash comment, as a source-string concatenation, as a JSON array on one line, or with double-escaped backslash-n separators is not body-shaped, the block ends at that line and the rest is stored verbatim; the tail of a block longer than maxPEMBodyLines survives by the same route. The RFC4716 armour (four dashes, BEGIN SSH2 ENCRYPTED PRIVATE KEY) does not match the bundled pem_private_key pattern at all, so no block is ever opened for it. Evidence: scanner.pemBodyLineRe and maxPEMBodyLines in internal/adapter/scanner/pem.go, DefaultPatterns pem_private_key in patterns.go. A fix must either widen the body-shape rule over these renderings or arm an entropy rule for a headerless base64 run (iss-96), and add the RFC4716 armour to the header pattern.
Two further residuals, from the fix round that added the block consumer and its evidence rule. (a) The same-line body in the bundled pem_private_key pattern (scanner.pemPrivateKeyPattern in patterns.go) orders its alternatives so that a short final padding chunk is taken unconditionally only in a CLOSED block, where the END marker after it is the evidence; in the open alternative the short chunk must END the line. So a one-line key rendering with no END marker that is followed by prose keeps its trailing padding chunk of under sixteen base64 characters — "pasted <BEGIN header> <64 base64 bytes> QQQ= and then prose" stores "QQQ=" verbatim. A fix must decide that chunk on the run before it rather than on what follows it.
(b) In the other direction, the evidence rule that opens a block over-claims: scanner.pemBodyOpened treats ANY single run of sixteen or more base64-alphabet characters carrying at least one alphanumeric byte, on a body-shaped line within pemEvidenceWindow of a header, as proof that a key body opened. A header that is merely MENTIONED in prose therefore still opens a block whenever a long single-token line follows within two lines — a CamelCase identifier alone on a line, "CertificateRotationPolicy" for instance, inside a fenced snippet is enough — and the consumed run deletes those lines from a committed record, ending mid-fence and orphaning the closing fence marker. A fix needs a stronger opener test (a base64 run at a key body's characteristic width, or the entropy rule iss-96 tracks) rather than a bare length threshold.
Sub-residual (b) is closed; this record stays open for the rest. scanner.pemBodyOpened no longer reads a bare sixteen-character length as key material. One line is conclusive evidence only at a key body's characteristic width (scanner.pemOpenerRunRe, forty — under RFC 7468's sixty-four, OpenSSH's seventy and MIME's seventy-six, so no canonically rendered body line falls beneath the bar), and a block re-wrapped narrower than that opens through scanner.pemBodyRunPair instead: two consecutive body-shaped lines that each carry a run long enough to be key material, which prose that merely names a header does not show. So a mentioned header followed by one fenced identifier opens nothing and the record it sits in keeps its bytes and its fences, while every real-key rendering the three advisories bought still collapses to the placeholder — the pair rule is what carries the narrow-wrapped open block that the width bar alone would have written out verbatim. What is left of (b) is the part only an entropy rule can decide (iss-96): one identifier of forty or more base64-alphabet characters alone on a line inside the window, and two of sixteen or more back to back. Untouched and still open: the body-shape widening over log-prefixed, CSV, XML and JSON renderings, the tail of a block longer than maxPEMBodyLines, the RFC4716 armour the bundled pem_private_key pattern never matches, and residual (a).
Correction to the paragraph above, from the round that closed iss-2609090932441377. A characteristic width is not the only single-line route, and on its own it was the wrong axis: a block truncated by a log rotation or re-wrapped to a narrow column shows ONE body line under forty characters, with no second line for scanner.pemBodyRunPair to read, so the raised bar wrote that line out verbatim while the header pattern went on reporting a handled secret. A run of sixteen or more opens a block on a single line too, under two conditions that say nothing about width and that both have to hold: the header's marker is the whole content of its line (scanner.pemPastedHeaderRe — a PASTED header, as against one named inside a sentence, which goes on past the marker), and the run carries a byte of the base64 alphabet outside [A-Za-z] (scanner.pemNarrowKeyRun — a digit, a plus, a slash or the pad, which material drawn from all sixty-four symbols effectively always carries and a CamelCase identifier does not). What is left of (b) is still only an entropy rule's to decide (iss-96), with two shapes added to it from the other direction: a real body of sixteen to thirty-nine characters whose header carries prose after its marker, and one whose run happens to be letters only.
Grounds
- pursued: every rendering the record names now collapses to the placeholder with the prose around it intact; a named rendering that still stores a body line, or a mentioned header that now consumes prose, would show it wrong
Frontmatter
| id | iss-2609020127210042 |
|---|---|
| type | issue |
| lifecycle | resolved |
| severity | major |
.abcd/work/issues/resolved/iss-2609020127210042-the-pem-block-consumer-redacts-the.md
open on GitHub ↗ · commit history ↗
mentions2
- iss-96Now that transcripts are captured automatically on every session end, the scanner's secret-pattern coverage becomes load-bearing in a way it was not when capture was a manual verb nobody ran. Verified by live test: the bundled patterns DO catch anchored tokens (AKIA... access key IDs, ghp_/gho_/sk-ant- style prefixes) and absolute home paths, but they do NOT catch unanchored high-entropy values — an AWS SECRET access key (the 40-char base64 value, no prefix), a bare password, or a generic API token with no recognisable prefix all pass through into the store verbatim. This is the standard prefix-matching limitation and is pre-existing, not a regression; the point is that the blast radius changed. Consider entropy-based detection or the opt-in gitleaks adapter for the transcript path specifically, where the input is unstructured prose rather than curated source.
- iss-2609090932441377Raising the single-line PEM opener bar from sixteen characters to forty removed coverage that the previous bar had. An open block, one carrying no END marker, whose visible body is a single line between sixteen and thirty-nine characters followed by any non-body line is no longer redacted, where it was before. The reachable shape is a real one rather than a contrived one: a private key re-wrapped to a narrow column by a mail client or a terminal, or truncated by a log rotation, leaves a header and one short line, and MIIEowIBAAKCAQEA, the canonical opening bytes of a DER-encoded RSA key, then survives verbatim. The failure is silent in the worst way, because the stage-two rescan through BlockingResidual reports no findings and the header pattern still fires, so the store records that a PEM secret was handled while a fragment of the key remains in the text. The argument the change rested on, that no canonically rendered body line falls under forty characters because RFC 7468 mandates sixty-four and OpenSSH writes seventy, is true only of an untruncated and un-rewrapped block, and a truncated or narrow-rewrapped paste is precisely the shape a transcript redactor exists to catch. Three independent adversarial reviews found this, one grading it critical. The fix must restore the lost shape without restoring the defect the raised bar was closing, which was a mentioned header opening a block on any sixteen-character word: the pair rule already distinguishes a wrapped body from prose, so the single-line path needs a discriminator that is not width alone. Detector: a header followed by one body line of thirty-two base64 characters and then a prose line must redact that body line, the two-consecutive-narrow-lines case must keep redacting, and the mentioned-header-plus-fenced-identifier case must still be left untouched.