Development record
GHSA-9wv7 residual, now narrowed to its second half. The container half is CLOSED: the byte branch decodes the formats the standard library gives it — zip, gzip, bzip2, zlib, tar, and PNG's compressed chunks (IDAT, zTXt, a compressed iTXt, iCCP) — keyed on MAGIC BYTES rather than the filename, and scans what comes out with the same byte rules, so a token in a gzip member, a zip entry or a DEFLATEd zTXt chunk is a finding where it used to ship invisible, and a zip renamed .png is labelled and decoded as a zip. Decoding draws on one budget per payload file: 4 MiB of inflated output (the byte-scan read cap, so a file costs at most twice what it costs today), 4 nesting levels, 1024 entries. A container that would exceed any of the three is refused rather than decoded — a decompressor with no output bound would be a denial of service. The formats abcd still cannot read — JPEG entropy streams, PDF FlateDecode, GIF LZW, WebP, the mp4/mov/webm/mp3 boxes and frames, xz, 7z, zstd, and any unrecognised magic — stay content_unverified exactly as before, and an archive holding one opaque entry stays unverified naming that entry; the report carries the decoded tier, the unverified tier, the reason per unverified path and the magic-keyed format separately, so "decoded and clean" is never folded into one green with "could not be read". xz, 7z and zstd have no standard-library decoder: decoding them needs a third-party dependency, which is a sign-off, not an omission. The decoded tier claims the bytes were READ, so each walk proves that rather than assuming it: a zip's catalogued entries must tile the archive and its directory records must tile the region up to its end record, which must close the file (a prepended local record, a gap anywhere, a trailer, or a declared directory size inflated to swallow one, all refuse it); an entry is read to the end of its catalogued extent, since a DEFLATE stream can stop short of it; a tar must end where its end-of-archive marker says; a compressed stream's trailer is covered like the stream; every entry is read from its physical extent, whatever its mode bits, typeflag or directory-shaped name claim, because an archive reader hands back an empty body for an entry that calls itself a directory however much data it carries; a nested region is scanned raw exactly as a top-level file is, since "the caller already scanned these bytes" holds only at the top; every span whose length or content the format lets the file choose — a zip's two entry names and two extra fields, its entry and archive comments, a gzip header's FEXTRA, filename and comment, a tar header's name, linkname, uname, gname and every PAX record and extended attribute it carries, plus the zero padding between an entry and its block boundary, and every PNG chunk the spec calls plaintext — is judged rather than taken on the spec's word, because a payload is whatever its length field says it is: a field carrying the signature of any known container refuses to promote the file, naming the field and the format, and the signature search is held to the format sniff by a totality test so a format the sniff names can never lack a signature. That rule is deliberately blunt rather than decoding what it finds, because the decoding version shipped a hole twice — a member that would not open was read as a member that was not there — and because decoding from every signature offset was quadratic in the field's length. Its price is noise where a signature is short ASCII: an ordinary source path inside an archive trips it (internal/ftype.go reads as mp4, docs/ID3-tags.md as mp3, RIFF.md as webp, cmd/ustar/ as tar), as does a chance zlib pair in a palette at roughly 1% for a 192-byte field, and a JPEG thumbnail in a camera-export PNG's eXIf chunk trips it honestly — all reported unverified naming the field; and a region counts as covered by the byte scan only when ALL of it reads as text rather than its first 8 KiB. Where a walk cannot account for a byte the file stays unverified with the reason — the bytes may be innocent, but nothing read them. Three residuals, stated rather than hidden: a member encoded so that no signature abcd knows appears anywhere in it can sit inside a field and be counted as read, which is the same residual the top-level branch has always had; a member buried inside a PNG's inflated pixel data or its ICC profile is scanned as bytes but not walked, because asking a region rule to vouch for either would refuse every real image; and the short NUL-terminated sub-fields inside a compressed PNG text chunk are stepped over.
GHSA-9wv7 residual, now narrowed to its second half. The container half is CLOSED: the byte branch decodes the formats the standard library gives it — zip, gzip, bzip2, zlib, tar, and PNG's compressed chunks (IDAT, zTXt, a compressed iTXt, iCCP) — keyed on MAGIC BYTES rather than the filename, and scans what comes out with the same byte rules, so a token in a gzip member, a zip entry or a DEFLATEd zTXt chunk is a finding where it used to ship invisible, and a zip renamed .png is labelled and decoded as a zip. Decoding draws on one budget per payload file: 4 MiB of inflated output (the byte-scan read cap, so a file costs at most twice what it costs today), 4 nesting levels, 1024 entries. A container that would exceed any of the three is refused rather than decoded — a decompressor with no output bound would be a denial of service. The formats abcd still cannot read — JPEG entropy streams, PDF FlateDecode, GIF LZW, WebP, the mp4/mov/webm/mp3 boxes and frames, xz, 7z, zstd, and any unrecognised magic — stay content_unverified exactly as before, and an archive holding one opaque entry stays unverified naming that entry; the report carries the decoded tier, the unverified tier, the reason per unverified path and the magic-keyed format separately, so "decoded and clean" is never folded into one green with "could not be read". xz, 7z and zstd have no standard-library decoder: decoding them needs a third-party dependency, which is a sign-off, not an omission. The decoded tier claims the bytes were READ, so each walk proves that rather than assuming it: a zip's catalogued entries must tile the archive and its directory records must tile the region up to its end record, which must close the file (a prepended local record, a gap anywhere, a trailer, or a declared directory size inflated to swallow one, all refuse it); an entry is read to the end of its catalogued extent, since a DEFLATE stream can stop short of it; a tar must end where its end-of-archive marker says; a compressed stream's trailer is covered like the stream; every entry is read from its physical extent, whatever its mode bits, typeflag or directory-shaped name claim, because an archive reader hands back an empty body for an entry that calls itself a directory however much data it carries; a nested region is scanned raw exactly as a top-level file is, since "the caller already scanned these bytes" holds only at the top; every span whose length or content the format lets the file choose — a zip's two entry names and two extra fields, its entry and archive comments, a gzip header's FEXTRA, filename and comment, a tar header's name, linkname, uname, gname and every PAX record and extended attribute it carries, plus the zero padding between an entry and its block boundary, and every PNG chunk the spec calls plaintext — is judged rather than taken on the spec's word, because a payload is whatever its length field says it is: a field carrying the signature of any known container refuses to promote the file, naming the field and the format, and the signature search is held to the format sniff by a totality test so a format the sniff names can never lack a signature. That rule is deliberately blunt rather than decoding what it finds, because the decoding version shipped a hole twice — a member that would not open was read as a member that was not there — and because decoding from every signature offset was quadratic in the field's length. Its price is noise where a signature is short ASCII: an ordinary source path inside an archive trips it (internal/ftype.go reads as mp4, docs/ID3-tags.md as mp3, RIFF.md as webp, cmd/ustar/ as tar), as does a chance zlib pair in a palette at roughly 1% for a 192-byte field, and a JPEG thumbnail in a camera-export PNG's eXIf chunk trips it honestly — all reported unverified naming the field; and a region counts as covered by the byte scan only when ALL of it reads as text rather than its first 8 KiB. Where a walk cannot account for a byte the file stays unverified with the reason — the bytes may be innocent, but nothing read them. Three residuals, stated rather than hidden: a member encoded so that no signature abcd knows appears anywhere in it can sit inside a field and be counted as read, which is the same residual the top-level branch has always had; a member buried inside a PNG's inflated pixel data or its ICC profile is scanned as bytes but not walked, because asking a region rule to vouch for either would refuse every real image; and the short NUL-terminated sub-fields inside a compressed PNG text chunk are stepped over.
One residual in the same branch was carried out of this record rather than closed by it, and lives on iss-2609090934372160: a real_name whose literal is a short single token (7 bytes or fewer — the threshold counts bytes, not runes, so a 4-rune CJK name passes as long) is dropped on bytes as chance noise, so such a name in PDF /Author metadata is not a finding there while it is on text. It is a threshold in the byte-scan identity policy (byteScanPolicy's bytePolicyKeepLongLiteral), not a coverage question, and decoding a container does not touch it: a decoded region is scanned with the same byte rules, so a short name inside a zip entry is dropped exactly as it is in the raw bytes. Closing it needs a way to tell a short name from a chance collision — an anchored context such as a PDF /Author or /Creator field or an EXIF artist tag, rather than a bare literal anywhere in a blob — or an accepted false-positive rate on binary bytes. That scope is iss-2609090934372160's, not this record's: the container half named above is what this record claims, and it is done.
Grounds
- pursued: we expect a secret inside a container abcd can open to be found before the payload is published, and a container it cannot open to be named as unread rather than counted as clean, so the gate's own report distinguishes decoded from unverified; a token inside a gzip member or a zip entry reaching a published payload unreported, a renamed archive classified by its extension, or a decode that exhausts the host on a crafted file would show it wrong. The short single-token real_name residual this record also carried is split to iss-2609090934372160 and is not claimed here.
Frontmatter
| id | iss-2608291832160371 |
|---|---|
| type | issue |
| lifecycle | resolved |
| severity | major |
.abcd/work/issues/resolved/iss-2608291832160371-container-payload-files-are-not-content.md
open on GitHub ↗ · commit history ↗
mentions2
- iss-2609090934372160A real_name whose literal is a short single token is dropped by the byte scan as chance noise, so a banned name in binary metadata is not a finding there while the same name in text is. byteScanPolicy carries byteScanLongLiteral at eight, and the threshold counts bytes rather than runes, so a four-rune CJK name passes as long while a seven-byte Latin one does not. The consequence is a name in a PDF Author field or an EXIF artist tag reaching a published payload unreported. Decoding containers does not touch this: a decoded region is scanned with the same byte rules, so a short name inside a zip entry is dropped exactly as in raw bytes. The threshold exists to hold down a false-positive rate on binary bytes, where a short Latin token appears by chance, so simply lowering it trades one defect for another. Closing it needs an anchored context that makes a match meaningful rather than incidental, a PDF Author key or an EXIF artist tag read structurally, or an accepted false-positive rate stated as a decision. Detector: a banned short single-token name placed in a PDF Author field must be reported by the payload scan, and a byte-identical run over binary content carrying the same token incidentally must not flood the report. Split out of iss-2608291832160371, whose container half shipped separately; this half was never addressed by it.
- iss-2610280103493414Scanner reports a multi-member gzip as fully decoded while later members' header fields go unscanned