Scanner reports a multi-member gzip as fully decoded while later members' header fields go unscanned

Frontmatter

.abcd/work/issues/resolved/iss-2610280103493414-private-report.md
open on GitHub ↗ · commit history ↗

mentions1

• iss-2608291832160371GHSA-9wv7 residual, now narrowed to its second half. The container half is CLOSED: the byte branch decodes the formats the standard library gives it — zip, gzip, bzip2, zlib, tar, and PNG's compressed chunks (IDAT, zTXt, a compressed iTXt, iCCP) — keyed on MAGIC BYTES rather than the filename, and scans what comes out with the same byte rules, so a token in a gzip member, a zip entry or a DEFLATEd zTXt chunk is a finding where it used to ship invisible, and a zip renamed .png is labelled and decoded as a zip. Decoding draws on one budget per payload file: 4 MiB of inflated output (the byte-scan read cap, so a file costs at most twice what it costs today), 4 nesting levels, 1024 entries. A container that would exceed any of the three is refused rather than decoded — a decompressor with no output bound would be a denial of service. The formats abcd still cannot read — JPEG entropy streams, PDF FlateDecode, GIF LZW, WebP, the mp4/mov/webm/mp3 boxes and frames, xz, 7z, zstd, and any unrecognised magic — stay content_unverified exactly as before, and an archive holding one opaque entry stays unverified naming that entry; the report carries the decoded tier, the unverified tier, the reason per unverified path and the magic-keyed format separately, so "decoded and clean" is never folded into one green with "could not be read". xz, 7z and zstd have no standard-library decoder: decoding them needs a third-party dependency, which is a sign-off, not an omission. The decoded tier claims the bytes were READ, so each walk proves that rather than assuming it: a zip's catalogued entries must tile the archive and its directory records must tile the region up to its end record, which must close the file (a prepended local record, a gap anywhere, a trailer, or a declared directory size inflated to swallow one, all refuse it); an entry is read to the end of its catalogued extent, since a DEFLATE stream can stop short of it; a tar must end where its end-of-archive marker says; a compressed stream's trailer is covered like the stream; every entry is read from its physical extent, whatever its mode bits, typeflag or directory-shaped name claim, because an archive reader hands back an empty body for an entry that calls itself a directory however much data it carries; a nested region is scanned raw exactly as a top-level file is, since "the caller already scanned these bytes" holds only at the top; every span whose length or content the format lets the file choose — a zip's two entry names and two extra fields, its entry and archive comments, a gzip header's FEXTRA, filename and comment, a tar header's name, linkname, uname, gname and every PAX record and extended attribute it carries, plus the zero padding between an entry and its block boundary, and every PNG chunk the spec calls plaintext — is judged rather than taken on the spec's word, because a payload is whatever its length field says it is: a field carrying the signature of any known container refuses to promote the file, naming the field and the format, and the signature search is held to the format sniff by a totality test so a format the sniff names can never lack a signature. That rule is deliberately blunt rather than decoding what it finds, because the decoding version shipped a hole twice — a member that would not open was read as a member that was not there — and because decoding from every signature offset was quadratic in the field's length. Its price is noise where a signature is short ASCII: an ordinary source path inside an archive trips it (internal/ftype.go reads as mp4, docs/ID3-tags.md as mp3, RIFF.md as webp, cmd/ustar/ as tar), as does a chance zlib pair in a palette at roughly 1% for a 192-byte field, and a JPEG thumbnail in a camera-export PNG's eXIf chunk trips it honestly — all reported unverified naming the field; and a region counts as covered by the byte scan only when ALL of it reads as text rather than its first 8 KiB. Where a walk cannot account for a byte the file stays unverified with the reason — the bytes may be innocent, but nothing read them. Three residuals, stated rather than hidden: a member encoded so that no signature abcd knows appears anywhere in it can sit inside a field and be counted as read, which is the same residual the top-level branch has always had; a member buried inside a PNG's inflated pixel data or its ICC profile is scanned as bytes but not walked, because asking a region rule to vouch for either would refuse every real image; and the short NUL-terminated sub-fields inside a compressed PNG text chunk are stepped over.