tarfiles can't open tgz files with gzip features like FEXTRA & FCOMMENT when mode='r|*'

[balmeida-nokia]

Bug report

If a tar file is streamed from network and read into tarfiles as:
with tarfile.open(mode='r|*', fileobj=stream) as tfile

however, if mode is r:*. I.e. with tarfile.open(mode='r:*', fileobj=stream) as tfile
this problem disappears.

reading fails within:

        while c < size:
            # Skip underlying buffer to avoid unaligned double buffering.
            if self.buf:
                buf = self.buf
                self.buf = b""
            else:
                buf = self.fileobj.read(self.bufsize)
                if not buf:
                    break
            try:
                buf = self.cmp.decompress(buf)
            except self.exception as e:
                raise ReadError("invalid compressed data") from e
            t.append(buf)
            c += len(buf)

at:

buf = self.cmp.decompress(buf)

with exception:

zlib.error: Error -3 while decompressing data: invalid distance too far back

Stack trace:

_read, tarfile.py:548
read, tarfile.py:526
_init_read_gz, tarfile.py:491
__init__, tarfile.py:375
open, tarfile.py:1827
<module>, test.py:2

This appears to happen with tgz files created by the helm charts tool but I'm having a hard time reproducing without having any NDA data or metadata.

Your environment

• CPython versions tested on: 3.11.4
• Operating system and architecture: Windows 10 21H2

Linked PRs

• gh-107398: Fix tarfile stream mode exception when process the file with the gzip extra field #126304
• [3.15] gh-107398: Fix tarfile stream mode exception when process the file with the gzip extra field (GH-126304) #150199
• [3.14] gh-107398: Fix tarfile stream mode exception when process the file with the gzip extra field (GH-126304) #150200
• [3.13] gh-107398: Fix tarfile stream mode exception when process the file with the gzip extra field (GH-126304) #150201

[encukou]

This might be a symlink/hardlink issue: On filesystems that don't support those, extracting them requires seeking. Does your archive contain them?
(On a *nix system you can check with tar -tvf <filename> | grep ^[lh])

The | mode docs should mention this.

[balmeida-nokia]

Does your archive contain them?

No. That command returns status code 1.
I find it highly dubious it's symlinks because the issue is when decoding gzip to obtain the tar file.

[balmeida-nokia]

The most frustrating in this part is that I don't know how to create a helm gzip with just a few files without having a whole setup. It just fails with an error of missing this or that or something not in place, etc... Maybe I can find an example online situation changed. See comment under

[balmeida-nokia]

I can hardly believe I didn't notice how simple the solution was and I was looking and looking for it. With some tips from a colleague it's very simple to get a minimal example 🤦‍♂️

Here's the minimal reproducible code. You will need to have helm installed. In my case, I have v3.7.1. Otherwise, you can remove the system calls and just go with opening the tar file

I recommend running this code in a "sanbox-ish" directory. It creates a lot of files.

import os
import tarfile

os.system("helm create helmBreaksTarfile")
os.system("helm package helmBreaksTarfile")

with tarfile.open("helmBreaksTarfile-0.1.0.tgz", mode='r|*') as tfile:
    pass

Example file. I had to place it inside a zip so github would accept it. I didn't want to or let github modify the helm tgz in any way.
helmBreaksTarfile-0.1.0.zip

[encukou]

Ah, got it. Tarfile reads the gzip header itself and ignores the flags, so it does not skip FEXTRA and FCOMMENT data.
See also: #61881

That's a bug. Unfortunately I can't commit to fixing it :(