Skip to content

GH-52024: [Python] Fix the codec list in show_info() - #51753

Open
Nishuuzz wants to merge 1 commit into
apache:mainfrom
Nishuuzz:minor-show-info-codecs
Open

Nishuuzz wants to merge 1 commit into
apache:mainfrom
Nishuuzz:minor-show-info-codecs

Conversation

@Nishuuzz

@Nishuuzz Nishuuzz commented Oct 5, 2026 •

Copy link
Copy Markdown
Contributor

Rationale for this change

pa.show_info() prints its Compression Codecs section from a hardcoded list:

codecs = ["brotli", "bz2", "gzip", "lz4_frame", "lz4", "snappy", "zstd"]

That list has drifted from _ensure_compression() in io.pxi, which is what
actually decides which codec names work. Two problems follow:

  • lz4 and lz4_frame both map to CCompressionType_LZ4_FRAME, so the same
    codec is reported twice under two names.
  • lz4_raw maps to CCompressionType_LZ4 and is a separate codec, but it is
    not listed at all, even though pa.Codec.is_available('lz4_raw') is True.

So seven lines are printed for six codecs, and the codec that is missing is a
real one.

Before:

Compression Codecs:
  brotli              : Enabled
  bz2                 : Enabled
  gzip                : Enabled
  lz4_frame           : Enabled
  lz4                 : Enabled
  snappy              : Enabled
  zstd                : Enabled

After:

Compression Codecs:
  brotli              : Enabled
  bz2                 : Enabled
  gzip                : Enabled
  lz4                 : Enabled
  lz4_raw             : Enabled
  snappy              : Enabled
  zstd                : Enabled

What changes are included in this PR?

One list in python/pyarrow/__init__.py: drop the duplicate lz4_frame and add
the missing lz4_raw. lz4 is kept as the name for the frame codec since that
is the one the docstrings lead with ('lz4' (or 'lz4_frame')).

On deriving this programmatically rather than hardcoding it — agreed, that is
the better fix. The canonical mapping lives in _ensure_compression() in
io.pxi, and neither Codec nor the C++ compression.h currently exposes the
set of supported codecs, so it needs a new accessor on the Cython side for
show_info() to read. I am happy to do that here, with the caveat that I cannot
compile the Cython extension in my environment and would be relying on CI to
verify it. If you would prefer, I can keep this change minimal and open a
follow-up issue for the programmatic version. Let me know which you would rather
have.

Are these changes tested?

Not by a new test. show_info() has no existing test coverage and its output is
a diagnostic print, so I did not want to pin the exact text without knowing
whether you would want that — happy to add one if you do.

Verified by applying the same change to an installed pyarrow and reading the
output back, and by walking every name _ensure_compression() accepts to
confirm which of them map onto the same C++ enum value:

bz2 -> BZ2, gzip -> GZIP, brotli -> BROTLI,
lz4 -> LZ4_FRAME, lz4_frame -> LZ4_FRAME,   # same codec, two names
lz4_raw -> LZ4,                             # distinct, was not listed
snappy -> SNAPPY, zstd -> ZSTD

Are there any user-facing changes?

Only the output of pa.show_info(), which now lists each supported codec once
and includes lz4_raw. No API change.

Was AI used for this PR?

In accordance to the AI generation guidelines, please disclose below whether and how AI was used in this PR.

PR code and description written by:

  • Human
  • AI

Reviewed before submission by:

  • Human
  • AI
  • Not reviewed

@raulcd raulcd left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the PR. A couple of things, could you use our PR template instead of removing the content? Could you also create an issue for this? Unfortunately this does not fit our MINOR definition as it changes a code file, not only docs and we require an issue.
Thanks.
As for the fix, I am wondering whether we could make this list being retrieved from the specific supported codecs programmatically instead of having a hardcoded list to avoid it from drifting in the future.

show_info() prints its Compression Codecs section from a hardcoded list
that has drifted from the names _ensure_compression() actually accepts.

"lz4" and "lz4_frame" both map to CCompressionType_LZ4_FRAME, so the
same codec was reported twice under two names, while "lz4_raw", which
maps to CCompressionType_LZ4 and is a separate codec, was not reported
at all. Seven lines were printed for six codecs, and the seventh codec
pyarrow supports was missing.

List "lz4" once and add "lz4_raw".
@Nishuuzz
Nishuuzz force-pushed the minor-show-info-codecs branch from e22f43e to 95365e4 Compare October 7, 2026 10:47
@Nishuuzz Nishuuzz changed the title MINOR: [Python] Fix the codec list in show_info() GH-52024: [Python] Fix the codec list in show_info() Oct 7, 2026
@Nishuuzz

Nishuuzz commented Oct 7, 2026

Copy link
Copy Markdown
Contributor Author

Both of those are done — filed #52024 for the bug and retitled this accordingly, and the description now follows the template.

The remaining question is yours: deriving the list programmatically rather than hardcoding it. I agree that is the better fix, and the obstacle is that the canonical mapping lives in _ensure_compression() in io.pxi and nothing currently exposes the supported set — neither Codec nor the C++ compression.h — so it needs a new accessor on the Cython side for show_info() to read. I am happy to write that here; the caveat is that I cannot compile the Cython extension in my environment, so I would be leaning on CI to verify it. Otherwise I can leave this as the minimal fix and open a separate issue for the programmatic version. Either is fine by me, whichever you prefer.

@github-actions github-actions Bot added awaiting change review Awaiting change review and removed awaiting changes Awaiting changes labels Oct 7, 2026
@github-actions

github-actions Bot commented Oct 7, 2026

Copy link
Copy Markdown

⚠️ GitHub issue #52024 has been automatically assigned in GitHub to PR creator.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants