Skip to content

Add FileIO.list_prefix - #3974

Open
rambleraptor wants to merge 4 commits into
apache:mainfrom
rambleraptor:fileio-list-prefix
Open

rambleraptor wants to merge 4 commits into
apache:mainfrom
rambleraptor:fileio-list-prefix

Conversation

@rambleraptor

@rambleraptor rambleraptor commented Sep 15, 2026

Copy link
Copy Markdown
Collaborator

Rationale for this change

We've had a lot of interest in #3361. I'm attempting to split that into some smaller PRs to make it easier for everyone.

This PR adds a list_prefix as a method on a SupportsPrefixOperations mixin. Thanks to @Fokko and @kevinjqliu for helping to point out the history around FileIO.

Are these changes tested?

Includes unit tests.

Are there any user-facing changes?

AI Disclosure: Claude did help here.

Recursively list files under a location, with size and modification time,
for the PyArrow and fsspec backends. Needed by maintenance actions such as
removing orphan files, which compare storage against table metadata, so
listed locations keep the scheme and authority recorded in that metadata.
@Fokko

Fokko commented Sep 17, 2026

Copy link
Copy Markdown
Contributor

Thanks for bringing this up, this is a very sensitive topic 😅 Iceberg is very opinionated about targeting object stores, rather than filesystems. In object stores, list operations (list-prefix) are both slow and expensive because they are paged responses. Therefore, we don't want to make them first-class citizens in the FileIO. The list operation is also missing from the reference implementation.

We could follow the same path there, where we have something like SupportPrefixOperation.

I can see PyIceberg being used to clean up small, low-volume orphan files. Ideally, I think you want to use something storage-specific, like storage-inventory.

rambleraptor and others added 2 commits September 17, 2026 19:44
Listing is slow and expensive on object stores, which is why the Java
reference implementation keeps listPrefix out of FileIO and exposes it
through the SupportsPrefixOperations extension instead.

Mirror that split: FileIO no longer carries a list_prefix stub, and
callers detect the capability with isinstance instead of catching
NotImplementedError.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@rambleraptor

Copy link
Copy Markdown
Collaborator Author

@Fokko thanks so much for bringing this up! This makes absolute sense and explains why Java has the SupportPrefixOperation. I agree that - at large scale - using an inventory product can help you identify orphaned files, but it's really tied to your underlying storage provider. I think there's still a place for PyIceberg to give you a quick CLI to perform some basic maintenance tasks.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants