Add FileIO.list_prefix - #3974
rambleraptor wants to merge 4 commits into
Conversation
Recursively list files under a location, with size and modification time, for the PyArrow and fsspec backends. Needed by maintenance actions such as removing orphan files, which compare storage against table metadata, so listed locations keep the scheme and authority recorded in that metadata.
|
Thanks for bringing this up, this is a very sensitive topic 😅 Iceberg is very opinionated about targeting object stores, rather than filesystems. In object stores, list operations (list-prefix) are both slow and expensive because they are paged responses. Therefore, we don't want to make them first-class citizens in the FileIO. The list operation is also missing from the reference implementation. We could follow the same path there, where we have something like SupportPrefixOperation. I can see PyIceberg being used to clean up small, low-volume orphan files. Ideally, I think you want to use something storage-specific, like storage-inventory. |
Listing is slow and expensive on object stores, which is why the Java reference implementation keeps listPrefix out of FileIO and exposes it through the SupportsPrefixOperations extension instead. Mirror that split: FileIO no longer carries a list_prefix stub, and callers detect the capability with isinstance instead of catching NotImplementedError. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
@Fokko thanks so much for bringing this up! This makes absolute sense and explains why Java has the |
Rationale for this change
We've had a lot of interest in #3361. I'm attempting to split that into some smaller PRs to make it easier for everyone.
This PR adds a
list_prefixas a method on aSupportsPrefixOperationsmixin. Thanks to @Fokko and @kevinjqliu for helping to point out the history around FileIO.Are these changes tested?
Includes unit tests.
Are there any user-facing changes?
AI Disclosure: Claude did help here.