Repository navigation
Conversation
This was referenced Oct 3, 2026
sundapeng
changed the title
sundapeng
changed the title
sundapeng
force-pushed
the
pypaimon-io-cache-routing
branch
4 times, most recently
from
October 4, 2026 08:01
ca21bcc to
72068f9
Compare
JingsongLi
marked this pull request as draft
October 4, 2026 08:23
sundapeng
force-pushed
the
pypaimon-io-cache-routing
branch
from
October 5, 2026 05:33
0dc1efe to
0af7167
Compare
When a REST table token vends io-cache endpoints, RESTTokenFileIO now
wraps the OSS FileIO so that immutable files are read through a cache
target chosen by file type, and every other request stays on the OSS
endpoint. The rules are the same as in the Java Jindo FileIO.
- io-cache.endpoint names one target. io-cache.targets with
io-cache.target.<name>.endpoint (and optional .region and
.path-style-access) names several, and io-cache.routes maps file types
to them. io-cache.whitelist and io-cache.policy keep their meaning, and
io-cache.origin.endpoint names the OSS endpoint when fs.oss.endpoint
points at a cache for older clients. io-cache.enabled set in the
catalog options overrides the token.
- Only read and meta requests use a target; exists counts as meta. Writes,
listings, deletes, renames, copies and non-oss paths stay on OSS, and a
client-side dlf.oss-endpoint turns routing off.
- Files rewritten in place (FileType.is_mutable) and metadata named by a
sequence id never use a target.
- Data files are recognized by name: a data or changelog file prefix,
including a custom one, followed by {uuid}-{count}.{extension}.
Manifests, indexes and statistics share the {uuid}-{count} shape but
have no extension, so a custom prefix such as manifest- cannot mix them
up; the manifest sidecar is excluded explicitly. Other names keep their
FileType, and names Paimon does not write stay on OSS.
- Errors from a target are raised as they are, without retrying on OSS.
- PyArrow, Lance and Vortex readers use the target's endpoint, scheme and
addressing style (fs.oss.second.level.domain.enable for path-style).
The routing cases in tests/resources/io_cache are shared with the Java
Jindo FileIO and paimon-rust.
sundapeng
force-pushed
the
pypaimon-io-cache-routing
branch
from
October 5, 2026 11:14
0af7167 to
a74a6cd
Compare
IoCacheRoutingFileIO inherited write_file, delete_quietly and the other FileIO helpers that check existence before a write or after a delete. Since exists() of a cacheable file now goes to a cache, write_file's check could reach a cache before the file existed. These helpers now delegate to the origin FileIO, and a test fails when a new FileIO method is neither overridden nor listed as a plain read.
Some cache endpoints, such as an OSS accelerator, also take writes and send them to OSS over their own write path, whose bandwidth adds to the direct one. With write in io-cache.policy, new_output_stream, the format writers (parquet, orc, avro, lance, blob, vortex, row), write_file and the pyarrow output streams of cacheable files use the file type's target. write_file still checks existence on origin first, and atomic writes, deletes, renames, copies, listings and writes of snapshots, hints and temp files stay on origin. A policy with only write routes writes only.
sundapeng
changed the title
Replace the JSON test vectors with assertions grouped by topic in io_cache_routing_test.py.
sundapeng
changed the title
sundapeng
marked this pull request as ready for review
October 6, 2026 16:02
exists answers whether a file is still there, and a cache that keeps the metadata of deleted objects answers yes after the file was deleted on OSS. meta now covers file status only; exists and exists_batch use a target only when io-cache.policy lists exists, as in the Java client.
…tes them The broad FileType classifier also matched names such as manifest.parquet, stat-2024.parquet or foo.index, which a Format Table may hold as ordinary files and replace in place, so a routed read could return the old file from a cache. A metadata or index file now uses a target only under the exact name Paimon writes, as in the Java and Rust clients; every other name goes to the origin.
paimon-rust's TableCommit names input-changelog manifests
manifest-{uuid}-changelog-{count}, optionally with .avro.sidecar. They are
written once under a unique name like every other manifest, so the
metadata name rule now accepts the -changelog infix for manifests (not
for manifest lists), matching paimon-rust and the Java client.
JingsongLi
reviewed
Oct 12, 2026
JingsongLi
left a comment
JingsongLi
left a comment
Contributor
There was a problem hiding this comment.
Reviewed this head in an isolated worktree. 195 routing/FileIO/token/native-format-option tests and 28 subtests pass, as do 162 focused tests on Python 3.8 and project Flake8. I also exercised real PyArrow S3 HTTP requests against local origin/target endpoints: Parquet data reaches the target and reads correctly. The two issues below were independently checked and reproduced; Lance was tested with supported pylance 0.39.0 and a real file/client.
…uting Lance and Vortex open files with their own storage clients, which take the endpoint as a virtual-hosted HTTPS address and cannot reach an HTTP, IP or path-style io-cache target. They now get the options of the origin FileIO, as without routing, and only Paimon's own FileIO calls use targets.
A child forked while another thread of the parent was creating a target inherited the held lock and waited on it forever. The routing FileIO now notices the new pid and starts with an empty target map and a new lock.
A PyArrowFileIO restored from a pickle, as in a Ray worker, recomputed whether the bucket goes into the endpoint without looking at path-style addressing, so with PyArrow below 16 it put the bucket in front of a path-style io-cache target again.
… routing Under io-cache routing the origin FileIO takes io-cache.origin.endpoint, which may be a URL, so Vortex, which reads with its own client from the origin, got https://bucket.https://host. It now gets the bucket in front of the host. The new test pins that Vortex reads the origin, not a path-style IP target, through RESTTokenFileIO; it replaces the Lance test that went away when upstream dropped the Lance file format.
With write in io-cache.policy, a routed write ran entirely on the target FileIO. On the PyArrow S3 backend (fs.oss.impl=legacy, also the fallback without pyjindosdk), PyArrowFileIO.new_output_stream first checks the parent directory and creates it, so the target received HEAD and LIST probes of directories, HeadBucket and directory-marker PUTs: requests for keys that are not routable file types. JindoSDK skips that step, which is why only the PyArrow backend showed it. new_output_stream now runs its parent directory step through create_parent_dirs, a no-op by default on FileIO. The routing FileIO runs that step on origin before a routed write, so origin sees the same directory requests as without routing, and target FileIOs open the object without it. The pyarrow filesystem streams of the routing FileIO are unchanged: they never had a parent directory step.
…n io-cache routing
The postpone writer names its files
{prefix}-u-{commit user}-s-{write id}-w-{uuid}-{count}.{extension}, which did
not match {prefix}{uuid}-{count}.{extension}, so their reads and writes always
used the origin. paimon-rust names its managed BLOB packs
{prefix}{uuid}.managed.blob, without the count Java writes. Both are written
once under a unique name and are now data files, with the .index and .blobref
files named after them, so all three clients route the same names
(apache/paimon-rust#1014).
test_whitelist also pins that io-cache.whitelist takes only the
contract's file types: a local-cache selector such as structured-data
is ignored, as paimon-rust does since 11c8613.
PyArrowFileIO.write_parquet and write_orc delete the file when the write fails. When io-cache routing sends the write to a target, that cleanup ran on the target's PyArrowFileIO, so the target got a DELETE and a HEAD for a path a target should only see PUT, GET and HEAD of finished files, and the origin, which unrouted holds the cleanup, got nothing. A target PyArrowFileIO now skips the cleanup (cleans_up_failed_writes, like creates_parent_dirs), and the routing FileIO runs delete_quietly on origin after a routed write_parquet or write_orc fails, then re-raises. Unrouted writes are unchanged. test_failed_routed_writes_are_cleaned_up_on_origin fails both writes on a recording backend: the target only sees the upload, origin and the unrouted FileIO each clean up the path once. It fails without the fix.
io-cache routing only knew the data file prefixes in the token and applied the same rules to every table the FileIO served. apache/paimon-rust#1014 review found gaps that this client has as well: - Other writers may replace Format Table files in place under the same name, and manage an Object Table's objects, so a target that caches them can serve stale content. - A table whose schema sets data-file.prefix or changelog-file.prefix writes data files the router did not recognize, so they always used the origin. Files written before such an option changed keep the old prefix, and a branch can set its own. FileIO gets two views with no-op defaults. origin_only() sends no request to a target; FormatTable, ObjectTable and IcebergTable (written by Iceberg writers) take it. for_table(path, prefixes, branch) is taken by FileStoreTable.__init__, which every load, copy, time travel and branch copy goes through, so no way of getting a table bypasses it. The view routes the prefixes of the table's schema, and once a name that may be a data file matches none of them, the prefixes of every schema of the branches it reads, listed from origin at most once. A copy keeps the prefixes and branches of the table it was made from: a branch made from a tag reads files main wrote. If the history cannot be read, the files it would have named stay on origin, which only loses io-cache. RESTTokenFileIO views keep the token and caches and apply the view to the FileIO cached for the token. A REST catalog may keep a branch's schema without a schema directory in storage, so a REST view also asks the catalog for the schema of each branch it reads. CachingFileIO wraps the view of its delegate and shares its cache. An IoCacheRoutingFileIO view shares the origin and targets of its base and closes nothing. Tests: io_cache_table_routing_test writes and reads real tables through a FileSystemCatalog whose FileIO routes to a target over the same storage; after ALTER TABLE changes data-file.prefix, a fresh load reads the older files through the target, and a branch copy reads the files written with the branch's prefix through it. rest_table_file_io_view_test checks the view every kind of REST table holds; the RESTTokenFileIO test checks that a branch view takes the prefix the catalog keeps; the routing and FileIO tests check that the history is only read for unknown data file names, once, from origin, that a failure leaves files on origin, and that a forked child does not wait on a listing the parent has in progress. The table tests fail without the change.
Routing decided by file name alone, so any URI read through a table's FileIO could reach a target if its name looked like a Paimon file. Row data references objects outside the table: BLOB descriptor fields, descriptors returned in blob-as-descriptor mode, multimodal blob reads. Other writers own those objects and may replace them under the same name. apache/paimon-rust#1014 review reproduced the stale read with an external oss://bkt/import/data-{uuid}-1.png that origin had replaced while the cache kept the old bytes; this client read it through the target the same way. A target now only serves paths under the table's location: the path a RESTTokenFileIO is built for, and the path of the table a view is for. Everything else is read from origin, as without routing. This covers every reader of row-data URIs at once, including ones added later, instead of each of them choosing a FileIO. Files the table writes under its location, branches included, keep their routing; blob views read the upstream table through that table's own FileIO and location. Data files under data-file.external-paths are outside the location and now use the origin, which is where they went without routing. io_cache_table_routing_test reads a data-evolution table whose blob-descriptor-field points at that external object: the routed read returns the origin's bytes and the target sees no request for it, while the table's own data files still go to the target. It returns the stale bytes without the change. The routing and RESTTokenFileIO tests check the location rule for a view and for the FileIO of a token.
Daft's IOConfig for a REST table and PVFS's ossfs or JindoSDK filesystem are built from the vended token and send every request to fs.oss.endpoint. Under io-cache routing that key is kept for clients that do not route, so it may name the cache: Daft read data files and the objects BLOB descriptors point at through it, and PVFS read snapshots, hints and every other file through it, the requests per-request routing keeps on origin. Like the Lance and Vortex readers, these clients now get the origin's options when the token enables routing: the origin endpoint, without the cache client keys. dlf.oss-endpoint still overrides everything, and an io-cache.enabled in PVFS's catalog options wins over the token, as it does for RESTTokenFileIO. Without routing the options are unchanged. test_single_endpoint_clients_get_the_origin_options and test_daft_gets_the_origin_options check the options both clients get with and without routing; they fail without the change.
…ting Opt-in native execution (read.native.enabled and the other native options) reuses the data token Python holds for a REST table and passes it to the paimon-rust core unchanged. Today's core reads no io-cache keys and sends every request to fs.oss.endpoint, which under routing is kept for clients that do not route and may name the cache. Native reads of snapshots, hints and manifests then went to the cache, the requests per-request routing keeps on origin. Like Daft, PVFS, Lance and Vortex, the core now gets the origin's options when the token enables routing: the origin endpoint, without the cache client keys. A dlf.oss-endpoint in the catalog options still turns routing off and leaves the token as it is. test_native_rest_cache_gives_the_core_the_origin_under_io_cache_routing checks the token the core receives with and without dlf.oss-endpoint; it fails without the change.
…actories Two FileIO wrappers dropped the view a table takes of io-cache routing, and two factories built FileIOs that could route URIs no table owns. - ResolvingFileIO (resolving-file-io.enabled) inherited origin_only() and for_table() from FileIO, which return the FileIO itself. A Format or Object Table on it was not origin-only, and no table's location, prefixes or schema history reached the FileIOs it resolves per scheme. Its views now share the resolved FileIOs and apply themselves to each one, as CachingFileIO does for its delegate; a view closes nothing. - As an io-cache origin or target, a ResolvingFileIO did not forward create_parent_dirs, and the target role did not reach the PyArrowFileIOs it resolves, so parent directories and the cleanup of a failed write ran on the target. Both now behave as for a PyArrowFileIO. - UriReaderFactory built from catalog options (the default uri_reader_factory of every FileIO) and the source FileIO of LeRobot and rosbag imports create FileIOs that serve no table. They now read through the origin-only view of those FileIOs, so a URI from row data never reaches a target by its name alone, the rule paimon-rust and the Java client follow. Today only RESTTokenFileIO builds a routing FileIO, and it wraps ResolvingFileIO rather than the other way round, so the first and third points are defensive; the second applies when a REST catalog sets resolving-file-io.enabled with routing on. io_cache_table_routing_test runs every table test again through a FileSystemCatalog with resolving-file-io.enabled whose resolved FileIO routes, and adds a Format Table whose file was replaced on origin and URIs read through a factory and a source FileIO; the routing FileIO test writes through ResolvingFileIO origin and targets. All fail without the change.
The previous commit read external HDF5, rosbag and LeRobot sources through the origin-only view of the ResolvingFileIO that _SourceFileIO builds. That view never changes a request: _SourceFileIO resolves each URI with FileIO.get and the caller's source options, and only RESTTokenFileIO builds a routing FileIO. It did break the multimodal tests, which put their own FileIO doubles in place of ResolvingFileIO and do not implement origin_only(). source_utils.py is back to upstream, and the routing test keeps only the UriReaderFactory case. The HDF5 and rosbag suites pass again with h5py and rosbags installed.
JingsongLi
reviewed
Oct 26, 2026
JingsongLi
left a comment
JingsongLi
left a comment
Contributor
There was a problem hiding this comment.
Requirement fit: SUPPORTED. Per-operation/file-type routing has end-to-end value, and the prior fork-lock issue is repaired; the obsolete Lance integration is removed upstream. Implementation: FINDINGS.
375 focused Python tests and 73 subtests passed, including bounded fork regressions and real-table controls; project-configured flake8 passed. I also ran actual PyArrow HTTP requests against origin/data/meta endpoints: routed Parquet and metadata writes/reads, origin existence checks, external/snapshot isolation through the routed FileIO, filesystem reads, and no fallback after target failure passed. Additional integration probes found the three gaps below. Current-head CI is green. Native evidence below is the real shared-token path plus captured bridge options and Rust source, not a local Rust HTTP run; no live OSS/Jindo service was exercised.
…s overrides to it RESTTokenFileIO._TOKEN_CACHE is shared by every instance for the same user and table, but refresh_token stored the token after merging in its own catalog's dlf.oss-endpoint and explicit io-cache.enabled. Another catalog with a different setting then reused that token. Python's own FileIO reapplied its catalog switch, but the native core and Daft merge the token last, so they followed the other catalog: with routing off in one catalog and on in the next, the second sent the native core and Daft to the legacy cache that fs.oss.endpoint names for old clients. The shared cache now keeps the token as the REST server vended it, and each instance applies its own catalog's overrides when it takes a token from the server or the cache. io_cache_routing computes the effective options in one place: with_catalog_overrides (dlf.oss-endpoint and an explicit io-cache.enabled) and effective_options (catalog options merged with the token, which wins, then the overrides). RESTTokenFileIO, Daft's REST options and single_endpoint_options, which gives the native core, Daft and PVFS the origin's options, all use them. io_cache_effective_options_test shares one vended token between two RESTTokenFileIOs whose catalogs set io-cache.enabled differently (false then true, false then unset, true then false) and checks the Python routing, the native data token and the Daft options of each, and that a dlf.oss-endpoint stays with its catalog; it fails without the change.
… origin PVFS backs OSS with one client that sends every request to one endpoint, so it takes the origin's options under io-cache routing. It looked for routing in the vended token alone, after copying only io-cache.enabled from the catalog. When the policy, targets and io-cache.origin.endpoint are set in the catalog and the token carries credentials and the legacy fs.oss.endpoint, routing was not recognized and PVFS read the legacy cache: cat_file returned the old object. PVFS now gives the token and its catalog options to single_endpoint_options, which applies the catalog's dlf.oss-endpoint and io-cache.enabled and looks for routing in both, as the native core and Daft do. io_cache_effective_options_test reads through PVFS.cat_file with the routing keys in the catalog only, with the REST calls and the ossfs constructor replaced at the client boundary; it fails without the change.
…io-cache routing Only RESTTokenFileIO builds a routing FileIO. Every other FileIO comes from FileIO.get, which for oss:// returns an OssFileIO that sends every request to fs.oss.endpoint. When io-cache routing is configured in the catalog options, fs.oss.endpoint may name the legacy cache that old clients use as a whole, so these FileIOs read from it. That covers the FileIOs a UriReaderFactory builds for row-data and BLOB URIs (its origin_only() returned the OssFileIO itself), the source FileIO of HDF5, LeRobot and rosbag imports, ResolvingFileIO and the FileIO of a filesystem or JDBC catalog. Daft's IOConfig for such catalogs and the native core of a table loaded from a filesystem or JDBC catalog took the same options. FileIO.get now gives an OssFileIO the origin's options when the options it gets enable io-cache routing, and Daft and the native core of those tables get single_endpoint_options. So that the routing FileIO's own origin and target FileIOs, also built by FileIO.get, keep their endpoints, their options no longer carry the io-cache keys they were derived from. The origin_only() call in UriReaderFactory is removed, as it changed nothing. io_cache_effective_options_test serves an old object from the legacy cache and the current one from the origin over HTTP to the real PyArrow client, with routing set in the catalog options: a URI reader, an import source FileIO, a filesystem catalog FileIO and a ResolvingFileIO read the current object without a request to the cache, a target FileIO still reads from its target, and Daft's options name the origin. native_plan_test checks the options the native core gets for a filesystem catalog table. Each fails without the change. The routing table test that read URIs through a FileIO.get replaced by the test harness is removed, as it could not see this path.
A branch created from a tag after data-file.prefix changed takes files named under the earlier prefix. Creating the branch copies the tag's schemas into it, so its schema history holds that prefix and the files reach the io-cache target. The new case reads such a branch through the catalog and fails when the view carries no schema history. A branch whose schemas exist only in the catalog cannot read such files: the scan loads each data file's schema from the branch's own schema directory and fails before any data request, so its history needs no other source.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
The pypaimon side of #10350: when a REST table token vends OSS cache endpoints,
RESTTokenFileIOwraps the OSS FileIO so that each request goes to a cache endpoint or to the origin, by operation and file type. The options and rules are the same as in #10350.How a request is routed
Every request goes either to a cache target or to the origin,
io-cache.origin.endpoint(by defaultfs.oss.endpoint):data-file.external-pathsor an object a BLOB descriptor points to, goes to the origin. A FileIO that serves no table reads such row-data URIs from the origin too. Paths that are notoss://, operations not listed inio-cache.policy(read,metafor file status,exists,write), temporary files, files rewritten in place and files named by a sequence id (snapshot-N,schema-N,changelog/changelog-N) go to the origin.data-file.prefixandchangelog-file.prefixof the table's current and earlier schemas, followed by{uuid}-{count}.{extension}, the same after the postpone writer's-u-{commit user}-s-{write id}-w-, or{uuid}.managed.blobfor a paimon-rust managed BLOB pack; its file index adds.index. Metadata ismanifest-,manifest-list-,index-manifest-orstat-followed by{uuid}-{count}, a changelog manifest written by paimon-rust ismanifest-{uuid}-changelog-{count}, and a manifest may add.avro.sidecar; a bucket index isindex-{uuid}-{count}; a global index is<kind>-global-index-{uuid}.index. Any other name, such as an external Format Table file calledmanifest.parquet, goes to the origin.io-cache.whitelist. Withio-cache.routes, the first rule naming the type picks the target; without it, the first target with an endpoint takes every type. A type no rule names goes to the origin.A target only ever sees files written once under a unique name, so it can be a read cache that keeps metadata.
existsis the exception: Paimon uses it to ask whether a file is still there, for example before replaying a commit on recovery, and a cache that keeps the metadata of deleted objects answers yes after the file is gone. Soexistsuses a target only when the policy lists it, which suits a target that answers existence as OSS does. The origin sees everything else, so it must answer every request as OSS does: OSS itself, or an endpoint that keeps no metadata cache, such as an OSS accelerator. Format, Object and Iceberg tables never use a target: other writers may replace their files in place under the same name, whatever the name looks like. A Format Table that should go through an OSS accelerator gets it by making the accelerator the origin.meta,read,exists,writemeta,readUsage
A read cache in front of the bucket
Reads and file status of immutable files use the cache; existence checks and everything else use OSS.
An endpoint that also takes writes
Data files are read, written and checked through the OSS accelerator, whose read, write and request paths to OSS add to the direct quotas; everything else uses OSS.
existsis listed because the accelerator keeps no metadata cache.The accelerator as origin, a read cache for data
Data files and global indexes are read through the cache. Snapshot probes, hint files, manifests, listings and all writes go through the accelerator.
The accelerator as origin, data straight to OSS
Data files are read and written on the OSS endpoint; metadata and the other request-heavy traffic go through the accelerator, so the two quotas add up.
Two targets by file type
Manifests and statistics are read through the accelerator, data files and global indexes through the cache, and the rest, existence checks included, uses OSS.
Options
The same options and values as in #10350, vended with the table token or set in the catalog options. None existed in pypaimon before this PR.
io-cache.enabledio-cache.policyread,meta(file status),exists,write;noneturns it off. Whole tokens, case-insensitive.io-cache.whitelistmeta,data,bucket-index,global-index,file-index, or*(the default).io-cache.endpointdefault.io-cache.targetswithio-cache.target.<name>.endpointio-cache.target.<name>.region,io-cache.target.<name>.path-style-accessdefaultincluded.io-cache.routesio-cache.origin.endpointfs.oss.endpoint, which may point at a cache for clients without routing.dlf.oss-endpoint(turns routing off) and the table optionsdata-file.prefix/changelog-file.prefix(read to recognise data files) are existing options this PR only reads.In pypaimon
new_input_stream,get_file_statusandget_file_sizefollow the read and meta rules,existsandexists_batchthe exists rule. Withwritein the policy,new_output_stream, the parquet, orc and avro writers,write_fileand the pyarrow output streams follow the write rule;write_filechecks existence on the origin first. When a write goes to a target, its parent directories are checked and created on the origin, as without routing (the PyArrow S3 backend does this before it opens a file, JindoSDK does not), so the target receives only the file itself.copy_file,copy_files,delete_quietlyand the other helpers that check existence before a write or right after a delete run on the origin. A test fails when a newFileIOmethod is neither handled by the routing FileIO nor listed as a plain read.FileIOinFileStoreTable.__init__, which every load, copy, time travel and branch copy goes through: the view routes only paths under the table's location and knows the prefixes of the table's schema; the prefixes of the earlier schemas of the branches it reads are listed once, through the origin, on the first request whose data file name has none of the current prefixes, and for a REST table also asked of the catalog, for a branch whose schema exists only there. Format, Object and Iceberg tables take an origin-only view.RESTTokenFileIO,CachingFileIOandResolvingFileIOviews share the FileIOs they were made from, apply the view to each, and close nothing; under routing, aResolvingFileIOorigin or target keeps the parent-directory and failed-write handling of the FileIOs it resolves.RESTTokenFileIObuilds a routing FileIO. Everything else that sends every request to one endpoint gets the origin's options when the options it is given enable routing: eachFileIOthatFileIO.getbuilds foross://(so the FileIOs of aUriReaderFactory, of HDF5, LeRobot and rosbag import sources, of aResolvingFileIOand of filesystem and JDBC catalogs), Vortex, Daft, PVFS, and the native core, whether it gets the token pypaimon holds or a filesystem catalog's options. The routing FileIO builds its origin and target FileIOs without the io-cache keys, so they keep their endpoints.write_parquetorwrite_orcto a target removes its partial file on the origin.dlf.oss-endpoint(turns routing off) and an explicitio-cache.enabled. The token cache shared by theRESTTokenFileIOs of a table keeps the token as vended, and each applies its own catalog's overrides, so catalogs with different settings never take each other's.fs.oss.second.level.domain.enablefor path-style). Vortex opens files with its own client, so it always gets the origin's options.#10350 and apache/paimon-rust#1014 implement the same rules.
Tests
io_cache_routing_test.py: the endpoint of each request with one and two targets, the policy (existson its own included), whitelist, routes and endpoint options, the type of each file name (custom prefixes that collide with reserved names, entropy and postpone directories, sidecars, temporary names, metadata directories, external names that only look like metadata or indexes such asmanifest.parquet), and the option parsing.io_cache_file_io_test.py: the endpoint eachFileIOand filesystem call reaches with and withoutwriteandexistsin the policy, per-target file systems, errors from a target, the write and delete helpers on the origin, the parent directories of routed writes created on the origin, and everyFileIOmethod classified.rest_token_file_io_test.pyandfile_io_test.pycover the token merge, the PyArrow endpoint and addressing, and that Vortex gets the origin's storage options.io_cache_file_io_test.pyalso forks while a parent thread holds the target lock, or while the schema history loads, and checks, with bounded waits, that the child's read returns.io_cache_table_routing_test.pywrites and reads real tables through aFileSystemCatalogwith a target that keeps an old copy:blob-descriptor-fieldsources outside the table come from the origin, files under an older schema prefix, branch files and the files a branch created from a tag took under an earlier prefix use the target, and table files keep the target whatever they are named like.rest_table_file_io_view_test.pyandrest_token_file_io_test.pycover the views of REST tables, their copies and catalog-only branch schemas;io_cache_file_io_test.pythe views themselves, failed routed writes and the origin options of Daft and PVFS;native_plan_test.pythose of the native core.io_cache_effective_options_test.pyserves an old object from a legacy cache and the current one from the origin over HTTP to the real PyArrow client, with routing set in the catalog options only: URI readers, import sources, filesystem catalog andResolvingFileIOFileIOs, PVFS and Daft read the origin, and a target FileIO keeps its target. TwoRESTTokenFileIOs sharing one vended token, with differentio-cache.enabledin their catalogs, each route as their catalog says and give the native core and Daft their own endpoint.Verified end to end
All three clients (Java, Python, Rust) created, wrote, read, time-travelled and dropped partitions of tables against local OSS mock endpoints that log every request, with one target, two targets by file type, write routing, and an OSS accelerator as origin with a read cache or OSS as the target. Every request a target saw was one these rules send to it. pypaimon also ran its cases through a real OSS accelerator from inside the VPC, as target and as origin, with both JindoSDK and PyArrow S3: two commits written, read back, time-travelled and read across both layouts.
API and Format
No storage format change. Routing is configured through options.
FileIOgetsorigin_onlyandfor_table, whose defaults return the FileIO itself, andcreate_parent_dirs, a no-op by default: the parent directory step thatPyArrowFileIO.new_output_streamnow runs through, so the routing FileIO can run it on the origin. ExistingFileIOimplementations are unaffected.Documentation
No docs page yet; the options are listed above.