Repository navigation
Conversation
sundapeng
changed the title
e24bc8a to
78eac7b
Compare
84d1500 to
30a7dac
Compare
When the options carry io-cache endpoints, for example vended with a REST
table token, the OpenDAL OSS FileIO now reads immutable files through a
cache target chosen by file type and keeps every other request on the
OSS endpoint. The rules are the same as in the Java Jindo FileIO and
pypaimon.
- io-cache.endpoint names one target. io-cache.targets with
io-cache.target.<name>.endpoint (and optional .path-style-access) names
several, and io-cache.routes maps file types to them. io-cache.whitelist
and io-cache.policy keep their meaning, and io-cache.origin.endpoint
names the OSS endpoint when fs.oss.endpoint points at a cache for older
clients. io-cache.enabled set in the catalog options overrides the token.
- Only read, stat and exists requests use a target. Writes, including
OutputFile::exists, listings, deletes, renames, copies and non-oss paths
stay on OSS, and a client-side dlf.oss-endpoint turns routing off.
- Files rewritten in place (FileType::is_mutable) and metadata named by a
sequence id never use a target.
- Data files are recognized by name: a data or changelog file prefix,
including a custom one, followed by {uuid}-{count}.{extension}.
Manifests, indexes and statistics share the {uuid}-{count} shape but
have no extension, so a custom prefix such as manifest- cannot mix them
up; the manifest sidecar is excluded explicitly. Other names keep their
FileType, and names Paimon does not write stay on OSS.
- Errors from a target are returned as they are, without retrying on OSS.
- Adds fs.oss.path-style-access for the OSS endpoint and documents the
options in getting-started.
The routing cases in testdata/io_cache are shared with the Java Jindo
FileIO and pypaimon.
30a7dac to
ef2a480
Compare
Some cache endpoints, such as an OSS accelerator, also take writes and send them to OSS over their own write path, whose bandwidth adds to the direct one. With write in io-cache.policy, OutputFile writers of cacheable files (data files, manifests, indexes) use the file type's target. OutputFile::exists, the existence check a writer makes, stays on origin, as do deletes, renames, copies, listings and writes of snapshots, hints and temp files. A policy with only write routes writes only.
sundapeng
changed the title
Replace the JSON test vectors with assertions grouped by topic in the cache_routing tests, and remove testdata/io_cache.
sundapeng
changed the title
JingsongLi
left a comment
JingsongLi
left a comment
There was a problem hiding this comment.
Requirement fit: SUPPORTED. Implementation: FINDINGS at 16bc555. Per-file-type OSS routing has a practical end-to-end use with REST-vended endpoints, but commit recovery currently trusts stale cache metadata and can publish a snapshot referencing cleaned data.
Validation: all 167 IO tests passed on the PR head, including an added real Parquet table write/commit/scan probe with both meta,read and meta,read,write; hiding the newly committed snapshot in the cache did not prevent discovery. The automatic merge with current main b262b5f is clean and all 3,772 core tests passed (6 ignored), including that probe. An additional actual commit/truncate/expire/delete/replay probe reproduced the missing-file recovery regression below. Routing rules were also compared with Java #10350 at 8bb021cd. No live OSS/cache cluster was available.
exists answers whether a file is still there: commit recovery checks the files it replays. A cache that keeps the metadata of deleted objects answers yes after the file was deleted on origin, so a replay could publish a snapshot that references a deleted file. meta now covers file status only; FileIO::exists and InputFile::exists use a target only when io-cache.policy lists exists, as in the Java and Python clients.
JingsongLi
left a comment
JingsongLi
left a comment
There was a problem hiding this comment.
Requirement fit: SUPPORTED. Implementation: FINDINGS. The new exists policy fixes the previous stale-positive recovery finding: a real Parquet write/commit/read, truncate, snapshot expiration, origin deletion and replay probe now rejects both recovery APIs and leaves snapshot 2 current even while the target retains the deleted data. All 188 IO, REST-token and snapshot tests pass.
A separate actual Format Table read reproduces the stale-row regression below: after replacing manifest.parquet on origin, a direct scan reads [2] while the routed scan reads [1]. No live OSS/cache deployment was available. The same Format Table regression reproduces after clean integration with main dd25596; a temporary origin-only FormatTable read control makes both probes pass.
…s them
The broad FileType classifier also matched names such as manifest.parquet,
stat-2024.parquet or foo.index, which a Format Table may hold as ordinary
files and replace in place, so a routed scan could return the old file
from a cache. A metadata or index file now uses a target only under the
exact name Paimon writes: manifest-, manifest-list-, index-manifest- and
stat-{uuid}-{n} (a manifest also with .avro.sidecar), index-{uuid}-{n},
and {kind}-global-index-{uuid}.index. Data files and their file indexes
keep the existing name rule; every other name goes to the origin.
test_format_table_file_named_like_a_manifest_uses_origin replaces such a
file on the origin while a target still holds the old one.
# Conflicts: # crates/paimon/src/io/file_io.rs
JingsongLi
left a comment
JingsongLi
left a comment
There was a problem hiding this comment.
Requirement fit: SUPPORTED. Implementation: FINDINGS at 98ae24a. The previous exists-policy and manifest.parquet findings are fixed. Independent actual Parquet write/commit/read and both recovery API probes pass with meta,read and meta,read,write; an origin-deleted file retained by a separate cache is rejected and snapshot 2 remains current.
Validation: all 3,913 core tests pass (6 ignored), including the original routing/REST/snapshot tests. Additional real Format Table reads against independent origin/cache endpoints confirm manifest.parquet now returns current rows, but a valid external file named data-{uuid}-1.parquet still returns stale rows; the origin-only control returns current rows. An actual PK input-changelog commit and incremental read also show the generated changelog manifest PUT/GET bypassing the configured metadata target. Both remaining findings reproduce after clean integration with main 2ec5871. No live OSS/cache deployment was available.
Please also update the getting-started io-cache.policy documentation: meta now means file status only; exists is a separate supported token.
A Format Table may hold files that other writers copy in or replace under
the same name, so even a Paimon-shaped name such as
data-{uuid}-1.parquet does not make its content immutable, and a routed
scan could return the old copy a cache still holds. Table::new and
Table::from_resolved_schema now give a Format Table the origin-only view
of its FileIO, so its reads, file status lookups and writes all use the
origin whatever io-cache.policy says; other tables keep the per-request
routing.
format_table_reads_replaced_files_from_origin_with_io_cache reproduces
the review probe: the target keeps id=1, the origin holds the
replacement id=2, the scan returns [2], an append stays on the origin,
and the target sees no request.
TableCommit names input-changelog manifests
manifest-{uuid}-changelog-{count}, optionally with .avro.sidecar, but the
metadata name rule only accepted manifest-{uuid}-{count}, so these
immutable files always went to the origin. The rule now accepts the
-changelog infix for manifests (not for manifest lists).
test_input_changelog_manifests_use_the_io_cache_target commits a
primary-key table with changelog-producer=input and manifest sidecars
under meta,read,write, reads the changelog incrementally, and checks
that the changelog manifest and its sidecar are written and read
through the target and never reach the origin.
meta now covers file status only and exists is a separate token, which the OSS Cache Routing section still described as one.
This reverts 129fad9. Format Table files may be replaced in place by scheduled jobs, but whether that can return old content depends on the target: one that forwards every request to OSS, such as an OSS accelerator, never does. The REST catalog that vends the targets knows which kind it vends and already resolves the io-cache options per table, database and catalog, so it decides whether a Format Table uses a target, as it does for exists. FileIO routes Format Table files by name like those of any other table again; the docs say who decides.
JingsongLi
left a comment
JingsongLi
left a comment
There was a problem hiding this comment.
Requirement fit: SUPPORTED. Per-request routing has end-to-end value for separating immutable data/cache traffic from mutable metadata and combining OSS endpoint capacity.
Implementation: FINDINGS. Two new routing gaps below were reproduced through real FileSystemCatalog create/load/write/commit/read APIs. The existing mutable Format Table stale-read finding also still reproduces on this head; I posted the current catalog-path evidence in its original thread.
Validation on this head integrated with current main: 173 IO tests, 8 REST token FileIO tests, snapshot commit/discovery and actual input-changelog manifest/sidecar tests all pass (183 standard targeted tests). An additional real commit-recovery probe passes: after the original snapshot and data are deleted on origin while retained by the target, both recovery APIs reject the replay and the latest snapshot remains unchanged. This confirms the exists-policy fix. Independent writer-routing and external-Parquet replacement assertions fail as described. These use local HTTP OSS-compatible endpoints and actual Arrow/Parquet/BLOB data; a live OSS accelerator/STS deployment was not available.
…che routing Other jobs may replace the files of a Format Table or an Object Table in place under the same name, so a Paimon-shaped name does not make them immutable. Leaving the choice to the catalog that vends the targets does not hold: FileSystemCatalog shares one FileIO across its tables, and REST external tables build theirs from the catalog options without a table token, so a catalog with a read cache had no way to exempt them. Table::new, Table::from_resolved_schema and copy_with_resolved_schema give a Format Table the origin-only view of its FileIO again, and ObjectTable keeps one for its objects, so their reads, file status lookups and writes use the origin whatever io-cache.policy says. A Format Table that should go through an OSS accelerator gets it by making the accelerator the origin with io-cache.origin.endpoint. format_table_reads_replaced_files_from_origin creates and loads a Format Table through FileSystemCatalog with catalog-level io-cache and meta,read. The target keeps row 7 and origin holds its replacement, row 9, under a Paimon-shaped name; the loaded table, a copy and a table built from the resolved schema all read 9, and the target sees no request. Without the change they read 7. object_table_reads_objects_from_origin does the same for an Object Table's FileIO.
A table's data-file.prefix and changelog-file.prefix live in its schema, but the io-cache router only read them from the FileIO options, so a table with a custom prefix sent every request for its data files to origin unless the prefix was copied into the catalog or token options. Each table's view of its FileIO now carries the prefixes of its schema, and every request passes them to the router, through RESTTokenFileIO too. copy_with_options and copy_with_resolved_schema add the prefixes of the new schema, so files written under a dynamic prefix use the target while earlier files keep using it. data_files_named_with_the_schema_prefix_use_the_target writes, commits and scans a FileSystemCatalog table whose schema alone sets data-file.prefix to custom-, under meta,read,write and the data whitelist: the PUT and GET of its data files, and of a copy with a dynamic- prefix, go to the target and none to origin. test_token_file_io_routes_files_named_with_table_prefixes covers the REST token path.
…ting
The postpone writer names its files
{prefix}-u-{commit user}-s-{write id}-w-{uuid}-{count}.{extension} and the
managed BLOB writer names its packs {prefix}{uuid}.managed.blob. Both are
written once under a unique name, but neither matched
{prefix}{uuid}-{count}.{extension}, so their reads and writes always used
origin. Both are now data files, with the .index and .blobref files named
after them. The other names the crate writes on current main (data,
changelog, blob, bucket and deletion vector indexes, global indexes,
manifests, statistics) already match.
postpone_bucket_files_use_the_target writes a bucket=-2 primary-key table
and managed_blob_packs_use_the_target writes and reads a primary-key BLOB
table through FileSystemCatalog under meta,read,write: the postpone file
and the packs are written and read on the target and never on origin.
JingsongLi
left a comment
JingsongLi
left a comment
There was a problem hiding this comment.
Requirement fit: SUPPORTED. Implementation: FINDINGS at 6c2f5c4. Per-request OSS endpoint routing has end-to-end value. The previous mutable Format Table stale-read finding and schema-prefix / postpone / managed-pack naming findings are fixed on their covered paths. Two additional production cases remain below.
Validation on the clean integration with main 272cce9: 174 IO tests, 9 REST-token FileIO tests, 5 real catalog/table routing tests, snapshot commit/discovery, and actual input-changelog routing pass (190 tests). Three independent real HTTP/Parquet/BLOB writer, external replacement, and commit-recovery probes also pass (193 total). Two further actual catalog/write/read probes reproduce the findings: old-prefix routing after persisted ALTER/reload, and stale external source bytes permanently copied into a managed BLOB pack. Retaining the old prefix in the reloaded FileIO context fixes the first probe; using origin-only FileIO for descriptor-source ingestion fixes the second. The temporary controls were restored after verification.
These checks use local HTTP OSS-compatible endpoints and actual persisted Arrow/Parquet/BLOB data. A live OSS accelerator / production STS deployment was not available.
Conflicts: - io/cache/file_type.rs: main split the Data file type into DedicatedData and StructuredData; keep that split and the pub(crate) visibility that io-cache routing needs, and expose is_dedicated_file to the router. - table/mod.rs: keep main's new instant module next to the io_cache_routing_tests module. Follow-ups so the routing behaviour survives the merge: - cache_routing.rs: data file names are classified as DedicatedData (.blob, .video, .vector.*) or StructuredData, so io-cache.whitelist=data keeps routing both, as before. - table/mod.rs: the options-only shortcut that main added to copy_with_resolved_schema also gets the table's io-cache FileIO view. - file_io.rs: a new main test deletes through the routed operator's origin.
…uting BlobFormatWriter copies the payload a BlobDescriptor points to into the .blob file of an append table or the .managed.blob pack of a primary-key table. The source is owned by whoever wrote it, not by the table, so its Paimon-shaped name says nothing about immutability: an import job may replace oss://bkt/import/data-<uuid>-1.png while an io-cache target still serves the old copy, and the table then persists the old payload for good. BlobFormatWriter, which both BLOB writer paths share, now reads descriptor sources through the origin-only view of its FileIO. The .blob files and managed packs the table writes are its own immutable files and keep their io-cache routing. TestOss can retain an old copy of an object, like a cache that has not seen the replacement. blob_descriptor_sources_are_read_from_origin writes a descriptor whose source is NEW on origin and OLD on the target into a primary-key and an append BLOB table: the source GET reaches origin only, the .blob PUT still goes to the target, and both the routed table and an origin-only client read NEW. Without the fix both tables persist OLD.
…hema A table's io-cache FileIO view only knew the data-file.prefix and changelog-file.prefix of the schema it was built from. After ALTER TABLE changes a prefix, the table loaded again, for example after a restart, sends the GETs of the data files committed under the old prefix to origin only, so those immutable files silently lose their io-cache traffic. Every table view now carries a shared, once-initialized loader of the prefixes of the table's persisted schemas. Table::new and from_resolved_schema attach it, copies share it, and copies that switch to another branch attach one for that branch's schemas. The loader runs at most once, on the first request whose file is named like a data file with none of the view's own prefixes, and only after the backend says io-cache routing is on; the token FileIO answers that from the token the request needs anyway. It lists and reads schema/ through an origin-only view, which has no loader and never uses a target, and shares the SchemaManager cache. A failure is logged once and those files stay on origin. Loading a table therefore does no extra I/O or token request, and a table without io-cache routing, or whose files all carry its current prefixes, never reads its schema history. Dynamic prefixes that were never persisted cannot be recovered; files named with them keep going to origin, which is safe. altered_data_file_prefix_keeps_routing_after_reloads mirrors the review probe with a FileSystemCatalog: data-file.prefix=older-, write and commit, ALTER to newer-, load_table and append, then a new catalog's get_table scans both rows, the schemas are listed from origin during that scan, and the GETs of the older- and newer- files reach the target, not origin. A catalog without io-cache routing scans the same rows without listing the schemas. rest_tables_keep_routing_for_prefixes_of_older_schemas builds a REST table whose token vends io-cache routing and whose response carries only the newer- schema: building it sends no OSS request and no token request beyond the one RESTTokenFileIO::build_file_io already makes, and reading the older- file reaches the target without another token request.
io-cache.whitelist and io-cache.routes reuse FileType::parse_whitelist, which since main split data into dedicated and structured data also accepts local-cache's structured-data selector. That value is not part of the io-cache contract (meta, data, bucket-index, global-index, file-index or *), so a whitelist naming it would route on this client but not on others. Parse io-cache file types through a wrapper that treats structured-data like any other unknown name: it is logged and ignored. data still selects both dedicated and structured data files. test_structured_data_is_not_an_io_cache_file_type pins it: whitelist=data routes Parquet and .blob files, whitelist=structured-data routes neither, structured-data,meta only routes metadata, and a route of structured-data alone is skipped like other malformed rules.
A column listed in blob-descriptor-field keeps the BlobDescriptor the writer supplied; the object it points to is not copied into the table and is owned by whoever wrote it. Reads still resolved those descriptors through the table's io-cache routing, so a source with a Paimon-shaped name such as oss://bkt/import/data-<uuid>-1.png that was replaced on origin could be served from a stale target copy. With data-evolution.row-sidecar.enabled, RowSidecarWriter also copies these payloads into the .row file while writing, which could persist that stale copy, like the BLOB writers fixed earlier. The data evolution reader now resolves descriptor fields, and RowSidecarWriter reads their sources, through the origin-only view of the table's FileIO. Managed packs, .blob files and .row files written by the table keep their io-cache routing. blob_descriptor_fields_read_their_sources_from_origin writes a descriptor whose source is NEW on origin and OLD on the target into a descriptor field, with and without a ROW sidecar: the scan returns NEW, every source GET reaches origin and none the target, and the table's data files are still written to and read from the target. Without the reader change the scan returns OLD; without the sidecar change the sidecar write reads the source from the target.
JingsongLi
left a comment
JingsongLi
left a comment
There was a problem hiding this comment.
Requirement fit: SUPPORTED. Implementation: FINDINGS at d686bfa. The new append/managed-pack ingestion, DE descriptor-field/ROW-sidecar and historical-prefix regression tests pass. Two additional supported paths remain uncovered: PK external descriptor reads, and the current prefix of a catalog-only REST branch reached through copy_with_branch. Details inline.
Validation: 228 IO tests, 9 REST-token FileIO tests, 9 real catalog/table routing tests, snapshot commit/discovery and input-changelog manifest/sidecar routing passed (248 targeted tests). An independent full public FileSystemCatalog create/write/prepare/commit/read probe and a unit-level HTTP probe reproduce stale PK descriptor payloads; disabling routing on the same stored table returns NEW. A separate ordinary REST branch-copy probe reproduces origin-only routing while directly loading that same branch uses the target. These use actual local HTTP OSS endpoints and persisted Parquet/BLOB data. Formatting, diff checks and clean integration with main 4cf3437 were checked. No live OSS accelerator/production STS deployment was available; current CI could not be verified because the Code Review backend cannot connect to GitHub.
…ache A primary-key table with blob-descriptor-field keeps the descriptors in its rows and resolves them through managed_blob_reader::resolve_batch with the table's FileIO, so an object an import job owns, named like a Paimon file, was read from the target, which may still hold a copy the origin replaced. The previous round fixed this at three call sites; every other reader of descriptors, such as BlobReader over the table's FileIO behind the C and Go bindings, still routed them by name. Data files the table writes under data-file.external-paths went to the target too, which serves the table's location and may not serve those paths at all. A table's view of the FileIO now carries the table's location, and a request through it uses io-cache only for a path under that location; anything else uses the origin. A FileIO that serves no table, such as the one BlobReader builds from storage options, cannot tell who owns a URI taken from row data, so UriInput and BlobReader read such URIs from the origin. The managed packs and .blob files a table writes lie under its location and keep their routing. The three call-site views of the previous round are removed.
Copying a REST table to a branch kept the source table's FileIO, so that its provider and FUSE mount carry over, and attached the branch's schema history. A branch whose schema exists only in the catalog has no schema directory, so the prefixes of its current schema were lost and its data files were read from the origin. table_view now builds the view for every constructor and every copy that moves to another branch: origin only for a Format Table, otherwise the table's location, the prefixes of the schema it uses and the lazily loaded prefixes of the branch's persisted schemas. The REST branch copy passes the schema the catalog returned. Copies within a branch keep the loaded history and add the prefixes of their options.
… io-cache routing
The postpone writer names its files
{prefix}-u-{commit user}-s-{write id}-w-{uuid}-{count}.{extension}, which did
not match {prefix}{uuid}-{count}.{extension}, so their reads and writes always
used the origin. paimon-rust names its managed BLOB packs
{prefix}{uuid}.managed.blob, without the count this client writes. Both are
written once under a unique name and are now data files, with the .index and
.blobref files named after them, so all three clients route the same names
(apache/paimon-rust#1014).
io-cache routing chose the endpoint of a request from the file name alone, for every table a FileIO served. Review of apache/paimon-rust#1014 found that a name shows neither who owns a file nor which prefixes a table has used, and this client has the same gaps: - Format, Object, Iceberg and Lance tables hold files that other writers may replace in place under the same name. Their FileIO is now origin only. - A row may point at any file. A blob-descriptor-field value, or a BLOB written from a descriptor, names a file outside the table that its writer may replace, even when it is named like a data file. A FileStoreTable now gives io-cache routing its location, and only files under it may use a cache target; files under data-file.external-paths are read from origin, as without routing. Descriptor readers created from the catalog context serve no table, so their FileIO is origin only. - Data files named with the data-file.prefix or changelog-file.prefix of the schema, of a copy's options or of an older schema were not recognized. The view passes the prefixes of the table's schema, and the router lists the branch's older schemas on OSS, once, the first time a file under the table is named like a data file with none of the known prefixes. FileIO gets two default no-op methods, originOnly() and withTableView(). AbstractFileStoreTable's constructor applies the view, so every copy, time travel and branch takes the one of its own schema. RESTTokenFileIO, ResolvingFileIO and CachingFileIO apply the view to the FileIO they wrap; a JindoFileIO view shares the file systems and the presigner of its base and closes nothing. IoCacheTableRoutingTest writes and reads through a FileSystemCatalog over a FileIO whose target keeps the first copy it served while origin holds a new one, as the paimon-rust review does: descriptor fields of primary-key and append tables, a BLOB copied from a descriptor, data files under an external path, a table read after ALTER of its prefix, and a branch with its own prefix. Each case but the branch one fails without its part of this change; the jindo tests cover the router.
…n io-cache routing
The postpone writer names its files
{prefix}-u-{commit user}-s-{write id}-w-{uuid}-{count}.{extension}, which did
not match {prefix}{uuid}-{count}.{extension}, so their reads and writes always
used the origin. paimon-rust names its managed BLOB packs
{prefix}{uuid}.managed.blob, without the count Java writes. Both are written
once under a unique name and are now data files, with the .index and .blobref
files named after them, so all three clients route the same names
(apache/paimon-rust#1014).
test_whitelist also pins that io-cache.whitelist takes only the
contract's file types: a local-cache selector such as structured-data
is ignored, as paimon-rust does since 11c8613.
io-cache routing only knew the data file prefixes in the token and applied the same rules to every table the FileIO served. apache/paimon-rust#1014 review found gaps that this client has as well: - Other writers may replace Format Table files in place under the same name, and manage an Object Table's objects, so a target that caches them can serve stale content. - A table whose schema sets data-file.prefix or changelog-file.prefix writes data files the router did not recognize, so they always used the origin. Files written before such an option changed keep the old prefix, and a branch can set its own. FileIO gets two views with no-op defaults. origin_only() sends no request to a target; FormatTable, ObjectTable and IcebergTable (written by Iceberg writers) take it. for_table(path, prefixes, branch) is taken by FileStoreTable.__init__, which every load, copy, time travel and branch copy goes through, so no way of getting a table bypasses it. The view routes the prefixes of the table's schema, and once a name that may be a data file matches none of them, the prefixes of every schema of the branches it reads, listed from origin at most once. A copy keeps the prefixes and branches of the table it was made from: a branch made from a tag reads files main wrote. If the history cannot be read, the files it would have named stay on origin, which only loses io-cache. RESTTokenFileIO views keep the token and caches and apply the view to the FileIO cached for the token. A REST catalog may keep a branch's schema without a schema directory in storage, so a REST view also asks the catalog for the schema of each branch it reads. CachingFileIO wraps the view of its delegate and shares its cache. An IoCacheRoutingFileIO view shares the origin and targets of its base and closes nothing. Tests: io_cache_table_routing_test writes and reads real tables through a FileSystemCatalog whose FileIO routes to a target over the same storage; after ALTER TABLE changes data-file.prefix, a fresh load reads the older files through the target, and a branch copy reads the files written with the branch's prefix through it. rest_table_file_io_view_test checks the view every kind of REST table holds; the RESTTokenFileIO test checks that a branch view takes the prefix the catalog keeps; the routing and FileIO tests check that the history is only read for unknown data file names, once, from origin, that a failure leaves files on origin, and that a forked child does not wait on a listing the parent has in progress. The table tests fail without the change.
Routing decided by file name alone, so any URI read through a table's FileIO could reach a target if its name looked like a Paimon file. Row data references objects outside the table: BLOB descriptor fields, descriptors returned in blob-as-descriptor mode, multimodal blob reads. Other writers own those objects and may replace them under the same name. apache/paimon-rust#1014 review reproduced the stale read with an external oss://bkt/import/data-{uuid}-1.png that origin had replaced while the cache kept the old bytes; this client read it through the target the same way. A target now only serves paths under the table's location: the path a RESTTokenFileIO is built for, and the path of the table a view is for. Everything else is read from origin, as without routing. This covers every reader of row-data URIs at once, including ones added later, instead of each of them choosing a FileIO. Files the table writes under its location, branches included, keep their routing; blob views read the upstream table through that table's own FileIO and location. Data files under data-file.external-paths are outside the location and now use the origin, which is where they went without routing. io_cache_table_routing_test reads a data-evolution table whose blob-descriptor-field points at that external object: the routed read returns the origin's bytes and the target sees no request for it, while the table's own data files still go to the target. It returns the stale bytes without the change. The routing and RESTTokenFileIO tests check the location rule for a view and for the FileIO of a token.
|
@JingsongLi both findings from the 10-25 review are addressed and the threads answered: bbc4a69 routes only the files under a table's own location, so PK external descriptor reads go to the origin, and 04517aa gives every table and branch copy, including I also checked the Rust client against the class raised in the latest Java and Python reviews, a component routing or caching with options other than the effective ones: |
…meta rule A selective read takes the size of the ROW sidecar it reads before reading it. That status request went through the origin operator, while the read of the same sidecar followed the read rule, so with io-cache.policy meta,read the HEAD of every sidecar stayed on origin. immutable_row_sidecar_size now resolves the routed operators like any other read and stats through OpClass::Meta, as get_status does. The local size cache keeps its origin-based key. The new routing test reads two rows of a 100-row file and checks that both the HEAD and the GET of the sidecar reach the target; it fails without the change.
…ng tests A branch created from a tag copies the schemas of the files it inherits, so its own schema history already holds the earlier data-file.prefix of main. The new routing test alters the prefix, creates a branch from a tag taken after the change and reads both files through copy_with_branch and the branch option; the file named with the earlier prefix reaches the target. It fails when the view carries no schema history. A branch whose schemas exist only in the catalog cannot read such a file at all: the reader loads the file's schema from the branch's own schema directory before it opens the file. So no fallback to main's schemas is needed.
|
Two more commits since my last comment, from a self-review against the same method:
fmt, clippy and |
7e0781c to
cac6733
Compare
JingsongLi
left a comment
JingsongLi
left a comment
There was a problem hiding this comment.
Requirement fit: SUPPORTED. Implementation: CLEAN at cac6733; no new functional code findings in this pass. The previous PK external-descriptor and catalog-only REST branch-prefix findings are addressed. Endpoint routing has concrete end-to-end value, and the table/URI boundaries now keep externally managed objects on origin.
Local validation in an isolated worktree, Rust 1.98.0 on macOS:
cargo test --locked --offline -p paimon --lib: 4,577 passed, 7 ignored, including persisted Parquet/BLOB table reads, branch/schema-history routing, stale external payloads, snapshot discovery, changelog sidecars, target failures, and local-cache invalidation.- An additional independent HTTP probe passed: after a REST token refresh changes both origin and target endpoints, already-created InputFile/OutputFile handles write/read through the new endpoints; immutable data reaches the new target, snapshots stay on the new origin, and neither old endpoint is touched.
- Formatting, diff checks, and
cargo clippy --locked --offline -p paimon --all-targets -- -D warningspassed.
Production readiness still needs two existing integration blockers cleared:
- Rebase onto current main (2300ad5):
git merge-treereports a conflict incrates/paimon/src/io/file_io.rs, insideInputFile::reader_with_size_hint. Preserve main's memory metadata-cache identity while selecting the routed read operator, then rerun the affected cache/read tests. - CI run 40402186905 fails dependency policy on the inherited zstd/zstd-safe/zstd-sys versions (RUSTSEC-2026-0391, 0398, 0403, 0410). This PR does not change those dependencies; main already contains #1229's dependency/security-policy update. Include that baseline and obtain green required CI after resolving the conflict.
Validation used local HTTP OSS stand-ins and persisted files; live OSS accelerator/STS signing and production throughput were not exercised.
Purpose
The paimon-rust side of apache/paimon#10350: when the options carry OSS cache endpoints, for example vended with a REST table token, the OpenDAL OSS FileIO sends each request to a cache endpoint or to the origin, by operation and file type. The options and rules are the same as in apache/paimon#10350 (Java) and apache/paimon#10351 (pypaimon); both carry the table rules of this PR (Format and Object Tables on the origin, only files under the table's location on a target, the table's own file prefixes including those of earlier schemas, postpone and managed BLOB names).
How a request is routed
Every request goes either to a cache target or to the origin,
io-cache.origin.endpoint(by defaultfs.oss.endpoint):data-file.external-pathsor an object a BLOB descriptor points to, goes to the origin. A FileIO that serves no table reads such row-data URIs from the origin too. Paths that are notoss://, operations not listed inio-cache.policy(read,metafor file status,exists,write), temporary files, files rewritten in place and files named by a sequence id (snapshot-N,schema-N,changelog/changelog-N) go to the origin.data-file.prefixandchangelog-file.prefixof the table's current and earlier schemas, followed by{uuid}-{count}.{extension}, the same after the postpone writer's-u-{commit user}-s-{write id}-w-, or{uuid}.managed.blobfor a managed BLOB pack; its file index adds.indexand its BLOB references.blobref. Metadata ismanifest-,manifest-list-,index-manifest-orstat-followed by{uuid}-{count}, a changelog manifest fromTableCommitismanifest-{uuid}-changelog-{count}, and a manifest may add.avro.sidecar; a bucket index isindex-{uuid}-{count}; a global index is<kind>-global-index-{uuid}.index. Any other name, such as an external Format Table file calledmanifest.parquet, goes to the origin.io-cache.whitelist. Withio-cache.routes, the first rule naming the type picks the target; without it, the first target with an endpoint takes every type. A type no rule names goes to the origin.A target only ever sees files written once under a unique name, so it can be a read cache that keeps metadata.
existsis the exception: Paimon uses it to ask whether a file is still there, for example before replaying a commit on recovery, and a cache that keeps the metadata of deleted objects answers yes after the file is gone. Soexistsuses a target only when the policy lists it, which suits a target that answers existence as OSS does. The origin sees everything else, so it must answer every request as OSS does: OSS itself, or an endpoint that keeps no metadata cache, such as an OSS accelerator. Format Tables and Object Tables never use a target: other jobs may replace their files in place under the same name, whatever the name looks like. A Format Table that should go through an OSS accelerator gets it by making the accelerator the origin.meta,read,exists,writemeta,readUsage
A read cache in front of the bucket
Reads and file status of immutable files use the cache; existence checks and everything else use OSS.
An endpoint that also takes writes
Data files are read, written and checked through the OSS accelerator, whose read, write and request paths to OSS add to the direct quotas; everything else uses OSS.
existsis listed because the accelerator keeps no metadata cache.The accelerator as origin, a read cache for data
Data files and global indexes are read through the cache. Snapshot probes, hint files, manifests, listings and all writes go through the accelerator.
The accelerator as origin, data straight to OSS
Data files are read and written on the OSS endpoint; metadata and the other request-heavy traffic go through the accelerator, so the two quotas add up.
Two targets by file type
Manifests and statistics are read through the accelerator, data files and global indexes through the cache, and the rest, existence checks included, uses OSS.
Options
The same options and values as in apache/paimon#10350, vended with the table token or set in the catalog options. None existed in paimon-rust before this PR.
io-cache.enabledio-cache.policyread,meta(file status),exists,write;noneturns it off. Whole tokens, case-insensitive.io-cache.whitelistmeta,data,bucket-index,global-index,file-index, or*(the default).io-cache.endpointdefault.io-cache.targetswithio-cache.target.<name>.endpointio-cache.target.<name>.region,io-cache.target.<name>.path-style-accessdefaultincluded. OpenDAL signs without a region, so.regionis accepted and ignored.io-cache.routesio-cache.origin.endpointfs.oss.endpoint, which may point at a cache for clients without routing.dlf.oss-endpoint(turns routing off) and the table optionsdata-file.prefix/changelog-file.prefix(read from the table schema, and from the catalog or token options, to recognise data files) are existing options this PR only reads.Brief change log
io/cache_routing.rsparses the options and picks the endpoint of each request.io-cache.target.<name>.regionis accepted but not needed, since OpenDAL signs OSS requests without a region.FileIOkeeps one OpenDAL operator for the origin and one per target. Reads,stat(including the size a selective read takes of a ROW sidecar) andexistsfollow the read, meta and exists rules, andOutputFilewrites follow the write rule;OutputFile::exists, the check a writer makes, uses the origin.UriInputandBlobReader), so it covers BLOB descriptors the writers or a ROW sidecar copy into the table,blob-descriptor-fieldvalues read from append, data-evolution and primary-key tables, and descriptors read withBlobReaderover a table's FileIO (as the C and Go bindings do); aBlobReaderbuilt from storage options serves no table and reads them all from the origin. The.blob,.managed.bloband.rowfiles the table writes lie under its location and keep their routing.io-cache.whitelistandio-cache.routesaccept only the file types listed below;datacovers both dedicated and structured data files.RESTTokenFileIO:io-cache.enabledset in the catalog options overrides the token.fs.oss.path-style-accessfor the OSS endpoint.Tests
cache_routing.rs: the endpoint of each request with one and two targets, the policy (existson its own included), whitelist, routes and endpoint options, the type of each file name (custom prefixes that collide with reserved names, a table's own prefixes, postpone and managed BLOB names with their.indexand.blobreffiles, entropy and postpone directories, sidecars, temporary names, metadata directories, external names that only look like metadata or indexes such asmanifest.parquet), and option parsing.file_io/cache_routing_tests.rsruns a FileIO against local OSS test servers: which operations reach the target with and withoutwriteandexistsin the policy, a target that still holds a file the origin deleted, a Format Table file namedmanifest.parquetread from the origin while the target holds an older copy, per-target operators, errors from a target, the local block cache.snapshot_manager.rs: committing and discovering snapshots never touches the cache.table_write.rs: a primary-key table withchangelog-producer=inputand manifest sidecars commits and reads its changelog incrementally; the changelog manifest and its sidecar are written and read through the target.rest_token_file_io.rs: options from the token and the catalog, and a table's own prefixes through the token FileIO.table/io_cache_routing_tests.rsdrives FileSystemCatalog tables over local OSS test servers: a Format Table whose file the origin replaced under a Paimon-shaped name, while the target kept the old copy, reads the new rows through the loaded table, a copy andTable::from_resolved_schema, and an Object Table reads its objects from the origin; a table whose schema alone setsdata-file.prefix, plus a copy with a dynamic prefix, abucket=-2primary-key table and a primary-key BLOB table write and read their data files, postpone files and managed packs through the target; afterALTER TABLEchangesdata-file.prefix, a table loaded from a new catalog, and a REST table whose response carries only the newer schema, read the files under the older prefix through the target; BLOB descriptor sources that the origin replaced while the target kept the old copy are copied and read from the origin, in append, data-evolution and primary-key tables (a primary-keyblob-descriptor-fieldread keeps its managed packs on the target), and throughBlobReaderover the table's FileIO or over storage options, while a file under the table's location keeps the target; a REST branch whose schema exists only in the catalog, loaded directly, throughcopy_with_branchor through thebranchoption, reads files named with its prefix through the target; a branch created from a tag afterALTER TABLEchanged the prefix reads the files it inherited under the earlier prefix through the target, since creating it copied their schemas; a selective read of a data-evolution table sends both the status request and the read of its ROW sidecar to the target; and every other way to derive a table (Table::from_resolved_schema,copy_with_options, snapshot and tag time travel, a resolved or pinned snapshot, a resolved schema on main or on a branch, a filesystem branch copy) reads files named with the schema's prefix through the target.Verified end to end
All three clients (Java, Python, Rust) created, wrote, read, time-travelled and dropped partitions of tables against local OSS mock endpoints that log every request, with one target, two targets by file type, write routing, and an OSS accelerator as origin with a read cache or OSS as the target. Every request a target saw was one these rules send to it. The pypaimon side (apache/paimon#10351) also ran its cases through a real OSS accelerator, as target and as origin.
API and Format
No public API or storage format change; routing is internal and configured through options.
Documentation
docs/src/getting-started.mddocuments the options under "OSS Cache Routing".