Skip to content

[python] Read VARIANT pages with Parquet OffsetIndex - #9962

Open
XiaoHongbo-Hope wants to merge 1 commit into
apache:masterfrom
XiaoHongbo-Hope:codex/late-materialize-variant-read
Open

XiaoHongbo-Hope wants to merge 1 commit into
apache:masterfrom
XiaoHongbo-Hope:codex/late-materialize-variant-read

Conversation

@XiaoHongbo-Hope

@XiaoHongbo-Hope XiaoHongbo-Hope commented Sep 18, 2026

Copy link
Copy Markdown
Contributor

What

  • Extend the Parquet OffsetIndex row-window reader from [python] Read Parquet row windows with OffsetIndex #9850 to standard VARIANT columns.
  • Read only the pages needed by a contiguous row window when it is beneficial.
  • Keep the ordinary reader fallback for missing indexes, scattered ranges, and unprofitable selections.

I/O trade-off

Selective reads can transfer fewer bytes but may issue more object-store range requests. This change does not alter the table format or page-index option semantics.

@XiaoHongbo-Hope XiaoHongbo-Hope changed the title [python] Add opt-in Parquet OffsetIndex reads for selected rows [python] Add opt-in late materialization using Parquet OffsetIndex Sep 18, 2026
@XiaoHongbo-Hope
XiaoHongbo-Hope force-pushed the codex/late-materialize-variant-read branch 2 times, most recently from fc224a8 to 4846dba Compare September 21, 2026 09:47
@XiaoHongbo-Hope XiaoHongbo-Hope changed the title [python] Add opt-in late materialization using Parquet OffsetIndex [python] Read VARIANT pages with Parquet OffsetIndex Sep 21, 2026
@XiaoHongbo-Hope
XiaoHongbo-Hope marked this pull request as ready for review September 21, 2026 11:24
@XiaoHongbo-Hope
XiaoHongbo-Hope force-pushed the codex/late-materialize-variant-read branch from 4846dba to d22f1da Compare September 21, 2026 11:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant