Optimize partial Flat random access with blocking I/O

[joseph-isaacs]

Summary

• add partial Flat-layout random access for fixed-width and ALPRD feature-vector data
• reconstruct directly from selected buffers and avoid repeated expression/metadata work
• batch and share segment-range requests while retaining the default blocking local-file path
• add an in-memory reconstruction benchmark

Why

Uniform feature-vector random access was spending significant CPU in request orchestration, repeated metadata validation, patch handling, and reconstruction. The updated path computes the selected ranges together, shares one exact batch, and reconstructs from the resolved buffers.

This PR is the blocking-I/O control variant. Applying the action/bench-random-access label runs the existing random-access workflow without enabling io_uring.

Measured results

On 8 pinned cores with hot page cache:

• feature-vectors/uniform: approximately 1.95 ms Vortex blocking versus 1.83 ms Lance
• feature-vectors/correlated: approximately 0.34 ms Vortex versus 0.54 ms Lance

Validation

• cargo test -p vortex-array
• cargo test -p vortex-layout
• cargo test -p vortex-io --lib (109 tests)
• cargo clippy -p vortex-io --lib --all-features -- -D warnings
• cargo test --doc -p vortex-io
• release random-access benchmark with 8-core pinning

[codspeed-hq]

Merging this PR will degrade performance by 7.91%

⚠️ Unknown Walltime execution environment detected

Using the Walltime instrument on standard Hosted Runners will lead to inconsistent data.

For the most accurate results, we recommend using CodSpeed Macro Runners: bare-metal machines fine-tuned for performance measurement consistency.

⚡ 3 improved benchmarks
❌ 6 regressed benchmarks
✅ 1991 untouched benchmarks
🆕 8 new benchmarks
⏩ 89 skipped benchmarks¹

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

	Mode	Benchmark	BASE	HEAD	Efficiency
❌	Simulation	sparse_is_constant	559.8 µs	891.2 µs	-37.18%
❌	Simulation	compact_sliced[(4096, 90)]	1.6 µs	1.9 µs	-13.97%
❌	Simulation	compact_sliced[(16384, 90)]	1.8 µs	2 µs	-13.01%
❌	Simulation	warm[(64, 256)]	10.8 ms	12.2 ms	-11.58%
❌	Simulation	compress_fsst[(1000, 64, 8)]	1 ms	1.2 ms	-11.54%
❌	Simulation	compress_fsst[(10000, 64, 4)]	8.6 ms	9.7 ms	-10.54%
⚡	WallTime	words_gather_scalar[65536]	9.4 µs	8.2 µs	+13.63%
⚡	Simulation	nullable_polygons_x_nullable_points	664.9 µs	587.2 µs	+13.24%
⚡	Simulation	nullable_polygons_90pct_x_constant_point	548.3 µs	487.3 µs	+12.52%
🆕	Simulation	arrow_export_100_predecoded_arrays	N/A	1.5 ms	N/A
🆕	Simulation	concat_100_canonical_one_row_arrays	N/A	131.6 µs	N/A
🆕	Simulation	container_concat_100_alprd_one_row_arrays	N/A	134.5 µs	N/A
🆕	Simulation	extract_100_prebuilt_arrays_to_arrow	N/A	5.6 ms	N/A
🆕	Simulation	rebuild_100_flat_arrays_from_resident_buffers	N/A	4.1 ms	N/A
🆕	Simulation	rebuild_100_flat_arrays_with_eager_leaf_decode	N/A	8.9 ms	N/A
🆕	Simulation	rebuild_and_extract_100_arrays_to_arrow	N/A	9.8 ms	N/A
🆕	Simulation	rebuild_with_eager_extract_100_arrays_to_arrow	N/A	10.2 ms	N/A

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.

---

_{Comparing ji/partial-flat-random-access-blocking (41615a2) with develop (b363fb7)}

Footnotes

1. 89 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

[github-actions]

Polar Signals Profiling Results

Latest Run

Status	Commit	Job	Attempt	Link
🟢 Done	41615a2	random-access-bench	1	Explore Profiling Data

Previous Runs (1)

Status	Commit	Job	Attempt	Link
🟢 Done	7cc750c	random-access-bench	1	Explore Profiling Data

---

Powered by Polar Signals Cloud

[github-actions]

Benchmarks: Random Access 📖

Verdict: No clear signal (medium confidence)
Attributed Vortex impact: -1.1%
Engines: random-access No clear signal (-1.1%, medium confidence)
Vortex (geomean): 0.982x ➖
Parquet (geomean): 0.993x ➖
Shifts: Parquet (control) -0.7% · Median polish -1.2%

How to read Verdict and Engines

• Verdict: Overall PR-level signal after subtracting baseline drift estimated from Parquet control rows. It can be Likely improvement, Likely regression, or No clear signal.
• Engines: Per-engine attribution. DataFusion is compared against DataFusion/Parquet controls; DuckDB is compared against DuckDB/Parquet controls. This answers whether each engine improved or regressed independently.
• Confidence: Based on directional consistency, share of rows above the noise floor, and control-run noise.

---

random-access / vortex-file-compressed / ns (0.982x ➖, 5↑ 6↓)

name	PR 41615a2 (ns)	base b363fb7 (ns)	ratio (PR/base)
random-access/feature-vectors/correlated/vortex-tokio-local-disk 🚀	275401	325491	0.85
random-access/feature-vectors/correlated/vortex-tokio-local-disk-footer	570860	616771	0.93
random-access/feature-vectors/uniform/vortex-tokio-local-disk 🚀	1636841	2563711	0.64
random-access/feature-vectors/uniform/vortex-tokio-local-disk-footer 🚀	2636770	3013585	0.87
random-access/nested-lists/correlated/vortex-tokio-local-disk 🚀	209197	293916	0.71
random-access/nested-lists/correlated/vortex-tokio-local-disk-footer 🚀	288635	340181	0.85
random-access/nested-lists/uniform/vortex-tokio-local-disk	1483052	1473034	1.01
random-access/nested-lists/uniform/vortex-tokio-local-disk-footer 🚨	1771681	1588092	1.12
random-access/nested-structs/correlated/vortex-tokio-local-disk	315947	325318	0.97
random-access/nested-structs/correlated/vortex-tokio-local-disk-footer 🚨	501160	450157	1.11
random-access/nested-structs/uniform/vortex-tokio-local-disk 🚨	1170063	960723	1.22
random-access/nested-structs/uniform/vortex-tokio-local-disk-footer 🚨	1527173	1216015	1.26
random-access/taxi/correlated/vortex-tokio-local-disk	813696	888603	0.92
random-access/taxi/correlated/vortex-tokio-local-disk-footer	1240182	1333316	0.93
random-access/taxi/uniform/vortex-tokio-local-disk 🚨	4546211	3967200	1.15
random-access/taxi/uniform/vortex-tokio-local-disk-footer 🚨	6659977	4684375	1.42
random-access/vortex-tokio-local-disk	547982	537780	1.02
random-access/vortex-tokio-local-disk-footer	942435	904694	1.04

random-access / parquet / ns (0.993x ➖, 0↑ 0↓)

name	PR 41615a2 (ns)	base b363fb7 (ns)	ratio (PR/base)
random-access/feature-vectors/correlated/parquet-tokio-local-disk	8761419444	8914077594	0.98
random-access/feature-vectors/correlated/parquet-tokio-local-disk-footer	8882219074	8781373858	1.01
random-access/feature-vectors/uniform/parquet-tokio-local-disk	8712869587	8796571130	0.99
random-access/feature-vectors/uniform/parquet-tokio-local-disk-footer	8880971449	8908534745	1.00
random-access/nested-lists/correlated/parquet-tokio-local-disk	137018448	137652053	1.00
random-access/nested-lists/correlated/parquet-tokio-local-disk-footer	136310510	140569020	0.97
random-access/nested-lists/uniform/parquet-tokio-local-disk	136708120	139050533	0.98
random-access/nested-lists/uniform/parquet-tokio-local-disk-footer	139747067	137892135	1.01
random-access/nested-structs/correlated/parquet-tokio-local-disk	20428685	20597244	0.99
random-access/nested-structs/correlated/parquet-tokio-local-disk-footer	20594045	20833053	0.99
random-access/nested-structs/uniform/parquet-tokio-local-disk	20024012	20334362	0.98
random-access/nested-structs/uniform/parquet-tokio-local-disk-footer	20211096	20609364	0.98
random-access/parquet-tokio-local-disk	182039111	182684728	1.00
random-access/parquet-tokio-local-disk-footer	182521703	181313582	1.01
random-access/taxi/correlated/parquet-tokio-local-disk	273123109	274106803	1.00
random-access/taxi/correlated/parquet-tokio-local-disk-footer	273507068	271756511	1.01
random-access/taxi/uniform/parquet-tokio-local-disk	287182327	290760652	0.99
random-access/taxi/uniform/parquet-tokio-local-disk-footer	289888896	290813261	1.00

random-access / lance / ns (0.993x ➖, 0↑ 0↓)

name	PR 41615a2 (ns)	base b363fb7 (ns)	ratio (PR/base)
random-access/feature-vectors/correlated/lance-tokio-local-disk	374587	355553	1.05
random-access/feature-vectors/correlated/lance-tokio-local-disk-footer	1078899	1117265	0.97
random-access/feature-vectors/uniform/lance-tokio-local-disk	1091053	1103367	0.99
random-access/feature-vectors/uniform/lance-tokio-local-disk-footer	1791448	1903521	0.94
random-access/lance-tokio-local-disk	650081	641508	1.01
random-access/lance-tokio-local-disk-footer	1355179	1346907	1.01
random-access/nested-lists/correlated/lance-tokio-local-disk	185248	185918	1.00
random-access/nested-lists/correlated/lance-tokio-local-disk-footer	600730	576459	1.04
random-access/nested-lists/uniform/lance-tokio-local-disk	889846	903831	0.98
random-access/nested-lists/uniform/lance-tokio-local-disk-footer	1319746	1330825	0.99
random-access/nested-structs/correlated/lance-tokio-local-disk	310620	308832	1.01
random-access/nested-structs/correlated/lance-tokio-local-disk-footer	456075	455628	1.00
random-access/nested-structs/uniform/lance-tokio-local-disk	2400577	2468432	0.97
random-access/nested-structs/uniform/lance-tokio-local-disk-footer	2555031	2600968	0.98
random-access/taxi/correlated/lance-tokio-local-disk	948969	963154	0.99
random-access/taxi/correlated/lance-tokio-local-disk-footer	1854762	1879591	0.99
random-access/taxi/uniform/lance-tokio-local-disk	9976336	10042761	0.99
random-access/taxi/uniform/lance-tokio-local-disk-footer	10472272	10776913	0.97