feat: add BatchNormalizer: re-chunk data source batches by rows and bytes (target_batch_size_bytes)#23390

Draft

adriangb wants to merge 11 commits into

apache:mainfrom

pydantic:batch-normalizer

Commits on Jul 8, 2026

feat: add BatchNormalizer for row- and byte-targeted batch re-chunking

adriangb
and
claude
committed
feat: add datafusion.execution.target_batch_size_bytes to normalize source batches

adriangb
and
claude
committed
feat: add large_values benchmark for very large row values

adriangb
and
claude
committed
feat: bound sort and merge output batches by target_batch_size_bytes

adriangb
and
claude
committed
docs: extend target_batch_size_bytes description to sort/merge/agg spill output

adriangb
and
claude
committed
feat: wire large_values benchmark into bench.sh

adriangb
and
claude
committed
fix: scale the compaction waste floor to target_batch_size_bytes

adriangb
and
claude
committed
chore: allow too_many_arguments on SortPreservingMergeStream::new

adriangb
and
claude
committed
chore: remove accidentally committed local benchmark results and scripts

adriangb
and
claude
committed

Commits on Jul 9, 2026