[RFC/Discussion] Partition-at-a-Time Streaming Evaluation in WindowAggExec

[Subodh-17]

[RFC/Discussion] Partition-at-a-Time Streaming Evaluation in WindowAggExec

• Status: Early Design Proposal — Seeking Maintainer & Community Feedback
• Target Area: datafusion-physical-plan (WindowAggExec, WindowAggStream)
• Author: Community Contributor

---

1. Problem Statement

In Apache DataFusion, window functions that require full partition awareness (such as ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING, LEAD, LAG, and RANK) are executed by WindowAggExec.

Currently, WindowAggStream acts as a pipeline-breaking operator:

1. It buffers every incoming RecordBatch into Vec<RecordBatch> until the upstream stream yields None (End-of-Stream / EOS).
2. It blocks downstream operators from consuming even a single output row until the entire upstream pipeline terminates.
3. At stream exhaustion, it invokes concat_batches across all accumulated batches, materializing a monolithic batch of $N$ rows before evaluating window expressions.

CURRENT BEHAVIOR (Whole-Stream Buffering Stall):
Input Batches:   [B0] ---> [B1] ---> [B2] ---> [B3] ---> ... ---> [Bn] ---> EOS
Buffering:       |----------------- FULL BUFFER (All N rows) ------------------|
Evaluation:                                                                    |--> concat_batches(N rows)
Emission:                                                                      |--> [Single Monolithic Batch]
Pipeline State:  |<--------------------- BLOCKED ----------------------------->|

Operational Drawbacks

• $O(N)$ Memory Retention: Working memory scales linearly with total stream length ($N$), even when partition cardinality is high and individual partitions are small ($|P_i| \ll N$). In local benchmarks on a 2.5-million-row dataset containing ~16.6k partitions, baseline DataFusion retained all 2.5M rows in memory, reaching a peak resident set size (Peak RSS) of ~724 MB.
• Pipeline Latency: Downstream operators that could stream data incrementally (such as LimitExec, streaming network sinks, or interactive displays) are starved of rows until the full upstream input is scanned.

---

2. Proposed Approach: Partition-at-a-Time Streaming

DataFusion's physical planner requires inputs to WindowAggExec to be pre-sorted by PARTITION BY keys (enforced via required_input_ordering()). Consequently, all rows belonging to any partition $P_k$ arrive contiguously in the stream.

Once the first row with partition key $P_{k+1}$ is observed, partition $P_k$ is guaranteed to be complete and will receive no further rows.

We propose updating WindowAggStream to evaluate and emit completed partitions incrementally:

1. Active Partition Buffer: Maintain active_partition_batches: Vec<RecordBatch> and active_partition_key: Option<Vec<ScalarValue>> for the currently open partition.
2. Boundary Detection (process_batch): Inspect partition keys in incoming batches using evaluate_partition_ranges:
   • Intra-batch partitions: Evaluate partitions that start and end within the batch directly on batch.slice(start, len) without cross-batch concatenation.
   • Cross-batch partitions: Accumulate slices until a boundary key transition occurs, concatenate only that partition's slices, evaluate, and emit.
3. Output FIFO & Backpressure: Enqueue evaluated partitions into a pending_batches: VecDeque<RecordBatch>. The stream drains pending_batches before polling upstream input, ensuring upstream ingestion pauses when downstream consumption is slow.
4. Unpartitioned Fallback: Queries without a PARTITION BY clause (OVER (ORDER BY ...)) take an explicit code branch (self.is_unpartitioned()) that falls back to the existing whole-stream buffering path, preserving current behavior. (In contrast, partitioned queries where all rows happen to share a single partition key run through the streaming state machine, buffering slices in active_partition_batches until EOF).

PROPOSED PARTITION-STREAMING PIPELINE:
Input Batch B0:  [ P0 (tail) ][ P1 (complete) ][ P2 (head) ]
                  |           |                 |
Finalize P0:      |---------> |-> Eval & Enqueue -> [pending_batches] -> Emitted Downstream!
Evaluate P1:                  |-> Eval & Enqueue -> [pending_batches] -> Emitted Downstream!
Buffer P2:                                      |-> [active_partition_batches]
Working Memory:   Bounded by max(|P_i|) + BatchSize

---

3. Working Memory Model & Inherent Limitations

3.1 Qualified Memory Bound

Rather than claiming constant $O(1)$ memory, the physical memory bound for partitioned streams accounts for Arrow buffer sharing and asynchronous buffering:

$$\text{Peak Working Memory} \le O(|P_{\max}|) + O(B_{\text{input}}) + O(B_{\text{pending}}) + O(\text{eval_temp})$$

Where:

• $O(|P_{\max}|)$: The raw row data of the largest single partition in the stream.
• $O(B_{\text{input}})$ (Buffer Pinning Slack): Arrow zero-copy slicing (RecordBatch::slice) retains Arc references to parent memory buffers. If an open partition spans across multiple batches, the head slice pins the first batch (holding unneeded prefix rows) and the tail slice pins the last batch (holding unneeded suffix rows). This unneeded pinned data can add up to approximately $2 \times B_{\text{input}}$ of buffer slack.
• $O(B_{\text{pending}})$: Output batches awaiting downstream polling, bounded by $B_{\text{input}} \times \text{output_width}$ due to backpressure.
• $O(\text{eval_temp})$: During partition finalization, finalize_active_partition() simultaneously holds the input slices, the concatenated partition batch, the newly computed window columns, and the output batch. We estimate this creates an unverified transient memory peak of roughly $3\times\text{ to }4\times$ the active partition's physical row data before intermediate buffers are freed.

3.2 The Single-Large-Partition Limitation

Partition-at-a-time evaluation does not solve memory accumulation when an individual partition spans the entire input ($|P| = N$) or exceeds available physical RAM. In that case, active_partition_batches accumulates all batches, degenerating to $O(N)$ memory with measured near-parity (observed $0.99\times$ speedup, $<0.1%$ peak RSS difference). Mitigating single-large-partition memory exhaustion requires out-of-core disk spilling or intra-partition sliding window operators (BoundedWindowAggExec).

---

4. Current Prototype Limitations: MemoryPool & OOM Risk

A critical architectural limitation of the current prototype is that it does not integrate with DataFusion's MemoryPool:

• WindowAggStream currently allocates active partition buffers, output queues, and evaluation temporaries directly via the system allocator without calling MemoryReservation::try_grow().
• Operational Risk: DataFusion's query engine currently has no visibility into the operator's heap usage. If an individual partition is excessively large (or data is unpartitioned), memory will expand unthrottled, risking an unmanaged process termination (OS OOM kill) rather than returning a clean DataFusionError::ResourcesExhausted error or triggering spills.
• We propose scoping pure in-memory streaming as Phase 1, with dynamic MemoryReservation integration as Phase 2.

---

5. Controlled Benchmark Observations (and Caveats)

To evaluate the mechanical impact of partition streaming, controlled benchmarks were run under isolated process invocations on Linux x86_64 (rustc 1.99.0 release build, 5 runs per workload):

• Query: SELECT c1, sn, v, COUNT(sn) OVER (PARTITION BY c1 ORDER BY sn) FROM t;
• Payload Schema: c1: Utf8 (100-byte string partition keys), sn: Int64 (ordering sequence), v: Float64.

Workload	Rows / Partitions	Baseline Median Query Time	Prototype Median Query Time	Observed Speedup	Baseline Peak RSS	Prototype Peak RSS	Physical RAM Saved
Small Partitions (1.0M rows)	1.0M / 6,666 (~150 rows/part)	119.98 ms	30.19 ms	3.97×	321.9 MB	173.3 MB	148.6 MB (46.2%)
Small Partitions (2.5M rows)	2.5M / 16,666 (~150 rows/part)	246.19 ms	77.76 ms	3.17×	724.0 MB	352.7 MB	371.2 MB (51.3%)
Large Partitions (2.0M rows)	2.0M / 4 (500k rows/part)	157.85 ms	134.44 ms	1.17×	574.3 MB	349.1 MB	225.2 MB (39.2%)
Single Partition (2.0M rows)	2.0M / 1 (2M rows/part)	136.59 ms	137.86 ms	0.99×	543.7 MB	543.6 MB	+0.1 MB (0.0%)
Micro-Partitions (1.5M rows)	1.5M / 15,000 (100 rows/part)	147.91 ms	56.93 ms	2.60×	454.2 MB	227.2 MB	227.0 MB (50.0%)

Essential Measurement Caveats

1. Sample Size & Variability: On the 1M-row small-partition workload, baseline query timings exhibited run-to-run variability between $98.01\text{ ms}$ and $133.04\text{ ms}$ (29.2% spread, mean $117.47 \pm 12.61\text{ ms}$), whereas prototype timings ranged from $30.05\text{ ms}$ to $31.25\text{ ms}$ (mean $30.41 \pm 0.49\text{ ms}$). Pairwise speedup across the observed extremes ranged from $3.14\times$ to $4.43\times$. Because sample sizes are small ($N=5$), we report observed ranges rather than parametric confidence intervals.
2. Causal Caveat: Baseline variability correlated with higher minor page faults (~70k vs. ~32k) and higher system CPU time ($0.16\text{ s}$ vs. $0.05\text{ s}$). We hypothesize that virtual memory reallocations during monolithic batch concatenation contributed to this jitter, though direct kernel-level tracing (e.g. clear_page) was not instrumented.
3. No Universal Speedup Guarantee: These speedups reflect synthetic workloads where avoiding monolithic batch reallocations saved significant CPU cycles. They are not universal guarantees for narrower payloads or compute-heavy expressions.
4. Single Partition Near-Parity: On single-partition streams (2.0M rows), the prototype exhibited performance and memory near-parity ($0.99\times$ speedup, $543.6\text{ MB}$ vs. $543.7\text{ MB}$ peak RSS—a 148 KB difference, well within typical OS page allocator variance).

---

6. Correctness Status & Test Gaps

What Has Been Formally Tested

Automated unit tests in the prototype verify:

• Cumulative frames: COUNT(sn) with ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW.
• Forward-looking frames: SUM(sn) with ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING.
• Ranking functions: RANK() and ROW_NUMBER() computing partition-local sequences on distinct inputs.
• Boundary edge conditions: Partitions spanning multiple batches, multiple partitions per batch, empty inputs, NULL partition keys across batch boundaries, and early stream cancellation.

Outstanding Test Gaps (Prior to PR)

• Unverified LEAD/LAG Boundary Assertions: While test_partitioned_lead_lag_rank_following_frame executes queries with LEAD and LAG without panicking, the test asserts only on the emitted rank and sum columns. It lacks explicit assert_eq! assertions verifying that LEAD and LAG emit correct boundary values and NULLs. Proving navigation semantics remains an outstanding test gap.
• Ties in RANK: Duplicate sort values within a streaming partition have not yet been explicitly tested.
• Complex Frames: GROUPS units, sliding offset frames (ROWS BETWEEN 2 PRECEDING AND 2 FOLLOWING), and RANGE intervals.
• Key Types: Multi-column composite keys (PARTITION BY col_a, col_b), DictionaryArray, and StringViewArray keys.

---

7. Phased Implementation Proposal

To keep review manageable, we suggest splitting this work into phases:

• Phase 1 (This RFC): Pure in-memory partition-at-a-time streaming in WindowAggStream, direct batch emission via pending_batches, unpartitioned fallback, and comprehensive edge-case unit tests. Zero public API changes.
• Phase 2: Dynamic MemoryReservation integration with DataFusion's MemoryPool to prevent unmanaged OOMs, plus vectorized intra-batch evaluation for micro-partitions.
• Phase 3 (Future Idea for Design Review): Explore potential out-of-core spilling mechanisms if an individual partition $|P_i|$ exceeds available memory allocation (e.g. leveraging DataFusion's disk management infrastructure), subject to future architectural design review.

---

8. Specific Feedback Requested from Maintainers

We would appreciate guidance on the following questions:

1. Output Batch Granularity vs. Coalescing: When partitions are small (e.g. 100 rows), emitting completed partitions directly yields a higher frequency of small batches. Should WindowAggExec rely on the physical optimizer inserting CoalesceBatchesExec downstream, or should WindowAggStream support an internal row threshold before emitting?
2. Planner Routing with BoundedWindowAggExec: DataFusion currently uses BoundedWindowAggExec for certain sliding frames. What architectural criteria or planner heuristics should guide the selection between BoundedWindowAggExec (which slides within a partition) and partition-streaming WindowAggExec (which streams across partitions) when both might apply?
3. MemoryPool Reservation Granularity (Phase 2): For dynamic memory tracking in Phase 2, should MemoryReservation be updated per incoming batch, or dynamically per partition transition?
4. Sorted Stream Debug Assertions: Should WindowAggExec include an optional runtime debug assertion verifying that input batches strictly adhere to declared partition sort order?

---

Disclaimer

This is an early architectural design proposal to align on direction with DataFusion maintainers. The prototype code is an experimental proof-of-concept and is not yet a production-ready or merge-ready implementation.

[Subodh-17]

Update: Prototype Milestone — Partition-at-a-Time Streaming & Memory Accounting Implemented

Context & Motivation

In Apache DataFusion, WindowAggExec currently buffers all incoming RecordBatches across the entire query input before evaluating window functions or emitting a single row. This imposes an $O(N)$ memory footprint and blocks pipeline streaming, even when the input stream is already sorted by the window expression's PARTITION BY keys. While BoundedWindowAggExec provides row-level streaming for sliding frames (ROWS BETWEEN ... PRECEDING AND CURRENT ROW), queries with non-sliding frames or general window functions (e.g., RANK(), DENSE_RANK(), ROW_NUMBER(), LEAD(), LAG(), or OVER (PARTITION BY ... ORDER BY ... ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING)) currently force full-dataset buffering.

Following up on the initial RFC in discussion #26425, we have implemented and validated a working prototype that adds partition-at-a-time streaming with integrated MemoryReservation accounting to WindowAggExec.

When the input stream's physical plan properties declare an output ordering that satisfies the window PARTITION BY keys, the operator can activate its partition-streaming path. On this path, the operator accumulates incoming rows only for the active partition, evaluates window expressions, and queues completed partitions for progressive pipelined emission downstream. This bounds active partition accumulation to $O(|P_{\max}|)$ rows (where $|P_{\max}|$ is the row count of the largest partition) rather than buffering the entire $O(N)$ input dataset before emitting. Logically, completed output batches awaiting emission in pending_batches add to retained operator memory across poll cycles, while physical resident memory remains subject to Arrow zero-copy buffer pinning (as detailed below). (If input ordering guarantees are absent or the query is unpartitioned, the operator falls back to bulk whole-dataset buffering).

---

Implemented Prototype Behavior

1. Partition-at-a-Time Execution:

   • Detects partition boundaries on incoming batches using DataFusion's evaluate_partition_ranges helper, which uses lexicographical partition-range detection.
   • Emits completed interior partitions immediately via a FIFO queue (pending_batches).
   • Slices trailing partitions at batch boundaries, retaining only the trailing slice in active_partition_batches to accumulate with subsequent incoming batches.
   • Automatically falls back to full-dataset bulk aggregation when the input is unpartitioned or lacks the necessary partition-key ordering.
   • Fuses stream error handling so any evaluation failure cleanly halts the stream, frees memory reservations, and returns Poll::Ready(None) on subsequent polls.

2. Logical Memory Accounting Model:

   • Sliced partition ranges from an incoming batch $B$ are allocated proportional shares of $B$'s physical allocation based on row count ($\sum_{i=0}^{M-1} r_mem_i \equiv \text{batch_size}$).
   • Within this operator, this avoids significant $M \times$ logical reservation overcounting that occurs if get_record_batch_memory_size() is called naively on zero-copy Arrow slices (downstream operators' independent measurements are noted below).
   • Batches waiting in pending_batches remain reserved in the operator's MemoryReservation across async poll cycles.
   • When partitions are emitted downstream, the reservation shrinks by the exact byte amount attributed to that partition.
   • Invariant: Every byte allocated via reservation.try_grow() has an exact, byte-for-byte matching shrink(). On stream exhaustion, error, or early drop, reservation.size() returns to exactly 0.

---

Proposed Future Work (Not Implemented in this Prototype)

To maintain a clean boundary between current work and future enhancements:

1. Physical Planner Optimizer Rules: No planner rule integration (e.g. in EnforceDistribution or physical optimizer passes) is included in this prototype. Execution modes are constructed and evaluated at the operator level.
2. Out-of-Core Spilling (Phase 3): Disk spilling is not implemented. When an individual partition $|P_i|$ exceeds the available pool capacity, the operator halts safely with ResourcesExhausted rather than spilling to disk.
3. Slice Compaction / Copying: Slices are zero-copy; optional deep-copy compaction of trailing slices to free parent buffers is deferred for maintainer input.
4. Micro-Partition Vectorization: Vectorized batching across micro-partitions before emission is left for downstream coalescing.

---

Validation & Test Coverage (Command-Verified Output)

All checks and tests pass cleanly under WSL Linux x86_64 (rustc 1.99.0):

• Unit Tests: cargo test -p datafusion-physical-plan --lib -- windows::window_agg_exec::tests passes 28/28 tests (0.17s).
• Physical Plan Window Suite: cargo test -p datafusion-physical-plan --lib -- windows passes 46/46 tests (4.43s).
• Linter: cargo clippy -p datafusion-physical-plan --lib -- -D warnings completes with 0 warnings.
• Formatting & Hygiene: cargo fmt --check and git diff --check complete with exit code 0.

Key scenarios validated by automated tests:

• Multi-partition single-batch slicing ($M = 3$) with verified non-zero intermediate reservations and clean return to 0.
• Partitions spanning across multiple incoming batches.
• Boundary detection at exact batch edges, interspersed empty batches, and rank ties.
• Graceful stream termination and error fusing with zero memory leaks upon memory-pool exhaustion (ResourcesExhausted).
• Safe RAII reservation cleanup on early stream drops and upstream source errors.

---

Disclosed Architectural Limitations

We explicitly distinguish logical MemoryPool reservation accounting from physical-memory guarantees:

1. Logical Accounting $\neq$ Hard Physical Bound: Proportional row attribution guarantees exact arithmetic balancing within DataFusion's MemoryPool, but does not bound actual resident process RAM.
2. Zero-Copy Arrow Buffer Pinning: Zero-copy slicing (batch.slice()) creates slice array headers pointing to shared Arc<Buffer> allocations. If an incoming 64 MB batch contains a trailing 1-row partition, that 1-row slice (logically accounting for ~6.7 KB) pins the underlying 64 MB physical buffer in RAM until that partition boundary is finalized.
3. Transient Concatenation Spikes: When a partition spans multiple batches, concat_batches() allocates a contiguous RecordBatch before dropping input batches, temporarily consuming ~$2\times$ memory during the concatenation step without a separate pre-reservation.
4. Kernel Pre-Allocation: Arrow window compute kernels allocate output arrays before reservation.try_grow(window_cols_size) is called.
5. Downstream Multiplied Reservation: If downstream operators call get_record_batch_memory_size() on output slices, they will independently measure the full parent buffer capacity for each slice.
6. No Disk Spilling: A single partition exceeding available memory pool budget will cleanly terminate with ResourcesExhausted; it cannot spill to disk.

---

Questions for Maintainers

We would appreciate maintainer guidance on two specific architectural questions before preparing a formal pull request:

1. Operator Topology: Unified WindowAggExec vs. Dedicated Physical Operator?
   In the current prototype, streaming partition execution is embedded directly within WindowAggExec via an internal mode branch, falling back to bulk evaluation when input ordering is insufficient.
   Question: Do maintainers prefer keeping partition streaming encapsulated within WindowAggExec (mirroring how SortPreservingMergeExec handles streaming merges), or should it be split into a distinct execution plan node (e.g., StreamingWindowAggExec or PartitionedWindowAggExec) selected by a physical optimizer rule?

2. Memory-Accounting Contract for Arrow Slices and Temporaries:
   The prototype uses proportional row attribution ($\sum r_mem_i = \text{batch_size}$) to provide an exact logical balance within MemoryPool, while retaining zero-copy Arrow slices.
   Question: Is proportional logical attribution acceptable to maintainers for partition streaming, or should trailing slices that span batch boundaries be copied/compacted to physically release parent Arc<Buffer> allocations at the cost of memcpy overhead? Additionally, should temporary memory during concat_batches and window-kernel execution be pre-reserved through a scratch consumer, or is transient unreserved allocation standard practice in DataFusion for intra-step kernel calls?

---

Disclaimer

This is a local design proposal to align on direction with DataFusion maintainers. The prototype demonstrates functional partition-at-a-time evaluation and exact MemoryPool accounting, but does not claim production readiness, physical memory bounds, or planner integration.

[Subodh-17]

Design Consultation: Partition-by-Partition Streaming for WindowAggExec

Hello DataFusion maintainers and community,

We have been investigating memory efficiency and time-to-first-batch latency in WindowAggExec for partitioned window queries (e.g. OVER (PARTITION BY ...) where the input physical plan already provides ordering on the partition keys).

1. Context & Motivation

In DataFusion, queries with bounded window frames can slide incrementally via BoundedWindowAggExec. However, queries planned with WindowAggExec (such as unbounded window frames or whole-partition aggregations) currently accumulate the entire input stream into an in-memory Vec<RecordBatch> before evaluating window expressions and emitting results upon EOF.

When queries contain many distinct partitions (e.g., millions of user sessions or sensor IDs) and the input stream is ordered by those partition keys, buffering the entire dataset introduces unnecessary peak memory accumulation and delays the emission of completed partition results.

We have developed an isolated prototype of partition-by-partition streaming directly within WindowAggExec and would welcome maintainer guidance on whether this direction fits the project's roadmap.

2. Mechanism & Conservative Eligibility

The prototype introduces incremental partition boundary detection:

• Eligibility Check: Streaming is enabled only when the physical input plan's declared properties (input.equivalence_properties()) satisfy all PARTITION BY expressions, and all window expressions in the operator share identical PARTITION BY expressions. The operator relies on declared plan ordering and does not perform runtime sort validation.
• Incremental Emission: Incoming batches are inspected for partition boundary transitions. Completed interior partitions are evaluated with compute_window_aggregates() and yielded downstream immediately without waiting for global EOF.
• Cross-Batch Partitions: Partitions that span across batch boundaries are buffered in active_partition_batches until the partition boundary arrives or input EOF occurs.
• Conservative Fallback: If the physical plan does not declare an ordering matching all partition keys, if partition expressions vary across window functions, or if the query is unpartitioned, the stream safely falls back to the existing bulk whole-stream accumulation path.

3. Verification & Semantic Equivalence

The prototype is covered by 30 focused unit and regression tests in windows::window_agg_exec::tests:

• Semantic Equivalence: Exact column-by-column, schema, and row count equality between the streaming path and the bulk fallback path on identical multi-batch datasets (test_t8_streaming_bulk_equivalence).
• Edge Cases Tested: Null/repeated partition keys, dictionary-encoded keys, composite partition keys, rank ties, interspersed empty batches, and large multi-batch macro-partitions.
• Memory Lifecycle & Fusing: Tested monotonic MemoryReservation growth during buffering, immediate quota shrinking upon downstream yield, clean ResourcesExhausted handling when memory pool limits are exceeded, and strict stream fusing to None across tested error routes (source errors, schema mismatch, and pool exhaustion).

4. Disclosed Architectural Boundaries

To ensure clear alignment with DataFusion's design principles, we want to highlight three current engine boundaries:

1. Logical Accounting vs. Physical RSS: MemoryReservation tracks estimated batch buffer byte sizes against the MemoryPool (cooperative quota). Because partition slicing uses zero-copy slicing (batch.slice()), underlying parent Arrow buffers remain held in physical RAM until all referencing slices from that batch are dropped downstream.
2. Concatenation Memory Overhead: While concat_batches() executes (in the bulk fallback path or when assembling partitions that span multiple batches), the operator's existing MemoryReservation remains active and charged for the buffered input data. However, concat_batches() allocates new contiguous Arrow buffers for the combined output before the source batch slices are dropped. During this step, both the input slices and the new concatenated batch concurrently reside in heap memory, creating a workload-dependent transient allocation peak. This additional temporary allocation is not accounted for by an incremental reservation against the memory pool.
3. No External Spilling & Single-Partition Limits: Partition streaming reduces operator memory accumulation by emitting completed partitions as boundaries arrive rather than buffering the entire multi-partition stream. However, it does not guarantee a hard physical-RAM bound. A single partition whose size exceeds the available memory pool budget will fail with ResourcesExhausted, as out-of-core disk spilling is not implemented.

5. Questions for Maintainers:

1. Operator Architecture: Do maintainers prefer evolving WindowAggExec in place to support partition streaming with conservative bulk fallback, or would you favor introducing a distinct physical plan node (e.g. PartitionedWindowAggExec) or an optimizer rule?
2. Arrow Memory Contract: When slicing batches across partition boundaries, is proportional logical reservation over zero-copy slices acceptable to the engine, or would maintainers require explicit deep buffer copying (allocating fresh arrays) to guarantee the immediate physical release of parent batch allocations?
3. Spilling Prerequisites: Is partition-by-partition streaming valuable as a standalone in-memory streaming optimization, or would maintainers consider disk spilling a prerequisite before accepting changes in this area?
4. GSoC 2027 Alignment: Does the community consider partitioned window streaming an impactful topic for formal contribution / GSoC project proposals?

We appreciate your feedback and architectural guidance!