Skip to content

Allow Range partitioned inputs to PartitionedTopK#23355

Draft
stuhood wants to merge 3 commits into
apache:mainfrom
paradedb:stuhood.range-partitioned-top-k
Draft

Allow Range partitioned inputs to PartitionedTopK#23355
stuhood wants to merge 3 commits into
apache:mainfrom
paradedb:stuhood.range-partitioned-top-k

Conversation

@stuhood

@stuhood stuhood commented Jul 6, 2026

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

What changes are included in this PR?

In two commits:

  • Improve the robustness of WindowTopN to intermediate RepartitionExec nodes, to ensure that PartitionedTopK can still be used when target_partitions mismatches physical partitions.
  • Use required_input_distributions to declare required partitioning for PartitionedTopKExec, BoundedWindowAggExec, and WindowAggExec.

Are these changes tested?

Yes, new slt tests are added for both the WindowTopN robustness fix, and for the use of required_input_distributions in PartitionedTopK.

Are there any user-facing changes?

No: only plan changes.

@github-actions github-actions Bot added physical-expr Changes to the physical-expr crates optimizer Optimizer rules core Core DataFusion crate sqllogictest SQL Logic Tests (.slt) physical-plan Changes to the physical-plan crate labels Jul 6, 2026
@stuhood stuhood changed the title Stuhood.range partitioned top k Allow Range partitioned inputs to PartitionedTopK Jul 6, 2026
@github-actions

github-actions Bot commented Jul 6, 2026

Copy link
Copy Markdown

Thank you for opening this pull request!

Reviewer note: cargo-semver-checks reported the current version number is not SemVer-compatible with the changes in this pull request (compared against the base branch).

Details
     Cloning apache/main
    Building datafusion v54.0.0 (current)
       Built [ 100.039s] (current)
     Parsing datafusion v54.0.0 (current)
      Parsed [   0.033s] (current)
    Building datafusion v54.0.0 (baseline)
       Built [  99.505s] (baseline)
     Parsing datafusion v54.0.0 (baseline)
      Parsed [   0.035s] (baseline)
    Checking datafusion v54.0.0 -> v54.0.0 (no change; assume patch)
     Checked [   0.592s] 223 checks: 223 pass, 30 skip
     Summary no semver update required
    Finished [ 202.361s] datafusion
    Building datafusion-physical-expr v54.0.0 (current)
       Built [  27.807s] (current)
     Parsing datafusion-physical-expr v54.0.0 (current)
      Parsed [   0.044s] (current)
    Building datafusion-physical-expr v54.0.0 (baseline)
       Built [  27.944s] (baseline)
     Parsing datafusion-physical-expr v54.0.0 (baseline)
      Parsed [   0.046s] (baseline)
    Checking datafusion-physical-expr v54.0.0 -> v54.0.0 (no change; assume patch)
     Checked [   0.332s] 223 checks: 222 pass, 1 fail, 0 warn, 30 skip

--- failure inherent_method_missing: pub method removed or renamed ---

Description:
A publicly-visible method or associated fn is no longer available under its prior name. It may have been renamed or removed entirely.
        ref: https://doc.rust-lang.org/cargo/reference/semver.html#item-remove
       impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.48.0/src/lints/inherent_method_missing.ron

Failed in:
  RangePartitioning::compatible_with, previously in file /home/runner/work/datafusion/datafusion/target/semver-checks/git-apache_main/257152542e7c03dcd1162d720af457922e203cca/datafusion/physical-expr/src/partitioning.rs:253
  Partitioning::compatible_with, previously in file /home/runner/work/datafusion/datafusion/target/semver-checks/git-apache_main/257152542e7c03dcd1162d720af457922e203cca/datafusion/physical-expr/src/partitioning.rs:416

     Summary semver requires new major version: 1 major and 0 minor checks failed
    Finished [  56.984s] datafusion-physical-expr
    Building datafusion-physical-optimizer v54.0.0 (current)
       Built [  37.013s] (current)
     Parsing datafusion-physical-optimizer v54.0.0 (current)
      Parsed [   0.021s] (current)
    Building datafusion-physical-optimizer v54.0.0 (baseline)
       Built [  37.083s] (baseline)
     Parsing datafusion-physical-optimizer v54.0.0 (baseline)
      Parsed [   0.022s] (baseline)
    Checking datafusion-physical-optimizer v54.0.0 -> v54.0.0 (no change; assume patch)
     Checked [   0.122s] 223 checks: 223 pass, 30 skip
     Summary no semver update required
    Finished [  75.220s] datafusion-physical-optimizer
    Building datafusion-physical-plan v54.0.0 (current)
       Built [  35.277s] (current)
     Parsing datafusion-physical-plan v54.0.0 (current)
      Parsed [   0.135s] (current)
    Building datafusion-physical-plan v54.0.0 (baseline)
       Built [  35.196s] (baseline)
     Parsing datafusion-physical-plan v54.0.0 (baseline)
      Parsed [   0.133s] (baseline)
    Checking datafusion-physical-plan v54.0.0 -> v54.0.0 (no change; assume patch)
     Checked [   0.607s] 223 checks: 223 pass, 30 skip
     Summary no semver update required
    Finished [  72.607s] datafusion-physical-plan
    Building datafusion-sqllogictest v54.0.0 (current)
       Built [ 173.814s] (current)
     Parsing datafusion-sqllogictest v54.0.0 (current)
      Parsed [   0.021s] (current)
    Building datafusion-sqllogictest v54.0.0 (baseline)
       Built [ 173.591s] (baseline)
     Parsing datafusion-sqllogictest v54.0.0 (baseline)
      Parsed [   0.023s] (baseline)
    Checking datafusion-sqllogictest v54.0.0 -> v54.0.0 (no change; assume patch)
     Checked [   0.087s] 223 checks: 223 pass, 30 skip
     Summary no semver update required
    Finished [ 350.758s] datafusion-sqllogictest

@github-actions github-actions Bot added the auto detected api change Auto detected API change label Jul 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

auto detected api change Auto detected API change core Core DataFusion crate optimizer Optimizer rules physical-expr Changes to the physical-expr crates physical-plan Changes to the physical-plan crate sqllogictest SQL Logic Tests (.slt)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Allow Partitioning::Range to satisfy PartitionedTopK Distribution::KeyPartitioned requirements

2 participants