Skip to content

[fix](be) Avoid finalized pipeline task submit crash#64946

Merged
morningman merged 1 commit into
apache:branch-4.0from
BiteTheDDDDt:codex/pick-64899-branch-4.0
Jun 30, 2026
Merged

[fix](be) Avoid finalized pipeline task submit crash#64946
morningman merged 1 commit into
apache:branch-4.0from
BiteTheDDDDt:codex/pick-64899-branch-4.0

Conversation

@BiteTheDDDDt

Copy link
Copy Markdown
Contributor

What problem does this PR solve?

Issue Number: None

Related PR: #64899

Problem Summary: Backport #64899 to branch-4.0. A runtime filter dependency can submit a pipeline task after the task has started close/finalize. On branch-4.0, HybridTaskScheduler::submit() calls PipelineTask::is_blockable(), which reads task sink/operator resources that finalize releases. This adds a submit gate so terminal close/finalize rejects late hybrid scheduler submits before is_blockable() can touch released resources.

Release note

None

Check List (For Author)

  • Test: Unit Test / Static check
    • build-support/clang-format.sh be/src/pipeline/pipeline_task.cpp be/src/pipeline/pipeline_task.h be/src/pipeline/task_scheduler.cpp be/test/pipeline/pipeline_task_test.cpp && build-support/check-format.sh be/src/pipeline/pipeline_task.cpp be/src/pipeline/pipeline_task.h be/src/pipeline/task_scheduler.cpp be/test/pipeline/pipeline_task_test.cpp
    • ./run-be-ut.sh --run --filter=PipelineTaskTest.TEST_CLOSED_TASK_REJECTS_HYBRID_SUBMIT_BEFORE_FINALIZE:PipelineTaskTest.TEST_FINALIZED_TASK_REJECTS_HYBRID_SUBMIT
  • Behavior changed: No
  • Does this need documentation: No

@BiteTheDDDDt BiteTheDDDDt requested a review from morningman as a code owner June 29, 2026 08:48
@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

### What problem does this PR solve?

Issue Number: None

Related PR: apache#64899

Problem Summary: Backport apache#64899 to branch-4.0. A runtime filter dependency can submit a pipeline task after close/finalize has started releasing task resources. On branch-4.0, HybridTaskScheduler::submit() calls PipelineTask::is_blockable(), which reads _sink and _operators. This adds a submit gate so terminal close/finalize rejects late hybrid scheduler submits before is_blockable() can touch released resources.

### Release note

None

### Check List (For Author)

- Test: Unit Test / Static check
    - build-support/clang-format.sh be/src/pipeline/pipeline_task.cpp be/src/pipeline/pipeline_task.h be/src/pipeline/task_scheduler.cpp be/test/pipeline/pipeline_task_test.cpp && build-support/check-format.sh be/src/pipeline/pipeline_task.cpp be/src/pipeline/pipeline_task.h be/src/pipeline/task_scheduler.cpp be/test/pipeline/pipeline_task_test.cpp
    - ./run-be-ut.sh --run --filter=PipelineTaskTest.TEST_CLOSED_TASK_REJECTS_HYBRID_SUBMIT_BEFORE_FINALIZE:PipelineTaskTest.TEST_FINALIZED_TASK_REJECTS_HYBRID_SUBMIT
- Behavior changed: No
- Does this need documentation: No

(cherry picked from commit 1533359)
@BiteTheDDDDt BiteTheDDDDt force-pushed the codex/pick-64899-branch-4.0 branch from ad030a0 to 314d0c4 Compare June 29, 2026 08:49
@BiteTheDDDDt

Copy link
Copy Markdown
Contributor Author

/review

@BiteTheDDDDt

Copy link
Copy Markdown
Contributor Author

run buildall

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I reviewed PR #64946 across the four changed files and the surrounding pipeline lifecycle paths. I did not find a blocking correctness issue or any inline comment worth submitting.

Critical checkpoints:

  • HybridTaskScheduler::submit() now checks the submit gate under _blockable_check_lock before calling PipelineTask::is_blockable(), so late submits after terminal close/finalize no longer read _sink or _operators.
  • PipelineTask::close(close_sink=true) and PipelineTask::finalize() close the submit gate before terminal resource release, while close(close_sink=false) keeps normal EOS/pending-finish resubmits available.
  • The new tests cover both close-before-finalize and post-finalize submit paths through the real HybridTaskScheduler::submit() call, including a counting sink assertion that is_blockable() is not evaluated after the gate closes.
  • Existing review context for this PR had no inline threads, and there was no additional user focus beyond the full PR review.

Validation:

  • git diff --check 5494d4eca3fc5ee0f19f3371a16ed21430c7bf83..314d0c451fc97a6771925d4b4d8ff1a86909258f -- be/src/pipeline/pipeline_task.cpp be/src/pipeline/pipeline_task.h be/src/pipeline/task_scheduler.cpp be/test/pipeline/pipeline_task_test.cpp passed.
  • I could not run the BE unit tests in this runner because thirdparty/installed, thirdparty/installed/bin/protoc, and thirdparty/installed/bin/thrift are missing.
  • I did not rely on local format tooling because this runner has clang-format 18.1.3 while the Doris scripts normally expect clang-format 16.

Subagent conclusions:

  • optimizer-rewrite: no optimizer/rewrite, semantic-equivalence, or parallel join/aggregate issue found; convergence round 1 returned NO_NEW_VALUABLE_FINDINGS.
  • tests-session-config: no test/session/config/style issue found; convergence round 1 returned NO_NEW_VALUABLE_FINDINGS.

@hello-stephen

Copy link
Copy Markdown
Contributor

BE Regression && UT Coverage Report

Increment line coverage 100.00% (17/17) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 71.58% (25594/35758)
Line Coverage 54.45% (271492/498633)
Region Coverage 51.97% (224871/432716)
Branch Coverage 53.45% (96912/181319)

@hello-stephen

Copy link
Copy Markdown
Contributor

BE Regression && UT Coverage Report

Increment line coverage 100.00% (17/17) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 71.57% (25593/35758)
Line Coverage 54.45% (271507/498633)
Region Coverage 51.98% (224911/432716)
Branch Coverage 53.45% (96917/181319)

@morningman morningman merged commit 0b34bee into apache:branch-4.0 Jun 30, 2026
29 of 31 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants