fix(orchestrator): mint distinct message IDs for cancel re-publishes - #505
Merged
Conversation
behinddwalls
marked this pull request as ready for review
August 4, 2026 01:53
behinddwalls
force-pushed
the
preetam/speculation-cancel-msgid
branch
from
August 4, 2026 01:58
733a5fd to
f2e6633
Compare
behinddwalls
force-pushed
the
preetam/speculation-buildsignal-paths
branch
from
August 4, 2026 01:58
c2d441b to
37428c4
Compare
This was referenced Aug 4, 2026
behinddwalls
force-pushed
the
preetam/speculation-cancel-msgid
branch
from
August 4, 2026 05:11
f2e6633 to
dda3dd3
Compare
behinddwalls
force-pushed
the
preetam/speculation-buildsignal-paths
branch
2 times, most recently
from
August 5, 2026 03:15
c33a2d6 to
6cabb60
Compare
behinddwalls
force-pushed
the
preetam/speculation-cancel-msgid
branch
from
August 5, 2026 03:15
dda3dd3 to
96feaf5
Compare
behinddwalls
force-pushed
the
preetam/speculation-buildsignal-paths
branch
from
August 5, 2026 03:26
6cabb60 to
8a077b1
Compare
behinddwalls
force-pushed
the
preetam/speculation-cancel-msgid
branch
from
August 5, 2026 03:26
96feaf5 to
d450509
Compare
behinddwalls
force-pushed
the
preetam/speculation-buildsignal-paths
branch
from
August 5, 2026 03:38
8a077b1 to
492d1cb
Compare
behinddwalls
force-pushed
the
preetam/speculation-cancel-msgid
branch
from
August 5, 2026 03:38
d450509 to
9227c40
Compare
behinddwalls
force-pushed
the
preetam/speculation-buildsignal-paths
branch
from
August 5, 2026 03:43
492d1cb to
65aad34
Compare
behinddwalls
force-pushed
the
preetam/speculation-cancel-msgid
branch
from
August 5, 2026 03:44
9227c40 to
dbd61ed
Compare
behinddwalls
force-pushed
the
preetam/speculation-buildsignal-paths
branch
from
August 5, 2026 04:11
65aad34 to
3e3666f
Compare
behinddwalls
force-pushed
the
preetam/speculation-cancel-msgid
branch
2 times, most recently
from
August 5, 2026 04:57
baf15bf to
82985ad
Compare
behinddwalls
force-pushed
the
preetam/speculation-buildsignal-paths
branch
from
August 5, 2026 04:57
3e3666f to
7f2f2fe
Compare
behinddwalls
force-pushed
the
preetam/speculation-buildsignal-paths
branch
2 times, most recently
from
August 5, 2026 19:05
b2a17a5 to
bf0df7a
Compare
behinddwalls
force-pushed
the
preetam/speculation-cancel-msgid
branch
from
August 5, 2026 19:05
fd83d99 to
64f7e91
Compare
behinddwalls
force-pushed
the
preetam/speculation-buildsignal-paths
branch
from
August 6, 2026 02:15
bf0df7a to
a6b5f8d
Compare
behinddwalls
force-pushed
the
preetam/speculation-cancel-msgid
branch
from
August 6, 2026 02:15
64f7e91 to
d51250d
Compare
behinddwalls
force-pushed
the
preetam/speculation-buildsignal-paths
branch
from
August 6, 2026 02:34
a6b5f8d to
60970f0
Compare
behinddwalls
force-pushed
the
preetam/speculation-cancel-msgid
branch
from
August 6, 2026 02:34
d51250d to
650a0d2
Compare
behinddwalls
force-pushed
the
preetam/speculation-buildsignal-paths
branch
from
August 6, 2026 03:04
60970f0 to
e780560
Compare
behinddwalls
force-pushed
the
preetam/speculation-cancel-msgid
branch
from
August 6, 2026 03:04
650a0d2 to
a351b6b
Compare
behinddwalls
force-pushed
the
preetam/speculation-buildsignal-paths
branch
from
August 6, 2026 03:17
e780560 to
4142ca7
Compare
behinddwalls
force-pushed
the
preetam/speculation-cancel-msgid
branch
from
August 6, 2026 03:17
a351b6b to
04e65c5
Compare
behinddwalls
force-pushed
the
preetam/speculation-buildsignal-paths
branch
from
August 6, 2026 03:32
4142ca7 to
bd68e1f
Compare
behinddwalls
force-pushed
the
preetam/speculation-cancel-msgid
branch
2 times, most recently
from
August 7, 2026 03:15
c167e22 to
b2e22cc
Compare
behinddwalls
force-pushed
the
preetam/speculation-buildsignal-paths
branch
2 times, most recently
from
August 7, 2026 03:30
ae3dfa9 to
c28713b
Compare
behinddwalls
force-pushed
the
preetam/speculation-cancel-msgid
branch
from
August 7, 2026 03:30
b2e22cc to
10ac3fc
Compare
mnoah1
approved these changes
Aug 7, 2026
behinddwalls
force-pushed
the
preetam/speculation-cancel-msgid
branch
from
August 7, 2026 19:48
10ac3fc to
fc33bc5
Compare
behinddwalls
force-pushed
the
preetam/speculation-buildsignal-paths
branch
2 times, most recently
from
August 7, 2026 19:54
350d26d to
ee4fcaa
Compare
behinddwalls
force-pushed
the
preetam/speculation-cancel-msgid
branch
from
August 7, 2026 19:54
fc33bc5 to
b8131db
Compare
behinddwalls
force-pushed
the
preetam/speculation-buildsignal-paths
branch
from
August 7, 2026 20:13
ee4fcaa to
6331c0f
Compare
behinddwalls
force-pushed
the
preetam/speculation-cancel-msgid
branch
from
August 7, 2026 20:13
b8131db to
17bbf48
Compare
## Summary ### Why? The cancel controller published its speculate hand-off with the bare batch ID as the message ID. The queue deduplicates on (topic, partition key, message ID) against every row it has not garbage-collected yet, consumed ones included, so a redelivery's re-publish for the same batch was a silent no-op — leaving a batch stuck Cancelling with nothing driving it to terminal. Same class of bug as b1f5795 in stovepipe. ### What? `publishBatchID` now goes through `submitqueue/core/publish`: `publish.UniqueID` mints a distinct message ID per publish (and documents the dedup rule once), and `publish.Message` owns the registry-lookup plumbing the controller previously hand-rolled. The batch-path test now asserts the payload still carries the batch ID while the message ID is distinct per publish, and the multi-batch tests assert on the payload's batch ID for the same reason. ## Test Plan ✅ `bazel test //submitqueue/orchestrator/controller/cancel/...` # Conflicts: # submitqueue/orchestrator/controller/cancel/BUILD.bazel # submitqueue/orchestrator/controller/cancel/cancel.go # Please enter the commit message for your changes. Lines starting # with '#' will be kept; you may remove them yourself if you want to. # An empty message aborts the commit. # # interactive rebase in progress; onto 8548220f # Last command done (1 command done): # pick c167e22 # fix(orchestrator): mint distinct message IDs for cancel re-publishes # No commands remaining. # You are currently rebasing branch 'preetam/speculation-cancel-msgid' on '8548220f'. # # Changes to be committed: # modified: submitqueue/orchestrator/controller/cancel/BUILD.bazel # modified: submitqueue/orchestrator/controller/cancel/cancel.go # modified: submitqueue/orchestrator/controller/cancel/cancel_test.go #
behinddwalls
force-pushed
the
preetam/speculation-cancel-msgid
branch
from
August 7, 2026 20:14
17bbf48 to
288ee84
Compare
behinddwalls
changed the base branch from
preetam/speculation-buildsignal-paths
to
main
August 7, 2026 20:14
behinddwalls
temporarily deployed
to
stack-rebase
August 7, 2026 20:23 — with 
GitHub Actions
Inactive
GitHub Actions
Inactive
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Why?
The cancel controller published its speculate hand-off with the bare batch ID as the message ID. The queue deduplicates on (topic, partition key, message ID) against every row it has not garbage-collected yet, consumed ones included, so a redelivery's re-publish for the same batch was a silent no-op — leaving a batch stuck Cancelling with nothing driving it to terminal. Same class of bug as b1f5795 in stovepipe.
What?
publishBatchIDnow goes throughsubmitqueue/core/publish:publish.UniqueIDmints a distinct message ID per publish (and documents the dedup rule once), andpublish.Messageowns the registry-lookup plumbing the controller previously hand-rolled. The batch-path test now asserts the payload still carries the batch ID while the message ID is distinct per publish, and the multi-batch tests assert on the payload's batch ID for the same reason.Test Plan
✅
bazel test //submitqueue/orchestrator/controller/cancel/...Issues