feat(orchestrator): dispatch builds per speculation path - #503
Merged
Conversation
behinddwalls
marked this pull request as ready for review
August 4, 2026 01:52
behinddwalls
force-pushed
the
preetam/speculation-build-dispatch
branch
from
August 4, 2026 01:58
9cd7353 to
7ff20d4
Compare
behinddwalls
force-pushed
the
preetam/speculation-path-build-store
branch
from
August 4, 2026 01:58
4443963 to
471898d
Compare
This was referenced Aug 4, 2026
behinddwalls
force-pushed
the
preetam/speculation-path-build-store
branch
from
August 4, 2026 05:11
471898d to
edc7d3a
Compare
behinddwalls
force-pushed
the
preetam/speculation-build-dispatch
branch
from
August 4, 2026 05:11
7ff20d4 to
2e32d07
Compare
behinddwalls
force-pushed
the
preetam/speculation-path-build-store
branch
from
August 5, 2026 03:15
edc7d3a to
28db660
Compare
behinddwalls
force-pushed
the
preetam/speculation-build-dispatch
branch
from
August 5, 2026 03:15
2e32d07 to
e37f128
Compare
behinddwalls
force-pushed
the
preetam/speculation-path-build-store
branch
from
August 5, 2026 03:26
28db660 to
b528041
Compare
behinddwalls
force-pushed
the
preetam/speculation-build-dispatch
branch
from
August 5, 2026 03:26
e37f128 to
70285fb
Compare
behinddwalls
force-pushed
the
preetam/speculation-path-build-store
branch
from
August 5, 2026 03:37
b528041 to
51fe6ef
Compare
behinddwalls
force-pushed
the
preetam/speculation-build-dispatch
branch
from
August 5, 2026 03:38
70285fb to
3983b14
Compare
behinddwalls
force-pushed
the
preetam/speculation-path-build-store
branch
from
August 5, 2026 03:43
51fe6ef to
226cfe9
Compare
behinddwalls
force-pushed
the
preetam/speculation-build-dispatch
branch
from
August 5, 2026 03:43
3983b14 to
57dcb4f
Compare
behinddwalls
force-pushed
the
preetam/speculation-path-build-store
branch
from
August 5, 2026 04:11
226cfe9 to
994904d
Compare
behinddwalls
force-pushed
the
preetam/speculation-build-dispatch
branch
2 times, most recently
from
August 5, 2026 04:57
125e288 to
f41ca0f
Compare
behinddwalls
force-pushed
the
preetam/speculation-path-build-store
branch
from
August 5, 2026 04:57
994904d to
7bad238
Compare
behinddwalls
force-pushed
the
preetam/speculation-build-dispatch
branch
from
August 5, 2026 17:05
f41ca0f to
d870cb4
Compare
behinddwalls
force-pushed
the
preetam/speculation-path-build-store
branch
from
August 5, 2026 17:05
7bad238 to
34add68
Compare
behinddwalls
force-pushed
the
preetam/speculation-build-dispatch
branch
from
August 5, 2026 17:28
d870cb4 to
5658532
Compare
behinddwalls
force-pushed
the
preetam/speculation-path-build-store
branch
from
August 5, 2026 17:28
34add68 to
0ecb87d
Compare
behinddwalls
force-pushed
the
preetam/speculation-build-dispatch
branch
from
August 5, 2026 19:05
5658532 to
d6273e5
Compare
behinddwalls
force-pushed
the
preetam/speculation-path-build-store
branch
2 times, most recently
from
August 6, 2026 02:15
4fe0ed0 to
6138ba9
Compare
behinddwalls
force-pushed
the
preetam/speculation-build-dispatch
branch
from
August 6, 2026 02:15
d6273e5 to
9ac443d
Compare
behinddwalls
force-pushed
the
preetam/speculation-path-build-store
branch
from
August 6, 2026 02:34
6138ba9 to
e9be5be
Compare
behinddwalls
force-pushed
the
preetam/speculation-build-dispatch
branch
2 times, most recently
from
August 6, 2026 03:04
a6435a5 to
cf21443
Compare
behinddwalls
force-pushed
the
preetam/speculation-path-build-store
branch
2 times, most recently
from
August 6, 2026 03:17
68e2cf0 to
51812bc
Compare
behinddwalls
force-pushed
the
preetam/speculation-build-dispatch
branch
from
August 6, 2026 03:17
cf21443 to
8d307b2
Compare
behinddwalls
force-pushed
the
preetam/speculation-path-build-store
branch
from
August 6, 2026 03:32
51812bc to
06bb174
Compare
behinddwalls
force-pushed
the
preetam/speculation-build-dispatch
branch
from
August 6, 2026 03:32
8d307b2 to
781562f
Compare
behinddwalls
force-pushed
the
preetam/speculation-path-build-store
branch
from
August 7, 2026 03:15
06bb174 to
de8085d
Compare
behinddwalls
force-pushed
the
preetam/speculation-build-dispatch
branch
from
August 7, 2026 03:15
781562f to
c149f02
Compare
behinddwalls
force-pushed
the
preetam/speculation-path-build-store
branch
from
August 7, 2026 03:30
de8085d to
9d4492c
Compare
behinddwalls
force-pushed
the
preetam/speculation-build-dispatch
branch
from
August 7, 2026 03:30
c149f02 to
f2a2cbe
Compare
mnoah1
approved these changes
Aug 7, 2026
behinddwalls
force-pushed
the
preetam/speculation-path-build-store
branch
from
August 7, 2026 19:48
9d4492c to
02917dc
Compare
behinddwalls
force-pushed
the
preetam/speculation-build-dispatch
branch
from
August 7, 2026 19:48
f2a2cbe to
031ab18
Compare
## Summary ### Why? The build stage triggered one build per batch, on the batch's full dependency list. Speculation needs one build per *path*, on the subset of dependencies the path assumes will succeed — that subset is what lets a batch be verified before the batches ahead of it resolve. ### What? The dispatch controller now reads the head's path set and starts a build for every pending path, on that path's own base. It **only starts builds** — stopping them belongs to the poll loop (next commit) — and it **never writes the path set**, which stays the speculate run's single-writer state; what this stage knows is the build it started, recorded in per-build rows of its own. The write order is Trigger → Build record → write-once link (`entity.PathBuild`) → signal; each write makes the previous one reachable, and the link is the idempotency point for redeliveries. A lost link race publishes both builds' signals and acks — the poll loop keeps the one the link names. On redeliveries only, every live linked path gets its signal re-published, closing the crash window between link and signal without forking duplicate poll chains past the queue's dedup horizon. The build signal carries only the runner's build ID and partitions on it, so one slow build cannot block a head's other paths. Known gap: a crash between `Trigger` and the link orphans that build; the fix is an idempotency key on `BuildRunner.Trigger`, marked TODO. Also adds `submitqueue/core/publish` — the registry-lookup-and-send plumbing and the message-ID deduplication rule in one place. ## Test Plan ✅ `bazel test //submitqueue/orchestrator/controller/build/... //submitqueue/core/publish/...` — write order, the base holding only assumed-success dependencies, redelivery republish without rebuild, lost-race double signal, halted batch starting nothing, redelivery healing for live linked paths, and a pin that no path-set write of any kind happens. ✅ `make fmt`, `make gazelle`, `make mocks` # Conflicts: # submitqueue/orchestrator/controller/build/BUILD.bazel # submitqueue/orchestrator/controller/build/build.go # submitqueue/orchestrator/controller/build/build_test.go # Please enter the commit message for your changes. Lines starting # with '#' will be kept; you may remove them yourself if you want to. # An empty message aborts the commit. # # interactive rebase in progress; onto 0ecb87d # Last command done (1 command done): # pick d870cb4 # feat(orchestrator): dispatch builds per speculation path # No commands remaining. # You are currently rebasing branch 'preetam/speculation-build-dispatch' on '0ecb87dd'. # # Changes to be committed: # new file: submitqueue/core/publish/BUILD.bazel # new file: submitqueue/core/publish/publish.go # new file: submitqueue/core/publish/publish_test.go # modified: submitqueue/orchestrator/controller/build/BUILD.bazel # modified: submitqueue/orchestrator/controller/build/build.go # modified: submitqueue/orchestrator/controller/build/build_test.go # # Conflicts: # submitqueue/orchestrator/controller/build/build.go # submitqueue/orchestrator/controller/build/build_test.go # Please enter the commit message for your changes. Lines starting # with '#' will be kept; you may remove them yourself if you want to. # An empty message aborts the commit. # # interactive rebase in progress; onto 5df7677d # Last command done (1 command done): # pick 781562f # feat(orchestrator): dispatch builds per speculation path # No commands remaining. # You are currently rebasing branch 'preetam/speculation-build-dispatch' on '5df7677d'. # # Changes to be committed: # new file: submitqueue/core/publish/BUILD.bazel # new file: submitqueue/core/publish/publish.go # new file: submitqueue/core/publish/publish_test.go # modified: submitqueue/orchestrator/controller/build/BUILD.bazel # modified: submitqueue/orchestrator/controller/build/build.go # modified: submitqueue/orchestrator/controller/build/build_test.go #
behinddwalls
force-pushed
the
preetam/speculation-build-dispatch
branch
from
August 7, 2026 19:54
031ab18 to
043f412
Compare
behinddwalls
temporarily deployed
to
stack-rebase
August 7, 2026 20:12 — with
GitHub Actions
Inactive
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Why?
The build stage triggered one build per batch, on the batch's full dependency list. Speculation needs one build per path, on the subset of dependencies the path assumes will succeed — that subset is what lets a batch be verified before the batches ahead of it resolve.
What?
The dispatch controller now reads the head's path set and starts a build for every pending path, on that path's own base. It only starts builds — stopping them belongs to the poll loop (next commit) — and it never writes the path set, which stays the speculate run's single-writer state; what this stage knows is the build it started, recorded in per-build rows of its own.
The write order is Trigger → Build record → write-once link (
entity.PathBuild) → signal; each write makes the previous one reachable, and the link is the idempotency point for redeliveries. A lost link race publishes both builds' signals and acks — the poll loop keeps the one the link names. On redeliveries only, every live linked path gets its signal re-published, closing the crash window between link and signal without forking duplicate poll chains past the queue's dedup horizon.The build signal carries only the runner's build ID and partitions on it, so one slow build cannot block a head's other paths. Known gap: a crash between
Triggerand the link orphans that build; the fix is an idempotency key onBuildRunner.Trigger, marked TODO.Also adds
submitqueue/core/publish— the registry-lookup-and-send plumbing and the message-ID deduplication rule in one place.Test Plan
✅
bazel test //submitqueue/orchestrator/controller/build/... //submitqueue/core/publish/...— write order, the base holding only assumed-success dependencies, redelivery republish without rebuild, lost-race double signal, halted batch starting nothing, redelivery healing for live linked paths, and a pin that no path-set write of any kind happens.✅
make fmt,make gazelle,make mocksIssues