feat(run-engine): fair virtual-time scheduling for the concurrency-key dequeue - #4367
Open
1stvamp wants to merge 50 commits into
Open
feat(run-engine): fair virtual-time scheduling for the concurrency-key dequeue#43671stvamp wants to merge 50 commits into
1stvamp wants to merge 50 commits into
Conversation
|
Contributor
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
WalkthroughAdds opt-in CK virtual-time scheduling to the run queue. The feature introduces environment and engine configuration, queue key helpers, Redis Lua commands for fair enqueue, dequeue, and nack handling, virtual-time state with TTL and cleanup behavior, and updated Redis command typings. New integration suites cover ordering, fairness, batching, concurrency, registration, garbage collection, disabled-mode compatibility, and Redis command overhead. Design, rollout, limitation, and research documentation are also included. 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
1stvamp
added
enhancement
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
1stvamp
force-pushed
the
feat/ck-virtual-time-scheduling
branch
from
July 24, 2026 17:48
bc6dc8d to
99eab9d
Compare
@trigger.dev/build
trigger.dev
@trigger.dev/core
@trigger.dev/python
@trigger.dev/react-hooks
@trigger.dev/redis-worker
@trigger.dev/rsc
@trigger.dev/schema-to-json
@trigger.dev/sdk
commit: |
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
1stvamp
force-pushed
the
feat/ck-virtual-time-scheduling
branch
3 times, most recently
from
July 27, 2026 14:15
f90585f to
9b34c9a
Compare
1stvamp
force-pushed
the
feat/ck-virtual-time-scheduling
branch
from
July 27, 2026 17:33
9b34c9a to
0abcf84
Compare
Member
Author
|
Benchmark: flag OFF vs ON. Relative numbers from a single local homelab box (production-like multi-cluster topology, not production; not prod scale).
Full method, scenarios, and tables: results-2026-07-27.md |
1stvamp
marked this pull request as ready for review
July 27, 2026 17:33
1stvamp
force-pushed
the
feat/ck-virtual-time-scheduling
branch
from
July 27, 2026 17:37
0abcf84 to
71e71cc
Compare
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
Member
|
@1stvamp this looks great! The one thing I'd be curious about is how do these changes effect the following:
How do both of those grow/react to changes in cardinality (e.g. all of a sudden there is a queue with 10k different concurrency keys). |
Member
Author
@ericallam good question, I'll do some testing. |
1stvamp
force-pushed
the
feat/ck-virtual-time-scheduling
branch
from
July 28, 2026 11:31
073b881 to
5722cdc
Compare
Member
Author
|
@ericallam ran it on the homelab box (dedicated redis, flag OFF vs ON, relative numbers only). Short version: memory is linear in key count, CPU barely moves and doesn't grow with cardinality.
Full tables and method: results-cardinality-2026-07-28.md. Shout if you'd like a bigger cardinality or a longer window. |
ericallam
previously requested changes
Jul 28, 2026
ericallam
left a comment
ericallam
left a comment
Member
There was a problem hiding this comment.
This all looks good! Just needs some cleaning up before we can merge. I think the whole e2e testing/bench thing shouldn't be included, along with the benchmark results and plan files and all that stuff. It should ideally be just the code and the changeset and the CI tests
Member
Author
💯 will do |
Member
Author
|
Stripped the e2e/bench harnesses, results, plans and design docs, so the branch is just the scheduler code, the CI tests and the server-changes note now. |
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
1stvamp
force-pushed
the
feat/ck-virtual-time-scheduling
branch
from
July 29, 2026 01:27
66a0d45 to
4c1fc25
Compare
Keeps the change to the scheduler code, the CI tests, and the server-changes note. The benchmark harnesses, e2e project, results, plans, and design references were only ever local validation aids and don't belong in the repo.
…me floor A concurrency-key variant that has queued work but nothing ready yet, which is what every nack with a retry backoff produces, stayed registered in ckVtime holding its old low tag. The floor is the lowest stored tag, so it froze there while the keys actually being served advanced. New keys register at the floor, so a key that arrived later started well below the established ones and won every pass-1 slot until it caught up, which is the starvation the feature is meant to remove. The dequeue path now de-registers a variant when it has work but none of it is ready, alongside the existing GC for variants with no work at all. It stays in ckIndex, so pass 2 still serves it in age order once its head is ready, and it rejoins the fair order at the current floor on its next enqueue, nack or serve. Idle keys no longer hoard priority credit either. Measured with a control against a treatment on the real dequeue path: with one future-headed variant present the floor stayed at 0 while served keys reached 25, and a newcomer took 20 of the next 20 serves. With the fix the same run matches the control, newcomer 5 of 20. Reported by Devin on #4367.
…ants The floor only tracked the lowest tag on record, so any registered variant that could not be served held it there: one sitting at its own concurrency ceiling, or one whose head is not ready yet. The keys actually being served advanced past it, and since new keys register at the floor, a later arrival started underneath the incumbents and took the fair pass until it caught up. The floor now also rises to the lowest tag that was servable on the call. Pass 1 walks candidates in ascending tag order, so that is a safe lower bound. The repair from the lowest tag on record stays, because both routes only ever raise it and it still recovers a floor that was lost while ckVtime survived. Also adds the fairness scenario the suite was missing. None of the six scenarios nacked or used future-scored messages, so a stalled variant never existed and this class of bug could not show up. In the window after it lands the latecomer now takes 5 of 20 serves, against 12 of 20 without the fix.
Reverts the de-registration added earlier on this branch. Dropping a variant from ckVtime when it had work but nothing ready stranded it: pass 1 is the only reader of ckVtime, and pass 2 is skipped whenever pass 1 fills the batch, so on a queue busy enough to keep filling it the variant was never looked at again. A blind review measured one sitting unserved for over two thousand calls after its head became ready, and every nack backoff produces exactly that shape, so a single steady key could hold up another key's retries indefinitely. That is worse than the floor pinning it was meant to address. The floor advance from servable variants handles both cases on its own, so the de-registration bought nothing. It now also only takes its bound from pass 1. Pass 2 picks candidates by message age, so its tag implies nothing about the entries it skipped, and letting it move the floor stepped over registered variants that were servable and simply never visited, confiscating their credit on the next serve. Adds the regression test the earlier tests were missing. They proved the variant was evicted but never that it came back, which is the half that was broken.
At the moment the flag is flipped, :ckVtime is empty and every already-queued variant is unregistered. The first dequeue has an empty pass 1, so pass 2 serves and registers up to actualMaxCount variants. From the next call on, pass 1 can serve one message per registered variant and actualMaxCount is often small, so pass 1 fills the batch off that cohort alone. Pass 2 was gated on dequeuedCount < actualMaxCount, so it never ran again, and the rest of the backlog stayed invisible until a registered variant fully drained or an enqueue or nack happened to land on it. A key that gets no further work has no other route into the fair order. Same reachability shape 1a6d1a5 fixed for registered-but-unservable variants, applied to the unregistered cohort. It also covers a mixed deploy, where an instance with the flag still off enqueues through the non-vtime command, and a :ckVtime that expired while ckIndex lived. Pass 2 now always runs. When the batch is already full it registers the variants pass 1 could not see, at the floor, instead of serving them, so the next call's pass 1 leads with them. Serving is still capped at actualMaxCount, so no serve happens that the old gate would have refused, and the fairness scenarios are byte-identical: ckSkew, ckTrickle, ckSybil, ckBalanced, ckManyKeys, ckHeavyIdle and ckStalledNewcomer all report the same numbers as before. Op cost is one extra fixed op per call. The pass-1 window read doubles as a free membership set, so nothing is registered twice, and the registrations are collected into a single variadic ZADD NX rather than one call each. Measured on the op-count budget test: 11.62 ops per dequeue over the flag-off path, against 10.90 before. The budget comment now counts 8 fixed ops rather than 7. Devin's suggestion of reserving a batch slot for pass 2 does not fix it. Pass 2 walks ckIndex in age order, so its one reserved slot always lands on a variant that is already registered and never reaches the cohort that is not. Measured against the new tests: identical to no fix at all, 12, 16 and 12 calls. Reported by Devin on #4367.
The scenario said a bounded first-serve delay was fine without asserting any bound, so nothing stopped that delay growing. It now pins the measured values: the light key is first served on step 9 with the flag on against 72 with it off, and cardinality above the pass-1 window costs no throughput (drain 79 on, 81 off). The harness has no wall-clock wait and no randomness, so those figures are exact; the assertions carry a little slack for tie-break churn only.
… idle polls writing Pass 1 now steps over a variant whose head is scheduled in the future without spending one of its window slots, so a retry storm across enough keys can no longer fill the window with variants that cannot be served and freeze the virtual-time floor. The variant stays registered and stays scanned, which is what keeps it reachable; only the budget is spared. The scan is capped at twice the window, so a wider block still degrades to pass 2's age order. A dequeue that serves nothing now persists nothing. Both things that block would write are re-derivable: minServableTag is only set inside a successful serve, and discovery only runs once the batch is full, so the floor read-repair is recomputed from ckVtime on the next call anyway. Refits the two tests whose premise these change: the freeze test now pins the residual beyond the scan cap, and the floor test pins that a zero-serve call persists nothing while the repair still lands on the next serving call.
Carries the fix from #4628 into the three CK scripts this branch adds, which do not exist on main and so could not be covered there. A concurrency key of '*' renders a variant name identical to the wildcard member the master queue uses for the base queue, and the unguarded transition cleanup then removed the entry the rebalance had just written, stranding every concurrency key on that queue. The pre-existing scripts are fixed in #4628; this is the same one-line guard applied to enqueueMessageCkVtimeTracked, enqueueMessageWithTtlCkVtimeTracked and nackMessageCkVtimeTracked.
…r order tryServe marks a variant attempted before the per-key concurrency gate, and pass 2's discovery step skips anything attempted, so a variant that was both gated and unregistered fell through every route: pass 1 could not see it without a ckVtime entry, pass 2's attempt was a no-op behind the gate, and discovery then passed over it. It stayed invisible to the fair pass on every call for as long as the gate held, and only a serve would have registered it. Unregistered only arises where a variant reached ckIndex without a vtime-aware write, which is the rollout case discovery already exists to repair: a backlog queued before the flag went on, an enqueue from an instance that still has it off, or a ckVtime that expired while ckIndex lived. Registration is NX so an already-registered variant keeps the tag it earned, and the state TTL is only written when the ZADD actually registered something, which is the path that can recreate a ckVtime key that expired out from under a live ckIndex. Adds a regression test that fails without the branch, and a second op-count budget covering the all-unservable scan. The existing budget only bounds the servable shape (its fixture acks immediately so nothing is ever gated or deferred), and its comment read as a general worst case, which it is not. Reported by Devin on #4367.
… TTL expiry and dead-letter Only the vtime dequeue removed a variant from :ckVtime. Ack, TTL expiry and the dead-letter path all ZREM a drained variant from ckIndex and left its ckVtime entry behind, with a tag that had stopped advancing, until some later scan happened to visit and collect it. Ack is much the most common of the three: it is what a cancellation of a still-queued run runs through. The limitations note called this bounded and self-healing, and it is, but on two weaker grounds than it claimed. The 24h state TTL is refreshed by every enqueue, nack and serving dequeue, so on a queue that is never quiet for a full day it never fires, leaving floor advance as the only collection route. And that route can be held still by a workload that keeps minting concurrency keys, because each fresh variant registers at the floor and its first serve records the floor as the minimum servable tag. It is also not fairness-neutral: registration is NX, so a REUSED key inherits the stale high tag and is deprioritised, which means a cancel-drain remembers history that a serve-drain forgets. Adds vtime variants of the three commands rather than editing them, so the flag-off scripts stay byte-identical by construction (this is a pure addition to the file). Ack and dead-letter take the key as one more KEYS slot from the call site; the TTL sweep derives it in Lua, as it already derives ckIndexKey, because it discovers the queues it touches inside the script. Tests cover all three paths and fail without the fix. The old stranded-entry test is kept rather than deleted, retargeted at the pre-fix command directly, since GC-on-scan is still load-bearing for an older instance during a rolling deploy. Reported by Devin on #4367.
A concurrency-key variant whose message zset drained was removed from ckVtime, throwing away the virtual-time tag it had accumulated. Its next enqueue re-registered it at the floor, so it came back with full credit. A variant holding a persistent backlog keeps advancing its tag instead, so it lost every pass-1 slot to variants that drain and reset each call, and pass 2 could not help once the batch was already full. Measured on a backlogged variant against five trickle variants: 1 serve out of 600 with the flag on, against 120 with it off, inverting the fairness the feature exists to provide. A parked tag now survives the drain in a sibling :ckVtimeIdle zset, and every registration path (enqueue, enqueue-with-ttl, nack, and the gated-variant branch) starts the variant at max(floor, idleTag) rather than at the floor. The out-of-band drains (ack, dead-letter, TTL expiry) park the tag too. Entries at or below the floor confer nothing, so a single ZREMRANGEBYSCORE per serving call reaps them and bounds the set. Deriving the floor differently was tried first and rejected by measurement: any variant with a tag that never advances, which includes any concurrency-gated key, pins the minimum and defeats it. Backlogged variant now lands on its round-robin share in every shape, including with a pinned-low gated or future-headed variant present, and with more trickle variants than batch slots. Op-count overhead goes from 587 to 641 against a budget of 900. The flag-off Lua is still byte-identical: 7 vtime command variants changed, the other 31 commands hash the same as HEAD.
The at-or-below-floor reap on :ckVtimeIdle is worth nothing while the floor is pinned, and a workload that keeps minting fresh concurrency keys pins it indefinitely: each new key registers at the floor and is served at it, so minServableTag never rises. A resource benchmark caught the set growing by the drain count every round and never shrinking, passing ckIndex in size by round 50 and reaching 12000 entries (1.77MB) over 60 rounds, with only the 24h state TTL bounding it. The mechanism was measured rather than inferred: a probe sampling the floor found it at 0 on every round while the lowest parked tag was 1, so ZREMRANGEBYSCORE could never match. Adds a rank cap, keeping the highest idleMaxEntries tags (default 10000, configurable), which does not depend on the floor moving. Trimming the lowest tags first drops the entries nearest the floor, whose remembered credit is worth least. Verified against an explicit cap of 3000: the set rises to it and stays flat there across 12000 drains with the floor still pinned at 0. The new ARGV is inserted before the metrics gauge arg, which has to stay last because the gauge fragment reads ARGV[#ARGV]. Also drops the node:test describe import from the new test file, which shadows vitest's own under globals:true. That is a wider pattern in this directory and is left alone elsewhere.
…path The idle lookup only matters on the call that actually registers a variant. ZADD NX is a no-op on one that is already registered, and its tag is already correct, so the ZSCORE preceding it was wasted on every enqueue after the first. Doing the ZADD first and the ZSCORE only when it reports an insert takes the common path from two ops to one. The per-registration EXPIRE of ckVtimeIdle went too: the park sites are the only writers that put anything in that key and they set its TTL themselves, so refreshing it on a call that may never write there was pure cost. Applies to all four registration sites: enqueue, enqueue-with-ttl, nack, and the gated-variant branch in the vtime dequeue. Six redis.call per registration becomes four in the common case, five or six on the rarer call that registers. Measured on a saturated benchmark (generator co-located with Redis, 1M invocations per arm, 3 interleaved cycles, Redis CPU/wall 0.97 on every arm, two independent cost measures agreeing to 0.04 usec). Against the flag-off enqueue script at 8.603 usec, the vtime path was 10.903 usec (+26.7%) and is now 10.177 usec (+18.3%), so this removes about 30% of the virtual-time enqueue overhead. That +26.7% independently reproduces the 26/23/24% total-CPU overhead the cardinality benchmark measured by a different method. A probe isolating the block gives the model behind it: roughly 1.38 usec fixed per EVALSHA plus 0.33 usec per redis.call, linear in call count for O(1) commands on small keys. Behaviour is unchanged. Final ckVtime tags are identical across already registered, unregistered, idle above floor, idle below floor and missing floor key, and the starvation suite that asserts exact tags still passes.
… order A concurrency-gated candidate cost two Redis calls: the SCARD that discovers it is gated, and a ZADD NX to make sure it is in the fair order. For anything pass 1 selected the second is a guaranteed no-op, because pass 1 draws its candidates from the ckVtime zset and being in that scan is what registration means. tryServe now takes a knownRegistered flag, so a gated visit from pass 1 costs only the SCARD, the same as the flag-off command. Pass 2 candidates can genuinely be unregistered, and ones outside pass 1's scanned prefix are indistinguishable from those, so both are collected and settled after pass 2 by a single variadic ZADD NX. Its return value is the count it inserted, so the idle-tag correction that stops a drained variant reclaiming full credit only runs when something actually registered. In the steady state nothing does and the whole batch costs that one call. Measured on the fully-gated shape, saturated (generator co-located with Redis, 1M invocations per arm, Redis CPU/wall 0.978-0.987, no state drift): N=1000 flag-off 21.05 before 76.63 (+55.6) after 54.56 (+33.5) -22.1us, 40% N=10000 flag-off 21.09 before 77.99 (+56.9) after 57.52 (+36.4) -20.5us, 36% A batched ZMSCORE reads the same information and measured better, -24.9us and -24.7us for 45% and 43%. It was rejected anyway: it would have been the first thing in this file to require Redis 6.2, and about 3 to 4 usec is a fair price for not raising the floor. The variadic ZADD NX is one call either way; it costs more because thirty skiplist lookups on the write path are dearer than thirty reads. Worth noting the saving per call removed is nearer 0.4 usec than the 0.33 usec measured previously on small keys, because the call being removed is a write against a zset holding thousands of members. Fully gated is the worst case by construction: a dequeue that serves exits earlier. Per-key concurrency limits are ordinary on ck queues, so it is worth having.
…scores The gated-registration path in the vtime dequeue was the one writer that touched :ckVtime without also touching :ckVtimeFloor. Everything else keeps the pair alive together: the enqueue and nack registrations EXPIRE both, and a serving dequeue SETs the floor with its own TTL. That path runs on calls that serve nothing, so the floor-persist block, which is guarded on having served, is skipped. A base queue whose variants are all sat at their per-key ceiling therefore refreshes the tags on every poll while the floor's TTL runs down underneath them. Once it expires the next registration reads GET ckVtimeFloorKey back as '0' and starts a brand-new variant below every established tag, so it leads pass 1 until it catches up. Same hole as the one the enqueue floor-TTL test was added to close, reached by a different path. Reported by Devin on #4367.
… change The note ran to three sentences, advertised a server env var a user cannot set, and described the pass-1 window degradation. .server-changes/README.md asks for a one-line description of behaviour rather than implementation, and says that needing a paragraph usually means you are describing the implementation. Reported by Devin on #4367; this is its suggested wording.
… at the floor A concurrency key seen for the first time registered at the virtual-time floor. The floor only rises to the lowest tag present or the lowest tag pass 1 actually served, so a variant registered at the floor and served at the floor leaves it exactly where it was. A workload minting a previously-unseen key per run (a uuid, a high-cardinality tenant id) therefore pinned the floor at the epoch forever, and any variant that had ever been served sat one quantum above it and lost every comparison to the next arrival. Being served once was a permanent penalty. Measured before, 120 calls at maxCount 5 against 8 fresh keys per call: the backlogged variant took 1 of 600 slots with the flag on, against 120 of 600 with it off. After: 120 of 600, and the floor ends at 119 rather than 0. A variant with a remembered idle tag is unchanged, since the parked tag already carries its history. One with none now joins one quantum behind the highest tag on record instead of at the floor, bounded at floor + arrivalCap (64 quanta by default) so a queue with a long-running leader cannot exile newcomers. Two details are load-bearing and came out of simulation rather than reasoning. Strictly behind the maximum, not level with it: registering at the maximum leaves an unbounded lex-ordered tie cohort under overload and a lex-late variant still starves. And falling back to the idle set's maximum when ckVtime shows nothing above the floor, because drained keys carry their credit out of ckVtime into the idle set, which an earlier draft measured at 0 of 600 for a lex-late hog. Every added call sits inside the branch that the ZADD NX actually inserts on, so an enqueue onto a registered variant costs exactly what it did before: the hot path is untouched by construction rather than by measurement. A registration pays one or two more reads, and skips the idle read when ckVtime already proves credit. Same root cause as the drain-and-re-register starvation fixed earlier on this branch. That was variants RETURNING to a stuck floor and was patched by remembering their tag; this is variants ARRIVING at one, which no amount of remembering can fix because a new key has nothing to remember. Unpinning the floor addresses both. Seven tag assertions move, each because the rule they encode has changed rather than because behaviour regressed; their names and rationale are updated with them. The idle-cap test loses its floor-pinned premise entirely, so it now asserts the floor advancing, which is the fix itself. Reported by Devin on #4367.
Two tests, both for holes the fresh-key work opened or exposed. The first pins the fix itself: a persistent backlog against eight brand-new concurrency keys per call, asserting it keeps pace with the flag-off arm and that the floor actually moves. Before the fix that shape served the backlog 1 time in 600 against flag-off's 120, so the assertion is nowhere near the boundary. The second covers the rank cap, which had quietly lost its only test. The idle set has two bounds, the at-or-below-floor reap and the cap, and the reap was dead while the floor sat pinned, so the existing test asserted a pinned floor to prove the cap was doing the work. Unpinning the floor killed that premise: the test now asserts the floor advances, which is the fix, and no longer says anything about the cap. The cap still matters, because a park only ever writes a tag ABOVE the floor, so anything parked faster than the floor climbs is out of the reap's reach. The new test seeds the idle set directly, far clear of any floor the fixture can reach, so the reap provably cannot be what trims it, and checks the survivors are the highest tags rather than an arbitrary subset. Mutation-checked: raising the configured cap while holding the assertions fixed fails it, so it is not vacuous.
The cap on how far above the floor a brand-new variant may register defaulted to 64 quanta, framed as a guard against one inflated tag propagating and as a bound on transitional spread. A blind multi-model review flagged that a burst larger than the cap piles up on the cap line, and chasing that turned up something worse than the tie cohort they described. While the floor rises about a quantum per call, the cap line at floor + 64q rises with it, and clamped arrivals stack 8 to 40 per quantum instead of the intended one. The serving front crosses that dense band slower than the floor climbs, so a backlogged variant's tag eventually climbs into the band, ties with the crowd, and key-name order decides who runs. Every finite cap therefore collides after O(cap) calls of sustained minting and restores a milder form of the starvation the arrival rule exists to prevent, in the hostile-tenant case specifically, with smaller caps failing sooner. Measured on Redis, 40 fresh keys per call over 300 calls at maxCount 5: a 2000-deep backlog kept 92 of 1500 slots at cap 64 and 293 of 1500 effectively uncapped. A discrete-event model of the same shape predicted 90 and 292 before the run. So the default goes to 2^32, leaving the clamp as a pure sanity bound against a corrupted tag rather than an operational limit. The reasoning and the numbers are in the option's doc comment, because the tempting instinct on reading "cap" is to lower it. A test pins the sustained-minting shape so it cannot drift back. Also documents why pass-2 discovery and the gated-pending batch still register at the floor while the enqueue paths stack. Two reviewers independently read that as an inconsistent application of the rule, which means the invariant needed writing down: those two sites only ever see established work that lost its tag, so "serve next" is right and there is nothing to stack behind, and a tenant cannot mint into them while the flag is on because a flag-on enqueue registers the variant first. Floor registration is for repair. A new enqueue-side path must stack, or fresh keys start entering at the floor again.
A candidate parked at its per-key concurrency ceiling fell out of tryServe returning nil, so pass 1 spent one of its window slots on a variant it could not serve. A gated variant's tag also stops advancing, so it keeps sorting to the front of ckVtime and is revisited first on every call. Enough of them and pass 1 serves nothing, ever, and the scheduler quietly runs on pass 2's age order instead. The original note on this said work conservation still held because pass 2 fills the batch, and that was the reason it was left alone. It does not hold. Where the gated variants are also the oldest, which is the ordinary case since a variant that has been queued longest is likely to be both old and saturated, pass 2's own window fills with the same variants and servable work behind them is reached by neither pass. The test added here starts from that shape and serves nothing at all before the fix, rather than serving in the wrong order. So a gated candidate now reports 'notReady', exactly as a future-scheduled head already did, and pass 1 reads past it without spending a slot. The read is bounded by scanLimit, which is already the cap on how far pass 1 will look. It is not free. On a fully gated call pass 1 now reads to scanLimit instead of stopping at the window: measured 53 Redis operations before and 80 after, every one of the 27 a SCARD, so roughly 10 usec. That is worth paying, because a fully gated call serves nothing either way, while a partially gated one goes from serving nothing to serving in fair order. A second test pins the op count against scanLimit plus the pass-2 window so the read cannot start running away. Reported by Devin on #4367.
… found bare A variant whose head is scheduled in the future reports 'notReady' so pass 1 declines to spend a window slot on it. That was added deliberately and its comment calls it load-bearing, but deleting the report left all four vtime suites green: the behaviour was defended by comment and by nothing else. Found by mutating the production Lua one change at a time and rerunning the suites, on the principle that a green run against a broken invariant is a hole. Seven of eight mutations were caught, some by more than twenty assertions; this was the one that walked through. It is the same shape as the concurrency-gate bug fixed in the previous commit, which is the uncomfortable part: the guard for one half of the problem had no test while the other half was actively broken. The test mirrors the gated one. Enough future-headed variants to fill the pass-1 window, sorted ahead on tag, and a servable variant behind them that fair order says to serve first and age order says to serve last. Verified in both directions: it passes against the real script and fails against the mutation with 'expected old to be owed', which is precisely the silent fall back to age order.
…tion A second mutation audit covered the paths the first one listed as untouched: the enqueue and nack registration, the ack, dead-letter and TTL-expiry drains, and the TypeScript plumbing between them. 14 of 25 mutations were caught, and the 11 green ones collapse into three real holes. The idle park was checked on the dequeue path and nowhere else, so ack, dead-letter and TTL expiry could all stop parking a drained variant's tag, and nack could stop restoring it, with every suite still green. Each test here gives the variant credit first, drains it by one route, then asserts the tag reaches the idle set and comes back on the next enqueue. The dead-letter test also covers its caller. The old one drove the Lua directly, so forcing that branch to the untracked command was green while the same damage done inside the script was caught. Reaching the drain through the public API needs the rollout shape, where a flag-off instance empties the variant queue without collecting it. The TTL enqueue command carries its own copy of the registration block and no test enqueued a run with a TTL under the flag, so all three of its behaviours were unverified. And no test configured a quantum, so the weight dimension had no coverage at all; this adds one at 0.3.
…e tag The gated-candidate block registers its batch with one variadic ZADD NX and then, if that added anything, walks every member of gatedPending applying its parked idle tag. The ZADD reports how many members it added but not which ones, so the correction lands on candidates it did not register. That reaches an already-registered variant whenever the pass-1 scan is truncated, since knownRegistered comes from that scan and it reads only scanLimit entries. A queue with more variants than that pushes registered ones into pass 2 as if they were new. Their idle entry also survives re-registration (the enqueue path reads the parked tag but never deletes it, and it is reaped only once the floor climbs past), so with floor < parked < live the XX write overwrites the live tag with the older one. The variant's clock winds back and it is served ahead of variants that are genuinely due, which is the opposite of what the feature is for and exactly what NX exists to prevent everywhere else. The current score is the discriminator: at the floor means the variant either just registered here or has no credit to lose, and above the floor means it has spent a turn and keeps its tag. Reading it first also skips the idle lookup for the advanced ones, so the branch gets cheaper rather than dearer, and the steady state is untouched because none of this runs unless something registered. Reported by Devin on #4367.
The rewind guard added alongside it was covered, but deleting the idle correction underneath it outright left all 59 vtime tests green, so the behaviour that correction exists for was still held up by a comment. A variant registering from the gated batch has to come back at its parked tag rather than at the floor, or draining under the gate hands it full credit. The comment saying so shrinks to the part the test cannot state, which is why the current score is the thing that tells an already-registered variant apart from one this call just added.
…not say
Comments only, no behaviour change: the non-comment bytes of index.ts are
identical before and after.
The 22 "NEW:" markers went. They meant "added relative to the non-vtime script
this one was copied from", which was useful while the copies were being written
and says nothing once the branch merges. Several carried a sentence of history
with them ("previously only the vtime dequeue removed a ckVtime entry"), which
belongs in the commit that changed it.
The long blocks came down to the part a reader cannot get from the code or from
a test. The gate report had nine lines describing a bug that now has two tests,
the arrival cap option had fifteen including measurements, and pass 2 had
seventeen. What survives in each is the invariant a future change could break
without noticing, most of all that floor registration is for repair only and a
new enqueue-side path has to stack instead.
The registration block appears three times and so did eleven lines explaining
it. Now once, with the other two pointing at it, which also stops the copies
drifting apart in prose while staying identical in code.
Test headers lose their provenance (which review or audit prompted them) and
keep the invariant plus the reason each fixture is shaped the way it is. One of
them also asserted that Lua truncates numbers to integers on the way into
Redis. It does not, measured on the 7.2 container the suites run against, so
that claim is gone rather than reworded.
…ires a run The vtime dequeue is a copy of dequeueMessagesFromCkQueueTracked, and main added a TTL re-registration to that command while this branch was open. A copy cannot pick up a change to its original, so the rebase left the vtime path behind: it drops an expired run from the queue sorted sets and defers to a TTL consumer that has no entry for it, because the entry is removed on first dequeue. The run is then orphaned. Only reachable with the flag on, so nothing shipped, but it would have gone out with the feature. Same four lines as the base command, verbatim; the expired branch is now identical between the two again. Copies drifting like this is a known cost of the vtime scripts being copies, kept deliberately so the flag-off path stays byte-identical to production. The drift is worth a test that pins each copy against its original, which would have caught this the day main merged.
1stvamp
force-pushed
the
feat/ck-virtual-time-scheduling
branch
from
August 21, 2026 14:29
b37a0e7 to
7f01aaf
Compare
Member
Author
|
One thing the rebase turned up. The vtime scripts are copies of the non-vtime ones (7 pairs, ~688 identical Lua lines). Main The copies are deliberate: flag-off runs byte-identical Lua to prod and I can prove that by So I'd rather pin each copy against its original in a test, and drift shows up as a red check. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Off by default.
When many concurrency-key variants share one task queue, the dequeue serves the oldest waiting run first, so one key's large backlog is served to exhaustion while keys queued behind it wait for the whole pile to drain. This adds an opt-in fair order: each key gets a virtual clock, the dequeue serves the smallest clock and advances it, so keys take turns instead of one pile draining. With the flag off, the existing scripts run unchanged.
How it works
:ckVtimeZSET (the virtual clocks) and a monotonic floor.ckIndexkeeps its head-timestamp domain, so time-eligibility, master-queue rebalancing, and every other writer stay untouched (this is what makes it mixed-deploy safe).Testing
Full run-queue suite is green with the flag off (no regression). Fairness is proven on the real batched dequeue path (not just one message per call), plus multi-consumer exactly-once, a per-dequeue op-count budget, and behaviour tests for the floor, tag advance, GC, and registration.
Rollout
Off by default behind
RUN_ENGINE_CK_VTIME_SCHEDULING_ENABLED. Enable on a staging cell, then production; rollback is flipping the flag off (leftover state expires within a day). During a rolling deploy, old instances serve in age order and are folded in by pass 2, so nothing is lost and no run is served twice. Every mutation is a single atomic Lua script, which is what makes those interleavings safe. That atomicity assumes the single-node Redis the run queue actually runs on: it has no cluster-mode setting (every other Redis inenv.server.tshas one,RUN_ENGINE_RUN_QUEUE_REDIS_*doesn't), and the master queue key sits outside the base queue's hash slot exactly as it does in the command this one is modelled on.Known limitations
A few review-flagged edges are bounded and self-heal rather than block the flag: variants drained by ack, TTL expiry, or the dead-letter path aren't removed from the virtual-time set, so a stale low-tag entry heals itself on the very next dequeue while a stale high-tag entry is just inert memory that clears within the 24h state TTL. Ties between variants sitting at the same virtual-time tag (a cold start, or a garbage-collected variant re-registering) break by queue-name lexical order rather than anything meaningful, a pre-existing effect of the old head-timestamp ordering that only affects who is served first, not long-run fairness.
The third edge is the pass-1 window itself. Enqueue and nack register a variant in the virtual-time set whether or not its head message is ready, so variants waiting on a retry backoff or a future start time still take up window slots. If enough of them do that at once to fill the window (
maxCount * RUN_ENGINE_CK_VTIME_WINDOW_MULTIPLIER, 3x by default), the fair pass serves nothing, the floor stops advancing, and every serve comes from pass 2 in today's age order. Work conservation still holds, so this is fairness degradation under a retry storm rather than a stall, and a key that arrives during one registers at the frozen floor and then leads by the virtual time the incumbents accrued while it lasted. Widen the multiplier if it shows up.ckVtime.test.tspins both halves: the degraded pass still serves on every call, and the recovery is bounded.