fix(cloudflare-hub): keep relay pairings across a Durable Object eviction - #224
Merged
Merged
Conversation
…ing state A relay hub holds its device registry and its pairings in memory, keyed by Connection. A host whose instance can be torn down while its connections stay open has no way to hand either to the successor instance, so a frame arriving on a surviving connection finds no pairing and is dropped. exportConnection renders one connection's state as device-ids and adverts alone, restoreConnections resolves a batch of those back onto the connections a successor still holds, and onConnectionStateChanged tells the host when a connection's value has changed so it has something current to persist.
…tion The instance is discarded after a short idle period while its sockets stay open, so the hub's registry and pairings went with it. A woken instance then saw a relay-data frame on a pairing it had never heard of and dropped it, and neither client could tell: no socket closed, so nobody re-established anything and the sender simply never got its answer. Anything answered within a few seconds beat the eviction, anything slower did not, which is exactly the case a relay matters most for. Each connection's relay state now rides on its own socket as a hibernation attachment, and the hub reassembles itself from ctx.getWebSockets() the first time a woken instance is asked to do anything. The attachment dies with the socket, so a client that goes away while no instance is running leaves nothing to sweep, and it is read synchronously, so the hub is whole before the frame that woke it is routed. The socket handling moves out of the Durable Object class into hibernating-hub.ts, which needs no cloudflare:workers import and can therefore be driven by a test that evicts the hub for real: build a second one over the same sockets and their attachments, which is precisely what the runtime preserves.
live-check.mjs relays as soon as it has paired, so it never leaves the hub quiet long enough to be evicted and cannot see the case that failed in production. The new script pairs, sends one request, then says nothing for well past the eviction threshold before the response goes back, and requires both directions to still work afterwards. Its own header states what a local run against wrangler dev does and does not prove, since workerd there does not evict on production's schedule.
Mearman
marked this pull request as ready for review
September 20, 2026 01:07
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
|
🎉 This PR is included in version 1.59.0 🎉 The release is available on: Your semantic-release bot 📦🚀 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #223
A response relayed through mesh.exadev.io was lost whenever the pairing had been quiet for more than about ten seconds. Confirmed the hypothesis in the issue by reading the code: the hub is a hibernating Durable Object, the instance is evicted after a short idle period while its sockets stay open, and
createRelayHub'spairingsandconnectionDevicemaps are ordinary instance state. A woken instance saw a relay-data frame on a pairing it had never heard of and dropped it, silently, with neither client in a position to notice: no socket closed, so nothing told either side to re-establish anything.Reproduced it before fixing it.
test/hibernation.integration.test.tsbuilds a hub over fake sockets, pairs two clients, then builds a second hub over the same socket objects and their attachments, which is exactly the pair of things the runtime preserves across an eviction. Five of its six cases failed against the old behaviour, the first with client A receiving nothing at all when B answered.Each connection's relay state now rides on its own socket as a hibernation attachment, and the hub reassembles itself from
ctx.getWebSockets()the first time a woken instance is asked to do anything. I chose an attachment over Durable Object storage for lifetime reasons rather than capacity: the runtime drops an attachment when its socket closes, which is exactly when the state stops meaning anything, so there is nothing to sweep when a client goes away while no instance is running. It also reads synchronously, so the hub is whole before the frame that woke it is routed, and a socket is the only thing that survives an eviction with an identity, so a storage row would have needed an attachment anyway just to say which row belonged to which socket. The 16KiB limit is far beyond a device-id, a few adverts and a pairing list.Adoption resolves the whole set of sockets at once rather than one at a time, because a pairing names its peer by device-id and that peer is a socket the new instance has yet to look at. A pairing whose peer is not among the survivors has genuinely gone and is dropped rather than half-restored. The legacy unaddressed-relay-data route travels in the same attachment, so a client that omits
to-devicestill reaches the peer it did before.wire-mesh-core gains
exportConnection,restoreConnectionsand an optionalonConnectionStateChanged.createRelayHub()keeps its existing no-argument call shape and nothing about the default behaviour changes, so the Node relay server, this repo's own tests and agent-comms' hub-helpers all carry on unchanged. The hub itself still knows nothing about sockets, attachments or Cloudflare: it reports that a connection's state changed and can render it as device-ids and adverts, and where that goes is the host's business.The Durable Object class shrinks to an adapter. Everything it used to do lives in
src/hibernating-hub.ts, which imports nothing fromcloudflare:workersand so can be driven by an ordinary Node test against a fake socket, which is what made the eviction reproduction possible at all.scripts/idle-relay-live-check.mjscovers the caselive-check.mjscannot, since that one relays the moment it has paired and so never leaves the hub idle long enough to be evicted. The new script pairs, sends a request, then goes quiet for thirty seconds before the response comes back. Both scripts pass against a localwrangler dev, but the script's own header says plainly that local workerd does not evict on production's schedule, so a local pass shows only that a long quiet period is survivable, not that an eviction is. That needs a run against the real deployment, which I will do once this is merged and deployed.The stale parts of the hub README went with it: it still described the DO as keeping connections alive for its own lifetime, listed hibernation as deferred work, and pointed at a
src/hub.tsthat no longer exists.