fix(core): bound startup recovery wait in shutdown to 15s - #2252
Open
kiwipaulrob wants to merge 1 commit into
Open
fix(core): bound startup recovery wait in shutdown to 15s#2252kiwipaulrob wants to merge 1 commit into
kiwipaulrob wants to merge 1 commit into
Conversation
core.shutdown() awaited startupRecoveryPromise with no timeout. With a large dirty episode and a slow/flaky LLM, the recovery reflect chain can take minutes, holding shutdown hostage until the systemd kill timer (observed 15 Aug 2026: SIGTERM 08:00:23 -> SIGKILL 08:10:23, 10-minute stop-sigterm wedge). Recovery is resumable: dirty episodes carry rewardDirty.failedAttempts and the periodic rescore re-runs them, so nothing is lost by proceeding after a short grace. The 15s bound still covers the fast init->shutdown SQLite race (issue MemTensor#1808) that the wait was introduced for.
kiwipaulrob
force-pushed
the
fix/shutdown-recovery-timeout
branch
from
August 15, 2026 04:29
cc25406 to
5fd4917
Compare
Collaborator
✅ Automated Test Results: PASSEDAll tests passed (41/41 executed). memos_local_plugin/unit: 41/41. Duration: 12s Branch: |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
core.shutdown()awaitedstartupRecoveryPromisewith no timeout. With a large dirty episode and a slow/flaky LLM, the startup-recovery reflect chain (up to 163 sequential LLM calls at ~2min each) can take minutes — holding shutdown hostage until the systemd kill timer fires.Observed 15 Aug 2026 on a production bridge: SIGTERM at 08:00:23, daemon stuck mid-recovery, systemd
TimeoutStopSec=600expired, SIGKILL at 08:10:23. A 10-minute stop-sigterm wedge from a single unguarded await.Fix
Wrap the recovery wait in the existing
withTimeout()helper (15s):Why 15s is safe:
init → shutdownrace closing SQLite mid-flush. A 15s grace still covers that (recovery of 0–1 episodes finishes in ms).rewardDirty.failedAttempts, and the periodic 10-minute rescore re-runs them. Proceeding after the grace loses no data.handle.shutdown()detaches subscribers and the daemon'sprocess.exit(0)fires — lingering LLM calls can no longer hold the process hostage.Verification
tsc --noEmitcleanTimeoutStopSectightened 600s → 90s on the systemd unit (ops-side belt-and-braces; with this fix graceful shutdown completes in ~20s)