Skip to content

fix(core): bound startup recovery wait in shutdown to 15s - #2252

Open
kiwipaulrob wants to merge 1 commit into
MemTensor:mainfrom
kiwipaulrob:fix/shutdown-recovery-timeout
Open

fix(core): bound startup recovery wait in shutdown to 15s#2252
kiwipaulrob wants to merge 1 commit into
MemTensor:mainfrom
kiwipaulrob:fix/shutdown-recovery-timeout

Conversation

@kiwipaulrob

Copy link
Copy Markdown

Problem

core.shutdown() awaited startupRecoveryPromise with no timeout. With a large dirty episode and a slow/flaky LLM, the startup-recovery reflect chain (up to 163 sequential LLM calls at ~2min each) can take minutes — holding shutdown hostage until the systemd kill timer fires.

Observed 15 Aug 2026 on a production bridge: SIGTERM at 08:00:23, daemon stuck mid-recovery, systemd TimeoutStopSec=600 expired, SIGKILL at 08:10:23. A 10-minute stop-sigterm wedge from a single unguarded await.

Fix

Wrap the recovery wait in the existing withTimeout() helper (15s):

await withTimeout(startupRecoveryPromise, 15_000, "startup_recovery_shutdown_timeout");

Why 15s is safe:

  • The wait was introduced for issue fix: [memos-local-plugin] Dreaming background processing starves Gateway event loop, causing WebSocket timeouts #1808 to prevent a fast init → shutdown race closing SQLite mid-flush. A 15s grace still covers that (recovery of 0–1 episodes finishes in ms).
  • Recovery is resumable by design: dirty episodes carry rewardDirty.failedAttempts, and the periodic 10-minute rescore re-runs them. Proceeding after the grace loses no data.
  • After the timeout, handle.shutdown() detaches subscribers and the daemon's process.exit(0) fires — lingering LLM calls can no longer hold the process hostage.

Verification

@Memtensor-AI Memtensor-AI added area:plugin OpenClaw & Hermes status:in-progress Someone or AI is working on it | 人工或 AI 正在处理 labels Aug 15, 2026
core.shutdown() awaited startupRecoveryPromise with no timeout. With a
large dirty episode and a slow/flaky LLM, the recovery reflect chain can
take minutes, holding shutdown hostage until the systemd kill timer
(observed 15 Aug 2026: SIGTERM 08:00:23 -> SIGKILL 08:10:23, 10-minute
stop-sigterm wedge).

Recovery is resumable: dirty episodes carry rewardDirty.failedAttempts
and the periodic rescore re-runs them, so nothing is lost by proceeding
after a short grace. The 15s bound still covers the fast init->shutdown
SQLite race (issue MemTensor#1808) that the wait was introduced for.
@kiwipaulrob
kiwipaulrob force-pushed the fix/shutdown-recovery-timeout branch from cc25406 to 5fd4917 Compare August 15, 2026 04:29
@Memtensor-AI

Copy link
Copy Markdown
Collaborator

✅ Automated Test Results: PASSED

All tests passed (41/41 executed). memos_local_plugin/unit: 41/41. Duration: 12s

Branch: fix/shutdown-recovery-timeout

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:plugin OpenClaw & Hermes status:in-progress Someone or AI is working on it | 人工或 AI 正在处理

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants