Daily OOM crash-loop: worker RSS far exceeds its own memory budget, then full reindex retriggers on every crash

[jbmx77]

Summary

The daemon repeatedly gets OOM-killed while indexing a mid-size repo (~9k files), and after each kill the next scheduled reindex treats the on-disk DB as invalid and reruns a full reindex instead of resuming/incremental, causing a daily OOM crash loop that has persisted for weeks.

Environment

• codebase-memory-mcp v0.10.8
• Linux x86_64, 15GB total RAM (total_ram_mb=15613)
• Repo being indexed: ~9096 files (pipeline.discover files=9096)

What happened

journalctl -k shows the kernel OOM-killer terminating the codebase-memory process (not --cbm-daemon-internal wrapper, the index worker itself) on 10+ separate days between 2026-07-15 and 2026-08-27, always around the same daily trigger time, with RSS between 5.6GB and 7.1GB:

Aug 27 07:23:31 kernel: Out of memory: Killed process 534979 (codebase-memory) total-vm:15045260kB, anon-rss:7082876kB ...
Aug 26 07:22:37 kernel: Out of memory: Killed process 2007241 (codebase-memory) total-vm:12947920kB, anon-rss:6201692kB ...
Aug 25 07:27:59 kernel: Out of memory: Killed process 3482383 (codebase-memory) total-vm:10850636kB, anon-rss:5667976kB ...
Aug 24 07:25:04 kernel: Out of memory: Killed process 765588 (codebase-memory) total-vm:14661524kB, anon-rss:6118132kB ...
Aug 22 07:21:04 kernel: Out of memory: Killed process 3712834 (codebase-memory) total-vm:13960104kB, anon-rss:5824416kB ...
Aug 21 07:23:59 kernel: Out of memory: Killed process 980081 (codebase-memory) total-vm:15476024kB, anon-rss:6758460kB ...
Aug 20 07:20:57 kernel: Out of memory: Killed process 2410488 (codebase-memory) total-vm:14279952kB, anon-rss:6112296kB ...

Notably the worker's own log reports a much smaller intended budget:

level=info msg=mem.init budget_mb=975 total_ram_mb=15613 source=daemon_worker_cap
level=info msg=parallel.mem.budget total_mb=975 per_worker_mb=121

and the daemon itself:

level=info msg=daemon.start version=0.10.8 memory_budget_bytes=4093071360 physical_job_limit=4 worker_memory_budget_bytes=1023267840
level=info msg=mem.init budget_mb=3903 total_ram_mb=15613 source=ram_fraction

So the process's own accounting caps it at ~975MB/worker (~3.9GB budget total), but actual RSS reaches 5.6–7.1GB — ~1.5-7x over its own stated budget. The budget appears to be advisory/accounting only and isn't enforced (no rlimit/cgroup self-imposed by the process), so it doesn't actually prevent the OOM.

The crash-loop mechanism

After each OOM kill, the in-progress DB write is left as a .stage.<random> file of 0 bytes (confirmed 3 consecutive days: home-ubuntu-hermes-agent.db.stage.2TZ0Qq, ...5QSpI1, ...PIBqTb, one per day, all 0 bytes). The next run then logs:

level=warn msg=pipeline.route path=full reason=invalid_existing_db
level=info msg=pipeline.route path=full

i.e. it treats the corrupted/incomplete DB as unusable and reruns a full reindex of all 9096 files from scratch — reproducing the same memory spike and getting OOM-killed again. This has repeated daily for at least 6 weeks with no self-recovery, because there's no partial/incremental-index fallback and no automatic re-attempt with reduced parallelism/budget after a detected crash.

Suggested fixes

1. Enforce the computed memory budget (e.g. via setrlimit/cgroup self-limit, or backpressure that pauses extraction workers) instead of only accounting it, so the process degrades gracefully instead of triggering the kernel OOM killer.
2. On invalid_existing_db, don't unconditionally do a full reindex — if the failure was itself caused by OOM/crash, retry with a smaller worker count / lower per-file retention before falling back to full.
3. Write the DB more atomically / checkpoint incrementally so a crash mid-index doesn't invalidate everything already indexed.
4. Expose the memory budget/parallelism as user-configurable via codebase-memory-mcp config set (currently only auto_index, auto_watch, ui_* are configurable) so operators on memory-constrained hosts can cap it below the automatic ram_fraction default.

Workaround applied

Deleted the stale .stage files to break the loop, and added MemoryHigh/MemoryMax cgroup limits to the parent systemd units that spawn the MCP daemon, so a future runaway is contained to that unit's cgroup instead of triggering the global kernel OOM killer (which was also killing unrelated sibling services on the same host).

[yohanchatelain]

Confirming this on v0.9.0, so the budget overrun is not a 0.10.x regression — it's at least as old as 0.9.0 (2026-07-08). Different host profile too: 7.5 GB RAM rather than 15 GB, desktop rather than server, with auto_index=true / auto_watch=true / auto_index_limit=50000.

Same signature as your report — the worker's own accounting says one thing:

level=info msg=mem.init budget_mb=1921 total_ram_mb=7685

and the kernel says another. Five kills in 35 minutes:

Aug 28 14:06:19 kernel: Out of memory: Killed process 356873 (codebase-memory) total-vm:8851232kB,  anon-rss:5049924kB
Aug 28 14:16:08 kernel: Out of memory: Killed process 359949 (codebase-memory) total-vm:9460300kB,  anon-rss:5037092kB
Aug 28 14:30:38 kernel: Out of memory: Killed process 361528 (codebase-memory) total-vm:9702044kB,  anon-rss:5304044kB
Aug 28 14:38:04 kernel: Out of memory: Killed process 362276 (codebase-memory) total-vm:9716644kB,  anon-rss:3963484kB
Aug 28 14:41:21 kernel: Out of memory: Killed process 363764 (codebase-memory) total-vm:13061928kB, anon-rss:6186904kB

That's 3.9–6.0 GB RSS against a stated 1921 MB budget, i.e. 2.0–3.1× over — consistent with the 1.5–7× you measured, on a host with half the RAM.

The escalation mechanism, for the "unrelated sibling services" part

You noted the global OOM killer was also taking out unrelated services. I can name the mechanism, and it's worse than it looks for anyone running this from a terminal.

An MCP server started by an editor/agent is a child of the terminal emulator's systemd scope, so that's the cgroup the kernel records for the victim:

task_memcg=/user.slice/user-1000.slice/user@1000.service/app.slice/snap.alacritty.alacritty-83004ebf….scope,task=codebase-memory

systemd scopes default to OOMPolicy=stop. So when the kernel OOM-kills the indexer, systemd does not stop at the one process — it tears down the entire scope:

Aug 28 14:42:51 systemd: snap.alacritty.alacritty-83004ebf….scope: Killing process 363555 (zellij) with signal SIGKILL.
Aug 28 14:42:51 systemd: snap.alacritty.alacritty-83004ebf….scope: Failed with result 'oom-kill'.

That SIGKILL is the zellij server — the terminal multiplexer hosting the whole session. Every shell, editor and long-running job in that terminal dies with it, and the user sees only Lost connection to the Zellij server, with nothing in zellij's own log but a clean shutdown. Nothing points at the indexer. It took a full journal correlation to find it.

So the blast radius isn't just sibling daemons: on a workstation it's the user's entire interactive session, and the true cause is invisible from where the failure surfaces.

Mitigation worth adding alongside MemoryMax

Your MemoryHigh/MemoryMax workaround caps the runaway. This one stops the escalation, and is a useful complement — a type-wide drop-in at ~/.config/systemd/user/scope.d/10-oom-continue.conf:

[Scope]
OOMPolicy=continue

A type-wide drop-in is necessary because these scopes are transient with random UUID names, so there's no stable unit name to target. After systemctl --user daemon-reload it applies to already-running scopes as well as new ones. An OOM-killed child then dies alone instead of taking the terminal with it.

For the cap itself, running the server in its own scope contains it and keeps it out of the terminal's cgroup entirely:

systemd-run --user --scope -p MemoryMax=3G -p MemorySwapMax=512M -p OOMPolicy=continue -- codebase-memory-mcp

This is transparent to MCP stdio — the JSON-RPC handshake comes back clean.

+1 on suggested fix #4

Confirming that on 0.9.0 config list exposes only auto_index, auto_index_limit, auto_watch, ui-lang — no memory or parallelism knob, so on a memory-constrained host there's no in-product way to cap the budget below the automatic ram_fraction default. Everything above is out-of-band systemd work that every affected operator has to rediscover independently. Even without full enforcement (#1), a settable ceiling would let people stay inside the product.