fix(lancedb): reclaim stale versions via write-locked prune

[gloryfromca]

Consolidated LanceDB hardening PR. Folds in #354 / #355 / #357 (now closed) plus the prune-under-lock storage fix + cascade health, all rebased on latest main. This mirrors the code the endurance soak actually exercised.

Commits

• fix(deps): pin lancedb to >=0.34.0,<0.35.0 (was fix(deps): pin lancedb to >=0.34.0,<0.35.0 #355) — 0.35 embeds lance-rust v9; pin the version the soak validated.
• fix(search): agent-agentic 500 + regenerate stale e2e seed/tests (was fix(search): agent-agentic 500 + regenerate stale e2e seed/tests #357).
• fix(lancedb): detect schema type drift + cascade rebuild recovery (was fix(lancedb): detect schema type drift and add cascade rebuild recovery #354).
• fix(lancedb): reclaim stale versions via write-locked prune — the main change (below).
• chore(rebase): adapt fix(lancedb): detect schema type drift and add cascade rebuild recovery #354's integration test to main's soft-embedding CascadeOrchestrator.

The prune-under-lock fix (why)

A storage-endurance soak (~100h across three runs, sustained churn + fault injection) reproduced the disk-bloat root cause: the bundled lock-free optimize(cleanup_older_than=...) loses its commit-conflict race against concurrent writes, so cleanup ran only 16 of ~250 times and superseded dataset versions piled up until the index dir hit the 40G disk guardrail and never reclaimed under load.

Split the maintenance path:

• LanceRepoBase.optimize() — compact-only, lock-free (a Retryable commit conflict here is benign; the next beat retries, so it must not stall writers).
• LanceRepoBase.prune(older_than) — new; cleanup_older_than + delete_unverified=True under the per-table write lock (no writer in flight → the Rewrite owns the manifest → cleanup completes every beat; aggressive deletion is safe). Also removes empty _indices/<uuid>/ husks (soak: 13061 dirs, 98% empty), threaded.
• Cascade worker: heavy beat calls prune(), light beat optimize(). Benign light-beat conflict → debug, doesn't count toward the failure streak or trigger a fallback rebuild.
• Prune retention decoupled from the prune cadence, defaulted to 60s (runs under the lock → only needs to outlive an in-flight read).

Plus cascade observability: CascadeOrchestrator.health() → CascadeHealth, surfaced as a typed cascade block on GET /health. healthy = operational only (drain / optimize / prune); failed_permanent is an informational count that does not flip healthy.

Soak result

• Disk bounded & fully reclaimable — sawtooths under load, reclaims to live size (~1.3G); 42G → 1.3G in 10 min once writes pause. vs run1 stuck at 40G until idle.
• 0 crashes / 0 OOM / 0 stuck cleanups across all runs. lance reports 2 live fragments / ~625MB per table.

Full write-up (code + conditions + results, with figures): DlabDev wiki → "EverOS LanceDB 存储浸泡测试 — 最终报告".

Tests

make lint + import-linter + dump_openapi.py --check + full unit suite (1882 passed).

Follow-up (not in this PR)

Embedding/vector errors during cascade indexing are currently classified permanent (retryable=False); the soak's fuzz showed a transient embedder blip permanently fails a file. Candidate: reclassify as RecoverableError.

Co-Authored-By: Claude Opus 4.8 noreply@anthropic.com

🤖 Generated with Claude Code

[gloryfromca]

Thanks for the thorough review — every finding was verified against the code and addressed. Resolution summary:

P0

P0-1 — backfill silently lost compaction/cleanup (masked by a stale test double)
_backfill.py still called the removed optimize(cleanup_older_than=…), which raised TypeError swallowed by the best-effort try/except, so cascade backfill skipped all reclaim — the exact bloat this PR fixes. CI stayed green because _FakeRepo kept the old signature. Fixed the call site to optimize() + prune(timedelta(0)); the fake now mirrors the real optimize()/prune() signatures (the drift that hid it), and the backfill tests pin the split. (df142c0)

P0-2 — delete_unverified=True guarded only by an in-process lock
Switched prune to delete_unverified=False. Measured on churned tables, False reclaims identically to True (both collapse superseded versions ~97%); True only additionally deletes in-flight/dangling files — exactly the cross-process corruption vector. So no cross-process lock is needed and the write-lock/commit fix (the real reclaim win) is unchanged. Runbook updated. (df142c0)

P1

P1-3 — /health could 500 on a SQLite hiccup
Wrapped orch.health() in try/except: on failure it reports healthy=false + a reason and keeps HTTP 200, so a locked/full/migrating DB can't flip liveness and restart the container. Added a probe-exception test. (df142c0)

P1-4 — cascade rebuild had no "stop the server" guard
Runbook now states rebuild requires stopping the server (it drops+recreates tables; a live daemon holds cached handles pointing at the dropped dataset). The cascade sync "safe in parallel" note is reworded to explain why it's safe (delete_unverified=False). (df142c0)

P2

• _is_benign_commit_conflict narrowed to the "commit conflict" phrase (dropped the over-broad "retryable" that would swallow unrelated recoverable errors).
• Added a timeout around the prune cleanup so a hung lance call can't wedge the per-table write lock.
• Corrected the stale _run_optimize_once / _prune_stale_seconds docstrings. (df142c0)

Follow-up (fb3dac7) — the schema-drift error's docstrings still described recovery as rm -rf …/.index/lancedb, which the runbook flags as the wrong recovery (it leaves the cascade queue done, so nothing re-indexes and the index comes back empty). Aligned them to everos cascade rebuild, and removed 15 dangling references to an internal numbered design-doc set that was never shipped to this repo.

Validation — run5 (targeted, on df142c0)

A 3h concurrent run exercising what single-process endurance can't:

• Cross-process integrity — 375 concurrent cascade sync prunes alongside the daemon: CLEAN (schema verify passed, all tables reopen + count OK, md↔lance counts consistent). Only 4 transient retryable commit-conflicts across 3h, each self-healed (optimize_failure_streak never left 0, /health stayed healthy).
• Disk bounded (delete_unverified=False) — du sawtoothed to ~4G under churn, collapsed to 328MB on write-stop, then flat for 2+ days (zero drift).
• P0-1 on real hardware — optimize() + prune(0) on a live table, no TypeError.
• Zero OOM / crashes / add-write errors throughout.

One non-blocking observation from run5: a heavy-beat (prune) retryable commit-conflict is currently counted toward optimize_failure_streak (unlike the light beat). Under 2s-interval hammering it triggered only 4× in 3h and self-healed every time, never nearing the alert threshold — so it's a non-issue in practice, but treating the heavy-beat retryable conflict as benign too would be a clean optional follow-up. Happy to fold it in if you'd like.