fix(lancedb): detect schema type drift and add cascade rebuild recovery by gloryfromca · Pull Request #354 · EverMind-AI/EverOS

gloryfromca · 2026-07-24T05:35:25Z

Problem — the root cause behind #337

verify_business_schemas (the startup guard) only compared column names against the current schema. A column whose on-disk Arrow type had drifted — name unchanged — sailed through and detonated much later, deep inside merge_insert, as the notoriously opaque:

RuntimeError: lance error: LanceError(IO): Execution error: Spill has sent an error, poll.rs:291:44

This is exactly #337: an episode.subject_vector column left as string (or null) by an older build, while the current schema declares a 1024-d fixed_size_list. Writing a real vector into that column is a pyarrow type error that lance wraps as the "Spill" IO error.

Verified

Reproduced byte-identically: a drifted subject_vector (string and null) column + a real 1024-d vector through the real LanceRepoBase.upsert() path yields the exact cascade worker fails to index episodes: LanceDB Spill has sent an error (IO) #337 error; a correctly-typed table accepts the same records.
No lancedb version produces the bad column (checked 0.13→0.34: ≤0.19 raise TypeError, 0.20+ render the correct fixed_size_list), and lance never silently adds a mistyped column — so the drift originates from a build that declared the field differently, and the guard is what should have caught it at startup.

Changes

1. Type-aware verify_business_schemas — compare each shared column's Arrow type against schema.to_arrow_schema() (the exact schema get_table builds tables from, so a healthy table never false-positives). On drift the startup error now names the drifted column and both types instead of the runtime Spill.

2. everos cascade rebuild — the safe recovery from a drifted/corrupt index:

drops the business LanceDB tables and re-indexes from markdown (the source of truth);
skips the verify guard (whose whole point is the drift it would otherwise trip on at startup — chicken-and-egg);
unlike rm -rf ~/.everos/.index/lancedb, re-populates already-done entries (reset_all clears the cascade queue so every md file re-enqueues) — a bare rm of the lancedb dir leaves the queue marked done and the index comes back empty;
unlike rm -rf ~/.everos/.index, preserves unprocessed_buffer (messages received but not yet extracted — not rebuildable from md).

Tests

test_verify_schemas.py — healthy tables pass (false-positive guard); subject_vector as string/null raises with type_drift; missing column raises (unchanged); drop_business_tables drops + recreates the correct vector type. The type-drift tests were confirmed to fail without the fix (old guard DID NOT RAISE).
test_md_change_state.py — reset_all clears every row regardless of status; noop on empty.
test_cascade_cli_integration.py — cascade rebuild runs despite a drift that trips a normal command's startup verify, recreates the table with the correct type, and re-indexes md from scratch.

Full unit suite + make lint (ruff, import-linter, datetime, openapi) green.

Fixes #337.

verify_business_schemas only compared column names, so a column whose on-disk Arrow type had drifted (name unchanged) slipped through and detonated later inside merge_insert as an opaque LanceError(IO) "Spill has sent an error" (#337). Now compare each shared column's Arrow type against schema.to_arrow_schema() — the exact schema get_table builds tables from, so a healthy table never false-positives. Reproduced #337 byte-identically: an episode.subject_vector column left as string or null by an older build, plus a real 1024-d vector on upsert, yields the exact crash. No lancedb version (0.13-0.34) renders Optional[Vector] as a non-vector type, so the startup guard is what should catch it — not the runtime. Add `everos cascade rebuild` as the safe recovery: it drops the business LanceDB tables and re-indexes from markdown, skipping the verify guard (which the drift would otherwise trip on startup). Unlike removing only .index/lancedb it re-populates already-done entries (reset_all clears the cascade queue); unlike removing all of .index it preserves unprocessed_buffer (messages not yet extracted). Fixes #337.

Add the `everos cascade rebuild` command to the runbook, CLI, and how-memory-works docs. Correct the old recovery guidance: a bare `rm -rf .index/lancedb` leaves md_change_state marked `done`, so the scanner skips those files and the index comes back empty — the runbook previously claimed a full repopulation that does not happen. `cascade rebuild` is the safe path (re-populates done entries, preserves unprocessed_buffer). Also document that verify now checks column types.

arelchan · 2026-07-27T12:04:07Z

在 1.1.3 上独立复现,确认这个 PR 的诊断和恢复路径都对。补几个数据点,顺便催一下合并。

1.1.3 仍复现,且已发布版本都还没带上这个修复

everos 1.1.3 + lancedb 0.33.0(0.34.0 也一样)/ pyarrow 25.0.0 / Python 3.12.13 / macOS arm64
表结构确认:episode.subject_vector 磁盘上是 string,而 Episode.to_arrow_schema() 要求 fixed_size_list<float>[1024] —— 与 cascade worker fails to index episodes: LanceDB Spill has sent an error (IO) #337 结论一致
现存 19 行的 subject_vector 全部为 null,符合"该列在一直为空时被建成 string"的推断
最新 release 1.2.0(7-24)不含本 PR(仍 OPEN),所以现在踩到的人依旧只能看到那个不可读的 Spill has sent an error

没有本 PR 的排查成本(正是它要消灭的)

在不知道 #337 的情况下,从症状定位到根因需要逐一排除:磁盘空间(剩余 237G)、文件描述符(116/1048576)、TMPDIR 可写性、双实例抢锁、everos 版本(1.1.3 与 1.2.0.dev1 均失败)、lancedb 版本(0.33.0 与 0.34.0 均失败)、数据损坏(读取/扫描/向量检索全正常)、批量大小(10/50/200 全部成功)。

最具误导性的一点:手动对同一张表做 merge_insert 是成功的(因为复制的老行 subject_vector 为 null),只有写入带真实 subject 向量的新行才炸。这让人很容易误判成"表是好的、是并发/资源问题"。

本 PR 的启动期类型比对能把这一整段直接变成一条明确报错,价值很大。

手动恢复已验证可行(给还没等到合并的人)

在 1.1.3 上按本 PR 的思路手动执行,结果正常:

停服务,备份 .index
按 Episode.to_arrow_schema() 重建 episode 表,把原有行拷回(subject_vector 全为 null,无损)
清掉卡住的 md_change_state(status in ('processing','failed'))
重启

结果:episode 19 行 -> 88 行(补回了积压 18 天的记录),其中 69 行带 subject_vector,cascade_worker_unrecoverable 从 330 次 -> 0 次,无数据丢失。

这正是本 PR 的 cascade rebuild 要自动化的流程,可以佐证该恢复路径是有效的。

两个本 PR 未覆盖的点

不影响本 PR 合并,单独说明(另开 issue 跟踪):

写入侧 API 仍会谎报成功:cascade worker 处于不可恢复失败时,POST /api/v1/memory/add 返回 200、POST /api/v1/memory/flush 返回 {"status": "extracted"},但实际什么都没落库。本 PR 通过"启动期拒绝启动"堵住了 schema 漂移这一类,但其他原因导致的写入失败仍会被静默吞掉,宿主应用无从察觉。
重试没有上限:单次运行观察到 330 条完全相同的 cascade_worker_unrecoverable kind=episode,只带来噪声和负载。

能否优先合并?目前每个新装/老库用户遇到这个问题,拿到的仍是那条看不懂的 IO 错误。

CascadeOrchestrator dropped the embedder param when embedding became a soft dependency (main); fold the schema-drift integration test onto it. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

gloryfromca · 2026-07-31T02:59:58Z

Folded into #379 (consolidated LanceDB hardening PR, rebased on latest main + validated by the storage soak). Commit preserved there.

* fix(lancedb): reclaim stale versions via write-locked prune The storage soak (48h, sustained churn + fuzz) proved the bundled lock-free `optimize(cleanup_older_than=...)` loses its commit-conflict race against concurrent writes — cleanup ran only 16 of ~250 times, so old dataset versions / FTS orphans piled up and the index dir grew to the 40G disk guardrail and never reclaimed under load. main still had that bundled call. Split the maintenance path: - `LanceRepoBase.optimize()` is now compact-only and lock-free (a commit conflict here is benign — the next beat retries, so it must not stall writers). - `LanceRepoBase.prune(older_than)` runs `cleanup_older_than + delete_unverified=True` **under the per-table write lock**, so no writer is in flight: the Rewrite has the manifest to itself (cleanup completes every beat) and aggressive deletion is safe. It also removes the empty `_indices/<uuid>/` husks cleanup leaves behind (soak: 13061 dirs, 98% empty), offloaded to a thread. - The cascade worker's heavy beat calls `prune()`; the light beat calls `optimize()`. A benign light-beat commit conflict is logged at debug and does not count toward the failure streak or trigger a rebuild. - Prune's retention window (`cleanup_older_than`) is decoupled from the prune cadence and defaulted short (60s). It runs under the write lock, so the window only needs to outlive an in-flight read; keeping it = cadence (300s) left ~2 cadences of superseded full-table copies on disk between beats (soak: transient ~15G/table peaks). 60s reclaims all but the last minute each beat — same live floor (~625MB/table), far lower transient footprint. Result on the re-run soak: disk sawtooths and reclaims to live-data size (~1.3G total) under active load — vs run1 stuck at 40G until writes stopped — with 0 crashes / 0 OOM / 0 stuck cleanups over 48h. Cascade projection health is now observable: - `CascadeOrchestrator.health()` -> `CascadeHealth`, combining the worker's in-memory signals (drain-loop failures, unrecoverable count, optimize streak, prune staleness) with the SQLite queue summary. - `GET /health` gains a typed `cascade` readiness block. `healthy` reflects operational health only (drain / optimize / prune); `failed_permanent` (files awaiting `cascade fix`) is a data-quality backlog reported as an informational count that does NOT flip `healthy` — otherwise the signal would sit red permanently. The scanner-side retry cap and the `_MAX_TOTAL_RETRIES` budget already on main handle re-enqueue storms, so no duplicate is added here. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(deps): pin lancedb to >=0.34.0,<0.35.0 The previous open-ended `>=0.13.0` let any environment float to an untested release, including 0.32-0.34 which carry a compaction offset-overflow regression (lance-format/lance#7653) that stalls version cleanup and grows the index dir without bound. - Floor 0.34.0: the current resolved version; runs safely thanks to the with_position=False FTS workaround shipped in #336. Verified that data written by lancedb 0.32.0 (lance v6) reads correctly under 0.34.0 (lance v8), so existing deployments upgrade cleanly. Never widen the floor below 0.34 -- older lance cannot read v8-format data. - Ceiling <0.35.0: 0.35 embeds lance-rust v9 (large encoding jump, not yet stable-released); it must pass the soak harness before we allow it. Resolved version is unchanged (still 0.34.0); this only tightens the declared constraint. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> (cherry picked from commit 94f9aa6) * fix(search): bridge agent-kind metadata into the agentic doc contract The agent AGENTIC path (`agent_case` / `agent_skill`) fed recall candidates straight into `aagentic_retrieve`, whose `_format_docs` (LLM sufficiency / multi-query prompt) reads `metadata["episode"]` as a `{subject, content}` dict plus a ms-epoch `timestamp`. Agent-kind rows carry their body in the recaller's `text_field` and time as a datetime, so `_format_docs` raised `TypeError: Candidate ... has no episode dict` and `POST /api/v*/memory/search` returned 500 for any `owner_type=agent` + `method=agentic` request. Mirror the episode path's bridge: reshape agent candidate metadata into the everalgo doc contract before `aagentic_retrieve`, and revert it before DTO shaping so the agent shapers still see a datetime timestamp. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> (cherry picked from commit bb9a3a5) * test(search): regenerate search seed to current linkage + migrate e2e The committed `search_seed` fixture and several search e2e tests were written for the pre-1.5 memcell fact-linkage model. Current extraction links atomic_facts to episodes via `parent_id == episode.entry_id` (parent_type="episode"), and user_memory clusters store episode entry_id members — so the stale fixture made VECTOR/AGENTIC recall and the cluster-narrowing path find nothing, and stale assertions checked an old error code. - Regenerate `search_seed/*` from a fresh corpus in the current entry_id format; facts now bridge across multiple episodes (richer agentic / hierarchical-eviction coverage). - Fix `_dump_search_seed.py` sampling: pick episodes that host facts first and keep facts by episode entry_id, so re-dumps stay coherent. - Migrate e2e tests to the entry_id model (hierarchical-eviction, session/timestamp filters, cluster seeding helper) and update the filter-error assertion to the current `INVALID_INPUT` code. - Provision `ome.toml` in the full-app pipeline fixture (the OME config reloader requires it; strategies are code-registered so the packaged default suffices), unblocking corpus regeneration. Full search e2e suite now green (49/49). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> (cherry picked from commit 44cd9ce) * fix(lancedb): detect schema type drift and add cascade rebuild recovery verify_business_schemas only compared column names, so a column whose on-disk Arrow type had drifted (name unchanged) slipped through and detonated later inside merge_insert as an opaque LanceError(IO) "Spill has sent an error" (#337). Now compare each shared column's Arrow type against schema.to_arrow_schema() — the exact schema get_table builds tables from, so a healthy table never false-positives. Reproduced #337 byte-identically: an episode.subject_vector column left as string or null by an older build, plus a real 1024-d vector on upsert, yields the exact crash. No lancedb version (0.13-0.34) renders Optional[Vector] as a non-vector type, so the startup guard is what should catch it — not the runtime. Add `everos cascade rebuild` as the safe recovery: it drops the business LanceDB tables and re-indexes from markdown, skipping the verify guard (which the drift would otherwise trip on startup). Unlike removing only .index/lancedb it re-populates already-done entries (reset_all clears the cascade queue); unlike removing all of .index it preserves unprocessed_buffer (messages not yet extracted). Fixes #337. * docs(cascade): document cascade rebuild and correct recovery guidance Add the `everos cascade rebuild` command to the runbook, CLI, and how-memory-works docs. Correct the old recovery guidance: a bare `rm -rf .index/lancedb` leaves md_change_state marked `done`, so the scanner skips those files and the index comes back empty — the runbook previously claimed a full repopulation that does not happen. `cascade rebuild` is the safe path (re-populates done entries, preserves unprocessed_buffer). Also document that verify now checks column types. * chore(rebase): adapt #354 integration test to soft-embedding main CascadeOrchestrator dropped the embedder param when embedding became a soft dependency (main); fold the schema-drift integration test onto it. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * test(cascade): freeze monotonic clock in prune-staleness health tests Both prune-staleness health tests fabricated `_started_at = time.monotonic() - (ALERT + 100)`, assuming monotonic() is a large value. On a fresh CI runner monotonic() is only ~100-180s, so the subtraction went negative, the source clamped the baseline to 0, and staleness read back as ~130s < 900s — failing on CI while passing on long-lived dev boxes where monotonic() is huge. Freeze the monotonic clock via monkeypatch so staleness is deterministic regardless of the runner's boot uptime. Source logic is unchanged; only the tests are made hermetic. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * test(cascade): wait for stable terminal state in rename scenarios `_wait_path_done` reached a terminal status, slept a 0.1s settle window, then asserted the status was still terminal — which contradicted its own docstring ("absorb any last-second re-enqueue"). A rename's delete event or an atomic-replace echo can flip a done row back to `processing` inside that window, so on a slow CI runner the assert fired ("flipped back to processing after reaching done"), failing test_rename_cross_owner_keeps_frontmatter_owner intermittently (seen on the 3.13 job). `make integration` runs without `--reruns`, so a single flake fails the whole job. Wait for a terminal state that *survives* the settle window instead: absorb a transient re-enqueue by waiting for terminal again, still bounded by `deadline` so a row that never settles surfaces as a timeout. Pre- existing flake on main, unrelated to the prune change; the scenario's real assertions (row counts, frontmatter owner) are untouched. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(cascade): cross-process prune safety + backfill reclaim fix Review of #379 found two P0s plus P1/P2s, all verified against the code: - P0-1: cascade backfill still called the removed optimize(cleanup_older_than=…) kwarg → TypeError swallowed by the best-effort try/except → backfill silently skipped all compaction + reclaim (the exact bloat this PR fixes). CI stayed green because the test double kept the stale signature. Fix: call optimize() + prune(0) at the call site; make the fake mirror the real signature so the drift can't hide again; pin prune in the backfill tests. - P0-2: prune ran delete_unverified=True guarded only by an in-process asyncio lock, but the runbook promises `cascade sync` is safe alongside a live server — and the CLI's first optimize beat does prune, in a separate process. It could delete files the daemon is mid-commit on. Fix: switch prune to delete_unverified=False. Measured to reclaim identically on churned tables (both collapse superseded versions ~97%); True only additionally deletes in-flight/dangling files — exactly the corruption vector. No cross-process lock needed; the write-lock/commit fix (the real reclaim win) is unchanged. - P1-3: /health called orch.health() (6 SQLite aggregates) with no guard → a locked/full/migrating DB would 500 the liveness probe and restart the container. Wrap it: unhealthy readiness + reason, HTTP stays 200. - P1-4: rebuild drops + recreates tables; a live daemon holds cached handles pointing at the dropped dataset. Runbook now says stop the server first — the one cascade command unsafe alongside a live server. - P2: narrow _is_benign_commit_conflict to the "commit conflict" phrase (a bare "retryable" swallowed unrelated recoverable errors); add a timeout around the prune cleanup so a hung lance call can't wedge the write lock; correct two stale docstrings. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs: fix schema-recovery guidance + drop dead internal doc refs Follow-up to the review; two doc issues neither the review nor the fix commit caught: - The schema-drift startup error's docstrings (verify_business_schemas and LanceDBLifespanProvider) still described the recovery as `rm -rf ~/.everos/.index/lancedb` — which the runbook explicitly calls the WRONG recovery (it leaves the cascade queue `done`, so nothing re-indexes and the index comes back empty). The raised error already points to `everos cascade rebuild`; align the docstrings to match. - 15 dangling references to an internal numbered design-doc set (12_/13_/16_/17_*.md) that was never shipped to this repo. Point the schema-recovery ones at docs/cascade_runbook.md; drop the rest (pure provenance in table/component docstrings) while keeping the substance. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs: align maintenance docstrings with split optimize/prune API Two docstring residuals from the #379 review's P2 list: - _run_optimize_once still described the pre-split bundled heavy beat ("same work plus cleanup_older_than ... older than one cadence"); the heavy beat now calls prune() under the write lock and the retention window is decoupled from the cadence (DEFAULT_OPTIMIZE_PRUNE_RETENTION_SECONDS). - _restore_shaper_metadata converts any numeric timestamp, wider than an exact inverse of the bridge; document that this is deliberate (the shaper contract requires a datetime either way). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * style(e2e): hoist fixture-body imports to conftest module top shutil / importlib.resources.files were imported inside the core_pipeline_runtime fixture body; move them to the module top to match the repo import style (#379 review P2). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(cascade): back off a hung prune via a separate attempt clock (N1) The write-lock timeout on prune (review P2) didn't achieve its goal: it was 300s — equal to the prune cadence — and `last_prune_at` only advanced on success. So a hung lance cleanup timed out after 300s, `should_prune` was still true (clock never moved), and the next beat re-pruned ~10s later — pinning the per-table write lock ~97% of the time, the exact write-starvation the timeout was meant to prevent. A real cleanup is milliseconds even on a heavily churned table (measured ~40ms at 320k writes / 100 versions), so the timeout is a pure hang-catcher: lower it to 60s (~1500x headroom, never fires normally, well below the 300s cadence). Split the prune clock so a failed prune backs off without masking the health signal: - last_prune_attempt_at (new) gates scheduling, advanced before the call whether it succeeds or times out → a hung prune waits a full cadence before retrying (light lock-free compaction runs meanwhile), so the lock is held at most ~timeout/cadence ≈ 17% in the worst case. - last_prune_at advances only on success and still drives the prune-staleness health signal, so a persistently failing prune surfaces as degraded instead of being hidden by an advanced schedule clock. Regression test: a raising prune advances the attempt clock (next beat is light, no immediate re-prune) but not the success clock. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: zhanghui <zhanghui@shanda.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>

gloryfromca mentioned this pull request Jul 24, 2026

cascade worker fails to index episodes: LanceDB Spill has sent an error (IO) #337

Closed

gloryfromca force-pushed the fix/verify-schema-type-drift branch from 07b4704 to fe65af2 Compare July 24, 2026 05:55

gloryfromca changed the title ~~🐛 fix(lancedb): verify_business_schemas compares column types, not just names~~ fix(lancedb): detect schema type drift and add cascade rebuild recovery Jul 24, 2026

gloryfromca closed this Jul 24, 2026

gloryfromca reopened this Jul 24, 2026

arelchan mentioned this pull request Jul 27, 2026

写入失败被上报为成功:cascade worker 不可恢复时 flush 仍返回 extracted;且失败无重试上限 #364

Open

gloryfromca mentioned this pull request Jul 31, 2026

fix(lancedb): reclaim stale versions via write-locked prune #379

Merged

gloryfromca closed this Jul 31, 2026

Navigation Menu

Uh oh!

fix(lancedb): detect schema type drift and add cascade rebuild recovery - #354

fix(lancedb): detect schema type drift and add cascade rebuild recovery#354
gloryfromca wants to merge 2 commits into
mainfrom
fix/verify-schema-type-drift

gloryfromca commented Jul 24, 2026 •

edited

Loading

Uh oh!

arelchan commented Jul 27, 2026

Uh oh!

gloryfromca commented Jul 31, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

gloryfromca commented Jul 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Problem — the root cause behind #337

Verified

Changes

Tests

Uh oh!

arelchan commented Jul 27, 2026

1.1.3 仍复现,且已发布版本都还没带上这个修复

没有本 PR 的排查成本(正是它要消灭的)

手动恢复已验证可行(给还没等到合并的人)

两个本 PR 未覆盖的点

Uh oh!

gloryfromca commented Jul 31, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

gloryfromca commented Jul 24, 2026 •

edited

Loading