P3: self-generated feature wiki + grounded retrieval & evaluation pipeline#11
Merged
Conversation
… eval Major changes: - Extract architecture knowledge from AGENTS.md to architecture.json (30 entries, source-verified) - AGENTS.md stripped to pure rules (tool order, answer format, navigation strategy) - implement: class skeleton mode (10K+ → ~500 tokens), ClassName.methodName support - implement: enforce search/graph before implement (session tracking) - search/graph/implement: navigation hints in responses - graph: architecture context injection from architecture.json - grep: sorted by relevance, limited to top 10 - Callee skeletons removed (graph(down) replaces at 1/10th cost) Bug fixes: - retriever.ts: callee skeleton bug (objects treated as strings) - Remove 5 unused deps, unused exports, stale params Results: - L1: 25/34 (unchanged, no artificial inflation) - L2 tokens: 69K → 29K avg/question (-58%) - L2 implement: 3,070 → 544 avg tokens (-82%) - Claude judge: 6 GOOD, 16 ACCEPTABLE, 9 WEAK, 3 WRONG (65% usable) Eval renames: tool-eval → layer1-tool-eval, agent-eval → layer2-agent-eval New: compare.ts generates 3-way comparison report, comparison-report.md
Remove unused eval layers (layer0-baseline, layer1-tool, compare) and their scripts. Add executive-report.md with manual quality assessment: 70% of answers rated GOOD/ACCEPTABLE. Update gemini-answers with latest eval run.
…swers with improved formatting - Add layer0-baseline-eval and layer1-tool-eval scripts and logs - Add compare.ts eval utility - Update all gemini answer logs with refined content - Update layer2-agent-eval with latest results - Replace executive-report.md with executive-report-flash.md
Adds the previously-untracked agy-session-runbook so the entire docs/ (proposal, eval log, diagrams, runbook) lives in one commit and stays recoverable after the next commit deletes the folder.
…ring; remove dead code - layer1: add LCS chain-order score (ordered Qs only, partial credit, report-only, not a pass gate) alongside precision/F1/Recall@k/MRR - layer0: fix to use loadTestcases (grouped-v1); was crashing on the new format - layer2: drop unused locals - retriever: remove unused getContext() + orphaned imports - add judge.ts (shared Claude judge), load-testcases.ts (grouped loader), agy-session-eval.ts (Track B) - testcases.json -> grouped-v1 (core/supporting/ordered)
benchmark-notools -> layer0-answers (pairs with layer0-baseline-eval.md) gemini-answers -> layer2-answers (pairs with layer2-agent-eval.md) benchmark -> reference-claude (gold reference, not a layer)
- src/eval: gen-gemini-nomcp / gen-gemini-mcp-selfloop (generators), eval-1..4 (one report per comparison), utils/ (eval-util, load-testcases, testcases.json) - drop layer0/1/2 naming and the API judge (judge.ts); semantic is Claude-judged - indexer: mtime/size fast-path in hasher so unchanged files skip read+md5 - package.json: gen:nomcp/gen:mcp + eval:1..4 scripts
Accuracy fixes (registry.ts, skeleton.ts): - impact mode no longer import-scopes event/pubsub edges, so event-driven dependents (callbacks.add / Meteor.subscribe) survive the blast radius - search runs a content-grep fallback before returning "no results" (stops the agent giving up on a wrong entry-symbol guess) - graph gains a `file` param + centrality-based root selection, so colliding symbols (Streamer, sendMessage) traverse the canonical definition - skeleton indexes module-top-level statements, so top-level callbacks.add registrations (notifications listening on afterSaveMessage) become visible Index loading (index.ts, local-db.ts): - incremental update patches only changed mappings instead of re-reading all ~7600 (minute-scale rebuild -> sub-second) - parallel batched reads + progress bar for the cold full rebuild - compact JSON for .global_index.json (drop pretty-print) - ensureIndex() unifies the load path across server + eval scripts Infra: - OUTPUT_DIR -> output.nosync to exclude the regenerable index from iCloud sync (the sync storm was throttling index reads) Eval (single run, new vs baseline; see logs/): - eval-3 semantic: baseline FAIL 4 -> 1; all 4 cut-targeted weak spots lifted (federation/apps-engine/impact-streamer/impact-settings), new-27 -> PASS - eval-2 deterministic: 29/34 (no regression), index 18.5k -> 19.2k symbols - eval-1 token regressed (Cut 3 surfaces larger correct subgraphs + token/loop workstream not yet done) — tracked as next step
search() was pure lexical fuzzy over symbol *definition sites*, so one query could only ever surface the seed's own file (recall ceiling ~1/N). Reframe retrieval as navigation: seed lexically, expand along the call graph (depth-2, incl. event/pubsub edges), then rank the neighbourhood by proximity + lexical + intra-cluster cohesion + centrality - hub penalty. Wire the ranked cluster into the search tool so the agent gets the whole subsystem on every query (incl. exact-symbol ones). Rank graph(down) children by relevance before truncating instead of arbitrary slice(0,6). Make eval-2 honest: promote single-query retrieval (R@10>=0.3) to the pass gate; demote feed-every-symbol substring recall to a sanity floor. eval-2: P@5 24.1->27.6, R@10 41->49, R@20 44->58, recall-miss 26->11, file-recall 97->99.5%; honest combined gate 24/34.
Add dependency-free load-env util so .env overrides stale shell-exported keys (e.g. an old GEMINI_API_KEY in ~/.zshrc that shadowed the file). Import it in both gemini generators and refresh regenerated answer logs and eval reports.
Generators (gen:nomcp / gen:mcp): temperature=0 greedy decode so answers are reproducible run-to-run — kills the ±3 PASS noise that made single-run comparisons meaningless. gen:mcp now records "Files Seen In Tool Results" per answer to power the synthesis split. eval-3: add a synthesis-recall split (retrieval-recall = tools surfaced it → synthesis-recall = agent then wrote it; the gap, not retrieval, is the lever), a deterministic FROZEN rubric verdict gated on core-spine coverage (not Claude's over-cited file list), and a hand-judged MANUAL verdict layer in logs/semantic-verdicts.json (answerHash ties each verdict to the answer it judged; stale ones are flagged loudly). 34/34 manually judged: PASS 16 / PARTIAL 14 / FAIL 4 (2 FAILs are gen-time rate-limit errors). eval-1: inline a token-/answer-size-matched naive (search-only) baseline as a control — answers "is the lift the graph or just more tokens?" (graph +17 pts). eval-2: promote chain-order LCS from report-only to a pass gate for ordered Qs. eval-1/2/3 now delete their own report before regenerating (no stale file on crash). Regenerated answers + reports.
- report.ts joins the eval-1/2/3 sidecars into one retrieval-funnel report: token efficiency, cumulative funnel (index -> retrieval -> synthesis), per-testcase detail, and a per-type summary (type = diagnostic lens). - eval-1/2/3 emit machine-readable *-data.json sidecars; eval-2 adds Recall@50 (the single-query ranking ceiling). - logs/ reorg: reports/ (per-eval .md appendices) + data/ (generated json, gitignored); report.md + semantic-verdicts.json stay at top. - README: retrieval-funnel section + metric dictionary. - package.json: refresh runs the report as its last step.
Reports: strip hardcoded editorial conclusions from the eval-1/2/3 and
unified-funnel generators — emit metrics/tables/definitions only, no
interpretive takeaways. Regenerate the four report .md files accordingly.
Indexer: extract cross-layer & dynamic-dispatch edges so previously
disconnected seams connect in the graph —
- REST: rest.get/post('/vN/x') <-> addRoute('x') / API.vN.get/post('x')
- service bus: broadcast('x') <-> onEvent('x')
- streamer: new Streamer('x') <-> sdk.stream('x')
- slash commands: slashCommands.add('x')
- class heritage: extends/implements -> type edges (service<->interface)
GENERATOR_VERSION 7->10 to force a full skeleton rebuild.
Verified: client<->server seams (e.g. livechat/room) now connect; R@10
unchanged (49.4%) — bottleneck is ranking, not reachability.
Weight BFS proximity by edge type instead of raw hop count — strong structural edges (call/jsx/new) keep a symbol close, weak `type` refs push it far, dynamic-dispatch edges sit between. Same 2-hop neighbourhood (recall unchanged); refines ranking so real-call neighbours outrank type-only references among equidistant candidates. eval-2: R@5 40.4->42.8 (+2.4), R@10 49.4->50.3 (+0.9), R@20/pass unchanged.
Regenerate gen:nomcp / gen:mcp answers against the cross-layer-edge index (GENERATOR_VERSION 10) and rebuild the eval-1 / eval-3 reports so they reflect the new agent runs (prior reports were on pre-edge answers).
The file-overlap auto rubric under-rates capability (right mechanism, different files). Replace the stale in-pipeline manual-verdict apparatus with a Claude- judged semantic verdict written to logs/reports/eval-3-mcp-agent-vs-claude-semantic-judgment.md (PASS 21 / PARTIAL 10 / FAIL 3, vs the auto rubric's PASS 10 / PARTIAL 14 / FAIL 10). - eval-3-mcp-vs-claude.ts: remove loadSidecar/answerHash/STALE-warning/manual column; keep the deterministic hard / synthesis-split / auto-verdict diagnostics. - report.ts: drop the now-dead MEAS classifier; label funnel rows as per-file pooled (they differ from eval-2's per-testcase mean R@k by design). - delete logs/semantic-verdicts.json (stale hand verdicts, superseded).
…, eval → gen/tools/token/report Server: registry becomes a thin MCP adapter; one tool per file (tools/), one algorithm per file (engine/: seeds, expand, down, up, source, common). New plan tool + intent.ts recipes set SESSION.intent; graph collapses direction+mode into move=expand|down|up with intent-driven defaults. implement renamed to details. Channel ablation (chanOn/ABLATE/ONLY), prefix/score search channels, AGENTS.md constitution and CodeRetriever all removed; ranking formula untouched. Eval: gen.ts (--mode=nomcp|mcp, --oracle intent), tools.ts (deterministic gate, pools search+graph(expand) at the new tool boundary), token.ts, report.ts (joins tools/token sidecars + answer-derived gather/synth + manual verdicts.md). Gate: tools eval 24/34 — per-testcase identical to the pre-refactor baseline (logs/reports/eval-2-BASELINE-pre-refactor.md); R@k/MRR bit-identical.
…seded reports & dead code - verdicts.md now carries the 34 manual verdicts (PASS 21/PARTIAL 10/FAIL 3) judged on the current answers — still valid, answers unchanged since judgment - delete superseded logs: eval-1/2/3 reports, ablation (ABLATE/ONLY) set, old semantic-judgment doc (all recoverable from git history), stale data sidecars - regenerate logs/report.md via the new pipeline (eval:token + report) - eval-util: drop unused LOGS export, unexport internal symMatch
…N.md - README gains a Roadmap section (phase 1 done / phase 2 = seed recall on the 6 retrievalRecall=0 concept queries) and a note on the frozen tools baseline - REFACTOR_PLAN.md removed — the plan is executed; recoverable from git history - rename eval-2-BASELINE-pre-refactor.md → tools-BASELINE-pre-refactor.md (the eval-2 script is now tools.ts; content byte-identical, still the frozen anchor)
…) post-refactor Re-ran gen:mcp (786k tok, 34/34, 0 err) and gen:nomcp on the refactored plan/search/graph/details engine; refreshed all answers. Re-judged every verdicts.md row against the new answers: 22 PASS / 9 PARTIAL / 3 FAIL (was 21/10/3). Headline ~flat but 8/34 rows flipped (4 up: claude-06, claude-07, new-17, new-19; 4 down: tour-07, tour-08, tour-10, new-10) — free-tier sampling noise, per the frozen refresh discipline. Regenerated reports: tools 24/34 (deterministic gate — refactor did not regress tool capability), token MCP 45% vs naive 12% vs no-MCP 14% (agent +33 pts), report funnel indexed 100% -> top50 55% -> surfaced 51% -> written 40% (bottleneck OK15 / G2 11 / G3 4 / G1-recall 3). gen.ts: log the full error message instead of truncating to 100 chars.
…rch/graph/details Design for reframing logs/report.md around the refactored pipeline (approach A x C: keep the cumulative funnel + add per-tool scorecards; PLAN routing as a lightweight parsed-accuracy diagnostic). No reindex, no GENERATOR_VERSION change, no new eval/Gemini run. Structure approved; pending spec review.
Untrack the report-redesign spec and gitignore docs/ — design specs stay on disk for ongoing work but are not pushed (per user preference). The spec file itself is unchanged locally.
4th `mode` column on all 34 rows: agent failure mode for the 12 non-PASS (no-pivot 5 / wrong-subsystem 2 / engine-unrankable 2 / sloppy-source 1 / dropped-synth 1 / gave-up 1), PASS rows —. Feeds report.ts §3 behavior diagnosis + agent/engine split. agent-fault 10 / engine-fault 2.
Add resolvedIntent (SESSION.intent after the loop) to AnswerRecord and write a `## Plan\n\nintent: <x>` section, so report.ts can read the routing decision untruncated (tool-call args are capped at 100 chars, which was clipping the intent). Forward-looking: current answers use report.ts's prefix/classifier fallback; next gen:mcp run is clean.
Widen verdicts parser to 4 cols (id|verdict|mode|reason); add resolveIntent (## Plan line → truncated-args prefix → classifyIntent fallback); compute per-row resolvedIntent/routeOk/failureMode/fault/bindingTool on J. Semantic mode from verdicts.md wins; mechanical fallback via R@50/gather/synth. Render unchanged this commit (backbone only).
…orecards §3 (new): agent-fault vs engine-fault diagnosis — 12 non-PASS split into 10 agent-fault / 2 engine-fault, per-mode fix-levers. §4 funnel relabeled to SEARCH+GRAPH rank / GRAPH loop / SYNTH with ENGINE/agent-fault annotations. §5 (new): per-tool scorecards (plan route 21/34, search, graph, synth) with binds# + fix-lever. §6 detail: route ✓/✗, mode, binding columns; binding now = the semantic failure mode's pipeline stage (consistent with mode). Old G1/G2/G3 classifier retired. §1/§2 byte-identical (regression guard).
…race rendering + gold scope/recall + drift guard) + unit tests
…m run trace + refresh kept in sync
…s core ⭐ (correctness pushed down into walk) + tests
…ly walk catches it / no hit + which step first hits) + tests
…the paid judge / default reads cache; PASS/PARTIAL/FAIL per question + summary) + tests
…move eval:tools script / refresh reference + delete orphaned metrics.md
… replaced by report.md)
…--semantic to use), remove the standalone CLI / verdicts.md output + drop judge from refresh
…ic, zero references / no script / no tests)
… script, zero references / no script / no tests)
…eport correctness + semantics; retired eval:tools / judge-CLI / metrics / verdicts)
…andidateModules · summarize summarizeOne orphan · verify --rewrite empty stub + unused MODEL_TIERS import)
…only production code) + rewrite relative imports + test glob → tests/**
…outside src/ after the move)
…wo copies were identical)
….ts (write 527→326; straighten out verify dependency: switch from write.js to citations.js)
…s → extract-edges.ts (extractCalls+extractHeritage+4 constants; skeleton 585→303, edge logic becomes its own module)
…nderWalk/renderAgent + fix scope duplication (name carries the score, drop the duplicate reason line) + tests
… steps · stop reason; dedupe stop reason on the top line; fold STOP into the group header) + tests
…sections + scope/walk use bullets/nested bullets; cut per-round affinity noise; core⭐ shown only on hit) — fixes rendering mashed into one blob
…es a short path (last two segments, to distinguish same-named functions/methods sendMessage.ts) + tests
…ty + ↳ sub-entries for reach/core hit including 0), carried via nested bullets + tests
…DME with current state + strip DeepWiki name repo-wide
This session:
- gen:mcp recovery: under the old key gemini-2.5-flash returned 404 (model retired), wiping out all 34/34 answers;
switched to a new key (model restored) and regenerated 34 agent answers, with new-20-proxify re-run for a single question (tool loop hadn't wrapped up)
- semantic evaluation: report --semantic real run → PASS 13 / PARTIAL 19 / FAIL 2 (scope 17/34, recall 56%)
- README fully aligned: removed DeepWiki-crawling narrative, deleted dead gen:nomcp command, fixed the refresh chain;
added wiki/ layer · wiki-site · wiki:gen 5-step pipeline; added latest semantic + Phase 3
- strip DeepWiki name: README/wiki-site frontend comments/write prompt/gate/package.json/prose-generated content
- gitignore: exclude .envrc · logs/reports/ · data/.wiki-prose-hashes.json
Merged in existing workspace changes (P3 wiki-generation):
- gen.ts sendWithRetry (backoff retry on Gemini transient 404/429/5xx, without affecting determinism)
- indexer/modulegraph · engine/down · wiki-site/app.js · scripts/wiki-serve · deleted wiki/extractors/prose.ts
- data/wiki-map · wiki-prose · wiki-meta/{deployment,development}
…+ build-site assembly script
- scripts/build-site.mjs: wiki-site/ + data/{wiki-map,wiki-prose}.json → dist/ (the viewer fetches via /data/ absolute paths, so the publish root must contain data/)
- vercel.json / netlify.toml: buildCommand=build-site, output directory dist
- package.json adds build:site; gitignore excludes dist/.vercel/.netlify
- verified locally: serve dist, both / and /data/*.json return 200, JSON is valid
…tomy, remove Roadmap - How it works: end-to-end two-stage throughline — offline (ts-morph extracts skeleton → extract-edges 12 edges → GLOBAL_INDEX → summarize+embed summary vectors) + online four tools (plan→search→graph:expand uses RRF to fuse structural score with semantic cosine → details uses ts-morph to read real source). Highlights the "zero runtime external dependency" north star, the self-generated wiki doubling as a routing layer, and the RRF two-signal complementarity motivation (see the selfgen-wiki / semantic-ranking design docs, only writing what's landed) - correct the stale assertion "No embeddings — pure graph arithmetic" → graph-primary + semantic cosine RRF fusion - Evaluation: rewritten as a "semantic pass/fail / trace pinpoints the culprit" dichotomy; delete the Retired bullet - delete the whole Roadmap section; clear the last remaining DeepWiki wording
… mistake report.md / verdicts-latest.json / wiki-verify.md brought into git (a snapshot of evaluation results, including the latest semantic 13/19/2 baseline); removed logs/reports/ from .gitignore. Note: report.md changes on every run, so after a refresh it must be recommitted to keep GitHub up to date.
…ason text of trace/walk/entry - report.ts rendering text, walk.ts walk stop reasons/affinity reason, entry.ts seed/page selection reason, trace.ts fallback reason all translated CN→EN (code comments remain in Chinese) - report.ts stopLabel includes-matching synced to walk.ts's new English keywords (coupling fix) - zero-API re-run of trace + report; numbers unchanged (34/34 · scope 17/34 · recall 56% · semantic 13/19/2) - regenerated the English logs/reports/report.md (zero residual Chinese)
…glish-only - Comments, JSDoc, log/error messages, and test titles across src/ and tests/ - README, scripts/, package.json, .gitignore - wiki-site: drop the zh dictionary and language toggle, render English-only UI
|
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR delivers Phase 3 (P3): a self-generated, citation-grounded feature wiki for an indexed codebase, together with the retrieval engine and low-/zero-API evaluation harness that support it. The wiki is now produced end-to-end from the local index (no external wiki fetching).
What's included
Wiki generation pipeline (
src/wiki/)generate.tsorchestrates a 5-step pipeline: outline → guide tree → write → citations/verify → diagrams.outline.tsplans chapters with deterministic file anchoring and coverage self-checks.write.tswrites per-page prose with enforcedpath:Lstart-Lendcitations, grounded against the index (unverified references are dropped).guide.ts,tree.ts,families.ts,taxonomy.ts) with MECE + stability gates.diagram.ts,src/wikimap/mermaid.ts).Retrieval engine (
src/server/engine/)bge-smallembeddings, affinity-guided directed walk with self-stop, entry-page/seed selection, and a candidate/architecture map.Indexer (
src/indexer/)Evaluation (
src/eval/)trace+ gold-check reporting (zero-API), an optional semantic judge, deterministic generation, and a per-question retrieval funnel.Static wiki site (
wiki-site/)Notes
npm test→ 223/224 passing. The single failing test (tests/indexer/modulegraph.test.ts—featureKey … → ee:livechat-enterprise) is a pre-existing issue on this branch, unrelated to the work in this PR.