feat(wiki-engine): add WASM tree-sitter AST track for code knowledge graph - #304
Open
m0Nst3r873 wants to merge 1 commit into
Open
feat(wiki-engine): add WASM tree-sitter AST track for code knowledge graph#304m0Nst3r873 wants to merge 1 commit into
m0Nst3r873 wants to merge 1 commit into
Conversation
…graph The code knowledge graph extractors were purely regex/line-based, which produced false-positive dependency edges (path-substring matching) and had no notion of call relationships. This adds a real AST track using web-tree-sitter (pure-WASM, no native toolchain) for TypeScript/JavaScript, Python, and Go, ported from the team-wiki reference implementation. - New ast/ module: WASM parser registry (async one-time init), tree-sitter queries, symbol/import/call-site walk, import & call resolvers, and fact/edge adapters. Emits precise file-to-file DEPENDS_ON / REFERENCES edges tagged source:"code-ast" with confidence weights. - Dual-track: runs alongside the regex heuristic track (which still covers Java/Rust/config); AST facts win on merge. Falls back to heuristic-only and records an AST_UNAVAILABLE gap when the runtime is unavailable or TEAMAI_SKIP_AST=1. - code-graph: AST relation facts build precise edges instead of being re-fuzzed through the path-substring matcher. - enrich: manifest edges now preserve real AST relation/source provenance (deterministic rank-based merge) instead of hardcoding DEPENDS_ON / code-heuristic. - Pinned web-tree-sitter@0.25.10 + tree-sitter-wasms@0.1.13 (only ABI-14 compatible pair; 0.26 rejects these grammars). - Docs (README + usage-guide, EN/zh-CN) and unit tests added. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What & Why
The code knowledge graph extractors (
src/wiki-engine/code-knowledge/) were purely regex / line-based. That has two concrete problems:file.includes(importPath)), so animport "./user"wrongly links to bothuser.tsanduser-repo.ts(any path containinguser).DEPENDS_ON(imports) existed; call sites were invisible.This PR adds a real AST track using
web-tree-sitter(pure-WASM, no native toolchain), ported from the team-wiki reference implementation, for TypeScript/JavaScript, Python, and Go. It runs alongside the existing regex heuristic track (which still covers Java/Rust/config), and AST results win on merge.How it works
ast/module: WASM parser registry (async one-time init), tree-sitter queries, symbol/import/call-site walk, import & call resolvers, and fact/edge adapters. Emits precise file-to-fileDEPENDS_ON/REFERENCESedges taggedsource: "code-ast"with confidence weights.TEAMAI_SKIP_AST=1is set, extraction falls back to heuristic-only and records anAST_UNAVAILABLEgap.code-graph: AST relation facts build precise edges directly, instead of being re-fuzzed through the path-substring matcher.enrich: manifest edges now preserve real AST relation/source provenance (deterministic rank-based merge) instead of hardcodingDEPENDS_ON/code-heuristic. Combined with upstream'sresolveImportToModuleresolver.web-tree-sitter@0.25.10+tree-sitter-wasms@0.1.13— the only ABI-14-compatible pair (0.26 rejects these grammars). Both are pure-JS deps resolved fromnode_modulesat runtime; no.wasmfiles are bundled intodist/.Accuracy (measured)
On a fixture with a naming-collision decoy (
user.tsreal target,user-repo.tsdecoy that nobody imports):TEAMAI_SKIP_AST=1)user-repo.ts)Test Plan
npx tsc --noEmit— cleannpx vitest run— 2074/2074 pass (includes newsrc/__tests__/ast-extract.test.ts, 11 cases covering TS/Python/Go extraction, merge precedence, gap recording, and enrich provenance)npm run build— successteamai codebase --extracton a multi-language sample repo producescode-astDEPENDS_ON+REFERENCESedges;TEAMAI_SKIP_AST=1falls back to heuristic-only + writesAST_UNAVAILABLEgapnpm pack+ install into a clean consumer — all.wasmfiles resolve fromnode_modules, AST track engages via the installed CLI (verifiesnpx/npm distribution works)Docs
README and usage-guide updated in both EN and zh-CN; new
TEAMAI_SKIP_ASTenv var documented.Compatibility notes for downstream consumers
recallgraph-boost already registersREFERENCESinRELATION_WEIGHTand usesmaxBoostsemantics, so the new edges participate correctly with no change and can't inflate scores via duplicate edges.GraphEdgeSourcealready containscode-ast; no schema change.🤖 Generated with Claude Code