Skip to content

feat(wiki-engine): add WASM tree-sitter AST track for code knowledge graph - #304

Open
m0Nst3r873 wants to merge 1 commit into
Tencent:mainfrom
m0Nst3r873:feature/ast-code-knowledge
Open

feat(wiki-engine): add WASM tree-sitter AST track for code knowledge graph#304
m0Nst3r873 wants to merge 1 commit into
Tencent:mainfrom
m0Nst3r873:feature/ast-code-knowledge

Conversation

@m0Nst3r873

Copy link
Copy Markdown
Collaborator

What & Why

The code knowledge graph extractors (src/wiki-engine/code-knowledge/) were purely regex / line-based. That has two concrete problems:

  1. False-positive edges. Dependency edges were built by path-substring matching (file.includes(importPath)), so an import "./user" wrongly links to both user.ts and user-repo.ts (any path containing user).
  2. No call relationships. Only DEPENDS_ON (imports) existed; call sites were invisible.

This PR adds a real AST track using web-tree-sitter (pure-WASM, no native toolchain), ported from the team-wiki reference implementation, for TypeScript/JavaScript, Python, and Go. It runs alongside the existing regex heuristic track (which still covers Java/Rust/config), and AST results win on merge.

How it works

  • New ast/ module: WASM parser registry (async one-time init), tree-sitter queries, symbol/import/call-site walk, import & call resolvers, and fact/edge adapters. Emits precise file-to-file DEPENDS_ON / REFERENCES edges tagged source: "code-ast" with confidence weights.
  • Dual-track with graceful fallback: if the WASM runtime can't load, or TEAMAI_SKIP_AST=1 is set, extraction falls back to heuristic-only and records an AST_UNAVAILABLE gap.
  • code-graph: AST relation facts build precise edges directly, instead of being re-fuzzed through the path-substring matcher.
  • enrich: manifest edges now preserve real AST relation/source provenance (deterministic rank-based merge) instead of hardcoding DEPENDS_ON / code-heuristic. Combined with upstream's resolveImportToModule resolver.
  • Pinned versions: web-tree-sitter@0.25.10 + tree-sitter-wasms@0.1.13 — the only ABI-14-compatible pair (0.26 rejects these grammars). Both are pure-JS deps resolved from node_modules at runtime; no .wasm files are bundled into dist/.

Accuracy (measured)

On a fixture with a naming-collision decoy (user.ts real target, user-repo.ts decoy that nobody imports):

Mode Edges False positives Precision
Regex only (TEAMAI_SKIP_AST=1) 3 1 (→ user-repo.ts) 67%
AST enabled 2 0 100%

Test Plan

  • npx tsc --noEmit — clean
  • npx vitest run — 2074/2074 pass (includes new src/__tests__/ast-extract.test.ts, 11 cases covering TS/Python/Go extraction, merge precedence, gap recording, and enrich provenance)
  • npm run build — success
  • E2E: real teamai codebase --extract on a multi-language sample repo produces code-ast DEPENDS_ON + REFERENCES edges; TEAMAI_SKIP_AST=1 falls back to heuristic-only + writes AST_UNAVAILABLE gap
  • Packaged form: npm pack + install into a clean consumer — all .wasm files resolve from node_modules, AST track engages via the installed CLI (verifies npx/npm distribution works)

Docs

README and usage-guide updated in both EN and zh-CN; new TEAMAI_SKIP_AST env var documented.

Compatibility notes for downstream consumers

  • recall graph-boost already registers REFERENCES in RELATION_WEIGHT and uses maxBoost semantics, so the new edges participate correctly with no change and can't inflate scores via duplicate edges.
  • GraphEdgeSource already contains code-ast; no schema change.

🤖 Generated with Claude Code

…graph

The code knowledge graph extractors were purely regex/line-based, which
produced false-positive dependency edges (path-substring matching) and had
no notion of call relationships. This adds a real AST track using
web-tree-sitter (pure-WASM, no native toolchain) for TypeScript/JavaScript,
Python, and Go, ported from the team-wiki reference implementation.

- New ast/ module: WASM parser registry (async one-time init), tree-sitter
  queries, symbol/import/call-site walk, import & call resolvers, and
  fact/edge adapters. Emits precise file-to-file DEPENDS_ON / REFERENCES
  edges tagged source:"code-ast" with confidence weights.
- Dual-track: runs alongside the regex heuristic track (which still covers
  Java/Rust/config); AST facts win on merge. Falls back to heuristic-only
  and records an AST_UNAVAILABLE gap when the runtime is unavailable or
  TEAMAI_SKIP_AST=1.
- code-graph: AST relation facts build precise edges instead of being
  re-fuzzed through the path-substring matcher.
- enrich: manifest edges now preserve real AST relation/source provenance
  (deterministic rank-based merge) instead of hardcoding DEPENDS_ON /
  code-heuristic.
- Pinned web-tree-sitter@0.25.10 + tree-sitter-wasms@0.1.13 (only ABI-14
  compatible pair; 0.26 rejects these grammars).
- Docs (README + usage-guide, EN/zh-CN) and unit tests added.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant