feat(research): investment-thesis research eval — material-facts-surfaced, a metric that needs investigation not retrieval

[drewstone]

What

Investment-thesis research eval: a domain reframe, a calibrated metric, and a controlled 3-arm topology A/B — the full result, on the branch and written up.

We moved the research eval off ML-paper retrieval — where a single web search already returns the answer, so the metric could only measure collection and the prior topology A/B came back a structural null — onto investment research, where the material facts are buried in 10-K footnotes and genuinely require investigation to surface. That is the domain where a smarter coordinator finally has room to beat blind collection, so the A/B is well-posed.

The three parts of the result

1. The held-out set — tests/eval/investment-thesis-set.ts: 5 public companies, each cutoff >= 18 months ago, 27 material facts across 8 analyst lenses, every fact read live out of the primary SEC EDGAR 10-K, every value knowable at the cutoff (the eventual collapse is recorded for the reader only and is never graded). Grading is a $0, model-free substring check; the checklist is firewalled.

2. Calibration — does the metric discriminate depth? (pnpm test investment-calibration, $0, offline, the binding gate): a deliberately shallow ticker summary scores 1/27 (4%), a deep filings-grounded thesis scores 27/27 (100%) — a +96-point gap, every company clearing the bars (shallow < 30%, deep > 70%). An anti-circularity guard asserts no deep thesis verbatim-embeds the checklist evidence, and calibration itself caught + closed three over-loose Silvergate grader groups before any A/B depended on them. This is the gate the ML deep-question exam passed only in spirit — same discrimination, materially harder domain.

3. The live 3-arm A/B (5 companies × 3 arms × 3 rounds, glm-5.2, $0.36, matched compute):

arm	material facts	cost (5 cos.)
collection (blind)	11/27 (41%)	$0.082
verify/dedup	10/27 (37%)	$0.125
driving (deepen)	16/27 (59%)	$0.157

Honest verdict: no topology significantly beats collection. Driving surfaces the most buried facts (+5 over collection, +18pp) for ~1.9× the cost, but at n=5 the paired-bootstrap CI crosses zero (P(Δ≤0)=0.08) — promising, under-powered, consistent with the project's prior topology nulls. Verify does not beat collection, and on two companies its strict primary-source gate zeroes the KB (autopsied: it correctly rejects the aggregator pages the worker fetched in place of the EDGAR filing). Every dollar is real per-call provenance from RouterClient.usage().

This PR's doc change

Unifies the writeup into one coherent section — docs/results/investment-thesis.md (§1 reframe, §2 calibration, §3-§5 the A/B verdict + autopsy) — folding in the standalone calibration doc so there is one result, one doc, matching the repo's docs/results/*.md convention. References it from the main paper (docs/two-agent-research-ab.md §9.1 + the per-result link list).

Provenance + verification

• Calibration numbers reproducible offline ($0): pnpm test investment-calibration — green.
• A/B numbers transcribed verbatim from the live run recorded in commit 338bc54.
• Per-fact provenance, curation-bias disclosure, and the drop log: docs/eval/investment-material-facts.md.
• Gates on the branch: pnpm typecheck clean, pnpm build success, pnpm lint exit 0 (2 pre-existing warnings in unrelated src/lint.ts/src/wikilinks.ts), pnpm test 184 passed / 12 live-gated skipped.

Limitations

Does not prove driving significantly beats collection (n=5, CI crosses zero). The 27-fact checklist carries a documented curation bias (downside-risk, distressed-name skew). Next rung: expand the held-out set past 5 companies and re-run at n>=24 so a +1-fact/company effect can clear a paired bootstrap; driving is the arm worth funding that test on, verify is not.

[tangletools]

✅ Auto-approved PR — 2bfc0f72

Blanket team auto-approval is enabled for this reviewer service.
The full PR reviewer audit still runs separately and will publish findings if it detects issues.

_{tangletools · auto-approval · reason: blanket_auto_approve · 2026-06-25T13:11:27Z}

[tangletools]

✅ Auto-approved PR — d3cbcb47

Blanket team auto-approval is enabled for this reviewer service.
The full PR reviewer audit still runs separately and will publish findings if it detects issues.

_{tangletools · auto-approval · reason: blanket_auto_approve · 2026-06-25T13:37:02Z}

[tangletools]

✅ Auto-approved PR — 338bc547

Blanket team auto-approval is enabled for this reviewer service.
The full PR reviewer audit still runs separately and will publish findings if it detects issues.

_{tangletools · auto-approval · reason: blanket_auto_approve · 2026-06-25T14:35:12Z}

[tangletools]

✅ Auto-approved PR — 64a9a706

Blanket team auto-approval is enabled for this reviewer service.
The full PR reviewer audit still runs separately and will publish findings if it detects issues.

_{tangletools · auto-approval · reason: blanket_auto_approve · 2026-06-25T14:41:43Z}