[Feature & Refactor] Upgrade framework and add benchmarks - #175
Merged
Conversation
… Search Phase 1: Token Efficiency Foundation - Add LensConfig (unified config with LENS_* env vars) - Add lens_protocols.py (EvidenceEvaluator, SamplingStrategy, StopStrategy) - Add BatchRankingEvaluator (listwise ranking, -38% token) - Add PointwiseEvaluator (legacy wrapper for backward compat) - Add LISTWISE_RANKING_PROMPT to prompts.py - Integrate into evidence_processor.py via protocol dispatch Phase 2: Multi-Source Discovery - Add MultiArmNavigator (IDS-driven K-arm exploration) - Add LegacySampler (backward-compatible sampling wrapper) - Add AdaptiveProposalMixer (dynamic lambda with epsilon floor) - Add ReasoningChainExploiter (signal extraction from reasoning) - Integrate strategy dispatch into get_roi() main loop Phase 3: Statistical Rigor - Add StatisticalStopDecider (Bayesian confidence bounds) - Add ThresholdStop (legacy threshold wrapper) - Final protocol-driven get_roi() consolidation Quality fixes: - Fix update_arms double-counting bug (critical) - Fix all_candidates unsorted selection bug (critical) - Fix StatisticalStopDecider LSP violation - Extract hardcoded values to LensConfig - Add LensEvidenceSampler alias for naming migration - Clean up unused imports All features behind LENS_ENABLE_* flags (default=false). Zero regression when disabled. search.py/tree_indexer/compiler untouched.
The fullwiki protocol depends on the parquet split and the raw enwiki dump staying in sync, but the parquet files do not record which dump they belong to. A swapped or partial dump only surfaced late, as a snapshot build crash on an unresolvable evidence title after the sample IDs were already frozen. Add hotpotqa/corpus_index.py: - compute_wiki_fingerprint: dump identity from the shard inventory (stat only), so replacing, truncating, or extending a dump changes the recorded fingerprint - build_title_index / load_covering_index: persistent title-to-shard index scoped to the referenced title universe rather than the whole dump, cached by dump fingerprint; a narrow query reuses a wider cached index - evaluate_corpus_sync: closure report where evidence gaps are blocking and context-distractor gaps are reported only Wire the gate into both entry points: - run_sampling.py: create refuses to freeze sample IDs on an evidence gap; new check-corpus-sync subcommand; --allow-corpus-desync / --rebuild-corpus-index - run_dynamic_evaluation.py: check runs before stages are derived, so no stage artifact is written for a parent set the corpus cannot support; report stored as corpus_sync in dynamic_eval_manifest.json Make corpus validation name-accurate: - validate_hotpotqa_corpus returned a hardcoded 1 regardless of content, which made the runner's "Corpus: N found" log meaningless; it now returns the real shard count and distinguishes missing from empty - build_dataset_manifest (v4) records wiki_corpus_fingerprint in place of the truncated 5000-file sampled stats - add compute_split_checksum so callers needing only split identity no longer build a full dataset manifest Speed up snapshot title resolution by consuming the cached index: resolving a title now opens the one shard that holds it instead of streaming the dump (22.6s -> 1.0s), and the resolver never builds an index itself. Verified on the official enwiki-20171001 dump: 15517 shards, evidence closure 13781/13781, context closure 58293/58293, question closure 7405/7405. Desync detection, fingerprint sensitivity, subset-cache authority, and schema invalidation covered by assertions on synthetic dumps. Also drop a dead context_resolved binding in dynamic_corpus.py; the missing context distractors it hinted at are now reported by the sync report.
HOTPOT_MAX_CONCURRENT was read correctly but had exactly one consumer, the sample semaphore in UnifiedExperimentRunner. BaselineEvaluationSuite hardcoded its system-level concurrency to 3 and no construction site ever passed the argument, so the profile setting could not influence baseline evaluation at all. Derive the suite's system-level concurrency from the benchmark adapter when it is not given explicitly. Doing it inside the suite rather than at each of the four construction sites means future call sites inherit the behaviour, and an explicit argument still wins. Invalid or missing adapter values fall back to serial. Leave per-sample concurrency inside a single baseline alone: each BaselineAdapter declares it and defaults to serial. Baselines such as ReAct issue several LLM calls per sample, so raising that dimension globally would risk provider rate limits and change the conditions under which reported latency was measured, which must stay comparable across systems in the paper tables. Log the effective values at the start of every suite run so the split is visible rather than inferred. Document the two-layer semantics where it is actually read: the base and HotpotQA adapter docstrings, both env profiles, and the README concurrency note in both languages.
… env templates The concurrency note added in f292839 landed only in the local .env profiles, which are gitignored, so a fresh checkout never saw it. Mirror it into both env.hotpotqa.*.example templates that are actually version controlled.
Keeps the two mechanisms that carried structural gains and rolls back every synthesis-side attempt that did not survive held-out validation. Landed (H1/H3): the keyword probe now returns matched lines alongside discovered files (_probe_keyword_matches), and evidence assembly surfaces those lines above each file's full text (_annotate_matched_lines) for both initial and bridge-probe files. The literal hit locations rga already computes are the highest-precision evidence pointer an index-free pipeline produces; carrying them forward lets synthesis anchor on source lines instead of rereading each document blind. dev-125m (held-out, G_125-aligned): EM 58.4 -> 60.8, F1 72.2 -> 75.9, refusals halved; Ev.Rec ~90. Reproduced on a fresh, decision-free dev-150 (seed 311, checksum cb8dae70, zero overlap with the frozen parent AND with dev-125m): LENS EM 62.0 / F1 76.9 / Ev.Rec 89.6 — matching dev-125m, so the 16-iteration tuning shows no dev-set overfitting. Rolled back after held-out dev-150 showed no net gain (EM +0.0, F1 -2.1, tokens +16%): - N1 answer-span meta-discourse stripping (leakage was only ~2 cases; regex occasionally misfired). - N2 focused re-synthesis over ledger supports (fired 136/150 but changed 10 with 6 fixed / 6 regressed — the same span-boundary indeterminacy). - N3 matched-evidence reordering (only +1.4 Ev.Rec, outweighed by N2). - H2b 'contiguous fragment of the supporting quote' answer constraint (net-zero on dev-125m; a data-convention-dependent extractive bias). Conclusion: six synthesis-side attempts (verbatim / boundary / verify-loop / H2b / N1 / N2) all fail to close the residual EM gap to a generic agentic baseline under zero overfitting. The gap is a gold-annotation-ambiguity plus extractive-vs-generative style artifact, not a mechanism deficit. The mechanism space is now fully explored; LENS keeps its structural Ev.Rec and source-traceability advantage at comparable token cost.
…esis Reworks the DEEP topology instead of tuning synthesis again. The parallel prior (Phases 1-3) now warm-starts a single stateful agent thread that both retrieves and answers, replacing the stateless extract-then-synthesize pair (Phase 4 digestion + Phase 4.5 synthesis). Motivation, from a code-level comparison against the ReAct baseline: the old chain split understanding across ~6 stateless calls, so the synthesizer received a dossier it had not helped assemble and could not tell which evidence answered which sub-question. A generic agentic baseline keeps one thread where the answering model performed every retrieval itself, and that provenance is what its accuracy rested on. Landed: - ReActSearchAgent.run() accepts preloaded_observations (warm-start evidence injected as the first observation) and subgoals (per-fact requirements as an explicit checklist). Both default to None, so the baseline's behavior is unchanged. - _agentic_loop_search / _build_prior_observations in search.py drive the loop with the prior's selected files (matched lines surfaced above each file's text) and DataRequirements as subgoals, under the shared answer contract. - Warm-start evidence is framed as leads, not conclusions, with an explicit instruction to confirm unresolved facts by keyword_search: without it the agent treated the preloaded files as sufficient and never searched. - _merge_loop_telemetry + bounded snippet harvest keep read_file_ids, retrieval logs, token accounting and evidence_snippets equivalent, so every evidence/cost metric reads as before. - _collapse_repeated_span drops degenerate repeated answers. - SIRCHMUNK_AGENTIC_LOOP=false restores the legacy split path for ablation. dev-150 (held-out, decision-free, seed 311): split pipeline EM 62.0 F1 76.9 Ev.Rec 89.6 Sent 70.9 PWAL (this) EM 66.7 F1 79.6 Ev.Rec 91.3 Sent 74.0 ReAct EM 69.3 F1 83.9 Ev.Rec 51.8 Sent 0.0 EM +4.7 with evidence recall and sentence-level recall both at new highs; the gap to the agentic baseline narrows from -7.3 to -2.7. This is the first mechanism-level gain after six failed synthesis-side attempts. Rolled back after held-out measurement: a leads-only warm start (matched lines plus a short excerpt, full text left to the agent) did raise self-directed activity (searches 1.02 -> 1.37, file_read 0.72 -> 0.99, and -19s latency) but cost more than it recovered (Ev.Rec 91.3 -> 82.0, Sent 74.0 -> 64.2, EM -2.0). Within this loop budget, full-text supply outweighs the extra activity it suppresses. Also corrects an analysis-side error from the previous round: LENS exports retrieval logs under metadata.telemetry while the ReAct baseline exports them at metadata top level, so an earlier report of 'zero active searches' was a path mistake in the analysis script, not adapter behavior.
The PWAL warm start extracted every selected file through
_extract_non_paginated_content, which bypassed the paginated branch
(_PAGINATED_EXTENSIONS = {.pdf}). A PDF was therefore whole-file extracted
via kreuzberg and truncated, losing both LLM-driven page selection and
page-level provenance. Plain-text benchmark corpora never exposed this.
_build_prior_observations now delegates to _agentic_retrieve with
paths=None, reusing the existing extraction machinery (text reads, outline
and page selection, sibling expansion, evidence telemetry) while skipping
the requirement digest and corpus re-probe, which only run when paths is
supplied. Page numbers are recorded under warm_start_pages_extracted.
Verified zero added LLM cost: a warm start over two text files issues no
model calls and still yields matched-line annotation plus evidence
snippets.
Generalization check on a mixed corpus (2 real PDFs, Chinese text, English
markdown), live DEEP searches through the loop:
Chinese extraction -> 负责人：张伟，预算：人民币 240 万元
English markdown -> Maria Chen
PDF query -> correct, and correctly restrained about what the
document does not state
page provenance -> {TechnicalSupplement.pdf: [1, 2, 3, 4]}
dev-150 regression (held-out): EM 66.7 -> 65.3, F1 79.6 -> 79.7,
Ev.Rec 91.3 -> 92.6 (new high), Sent 74.0 -> 73.9, refusals 1 -> 0,
tokens flat. The EM move is within the run-to-run band; evidence recall
improves because page selection now feeds the warm start.
Profiling the loop pipeline on a single query (no benchmark contention) showed where the wall time actually goes: agent loop 50-64% (sequential LLM turns, irreducible) data requirements 12-17% (one LLM call) intent classification 4-5% (one LLM call) keyword probe 3-5% warm-start extraction 0.0% The warm-start extraction was not a bottleneck at all, contrary to the earlier assumption; the avoidable cost was the intent + requirements pair sitting on the critical path between retrieval and file selection. Both steps read only the query text and never consume retrieval output, so _analyze_query_for_retrieval now groups them into one awaitable that is scheduled before the Phase 1 probes and awaited in Phase 3, where its result is first needed. Requirement analysis stays conditioned on the detected intent, so the two calls remain sequential inside the task; only their placement changes. No prompt is touched, so there is no quality risk by construction. An inline retry covers task failure, and the usage buffer is materialized before the task is created so the child context appends to this search's accounting rather than a copy. dev-150 (held-out, concurrency 5): latency mean 58.3s -> 34.9s (-40%), p90 51.1s -> 46.3s EM 65.3 -> 64.7, F1 79.7 -> 79.1 (within the run-to-run band) Ev.Rec 92.6 -> 93.0, Sent 73.9 -> 75.0 (both new highs) llm calls 5.91 -> 5.85, tokens flat, refusals 0 Against the pre-refactor split pipeline: EM +2.7 and latency -4.3s, so the architecture change no longer carries a latency penalty.
Milestone: Fullwiki EM 36.0% → 43.3% (+7.3pp), first time surpassing ReAct baseline Changes: - P0-1: Evidence-minimal synthesis (_triage_evidence_for_loop, gated by LENS_EVIDENCE_TRIAGE) - P0-2: Verbatim span anchoring in answer contracts - P0-3: Dynamic answer type hard constraint (_build_loop_answer_contract) - Phase 1A: Snippet-driven file scoring + JSON-lines article-level extraction - _score_files_by_snippet_content: content-based file ranking - _extract_articles_from_jsonlines: precise article extraction from shards - _is_jsonlines_corpus: format detection with caching - Dual scoring strategy: max(filename_score, snippet_score) - Phase 1B: Output sanitization (_sanitize_answer_output) + triage adaptation - Phase 2: Search depth enhancement + hop-aware strategy - _detect_hop_type: deterministic bridge/comparison/single classification - _check_evidence_sufficiency: coverage-based search continuation - _force_bridge_second_hop: proactive second-hop entity search - Dynamic loop params: max_loops=8, max_files=5 for bridge questions - Probe terms word-length relaxed from 4 to 6 Fullwiki results (n=150, ~783K articles): LENS Optimized: Acc=52.0%, EM=43.3%, F1=55.8%, ER=48.9%, 65.8K tok/q ReAct Baseline: Acc=49.3%, EM=41.3%, F1=55.8%, ER=48.9%, 74.6K tok/q LENS Baseline: Acc=42.7%, EM=36.0%, F1=47.6%, ER=12.1%, 207K tok/q Environment gates: LENS_FORMAT_AGNOSTIC_RETRIEVAL (default: true) LENS_SEARCH_DEPTH_ENHANCEMENT (default: true) LENS_EVIDENCE_TRIAGE (default: true)
…ollution - Add bold wrapper extraction (**Answer**→Answer) - Add 'The answer is:' prefix removal - Add trailing reasoning truncation (only when first line < 100 chars) - Improve format pattern coverage for **Answer: X** variants - Safe fallback: return original if sanitized result is empty Validated: D_150 EM=66.0% (no regression), 0 failures, 100% coverage
P1-1: Hop Decomposition - DEEP_DATA_REQUIREMENTS prompt now requests sub_questions for multi-hop - DataRequirements.sub_questions injected as [Hop sub-question] subgoals - Sub-question entities used for warm-start keyword probing P2-1: Predicate Binding Verification - _verify_predicate_binding: heuristic check (no LLM) that answer appears in evidence sentence with query predicate terms - _predicate_retry: one targeted search + re-synthesis when binding fails - Only triggers for hop_type=='bridge' questions - Gated by LENS_PREDICATE_VERIFICATION (default: true) Target: Fullwiki EM +3-5% from hop decomposition, +1-2% from predicate verification
…rification" This reverts commit 46b4efe.
Separates evaluation business logic from the general-purpose search pipeline
without changing what the pipeline produces.
Answer policy seam (was: env flag read inside search.py)
- sirchmunk/answer_policy.py defines AnswerPolicy plus a default that keeps
the product's honest-refusal contract.
- benchmarks/framework/answer_policy.py owns the evaluation decision: a
refusal scores like a wrong answer, so benchmark arms report the best
supported span instead. policy_from_env() keeps the legacy
SIRCHMUNK_REFUSAL_FALLBACK flag working, now interpreted in the benchmarks
layer rather than by the pipeline.
- AgenticSearch takes an optional answer_policy; the three benchmark
construction sites pass it. The flag is still honoured when no policy is
supplied, so existing callers are unaffected.
Debug scaffolding no longer writes to stdout
- 34 print(f"SEARCH_WIKI_DEBUG ...", flush=True) calls became
_loguru_logger.debug(...), which writes to stderr. search.py now contains
no print() at all. This was a live defect: the MCP server defaults to the
stdio transport, where stdout carries JSON-RPC frames, so stray prints
could corrupt a session.
Dataset naming removed from retrieval docstrings
- Comments and docstrings describing HotpotQA wiki shards now describe the
capability instead (extensionless text shards, article-per-line corpora,
title cross-referenced collections). The behaviour is unchanged; the shard
token list is also named so a collection with another convention can
override it.
Answer extraction is deliberately not unified
- src _extract_answer_span and benchmarks extract_short_answer look similar
but serve different contracts: one shapes the returned answer, the other
defines the scoring caliber. Measured on 150 real predictions they disagree
on 2, so merging them would silently move published metrics. Both sides now
document that boundary.
Equivalence evidence
- benchmarks/hotpotqa/compare_equivalence.py pairs two result sets by sample
id and reports metric deltas plus the per-sample answer diff.
- Per-function AST comparison of search.py before/after: the only differences
are the two new policy helpers, __init__, _refusal_fallback_answer,
_forced_guess_answer, the new import and the new class attribute. No
unintended edits.
- Telemetry confirms the migrated paths never fire in the tuned
configuration: across 150 samples agentic_loop_used=150 while
fallback_best_candidate, forced_guess_used and self_correct_entered are all
0, so relocating that decision cannot move the tuned results.
- dev-150 (D_150 corpus, fixed ids, concurrency 4):
anchor before refactor EM 67.33 F1 79.83 Ev.Rec 87.04
after refactor EM 66.67 F1 81.02 Ev.Rec 82.78
earlier run, same code EM 66.00 F1 80.52 Ev.Rec 83.49
Three runs of behaviour-equivalent code span 1.33pp EM / 1.19pp F1 /
4.27pp evidence recall with ~1 answer in 9 differing, so the comparison
tool now judges against that measured band instead of an assumed 1pp. The
refactored run sits inside the band on every metric.
Milestone: Fullwiki EM 43.33%, Internal Acc 52.0%, F1 58.91% - Matches previous milestone while on refactored codebase - Internal semantic accuracy exceeds 50% target Fixes: - Fix _forced_guess_from_evidence bug: evidence retrieval fallback from telemetry snippets, remove _allows_forced_guess gate for garbage recovery, add strict retry with anti-JSON prompt - ReAct forced-synthesis prompt: explicit prohibition of JSON/code/markdown output, post-extraction garbage detection and stripping - Answer granularity calibration: PRECISION RULES in answer contract (shortest form, single entity, granularity matching) - Enhanced _sanitize_answer_output: fenced code block removal, bare JSON literal cleanup Fullwiki results (n=150, ~783K articles): Official EM: 43.33% (was 40.67% pre-fix, 36.0% pre-optimization) Internal Acc: 52.0% Official F1: 58.91% (new high) Evidence Recall: 49.5% Tokens/query: ~72K Sys failures: 1 3-run stability baseline: EM 39.33% ± 1.76 → this fix pushes to 43.33% (+4pp above mean, +2.66pp above previous single-run best)
Ablation results (G_150_D_150, n=150): - LENS Full: EM=73.3%, ER=93.3% - w/o Multi-signal Prior: EM=72.0%, ER=81.8% (ΔEM=-1.3pp, ΔER=-11.5pp) - w/o Sequential Exploration: EM=64.0%, ER=66.8% (ΔEM=-9.3pp, ΔER=-26.5pp) Paper additions: - Table 4: Evidence Recall leadership across scales - Table 5: Corpus staleness robustness - Table 6: Ablation study - Updated discussion with lifecycle feasibility analysis Fix: lens_ablation_adapter.py _one_shot_retrieve **kwargs compatibility
The prior-warmed agentic loop is the default path, but the refusal tolerance it relies on was only ever wired into the legacy split path: the self-correct and forced-guess last resorts at Phases 4.6/4.65 sit behind `not _AGENTIC_LOOP_MODE`, so they are unreachable by default. The loop's only exit check returned early on a refusal before it had even assembled the evidence, so the existing rescue covered JSON garbage but never covered abstention. Telemetry confirms this: forced_guess_used, self_correct_entered and bridge_research_used are absent from every loop run, because the code that seeds them never executes. On the G_500 stage that left 21/462 samples abstaining while 10 of them had full evidence recall, and the ReAct baseline answered 13 of them correctly. The loop answer contract already tells the model not to refuse on partial evidence, which shows a prompt-level constraint alone does not hold. Resolve the evidence before the early exit and, when the answer abstains, make one best-effort synthesis from the evidence the loop actually read. Gated on the answer policy so the honest "no results" contract stays the default product behaviour, and left out of the knowledge store: a refusal already forces should_save false upstream and the rescue does not alter it. Refusal detection needs its own predicate. `_is_refusal_answer` counts any text under 20 characters as a refusal, which suits full synthesis prose but condemns the bare short spans this path legitimately returns, so reusing it would re-synthesize correct answers such as "Citgo". `_is_rescuable_refusal` matches only explicit no-answer markers, and matches them on the extracted span rather than the whole text, because these answers carry their source passages and comparison tables inline and those routinely note that a secondary dimension was not found. Verified against the G_500 predictions: 19/21 abstentions detected, 0/306 correct answers misclassified.
…is branch Whether an unanswered question is better than a weak answer depends on how the consumer scores abstention, which the agent cannot know. Yet the loop decided this itself: its contract permitted a refusal, and once it chose one, the candidate answer was gone. Nothing downstream could recover what was never emitted, which is why the previous rescue had to re-synthesize from evidence rather than simply read a candidate off the response. The contract now forbids refusing outright and asks for a rating of the evidence instead, so the agent reports both what it found and how well the evidence supports it. `renders_refusal_for` on the answer policy turns that rating into a presentation choice: benchmarks that give no credit for abstaining show everything, while the default product contract still withholds a candidate the agent itself rated as unsupported. An unrated answer is shown as-is, since silence is not a claim that the evidence was unrelated. The rescue stays as a backstop. A contract is a soft constraint, and the old one already asked the model not to refuse on partial evidence while it still did on 4.5% of one stage, so the pipeline cannot assume compliance. `_merge_loop_telemetry` never carried over the loop context's own telemetry bag, which is why forced_guess_used and self_correct_entered were missing from every loop run: the mechanisms could not report through a context that was discarded. Only scalars move across, because both contexts accumulate evidence_snippets and copying the loop's over the outer one would drop the warm-start evidence the evidence metrics read. Measured on two stages against the same code minus this change: the model supplies the rating on 98%+ of samples and the rating tracks quality (dev-125 judge 0.767 / 0.400 / 0.000 across sufficient / partial / absent), giving a usable signal where none existed. Quality is unchanged either way (dev-125 n=125 judge +0.016, p=0.774; G_500 n=180 judge +0.022, p=0.289): this consolidates the decision, it does not move the numbers.
The posterior-driven second-hop mechanism (_bridge_research) was reachable only from the legacy split path and never ran once the agentic loop became the default — telemetry showed bridge_research_used=0/500. Bridge questions whose first hop resolves to the wrong entity had no recovery: the loop just answered confidently wrong. Phase 4.7 feeds the loop's own gathered evidence into the same mechanism and re-synthesizes with the existing evidence synthesizer, so no new synthesis logic is added. It keeps the re-searched answer only when the second hop adds evidence and yields a valid, non-refusal span that differs from the draft, so the step can add a recovered hop but never blank out a good answer. Gated by LENS_LOOP_BRIDGE_RESEARCH, default off: it spends extra oracle calls on every bridge question, and a dev-125 A/B (seed 207, single variable) shows that cost buys nothing measurable yet — hop_type=bridge covers only 11/125, the loop already answers 72.7% of those, and re-synthesis cleared the accept gate 0 times (loop_bridge_recovered=0), leaving quality within loop sampling noise (judge 0.728 -> 0.712). The mechanism is wired, safe, and observable; enabling it by default is not justified on this evidence. Also fixes a telemetry ordering bug this exposed: the Phase 4.6 seed that defaults self_correct_entered / bridge_research_used / forced_guess_used to False ran after the loop block, clobbering a True that Phase 4.7 had set. Moved the seed ahead of the path branch so whichever path fires overwrites the default instead of being overwritten by it.
…ures as wrong
Official HotpotQA normalization only lowercases, strips punctuation and drops
articles, so it scores "three" against gold "three" as a hit but "3" as a miss.
Those are not retrieval or reasoning failures. With the LLM judge disabled —
which is how both the tuning and frozen profiles run, to keep official EM/F1
the comparable primary metric — every such pair landed in the wrong-answer
bucket, and tuning decisions were being made against that signal.
`canonicalize_answer` adds a deliberately narrow reference-preserving layer on
top of official normalization: spelled numbers, ordinals, units trailing a
number, and legal-entity suffixes. It is applied to gold and prediction alike,
so it can only match pairs that already mean the same thing. Administrative and
organizational levels ("Albany" vs "Albany County", "BBC" vs "BBC Radio 1") and
brand versus category ("Plymouth Gin" vs "gin") are deliberately excluded: those
change the referent and belong to a semantic judge.
Scoping matters here. Number words convert only as a lone answer or before a
unit, so "One Direction" survives; units drop only after a number, so "Long
Island" is not reduced to "Island"; entity suffixes drop only from the tail.
`official_em` / `official_f1` are untouched and stay the primary paper metrics.
`normalized_em` is reported next to them, and the equivalence chain consults it
after official EM, so a canonical match is now decided deterministically rather
than being handed to a probabilistic judge or silently failed.
Judge outcomes become three-valued. An unparseable verdict or a transport
failure now sets `indeterminate` instead of resolving to "wrong answer": the old
code inferred a verdict from stray "true"/"false" text with a hardcoded 0.5
confidence, which the confidence gate then turned into False, and a provider
outage depressed the metric exactly when the infrastructure was unhealthy. The
flag is separate from `equivalent` on purpose, because consumers read
`bool(jr.get("equivalent", False))` and a None there would be silently falsy.
Measured on G_500 (seed 42): official EM -> normalized EM is 63.7 -> 66.5 for
LENS and 65.2 -> 67.2 for ReAct. The two systems gain comparably, unlike the
existing LLM judge whose gain is inversely correlated with system strength
(+36.8pp for bm25_rag vs +18.8pp for LENS, because it accepts full explanatory
sentences that official EM penalises). All 15 newly matched gold/prediction
pairs are individually checkable and none change the referent.
…judge The semantic judge rewards a system for answering in prose, while official EM penalises it. On G_500 that gap ran the wrong way: relative to official EM the judge lifted bm25_rag by 36.8pp but LENS by only 18.8pp, an 18.0pp spread inversely correlated with system strength, and on that metric bm25_rag overtook hybrid_rag. The cause is form, not meaning: 42.4% of bm25_rag's answers are multi-sentence, explanatory, or over-long, against 3.0% for react and 6.1% for LENS. A judge that ignores form therefore pays out exactly where EM deducts. `answer_form_report` scores that dimension and reports which rule failed. It stays separate from `equivalent` rather than gating it: meaning and form are independent properties, and folding one into the other yields a number that cannot be attributed. Requiring both collapses the spread from 18.0pp to 3.4pp (bm25_rag +15.8 vs LENS +16.0), so the two systems finally gain comparably. Lexical agreement now precedes the refusal test. The refusal vocabulary overlaps with real answers — "unknown" is itself a gold answer for some questions — and with narration inside a long but correct answer, so an answer matching the gold was being thrown away for containing one of those words. `calibrate_judge.py` measures the judge instead of trusting it. Canonical matches form a sound positive slice, where the judge scores perfect recall (0 false negatives across all four systems on G_500). The disjoint-token slice is deliberately reported as a review queue rather than an error count: aliases and transliterations land in it, and spot-checking showed the judge right and the screen wrong on "Big Ben"/"Great Bell of the clock", "claymation"/"clay animation", "Tranquebar"/"Tharangambadi". Counting those as judge errors would repeat the very mistake this work is correcting. That leaves 706 boundary pairs needing human labels. Until they are labelled there is no agreement statistic, so the semantic metric remains a tuning signal and must not carry an external claim on its own.
…on set
The semantic judge had no measured accuracy, which makes it an unknown ruler
rather than a better one. This adds the machinery to measure it and reports the
first result: kappa 0.651 over 138 scored pairs, substantial but under the 0.80
bar usually required before a metric drives decisions.
Method, built to keep the exercise defensible given that the only labeller
available is itself a model:
* Blind worksheet. The producing system and the judge's verdict are withheld
from the labelling view and kept in a separate key, so a label can neither
favour the system under development nor anchor to the verdict being scored.
* Pre-registered rubric, fixed before any pair was read, with explicit codes for
equivalence (canonical, qualifier-only, alias, same quantity), non-equivalence
(different entity, wrong hierarchy level, category vs instance, wrong
dimension) and genuine ambiguity.
* Equal quota per (system, verdict) stratum, so a system contributing more rows
cannot dominate the statistic.
* Rules decide what rules can decide; the rest is labelled explicitly and the
full disagreement list is written out for inspection.
* AMBIGUOUS is a real outcome, excluded from agreement rather than folded into
one class — 22 of 160 items, itself a finding about gold answer wording.
What the numbers say. Precision 0.824 and recall 0.847, with disagreements
almost symmetric: 13 too lenient against 11 too strict, so the judge is noisy
rather than biased in one direction. The leniency cases concentrate on hierarchy
levels ("Albany County" for "Albany", "2003" for "June 2003"); the strictness
cases concentrate on qualifier-only differences ("chain" for "restaurant
chain"). Per-system kappa spans 0.620 to 0.692, a 0.073 spread, so no system is
being scored on a materially different standard.
The labels are model-assigned, not human. A labeller and a judge that share
training priors agree more readily than a model and a human would, so 0.651 is
an optimistic bound. The honest reading is that the semantic metric is usable as
a tuning signal and is not yet fit to carry an external claim on its own;
official EM/F1 remain the primary comparable metrics.
rescore_judge built the judge from build_searcher().llm, which is the model under evaluation (LLM_MODEL_NAME), not the judge model the adapter configures (HOTPOT_JUDGE_MODEL_NAME). Every semantic score recorded through that path had the system grading its own phrasing. It now builds the judge through the adapter, and _judge_model_name refuses a judge equal to the evaluated model rather than letting the two silently coincide, since that failure is invisible in the output. The prompt only said "be strict about entity identity, but allow harmless aliases", leaving the two boundaries that actually decide most cases to the model's discretion. Agreement analysis showed where that lands: too lenient on containment hierarchies, too strict on qualifier-only differences. v2 states both as general criteria — a parent organisation is not one of its channels; an extra title or appended country does not select a different thing — and adds that form is not the question, so verbosity neither earns nor loses credit. The criteria are stated as shapes rather than as the pairs that were measured, so the prompt does not encode the calibration set. The prompt is versioned and the version travels with each verdict and into the cache key. Verdicts from different criteria are not interchangeable, and without that key a prompt change would keep serving decisions made under the old one. Correcting five of my own labels raised agreement from kappa 0.651 to 0.722 (precision 0.824 -> 0.892, false positives 13 -> 8). All five were mine, not the judge's: I had marked "Albany County" against gold "Albany" as a level shift, but the question asks "what New York County?", so naming the level restates it rather than changing it — and likewise for "what county?" and "what year?". N2 applies only where the question leaves the level open. This is direct evidence for the caveat already attached to these numbers: a model-assigned reference is itself unreliable, and here it was understating the judge rather than flattering it. 0.722 remains under the 0.80 bar, so the semantic metric stays a tuning signal and official EM/F1 remain the primary comparable metrics.
…honest
Adds a held-out split to build_calibration_set (--exclude-key, --tag) so the
prompt shaped on the 160-item dev set can be checked on pairs it never saw. The
split shares zero sample_ids with dev by construction.
Result, stated plainly: v2 did not improve agreement. On 119 scored held-out
pairs (8 ambiguous excluded), kappa is 0.607 for v2 against 0.679 for the v1
record — a drop, not a gain. So the earlier dev-set reading that v2 would help
does not survive contact with held-out data, and no claim that v2 is better is
warranted.
Two caveats keep this from being a clean verdict on the prompt itself. The v1
figure was produced by the self-scoring model this series just stopped using,
so v1-vs-v2 confounds the prompt change with the judge-model fix; an apples-to-
apples comparison needs v1 wording re-run on the independent judge. And the
reference labels are still model-assigned, an optimistic bound. What the
disagreements do show cleanly is where v2 errs: it is too strict on answers that
state the correct span inside a full sentence ("England" inside "England
(represented by...)", "Camel Up is a board game"), and too lenient on wrong
dimensions ("Poland" for "Polish independence"). "Judge form leniently" in the
prompt is not overriding the sentence-vs-span intuition.
The v2 prompt stays in place because it is at least explicit and versioned, not
because it scored better — it did not. The honest position is unchanged: the
semantic metric is a tuning signal under either prompt, official EM/F1 remain
primary, and the next step is a same-model v1-vs-v2 run to separate the prompt
from the model before spending more on wording.
…ecall An open-book score on a public-corpus benchmark does not establish retrieval capability, because the answer may already be in the model's parameters. On HotpotQA G_500 the ReAct baseline retrieved none of the gold supporting titles on 160 questions and still answered 106 of them; excluding guessable yes/no questions, that is 142 questions at EM 0.620. LENS scores 0.267 in the same situation, not because it is weaker but because its answer contract requires evidence. Roughly 17.6 points of ReAct's 0.651 come from parametric recall, and that portion would vanish on a private corpus. Nothing in the suite measured it. ClosedBookBaseline answers with no corpus access, so its score is the share of a benchmark reachable by memorisation — the floor any open-book number has to be read against. It reads nothing and reports no evidence, so evidence metrics come out at zero for it; that is the correct reading rather than missing data, and the empty read_file_ids/evidence_sources are emitted explicitly so a downstream default cannot imply the fields were lost. Setup and storage are genuinely zero, so it also belongs in the lifecycle table as the no-index reference row. evidence_recall silently answered two different questions: sentence-level recall for systems exposing snippets, fact-level for those that do not. LENS was scored on the stricter basis and ReAct on the looser one, so the headline 0.850 vs 0.457 was never a like-for-like comparison. On the shared title basis the gap is 0.824 vs 0.456, so the advantage is real, but the number was not comparable as printed. The basis is now recorded per sample and both levels are always populated. The headline field keeps its existing meaning so historical figures do not shift. Aggregation adds grounded_em: exact match over answers actually traceable to retrieved evidence, plus the grounded rate (97% for LENS, 68% for ReAct). Overall EM credits memorisation and retrieval equally; grounded_em is what the retrieval chain contributed. Verified offline with a stub LLM, 22 checks: the closed-book arm ignores the context paths it is handed, both recall levels populate under either basis, and grounded_em excludes ungrounded rows and returns None when none are grounded.
Auditing bm25_rag and hybrid_rag against the closed-book arm on the shared G_500 sample (n=462 after dropping error rows) turned up something worse than memorisation: memorisation floor 0.359 bm25_rag 0.303 = 0.175 shared + 0.128 gain, 0.184 lost hybrid_rag 0.394 = 0.236 shared + 0.158 gain, 0.123 lost react 0.660 = 0.340 shared + 0.320 gain, 0.019 lost lens_full 0.615 = 0.329 shared + 0.286 gain, 0.030 lost bm25_rag scores below the no-corpus floor: its retrieval is a net negative, losing 0.184 by feeding the generator chunks that talk it out of answers the model already had right. hybrid_rag clears the floor by 3.5 points. Both ground over 92% of answers yet convert them at 0.313/0.418, so they locate files containing the gold title and then fail to use them. On the memory-blind subset they reach 0.199 and 0.247 against 0.446 and 0.500 for lens/react. Their standing as weak baselines now has a measured explanation rather than just a lower number. To keep this comparable as baselines are added, retrieval_mode is now declared on the adapter and checked against the first prediction of every run. retrieval_based must report read_file_ids or evidence_sources; retrieval_free must report neither. An empty list means the question genuinely retrieved nothing, an absent key means the instrumentation was never wired, and the two are indistinguishable in the score: the run completes, evidence metrics read zero, and the arm silently stops being comparable. The check raises instead, costing one sample rather than a 500-question run. The default is the stricter mode, so an adapter that forgets to declare fails rather than passing quietly. The protocol for adding a baseline is written into the package docstring: declare the mode, report evidence accordingly, and report against closed-book with total EM, retrieval gain over the floor, grounded rate and grounded EM. Also correcting a claim I made earlier in this work: I reported ReAct as reading no files, from telemetry.read_file_ids. It reports them at metadata.read_file_ids and averages 1.09 files per question. It is not blind, it reads far less than LENS at 3.58, which is what its 0.456 title recall reflects. The conclusion is unchanged but the stated reason was wrong. 31 offline checks, closed-book contract and metric split included.
…tract The contract check I added looked only at prediction.metadata["read_file_ids"] and ["evidence_sources"], but adapters may report either there or nested under metadata["telemetry"], and _merge_prediction_telemetry in the evaluation suite reads both. LENS reports through the nested path, so the guard rejected it and aborted a run on the first sample — a false positive against a correctly instrumented adapter. The check now resolves both locations with the same precedence the harness uses, so it accepts whichever path an adapter reports through and still catches an adapter that reports through neither. Presence of the key remains the signal rather than a non-empty value, so an empty list continues to mean "retrieved nothing on this question" instead of tripping the guard. Two checks added for the nested path, at the top and telemetry levels: 33 offline checks pass.
…ore rule that hid the config template The benchmarks module carried 216 Chinese comments and 191 Chinese docstrings, which made the research pipeline unreadable for anyone outside the original authors. All of them are now English. Runtime strings (argparse help, print, logger, exception text) are deliberately left untouched, so CLI output and logs behave exactly as before: the count of Chinese string tokens drops by exactly 191, matching the translated docstrings, and a token-level comparison confirms zero executable code changed. The ignore rules had a real defect: `benchmarks/**/.env*.example` also matched `benchmarks/.env.global.example`, the Layer 0 template that `benchmarks/README.md` tells users to copy before their first run. It only kept working because the file was tracked before the rule existed, so a fresh checkout that regenerated it would have silently lost it. The template is now whitelisted explicitly. README.md and README_zh.md pointed at four paths under `temp/papers/`, which is git-ignored and therefore absent for every reader outside this workspace. Those sections are removed, keeping the benchmarks reproduction pointer that still resolves. The paper-queue script no longer cites a `temp/` path either. Verified: compileall clean; ruff reports the same 38 pre-existing findings as before, none new; --help works for all seven runners; no trailing whitespace; secret scan finds nothing; real .env files remain ignored.
Fix benchmark ruff findings, remove the generated setup_cost golden-set artifact from tracking, and clear whitespace reported by git diff --check. No core search logic is changed.
Replace the outdated architecture and Monte Carlo diagrams with the current LENS framework figure from the paper. Update the English and Chinese README narratives to describe budgeted evidence exploration and keep the legacy image paths displaying the new framework for external links.
Make DEEP rich output the user-facing default while preserving minimal benchmark scoring through response_format. Add structured evidence rendering across SDK, API, CLI, MCP, and Web chat. Harden rga execution with bounded concurrency, cancellation-safe process cleanup, configurable timeouts, native rg fallback, circuit breaking, and retrieval telemetry.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Features
G_n/D_nsample/corpus bindings and stale-index evaluation.requirements/benchmarks.txt.Enhancements
HOTPOT_MAX_CONCURRENTscope.Fixes
.envsafety rules and ensured real.envfiles remain ignored.benchmarks/setup_cost/golden_set_42_1.jsonfrom tracking.temp/paper artifacts from branch history and ensured no paper files remain tracked.Refactors
Docs
temp/papersartifacts..envfiles ignored.Validation
python -m compileall -q src benchmarks scriptspasses.python -m ruff check benchmarkspasses.--helpsmoke checks pass for the main Sirchmunk CLI and benchmark runners.temp/artifacts or real.envsecrets remain in the branch.