Measure dollar typing against committee reports, with a resolver that agrees on 87.5% of them

[banjtheman]

Related issue

Refs #147 (epic: financial semantics research: type each dollar amount before money returns to the report). This adds evidence, not a closing change.
Refs #115 (split each account's money into typed sub-amounts). It covers itemised lines and multi-piece accounts. Each figure's role and effect aren't typed yet.
Refs #796 (proposed ADR 0025). This measures the bar it proposes: the figure typed as an account's appropriation must equal the committee report's figure for that account.

What does this change?

Research only. No product code changes, money stays out of the report, and classify_bill.py's rules are untouched (#736 copies them into the package and checks parity).

At Civic Tech DC night the organizers asked how good the regex classifier's dollar typing is. This answers it, and improves it.

• committee_agreement.py scores the figure typed as each account's appropriation against the answer key's figure, for every account the answer keys (tests/data/validation_*.json) map to a bill node: 1,163 of 1,267, all committed fixtures, under a second to run. It reports each evidence source separately:

  • Committee reports: thirteen keys, each built from a Senate committee report (FY2024 and FY2025 reported bills), 752 mapped accounts.
  • The Legislative Branch spreadsheet: validation_leg_branch.json comes from House_and_Senate_Appropriations_FY_1994_to_present.xlsx, not a committee report, and is checked against the FY2014–FY2020 enrolled bills. 411 mapped accounts.

• appropriation_resolver.py reads a paragraph the way appropriations are drafted:

  • nothing after the first "Provided, That" counts (an "in all" there caps one use of the money);
  • an "in all" total before it wins;
  • an explicit "is hereby appropriated" names the figure;
  • otherwise it takes the figure that closes the purpose clause, skipping allowances inside it, loan levels and carve-outs;
  • it abstains rather than guess.

  It also does two more things:

  • Itemised lines: ledger types each line with its own name ("including: Office of the Speaker, $6,645,417; ..."). The answer key's name is used only to find a line, never to choose a figure, and a figure selected for a line is scored against that line's own figure.
  • Multi-piece accounts: resolve_total adds the pieces an account appropriates in: "together with $X from the trust fund", ", and in addition $X", a second purpose, and an advance appropriation available this year.

• Review fixes from @willhea (Review fixes for civictechdc/DeltaTrack#845: line scope, proviso boundary, evidence source banjtheman/DeltaTrack#1, merged into this branch):

  • An itemised line is scored by the scope its selection carries. Before, a wrong line figure could be excused as "finer than the paragraph" because an unrelated resolver happened to type the same number.
  • "In all" is read only before the first proviso. Before, "…Provided, That travel shall not exceed, in all, $500,000" replaced a $10,000,000 appropriation.
  • Agreement is reported per evidence source.

  No count moves, because neither bug fires on the committed answer keys. Tests fail if either comes back.

• rollup_agreement.py adds typed figures up per agency and compares them with the report's "Total, " lines. A total is attributed by the report's own arithmetic, so every attribution is checked by construction. Totals are compared net and gross (report totals net out fee offsets and rescissions).

Results (all reproduced by the commands below):

Typed figure equals the answer key's	classify_bill today	this PR
Committee reports (752 accounts)	636 (84.6%)	658 (87.5%)
Legislative Branch spreadsheet (411 accounts)	284 (69.1%)	403 (98.1%)
Combined (1,163 accounts)	920 (79.1%)	1,061 (91.2%)

Errors and roll-ups	classify_bill today	this PR
Typed a different figure while the answer key's is right there in the text	37	1 (gross versus net, TSA)
Figures typed that the source contradicts or does not confirm	83	37, all on committee-report accounts
Agency roll-ups reproducing the committee report's total (of the complete ones)	116 of 159	124 of 154

Against committee reports alone, this PR agrees on 87.5%. The rise from 82.1% to 90.4% combined is entirely the spreadsheet's itemised office lines, so committee-report agreement doesn't change with them (648 of 752 either way).

The roll-up caught one thing no account check could: a liquidating appropriation typed as new money (FTA: "For payment of obligations incurred …, $14,279,000,000, to be derived from the Mass Transit Account", which the report keeps out of new budget authority).

How far to trust the numbers (the README says this at length):

• The first rules were written from 6 answer-key files and frozen at d394669 before the other 8 were scored. On those 8 held-out files, wrong figures went 3 → 0.
• Later steps were written after reading the held-out files, so their held-out number (85.5%) is no longer an unbiased estimate.
• The combined 91.2% includes the Legislative Branch spreadsheet, which is not a committee report. Its 411 accounts are over a third of the answer keys and carry all 96 itemised-line gains. The committee-report figure is 87.5%.
• A fresh answer key (another fiscal year) is the next honest test.

Deliberately left out:

• Moving any of this into the engine (Show each version's dollar amounts, typed, in three report views #736 owns the classifier's move into the package).
• The fix that makes classify_bill type "In addition to amounts made available elsewhere in this Act, $X is hereby appropriated" (Show each version's dollar amounts, typed, in three report views #736's rules).
• Fixing answer-key lines the builder mapped to the wrong paragraph (scripts/build_validation.py).
• Typing each figure's effect (gross versus net), which ADR 0025 proposes.

How to test

uv run python docs/research/financial-semantics/committee_agreement.py                    # every resolver, all answer keys
uv run python docs/research/financial-semantics/committee_agreement.py --split held-out
uv run python docs/research/financial-semantics/committee_agreement.py --resolver summed --details wrong_figure
uv run python docs/research/financial-semantics/rollup_agreement.py --resolver summed --details mismatch
uv run pytest tests/test_appropriation_resolver.py tests/test_rollup_agreement.py

What I ran and checked:

• Hand-traced disagreements:
  • EEOC: the regex took "up to $32,500,000 for payments to State agencies" over the $488,221,000 appropriation.
  • Legislative Branch: "$7,110 each…; in all, $28,440".
  • Promoting Safe and Stable Families: $345,000,000 + $72,515,000.
  • TSA: gross versus net.
  • NASA: the report's $3,044,440,000 against the bill's $3,044,400,000, which looks like a report typo.
• New tests fail when broken: every unit test was checked by reverting its rule and watching it go red (for example, the loose "in addition" rule wrongly adds the Fiscal Service's $235,000; removing the subtotal jump double-counts).
• Review fixes:
  • Both bugs reproduce on the pre-fix branch: the proviso example typed $500,000, and a wrong line was scored finer_than_node once another resolver typed its figure.
  • Both are fixed after the merge.
  • Each new test goes red with its fix reverted.
  • The measurement output is unchanged apart from the per-source lines, and the roll-up output is byte-identical.
• Slow gates:
  • They pin the wrong-figure sets: structured: two known; ledger and summed: TSA only.
  • They pin minimum agreement counts (for summed, per source: at least 658 committee-report, at least 403 spreadsheet) and the roll-up outcome counts, so the README's numbers can't drift unnoticed.
  • The two new modules are added to CI's remaining-slow shard 2.
• CI gates, locally:
  • ruff check and ruff format --check: clean.
  • Fast: 2309 passed.
  • Browser: 30 passed.
  • Slow (-m slow --deselect tests/test_govinfo_corpus_parity.py): 2192 passed.

@willhea @mattzamora, this measures #796's bar against #736's rules. Your read on whether it's the right shape matters most.

Checklist

• Linked the issue above (Closes #...): Refs only, as this is research evidence that doesn't finish Epic: financial semantics research: type each dollar amount and reconcile totals before money returns to the report #147 or Research: split each account's money into typed sub-amounts and summarise how they changed between versions #115
• Ran the CI gates locally and they pass (see What CI checks)
• New or changed behavior has tests
• For a bug fix: the test fails without the fix, and I ran it both ways to check (each rule's test proved red with the rule reverted)
• Disclosed AI assistance below, if any

AI assistance

Written with Claude Code (Claude Opus 5.5) at Civic Tech DC night. I reviewed the change and the evidence above.

🤖 Generated with Claude Code

[willhea]

Thanks for this, @banjtheman. Scoring the typed figure against an independent source is the check ADR 0025 needs, and the README is candid about where the numbers come from and how far to trust them.

Review turned up three things worth fixing before this merges. Rather than go back and forth here, I put working fixes in banjtheman#1, opened against your branch so you can merge, change or decline them:

1. Scoring scope depended on the other resolvers. measure() treated a figure as paragraph-level when it equalled resolve(), resolve_total() or baseline(), so a wrong named-line figure could become finer_than_node just because an unrelated resolver changed. The fix carries the scope with the selection itself.
2. A proviso's "in all" could replace the appropriation. resolve() searched the whole paragraph before cutting at the first proviso, so "For necessary expenses, $10,000,000: Provided, That payments for travel shall not exceed, in all, $500,000." typed $500,000.
3. The Legislative Branch evidence comes from a spreadsheet, not a committee report. validation_leg_branch.json is built from House_and_Senate_Appropriations_FY_1994_to_present.xlsx, and its 411 accounts carry all 96 itemised-line gains. Against committee reports alone, summed agrees on 658 of 752 (87.5%). The spreadsheet is 403 of 411, and the 91.2% combines the two.

No count moves with the fixes, because neither bug fires on the committed answer keys. What changes is how the results are attributed, plus tests that fail if either bug comes back.

Feedback welcome, on the fixes themselves or on whether the per-source framing reads right to you.

[banjtheman]

Thanks @willhea, this was a really useful review, and I've merged banjtheman#1 into this branch.

Before merging I checked the fixes locally:

• Both bugs reproduce on the pre-fix branch. The proviso example typed $500,000, and the Speaker's wrong line was scored finer_than_node once another resolver typed the same figure.
• Both are fixed after the merge. Each new test goes red when its fix is reverted.
• No counts move. The measurement output only gains the per-source lines, and the roll-up output is identical. The README's per-source numbers reproduce: 658/752 (87.5%) against committee reports, 403/411 (98.1%) against the spreadsheet.
• Gates pass: ruff clean, fast 2309, slow 2192, browser 30.

On the framing: yes, per source reads right to me. Agreement with the Legislative Branch spreadsheet shouldn't be presented as committee-report agreement. I've updated the PR title and description to lead with 87.5% against committee reports, with 98.1% and the combined 91.2% beside it.

(Checked with Claude Code.)