fix(pdf): rejoin every printed word break, deciding the hyphen from the document (#650) - #682
Draft
willhea wants to merge 9 commits into
Draft
fix(pdf): rejoin every printed word break, deciding the hyphen from the document (#650)#682willhea wants to merge 9 commits into
willhea wants to merge 9 commits into
Conversation
…he document (#650) A word GPO's printer broke across a line reached `full_text` broken whenever the continuation was uppercase, so the Title I heading of 118-hr-8752 exported as `INTEL-` / `LIGENCE` and its governing statute as `McKinney-` / `Vento`. The merge required `.islower()` on the continuation, so it declined to join at all. Swept over the fixture corpus with GPO's XML as an oracle, the defect is larger than the uppercase case it was filed for, and in two directions the issue did not name: 1,031 uppercase continuation, never joined -- the filed defect 423 lowercase continuation, joined but WELDED -- `project-based` -> `projectbased` 1,542 break at a PAGE SEAM, never joined -- `_merge_print_lines` runs per page Whether the hyphen belongs to the word is not decidable at the break: GPO breaks a compound at its own hyphen and PDFium marks that identically to a syllable break (`McKinney<FFFE>22 Vento`). It IS decidable from the rest of the document, which almost always spells the word out somewhere it did not have to break. `BreakEvidence` indexes those spellings and decides 47,875 of 50,679 breaks with zero errors against the XML; pooling the compared pair's two documents adds 826 more, also with zero; the remainder falls to letter case. Measured paired on the 50,679 breaks the XML can decide: 2,980 wrong -> 118 wrong, 1,336 fixed, no site regressed. Through the pipeline's own output the corpus goes from 3,201 defective sites to 213, and the 180 that remain are enumerated in tests/data/pdf/word_break_residuals.json with the reason each could not be decided. Not gated on PDFium's U+FFFE marker, though it names the same breaks where it appears. Swept corpus-wide, 1,390 of 52,044 line-final hyphens carry no marker -- every one in an enrolled print, plus a 0-4% tail of ordinary syllable breaks in numbered prints -- so gating on it would stop joining all of them. ADR 0020 evidence for the correspondence that moved: of 23 pinned pairs, 6 changed and all 6 emit FEWER changes (-32 total, almost entirely spurious `added`), none more. That is the #31 defect class -- a word split in one version and whole in the other reading as a change -- clearing. One round-1 revocation moved with it, in 115-hr-5895/3->4 (72 -> 71); the other 34 pairs are identical under both extractors. The gate is hyphen-sensitive, which nothing else in the suite is: `normalize_for_cross_format` deletes hyphens before matching and says so, and `normalize_for_recall` rewrites `Child- Rescue` back into `Child-Rescue` at compare time. Both are right for a recall question and both make this defect invisible. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…e-deriving them (#650) Schema 2.1 adds optional `join_points: {v1, v2}`: where the printer broke a word across a line inside `full_text`, and whether reflowing that break drops the hyphen. Additive; `null` wherever the text has no printed line breaks to reflow, which is the XML pipeline always and the PDF pipeline's reflowed rendering. The field exists because the answer is not recoverable from the text it describes. GPO prints a syllable break and a compound broken at its own hyphen identically, so a consumer looking at `INTEL-` / `LIGENCE` and `McKinney-` / `Vento` cannot tell which closes up and which keeps its hyphen. The extractor decides it from evidence the consumer does not have -- how the rest of the document spells that word -- and now carries the decision instead of discarding it. That is #653's rule: a consumer may derive by applying facts the document carries, and may not re-infer facts it omits. In-browser search was the third implementation of the rule and the one that disagreed with the parser in both directions. It now applies a `data-join` attribute the renderer stamps on the row from the carried points. Its own test asserted agreement with the parser's merged text and passes by construction rather than coincidence. Encoding: `at` is delta-encoded and `drop` is a bitstring, not an array of objects. Measured on 118-hr-8752, 2.7 kB against 14.4 kB -- 2% of the rendered text rather than 11%. Both were built and compared rather than assumed. Verified by reconstruction: applying every carried point to the print-faithful text reproduces `pdf_full_text` exactly, and `pdf_full_text` is computed without reference to the points. A wrong offset, a wrong delta or a flipped bit diverges. Baseline churn is the version string alone -- 17 digests move, no pair's change count does. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… measurement (#650) `pdf_blocks._rejoin_cross_page_hyphens` and `pdf_anchors._join_major_run`'s de-hyphenating branch each still describe how to reflow a printed word break, and each describes the pre-#650 answer: lowercase-continuation-only in the first, drop the hyphen unconditionally in the second. Neither runs any more -- `extract_clean_pages` resolves every break before either sees a `Page` -- but a reader has no way to tell a dormant rule from a live one, and four copies of this rule quietly disagreeing is the whole of #650. Measured rather than asserted, over the fixture corpus: pdf_blocks._rejoin_cross_page_hyphens 0 joins performed pdf_anchors._join_major_run (hyphen) 0 de-hyphenations performed Kept rather than deleted: both remain reachable for a `Page` assembled by hand (tests, and the anchor parser's single-page `parse_lines`), neither of which can carry a page seam. Each docstring now says it is dormant, that it is not the project's rule, and that `pdf_text.BreakEvidence` is. Deleting them is a separate change with its own test surface, and is not bundled here. Comment-only, and shown to be: regenerating the round-1 sentinel moves 27 `parser_revision` fields -- the digest covers module bytes by design, so a comment moves it -- and 0 pairing streams. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Review against the amended contract found the repair passing its own gate while being wrong in three ways. Each is fixed and pinned by a negative control that was confirmed to go red when the mistake is reintroduced. 1. The repair joined broken words to running-header chrome. PDFium floats `H. R. 3547—61` to the top of the next page's reading order, where it lands between a page-seam break and its continuation. It starts with an alphanumeric, so the "continuation starts with a letter" guard let it through, producing `evidence-H.`, `project-H.`, `Hart-Scott-H.` -- and all 16 were then blessed in the residual fixture, so the gate certified them. In the corpus this shape outnumbers the genuine cross-page uppercase breaks 16 to 5, so the dominant cross-page case was corrupted while the named examples passed. `_SEAM_CHROME` now declines the join; the 16 entries are gone from the fixture. 2. Pooling let one version overrule the other about its own text. Both sides were merged into one index and the majority won, so a version that writes `Non-Dedicated` once and never `NonDedicated` is rendered `NonDedicated` when the other version writes the closed form more often. That corrupts a version that was never ambiguous AND erases a real spelling change, so the diff stops reporting a difference the documents have. Evidence is now ordered tiers: a document's own text first, the sibling consulted only where it is silent, then the case-shape fallback. The earlier "0 cross-version disagreements after pooling" measurement was vacuous. Pooling applies identical evidence to both sides, so agreement was guaranteed by construction; it measured a tautology. 3. The gate used the vocabulary oracle the contract rejects. It asked whether a produced form appears anywhere in the version's XML, which certifies a reconstruction from an unrelated occurrence and cannot choose at all when a bill uses both spellings. `XmlOracle` now matches the reconstruction against its PDF neighbours, trigram first and bigram either side as a fallback, comparing both candidates WITHIN a tier before falling to the next -- the negative control caught that mixing tiers reads a bigram for one candidate as competing with a trigram for the other. Sites the oracle cannot decide are now counted per version and asserted, so silent growth of the uncovered set is itself a failure. Residuals 180 -> 167 and 53 sites recorded as undecided. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…650 The amended #650 states that in-browser search is out of scope: search reads the print-faithful DOM and normalizes it independently, so repairing `full_text` does not change it, and the two fail differently. `full_text` leaves all six named examples split, while search is already correct for the compounds and wrong only for `INTEL-` / `LIGENCE`. Shipping the search rewrite here would contradict the contract this PR is meant to satisfy. Removed from this branch, and preserved on `issue653-join-points-wip` for a separate pull request against #653: - `join_points` on the canonical, and the schema 2.1 bump that carries it - `pdf_print_join_points` / `print_join_points` - the `data-join` row attribute and the in-browser search rewrite that reads it - the fixtures and version pins those required What stays is what #650 asks for: `full_text` reconstructs a word the printer broke, the print-faithful rendering is untouched, and nothing here changes the public contract. `schema_version` returns to 2.0 and the regenerated baselines lose the churn the version string caused, so the remaining diff in those artifacts is the word-break repair alone. Verified: full suite green, and the canonical baselines regenerate with no change to any pair's change count. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…pooling claims (#650) Three corrections from review. The undecided-site budget failed OPEN. It was read with a default, so deleting a version's entry silently disabled the control for that version -- the one edit most likely to be made to quiet it, and precisely the weakening the count exists to prevent. A missing entry is now a failure, and a companion test asserts the budget covers exactly the collected versions, so a stale entry is caught alongside a missing one. Confirmed red by deleting a key. `BreakEvidence` still advertised the pooling method and its result -- "pooling a compared pair's two documents adds 826 more, also with zero errors" -- for a method that no longer exists and a measurement this branch has already acknowledged was vacuous. `PrintPages` and the regeneration script described pooling too. All three now describe own evidence first with the sibling as fallback, and the docstring records why pooling was withdrawn rather than leaving the discredited claim in place for the next reader to trust. The result quantities are re-derived. The figures previously quoted came from the vocabulary oracle the contract rejects, so they could not stand once the oracle changed. Measured with the shipped `XmlOracle`, evaluating `develop`'s parser and this branch's with identical code over the same population of 50,785 judged breaks: today 3,027 wrong repair 193 wrong fixed 2,834 undecided 47 -> 53 The 193 decomposes exactly: 193 occurrences over 167 distinct fragment pairs, which is the residual fixture, and ZERO splits left unjoined. The fallback figures were also re-derived and are unchanged (2,001 sites, case shape wrong on 118, 5.9%). #650 is amended with the corrected population and an account of the 74 -> 53 movement. Sentinel regeneration moves 27 parser_revision fields and no pairing stream, so the docstring edits are behaviour-neutral. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… can (#650) Docstring-only. Audited after the same failure was found in a sibling branch's test: a description claiming more than the code does. This file had four instances. The oracle paragraph still described VOCABULARY membership -- "every word form the reflowed PDF produces at a break should be a word form the XML has" -- which is the framing #650 explicitly rejects and which the code stopped doing when `XmlOracle` became positional. A reader checking the test against the contract would have found the file arguing for the contract's discarded alternative. Neither clause stated its boundary. Both are scoped to sites the oracle can judge, and a break attested in neither form or in both equally is EXCLUDED and counted, not passed. That set is where the #535 chrome seams sit, so "no word left split" was never the guarantee: it is "none left split where the XML can say so". Written down explicitly, because a reader who took the stronger reading would stop looking for #535, which is still open. `_unjoined_words` and `_RESIDUALS` were both renamed and the prose was not updated. The class's comparative claim was carried over from a standalone probe whose tiering differed from the shipped class, and was quoted after that tiering was corrected. A caveat is binding on the person who wrote it, so it is re-derived here rather than trusted: measured with `XmlOracle` over the sites the repair produces, aligned matching decides 50,785 against vocabulary's 50,716, contradicting it at zero sites. The claim holds; it just had no current measurement behind it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
#650) The residual paragraph said both candidate forms are absent from the document's own text "and from the other version compared with it". This gate runs the SINGLE-DOCUMENT path (`cached_pages` -> `extract_clean_pages`) and never consults a sibling, so it established no such thing. The sibling-borrowing path exists, in `compare/pdf.py`, but it is not what these counts measure, and reading them as though it were would overstate what a comparison's reader is exposed to in the wrong direction. Now says what the gate exercises, and notes separately that the production comparison can borrow the other version after the document's own evidence is silent, so the residual set here is an upper bound. `_residuals` carried the same claim and is fixed with it. Also runs `ruff format`, which is why `lint-format` failed the last hosted CI run on this branch (`e171b36f`); every test job in that run passed. Three files this branch introduced were unformatted. The diff is line rejoining and trailing blank lines, with no semantic change, and it is included here rather than left because the branch cannot go green without it. Regenerating the round-1 sentinel afterwards moves 27 parser_revision fields and no pairing stream: the revision digest covers module bytes by design, so reformatting moves it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…word-breaks # Conflicts: # examples/hr8752_pdf_diff.html # tests/data/pdf_canonical_baseline.json # web/webapp/sample/example.html
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #650. Reviewed against the amended contract in that issue.
What this fixes
A word GPO's printer broke across a line reached
full_textbroken._merge_print_linesrequired.islower()on the continuation, soINTEL-/LIGENCEandMcKinney-/Ventowere never joined at all.Measured against each version's own XML, the defect is larger than the uppercase case it was filed for:
project-based→projectbased_merge_print_linesruns per pageWhether the hyphen belongs to the word is not decidable at the break, but it is decidable from how the rest of the document spells that word.
Measured with the shipped aligned oracle, evaluating
develop's parser and this branch's with identical code over the same 50,785 judged breaks:The 193 decomposes exactly: 193 occurrences over 167 distinct per-version residual keys (
version,left,right), which is the residual fixture row-for-row, and zero splits left unjoined among the sites the oracle judges. Those 167 keys cover 102 globally distinct fragment pairs, since the same pair recurs across versions.An earlier revision of this PR claimed 2,980 → 118 with 1,336 fixed. Those came from the vocabulary oracle the contract rejects and are withdrawn, not restated: the numbers above are re-derived on the population every other figure here is now stated over.
All six named examples reflow correctly, each against its own version's XML:
INTELLIGENCE,McKinney-Vento,E-Verify,Non-Dedicated,Climate-Related,Child-Rescue.Three false greens found in review, now fixed
The first version of this PR passed its own gate while being wrong in three ways. Each fix is pinned by a negative control, and each control was confirmed to go red when the mistake is reintroduced.
1. The repair joined broken words to running-header chrome. PDFium floats
H. R. 3547—61to the top of the next page, where it lands between a page-seam break and its continuation. It starts with an alphanumeric, so the guard let it through, producingevidence-H.,project-H.,Hart-Scott-H.— and all 16 were then blessed in the residual fixture, so the gate certified them. In the corpus this shape outnumbers the genuine cross-page uppercase breaks 16 to 5, so the dominant cross-page case was corrupted while the named examples passed.2. Pooling let one version overrule the other about its own text. Both sides were merged into one index and the majority won. A version writing
Non-Dedicatedonce and neverNonDedicatedwas renderedNonDedicatedwhen the other version wrote the closed form more often — corrupting a version that was never ambiguous, and erasing a real spelling change so the diff reports no change where there is one. Evidence is now ordered tiers: own text first, sibling only where silent, then the fallback.3. The gate used the vocabulary oracle the contract rejects. It asked whether a produced form appears anywhere in the version's XML, which certifies a reconstruction from an unrelated occurrence and cannot choose at all when a bill uses both spellings.
XmlOraclenow matches against PDF neighbours, trigram first then bigram, comparing both candidates within a tier before falling to the next. Writing that control caught a further bug in my own oracle: mixing tiers reads a bigram for one candidate as competing with a trigram for the other. Sites the oracle cannot decide are now counted per version and asserted, so silent growth of the uncovered set is itself a failure.Scope
The
join_pointsschema 2.1 change and the in-browser search rewrite have been removed from this PR and preserved onissue653-join-points-wipfor a separate PR against #653. The amended #650 states search is out of scope: it reads the print-faithful DOM and normalizes independently, so repairingfull_textdoes not change it, and the two fail differently.The net diff against the base now touches no file under
schema/orsrc/deltatrack/formatters/.schema_versionstays 2.0.Verification
tests/test_pdf_word_break_recall.py, the only hyphen-sensitive check in the suite.normalize_for_cross_formatdeletes hyphens before matching and says so;normalize_for_recallrewritesChild- Rescueback at compare time. Both are right for a recall question and both make this defect invisible.ruff check .andruff format --check .(255 files). See Base below for hosted status.ADR 0020 evidence
Of 23 pinned pairs, 6 changed and all 6 emit fewer changes (−32 total, almost entirely spurious
added); none emits more. That is the #31 defect class clearing. One round-1 revocation moved with it,115-hr-5895/3→4(72 → 71); the other 34 pairs are identical under both extractors, andtest_pdf_round1_revocation's pinned figure is updated 224 → 223 with that attribution recorded.Known limits
† HR 4366 EASfooter sits between break and continuation. That is Running page header/footer is read as bill text and splices into the middle of a word #535, untouched here. These sites are excluded rather than passed: the chrome token is not a word, so the oracle cannot judge them, which is why they sit outside the "zero left unjoined" result above rather than contradicting it.Base and hosted status
origin/develophas been merged into this branch and pushed. GitHub now reports thepull request mergeable; the three conflicts were confined to derived artifacts
(
examples/hr8752_pdf_diff.html,tests/data/pdf_canonical_baseline.json,web/webapp/sample/example.html) and were resolved by regenerating them on the newbase rather than hand-merged. No non-derived implementation code conflicted:
developchanged nothing undersrc/deltatrack/parsers/orsrc/deltatrack/compare/,which is the code this branch touches.
Two things to know when reading the artifact diff:
developreworkedsrc/deltatrack/formatters/in the same span, so the regeneratedexamples and canonical baseline carry those changes as well as this branch's. That
movement is not word-break drift.
parser_revisionfields move whenever the parser's byteschange, including for comments and formatting. Every regeneration on this branch has
moved those fields and no pairing stream, which is what makes those edits
demonstrably behaviour-neutral.
Hosted CI is running on the merged head. The test figures quoted above are local; the
hosted verdict is whatever the run reports, and this PR should not be read as green
until it lands.
🤖 Generated with Claude Code