Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -273,7 +273,7 @@ Most modules define a `frozenset` of known name pieces; `capitalization.py` and
- `suffixes.py` — `SUFFIX_ACRONYMS` (with periods, e.g. "M.D.") and `SUFFIX_WORDS` (e.g. "Jr."), plus `GLUED_HONORIFICS` (#308), the subset of `SUFFIX_WORDS` the peel may split off the END of a name token — a separate, harsher set, since the glued position has no writer-drawn boundary to lean on
- `particles.py` — `PARTICLES` (family-name particles, e.g. "de", "van") and `NON_GIVEN_NAME_PARTICLES`, the curated subset that is *never* a standalone given name (a name whose opening PIECE is one of them, standing alone, is all surname — "de Mesnil" — under EVERY `name_order` since #359, and the degenerate bare "de" with nothing to fold into still stays as it is. What post_rules rule 1b enforces is one clause wider than the leading shape, and reading it as leading-only is how the FAMILY_FIRST bug got in: where a member stands ALONE as a piece, either opening the name or in the given position, the name is left with no given name at all — the given and the middles fold into the family. Two shapes, one repair — opening the name it pulls the rest in, and in the given position (`"Mesnil de"` under `FAMILY_FIRST`, where the given position is the trailing piece) it folds into the family beside it. So the rule asks by opening POSITION, read off `pieces`, as well as by the GIVEN role; the role test alone caught both shapes only because under the default order the opening piece IS the given. It is a lone PIECE throughout, and stating it any wider is false: under `FAMILY_FIRST` the given position of `Juan de la Vega` holds the whole chain `de la Vega`, a three-token piece rather than a lone particle, so 1b declines and reports it — #359 records that case as working as intended — and the degenerate bare `de` keeps given `de` because it has nothing to fold into. (`Sir de Mesnil` reported given `de Mesnil` and no family at all in the default order, which was the same guard declining on a chained piece; #367 removed the chain rather than touching this rule — a title is now transparent to the leading-particle exception, so the piece is lone again, 1b fires, and the name reads family `de Mesnil` like the untitled form.) A leading particle OUTSIDE the set is genuinely order-dependent and still splits — "van Gogh" is family "van", given "Gogh" under both family-first orders — since a word that CAN be a given name leaves `name_order` a real question to answer; what the set decides under any of the three orders is that such a leading particle records a `PARTICLE_OR_GIVEN` ambiguity and one inside it records none); `Lexicon.particles_ambiguous` is its complement within `PARTICLES`, so the two mark OPPOSITE sets — see the flip warning in `docs/migrate.rst` before translating either
- `bound_given_names.py` — `BOUND_GIVEN_NAMES` (bound given-name prefixes, e.g. "abdul", "abu"); a group-stage rule joins the first non-title piece to its following piece before roles are assigned, reserving a family word unless a family comma has already fixed it or a given-name title stands ahead (#369, rules.md#P5) (v1's `_join_bound_first_name`, ported into `_pipeline/_group.py` and gone from the tree — the v1 descriptions further down are history, not current code)
- `conjunctions.py` — `CONJUNCTIONS` (e.g. "and", "of") used to chain multi-word titles
- `conjunctions.py` — `CONJUNCTIONS` (e.g. "and", "of") used to chain multi-word titles and to join name parts (rules.md#P3), plus `CONJUNCTIONS_AMBIGUOUS` (#383/#479), the subset of single cased letters that read as an INITIAL in a name written wholly in one case — a marker set with no v1 `Constants` attribute of its own, so the only v1 knob that reaches it is deleting the conjunction
- `maiden_markers.py` — `MAIDEN_MARKERS` (e.g. "née", "geb.") routing the following name to `maiden`
- `surnames.py` — `KOREAN_SURNAMES`, the census list the 2.0 API splits unspaced hangul on (#271). With `maiden_markers.py` it is one of the two data modules `Constants` has **no** attribute for: both reach the parse only through `Constants._snapshot()` → `Lexicon`, so the v1 surface stays frozen and there is no v1 knob to turn either off (the opt-out is the 2.0 `Policy`)
- `capitalization.py` — `CAPITALIZATION_EXCEPTIONS` mapping (e.g. `{'phd': 'Ph.D.'}`)
Expand Down Expand Up @@ -317,7 +317,7 @@ The 2.0 rewrite lands as underscore-private modules alongside the v1 code. These
- **Parser owns config-dependent conveniences**: `Parser.matches`/`Parser.capitalized`/`Parser.revise` exist because the `ParsedName` equivalents fall back to DEFAULT config for str/omitted arguments (documented loudly in both docstrings). `revise` harvests tokens from a full sub-parse of each replacement value (tags kept minus `FOLDED_TAG`, roles forced, the R1 entry pass `suffix_entries` re-run over the forced state so a suffix value's entries follow its own commas, ambiguities discarded); the merge tail is shared with `replace()` via `ParsedName._with_field_tokens`. `Parser.capitalized` delegates through `name.capitalized(self.lexicon)` specifically so `_parser` never imports `_render` — keep it that way.
- **Per-word vocabulary fields warn on multi-word entries** (`_normset`/`_normpairs` via `_warn_dead_entry`, UserWarning, never a raise — see the given_name_titles Gotcha for why raising is wrong). `given_name_titles` is the one multi-word-matched field and is exempt; `_edit` passes `warn=False` (add() warns once via the new instance's `__post_init__`; remove() stores nothing). The default vocabulary and every locale pack must stay warning-free (`test_default_lexicon_builds_warning_free`, `test_pack_vocabulary_entries_are_single_words`).
- **Invariants guard harm, not no-ops**: add a constructor check when violating it produces a *wrong parse*, not when it produces *nothing*. A false positive costs a working configuration; a true positive on an inert condition costs the user nothing, so that trade is never worth taking. `suffix_acronyms_ambiguous ∩ suffix_words` is guarded because the overlap loses a family name; `given_name_titles` is not, because an unreachable entry is simply never consulted (see Gotchas). Before adding one, construct the config it forbids and check what actually breaks. **Between raise and silence sits the construction-time `UserWarning`**, for a gap that is statically decidable, harmless to SOME deliberate caller, and indistinguishable-from-working for everyone else: the segmenterless activation (#337 — `parser_for(locales.JA)` without a segmenter behaved exactly like a working parser minus the feature) warns rather than raises because the inert JA registration is itself a pinned property, and a warning is filterable by the caller who wants exactly that. The message must carry every applicable remedy and no inapplicable one (the `ja_segmenter` hint fires only when a Japanese script is among the dead ones). Test fuzzers that legitimately construct such configs suppress the warning by MESSAGE, never by category — a blanket `UserWarning` ignore would mask the next construction diagnostic (`_quiet_parser` in `tests/v2/test_properties.py` is the pattern).
- **The shim TRANSLATES; it never raises on a config v1 accepted, and never silently changes the parse**: `Constants._snapshot()` is a translation boundary between v1's model and v2's invariants, and every transformation there carries its v1-reachability argument in a comment. Six exist today — `first_name_titles` re-folded per word (v1 joins-then-`lc`, v2 normalizes-then-joins), `suffix_acronyms_ambiguous ∩ acronyms` (a provable no-op), `suffix_words − ambiguous` (v1 already accepts the word via the acronym branch, so the addition is inert there), `particles_ambiguous ∪ (bound ∩ particles)` (a pinned deviation, `test_bound_never_given_prefix_deviates_on_two_pieces`), `honorific_tails = GLUED_HONORIFICS ∩ suffix_words` (#308 behavior with no v1 manager of its own, so the one v1 knob that reaches it is deleting the suffix word — which turns the peel off, `test_snapshot_removing_a_honorific_word_turns_the_peel_off`), and `maiden_delimiters − nickname_delimiters` on the POLICY half of the same method (v1 precedence: a pair in both v1 buckets parses as a nickname, while `Policy` resolves the overlap the other way, so the subtraction is what keeps the facade at v1 behavior, `test_snapshot_overlap_keeps_v1_nickname_precedence`). Note that last one is on the `Policy`, not the `Lexicon` — the roster is per-`_snapshot()`, not per-vocabulary-field, so a sweep that only reads the `Lexicon(...)` call misses it. When a v1 config cannot satisfy a v2 invariant, work out what v1 actually *does* with it — usually nothing — and reproduce that; weakening the invariant or letting the raise through are both wrong. **Test the case the translation decides**, not one where both branches agree: a test using an input v1 parses identically with and without the config pins nothing.
- **The shim TRANSLATES; it never raises on a config v1 accepted, and never silently changes the parse**: `Constants._snapshot()` is a translation boundary between v1's model and v2's invariants, and every transformation there carries its v1-reachability argument in a comment. Seven exist today — `first_name_titles` re-folded per word (v1 joins-then-`lc`, v2 normalizes-then-joins), `suffix_acronyms_ambiguous ∩ acronyms` (a provable no-op), `suffix_words − ambiguous` (v1 already accepts the word via the acronym branch, so the addition is inert there), `particles_ambiguous ∪ (bound ∩ particles)` (a pinned deviation, `test_bound_never_given_prefix_deviates_on_two_pieces`), `honorific_tails = GLUED_HONORIFICS ∩ suffix_words` (#308 behavior with no v1 manager of its own, so the one v1 knob that reaches it is deleting the suffix word — which turns the peel off, `test_snapshot_removing_a_honorific_word_turns_the_peel_off`), `conjunctions_ambiguous = CONJUNCTIONS_AMBIGUOUS ∩ conjunctions` (#383/#479 behavior with no v1 manager of its own either, so the one v1 knob that reaches it is deleting the conjunction — which turns the marking off, `test_snapshot_removing_a_conjunction_turns_the_marker_off`), and `maiden_delimiters − nickname_delimiters` on the POLICY half of the same method (v1 precedence: a pair in both v1 buckets parses as a nickname, while `Policy` resolves the overlap the other way, so the subtraction is what keeps the facade at v1 behavior, `test_snapshot_overlap_keeps_v1_nickname_precedence`). Note that last one is on the `Policy`, not the `Lexicon` — the roster is per-`_snapshot()`, not per-vocabulary-field, so a sweep that only reads the `Lexicon(...)` call misses it. When a v1 config cannot satisfy a v2 invariant, work out what v1 actually *does* with it — usually nothing — and reproduce that; weakening the invariant or letting the raise through are both wrong. **Test the case the translation decides**, not one where both branches agree: a test using an input v1 parses identically with and without the config pins nothing.
- **Reprs are bounded**: render which fields deviate from a named baseline and by how much, never contents (`Lexicon(default + titles: +2)`). `PolicyPatch`'s repr shows only set (non-UNSET) fields; `_order_repr` must never raise even on an unvalidated patch's garbage `name_order` (PolicyPatch defers validation to apply time); the sweep test in `tests/v2/test_reprs.py` pins that no config repr leaks the UNSET sentinel.
- **Every pipeline stage is one module holding one public function of the same name** (`_tokenize.py`/`tokenize`), with the exceptions named where they stand — `_group.py`'s `marker_run_length`, a shared predicate (mechanisms.md#ONE-PREDICATE-PER-QUESTION), and since #511 `_post_rules.py`'s `suffix_entries`, the R1 entry pass as a function, because `Parser.revise` runs that one pass over a forced-role sub-parse and a stage's tail cannot be called on its own — **and its module docstring declares the contract in three labelled lines**: `Consumes:` what it takes from `ParseState`, `Produces:` what it hands back, `Reads:` which `Policy`/`Lexicon` fields it consults. `Reads:` is the load-bearing one — it makes "which stage do I touch for this feature?" a grep rather than a read-through. The authority for the stage set is `_pipeline/__init__.py`'s `STAGES` (eight, and not public API) with the field-ownership map in `ParseState`'s docstring, pinned by `tests/v2/pipeline/test_state.py`; NOT rules.md, which is implementation-free by its own preamble and whose `implemented:` names modules honoring a rule rather than stages. A `_pipeline/` module that is not a stage says so in its first line instead (`_assemble.py`: "Not a stage: …", omitting `Reads:` because it consults neither), so absence of the three lines is a claim about the module rather than an oversight. Provenance: §5 of the 2026-07-11 conventions spec, recorded here 2026-08-16.
- **A claim about WHICH STAGE or WHICH LAYER does something is checkable — check it before writing it.** The pipeline is eight stages with a written ownership map (`ParseState`'s docstring, pinned by `tests/v2/pipeline/test_state.py`), and `parse(s).tokens` prints every token's role and tags, so "extract assigns this", "classify never sees that", "group consumes it" each have a one-command answer. #329's prose claimed delimited maiden content is *"claimed whole before classify has tagged anything inside it"*; measured, `classify` tags the marker fine and only the CONSUMING is missing, because `_group`'s rule walks `pieces` and a token that already carries a role is not in `pieces`. Two different mechanisms, one plausible sentence covering both. That single claim then shipped SIX times across three correction rounds, which is the part worth internalizing: **when a mechanism claim turns out wrong, sweep for where else you wrote it, and sweep again at the END of the change over the words the change itself just added.** Prose density here means one idea lives in a docstring, a case note, a release-log entry and this file at once; the implementer working against a wrong mechanism is the person most likely to restate it; and rewriting a mechanism claim is writing one, so the correction earns the same one-command check as the original — two of the six instances were fresh errors introduced by the sentence fixing the previous one. **Adjacency is the trap.** The claim that feels already-known is the one about the neighbouring stage or the neighbouring layer: one comment block in `tests/v2/test_facade_cases.py` got the exception type, the raising layer, the skip mechanism, the count of skipped rows, and which row was blocked all wrong at once — every one a claim about `_config_shim` versus real 1.4.0, written from reasoning, in the file whose whole job is translating between them. What finally held was not better prose but moving the claim into a test (`_CORE_ONLY_IDS`), which cannot be wrong the way a sentence can.
Expand Down
76 changes: 63 additions & 13 deletions docs/customize.rst
Original file line number Diff line number Diff line change
Expand Up @@ -99,15 +99,20 @@ Removing works the same way, and drops the word from recognition:

A few fields mark a subset of another — ``given_name_titles`` over
``titles``, ``particles_ambiguous`` over ``particles``,
``suffix_acronyms_ambiguous`` over ``suffix_acronyms``, and
``suffix_acronyms_ambiguous`` over ``suffix_acronyms``,
``conjunctions_ambiguous`` over ``conjunctions``, and
``honorific_tails`` over ``suffix_words``. Entries belong in the base
field too, so add to both and remove from the marker first. The last
three enforce that: anything else raises ``ValueError`` naming the
orphans rather than leaving a marker entry that no rule will ever
consult. ``given_name_titles`` is deliberately unchecked — a title run
is matched as one space-joined string, or by that run's last word, so a
legitimate entry like ``"sir and dame"`` is no single word in
``titles`` — and an orphan there is inert rather than harmful.
field too, so add to both and remove from the marker first. Three of
them enforce that — ``particles_ambiguous``, ``suffix_acronyms_ambiguous``
and ``honorific_tails`` raise ``ValueError`` naming the orphans, because
an orphan in each of those does real harm rather than nothing. The
other two are deliberately unchecked because an orphan there is inert:
``given_name_titles`` matches a title run as one space-joined string, or
by that run's last word, so a legitimate entry like ``"sir and dame"``
is no single word in ``titles``; and a ``conjunctions_ambiguous`` entry
is only ever read for a word that is a conjunction, so
``remove(conjunctions={"e"})`` simply works and the stale marker entry
is never consulted.

Turning title detection off
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Expand Down Expand Up @@ -187,11 +192,13 @@ each way a source might punctuate it.
Words that are also ordinary names
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Two fields — ``suffix_acronyms_ambiguous`` and ``particles_ambiguous``
— mark entries from ``suffix_acronyms`` and ``particles`` that are also
plausible as ordinary name words on their own (an acronym suffix that
doubles as a nickname, a particle that doubles as a given name). They
don't add new vocabulary by themselves; they narrow how an existing
Three fields — ``suffix_acronyms_ambiguous``, ``particles_ambiguous``
and ``conjunctions_ambiguous`` — mark entries from ``suffix_acronyms``,
``particles`` and ``conjunctions`` that are also plausible as ordinary
name words on their own (an acronym suffix that doubles as a nickname,
a particle that doubles as a given name, a connective letter that
doubles as an initial). They don't add new vocabulary by themselves;
they narrow how an existing
entry is read when it appears alone. If you're not sure whether a word
you're adding is one of these ambiguous cases, weigh how often it is a
name against how often it is the credential. Marking it ambiguous is
Expand Down Expand Up @@ -270,6 +277,49 @@ ambiguity is recorded and it becomes part of the surname — under any
>>> Parser(lexicon=lex).parse("van Gogh").family
'van Gogh'

``conjunctions_ambiguous`` is the same idea for one-letter connectives.
A single letter written against the name's own case is an initial and
one written with it is the connective — but a name written wholly in
one case, all upper or all lower, says nothing either way, and this is
the set that decides it there. ``e`` is the one entry shipped: a bare
``E`` initial is common where an ``e`` between two surnames is rare, and
``y`` runs the other way, so ``y`` joins even written as a bare capital.

.. doctest::

>>> parse("jose e maria santos").middle # 'e' reads as an initial
'e maria'
>>> parse("JUAN GARCIA Y LOPEZ").family # 'y' joins
'GARCIA Y LOPEZ'
>>> parse("Jose e Maria Santos").given # mixed case decides itself
'Jose e Maria'

A member also reports the fork, so a caller can see which reading was
taken:

.. doctest::

>>> [a.kind for a in parse("jose e maria santos").ambiguities]
[<AmbiguityKind.CONJUNCTION_OR_INITIAL: 'conjunction-or-initial'>]

If your data is Portuguese, where ``e`` links surnames the way ``y``
does in Spanish, take it out and the connective reading comes back:

.. doctest::

>>> lex = Lexicon.default().remove(conjunctions_ambiguous={"e"})
>>> Parser(lexicon=lex).parse("jose e maria santos").given
'jose e maria'

If your data is Dutch, where a bare single letter is an initial and
never a connective, add the other one instead:

.. doctest::

>>> lex = Lexicon.default().add(conjunctions_ambiguous={"y"})
>>> Parser(lexicon=lex).parse("juan garcia y lopez").middle
'garcia y'

Bound given names
~~~~~~~~~~~~~~~~~~

Expand Down
Loading