Bring the user docs in line with 3.0's readings, views and language choices

docs/AGENTS.md

@@ -40,6 +40,8 @@ The rules below were distilled from PR #588, which restructured `customize.rst`
 
 **Check a claim end to end, through the call the reader makes.** "One `phd` key covers `Phd`" was true of the key match and false of `capitalized()`'s output, which leaves mixed case alone unless `force=True`. Verify against what the reader would observe, not an intermediate step.
 
+**A claim about what a language choice settles needs a word whose readings sit in different files.** A plain choice keeps every name reading and switches off the other files' vocabulary readings of a word its own file lists as vocabulary, so it leaves a fork where one file holds both readings and ends one where the readings are split. #1146's first draft tested `van` (both in `nl.toml`) and `MA` (both in `en.toml`), and wrote that no choice settles a word with a name reading; review ran `JOSE SILVA DO` (credential in `en`, particle in `pt`), which `locales.PT` reads as an unreported family name. The second and third reviews found the strict half: a strict choice keeps only the readings its chosen files hold, name readings included, so a strict `nl` choice reads `JOHN SMITH MA` family `MA` and a strict `pt` one `Do Smith` family `Do Smith` (`do`'s particle in `pt`, its given reading in `en`), both unreported, while the plain `pt` choice keeps `Do` given and reports it. The second fix scoped this to a word no chosen file holds, which the `Do` case refutes: state what a choice keeps reading by reading, never word by word. `Lexicon.default().readings_of(word)` shows which file holds each reading; put one word of each layout in the check, under a plain and a strict choice: both readings in the chosen file, the vocabulary reading in another file, the name reading in another file, and a word the chosen file does not hold.
+
 **A claim about code needs the code.** "An empty `honorific_tails` stops the peel at its first guard" is checked by reading the guard, not by parsing a name. Find it through the stage headers: each stage module (the eight in `nameparser/_pipeline/__init__.py`'s `STAGES`) declares the `Policy`/`Lexicon` fields it reads on a `Reads:` line. The shared helpers `_vocab.py` and `_pieces.py` have no such line and read the lexicon too, so grep them as well.
 
 ## Linking

docs/concepts.rst

@@ -129,9 +129,9 @@ three places, and which one is decided by a single question: what does
 this setting vary with?
 
 * :class:`~nameparser.Lexicon` holds everything that varies by
-  **language**: the vocabulary — titles, particles, suffixes,
-  conjunctions, and the rest of the word lists the parser matches
-  against.
+  **language**: the vocabulary, written as readings — what each word
+  may be read as (a title, a particle, a credential, a given name...),
+  each held in the file of the language whose names it occurs in.
 * :class:`~nameparser.Policy` holds everything that varies by
   **data source or application**: the behavior switches — name order,
   patronymic rules, delimiters, strip flags — anything that changes
@@ -156,6 +156,15 @@ of the same parsed data wants "Given Family" is a fact about where the
 string is going next, decided at the moment you render it — that's a
 rendering argument, not something baked into how the name was parsed.
 
+The default lexicon reads every language's words at once, which is
+why it serves data whose language nobody can declare. Where you can
+declare it, a :class:`~nameparser.Locale` carries the declaration: a
+choice of languages, which selects among the shipped readings by the
+files they are written in, beside a lexicon fragment and a policy
+patch, applied together by :func:`~nameparser.parser_for`. A locale
+pack is a :class:`~nameparser.Locale` that nameparser ships; see
+:ref:`choosing-languages`.
+
 This replaces 1.x's single configuration object, which mixed all three
 concerns — vocabulary, behavior, and output formatting — into one
 mutable bag plus a ``string_format`` template string. Sorting a

docs/customize.rst

@@ -11,6 +11,17 @@ why the split is drawn there.
 Vocabulary: Lexicon
 --------------------
 
+When your data is in a known language, choosing it is the first thing
+to try, before editing words one at a time. A choice of languages
+(:ref:`choosing-languages`) makes the chosen language's reading win on
+a word it lists as vocabulary, switching off another language's
+(Portuguese ``do`` the particle over English ``DO`` the credential),
+and turns on the readings its file holds off by default; a strict
+choice (:ref:`strict-choice`) also drops the other languages' words
+written in Latin letters. The edits below are for what a choice
+leaves open, and for data whose language you cannot declare, which the
+default vocabulary, reading every language at once, already serves.
+
 Configuring by reading
 ~~~~~~~~~~~~~~~~~~~~~~~
 
@@ -425,6 +436,20 @@ when it appears alone, and the parse reports the fork as an ambiguity.
 A title with a ``given`` reading works differently, being a title only
 inside a phrase (:ref:`titles-also-given-names`).
 
+A plain choice of languages keeps every name reading, and a strict one
+(:ref:`strict-choice`) every reading its chosen files hold, so neither
+settles these words where the chosen language's own file holds both
+readings. English lists ``ma`` as a credential and as a surname, so a
+strict English choice still reads ``JOHN SMITH MA`` by the count and
+reports it, and Dutch lists ``van`` as a particle and as a given name.
+A choice ends such a fork where it switches off another language's
+vocabulary reading of the word (below), and a strict choice also
+wherever one of the two readings sits in a file it does not choose,
+which it drops, name readings included: a strict Dutch choice reads
+``JOHN SMITH MA`` family ``MA``, and a strict Portuguese one ``Do
+Smith`` family ``Do Smith``, both unreported. The recipes in this
+section are the way to settle the rest.
+
 Credentials that are also surnames
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
@@ -518,6 +543,22 @@ free:
   one adds it back with ``Lexicon.default().add("cha",
   kind="credential")``.
 
+Where the credential reading is another language's, choosing the
+language settles the word with no edit. English lists ``DO`` as a
+credential (Doctor of Osteopathy), and Portuguese lists ``do`` as a
+particle, so choosing Portuguese switches the credential reading off
+and a trailing ``DO`` is a name word, with no fork left to report
+(:ref:`choosing-languages`):
+
+.. doctest::
+
+    >>> from nameparser import locales, parser_for
+    >>> parse("JOSE SILVA DO").suffix, parse("JOSE SILVA DO").ambiguities
+    ('DO', (Ambiguity('suffix-or-name': 'DO'),))
+    >>> portuguese = parser_for(locales.PT).parse("JOSE SILVA DO")
+    >>> portuguese.family, portuguese.ambiguities
+    ('DO', ())
+
 .. _titles-also-given-names:
 
 Titles that are also given names
@@ -545,6 +586,15 @@ a title only inside a title phrase: behind a style
     >>> parse("Major General John Smith").title
     'Major General'
 
+A language can run the other way, holding a given-name reading of a
+title that its file leaves off until the language is chosen. ``Lady``
+is a given name under a Spanish choice (``locales.ES``), ``Hon`` under
+a Chinese choice of languages (a :class:`~nameparser.Locale` with
+``languages=frozenset({"zh"})``; the ``zh`` pack makes none, so it
+leaves ``Hon`` a title), and ``Guru`` under ``locales.NE``, each staying a title
+inside a phrase; :ref:`choosing-languages` shows the first two, and the table of shipped
+packs in :doc:`locales` the third.
+
 Where such a word opens the name and a listed phrase could have held
 it, the parse reports a ``title-or-name`` ambiguity, its alternative
 the title reading (:ref:`reading-not-taken`). A word no listed phrase
@@ -929,8 +979,13 @@ right and then reverses the remaining two, so
 
 Nothing keys this order to a script the way the East Asian defaults
 below do — Vietnamese is written in the Latin alphabet, which carries
-no order of its own — so it applies only where you set it, and there
-is no ``vn`` locale pack.
+no order of its own — so it applies only where you set it. For
+Vietnamese data, the ``vi`` pack (``parser_for(locales.VI)``) sets this
+order and also keeps the Vietnamese syllables other languages read as
+particles or connectives (``Van``, ``Le``, ``Do``, ``The`` and others)
+as name words, and ``Chi``, ``Ba`` and ``Do`` out of the credentials (see
+:ref:`vietnamese-names`);
+the bare ``Policy`` above is for data that wants the order alone.
 
 Declaring the order settles where a surname ends
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
@@ -1041,7 +1096,8 @@ consulted at all:
   obvious ``"Nguyen Van Minh"``: ``Van`` is the Dutch particle ``van``,
   so that name reads family ``Nguyen`` with ``Van Minh`` given under
   *both* family-first orders, and the choice between them makes no
-  difference.
+  difference. The ``vi`` pack takes the particle reading out, so under
+  it the name reads middle ``Van``, given ``Minh``.
 - **A never-given particle opening the name** overrides the declared
   order outright: where a particle that can never be a given name
   stands alone as the opening piece, the whole name is the surname, in

docs/locales.rst

@@ -191,7 +191,12 @@ alone, reads the other's names by its own word:
 
 Choosing both keeps both readings, as the default does. A reading
 that a word is borne as a name never settles anything: it keeps the
-name around the word in charge, under any choice.
+name around the word in charge, under any plain choice. A strict
+choice keeps only the readings its chosen files hold, beside the
+shared words and those in other scripts, so it drops a name reading
+written in Latin letters in another language's file, as it drops any
+other
+(`Reading one language only`_).
 
 A name reading held off by default is how a language claims a word
 another language's titles use. ``Hon`` is the British style (the
@@ -504,7 +509,9 @@ A Czech or Slovak woman who keeps her own surname beside her
 husband's writes the two without a hyphen, ``Alena Pešová Řehořová``,
 which the default reads as a middle name and a family name. The two
 packs read two family names (``two_family_names``, see
-:ref:`two-family-names`), so the double surname stays whole. Two given
+:ref:`two-family-names`), so the double surname stays whole, each
+name in its own view (``surname1`` ``Pešová``, ``surname2``
+``Řehořová``). Two given
 names pay for it: with no Czech or Slovak given-name list to keep a
 given name out of the family (none is published for the whole
 population), ``Petr Pavel Novák`` reads family ``Pavel Novák``. A family
@@ -612,7 +619,7 @@ Shipped packs
        ``mjr.``) and the degrees written
        after the name, and reads two
        family names (``Alena Pešová Řehořová`` → family ``Pešová
-       Řehořová``), so two given names read as a given name and a family
+       Řehořová``, ``surname2`` ``Řehořová``), so two given names read as a given name and a family
        name (see :ref:`west-south-slavic-packs`).
    * - ``sl``, ``hr``
      - Slovenian and Croatian vocabulary — a choice of that one
@@ -626,7 +633,8 @@ Shipped packs
        ``Lady`` as a given name (``Lady Heidenreich`` → given ``Lady``), and
        reads two family names, a list of Spanish given names keeping a
        given name out of the family (``Gabriel García Márquez`` →
-       family ``García Márquez``, ``José Luis Pérez`` → family
+       family ``García Márquez``, ``surname2`` ``Márquez``;
+       ``José Luis Pérez`` → family
        ``Pérez``). See `Spanish and Portuguese titles`_.
    * - ``es_ar``
      - Argentine Spanish — ``es`` with a longer given-name list, one
@@ -901,6 +909,19 @@ stays out of it:
     >>> spain.parse("José Luis Pérez").family
     'Pérez'
 
+``family`` keeps both names, and the parse gives each its own view:
+``surname1`` and ``surname2``, with ``surname1_base`` for filing by
+the first surname without its particles (see :ref:`aggregate-views`):
+
+.. doctest::
+
+    >>> marquez_es = spain.parse("Gabriel García Márquez")
+    >>> marquez_es.surname1, marquez_es.surname2
+    ('García', 'Márquez')
+    >>> vega_es = spain.parse("Juan de la Vega del Valle")
+    >>> vega_es.surname1, vega_es.surname1_base
+    ('de la Vega', 'Vega')
+
 Data from systems without the letter ñ writes ``Begoña`` as
 ``Begona``, so each listed name written with ñ is listed with n too,
 where that spelling is still mostly a given name. A name written with ñ
@@ -1018,13 +1039,15 @@ of a name by default, packs or no. ``Pandit``, ``Sardar``, ``Kumari``,
 ``Babu``, ``Acharya`` and ``Begum`` are titles under no pack: each opens
 far more names as a given name or a surname than as the title.
 
+.. _vietnamese-names:
+
 Vietnamese names
 ^^^^^^^^^^^^^^^^
 
 A Vietnamese name is written family name first, then the middle name,
 then the given name, and the default, reading given-first, reads it the
 other way round, diacritics or none. ``vi`` reads it in its own order,
-keeps ``Van`` (Văn), ``Le`` (Lê), ``Do`` (Đỗ) and ``The`` (Thế) as name
+keeps syllables such as ``Van`` (Văn), ``Le`` (Lê), ``Do`` (Đỗ) and ``The`` (Thế) as name
 words where other languages' particles and connectives would join them
 to the words beside them, keeps ``Chi``, ``Ba`` and ``Do`` from being
 read as a credential ending a name written in capitals

docs/usage.rst

@@ -175,7 +175,8 @@ reading kind is what :meth:`Lexicon.words(kind=...)
      - Field
      - Example
    * - Titles (``title``)
-     - adjacent titles
+     - adjacent titles; a title that is also a given name, only inside
+       a title phrase (:ref:`titles-also-given-names`)
      - ``title``
      - ``Asst. Vice Chancellor John`` → ``Asst. Vice Chancellor``
    * - Suffixes (``suffix``)
@@ -191,6 +192,11 @@ reading kind is what :meth:`Lexicon.words(kind=...)
        (``del Carmen``), which is ``middle``
      - ``family``
      - ``Juan de la Vega`` → ``de la Vega``
+   * - Ranks (``rank``)
+     - the following surname, inside the name; opening it, most are
+       titles (below)
+     - ``family``
+     - ``Jan Freiherr von Berg`` → ``Freiherr von Berg``
    * - Maiden markers (``birth_marker``)
      - the following name
      - ``maiden``
@@ -211,11 +217,63 @@ names together as easily as two surnames:
     >>> parse("Juan de la Vega y Rodriguez").family
     'de la Vega y Rodriguez'
 
-One shipped vocabulary works the other way round and so is not in the
-table above: the Korean surnames (``Lexicon.default().surnames``) *split* a
-word instead of merging two, and is covered under `East Asian names`_.
-:doc:`customize` covers how to change which words are in each of these
-sets.
+Most ranks (``Freiherr``, ``Graf``, ``Prinz``) are titles as well as
+particles, and read by where they stand: opening the name each is a
+title, and inside it the surname takes it in. ``Freiherrin`` holds no
+title reading, so opening a name it is a name word. ``St`` reads the same
+way, a title and a ``saint`` particle; the other ``saint`` words
+(``San``, ``Sta``, ...) hold no title reading, so opening a name none
+of them is a title:
+
+.. doctest::
+
+    >>> parse("Freiherr Hans von Berg").title
+    'Freiherr'
+    >>> parse("Jan Freiherr von Berg").family
+    'Freiherr von Berg'
+    >>> parse("St John Smith").title, parse("John St Clair").family
+    ('St', 'St Clair')
+
+Two shipped vocabularies work the other way round and so are not in
+the table above: they *split* a word instead of merging two. The
+Korean surnames (``Lexicon.default().surnames``) divide an unspaced
+hangul name, and the glued honorifics, the ``honorific`` readings
+carrying ``attached`` (``Lexicon.default().words(kind="honorific",
+attached="after")`` lists them), split off the end of a Chinese,
+Japanese or Korean name. Both are covered under `East Asian names`_.
+:doc:`customize` covers how to add and remove the readings behind each
+of these rows.
+
+Inspecting the vocabulary
+~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Two questions explain why a word read as it did, and each has its own
+call. What a word *may* be read as is the lexicon's answer,
+:meth:`Lexicon.readings_of <nameparser.Lexicon.readings_of>`, each
+reading naming the language files that hold it. What the parse *did*
+read it as is the token's, :attr:`Token.reading
+<nameparser.Token.reading>` (see :ref:`reading-taken`):
+
+.. doctest::
+
+    >>> from nameparser import Lexicon
+    >>> [(r.kind, r.languages) for r in Lexicon.default().readings_of("van")]
+    [('preposition', ('nl',)), ('given', ('nl',))]
+    >>> parse("Ludwig van Beethoven").tokens[1].reading
+    Reading('particle' as 'family', 'vocabulary', in nl)
+
+A lexicon answers for the readings it has on. A reading its file holds
+off until a caller chooses the language (:ref:`choosing-languages`) is not among
+the default lexicon's: ``Lady`` is a given name only under a Spanish
+choice, and only that parser's lexicon lists the reading:
+
+.. doctest::
+
+    >>> from nameparser import locales, parser_for
+    >>> [r.kind for r in Lexicon.default().readings_of("lady")]
+    ['nobility']
+    >>> [r.kind for r in parser_for(locales.ES).lexicon.readings_of("lady")]
+    ['nobility', 'given']
 
 A particle at the start of a name
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
@@ -721,6 +779,8 @@ and prints a note naming the install. :func:`~nameparser.parser_for` never
 does this on its own, so the same code parses alike wherever it runs.
 :doc:`customize` covers turning any of these behaviors off.
 
+.. _aggregate-views:
+
 Aggregate views
 ----------------
 
@@ -791,9 +851,11 @@ name written without the particle has none:
 
 ``surname2`` is the family's second name, where the parse read the
 family as two: with ``two_family_names`` on
-(:ref:`two-family-names`), or for a Chinese name opening with two
-surnames under ``zh`` (:ref:`east-asian-names`), and ``family`` keeps both. Every other name
-has none, a family comma's and a hyphenated one included:
+(:ref:`two-family-names`), as the ``es``, ``es_ar``, ``cs`` and ``sk``
+packs turn it on (:doc:`locales`), or for a Chinese name opening with
+two surnames under ``zh`` (:ref:`east-asian-names`), and ``family``
+keeps both. Every other name has none, a family comma's and a
+hyphenated one included:
 
 .. doctest::
 
@@ -1278,7 +1340,10 @@ The reading each word took
 
 Since 3.0 every token also says how the parse read it, whether or not
 a fork was reported. ``Token.reading`` is a :class:`~nameparser.Reading`
-of the same three parts, its role the token's own field:
+with the same parts, its role the token's own field, and two flags
+besides, ``first_surname`` and ``second_surname``, marking the words
+``surname1`` and ``surname2`` hold where the family was read as two,
+or had middle names folded into it (``middle_as_family``):
 
 .. doctest::