[derek73]

Two new local corpora of Vietnamese names, for measuring the Vietnamese pack proposed in #1108. As with the other local corpora, they are fetched into the cache and never committed.

What they are

corpus	source	records (2026-11-05)	parts
uit-vinames	UIT-ViNames (To et al., NLPIR 2020), the zip in the authors' GitHub repository, pinned by commit and sha256	21,090 distinct names of two words or more, family first, with diacritics (all but 109)	family / middle / given derived by the Vietnamese convention: first word, the words between, last word (local_uit.split, versioned by SPLIT_VERSION)
vn-assembly	the 15th National Assembly's deputies (dbqh.quochoi.vn), from OpenSanctions' vn_national_assembly export (CC BY-NC 4.0)	499 deputies: 7 GS.TS., 9 PGS.TS., 142 TS., 241 ThS., 100 untitled	each name composed as <title> <official name>, with the title read off the education the deputy states (title_of, versioned); the parts are derived by the same split

Neither source states the parts of a name, so the score measures agreement with the convention a reader assumes. For Kinh names that is the name's structure. For some other ethnicities' names (Ê-đê, for example) it is only position, and a score cannot see the difference. vn-assembly keeps the ethnicity so that its report can group by it.

Changes

• tools/differential/local_uit.py and local_vnassembly.py, each registered in local_corpora.SOURCES.
• local_agreement.FIELDED gains a row for each source, so both are scored on family, middle and given; vn-assembly is scored on its title too, which is never set aside.
• Fielded gains a group_floor field, so vn-assembly's ethnicity groups print from 10 records instead of 1,000. The field is repr=False, so scorer_fingerprint is unchanged for the four existing sources: psc-individuals, orcid-researchers, patentsview-inventors and cz-candidates all match master.
• Wholeness checks (AGENTS.md Gotchas):
  • UIT: a CSV, but the zip's sha256 is pinned, so a cut or changed body is refused before it is parsed, and the row count is pinned too.
  • OpenSanctions: JSONL. The index states the file's size and sha1, and both are checked. The deputy count is checked against the index's target_count. Lines are read with corpus_build.jsonl_lines, and records are written through the shared jsonl_line.
• Documentation: the README "Local corpora" table rows and a fetch paragraph, and an AGENTS.md pointer for changes to Vietnamese names.
• Tests: tests/v2/test_local_uit.py and tests/v2/test_local_vnassembly.py, offline, with invented names only.

Scores on the corpora (2026-11-05)

local_agreement.py <source> under the default parser: uit-vinames 13 of 21,090 agree, vn-assembly 0 of 499. Both are expected, since both corpora are written family first. The scores of the #1108 pack go in that PR's decisions entry.

Review

Reviewed by code-reviewer, pr-test-analyzer (37 mutations; the gaps it found are now covered: middle-only misses, the group floor, zero or two names, the boundary of the titled share), comment-analyzer, and a /simplify pass with four angles. A second review round covered the fix commit.

Skipped from /simplify:

• a shared URL-map urlopen stub in _differential_fixtures.py, which would touch test modules outside this change;
• a shared body-digest reader, which has only two callers.

🤖 Generated with Claude Code