Skip to content

fix(added vocab): output original content for normalized vocab - #2723

Draft
SBrandeis wants to merge 4 commits into
mainfrom
fix/normalized-added-content
Draft

SBrandeis wants to merge 4 commits into
mainfrom
fix/normalized-added-content

Conversation

@SBrandeis

@SBrandeis SBrandeis commented Nov 19, 2026 •

Copy link
Copy Markdown
Collaborator

TL;DR

Store AddedTokens in full in BucketAddedVocabulary, which has the nice property of storing the original content of the added tokens

Fixes #2693
Supersedes #2485 (same bug on the write path: a normalized: true added token was written back by its normalized form, so ids moved after save or pickle).

What changes

  • AddedVocabulary keeps added_tokens: Box<[AddedToken]> by internal id; AddedTokenFlags is gone.
  • get_vocab, get_added_tokens_decoder and to_json report each token's declared content. Matching and decoding still use the normalized form, so ids and decoded text are unchanged.
  • llama-2.json: get_vocab_size(true) is 32000 again (32003 on main, 32000 on 0.23).
  • round_trip_preserves_ids_on_every_real_config is enabled and green.

Not in this PR: token_to_id still looks up the matcher form, so token_to_id("Hello") is None for a normalized token under a lowercasing normalizer.

Throughput (tokbench, HF Jobs)

Job 6afed1bf: EPYC 7R13, one pinned core, 30 rotated rounds, max-of-N, pipeline engine. Three arms: main 1a9ec50b1, a second build of main (A/A control) and this branch at 96c25fdb. Ids and decoded text are identical across arms and rounds. Read the change column against the A/A control column, which is the noise of two builds of the same source.

MB/s is the best of the 30 rounds. "A/A control" is the same change measured on the second build of main.

Encode (the added-* corpora are where the matcher reads the larger per-token entry):

model corpus main MB/s branch MB/s change A/A control
llama-3 english 105.5 105.5 +0.0% +0.1%
llama-3 added-special-dense 231.4 232.3 +0.4% -0.5%
llama-3 added-special-sparse 301.7 301.1 -0.2% -0.6%
llama-3 added-normalized-dense 340.5 340.2 -0.1% -0.6%
llama-3 added-normalized-sparse 333.5 332.5 -0.3% -0.5%
deepseek-v4 english 118.0 110.3 -6.5% -2.7%
deepseek-v4 added-special-dense 124.8 124.4 -0.3% -0.4%
deepseek-v4 added-special-sparse 183.9 183.2 -0.4% -0.5%
deepseek-v4 added-normalized-dense 252.0 251.0 -0.4% -1.2%
deepseek-v4 added-normalized-sparse 252.2 253.2 +0.4% -0.7%
bert-base-uncased english 22.5 22.2 -1.2% -2.2%
bert-base-uncased added-special-dense 22.1 21.9 -1.1% -3.9%
bert-base-uncased added-special-sparse 23.0 22.6 -1.6% -3.9%
bert-base-uncased added-normalized-dense 24.4 24.0 -1.5% -3.6%
bert-base-uncased added-normalized-sparse 24.2 23.8 -1.8% -3.0%
geomean -1.0% -1.6%

Decode:

model corpus main MB/s branch MB/s change A/A control
llama-3 english 310.3 323.6 +4.3% +1.3%
llama-3 added-special-dense 370.8 395.3 +6.6% +0.4%
llama-3 added-special-sparse 465.1 494.4 +6.3% +1.6%
llama-3 added-normalized-dense 386.2 403.2 +4.4% -1.3%
llama-3 added-normalized-sparse 433.4 471.5 +8.8% +2.6%
deepseek-v4 english 238.0 248.0 +4.2% +4.2%
deepseek-v4 added-special-dense 201.9 209.0 +3.5% +4.9%
deepseek-v4 added-special-sparse 290.5 302.1 +4.0% +5.9%
deepseek-v4 added-normalized-dense 271.5 281.3 +3.6% +3.9%
deepseek-v4 added-normalized-sparse 319.3 338.1 +5.9% +7.0%
bert-base-uncased english 7.3 7.3 +0.2% +1.3%
bert-base-uncased added-special-dense 3.4 3.4 -0.1% -0.1%
bert-base-uncased added-special-sparse 4.8 4.8 -0.7% +0.7%
bert-base-uncased added-normalized-dense 5.2 5.2 -0.7% +1.4%
bert-base-uncased added-normalized-sparse 6.4 6.4 +0.0% +1.2%
geomean +3.3% +2.3%

The branch is inside the A/A band everywhere except two places: deepseek-v4 encode on english (−6.5% against −2.7% for A/A) and llama-3 decode (3 to 6% above A/A). One job does not tell a real effect from a build-layout one for either.

Memory (tokbench measure memory, HF Jobs)

Job 6afede23: live heap, main 1a9ec50b1 against 96c25fdb, english corpus (added-special-dense gives the same deltas).

model added tokens heap after load, main branch heap after encode, main branch
deepseek-v4 1283 6.357 MB +87.6 KB 8.525 MB +88.5 KB
llama-3 256 8.654 MB +20.5 KB 10.822 MB +21.4 KB
t5-base 103 24.759 MB +7.8 KB 27.050 MB +8.1 KB
gpt2 1 2.740 MB −0.4 KB 4.909 MB −1.2 KB
bert-base-uncased 5 1.792 MB 0 4.061 MB +0.1 KB

The cost is the stored content plus the larger per-token entry: about 70 to 85 bytes per added token, 1.4% of the load heap on deepseek-v4 and under 0.3% elsewhere.

@HuggingFaceDocBuilderDev

Copy link
Copy Markdown

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@SBrandeis
SBrandeis marked this pull request as ready for review November 19, 2026 11:39
@SBrandeis
SBrandeis force-pushed the fix/normalized-added-content branch from 96c25fd to b173163 Compare November 19, 2026 16:07
@SBrandeis
SBrandeis marked this pull request as draft November 19, 2026 16:08

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[v1] get_vocab() / get_vocab_size() include the normalized form of normalized: true added tokens (Llama-2: 32000 → 32003)

2 participants