Skip to content

fix(bpe): decorate bytes, not characters, in byte-level models with affixes - #2592

Merged
SBrandeis merged 3 commits into
huggingface:mainfrom
patrasq:fix-byte-level-affixes
Nov 19, 2026
Merged

SBrandeis merged 3 commits into
huggingface:mainfrom
patrasq:fix-byte-level-affixes

Conversation

@patrasq

@patrasq patrasq commented Oct 26, 2026

Copy link
Copy Markdown
Contributor

A byte-level word reaches the model spelled one character per byte, so end_of_word_suffix belongs on its last byte and continuing_subword_prefix on every byte but the first, which is what 0.23.x does. On main, convert_affixed decorates whole Unicode characters instead. As a result CLIP (byte-level, end_of_word_suffix: "</w>") returns different ids for any non-ASCII word.

Repro (main vs 0.23.3):

from tokenizers import Tokenizer

tok = Tokenizer.from_pretrained("openai/clip-vit-base-patch32")
print(tok.encode("fotografie cu pisică în București").ids) # photo of a cat in Bucharest (romanian)
# 0.23.3: [49406, 23687, 10348, 11224, 9230, 535, 128, 481, 48704, 333, 609, 36758, 132, 247, 2605, 49407]
# main:   [49406, 23687, 10348, 11224, 9230, 535, 128, 225, 48704, 333, 609, 36758, 132, 247, 2605, 49407]

pisică should end in Ä ĥ</w> (128, 481) but ends in Ä ĥ (128, 225) on main.

Cause: the decorated character (Äĥ</w> for ă) is not in the vocab, so </w> is dropped and the character falls back to its bytes. Where a whole character is a vocab entry, merging starts from it instead of from its bytes.

Fix: convert now dispatches affixed models on their atoms, as it already does for models without affixes. Byte-level models go to a new convert_affixed_bytes. It decorates each byte and looks it up with the affixes mapped the way byte_level::transform_vocab maps the vocab keys. A decorated byte missing from the vocab falls back to the plain byte, which is what push_unknown did for byte-level characters. That arm of push_unknown is no longer reachable, so this removes it.

This is independent of #2484, and CLIP needs both. I ran 4,020 seeded multilingual and random-Unicode texts. CLIP differs from 0.23.3 on 3,550 of them with #2484 alone and on 10 with both. All 10 contain codepoints unassigned in Unicode 14, which the v1 Split breaks into words differently; main already does that. On the same texts, the ids for openai-gpt, gpt2 and roberta-base don't change. #2550 renames the u32::MAX sentinel used here, so whichever PR lands second needs a one-line rebase.

Test: byte_level_end_of_word_suffix_decorates_the_last_byte in tk-encode/src/models/bpe/tests.rs. Both of its assertions fail on main: [120, 258] instead of [120, 195, 257], and [258, 256] instead of [195, 259]. make lint, make test and make oracle pass. CLIP isn't in the oracle fixtures, and I'm happy to add it once its tokenizer.json is in hf-internal-testing/tokenizers-test-data.

AI-assisted. I reviewed every changed line.

patrasq and others added 2 commits November 19, 2026 14:53
…ffixes

A byte-level word reaches the model one character per byte, so
`continuing_subword_prefix` and `end_of_word_suffix` belong on bytes.
`convert_affixed` decorated whole Unicode characters instead, which
moved ids for CLIP on any non-ASCII text: `</w>` landed on a character
missing from the vocab and was dropped, and merging started from
whole-character tokens.
@SBrandeis
SBrandeis force-pushed the fix-byte-level-affixes branch from 5f7e5f7 to ab56e7d Compare November 19, 2026 14:26
@SBrandeis
SBrandeis force-pushed the fix-byte-level-affixes branch from 2510701 to 3b883b4 Compare November 19, 2026 14:38

@SBrandeis SBrandeis left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks a lot for fixing this, LGTM

@HuggingFaceDocBuilderDev

Copy link
Copy Markdown

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@SBrandeis

Copy link
Copy Markdown
Collaborator

Encode throughput: this PR vs its parent

  • CLIP on multi-byte scripts (Russian, Chinese) is 7-9% slower, in the cells where the fix changes the ids.
  • CLIP on English and openai-gpt are unchanged (within ±1.5%, less then A/A rebuild noise).
  • gemma-4 is inconclusive on Russian and Chinese: 3.5% and 5.4% slower, with intervals (±2.6, ±4.7) that overlap the rebuild noise. A cost of a few percent on convert_chars is not ruled out.
  • llama-3, whose code path this PR does not touch, moves 1.2-1.8%: that is the noise floor of this setup.

Measured with tokbench (pipeline engine, encode only) in one HF Job (cpu-performance): three builds run back to back for 20 rotated rounds, pinned to one core. Cells are the median ± 95% bootstrap CI. "parent, rebuilt" is a second build of the same parent commit: it is the noise floor (up to ~2% here), so read "this PR vs parent" against that row.

llama-3: control, goes through convert_bytes, which this PR does not touch. Same ids.

english russian chinese
parent 104.7 ± 0.6 MiB/s 39.0 ± 0.2 MiB/s 30.4 ± 0.2 MiB/s
parent, rebuilt 103.4 ± 1.0 MiB/s 39.1 ± 0.1 MiB/s 30.1 ± 0.1 MiB/s
this PR 103.0 ± 0.8 MiB/s 38.4 ± 0.2 MiB/s 29.8 ± 0.1 MiB/s
parent, rebuilt vs parent 1.0% slower ± 0.7 0.2% slower ± 0.5 1.1% slower ± 0.8
this PR vs parent 1.2% slower ± 0.9 1.4% slower ± 0.5 1.8% slower ± 0.9

gemma-4: convert_chars. Same ids.

english russian chinese
parent 10.4 ± 0.1 MiB/s 19.9 ± 0.3 MiB/s 49.2 ± 2.4 MiB/s
parent, rebuilt 10.3 ± 0.0 MiB/s 19.4 ± 0.3 MiB/s 49.0 ± 1.1 MiB/s
this PR 10.3 ± 0.1 MiB/s 19.1 ± 0.2 MiB/s 46.8 ± 1.3 MiB/s
parent, rebuilt vs parent 1.9% slower ± 0.9 1.6% slower ± 2.6 1.6% slower ± 2.9
this PR vs parent 1.2% slower ± 0.7 3.5% slower ± 2.6 5.4% slower ± 4.7

openai-gpt: convert_affixed with character atoms. Same ids.

english russian chinese
parent 19.6 ± 0.1 MiB/s 28.3 ± 0.1 MiB/s 22.5 ± 0.1 MiB/s
parent, rebuilt 19.6 ± 0.1 MiB/s 28.2 ± 0.1 MiB/s 22.5 ± 0.1 MiB/s
this PR 19.8 ± 0.1 MiB/s 28.7 ± 0.0 MiB/s 22.7 ± 0.0 MiB/s
parent, rebuilt vs parent 0.1% slower ± 0.3 0.2% slower ± 0.5 0.4% slower ± 0.5
this PR vs parent 1.3% faster ± 0.3 1.4% faster ± 0.4 0.6% faster ± 0.4

clip (openai/clip-vit-base-patch32): convert_affixed with byte atoms, the case this PR fixes. The ids differ from the parent, so the two builds do different work.

english russian chinese
parent 15.1 ± 0.1 MiB/s 19.5 ± 0.1 MiB/s 15.6 ± 0.1 MiB/s
parent, rebuilt 15.4 ± 0.1 MiB/s 19.7 ± 0.0 MiB/s 15.8 ± 0.1 MiB/s
this PR 15.2 ± 0.1 MiB/s 17.8 ± 0.0 MiB/s 14.5 ± 0.0 MiB/s
parent, rebuilt vs parent 1.9% faster ± 0.4 1.2% faster ± 0.4 1.7% faster ± 0.3
this PR vs parent 1.0% faster ± 0.6 8.8% slower ± 0.4 7.4% slower ± 0.5
clip tokens produced english russian chinese
parent 228,489 534,764 847,774
this PR 228,498 567,017 848,297

Notes:

  • llama-3 sits at the noise floor, as expected for an untouched path.
  • gemma-4 on Russian and Chinese leans slower, but the intervals are wide and overlap the rebuilt-parent row: this run neither shows nor rules out a few percent there.
  • Every build enables fancy-regex in tokbench's pipeline engine (CLIP's patterns need it), so the absolute numbers are not comparable with other tokbench runs.
  • This run checks that ids agree or differ between builds. It does not check the new CLIP ids against a released version.

@SBrandeis
SBrandeis merged commit 006a108 into huggingface:main Nov 19, 2026
30 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants