Repository navigation
fix(bpe): decorate bytes, not characters, in byte-level models with affixes - #2592
Conversation
…ffixes A byte-level word reaches the model one character per byte, so `continuing_subword_prefix` and `end_of_word_suffix` belong on bytes. `convert_affixed` decorated whole Unicode characters instead, which moved ids for CLIP on any non-ASCII text: `</w>` landed on a character missing from the vocab and was dropped, and merging started from whole-character tokens.
5f7e5f7 to
ab56e7d
Compare
2510701 to
3b883b4
Compare
SBrandeis
left a comment
There was a problem hiding this comment.
Thanks a lot for fixing this, LGTM
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
Encode throughput: this PR vs its parent
Measured with tokbench ( llama-3: control, goes through
gemma-4:
openai-gpt:
clip (
Notes:
|
A byte-level word reaches the model spelled one character per byte, so
end_of_word_suffixbelongs on its last byte andcontinuing_subword_prefixon every byte but the first, which is what 0.23.x does. On main,convert_affixeddecorates whole Unicode characters instead. As a result CLIP (byte-level,end_of_word_suffix: "</w>") returns different ids for any non-ASCII word.Repro (main vs 0.23.3):
pisicăshould end inÄĥ</w>(128, 481) but ends inÄĥ(128, 225) on main.Cause: the decorated character (
Äĥ</w>foră) is not in the vocab, so</w>is dropped and the character falls back to its bytes. Where a whole character is a vocab entry, merging starts from it instead of from its bytes.Fix:
convertnow dispatches affixed models on their atoms, as it already does for models without affixes. Byte-level models go to a newconvert_affixed_bytes. It decorates each byte and looks it up with the affixes mapped the waybyte_level::transform_vocabmaps the vocab keys. A decorated byte missing from the vocab falls back to the plain byte, which is whatpush_unknowndid for byte-level characters. That arm ofpush_unknownis no longer reachable, so this removes it.This is independent of #2484, and CLIP needs both. I ran 4,020 seeded multilingual and random-Unicode texts. CLIP differs from 0.23.3 on 3,550 of them with #2484 alone and on 10 with both. All 10 contain codepoints unassigned in Unicode 14, which the v1
Splitbreaks into words differently; main already does that. On the same texts, the ids for openai-gpt, gpt2 and roberta-base don't change. #2550 renames theu32::MAXsentinel used here, so whichever PR lands second needs a one-line rebase.Test:
byte_level_end_of_word_suffix_decorates_the_last_byteintk-encode/src/models/bpe/tests.rs. Both of its assertions fail on main:[120, 258]instead of[120, 195, 257], and[258, 256]instead of[195, 259].make lint,make testandmake oraclepass. CLIP isn't in the oracle fixtures, and I'm happy to add it once itstokenizer.jsonis inhf-internal-testing/tokenizers-test-data.AI-assisted. I reviewed every changed line.