Skip to content

fix(bpe): apply end_of_word_suffix to single-character words - #2484

Merged
SBrandeis merged 1 commit into
huggingface:mainfrom
MohammadHijjawi97:fix-bpe-single-char-end-of-word-suffix
Nov 19, 2026
Merged

SBrandeis merged 1 commit into
huggingface:mainfrom
MohammadHijjawi97:fix-bpe-single-char-end-of-word-suffix

Conversation

@MohammadHijjawi97

Copy link
Copy Markdown
Contributor

With end_of_word_suffix set, a one-character word must encode as char + suffix (it is both the first and the last character), which is what convert_affixed does and what 0.23.x returns. On main it encodes as the bare character instead.

Repro (1.0.0rc2 vs 0.23.2):

import json, os, tempfile
from tokenizers import Tokenizer

config = {
    "version": "1.0", "truncation": None, "padding": None, "added_tokens": [], "normalizer": None,
    "pre_tokenizer": {"type": "WhitespaceSplit"}, "post_processor": None, "decoder": None,
    "model": {"type": "BPE", "dropout": None, "unk_token": None, "continuing_subword_prefix": "##",
              "end_of_word_suffix": "</w>", "fuse_unk": False, "byte_fallback": False, "ignore_merges": False,
              "vocab": {"a": 0, "b": 1, "a</w>": 2, "b</w>": 3, "##b</w>": 4, "ab</w>": 5},
              "merges": [["a", "##b</w>"]]},
}
path = os.path.join(tempfile.mkdtemp(), "tokenizer.json")
json.dump(config, open(path, "w", encoding="utf-8"))
print(Tokenizer.from_file(path).encode("ab a").ids)
# 0.23.2:     [5, 2]   ("ab</w>", "a</w>")
# 1.0.0rc2:   [5, 0]   ("ab</w>", "a")

Cause: prove_fold marks every single-character vocab entry as foldable ("a single atom has no pair to merge"), so the fold emits a for the pretoken a before convert_affixed gets to decorate it. That shortcut only holds when there is no suffix (a prefix never applies to the first character). With a suffix, the entry now goes through merge_word like any other, which proves it is not its own encoding.

Test: end_of_word_suffix_applies_to_single_char_words in tk-encode/src/models/bpe/tests.rs (fails on main with [0] vs [2]). cargo test -p tk-encode, cargo fmt --check and cargo clippy -p tk-encode --all-targets --all-features -D warnings pass.

@SBrandeis SBrandeis left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Clean and simple, thank you for fixing this!

@SBrandeis
SBrandeis requested a review from ArthurZucker October 1, 2026 09:15
@HuggingFaceDocBuilderDev

Copy link
Copy Markdown

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@ArthurZucker ArthurZucker left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a good "patch", but I am wondering if we can't create the folds with the suffix? (in the meantime fine to merge as is indeed, probably a todo for perfs)

@MohammadHijjawi97

Copy link
Copy Markdown
Contributor Author

Thanks @ArthurZucker! Agreed, this is a correctness patch and folding with the suffix is a good perf follow-up.

As I read it, with an end_of_word_suffix set, a pretoken w encodes to the w</w> entry while the fold lookup finds the bare w entry, so prove_fold cannot mark entries foldable and the fold fast path is effectively off; this PR just makes that hold for single characters too. Doing it properly would mean proving at load time that each w + suffix entry is what merging w produces, then letting a lookup on w return that id, which needs either a "fold to" id stored on the bare entry or a small separate lookup keyed by the stripped words.

`prove_fold` marked every single-character vocab entry as foldable, on
the grounds that a lone atom has no pair to merge. With an
`end_of_word_suffix` that does not hold: a one-character word is both
its first and last character, so `convert_affixed` looks it up as
`char + suffix`. The fold short-circuited that, so "a" encoded as `a`
instead of `a</w>`, unlike 0.23.x.

Only take the single-character shortcut when the model has no suffix,
and let the merge engine prove the entry otherwise.
@SBrandeis
SBrandeis force-pushed the fix-bpe-single-char-end-of-word-suffix branch from e405bce to 078953a Compare November 19, 2026 13:42
@SBrandeis
SBrandeis merged commit 0af3d4f into huggingface:main Nov 19, 2026
30 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants