Skip to content

fix(bitcannon): backwards compatibility: use onig to generate atom_tables - #2558

Open
SBrandeis wants to merge 8 commits into
mainfrom
fix-unicode17-atom-tables
Open

SBrandeis wants to merge 8 commits into
mainfrom
fix-unicode17-atom-tables

Conversation

@SBrandeis

@SBrandeis SBrandeis commented Oct 21, 2026 •

Copy link
Copy Markdown
Collaborator

Closes #2557

TL;DR

bitmap_gen used Unicode 17 to generate the atom tables used by bitcannon

This PR refatcors it to use onig to classify unicode characters and generate the atom_tables powering bitcannon
Ensures backwards-compat with how 0.23 would classify each char

@HuggingFaceDocBuilderDev

Copy link
Copy Markdown

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@SBrandeis SBrandeis changed the title fix(bitcannon): backwards compatibility with Unicode 16 fix(bitcannon): backwards compatibility: use onig to generate atom_tables Oct 21, 2026
@SBrandeis
SBrandeis marked this pull request as ready for review October 21, 2026 09:55

@ArthurZucker ArthurZucker left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

very critical / needed let's make aboslute certain we follow 0.23 and that this is amply tested!

Overall Okay!

Comment thread tokenizers/bitmap_gen/src/lib.rs Outdated
Comment on lines +8 to +9
/// The oracle: onig, the regex engine of tokenizers 0.23, answering "is `c` in `prop`?".
///

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

onig was not the default engine! no?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It was: onig is the default regex engine in every 0.23 release, and it's what the wheels ship. It has been the engine since May 2020.

  • History: onig came in on 2020-05-27 (dde08b0, "Rust - Use Onig for byte-level") as a plain dependency. It became an optional feature, on by default, on 2022-06-10 (Adding unstable_wasm feature + example to run tokenizers on wasm. #1009, which added unstable_wasm and fancy-regex as the alternative). The default hasn't changed since.
  • Features: default = ["progressbar", "onig", "esaxx_fast"] at v0.23.0rc0, v0.23.1, v0.23.2 and v0.23.3 (Cargo.toml#L100). fancy-regex is only picked when onig is off (utils/mod.rs#L5-L12), and the Python and Node bindings use the default features.
  • Version: bindings/python/Cargo.lock pins onig 6.5.3 / onig_sys 69.9.3 for 0.23.1 to 0.23.3. That Oniguruma carries Unicode 16.0 data, hence the =69.9.3 pin in bitmap_gen.
  • Released wheels: checked on 0.23.1, 0.23.2 and 0.23.3 from PyPI:
from tokenizers import Regex
from tokenizers.pre_tokenizers import Split

def split(pattern, text):
    return [p for p, _ in Split(Regex(pattern), "isolated").pre_tokenize_str(text)]

split(r"\h", "xfx")               # ['x', 'f', 'x']: \h is a hex digit (onig), not horizontal space (PCRE2)
split(r"\h", "x x")               # ['x x']
split(r"\p{Cn}", "a\U0001FAE9b")  # not split: U+1FAE9 (Unicode 16) is assigned
split(r"\p{Cn}", "a\U0001FAEAb")  # split: U+1FAEA (Unicode 17) is unassigned

The Unicode 16 data has been there since 0.21.2 (2025-06-24; 0.21.1 still treats U+1FAE9 as unassigned), so this exact behaviour is what users have had for about 17 months.

@SBrandeis
SBrandeis force-pushed the fix-unicode17-atom-tables branch 2 times, most recently from c45a68c to 5bb2ef5 Compare November 12, 2026 12:17
@SBrandeis
SBrandeis force-pushed the fix-unicode17-atom-tables branch from ac6bd44 to 42a159d Compare November 19, 2026 16:34
The numeric atom comes from char::is_numeric, whose Unicode version follows the
toolchain. Rust 1.100 (Unicode 18) left 323 code points stale without any test failing.
@SBrandeis
SBrandeis force-pushed the fix-unicode17-atom-tables branch from 42a159d to 472c411 Compare November 19, 2026 16:53

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[1.0.0-rc.3] DeepSeek encode returns different token IDs from 0.23.2

3 participants