Repository navigation
Conversation
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
SBrandeis
marked this pull request as ready for review
October 21, 2026 09:55
ArthurZucker
reviewed
Nov 6, 2026
ArthurZucker
left a comment
Collaborator
There was a problem hiding this comment.
very critical / needed let's make aboslute certain we follow 0.23 and that this is amply tested!
Overall Okay!
Comment on lines
+8
to
+9
| /// The oracle: onig, the regex engine of tokenizers 0.23, answering "is `c` in `prop`?". | ||
| /// |
Collaborator
There was a problem hiding this comment.
onig was not the default engine! no?
Collaborator
Author
There was a problem hiding this comment.
It was: onig is the default regex engine in every 0.23 release, and it's what the wheels ship. It has been the engine since May 2020.
- History: onig came in on 2020-05-27 (dde08b0, "Rust - Use Onig for byte-level") as a plain dependency. It became an optional feature, on by default, on 2022-06-10 (Adding
unstable_wasmfeature + example to runtokenizerson wasm. #1009, which addedunstable_wasmandfancy-regexas the alternative). The default hasn't changed since. - Features:
default = ["progressbar", "onig", "esaxx_fast"]atv0.23.0rc0,v0.23.1,v0.23.2andv0.23.3(Cargo.toml#L100).fancy-regexis only picked whenonigis off (utils/mod.rs#L5-L12), and the Python and Node bindings use the default features. - Version:
bindings/python/Cargo.lockpinsonig6.5.3 /onig_sys69.9.3 for 0.23.1 to 0.23.3. That Oniguruma carries Unicode 16.0 data, hence the=69.9.3pin inbitmap_gen. - Released wheels: checked on 0.23.1, 0.23.2 and 0.23.3 from PyPI:
from tokenizers import Regex
from tokenizers.pre_tokenizers import Split
def split(pattern, text):
return [p for p, _ in Split(Regex(pattern), "isolated").pre_tokenize_str(text)]
split(r"\h", "xfx") # ['x', 'f', 'x']: \h is a hex digit (onig), not horizontal space (PCRE2)
split(r"\h", "x x") # ['x x']
split(r"\p{Cn}", "a\U0001FAE9b") # not split: U+1FAE9 (Unicode 16) is assigned
split(r"\p{Cn}", "a\U0001FAEAb") # split: U+1FAEA (Unicode 17) is unassignedThe Unicode 16 data has been there since 0.21.2 (2025-06-24; 0.21.1 still treats U+1FAE9 as unassigned), so this exact behaviour is what users have had for about 17 months.
SBrandeis
force-pushed
the
fix-unicode17-atom-tables
branch
2 times, most recently
from
November 12, 2026 12:17
c45a68c to
5bb2ef5
Compare
SBrandeis
force-pushed
the
fix-unicode17-atom-tables
branch
from
November 19, 2026 16:34
ac6bd44 to
42a159d
Compare
The numeric atom comes from char::is_numeric, whose Unicode version follows the toolchain. Rust 1.100 (Unicode 18) left 323 code points stale without any test failing.
SBrandeis
force-pushed
the
fix-unicode17-atom-tables
branch
from
November 19, 2026 16:53
42a159d to
472c411
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #2557
TL;DR
bitmap_genused Unicode 17 to generate the atom tables used by bitcannonThis PR refatcors it to use
onigto classify unicode characters and generate the atom_tables powering bitcannonEnsures backwards-compat with how
0.23would classify each char