Skip to content

Fix FalconMamba tokenizer conversion with nonterminal ByteLevel - #2672

Open
voropaevv wants to merge 4 commits into
huggingface:mainfrom
voropaevv:codex/legacy-bytelevel-sequence
Open

voropaevv wants to merge 4 commits into
huggingface:mainfrom
voropaevv:codex/legacy-bytelevel-sequence

Conversation

@voropaevv

Copy link
Copy Markdown

Fixes #2670.

The FalconMamba tokenizer runs Digits and Split on the output of ByteLevel. Preserve the byte mapping at that point in the converted sequence and pass the resulting spans to character BPE. Terminal ByteLevel keeps its existing fused path.

The regression checks a split inside the mapped bytes of β, both regex settings, and a canonical JSON round trip. With the original FalconMamba tokenizer, compile().encode("Hello\tworld 123").ids returns [9856, 192, 9091, 204, 10963], matching 0.23.3.

Checked the Rust workspace, Clippy, the local Python suite, and 566 FalconMamba inputs against 0.23.3 for token IDs and decoded text.

@voropaevv
voropaevv marked this pull request as ready for review November 11, 2026 02:57
Merge main and route mapped sequences through the validated PreTokens factory. Preserve private construction, mapped-text lifetimes and ByteMap length errors. Cover an empty prefix as well as existing split sequences.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[1.0.0rc4] FalconMamba tokenizer fails to load: a ByteLevel pre-tokenizer must be the last member of its Sequence

1 participant