Repository navigation
feat(bindings): e2e and golden tests for Python bindings - #2235
Merged
SBrandeis merged 9 commits intoJul 23, 2026
Merged
Conversation
Same suite, two builds: `make golden-release` runs it on the tokenizers wheel transformers resolves (always green, defines the target behavior); `make golden` runs it on the in-tree build, where the failures enumerate what the 1.0 rewrite is still missing. Four layers: the tokenizers API driven directly, the transformers tokenizer API, inference (generate/ pipeline), and training (Trainer/collators). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Scenarios lifted from the maintained examples/pytorch/ scripts at v5.14.1: run_qa's overflow/stride/sequence_ids/offsets span labeling, run_ner's word_ids label alignment, run_mlm's special_tokens_mask + masking collator, run_translation's text_target and seq2seq collator, plus token-classification pipeline spans, encoder-decoder generate, and the T5 Unigram+Metaspace archetype in the direct-API layer. 31 tests, all green on the release wheel. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Story-shaped tests mirroring how production stacks drive the tokenizer: chat serving (ChatML template + control tokens as added specials + history truncation to a token budget + left-padded batch generate + TextIteratorStreamer streaming == one-shot decode), RAG (token-budget chunking via offsets with exact source spans + mean-pooled embeddings + retrieval), and SFT (TRL-style completion-only masking, tokenizer pickled through datasets.map(num_proc=2), Trainer steps). 34 tests green on the release wheel. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
tests/golden now holds true golden-master tests: goldens/*.jsonl record everything the released wheel (0.23.1) produces on the 8 bench-model tokenizers — ids, tokens, offsets, masks, word ids, decodes and pair encodings for fixture excerpts and edge-case strings, plus ids digests over the fixture corpora. generate.py regenerates them from .release/ (never hand-edited, guarded); test_golden.py replays the inputs on the current build, one test per output domain per model, so failures read as a conformance matrix. Opt-in via TOKENIZERS_GOLDEN (make golden) until the rewrite is green, then the gate should be dropped. The transformers scenario suite moves to tests/e2e (make e2e / e2e-release) — flows there, values here. CI: e2e-release and golden-freshness (regen must reproduce the committed goldens byte for byte) required; one informational in-tree job runs both. First catch, dev build vs release on llama-2: identical ids but tokens[0] renders as '▁<s>' instead of '<s>' — the Prepend normalizer leaks into the inserted special token's string. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
from_pretrained needs it (the wheel's optional hub extra), so the network-marked pretrained test failed in a fresh venv — it only worked after `make examples` pulled the dep in via datasets. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
rustdoc compiles indented doc blocks as Rust doctests; the example is an illustration, not compilable code. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
ruff 0.16.0 widened the default rule set, breaking CI with no code change: import sorting (I001), loop vars captured by bench lambdas (B023), implicit string concat in lists (ISC004), open() without a context manager (SIM115), zip-for-pairwise (RUF007), and a blind pytest.raises(Exception) (B017). Pin the CI ruff so rule-set changes land deliberately. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
TL;DR