Skip to content

feat(bindings): e2e and golden tests for Python bindings - #2235

Merged
SBrandeis merged 9 commits into
feat/pipeline-python-apifrom
feat/python-transformers-tests
Jul 23, 2026
Merged

SBrandeis merged 9 commits into
feat/pipeline-python-apifrom
feat/python-transformers-tests

Conversation

@SBrandeis

@SBrandeis SBrandeis commented Jul 23, 2026 •

Copy link
Copy Markdown
Collaborator

TL;DR

  • Implement a set of E2E tests exercising the Python package through its own API and transformers' API
  • Those E2E also exercise ~real world flows to make sure those are supported
  • Implement a set of "golden tests" that ensure the output from the new code is byte-identical to the one from the latest released version (v0.23)
  • tooling (Makefile) and docs

SBrandeis and others added 5 commits July 23, 2026 17:01
Same suite, two builds: `make golden-release` runs it on the tokenizers
wheel transformers resolves (always green, defines the target behavior);
`make golden` runs it on the in-tree build, where the failures enumerate
what the 1.0 rewrite is still missing. Four layers: the tokenizers API
driven directly, the transformers tokenizer API, inference (generate/
pipeline), and training (Trainer/collators).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Scenarios lifted from the maintained examples/pytorch/ scripts at
v5.14.1: run_qa's overflow/stride/sequence_ids/offsets span labeling,
run_ner's word_ids label alignment, run_mlm's special_tokens_mask +
masking collator, run_translation's text_target and seq2seq collator,
plus token-classification pipeline spans, encoder-decoder generate,
and the T5 Unigram+Metaspace archetype in the direct-API layer.
31 tests, all green on the release wheel.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Story-shaped tests mirroring how production stacks drive the tokenizer:
chat serving (ChatML template + control tokens as added specials +
history truncation to a token budget + left-padded batch generate +
TextIteratorStreamer streaming == one-shot decode), RAG (token-budget
chunking via offsets with exact source spans + mean-pooled embeddings +
retrieval), and SFT (TRL-style completion-only masking, tokenizer
pickled through datasets.map(num_proc=2), Trainer steps). 34 tests
green on the release wheel.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
tests/golden now holds true golden-master tests: goldens/*.jsonl record
everything the released wheel (0.23.1) produces on the 8 bench-model
tokenizers — ids, tokens, offsets, masks, word ids, decodes and pair
encodings for fixture excerpts and edge-case strings, plus ids digests
over the fixture corpora. generate.py regenerates them from .release/
(never hand-edited, guarded); test_golden.py replays the inputs on the
current build, one test per output domain per model, so failures read
as a conformance matrix. Opt-in via TOKENIZERS_GOLDEN (make golden)
until the rewrite is green, then the gate should be dropped.

The transformers scenario suite moves to tests/e2e (make e2e /
e2e-release) — flows there, values here. CI: e2e-release and
golden-freshness (regen must reproduce the committed goldens byte for
byte) required; one informational in-tree job runs both.

First catch, dev build vs release on llama-2: identical ids but
tokens[0] renders as '▁<s>' instead of '<s>' — the Prepend normalizer
leaks into the inserted special token's string.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
from_pretrained needs it (the wheel's optional hub extra), so the
network-marked pretrained test failed in a fresh venv — it only worked
after `make examples` pulled the dep in via datasets.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@SBrandeis SBrandeis changed the title golden end-to-end tests through transformers v5 feat(bindings): e2e and golden tests for Python bindings Jul 23, 2026
SBrandeis and others added 4 commits July 23, 2026 21:33
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
rustdoc compiles indented doc blocks as Rust doctests; the example is an
illustration, not compilable code.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
ruff 0.16.0 widened the default rule set, breaking CI with no code
change: import sorting (I001), loop vars captured by bench lambdas
(B023), implicit string concat in lists (ISC004), open() without a
context manager (SIM115), zip-for-pairwise (RUF007), and a blind
pytest.raises(Exception) (B017). Pin the CI ruff so rule-set changes
land deliberately.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@SBrandeis
SBrandeis marked this pull request as ready for review July 23, 2026 20:00
@SBrandeis
SBrandeis merged commit 4162de8 into feat/pipeline-python-api Jul 23, 2026
42 of 44 checks passed
@SBrandeis
SBrandeis deleted the feat/python-transformers-tests branch July 23, 2026 20:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant