Skip to content

fix(tk-encode): take &str in PatternMatcher::extract_next - #2712

Merged
SBrandeis merged 2 commits into
mainfrom
fix/extract-next-str
Nov 19, 2026
Merged

SBrandeis merged 2 commits into
mainfrom
fix/extract-next-str

Conversation

@SBrandeis

@SBrandeis SBrandeis commented Nov 18, 2026 •

Copy link
Copy Markdown
Collaborator

TL;DR

extract_next uses &str instead of &[u8] to remove potential UB surface

Performance

Result: no regression. This PR is as fast as its parent.

  • Normal text (English): same speed. On llama-3 all builds are within 1.4 MiB/s of each other at about
    102 MiB/s; on bert, within 0.5 MiB/s at about 22 MiB/s.
  • Text packed with added tokens: same speed, within build noise. Building the parent twice, with
    no code change at all, already gives results up to 7.3% apart. This PR is at most 4.2% from the
    parent, and its median is never below the slower of the two parent builds.

So where a table below says "slower" for this PR, compare it with the "parent, rebuilt" row of the
same table: that row is the parent's own code, and it says "slower" by a similar or larger amount. Those
percentages are noise between builds, not a cost of this change.

Why there is nothing to pay for: the three from_utf8_unchecked are removed and no runtime check
replaces them. The helpers already worked on a str.

Full tables: tokbench on HF Jobs, this PR vs its parent vs the parent rebuilt

tokbench, pipeline engine, single thread, HF Jobs
cpu-performance (EPYC 7R13, rustc 1.93.0). Three builds: this PR (0490595), its parent
(1a9ec50), and the parent built a second time from another directory. 20 rounds, builds rotated
within a round, pinned to one CPU. Cells are the median over rounds ± half of its 95% bootstrap
interval; the "vs parent" rows are the median per-round change. Ids are identical across builds and
rounds.

No tokbench model has an lstrip, rstrip or single_word added token, so the first five models
are llama-3 plus the five <|xs0..4|> tokens of the added-special-* corpora, flagged as named.

lstrip tokens

english added-special-dense added-special-sparse
parent 102.1 ± 1.3 MiB/s 102.3 ± 0.1 MiB/s 112.0 ± 0.2 MiB/s
parent, rebuilt 103.5 ± 0.8 MiB/s 99.8 ± 0.2 MiB/s 108.8 ± 0.4 MiB/s
this PR 102.8 ± 1.2 MiB/s 101.2 ± 0.4 MiB/s 110.6 ± 0.3 MiB/s
parent, rebuilt vs parent 1.6% faster ± 1.7 2.4% slower ± 0.2 2.8% slower ± 0.2
this PR vs parent 0.7% faster ± 1.6 1.4% slower ± 0.4 1.0% slower ± 0.4

rstrip tokens

english added-special-dense added-special-sparse
parent 102.4 ± 1.0 MiB/s 99.4 ± 0.1 MiB/s 108.3 ± 0.3 MiB/s
parent, rebuilt 102.8 ± 1.3 MiB/s 97.9 ± 0.4 MiB/s 106.1 ± 0.1 MiB/s
this PR 102.4 ± 1.0 MiB/s 98.7 ± 0.2 MiB/s 107.9 ± 0.2 MiB/s
parent, rebuilt vs parent 1.2% faster ± 1.4 1.2% slower ± 0.3 2.0% slower ± 0.4
this PR vs parent 0.2% faster ± 1.9 0.6% slower ± 0.3 0.5% slower ± 0.3

single_word tokens

english added-special-dense added-special-sparse
parent 102.8 ± 1.2 MiB/s 48.7 ± 0.1 MiB/s 84.3 ± 0.5 MiB/s
parent, rebuilt 102.1 ± 0.8 MiB/s 45.1 ± 0.2 MiB/s 80.3 ± 0.4 MiB/s
this PR 102.9 ± 1.0 MiB/s 46.6 ± 0.1 MiB/s 81.6 ± 0.5 MiB/s
parent, rebuilt vs parent 0.4% slower ± 1.2 7.3% slower ± 0.4 4.6% slower ± 1.0
this PR vs parent 0.0% slower ± 1.5 4.2% slower ± 0.2 3.2% slower ± 0.5

all three flags

english added-special-dense added-special-sparse
parent 102.4 ± 1.7 MiB/s 72.9 ± 0.2 MiB/s 95.8 ± 0.5 MiB/s
parent, rebuilt 103.2 ± 1.2 MiB/s 71.6 ± 0.1 MiB/s 92.5 ± 0.2 MiB/s
this PR 103.3 ± 1.2 MiB/s 72.2 ± 0.1 MiB/s 92.5 ± 0.4 MiB/s
parent, rebuilt vs parent 0.3% faster ± 1.1 2.0% slower ± 0.3 3.4% slower ± 0.5
this PR vs parent 0.6% faster ± 1.1 0.9% slower ± 0.2 3.7% slower ± 0.3

no flag

english added-special-dense added-special-sparse
parent 102.3 ± 1.1 MiB/s 59.6 ± 0.2 MiB/s 97.1 ± 0.3 MiB/s
parent, rebuilt 102.5 ± 1.1 MiB/s 55.7 ± 0.2 MiB/s 94.5 ± 0.2 MiB/s
this PR 103.5 ± 1.8 MiB/s 58.7 ± 0.2 MiB/s 96.9 ± 0.3 MiB/s
parent, rebuilt vs parent 0.2% slower ± 0.8 6.5% slower ± 0.4 2.7% slower ± 0.6
this PR vs parent 0.7% faster ± 1.7 1.5% slower ± 0.6 0.2% slower ± 0.4

stock llama-3 (the corpora's tokens are not in its vocabulary)

english added-special-dense added-special-sparse
parent 102.2 ± 0.6 MiB/s 230.3 ± 0.8 MiB/s 296.9 ± 2.0 MiB/s
parent, rebuilt 103.0 ± 1.1 MiB/s 228.7 ± 1.1 MiB/s 297.1 ± 1.5 MiB/s
this PR 103.3 ± 0.9 MiB/s 233.4 ± 1.0 MiB/s 300.1 ± 3.6 MiB/s
parent, rebuilt vs parent 0.8% faster ± 1.3 0.9% slower ± 0.5 0.2% slower ± 0.8
this PR vs parent 1.2% faster ± 1.5 1.8% faster ± 0.5 0.7% faster ± 1.1

stock bert-base-uncased

english added-special-dense added-special-sparse
parent 22.0 ± 0.1 MiB/s 21.5 ± 0.1 MiB/s 22.4 ± 0.1 MiB/s
parent, rebuilt 21.5 ± 0.1 MiB/s 20.9 ± 0.1 MiB/s 21.8 ± 0.1 MiB/s
this PR 21.6 ± 0.1 MiB/s 21.1 ± 0.1 MiB/s 22.1 ± 0.1 MiB/s
parent, rebuilt vs parent 1.8% slower ± 0.5 3.0% slower ± 0.8 3.0% slower ± 0.6
this PR vs parent 2.0% slower ± 0.6 2.1% slower ± 0.9 2.2% slower ± 1.1
Reproduce: one HF Job, about 20 minutes

Save the script as hf-job.sh, then:

hf jobs run --detach --flavor cpu-performance --timeout 1h --secrets HF_TOKEN rust:1.93.0-bookworm bash -c "$(cat hf-job.sh)"

The tables above are printed at the end of the job log.

#!/usr/bin/env bash
# A/B on one HF Jobs host: tokenizers#2712 (04905957f) vs its parent, plus a second copy of the parent
# as an A/A control. No tokbench model has an lstrip, rstrip or single_word added token, so the
# xs-* models are llama-3 plus the five <|xs0..4|> tokens of the added-special corpora, flagged.
# Prints one table per model to the job log.
set -euo pipefail

MAIN_SHA=1a9ec50b1cdf1cf97da689aad32003aa73d49755
BRANCH_SHA=04905957f54d06670174b6e0168ea6e7bc664718
TOKBENCH_SHA=76d9f254678d52fb7bd8765b7a3af9eb4b91699e
ROUNDS=${ROUNDS:-20}
MODELS=(llama-3 bert-base-uncased)
VARIANTS=(xs-plain xs-lstrip xs-rstrip xs-single-word xs-all)
CORPORA=(english added-special-dense added-special-sparse)
ARMS=(main main2 branch)
W=/w

log() { echo "[$(date -u +%T)] $*"; }

log "phase: tools"
apt-get update -qq
apt-get install -y -qq --no-install-recommends clang cmake pkg-config libssl-dev python3 util-linux >/dev/null
curl -LsSf https://astral.sh/uv/install.sh | sh >/dev/null 2>&1
export PATH="$HOME/.local/bin:$PATH"
lscpu | grep -E "Model name|^CPU\(s\)|Thread|MHz" || true
rustc --version

log "phase: sources"
mkdir -p $W/out
git clone -q https://github.com/huggingface/tokbench $W/tokbench
git -C $W/tokbench checkout -q $TOKBENCH_SHA
git clone -q https://github.com/huggingface/tokenizers $W/tokenizers
git -C $W/tokenizers fetch -q origin $MAIN_SHA $BRANCH_SHA
git -C $W/tokenizers worktree add -q --detach $W/arm-main $MAIN_SHA
git -C $W/tokenizers worktree add -q --detach $W/arm-main2 $MAIN_SHA
git -C $W/tokenizers worktree add -q --detach $W/arm-branch $BRANCH_SHA

log "phase: data"
HF=(uvx --from huggingface_hub hf)
mkdir -p $W/data/models $W/data/fixtures
for c in "${CORPORA[@]}"; do
  "${HF[@]}" download huggingface/tokbench-corpora "fixtures/$c.txt" --repo-type dataset --local-dir $W/dl >/dev/null
  cp "$W/dl/fixtures/$c.txt" $W/data/fixtures/
done
for m in "${MODELS[@]}"; do
  case "$m" in
    bert-base-uncased) src="$m.json" ;;
    *) src="models/$m/tokenizer.json" ;;
  esac
  "${HF[@]}" download hf-internal-testing/tokenizers-test-data "$src" --repo-type dataset --local-dir $W/dl >/dev/null
  mkdir -p "$W/data/models/$m" && cp "$W/dl/$src" "$W/data/models/$m/tokenizer.json"
done
python3 - <<'EOF'
import copy, json, os

base = json.load(open("/w/data/models/llama-3/tokenizer.json"))
top = max(max(base["model"]["vocab"].values()), max(t["id"] for t in base["added_tokens"]))
variants = {
    "xs-plain": {},
    "xs-lstrip": {"lstrip": True},
    "xs-rstrip": {"rstrip": True},
    "xs-single-word": {"single_word": True},
    "xs-all": {"lstrip": True, "rstrip": True, "single_word": True},
}
for name, flags in variants.items():
    config = copy.deepcopy(base)
    for i in range(5):
        token = {"id": top + 1 + i, "content": f"<|xs{i}|>", "single_word": False, "lstrip": False,
                 "rstrip": False, "normalized": False, "special": True}
        config["added_tokens"].append({**token, **flags})
    os.makedirs(f"/w/data/models/{name}", exist_ok=True)
    json.dump(config, open(f"/w/data/models/{name}/tokenizer.json", "w"), ensure_ascii=False)
EOF
MODELS+=("${VARIANTS[@]}")
ls -la $W/data/fixtures $W/data/models/*

U="https://github.com/huggingface/tokenizers"
for arm in "${ARMS[@]}"; do
  log "phase: build $arm"
  T=$W/arm-$arm/tokenizers
  cfg=(--config "patch.\"$U\".tk-encode.path=\"$T/tk-encode\""
       --config "patch.\"$U\".tk-serialize.path=\"$T/tk-serialize\""
       --config "patch.\"$U\".tk-convert.path=\"$T/tk-convert\"")
  cp -R $W/tokbench $W/tokbench-$arm
  cd $W/tokbench-$arm
  export CARGO_TARGET_DIR=$W/target-$arm
  cargo update -q -p tk-encode -p tk-serialize -p tk-convert "${cfg[@]}"
  tree=$(cargo tree -p tokbench --features pipeline -i tk-encode "${cfg[@]}" | head -1)
  echo "$tree"
  [[ "$tree" == *"$T/tk-encode"* ]] || { echo "patch not applied for $arm" >&2; exit 1; }
  cargo build -q --release -p tokbench --features pipeline "${cfg[@]}"
  ls -la $CARGO_TARGET_DIR/release/tokbench
done

model_args=(); for m in "${MODELS[@]}"; do model_args+=(--model "$m"); done
corpus_args=(); for c in "${CORPORA[@]}"; do corpus_args+=(--corpus "$c"); done

log "phase: run, $ROUNDS rounds x ${#ARMS[@]} arms, pinned to cpu 4"
start=$(date +%s)
for r in $(seq 1 "$ROUNDS"); do
  for i in 0 1 2; do
    arm=${ARMS[$(( (r + i) % 3 ))]}
    taskset -c 4 $W/target-$arm/release/tokbench --reps 5 --no-memory --engine pipeline \
      --models $W/data/models --corpora $W/data/fixtures "${model_args[@]}" "${corpus_args[@]}" \
      --out $W/out/$arm-$r.json >$W/out/$arm-$r.log 2>&1 \
      || { tail -20 $W/out/$arm-$r.log; exit 1; }
  done
  now=$(date +%s)
  log "round $r/$ROUNDS done, elapsed $((now - start))s, eta $(( (now - start) * (ROUNDS - r) / r ))s"
done

log "phase: analyze"
python3 - <<'EOF'
import collections, glob, json, random, statistics

ARMS = {"main": "parent", "main2": "parent, rebuilt", "branch": "this PR"}
CORPORA = ["english", "added-special-dense", "added-special-sparse"]
obs = collections.defaultdict(dict)
hashes = collections.defaultdict(set)
for arm in ARMS:
    for path in glob.glob(f"/w/out/{arm}-*.json"):
        rnd = int(path.rsplit("-", 1)[1].split(".")[0])
        for run in json.load(open(path))["runs"]:
            meta, res = run["dataset_metadata"], run["results"][0]
            key = (meta["model"], meta["corpus"])
            obs[key + (arm,)][rnd] = res["mbps"]
            hashes[key].add((res["ids_hash"], res["total_tokens_produced"]))
print("RAW", json.dumps({"|".join(k): v for k, v in obs.items()}))
mismatches = [k for k, h in hashes.items() if len(h) != 1]
print("output mismatches across arms and rounds:", mismatches or 0)

rng = random.Random(0)

def median_ci(values):
    """Median and the half-width of its 95% bootstrap interval."""
    medians = sorted(statistics.median(rng.choices(values, k=len(values))) for _ in range(10_000))
    return statistics.median(values), (medians[9_749] - medians[250]) / 2

def throughput(key):
    median, half = median_ci(list(obs[key].values()))
    return f"{median:.1f} ± {half:.1f} MiB/s"

def change(key, arm):
    """Median of the per-round ratios to the parent: arms of one round ran back to back."""
    base, other = obs[key + ("main",)], obs[key + (arm,)]
    median, half = median_ci([other[r] / base[r] for r in sorted(base)])
    percent = (median - 1) * 100
    return f"{abs(percent):.1f}% {'faster' if percent >= 0 else 'slower'} ± {half * 100:.1f}"

for model in sorted({k[0] for k in obs}):
    print(f"\n**{model}**\n")
    print("| | " + " | ".join(CORPORA) + " |")
    print("|---|" + "---|" * len(CORPORA))
    for arm, label in ARMS.items():
        print(f"| {label} | " + " | ".join(throughput((model, c, arm)) for c in CORPORA) + " |")
    for arm in ("main2", "branch"):
        print(f"| {ARMS[arm]} vs parent | " + " | ".join(change((model, c), arm) for c in CORPORA) + " |")
EOF
log "done"

@HuggingFaceDocBuilderDev

Copy link
Copy Markdown

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@SBrandeis
SBrandeis marked this pull request as ready for review November 18, 2026 12:25
@SBrandeis SBrandeis changed the title fix(tk-encode): take &str in PatternMatcher::extract_next fix(tk-encode): take &str in PatternMatcher::extract_next Nov 18, 2026
AddedVocabulary's single_word, lstrip and rstrip checks reinterpreted the
input with from_utf8_unchecked, but the public trait method took &[u8]:
a single_word token and the bytes b"\x80[T]" walked off the slice decoding
an invalid str (miri: entering unreachable code). The only caller,
SpecialSegmentIterator, already holds a &str, so the type now says it and
the three unsafe blocks are gone.
@SBrandeis
SBrandeis force-pushed the fix/extract-next-str branch from fc09919 to 2b36ca3 Compare November 19, 2026 10:05
The lstrip check sliced the input at match_end before a helper that only
reads s[..match_start]. As a str slice that cut costs a char-boundary check
per lstrip match and nothing uses it. Without it, SpecialSegmentIterator::next
is 3 instructions smaller than before the &str change (x86-64, rustc 1.93).
@SBrandeis
SBrandeis marked this pull request as draft November 19, 2026 14:45
@SBrandeis
SBrandeis marked this pull request as ready for review November 19, 2026 15:39
@SBrandeis
SBrandeis merged commit 9817e71 into main Nov 19, 2026
31 checks passed
@SBrandeis
SBrandeis deleted the fix/extract-next-str branch November 19, 2026 16:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants