Repository navigation
Conversation
SBrandeis
left a comment
There was a problem hiding this comment.
Clean and simple, thank you for fixing this!
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
ArthurZucker
left a comment
There was a problem hiding this comment.
This is a good "patch", but I am wondering if we can't create the folds with the suffix? (in the meantime fine to merge as is indeed, probably a todo for perfs)
|
Thanks @ArthurZucker! Agreed, this is a correctness patch and folding with the suffix is a good perf follow-up. As I read it, with an |
`prove_fold` marked every single-character vocab entry as foldable, on the grounds that a lone atom has no pair to merge. With an `end_of_word_suffix` that does not hold: a one-character word is both its first and last character, so `convert_affixed` looks it up as `char + suffix`. The fold short-circuited that, so "a" encoded as `a` instead of `a</w>`, unlike 0.23.x. Only take the single-character shortcut when the model has no suffix, and let the merge engine prove the entry otherwise.
e405bce to
078953a
Compare
With
end_of_word_suffixset, a one-character word must encode aschar + suffix(it is both the first and the last character), which is whatconvert_affixeddoes and what 0.23.x returns. On main it encodes as the bare character instead.Repro (1.0.0rc2 vs 0.23.2):
Cause:
prove_foldmarks every single-character vocab entry as foldable ("a single atom has no pair to merge"), so the fold emitsafor the pretokenabeforeconvert_affixedgets to decorate it. That shortcut only holds when there is no suffix (a prefix never applies to the first character). With a suffix, the entry now goes throughmerge_wordlike any other, which proves it is not its own encoding.Test:
end_of_word_suffix_applies_to_single_char_wordsintk-encode/src/models/bpe/tests.rs(fails on main with[0]vs[2]).cargo test -p tk-encode,cargo fmt --checkandcargo clippy -p tk-encode --all-targets --all-features -D warningspass.