tokenize.rs

ACL 2026 ยท arXiv 2512.12641

Which Pieces Does Unigram Tokenization Really Need?

Sander Land, Yuval Pinter

Read the paper on arXiv ↗

Unigram is the elegant, probabilistic alternative to Byte-Pair Encoding. Almost everyone uses BPE anyway, because Unigram's only real implementation is buried inside SentencePiece.

Scatter plot of change in training objective versus change in token count for each hyperparameter variant, showing Flat Score Prune achieving better compression at a small objective cost.
Every setting trades objective for compression, and none wins both. Flat Score Pruning buys the most compression for the least loss.

The problem

Unigram's theory is clean; its practice is not. The working implementation lives inside SentencePiece, covering seed initialization, EM, iterative pruning, and finalization, with the details poorly documented and hard to reverse-engineer. That complexity has kept Unigram out of most tokenizer pipelines.

What we do

We document what SentencePiece actually does and which parameters matter, and find that much of it is optional. Multiple EM iterations, the digamma transform, and early pruning thresholds barely change the result. We also introduce Flat Score Pruning, which prunes the lowest-probability tokens directly instead of running the expensive iterative loop that accounts for how tokens interact.

Why it matters

No configuration improves both training loss and compression at once, but Flat Score Pruning trades a little loss for compression that even edges BPE on monolingual data. Unigram also aligns far better with real morpheme boundaries than BPE, 0.62 against 0.22 at a 16K vocabulary, which is a good reason to use it now that it is easy to build.

Read the paper on arXiv ↗ Browse the code ↗