ACL 2026 ยท arXiv 2512.12641
Which Pieces Does Unigram Tokenization Really Need?
Read the paper on arXiv ↗Unigram is the elegant, probabilistic alternative to Byte-Pair Encoding. Almost everyone uses BPE anyway, because Unigram's only real implementation is buried inside SentencePiece.
The problem
Unigram's theory is clean; its practice is not. The working implementation lives inside SentencePiece, covering seed initialization, EM, iterative pruning, and finalization, with the details poorly documented and hard to reverse-engineer. That complexity has kept Unigram out of most tokenizer pipelines.
What we do
We document what SentencePiece actually does and which parameters matter, and find that much of it is optional. Multiple EM iterations, the digamma transform, and early pruning thresholds barely change the result. We also introduce Flat Score Pruning, which prunes the lowest-probability tokens directly instead of running the expensive iterative loop that accounts for how tokens interact.
Why it matters
No configuration improves both training loss and compression at once, but Flat Score Pruning trades a little loss for compression that even edges BPE on monolingual data. Unigram also aligns far better with real morpheme boundaries than BPE, 0.62 against 0.22 at a 16K vocabulary, which is a good reason to use it now that it is easy to build.