Fishing for Magikarp

Why does SolidGoldMagikarp break language models? It sits in the tokenizer but barely appears in the training data. This paper finds these glitch tokens automatically.

The problem

Tokenizers are built separately from models, so vocabularies fill with tokens the model almost never saw during training. These under-trained glitch tokens cause strange, unwanted behaviour, and there was no reliable way to find them beyond stumbling onto them.

What we do

Under-trained tokens leave a fingerprint in the model's own weights: their embeddings drift toward a shared, telltale direction. We combine these weight-based indicators with tokenizer analysis and a prompting step that confirms each candidate. On open models, the indicators track how often a token appeared in training across about ten orders of magnitude.

Most glitch tokens have oddly specific origins:

Token	Where it came from
SolidGoldMagikarp	A Reddit username that gave the phenomenon its name
BjKPZFq	One document full of base64-encoded strings (StarCoder2)
Ostschwizertütsch	A list of Wordle answers by dialect (StarCoder2)
天天中彩票APP	Chinese lottery-app ad text (GPT-4o)
\uEFC0	An unassigned private-use Unicode character (Mistral)

Why it matters

We scan around two dozen open models plus closed ones such as GPT-4 and Claude. Glitch tokens show up in all of them, from a handful in some to thousands in others, and the method surfaced real tokenizer bugs that vendors later fixed. Detecting them makes models safer and more efficient, and helps close the gap between tokenizer and training.