What is a Perplexity (Metric)?

Intuitively, perplexity answers "how surprised was the model by this text?" A model with perplexity of, say, 20 on a dataset behaves, on average, as if it were choosing uniformly among 20 equally likely next tokens at each step; a perplexity of 2 means it was rarely surprised at all. It is computed directly from the model's own predicted probability distribution over the vocabulary at each position, compared against the actual next token. Perplexity is most useful as a relative, not absolute, signal: comparing two checkpoints of the same model on the same held-out dataset during pre-training tells you whether training is still improving the model, and comparing perplexity across domains tells you where a model is weakest (e.g., higher perplexity on legal text than on casual writing signals a knowledge gap). It is less useful for comparing across different tokenizers or vocabularies, since the token boundaries themselves change what "predicting the next token" even means. Critically, low perplexity does not guarantee a model is helpful, safe, or factually correct — it only measures how statistically predictable held-out text was to the model, which is why downstream task benchmarks and human evaluation remain necessary alongside it.