Evaluating language models with perplexity

Suppose you have two speech recognizers that are identical in every way except their Language model. The most convincing way to decide which LM is better is to run both systems on the same real audio and count how many words each gets wrong. Whichever has the lower word error rate has the better LM, for this task, on this data.

That is Extrinsic evaluation: you embed the model in an application and measure the application. SLP3 calls it "the only way to know if a particular improvement in the language model ... is really going to help the task at hand". Its weakness is cost. Every candidate model needs a full end-to-end run, and if you are trying twenty smoothing settings in an afternoon, twenty ASR decodes is not an option. Intrinsic evaluation trades realism for speed: it scores the model by itself, independent of any application. The standard intrinsic metric, for n-gram models and for neural LLMs alike, is Perplexity, which this part builds up step by step.

What "better" means: the Shannon game

To score a model by itself we need a definition of good that does not mention any application. The Shannon game supplies one. Cover the next word and ask the model to bet on it: "I always order pizza with cheese and ___". A good model spreads its bets over sensible continuations, say 0.1 on mushrooms, 0.1 on pepperoni, 0.01 on anchovies, a sliver on fried rice, and essentially nothing (10^-100 on the slide) on the word "and". Shannon ran the original version of this experiment in 1951 with letters rather than words, asking people to guess the next character of English text to estimate how predictable the language is. Contexts differ in how open they are: after "The 33rd President of the US was" almost all the probability belongs to one name, while after "I saw a" thousands of nouns are reasonable. The last concept of this part turns that difference into a number.

A Unigram model plays this game badly. It ignores the context entirely, so after "cheese and" it bets the same way it bets everywhere, heavily on frequent words like "the". The rule that falls out is simple: a better model is one that assigns a higher probability to the word that actually occurs. A perfect model would give each real next word probability 1. Applied to a whole held-out text, the better model gives the test text a higher probability, and perplexity is just a length-normalized way of reporting that probability.

You build a Bigram model and, by accident, five of your test sentences are also in the training corpus. Every bigram in those sentences now has a nonzero count, so the model assigns them a much higher probability than it would to genuinely new text, and the Perplexity drops. The model has not improved at all. It has simply seen the answers.

SLP3 names this precisely: if a test sentence is part of the training corpus, "we will mistakenly assign it an artificially high probability", a situation called training on the test set, or Data contamination, and it "causes huge inaccuracies in perplexity". The cure is to split the data into three disjoint parts with different jobs: a Training set, a Development set and a Test set.

The devset exists because models have knobs. Interpolation weights, the k of add-k smoothing and the discount in Kneser-Ney are all hyperparameters: they are not learned from counts, so they must be chosen by trying values and keeping the best. If you choose them by checking the test set, the test set has become part of training by another route. SLP3's rule is to do all testing on the devset "until the very end" and to run the test set "once, or a very few number of times".

Two further requirements make the test set meaningful. It should be drawn from the domain you care about: if the model will transcribe chemistry lectures or hotel booking requests, the test text should be chemistry lectures or booking requests, and the devset should look like the test set. And it should be large enough to give the statistical power to show a significant difference between two candidate models. A ten-sentence test set can make any two models look different by luck.

The obvious intrinsic score is the probability the model assigns to the test set. It has one fatal flaw. The same Bigram model might give a 5-token sentence a probability around 10^-5 and a 50-token paragraph around 10^-50. Every extra token multiplies in another factor below 1, so raw probability mostly measures length, and two test sets of different sizes cannot be compared.

Perplexity fixes this by taking the Nth root, which turns the product of N factors into a per-token average, and by inverting it so that the number grows as the model gets more surprised. For a test set W = w_1 ... w_N:

Expanding the joint probability with the Chain rule of probability turns it into a product of next-word probabilities, so perplexity is the geometric mean of the inverse next-word probabilities:

A specific model only changes what goes in the denominator. A Unigram model uses P(w_i); a bigram model, under the Markov assumption, uses P(w_i | w_(i-1)).

Because the exponent is negative, lower perplexity means higher test probability. For a fixed test set N is fixed, and P^(-1/N) is strictly decreasing in P, so minimizing perplexity is exactly the same as maximizing the probability of the test set. The inversion is not arbitrary: it comes from the original definition of perplexity as two raised to a cross-entropy rate, which Part 10 develops.

A worked example with the Berkeley bigrams

Compute it in log space

On a real test set of a million tokens the product underflows any floating-point type, so in practice you sum log probabilities instead, average them, and exponentiate once at the end:

The exponent is the average number of bits the model needs per token, which is the bridge to cross-entropy in Part 10. The base does not matter as long as it matches: Hugging Face defines perplexity as "the exponentiated average negative log-likelihood of a sequence", using natural logs and exp, which gives the same number. The same documentation notes that perplexity is not well defined for masked models such as BERT, because they do not assign a left-to-right probability to the sequence.

SLP3 trained Unigram, Bigram and Trigram models on 38 million words of Wall Street Journal text and measured Perplexity on a 1.5 million word test set. The results were 962, 170 and 109.

More context gives the model more information about the next word, so it is less surprised and its perplexity falls. Notice the diminishing return: the first word of context cuts perplexity by a factor of more than five, the second by about a third. The catch is data. A trigram model needs reliable counts for three-word sequences, and Sparsity grows fast with the order, so the gain holds only when the training set is large enough to estimate the longer n-grams. With too little data a higher order can be worse, which is the subject of Part 06.

Conventions that change the number

A real test corpus is many sentences, not one, so the model is run over the whole stream and the probability runs across sentence boundaries. If you use Sentence boundary tokens, they enter the count. SLP3 includes one token per sentence in N: the end marker </s>, but not the start marker <s>. The reason is that </s> is a genuine prediction (the model must decide where the sentence ends), while the step from </s> to the next <s> happens with probability almost 1, so counting it would add an almost-free token and flatter the score.

The deeper rule is that perplexity is a property of a model, a test set and a vocabulary together. SLP3 states that the perplexity of two language models "is only comparable if they use identical vocabularies". A model with a 10k-word vocabulary maps every rarer word to a single unknown token and then predicts that token easily, so its perplexity is lower on a task that is genuinely easier. In modern subword models the same applies to tokenization: Hugging Face warns that "the tokenization procedure has a direct impact on a model's perplexity". Even the evaluation procedure matters. GPT-2 large on WikiText-2 scores 19.44 with non-overlapping 1024-token windows and 16.44 with a sliding window of stride 512, because each token gets more context.

Perplexity is not the task

Lower perplexity often correlates with better downstream performance, which is why everyone uses it, but SLP3 is explicit that an intrinsic improvement "does not guarantee an (extrinsic) improvement". When the decision matters, confirm with the task metric: Extrinsic evaluation with word error rate for speech, BLEU or COMET for translation.

Take a toy language with three words, red, blue and green, where any word can follow any word. At every position there are three possible next words, so its Weighted branching factor is 3. Model A knows nothing and gives each word probability 1/3. On any test set of five words, PP_A = ((1/3)^5)^(-1/5) = 3. A uniform model's perplexity is exactly the number of choices.

Now Model B has learned that red is common: P(red) = 0.8, P(blue) = P(green) = 0.1. On the textbook Test set "red red red red blue", the probability is 0.8^4 × 0.1 = 0.04096 and PP_B = 0.04096^(-1/5) ≈ 1.89. Three words are still possible at every step, but the model is effectively choosing among fewer than two.

This is the intuition SLP3 offers: perplexity is the weighted average branching factor. The plain branching factor counts the possible next words. Perplexity weights them by how much probability the model puts on the words that actually occur, and reports the size of a uniform choice that would leave you equally uncertain. So the WSJ Trigram's 109 means that, on average, the model was as unsure as if it were picking uniformly from about 109 words at each step, out of a vocabulary of tens of thousands.