Glossary

A single vocabulary item that stands for every out-of-vocabulary word; it changes the vocabulary, so perplexities are comparable only with the same UNK handling, and in Kneser-Ney it is a normal word with count 0 that gets mass only from the uniform floor.

• Slide 97

Subtracting a fixed discount d (about 0.75) from each non-zero count and interpolating with a lower-order distribution.

• Slide 79
• Slide 80

Adding a fractional count k (such as 0.1 or 0.01, tuned on dev data) to every n-gram instead of one.

• Slide 67

Using the highest-order n-gram if it was seen, otherwise falling back to a lower order.

• Slide 72
• Slide 73

The normalizing weight d / C(h) times the number of distinct continuations of h, which redistributes exactly the discounted mass.

• Slide 86
• Slide 90
• Slide 94

Treating text as an unordered collection of words; a unigram model is one, since it ignores context and word order.

• Slide 16

A small dialog corpus of 9332 restaurant queries with a vocabulary of about 1446 words, used to illustrate bigram tables.

• Slide 22
• Slide 23
• Slide 24
• Slide 25

A two-word sequence; a bigram model conditions each word on the one preceding word, P(w_i | w_{i-1}).

• Slide 10
• Slide 13
• Slide 15
• Slide 16
• Slide 17

Decomposes a sequence probability into a product of next-word probabilities, each conditioned on all previous words.

• Slide 12
• Slide 35

Bigrams with training count c (2 to 9) occur about c minus 0.75 times on average in an equal-size held-out set.

• Slide 77
• Slide 78

Linear interpolation whose lambda weights depend on the context, trusting higher-order n-grams more when that context has strong counts.

• Slide 70

The number of distinct words that precede w in the training data, |{v : C(vw) > 0}|.

• Slide 83

How likely w is to appear as a novel continuation: its continuation count divided by the number of bigram types.

• Slide 82
• Slide 84
• Slide 89

The number of distinct n-grams seen exactly once (n1) or exactly twice (n2); Ney et al. use them to estimate the discount d = n1 / (n1 + 2 n2).

• Slide 80

The entropy of the true distribution measured under a model m; it upper-bounds the true entropy, and perplexity equals 2 to its value.

• Slide 103

Test sentences leaking into training, making probabilities look artificially high and perplexity misleadingly low.

• Slide 32

Held-out data used to tune hyperparameters such as interpolation weights or k in add-k.

• Slide 31
• Slide 32
• Slide 67
• Slide 69
• Slide 71

Reducing the counts of observed n-grams to free probability mass for unseen ones.

• Slide 59
• Slide 76

A measure of uncertainty, H(X) = -sum p(x) log2 p(x), in bits; lower entropy means outcomes are easier to predict.

• Slide 102

Evaluating an LM by its effect inside an application such as speech recognition or machine translation.

• Slide 29
• Slide 39

An LM encodes the statistics of its training corpus, so a model trained on one genre (Shakespeare) fits another (newswire) poorly.

• Slide 56
• Slide 57

Always emitting the most probable next word instead of sampling from the distribution; deterministic and prone to bland, repetitive text.

Data kept out of training (a dev corpus, not the test set) used to set hyperparameters such as interpolation weights and discounts by maximizing its likelihood, equivalently minimizing its perplexity.

• Slide 69
• Slide 71
• Slide 76
• Slide 77
• Slide 100

A setting such as an interpolation weight or smoothing constant chosen on held-out data, not estimated from training counts.

• Slide 31
• Slide 71

Evaluating LM quality directly, independent of any application, typically with perplexity.

• Slide 29
• Slide 39
• Slide 104

The count used in recursive Kneser-Ney: raw count at the highest order and continuation count at lower orders.

• Slide 95
• Slide 96

Absolute discounting whose lower-order distribution is the continuation probability rather than raw word frequency.

• Slide 85
• Slide 93
• Slide 100

A model that assigns a probability distribution over the next word or token, equivalently a probability to a whole word sequence.

• Slide 3
• Slide 8
• Slide 104

Adding one to every count and V to the denominator; simple but moves too much mass to unseen events.

• Slide 62
• Slide 63
• Slide 64
• Slide 65
• Slide 66

Mixing unigram, bigram and trigram estimates with weights lambda that sum to 1, learned on held-out data.

• Slide 68
• Slide 69
• Slide 70

Storing and adding logarithms of probabilities instead of multiplying them, to avoid numerical underflow.

• Slide 25
• Slide 27

Approximates the full history by only the last N-1 words when predicting the next word.

• Slide 13
• Slide 14
• Slide 104

Estimating n-gram probabilities as relative frequencies: the n-gram count divided by the count of its prefix.

• Slide 17
• Slide 18
• Slide 21
• Slide 104

Kneser-Ney with three discounts D1, D2 and D3+ for counts of 1, 2 and 3 or more, estimated per order from counts of counts; the standard n-gram baseline in KenLM and SRILM.

• Slide 98
• Slide 99
• Slide 104

A contiguous sequence of n words or tokens, and by extension the probabilistic model built on their counts.

• Slide 10

An LM that represents words or tokens in continuous space and generalizes via similarity instead of exact context matches.

• Slide 9
• Slide 101

A product of many probabilities (each at most 1) becoming too small for floating point and rounding to zero; avoided by adding log probabilities.

• Slide 27

Higher-order n-grams on small corpora copying training fragments because few continuations follow a long prefix.

• Slide 42
• Slide 54
• Slide 104

The inverse probability of a test set normalized by the number of tokens, PP(W) = P(W)^{-1/N}; lower is better.

• Slide 29
• Slide 33
• Slide 34
• Slide 35
• Slide 36
• Slide 37
• Slide 38
• Slide 104

An imaginary count added to every type or possible continuation before normalizing: one in add-one smoothing, k in add-k.

• Slide 62
• Slide 63
• Slide 67

The count C* implied by a smoothed model, its smoothed probability times the prefix count, compared with the raw count to see how much smoothing changed it.

• Slide 65
• Slide 66

A count normalized by the total count of its conditioning context so the values sum to 1.

• Slide 17
• Slide 74

Generating text by repeatedly drawing the next word from the model's conditional distribution until </s> is produced.

• Slide 42
• Slide 43
• Slide 45
• Slide 46
• Slide 104

The symbols <s> and </s> that mark sentence start and end so the model gives a proper distribution over all sentence lengths.

• Slide 15
• Slide 19
• Slide 28
• Slide 37

Guessing the next word in a sentence, used as the intuition that a better model gives higher probability to the word that actually occurs.

• Slide 30

Shaving probability mass from seen events and giving it to unseen ones so no plausible sequence gets probability zero.

• Slide 59
• Slide 60
• Slide 61
• Slide 104

Most possible n-grams never appear in training data, so count tables are mostly zeros, worse as n grows.

• Slide 14
• Slide 21
• Slide 22
• Slide 28
• Slide 54

A web-scale backoff score that uses the relative frequency if seen, otherwise 0.4 times the lower-order score; not a true distribution.

• Slide 73
• Slide 74

Representing text as subword or byte tokens (as in BPE), so any word can be spelled and a test set need not contain unseen tokens even when it contains unseen words.

• Slide 58

Unseen data used only for final evaluation, run once or very few times.

• Slide 31
• Slide 32

The data used to estimate model parameters (counts).

• Slide 31

A three-word sequence; a trigram model conditions each word on the two preceding words.

• Slide 10
• Slide 13
• Slide 16
• Slide 28

A single word; a unigram model uses no context and estimates P(w).

• Slide 16
• Slide 43

Interpretation of perplexity as the effective number of equally likely next words given the model's probabilities.

• Slide 40
• Slide 41

An unseen test n-gram gets probability 0, making the whole sequence probability 0 and perplexity undefined.

• Slide 59