How do n-gram language models work?

A language model assigns probabilities to the words that could come next. We can build one by counting short sequences in a collection of text. We'll use those counts to make a prediction, then improve the estimate when the text contains a phrase we haven't seen.

Count short sequences

Let's predict the last word in this sentence:

we sat on the bank of the ___

An n-gram is a sequence of $n$ consecutive words. A bigram contains two words, such as "the river"; a trigram contains three, such as "of the river". An n-gram language model predicts a word using the preceding $n-1$ words as its context [1].

For our prediction, a bigram model uses "the" as context. A trigram model uses "of the". Keeping a shorter context lets us count matching sequences across many sentences, rather than requiring a match for "we sat on the bank of the" in full.

Bigram

bankoftheriver

Trigram

bankoftheriver

We'll start with the bigram model. Our training text is the collection we use to estimate its probabilities. If "the" appears ten times as a context, and four of those occurrences are followed by "river", we estimate:

$$\widehat{P}(\text{river}\mid\text{the}) = \frac{4}{10} = 0.4$$

We apply the same calculation to every word in the vocabulary, the set of words the model can predict. Here are the counts after "the":

Next word	Count	Estimated probability
river	4	0.4
bank	3	0.3
road	3	0.3
lake	0	0
All remaining words	0	0

The counts sum to ten. Dividing each by ten gives a probability distribution that sums to one. In general, for a context $c$ and a next word $w$:

$$\widehat{P}(w\mid c) = \frac{\operatorname{count}(c,w)}{\operatorname{count}(c)}$$

This is the maximum likelihood estimate: it chooses the probabilities that make the observed training text most likely. We can build a trigram model in the same way, counting "of the river" and dividing by the count of "of the".

Give unseen phrases a probability

Our table assigns zero probability to "lake" after "the". Even if "lake" appears elsewhere in the training text, the bigram "the lake" has no examples here. The model therefore rules out a reasonable continuation.

This also affects longer sequences. We calculate a sequence's probability by multiplying its next-word probabilities. If we change the last word of our sentence to "lake", the final factor is zero, so the probability of the entire sequence becomes zero.

Collecting more text reduces the number of missing counts, but new combinations of familiar words keep appearing. Smoothing adjusts the estimates so that unobserved combinations can still receive probability [2].

The simplest method is add-one smoothing. We add one to every possible next-word count, including the zeros. If the vocabulary contains $V$ words, the total count increases by $V$, so we adjust the denominator too:

$$P_{\text{add-one}}(w\mid c) = \frac{\operatorname{count}(c,w)+1}{\operatorname{count}(c)+V}$$

With ten words in our vocabulary, the total after "the" becomes $10+10=20$. The count for "lake" becomes one, giving it probability $1/20=0.05$. The count for "river" becomes five, but its probability falls from $4/10=0.4$ to $5/20=0.25$ because the denominator has grown.

Next word	Before	After
river	0.4	0.25
bank	0.3	0.2
road	0.3	0.2
lake	0	0.05
Other 6 words	0	0.3

We have made room for unseen pairs by reducing the probability assigned to observed ones. Add-one smoothing shows how this works, but it can shift too much probability away from the data. With a large vocabulary, adding one for every word greatly increases the denominator. It also gives every unseen continuation the same probability, even when some words are much more common than others.

Use shorter contexts as well

We already have information that can help with an unseen pair: how often its next word appears elsewhere. A unigram model estimates a word's probability from its overall frequency, without any preceding context. If "lake" is common in the collection, that should affect our estimate even when "the lake" is missing.

One way to use this information is backoff. We use a shorter context when the longer sequence lacks enough evidence. For a missing trigram "of the lake", we can consult the bigram "the lake", then the unigram "lake". A backoff model reserves some probability for these shorter-context estimates by reducing, or discounting, the estimates for observed sequences. Simply replacing zeros with shorter-model probabilities would make the total exceed one.

$$\begin{gathered} P(\text{river}\mid\text{bank of the}) \\ \downarrow \\ P(\text{river}\mid\text{of the}) \\ \downarrow \\ P(\text{river}\mid\text{the}) \\ \downarrow \\ P(\text{river}) \end{gathered}$$

Another approach is interpolation: we blend the estimates rather than choosing only one. For a bigram model and a unigram model:

$${P_{\text{mix}}(w\mid c)} = {\lambda\widehat{P}(w\mid c)} + {(1-\lambda)\widehat{P}(w)}$$

Here $c$ is the preceding word, and $\lambda$ controls how much weight we give the bigram estimate. Both component distributions sum to one, and their weights sum to one, so the mixture remains a valid probability distribution.

Suppose "lake" accounts for 2% of the words in the training text. Its unigram estimate is $0.02$, while our bigram estimate after "the" is still zero. Giving the bigram model 80% of the weight and the unigram model 20% gives:

$$\begin{aligned} P_{\text{mix}}(\text{lake}\mid\text{the}) &= 0.8\times0 + 0.2\times0.02 \\ &= 0.004 \end{aligned}$$

Unlike add-one smoothing, this mixture gives more probability to unseen pairs whose next words are common. We can also blend trigram, bigram, and unigram estimates. If a context never appears at all, its count ratio is undefined; we use the shorter model directly instead.

The best balance depends on the data. We choose interpolation weights using a validation set, a separate collection of text that was not used to count the n-grams. This lets us check whether giving more weight to a longer context improves predictions beyond the training examples.

Measure predictions on new text

To compare the finished models, we need a test set that was used for neither counting nor choosing the weights. We ask each model to predict the words in that text and record the probability it assigns to each word that actually occurs.

Perplexity combines these probabilities into one score. For $T$ test words, with $c_i$ the context available when predicting word $w_i$:

$$\operatorname{PP} = \exp\left(-\frac{1}{T}\sum_{i=1}^{T}\ln P(w_i\mid c_i)\right)$$

If a model assigns probability $1/4$ to every observed test word, its perplexity is four. Assigning $1/8$ to each gives perplexity eight. When the probabilities vary, the formula averages their log probabilities before converting the result back to this scale.

$$\begin{array}{cc} P(w_i\mid c_i)=\frac{1}{4} & \qquad P(w_i\mid c_i)=\frac{1}{8} \\[0.8em] \operatorname{PP}={\color{#c9342d}4} & \qquad \operatorname{PP}=8 \end{array}$$

Lower perplexity means the model assigned higher probability to the observed text overall. The comparison needs the same test text, vocabulary, and tokenization; changing how we divide text into words or tokens changes what is being measured. A zero probability makes perplexity infinite, which is another reason to smooth the estimates.

We can now compare context lengths and smoothing methods by how well they predict new text. These improvements still operate on separate word counts: occurrences of "the river" do not directly contribute to the count of "the stream", and words outside the context window cannot affect the prediction. Smoothing improves estimates within that representation; it does not teach the model how different words are related.

References

1. Jurafsky, D., & Martin, J. H. (2026). N-gram Language Models. Speech and Language Processing, third edition draft, Chapter 3. web.stanford.edu ↩
2. Chen, S. F., & Goodman, J. (1996). An Empirical Study of Smoothing Techniques for Language Modeling. Proceedings of ACL. doi:10.3115/981863.981904 ↩