What counts as a word

The whole lecture answers one question: what unit does a model operate on, and how do we get from raw bytes to that unit? Everything else is a stop on the way. The roadmap has eight stops, and it is worth holding the shape in mind because the lecture overview page mirrors it exactly.

1. Words, with Zipf and Heaps. How to count units, and the two laws that govern how frequency and vocabulary behave as a corpus grows.
2. Morphemes. The meaningful parts inside a word, which explain why walk, walks and walked are one lemma but three types.
3. Corpora. Where the text comes from, and why genre, domain and time shift every count in this part.
4. Unicode. How characters become bytes, so that a tokenizer can promise to represent any script without an unknown symbol.
5. Tokenization with BPE. The algorithm that learns its own units from data instead of trusting spaces.
6. Sentence segmentation. The period that ends a sentence versus the period inside Dr. or U.S.
7. Regular expressions. The pattern language behind pre-tokenizers, cleaners and feature extractors.
8. Minimum edit distance. A distance between strings that scores spelling correction and speech recognizers alike.

The first stop carries Zipf's law and Heaps' law. The second is the Morpheme. The third is the Corpus. The fourth turns a Code point into UTF-8 bytes. The fifth is Tokenization and byte-pair encoding. Then come Sentence segmentation, the Regular expression and, finally, Minimum edit distance. This part covers only the opening of the first stop, but it sets the vocabulary every later part reuses.

The seven learning goals on slide 3 map onto the same stops: know why a word is not a universal unit, distinguish types from tokens and explain why vocabulary keeps growing, know what morphemes are, explain Unicode and byte-level processing, describe BPE and tokenizer design choices, use regular expressions, and compute minimum edit distance with dynamic programming. The first two goals are this part.

Count the words in this sentence before reading on: They picnicked by the pool, then lay back on the grass and looked at the stars. Most people say sixteen. Some say eighteen. Both are right, and the reason they can both be right is the subject of this concept.

In many writing systems spaces suggest boundaries, but a space is only a suggestion. The moment you want a number, you have to decide what to do with everything that a space does not settle, and each decision is tied to what the text is for. Splitting text into units is Tokenization, and the resulting units are tokens. Jurafsky and Martin put it plainly: how we define words depends on the task. The table below pairs each question from the slide with a task that answers yes and a task that answers no.

Speech: where the units fall apart

Written text at least offers spaces. A transcript of spontaneous speech offers something stranger. Consider the utterance I do uh main- mainly business data processing. There is a broken-off word, main-, which Jurafsky and Martin call a fragment, and a filler, uh, which they call a filled pause. Together these are a Disfluency. Should a tokenizer keep them?

For speech recognition the answer is yes, and for a reason that surprises people: fillers help predict the upcoming word, because they signal that the speaker is restarting the clause or idea, and they are a cue to speaker identification. Clark and Fox Tree went further and argued that uh and um are conventional English words: uh announces an expected minor delay, um a major one. A parser or a translation system, on the other hand, wants the fluent sentence underneath, so it drops them. The Switchboard portion of the Penn Treebank release is explicitly annotated for disfluencies, so this decision is built into a real corpus, not a thought experiment.

Languages with no spaces at all

The space is a convention of some scripts, not of language. Chinese, Japanese and Thai are written without spaces between words, so there is no orthographic word to count. Jurafsky and Martin give a Chinese sentence about Yao Ming reaching the finals and show that its units depend on which standard you adopt. Chinese words average only 1.5 to 1.9 characters, so the character is often the practical unit, and the question "how many words?" simply has no answer until a segmentation standard is chosen.

Once a policy is fixed, two numbers describe any text. Take Hamlet's to be or not to be. Six units are written down in order. Only four different ones appear, because to and be each occur twice. Those two numbers have names, and the whole of corpus statistics is built on them.

A Token is one occurrence of a unit in running text, and the total count of tokens is written N. A Type is a distinct vocabulary item, and the number of types is the size of the Vocabulary, written |V|. So to be or not to be has N = 6 tokens and |V| = 4 types: to, be, or, not. The two are tied together by a bookkeeping identity: every token belongs to exactly one type, so the type counts add up to the token count.

Try it yourself. The counter below applies a tokenization policy to a sentence and recomputes both numbers live. Start with the picnic sentence, switch on punctuation splitting and watch N go from 16 to 18. Then load the Apple sentence and flip lowercase: N stays put while |V| drops, because Apple and apple have become one type. That asymmetry is the whole lesson of the next concept in miniature.

Five corpora, one table

Jurafsky and Martin collect rough counts for several English corpora, and the lecture reproduces their table. Two columns are on the slide. The two on the right are computed here: types per thousand tokens, and its reciprocal, tokens per type. Read those computed columns first, because they tell a story the raw counts hide.

Down the table, N grows by six orders of magnitude while |V| grows by less than three. The ratio of types to tokens falls from about 35 per thousand for Shakespeare to about 0.013 per thousand for the Google corpus. This is the first preview of Heaps' law: vocabulary grows without bound but sublinearly, |V| = k N^β with β between 0 and 1, and Jurafsky and Martin report values from 0.44 to 0.56 or higher, so vocabulary goes up a little faster than the square root of the text length. Part 02 fits that law properly; here it is enough to see the bend.

Now read across rows instead of down. Brown has one million tokens and 38 thousand types. Switchboard has more than twice as many tokens and only 20 thousand types, fewer even than Shakespeare with a third of Switchboard's tokens. Size alone cannot explain that. Switchboard is about 2,400 telephone conversations among 543 speakers on about 70 everyday topics, and casual conversation reuses a small everyday vocabulary. Brown samples 500 texts across 15 written genres, and Shakespeare's plays reach for coinages and archaic forms. This is the point of slide 6: even within one language, genre and domain shift the vocabulary, so |V| depends on the Corpus, not only on N.

Why the vocabulary never stops growing

The falling ratio and the endless growth have the same cause. Function words like the and of are seen early and then only repeat. Content words keep arriving. Slide 8 lists where they come from, and each source is a reason your model will meet a word it has never seen.

• Proper names and new products: every person, place, company and gadget is a type, and new ones appear every day.
• Creative spellings and typos: cooool, gr8 and every misspelling is a fresh type under a wordform policy.
• Code-switching: speakers mix languages within one utterance, so an English corpus carries Spanish, Hindi or Arabic types as well.
• Morphology: walk, walks, walked and walking are four wordform types but one Lemma. Inflection multiplies surface forms, and morphologically rich languages multiply them far more.
• Compounding and multiword expressions: New York and ice cream behave as single lexical units but span a space. Sag and colleagues called multiword expressions a pain in the neck for NLP, and Jurafsky and Martin note that tokenizing them needs a dictionary and is tied up with named entity recognition.

The consequence is that out-of-vocabulary words are unavoidable under any word-level policy. However large the training corpus, tomorrow's text contains a name or a typo that is not in V. That is the problem subword tokenization solves later in this lecture, by building rare words out of pieces that are in the vocabulary.

Every policy in this part has been described in words. Here is one written down as rules and run on a real sentence. The Penn Treebank convention is the standard for parsed corpora of English, and Jurafsky and Martin use this example to show its output.

The four decisions

The worked example silently made four decisions, and slide 9 names them. Each has options, each helps some task and each hurts another. The table is the answer to the exam question "list four preprocessing decisions and one downstream consequence of each".

Case folding is the most tempting shortcut. Manning, Raghavan and Schütze observe that for information retrieval, lowercasing everything often remains the most practical solution, and in the same breath list what it destroys: proper nouns distinguished only by case, such as General Motors, the Fed against fed, and Bush against bush. Their alternative is truecasing, restoring the likely original case with a classifier. Jurafsky and Martin add that for other tasks capitalization is a useful feature and is retained, so some teams keep a cased and an uncased model side by side.

Normalization is the quiet one. A Unicode normalization form decides whether the precomposed letter Ç at U+00C7 and the sequence C plus combining cedilla are the same type. Unicode Standard Annex 15 defines four forms, NFC, NFD, NFKC and NFKD, and the same logic applies to four kinds of quotation mark and three kinds of dash that a keyboard and a word processor scatter through any corpus. Skip it and your vocabulary silently holds several copies of the same word.

Numbers show the trade-off most sharply. Kept raw, every price and every year is a new type and |V| explodes. Mapped to a single <NUM> placeholder, the vocabulary shrinks but the model can no longer tell 12 from 12,000. Modern tokenizers pick a middle path: Jurafsky and Martin describe the GPT-4o tokenizer chunking numbers into groups of up to three digits, so magnitude survives as a sequence of small pieces.

The four ambiguous boundaries

Even after the four decisions are made, some characters refuse to settle. Slide 10 collects them. In each case the same character is a boundary in one context and part of a unit in another, and a tokenizer has to pick a default and accept the errors it causes.

A Clitic is the trickiest of the four because the apostrophe is three-way ambiguous. Jurafsky and Martin point out that it marks the genitive (the book's cover), quotation ('The other class', she said) and the contracted clitic (we'll, I'm). Hyphens run from co-education, where the pieces are meaningless alone, to Hewlett-Packard, where each is a name, and Manning and colleagues show there is no rule that suits both. Abbreviation periods will return as the central problem of Sentence segmentation later in the lecture. Social media adds hashtags, handles and URLs that should be kept whole, since #NLP or a web address split at its punctuation is worthless. A Treebank-style tokenizer will not do this by itself: it pads # and @ with spaces and breaks a URL at its colon, so a social-media pipeline adds hashtag, handle and URL patterns in front of it.