Words and tokens

The whole lecture answers one question: what unit does a model operate on, and how do we get from raw bytes to that unit? Everything else is a stop on the way. The roadmap has eight stops, and it is worth holding the shape in mind because the lecture overview page mirrors it exactly.

1. Words, with Zipf and Heaps. How to count units, and the two laws that govern how frequency and vocabulary behave as a corpus grows.
2. Morphemes. The meaningful parts inside a word, which explain why walk, walks and walked are one lemma but three types.
3. Corpora. Where the text comes from, and why genre, domain and time shift every count in this part.
4. Unicode. How characters become bytes, so that a tokenizer can promise to represent any script without an unknown symbol.
5. Tokenization with BPE. The algorithm that learns its own units from data instead of trusting spaces.
6. Sentence segmentation. The period that ends a sentence versus the period inside Dr. or U.S.
7. Regular expressions. The pattern language behind pre-tokenizers, cleaners and feature extractors.
8. Minimum edit distance. A distance between strings that scores spelling correction and speech recognizers alike.

The first stop carries Zipf's law and Heaps' law. The second is the Morpheme. The third is the Corpus. The fourth turns a Code point into UTF-8 bytes. The fifth is Tokenization and byte-pair encoding. Then come Sentence segmentation, the Regular expression and, finally, Minimum edit distance. This part covers only the opening of the first stop, but it sets the vocabulary every later part reuses.

The seven learning goals on slide 3 map onto the same stops: know why a word is not a universal unit, distinguish types from tokens and explain why vocabulary keeps growing, know what morphemes are, explain Unicode and byte-level processing, describe BPE and tokenizer design choices, use regular expressions, and compute minimum edit distance with dynamic programming. The first two goals are this part.

Count the words in this sentence before reading on: They picnicked by the pool, then lay back on the grass and looked at the stars. Most people say sixteen. Some say eighteen. Both are right, and the reason they can both be right is the subject of this concept.

In many writing systems spaces suggest boundaries, but a space is only a suggestion. The moment you want a number, you have to decide what to do with everything that a space does not settle, and each decision is tied to what the text is for. Splitting text into units is Tokenization, and the resulting units are tokens. Jurafsky and Martin put it plainly: how we define words depends on the task. The table below pairs each question from the slide with a task that answers yes and a task that answers no.

Speech: where the units fall apart

Written text at least offers spaces. A transcript of spontaneous speech offers something stranger. Consider the utterance I do uh main- mainly business data processing. There is a broken-off word, main-, which Jurafsky and Martin call a fragment, and a filler, uh, which they call a filled pause. Together these are a Disfluency. Should a tokenizer keep them?

For speech recognition the answer is yes, and for a reason that surprises people: fillers help predict the upcoming word, because they signal that the speaker is restarting the clause or idea, and they are a cue to speaker identification. Clark and Fox Tree went further and argued that uh and um are conventional English words: uh announces an expected minor delay, um a major one. A parser or a translation system, on the other hand, wants the fluent sentence underneath, so it drops them. The Switchboard portion of the Penn Treebank release is explicitly annotated for disfluencies, so this decision is built into a real corpus, not a thought experiment.

Languages with no spaces at all

The space is a convention of some scripts, not of language. Chinese, Japanese and Thai are written without spaces between words, so there is no orthographic word to count. Jurafsky and Martin give a Chinese sentence about Yao Ming reaching the finals and show that its units depend on which standard you adopt. Chinese words average only 1.5 to 1.9 characters, so the character is often the practical unit, and the question "how many words?" simply has no answer until a segmentation standard is chosen.

Once a policy is fixed, two numbers describe any text. Take Hamlet's to be or not to be. Six units are written down in order. Only four different ones appear, because to and be each occur twice. Those two numbers have names, and the whole of corpus statistics is built on them.

A Token is one occurrence of a unit in running text, and the total count of tokens is written N. A Type is a distinct vocabulary item, and the number of types is the size of the Vocabulary, written |V|. So to be or not to be has N = 6 tokens and |V| = 4 types: to, be, or, not. The two are tied together by a bookkeeping identity: every token belongs to exactly one type, so the type counts add up to the token count.

Try it yourself. The counter below applies a tokenization policy to a sentence and recomputes both numbers live. Start with the picnic sentence, switch on punctuation splitting and watch N go from 16 to 18. Then load the Apple sentence and flip lowercase: N stays put while |V| drops, because Apple and apple have become one type. That asymmetry is the whole lesson of the next concept in miniature.

Five corpora, one table

Jurafsky and Martin collect rough counts for several English corpora, and the lecture reproduces their table. Two columns are on the slide. The two on the right are computed here: types per thousand tokens, and its reciprocal, tokens per type. Read those computed columns first, because they tell a story the raw counts hide.

Down the table, N grows by six orders of magnitude while |V| grows by less than three. The ratio of types to tokens falls from about 35 per thousand for Shakespeare to about 0.013 per thousand for the Google corpus. This is the first preview of Heaps' law: vocabulary grows without bound but sublinearly, |V| = k N^β with β between 0 and 1, and Jurafsky and Martin report values from 0.44 to 0.56 or higher, so vocabulary goes up a little faster than the square root of the text length. Part 02 fits that law properly; here it is enough to see the bend.

Now read across rows instead of down. Brown has one million tokens and 38 thousand types. Switchboard has more than twice as many tokens and only 20 thousand types, fewer even than Shakespeare with a third of Switchboard's tokens. Size alone cannot explain that. Switchboard is about 2,400 telephone conversations among 543 speakers on about 70 everyday topics, and casual conversation reuses a small everyday vocabulary. Brown samples 500 texts across 15 written genres, and Shakespeare's plays reach for coinages and archaic forms. This is the point of slide 6: even within one language, genre and domain shift the vocabulary, so |V| depends on the Corpus, not only on N.

Why the vocabulary never stops growing

The falling ratio and the endless growth have the same cause. Function words like the and of are seen early and then only repeat. Content words keep arriving. Slide 8 lists where they come from, and each source is a reason your model will meet a word it has never seen.

• Proper names and new products: every person, place, company and gadget is a type, and new ones appear every day.
• Creative spellings and typos: cooool, gr8 and every misspelling is a fresh type under a wordform policy.
• Code-switching: speakers mix languages within one utterance, so an English corpus carries Spanish, Hindi or Arabic types as well.
• Morphology: walk, walks, walked and walking are four wordform types but one Lemma. Inflection multiplies surface forms, and morphologically rich languages multiply them far more.
• Compounding and multiword expressions: New York and ice cream behave as single lexical units but span a space. Sag and colleagues called multiword expressions a pain in the neck for NLP, and Jurafsky and Martin note that tokenizing them needs a dictionary and is tied up with named entity recognition.

The consequence is that out-of-vocabulary words are unavoidable under any word-level policy. However large the training corpus, tomorrow's text contains a name or a typo that is not in V. That is the problem subword tokenization solves later in this lecture, by building rare words out of pieces that are in the vocabulary.

Every policy in this part has been described in words. Here is one written down as rules and run on a real sentence. The Penn Treebank convention is the standard for parsed corpora of English, and Jurafsky and Martin use this example to show its output.

The four decisions

The worked example silently made four decisions, and slide 9 names them. Each has options, each helps some task and each hurts another. The table is the answer to the exam question "list four preprocessing decisions and one downstream consequence of each".

Case folding is the most tempting shortcut. Manning, Raghavan and Schütze observe that for information retrieval, lowercasing everything often remains the most practical solution, and in the same breath list what it destroys: proper nouns distinguished only by case, such as General Motors, the Fed against fed, and Bush against bush. Their alternative is truecasing, restoring the likely original case with a classifier. Jurafsky and Martin add that for other tasks capitalization is a useful feature and is retained, so some teams keep a cased and an uncased model side by side.

Normalization is the quiet one. A Unicode normalization form decides whether the precomposed letter Ç at U+00C7 and the sequence C plus combining cedilla are the same type. Unicode Standard Annex 15 defines four forms, NFC, NFD, NFKC and NFKD, and the same logic applies to four kinds of quotation mark and three kinds of dash that a keyboard and a word processor scatter through any corpus. Skip it and your vocabulary silently holds several copies of the same word.

Numbers show the trade-off most sharply. Kept raw, every price and every year is a new type and |V| explodes. Mapped to a single <NUM> placeholder, the vocabulary shrinks but the model can no longer tell 12 from 12,000. Modern tokenizers pick a middle path: Jurafsky and Martin describe the GPT-4o tokenizer chunking numbers into groups of up to three digits, so magnitude survives as a sequence of small pieces.

The four ambiguous boundaries

Even after the four decisions are made, some characters refuse to settle. Slide 10 collects them. In each case the same character is a boundary in one context and part of a unit in another, and a tokenizer has to pick a default and accept the errors it causes.

A Clitic is the trickiest of the four because the apostrophe is three-way ambiguous. Jurafsky and Martin point out that it marks the genitive (the book's cover), quotation ('The other class', she said) and the contracted clitic (we'll, I'm). Hyphens run from co-education, where the pieces are meaningless alone, to Hewlett-Packard, where each is a name, and Manning and colleagues show there is no rule that suits both. Abbreviation periods will return as the central problem of Sentence segmentation later in the lecture. Social media adds hashtags, handles and URLs that should be kept whole, since #NLP or a web address split at its punctuation is worthless. A Treebank-style tokenizer will not do this by itself: it pads # and @ with spaces and breaks a URL at its colon, so a social-media pipeline adds hashtag, handle and URL patterns in front of it.

Take a novel, count every word, and sort the counts from largest to smallest. Manning and Schütze did exactly this for Mark Twain's Tom Sawyer: 71,370 tokens and 8,018 distinct types. The most frequent type is "the" with 3,332 occurrences, then "and" with 2,972, then "a" with 1,775. At the other end sit thousands of words like "applausive" that appear exactly once. The hundred most common types alone cover 50.9% of every token in the book.

Give each type a rank R, with R = 1 for the most frequent word. Now multiply frequency by rank. For most of the list the product hovers around a constant, roughly 8,000 to 10,000, a relationship Estoup noticed in 1916 and George Kingsley Zipf made famous in the 1930s and 1940s, so it carries Zipf's name. Zipf's law says that frequency falls off as a power of rank.

Because the law is a proportionality, the useful working form anchors it to the top word. If you know how often the most frequent word occurs, everything else follows from the rank alone.

With α = 1 the arithmetic is nothing more than division. The word at rank 10 occurs about a tenth as often as the top word, the word at rank 1,000 about a thousandth as often. Slide 12 will show that natural language has α slightly below one, so the fall is a little gentler and the rare words a little less rare than the pure 1 / R rule predicts.

Read the table across a row and you see the fall; read it down a column and you see what α does. A smaller exponent leaves more mass in the tail, which for a language model means more probability spread across rare words. A larger exponent concentrates the text in fewer types. The exponent is a property of the corpus, so it differs slightly between languages, genres and preprocessing choices such as case folding.

Slide 12 plots rank against frequency for three corpora, English Wikipedia, Arabic Wikipedia and an undotted Arabic Wikipedia in which the dots that distinguish letters such as ب, ت and ث have been removed. Both axes are logarithmic, labelled from 10^0 to 10^6 in rank (the Arabic curve runs on to about 2 × 10^6) and to 10^7 in frequency, and two dashed straight lines, each pinned to the top word of its corpus, are drawn with α = 0.9252 for Arabic and α = 0.9290 for English. The first thing to understand is why anyone would expect a straight line at all.

Write Zipf's law as an equation with a constant, f(R) = C · R^−α, and take the logarithm of both sides. A product becomes a sum and a power becomes a multiplier, which is the whole trick.

On axes that show log R and log f, this is the equation of a straight line with slope −α and intercept log C, which is log f(1) since log 1 = 0. The Stanford IR book states the special case directly: log cf_i = log c + k log i with k = −1. So the two fitted slopes on the slide, −0.9252 and −0.9290, are the exponents of the two Zipf fits, and the fact that they are nearly parallel says Arabic and English share the same kind of head-and-tail shape.

Two places where the real curve leaves the line

Look at the slide again and the solid curves do not follow the dashed lines everywhere. They deviate in two characteristic ways, and both are diagnostic rather than noise. At the head, the curve is flatter than the power law: the top few words are common, but the drop from rank 1 to rank 10 is smaller than R^−α demands. This is the same shortfall the Tom Sawyer table showed: f × R was 3,332 for "the" against 8,000 to 10,000 further down, and it is why Mandelbrot added the shift term (R + ρ).

How that flat head looks on a plot depends on where the line is anchored. On slide 12 both fits pass through f(1). The blue English curve bulges slightly above the purple line only for ranks 2 to about 15, then runs just under it. The Arabic curves behave differently: they dip below the red line for ranks 3 to about 50 and cross back above it. At Wikipedia scale the head deviation is small, and the head is not uniformly flatter. The flat head is clearest in small texts such as Tom Sawyer and the explorer's sample; on slide 12 the dominant departure is the tail.

At the tail, beyond about rank 10^4, the curves steepen sharply below both lines and then turn into a staircase with horizontal steps at frequencies 3, 2 and 1, stretching out to about rank 2 × 10^6 for Arabic. The reason is that a frequency is a count, and counts are integers. There is no word that occurs 1.5 times. If 400,000 types each occur exactly once, they occupy 400,000 consecutive ranks at height f = 1, which draws as one long flat step. The step at f = 2 is shorter because fewer types occur exactly twice, and so on upward until the counts are large enough that adjacent ranks rarely share a value and the curve looks smooth. The bottom step is the longest because types seen once are the largest single group, the hapax legomena that concept four returns to.

One more thing the slide shows without saying it. In the tail, the Arabic curves sit above the English one: at any low count, Arabic has more types occurring at least that often. Templatic morphology and attached clitics mean that a single Arabic lexeme surfaces as many distinct spelled forms, each of which is a separate type for a whitespace tokenizer. That is a preview of the next concept, where the same fact shows up as a larger k in Heaps' law.

Zipf describes a Corpus frozen at one size. Now let the corpus grow. Read the first thousand words of Pride and Prejudice and you meet 361 distinct types. After ten thousand words the count is 1,698; after fifty thousand, 4,304; at the end of the novel, 123,627 tokens have produced 6,298 types. Each new stretch of text adds fewer new words than the last, but it never adds none. The same pattern holds across corpora of every size.

The rule behind these numbers is Heaps' law, also called Herdan's law after the linguist who described it in 1960, nearly two decades before Heaps (1978). The Vocabulary size is a power of the token count, with an exponent strictly between zero and one.

The two halves of the constraint on β say two different things. Because β < 1, growth is sublinear: doubling the corpus does not double the vocabulary. Because β > 0, growth is unbounded: the derivative d|V| / dN = k β N^(β − 1) is positive for every N, so the curve keeps rising however far you extend it. SLP3 phrases the second half bluntly: no matter how big our vocabulary, we will never have a vocabulary that captures all the possible words that might occur.

The slide's three fits, and what the constants mean

Slide 14 fits the law to the same three corpora as before and reports k and β for each. The constants are worth reading as a pair. k is a multiplicative scale (the predicted |V| at N = 1) that shifts the whole curve up, and β sets the slope on log-log axes: the ratio between two corpus sizes, (N2 / N1)^β, depends on β alone. Plugging in three corpus sizes makes the comparison concrete.

Now compare languages. At a million tokens Arabic Wikipedia reaches 115,672 types, about two and a half times the English figure, and undotted Arabic reaches 98,312. Arabic gets there because of its k, 73 against 15: templatic morphology and attached clitics such as wa-, bi- and -hu multiply the surface forms of every lexeme, so the vocabulary is large from the very first pages. Its β is lower, though, so the curve bends sooner: a hundredfold larger corpus multiplies the English vocabulary by 100^0.583 ≈ 14.7 but the Arabic vocabulary by only 100^0.533 ≈ 11.6. Removing the dots collapses distinct letters into one shape, and the fit agrees that this shrinks the vocabulary: undotted Arabic is below dotted Arabic at every size in the table. Its larger k (90 against 73) is offset by its smaller β, and the two curves cross below about 2,400 tokens, so k alone never tells you which vocabulary is bigger; evaluate k N^β at the N you care about. Parts 03 and 04 explain the morphology.

Typical values, and how the slide compares

The constants reported across five reference points are easiest to read side by side.

SLP3 summarizes the law as vocabulary growing a little faster than the square root of the corpus length in words, with β depending on corpus size and genre. The Arabic exponents, 0.533 and 0.506, fall inside SLP3's 0.44 to 0.56 band; the English 0.583 is just above it, which SLP3's "or even higher" allows. The English k of 14.9 is on the low side, which is what you get when a fit trades a smaller k for a larger β; the two constants are not independent estimates.

Put the two laws together and a design problem falls out. In Tom Sawyer, 3,993 of the 8,018 types occur exactly once, which is 49.8%. In the three chapters of Pride and Prejudice inside the explorer, 475 of 845 types occur once, or 56%. A type seen exactly once is a Hapax legomenon, from the Greek for "read only once", and in corpus after corpus these are about half of everything a model must store.

That one half is not a coincidence; it falls out of Zipf's law. With α = 1, the type at rank R has count C / R, so the number of types with count at least m is the rank at which the count drops to m, namely C / m. The types with count exactly m are then C / m − C / (m + 1) = C / (m (m + 1)), and dividing by the total C gives a share of 1 / (m (m + 1)): one half seen once, one sixth seen twice, one twelfth seen three times.

Baayen (2001) develops this connection between the frequency spectrum and vocabulary growth in full, and it is why the bottom step of the staircase on slide 12 is the longest.

Slides 15 and 16 list the consequences, and they form a chain rather than a list. Start with size. Heaps' law says a word-level Vocabulary on a large corpus runs to hundreds of thousands or millions of types, and every one of them needs a row in an embedding matrix and a column in the output softmax. Memory grows with |V| × d, where d is the embedding width. Then comes sparsity, which is Zipf's contribution: since half of those types belong to hapax legomena, half of the parameters in that matrix have been trained on a single example. Whatever the model learned about "applausive" it learned from one sentence.

Next comes the guarantee. Because Heaps' law has a positive derivative at every N, any finite training corpus stops somewhere on a curve that is still rising. New text therefore always contains types that were never seen, and those types are out-of-vocabulary. This is not a defect of a small corpus that a bigger one would cure; a bigger corpus only pushes the OOV rate down along the same curve, never to zero. Finally, evaluation inherits the problem. Earlier SLP3 drafts warned that mapping unseen words to an <UNK> special token lets a model buy a lower perplexity simply by choosing a smaller vocabulary, since more of the test set collapses onto one easy symbol. The current chapter 3 sidesteps the issue by making the vocabulary subword tokens, so that, in its words, the test set can never contain unseen tokens.

• Huge word-level vocabularies: memory in the embedding and output layers scales with |V|.
• Sparsity: about half of all types are hapax legomena, so about half of the rows have one training example.
• Guaranteed OOV: Heaps' law keeps rising, so new text always brings unseen types.
• Evaluation sensitivity: mapping unseen words to <UNK> changes what a perplexity number means.

Two answers to the same problem

The two families of language model respond to the chain at different points. N-gram models keep the word as the unit and repair the probability estimate: smoothing moves a little probability mass from seen events to unseen ones, and backoff falls back to a shorter context when the longer one was never observed. Lecture 3 and SLP3 chapter 3 develop both. Neural models change the unit instead. With Subword tokenization, learned by byte-pair encoding as Sennrich, Haddow and Birch showed for machine translation in 2016, a word never seen in training is spelled out of pieces that were, and with byte-level fallback there is nothing left that cannot be represented. The vocabulary becomes a design choice of tens of thousands of units rather than a fact about the corpus, which is exactly the trade part 6 examines.

The word cats has four letters but only two units of meaning: cat, the animal, and -s, the signal that there is more than one. The word unhappiness has three: un- for negation, happy for the core idea, and -ness to turn the adjective into a noun. Each of those units is a Morpheme: the smallest unit that carries meaning or a grammatical function. Jurafsky and Martin put it in one line: a morpheme is a minimal meaning-bearing unit in a language, so fox is one morpheme while cats is two (SLP3, section 2.2).

Try the segmentation before the theory. In SLP3 the sentence Doc worked carefully washing the glasses is cut as Doc work-ed care-ful-ly wash-ing the glass-es. Every hyphen marks a boundary where one meaning stops and another begins. Doc and the are single morphemes; carefully is three. Notice what the cuts are not: they are not syllables, and they are not places where the spelling changes. They are places where meaning changes.

Free and bound

The first sorting of morphemes is by independence. A free morpheme can occur by itself as a word: cat, walk, happy. A bound morpheme never occurs alone and is always attached to some other morpheme, the way -s in dogs is (SIL Glossary of Linguistic Terms). Bound morphemes are the affixes: every un-, -ed, -ing and -ness you will ever segment. The test is simple. Could you say it on its own and be understood? happy, yes. ness, no.

Roots, stems and affixes

The second sorting is by job. SLP3 divides the morphemes of a word into roots, the central morpheme supplying the main meaning, and affixes, which add meanings of various kinds: in worked, work is the root and -ed the affix; in glasses, glass is the root and -es the affix (SLP3, section 2.2). The root is the core lexical material, and in English it is usually a free morpheme.

The lecture adds a third word that students often blur into root: the Stem. A stem is the inflectable base, the thing the grammatical endings attach to. In a simple word like walk the root and the stem are the same string, which is why one slide writes them as root/stem. They come apart once a derived word is inflected. In unhappinesses the root is happy, the core of the meaning, but the plural -es attaches to the whole stem unhappiness. The root is what the word is about; the stem is what the grammar sees.

Affixes are then named by where they sit. A prefix goes before the stem (un-). A suffix goes after it (-ed, -ing). An infix is inserted inside the stem, and a circumfix wraps around it in two parts at once. Both are rare or absent in English. The standard textbook circumfix is the German past participle, as in ge-spiel-t, played, where ge- and -t together express one meaning (Haspelmath and Sims, chapter 2), and the usual English illustration of infixation is the jokey abso-bloody-lutely. Keep these two as illustrations; the slide only says they exist and are rare. Compounding is different again: it joins two roots without any affix, so snowman is snow plus man and laptop is lap plus top, two free morphemes in one word.

Arabic roots: a skeleton, not a string

Everything above assumed the root is a contiguous piece of the word. Arabic breaks that assumption, and it is the reason this part matters so much for your work. The Arabic root for writing is K-T-B, the three consonants ك ت ب. Those three letters never appear next to each other as a word. Instead they are slots, and a pattern of vowels and sometimes extra consonants is threaded between them. The same root with three patterns gives kataba كَتَبَ he wrote, kitāb كِتَاب book, and kātib كَاتِب writer. Look at the Arabic script, where the radicals are highlighted: ك, ت, ب keep their order in every form, and only the short-vowel marks and the long vowel ا between them change.

This is Templatic morphology, also called root-and-pattern morphology. The root is an abstract consonantal skeleton, and actual words are formed by adding the vowels and non-root consonants that go with a particular morphological category around the root consonants (Wikipedia, Semitic root). Ryding's reference grammar treats the root and pattern as the two fundamental building blocks of an Arabic word: the root supplies the semantic field, the pattern supplies the grammatical and derivational shape (Ryding, chapter 3). You will see the mechanism itself in the third concept of this part; for now the point is that K-T-B is a root in exactly the sense defined above, core lexical material, but it is discontinuous.

Why NLP cares

SLP3 names three candidates for what a Token should be: words, morphemes and characters (SLP3, section 2.4). Earlier it observes that because morphemes are hard to define consistently across languages, it is very difficult to use morphemes as a standard for Tokenization cross-lingually (SLP3, section 2.2). That difficulty is the whole reason subword tokenizers exist, and you will meet them later in this lecture. Arabic root-and-pattern words are the hard case: there is no contiguous substring that means writing shared between kataba, kitāb and kātib, so a tokenizer that only cuts strings cannot see that they are related.

Open a dictionary. walk, walked and walks share one entry; the endings only tell you when the walking happened or who did it. happy and happiness get two entries: an adjective and a noun that names its quality. That single observation is the whole distinction between the two functions an Affix can have. Inflection marks grammar (tense, number, case, agreement) and usually keeps the same Lexeme (the slide's hedge, consistent with SLP3's continuum). Derivation creates a new lexeme, and often changes the part of speech.

SLP3 describes the two ends of the scale. Inflectional morphemes are grammatical morphemes that tend to play a syntactic role, such as marking agreement, the plural -s on nouns or the past tense -ed on verbs; they tend to be productive, often obligatory, and their meanings tend to be predictable. Derivational morphemes are more idiosyncratic in their application and meaning, usually apply only to a specific subclass of words, and result in a word of a different grammatical class than the Root, often with a meaning hard to predict exactly (SLP3, section 2.2). SLP3's own example is care, a noun, plus -ful giving an adjective, plus -ly giving an adverb: two derivations in a row, each moving the word to a new category.

The English slide examples line up with that. walk to walk-ed and walk-s is inflection: every regular verb takes these endings, the meaning is exactly past tense (or past participle) or third person singular, and the word stays a verb. happy to happi-ness is derivation: -ness attaches mainly to adjectives, not every adjective takes it comfortably, and the result is a noun.

Derivation with nothing added

Here is the case that catches people. The noun an email became the verb to email without any affix at all. Nothing was attached, the spelling did not change, and yet a new lexeme with a new part of speech now exists. This is Conversion (zero derivation), also called zero derivation: the creation of a word of a new part of speech from an existing word of a different part of speech without any change in form (Wikipedia, Conversion). It is derivation by the test that matters, a new dictionary entry, even though there is no morpheme to point at.

The same two functions in Arabic

Arabic uses both functions too, and the slide picks an example of each. Inflection: katab-tu كَتَبْتُ I wrote is the Stem katab plus the suffix -tu, which marks first person singular past. The lexeme is still the verb to write; only who and when changed. Wikipedia's overview of Arabic verbs confirms katabtu as I wrote and notes that person, gender, number, tense, mood and voice are specified by changes to the vowels between the consonants along with prefixes or suffixes (Wikipedia, Arabic verbs). Derivation: kitāb كِتَاب book and kātib كَاتِب writer share the root K-T-B but are two lexemes, a noun for the object and an agent noun for the person, made by choosing two different patterns. Do not read a division of labour into that pair. The slide's inflection example happens to use a suffix and its derivation example a pattern, but Arabic uses both mechanisms for both jobs: the broken plural kutub كُتُب books inflects kitāb by changing the pattern alone, and maktab مَكْتَب office derives a place noun with the maCCaC pattern, whose ma- sits in front of the root like a prefix.

Four tests to decide

1. Lexeme: would a dictionary list the two forms under one entry? One entry means inflection.
2. Category: did the part of speech change? A change points to derivation, though derivation can also keep it, as un-happy does.
3. Productivity: does the affix apply to essentially every word of its class with a predictable meaning? Every regular verb takes -ed; not every adjective takes -ness.
4. Position: inflection closes the word. In un-happi-ness-es the derivational pieces sit inside and the plural sits outside. Haspelmath and Sims describe this ordering as a strong cross-linguistic tendency (chapter 5).

Take scissors to walked and you can cut it into walk and ed. Take scissors to sang and there is nowhere to cut: the past tense lives in the vowel, in the middle of the root, and no substring means past. Both words are past-tense Inflection. They differ not in function but in form. That is the second axis of this part, independent of the first: Concatenative morphology attaches segments in a line, and Non-concatenative morphology changes the inside of the word instead.

Concatenative morphology covers everything you have segmented so far: a prefix before, a suffix after, or two roots side by side. cat-s, walk-ed, un-happy and snow-man are all built by placing one contiguous piece next to another. The pieces keep their shape, and you can recover them by cutting. Non-concatenative morphology is defined by what it lacks: word formation and inflection in which the Root is modified and which does not involve stringing morphemes together sequentially (Wikipedia, Nonconcatenative morphology). The slide groups it into internal change, templatic patterns and Suppletion.

Concatenation taken to its limit: Turkish

Turkish shows how far the concatenative strategy can go. The word evlerimden means from my houses, and it is four pieces in a row: ev-ler-im-den. Turkish is a highly agglutinative language, in that much of the grammar is expressed by means of suffixes added to nouns and verbs, and evlerden from the houses is analysed as ev house, -ler plural, -den ablative (Wikipedia, Turkish grammar). Adding the first person possessive -im between them gives the slide's form. Each suffix has a clean boundary and stays recognisable, which is what makes the word so easy to cut (one suffix, -im, still bundles person, number and possession, hence the periods in its gloss).

Three ways to change the inside

Internal change is the English strong-verb pattern. sing, sang, sung (and the noun song) differ only in the vowel of the root. This vowel alternation is called Ablaut, a system of regular vowel variations inherited from Proto-Indo-European (Wikipedia, Indo-European ablaut). The root and the tense are fused in one syllable; there is no -ed to remove.

Templatic patterns are the Arabic case from the first concept, now seen as a mechanism. The root ك ت ب stays fixed in order, and a template of vowels and extra consonants is woven through it: kataba كَتَبَ he wrote, kitāb كِتَاب book, kātib كَاتِب writer. The pieces of the pattern are not contiguous with each other and neither are the pieces of the root, so there is no cut that separates the two morphemes. Watch how the radicals never move while the material between them changes.

Look closely at yaktubu يَكْتُبُ he writes, and you will see both mechanisms at once. The prefix ya- and the suffix -u are concatenative; the Stem shape -ktub-, with no vowel after k and u after t, the pattern CCuC woven through the radicals, is templatic. Arabic words are typically built by wrapping affixes around a root-and-pattern core, which is why an Arabic morphological analyzer has to do two different kinds of work.

Suppletion is the extreme case. The past tense of go is went, and no rule relates the two strings. Suppletion is the use of one word as the inflected form of another word when the two words are not cognates; went was originally the past tense of the verb wend (Wikipedia, Suppletion). Haspelmath and Sims treat it as the extreme case of allomorphy, where the two forms share no phonological material, and the slide lists it under non-concatenative because the outcome is the same: a grammatical change with no segment you can cut off. Suppletion also covers good to better.

Two axes, not one

Function and form are independent. walked and sang are both inflection; one is concatenative and one is not. happiness and kitāb are both Derivation; one is concatenative and one is templatic. When you analyse a form, answer both questions separately: what does the change do, and how is the change realised?

The form axis is the one that decides what a tokenizer can see. Subword tokenization methods such as Byte-pair encoding (BPE) work on contiguous substrings, so they approximate concatenative morphology well: walk can be shared between walked and walking because it is a substring of both. A discontinuous root like K-T-B is never a substring of anything, so the same tokenizer cannot share the writing meaning across kataba, kitāb and kātib. That is SLP3's cross-lingual difficulty made concrete, and it is the hook for most Arabic tokenization research.

Start with an English sentence that Zwicky and Pullum (1983) use in section 2 of their paper: The person I was talking to's going to be angry. The 's is a reduced is, and it is hanging off the preposition to. Now vary the host: The ball you hit's just broken my window puts it on a verb, Any answer not entirely right's going to be marked on an adjective, The drive home tonight's been really easy on an adverb. One element, four host categories. It is a reduced auxiliary throughout, is in the first and third sentence and has in the second and fourth, and in every case it does exactly what the separate word is or has would do.

Compare the plural -s of cats. It attaches to nouns and to nothing else, and it never does the job of a word: there is no sentence in which plural -s is the verb or the subject. That difference in behavior is what separates a Clitic from an Affix. Jurafsky and Martin define a clitic as "a morpheme that acts syntactically like a word but is reduced in form and attached (phonologically and sometimes orthographically) to another word" (SLP3, chapter 2). The two halves of that definition pull in opposite directions, which is why clitics are confusing: in the syntax they are words, in the sound stream they are part of a word.

Why spelling is not the test

English happens to mark most clitics with an apostrophe (I'm, we've, possessive 's), and French does the same with l' in l'opéra. That is a writing convention, not a grammatical fact. Arabic writes every clitic solidly attached with no separator at all, so if attachment in writing were the criterion, Arabic would have no clitics and English would have almost none outside contractions. The slide's key sentence is worth memorizing verbatim: attachment in writing is not enough, use diagnostics.

The diagnostics come from Zwicky and Pullum. Their first criterion is the one this course uses as the primary test, and they state it as: "Clitics can exhibit a low degree of selection with respect to their hosts, while affixes exhibit a high degree of selection with respect to their stems." This is Selectivity. In a footnote they give it as a ladder: an element that attaches to any word of a major class is least selective, then one that attaches only to verbs, then only to a subclass of verbs, then only to a few specified verbs. The further down the ladder, the more affix-like the element. The second test the slide names is paradigm membership: an affix fills one cell of a table of forms, a clitic does not. All six of their criteria are in the table below alongside the slide's two tests, and you should be able to name at least selectivity, arbitrary gaps and idiosyncrasy.

Three Arabic cases and a fourth that refuses to settle

Arabic is where the test earns its keep, because the writing system gives you no hint. Take وَبِالْبَيْت, wa-bi-al-bayt, roughly "and in the house". The wa- is the conjunction "and" and the bi- is the preposition "with, by, in". Both are function words in the syntax, and both attach to whatever word begins the phrase: a noun here, a verb in wa-kataba, a pronoun, a particle. Lowest rung of the selectivity ladder, no paradigm. They are proclitics, clitics that lean forward onto the following word.

Now كِتَابُهُ, kitāb-u-hu, "his book". The -u is the nominative case vowel, an inflectional ending. The -hu is the pronoun "he" in possessive position, and it is the possessor of the noun in the syntax. It attaches to nouns (kitāb-u-hu), to verbs as an object (kataba-hu, "he wrote it") and to prepositions (بِهِ, bi-hi, "with it"). Three host categories, a syntactic role of its own: an Enclitic, leaning backward onto the preceding word.

Finally كَتَبْتُ, katab-tu, "I wrote". The -tu also means something pronoun-like (first person singular), which is exactly what tempts students to call it a clitic. Apply the diagnostics. Selectivity: -tu attaches to verb stems and to nothing else. Paradigm: it is one cell of the perfective conjugation, next to -ta (you, masculine), -ti (you, feminine), -a (he) and -at (she). Syntactic role: it is subject agreement on the verb, and a separate subject pronoun can still appear alongside it. Every test says Inflection, so -tu is an inflectional affix on the Stem katab.

The fourth row is the one the slide flags as borderline, and it is worth understanding why rather than memorizing the label. The article al- is a function element like French l', which argues clitic. But it attaches only to nouns and adjectives, and it alternates with the indefinite ending -n (bayt-un versus al-bayt-u) inside a definiteness system, which argues prefix. Arabic NLP tools split the difference. Pasha et al. (2014) list two segmentation schemes for MADAMIRA: ATB, which "tokenizes all clitics except for the definite article", and D3, in which "basically all clitics are tokenized". ATB also leaves the future particle sa- attached to the verb, where D3 splits it off. Alotaiby et al. (2010) note that their parse-it-as-a-separate-word test "covers most of the clitics except the definite article". When the tools disagree, borderline is the honest verdict.

Clitic status and tokenization policy are different questions

The slide ends on a practical fork: keep the clitic attached, or split it? Linguistics does not answer that Tokenization question for you, and the cleanest proof is English n't. Zwicky and Pullum devote their paper to showing that n't passes all six criteria as an inflectional affix: it attaches only to auxiliaries, it has arbitrary gaps (there is no amn't), it has idiosyncratic forms (won't, shan't), and it cannot attach to a host that already carries a clitic (I'd've is fine, I'dn't is not). Yet the Penn Treebank tokenization standard splits it, producing does n't (SLP3, section 2.8). The tokenizer made a modeling decision (a negation token is useful) that contradicts the linguistic analysis, and it was right to.

Modern subword tokenizers make the same kind of decision by frequency rather than by rule. Jurafsky and Martin observe that the GPT-4o tokenizer segments 's off proper nouns such as Jane but keeps it inside frequent words such as she's (SLP3, section 2.4.3). For Arabic the choice is explicit and named: whitespace, ATB or D3. Try the three on the same two strings.

Take four languages and watch how each packages grammar. English and Turkish can share one meaning, "from my houses". English spends three words: from, my, house-s, four morphemes spread over three words. Turkish spends one: ev-ler-im-den, house-PL-1SG.POSS-ABL, four morphemes in one word, each suffix doing exactly one job in a fixed order. Spanish habl-o, "I speak", is a different meaning chosen to show a different trick: the single vowel -o carries person, number, tense and mood all at once, two morphemes carrying four features, and you cannot cut it into a piece for each. Mandarin wǒmen míngtiān qù Běijīng, "we tomorrow go Beijing", has almost no Inflection (the plural -men of wǒmen is about as far as it goes); word order and free words carry the grammar.

Turkish, Spanish and Mandarin illustrate three of the four labels on slide 24 (the fourth, polysynthetic, comes below), and English is the analytic baseline. The labels line up on two independent axes rather than one scale. The first axis is how many morphemes a word carries, from isolating (about one) to polysynthetic (a word can hold what English needs a clause for). The second axis is how cleanly the morphemes can be cut apart, from agglutinative (one morpheme, one feature, clean seams) to fusional (one segment bundles several features). This two-axis picture goes back to Sapir's 1921 classification, which separated degree of synthesis from technique of combination (Arkadiev 2019, and SLP3 section 2.2 uses the same two ideas).

The Turkish and Spanish rows deserve a closer look, because the difference between them is the second axis. Bickel and Nichols (WALS chapter 21) call Turkish case and number monoexponential: -ler means plural and nothing else, and the case suffixes -in, -i, -e each mean one case. In a fusional language such as Russian, they note, there is "no element that exclusively expresses case or number": the genitive plural ending says both at once. English has the same fusion in miniature. Jurafsky and Martin point out that the -s of she reads encodes third person, singular and present tense in one segment, exactly as Spanish -o does.

The polysynthetic row borrows its form from a neighboring language on purpose. The slide names Inuktitut but prints no word, warning that examples vary by dialect and orthography and that the key property is morpheme density. Jurafsky and Martin's example is Koryak: t-ə-nk'e-mejŋ-ə-jetemə-nni-k, one word meaning "I sewed a lot of yurt covers in the middle of a night" (SLP3, example 2.9), and that is the form to quote for the polysynthetic type. Gupta and Boulianne (2020) describe Inuktitut as showing "a much higher degree of polysynthesis than other agglutinative languages usually considered in ASR, such as Finnish or Turkish", which is a useful reminder that polysynthetic languages are usually agglutinative too: the axes are independent.

Tendencies, not boxes

The slide's mental model is that a language can be "more X" in one subsystem and "more Y" in another, and this is not a hedge. Jurafsky and Martin write that "languages can make use of different morphological systems so it would be more accurate to talk about these as general tendencies." Bickel and Nichols (WALS chapter 22) give an example inside one language: "the English past, for example, is more synthetic than the future", because walked packs tense into the verb while will walk spends a separate auxiliary. So when you classify, name the subsystem: English verbs are mostly analytic with a fusional pocket, English Derivation is productive and concatenative, and Arabic verbs are fusional while Arabic attached function elements are clitic-like.

Where English and Arabic sit

Slide 26 places the two languages this course cares about. English is largely analytic: auxiliaries and word order do most of the grammatical work, and inflection is limited to a handful of endings, -s, -ed, -ing. But it has fusional pockets. am, is and are each bundle person, number and tense in a form with no separable ending at all, and went replaces the expected go-ed wholesale, a case of Suppletion. Where English is genuinely productive is derivation and Compounding: un-happi-ness, laptop, and every new noun coined this year.

Modern Standard Arabic sits elsewhere on both axes. Word formation follows templatic morphology, so a consonantal Root such as K-T-B is interleaved with vowel patterns, a non-concatenative mechanism that no amount of prefix and suffix cutting will recover. The verb system is largely fusional, and it is large: Obeid et al. (2020) note that this richness "leads to MSA verbs with upwards of 5,400 forms". On top of the templatic core sit the concatenative pieces from the previous concept, proclitics wa-, bi-, li- and the pronominal enclitics, which behave as clitics in NLP, plus the borderline al-. Sapir's own table, as reproduced by Arkadiev, files English as fusional and analytic and Arabic as fusional, symbolic and synthetic, which is the same placement in older words.

The typology labels can be turned into a single number. Take a long text, count its morphemes, count its words, and divide. Joseph Greenberg proposed this ratio in 1960 as the Index of synthesis, "calculated as the ratio of morphemes to words in a sufficiently long text" (Arkadiev 2019). Try it on four English words: The cats walked home is the (1), cat-s (2), walk-ed (2), home (1), six morphemes over four words, an index of 1.5. Turkish ev-ler-im-den alone is 4 / 1 = 4.0, and the English equivalent from my houses is 4 / 3 ≈ 1.33.

Greenberg computed the index for eight languages, and the result is the number line on slide 27, reproduced from the same figure in Jurafsky and Martin (SLP3, figure 2.3). Vietnamese sits at the bottom with almost exactly one morpheme per word. English is close behind, which confirms the analytic label of the previous concept. Old English, Yakut, Swahili and Sanskrit cluster in the middle band, and Greenlandic (Eskimo in Greenberg's paper) stands far to the right at almost four morphemes per word. The bands drawn through this line are the ones you already know: analytic below about 2, synthetic between 2 and 3, polysynthetic above 3.

Greenberg's paper contains several more indices, and two of them explain what the synthesis number hides. Eskimo scores 2.72 on suffixation and 1.75 on inflection against English's 0.64 and 0.53; Swahili scores 1.16 on prefixation against English's 0.04. Swahili sits mid-synthetic not because it stacks suffixes but because its noun-class and agreement system lives in prefixes. Same band, different subsystem, which is the tendencies-not-boxes lesson in numbers.

Why the index decides your tokenizer

Slide 28 closes the morphology chapter with one sentence: "word" is not universal, because morphology and clitics change what counts as a Token. The index of synthesis makes that quantitative. Every extra morpheme per word multiplies the number of distinct surface forms a Lemma can take, and every clitic that attaches multiplies it again. Since a model's Vocabulary is a set of surface forms, a language with a high index and heavy cliticization will have a higher Type-token ratio (TTR), far more types for the same number of tokens, and far more of them will be Out-of-vocabulary (OOV) at test time. Recall Heaps' law from part 02: rich morphology tends to raise the exponent, so the vocabulary keeps growing quickly as the corpus grows.

At the polysynthetic end the effect is far larger. Schwartz et al. (2020) compare the parallel Inuktitut and English sides of the Nunavut Hansard. The numbers below come from their table 3.2.

Inuktitut produces about 26 times as many types from about half as many tokens. Gupta and Boulianne (2020) report the practical consequence for speech recognition: even with a 1.3 million word vocabulary, held-out Inuktitut stories still have "more than 60% of words out-of-vocabulary". A word-level vocabulary cannot be made large enough. That is the open-vocabulary problem, and slide 28 lists three responses to it.

1. Subword tokenization with BPE or WordPiece. Sennrich, Haddow and Birch (2016) motivated BPE for translation precisely because vocabularies are "typically limited to 30 000 to 50 000 words, but translation is an open-vocabulary problem", "especially for languages with productive word formation processes such as agglutination and compounding". WordPiece, from Wu et al. (2016), divides words "into a limited set of common sub-word units". Part 06 builds BPE from scratch.
2. Morphological segmentation when linguistic structure matters. A frequency-driven tokenizer will not reliably cut وَبِالْبَيْت at the clitic boundaries. MADAMIRA and CAMeL Tools do, using an analyzer and an explicit scheme (ATB or D3), which is what you want when a downstream task needs the bare lemma.
3. Careful preprocessing: case, punctuation and a stated clitic splitting policy. Jurafsky and Martin describe the pre-tokenization stage as "breaking the input at spaces and punctuation, stripping off clitics, and breaking numbers into sets of digits" (SLP3), which is the tokenization pipeline's first pass.

The last thread to pull runs into part 07. A multilingual subword vocabulary is trained mostly on English, and Jurafsky and Martin note that such tokenizers "use most of the tokens for English" while "the other languages tend to get their words split up into shorter tokens" (SLP3, section 2.4.3). Arabic suffers twice: it has less training data in the mix, and its clitics and templatic verbs (upwards of 5,400 forms per verb) spread that data across far more surface forms, so fewer of them earn a merge. The same sentence costs more tokens in Arabic than in English, which means more compute per sentence and a shorter effective context.

Two datasets both labelled "Arabic corpus" can be almost unrelated. One is Modern Standard Arabic newswire from 2005, written by trained journalists in a formal register. The other is Gulf-dialect tweets from 2024, full of Arabizi, emoji and code switching into English. A model trained on the first will stumble on the second, and a vocabulary learned from either will miss most of the other. Same language label, different domain, genre, time, demographics and variety.

A Corpus is a structured collection of texts, often annotated, drawn from sources such as books, news, web crawls, social media and speech transcripts. The word "structured" matters: a corpus is a deliberate sample with metadata, not a pile of files. Jurafsky and Martin describe the dimensions along which such samples vary as language and variety (including code switching), genre, the demographics of the writer or speaker, and time. The table below adds domain, which the slides list separately, and gives an Arabic example for each.

Models inherit their corpus

A language model is a compressed description of its training corpus, so it inherits the corpus's biases and blind spots. Bender and Friedman argue that systems perform best for the populations represented in their training data, so bias and exclusion arise when that data does not match the people the system is deployed for, and the fix begins with knowing what the data contains. Without that knowledge, a result on a benchmark cannot be interpreted: a 95% accuracy on 2005 newswire says nothing about 2024 tweets. To interpret any result you need the metadata below.

Data statements and datasheets

Two papers turned that metadata list into a standard document. Bender and Friedman (2018) proposed the data statement for NLP datasets, a short structured write-up with lettered sections from curation rationale through language variety, speaker and annotator demographics, speech situation and text characteristics. Gebru and colleagues (2021) proposed the datasheet for any machine learning dataset, borrowing the idea from electronics: a component datasheet lists operating ranges and test conditions, so a dataset should too. The slides merge the two into six fields.

The payoff is transparency, reproducibility and responsible use. Transparency because a reader can judge whether your corpus matches their population. Reproducibility because someone else can rebuild or extend the sample. Responsible use because a datasheet states what the data must not be used for, which is the only place such a limit can live once the files are downloaded.

Type the word café on two computers. On a Mac the é arrives as one unit, U+00E9. On a system with a decomposed input method it arrives as two, the plain letter U+0065 followed by a combining acute accent U+0301. On screen the two strings are pixel for pixel identical. In a program they are unequal, they have different lengths, and a tokenizer trained on one will split the other into pieces it never saw during training. Arabic has the same trap: a legacy keyboard emits the lam-alef ligature U+FEFB as one character, while a modern one emits U+0644 then U+0627.

To reason about this you need three separate words. Unicode assigns every character a Code point, an integer between 0 and 0x10FFFF written with a U+ prefix in hexadecimal, such as U+0061 for the letter a. That gives 1,114,112 possible positions, and the standard covers Latin, Cyrillic, Arabic, Han, emoji and over 170 other scripts within them. A Glyph is the rendered shape a font draws for a code point. The Unicode core specification is explicit that an abstract character should not be confused with a glyph: the code point is the identity, the glyph is one of many possible pictures of it.

The mapping runs many to many in both directions. One glyph can stand for several code points: the é above is one shape and two encodings, and Latin A (U+0041), Cyrillic А (U+0410) and Greek Α (U+0391) look identical in most fonts. One code point can also have several glyphs: the Arabic letter beh U+0628 is drawn as isolated, initial, medial or final form depending on its neighbours, yet the text stores a single code point and the font chooses the shape.

ASCII: the first 128 code points

The oldest slice of Unicode is ASCII, a 7-bit code with 128 positions defined in RFC 20, which occupies code points U+0000 to U+007F unchanged. The slide shows a fragment of the table in hexadecimal and decimal. The version below adds the binary column, because the binary reveals a fact you will use every time you write a case-folding rule.

Compare A at 0100 0001 with a at 0110 0001. They differ in exactly one bit, bit 5, which is worth 0x20 or 32. Every uppercase letter sits 32 positions before its lowercase partner, so Case folding in ASCII is a single OR with 0x20, which is why it is so fast and why it breaks the moment text leaves ASCII. Arabic has no case, and Greek final sigma has no simple bit rule, so Unicode defines case folding as a table lookup instead.

Normalization: one canonical spelling for each character

Back to café. Unicode calls U+00E9 the precomposed form and U+0065 U+0301 the decomposed form, and declares them canonically equivalent: they must be treated as the same character. Arabic has the same pair. Alef with hamza above, U+0623, decomposes canonically to bare alef U+0627 plus the combining hamza above U+0654, as the Unicode code chart records. A different, weaker relation is compatibility equivalence: the lam-alef ligature U+FEFB is compatible with U+0644 U+0627, and the ligature fi (U+FB01) with the two letters f and i. Compatibility variants look different or carry formatting, so folding them loses information.

Unicode normalization converts text to one of four standard forms defined in Unicode Standard Annex 15. The D forms decompose, the C forms decompose and then recompose, and the K forms also apply the lossy compatibility mappings. The table gives the result for é, for the lam-alef ligature and for the fi ligature, with the UTF-8 byte count of each result so you can see that normalization changes lengths, not only identities.

Here is the tokenizer bug normalization prevents. Suppose a BPE vocabulary is learned on NFC text, so "café" is one frequent token. Now NFD input arrives. The tokenizer sees c a f e U+0301, finds no merge that includes a lone combining mark, and emits "cafe" plus a separate accent token that no training word ever contained. Equality tests, dictionary lookups and Tokenization all fail in the same way, and all three are fixed by normalizing once, at ingestion, before anything else runs. Note that alef normalization in Arabic NLP toolkits, which folds U+0623, U+0625 and U+0622 into bare alef, is an application rule on top of Unicode, not one of the four forms.

The Arabic word كتاب has four letters, and Python's len agrees: four. Encode it and count the bytes: eight. The English word "Hi" is two and two. The difference is the whole content of this concept. Every Code point above U+007F needs more than one byte, and the rule that decides how many, and which bits go where, is small enough to learn by heart.

UTF-8 is a variable-length encoding form that maps each Unicode scalar value (a code point that is not a surrogate) to a sequence of one to four bytes. It is defined in the Unicode core specification and, for the Internet, in RFC 3629. Code points below 0x80 take one byte, and most other characters take two, three or four. The slide's mapping table is the exact rule, reproduced here with the payload capacity of each row.

Read the templates as two kinds of byte. The first byte of every sequence is the lead byte, and its run of leading ones announces the length: 0 means one byte, 110 two, 1110 three and 11110 four. Every later byte is a continuation byte and starts 10, leaving six payload bits. The payload capacities are 7, 11, 16 and 21 bits, and the last is no accident: the largest code point U+10FFFF is exactly a 21-bit number, so four bytes suffice for the whole codespace.

Now do it live. The simulator below encodes whatever you type, one character at a time, and shows the row, the payload bits, the templates and the resulting bytes. It preloads Hi كتاب 😀: nine code points, ten UTF-16 units and sixteen bytes. The € 中 preset exercises the three-byte row, the only one with a 1110 lead byte. Switch the normalization control to NFD with the café preset, or to NFKC with the ligature preset, and watch the byte count change before a single byte is encoded.