Tokenizers in practice and sentence segmentation

Type the sentence Anyhow, she's seen Jane's 224123 flowers anyhow! into the tiktokenizer web app and pick GPT-4o. You get thirteen tokens, and Jurafsky and Martin use exactly this figure in SLP3 to introduce tokenizers in practice. Three of those tokens are worth staring at before any theory: Anyhow at the start is two tokens, Any and how, while the same word at the end is one token, ·anyhow. Jane's becomes ·Jane plus 's, but she's stays whole. And 224123 becomes 224 and 123, with a lone space token in front of it.

Every one of these splits has a mechanical explanation, and none of it is in the BPE trainer from part 06. A shipping BPE tokenizer is a pipeline. Hugging Face documents it as four stages: normalization, pre-tokenization, the model (BPE, unigram or WordPiece), and post-processing that adds special tokens. The slide lists the families: most modern tokenizers are BPE-like, with unigram (Kudo 2018) and WordPiece (Schuster and Nakajima 2012) as the variants, and almost all of them sit behind a regex-based Pre-tokenizer.

The pre-tokenizer sets the fences

The pre-tokenizer is a Regular expression that cuts the text into pieces before any merge is counted. GPT-2 uses one pattern, and cl100k and o200k are tightened variants of it. Its alternatives are tried left to right at every position, which is why the order in the table below matters.

On We're 350 dogs! Um, lunch? this regex yields We, 're, ·350, ·dogs, !, ·Um, ,, ·lunch, ?, the example SLP3 prints asFigure 2.15. The consequence for BPE is the single most useful rule of this part. Pair counts during training and merges during encoding are computed inside each piece. A pair whose two halves sit in different pieces is never a candidate, so the pre-tokenizer sets an upper bound: the final tokens are always parts of its pieces, never unions of them.

Now the three observations explain themselves. The leading space is part of the word piece, so ·anyhow and Anyhow are different byte strings. The lowercase, space-prefixed form is common in running text and earned a merge up to a whole token; the capitalized, sentence-initial form is rarer and stops at Any plus how. Position matters because the space marker is inside the piece. Second, the Clitic 's: under GPT-2 and cl100k the first regex alternative cuts every contraction into its own piece, so ·she and 's can never merge. o200k attaches the contraction to the word alternative, so the piece is ·she's, and since she's is a very frequent English word it has a token of its own, while ·Jane's does not and falls back to ·Jane plus 's. SLP3 states the same asymmetry: clitics are segmented off proper nouns like Jane but counted as part of frequent words like she's. Third, the digits: cl100k and o200k replace ·?\p{N}+ with \p{N}{1,3}, which caps any digit run at three per piece and, because the new alternative has no optional space, leaves the space before 224123 as a piece of its own (token id 220 in both vocabularies).

Vocabulary size is a parameter, not a result

The Vocabulary of a BPE tokenizer is its base symbols plus one entry per merge, and the trainer stops after k merges. Nothing in the algorithm chooses k. The slide gives a typical range of 30k to 100k; real systems now go further, and SLP3 quotes 50,000, 100,000 or even 200,000.

A larger vocabulary buys shorter sequences and more whole-word tokens, at the cost of a bigger embedding table and rarer training examples per token. The number is chosen, and the next concept shows that who benefits from that budget is also chosen.

Byte-level BPE: 256 symbols and no unknown character

The base symbols of the toy BPE in part 06 were characters. Radford et al. (2019) explain why GPT-2 did not do that: a base vocabulary of Unicode code points would exceed 130,000 entries before a single merge, while a byte-level version needs only 256. So GPT-2 runs merges over the UTF-8 bytes of the text. That is Byte-level tokenization, and its three advantages on the slide follow from one fact: every string in every script is a sequence of bytes from a set of 256, so there is no OOV character, any script works, and noisy text with typos or stray symbols still tokenizes. GPT-2 also adds a rule that prevents merges across character categories, with an exception for spaces, so that dog., dog! and dog? do not each become a token.

The con on the slide, that tokens may be less interpretable, is not abstract for Arabic. Arabic letters live in U+0600 to U+06FF, inside the two-byte range of UTF-8, so every letter is two bytes and a merge boundary can fall between them.

Try both ideas yourself. The playground below runs the GPT-2 regex on any sentence, replays a small merge list inside each piece and reports which merges the fences blocked, and shows the UTF-8 bytes with the token brackets measured above.

Take one meaning and write it twice. The book is on the table. is six words and a period, and its Arabic translation الكتاب على الطاولة. is three words and a period. Run both through cl100k and the English costs seven tokens while the Arabic costs twelve. Same meaning, half the words, nearly twice the tokens.

The o200k column shows the fix: a 200K vocabulary trained on more multilingual text spent merges on Arabic, so كتاب became one token and the Arabic sentence dropped to 6 tokens. The premium comes from how the budget was spent, not from the script.

The cause is the merge budget of the previous concept. BPE spends its k merges where the frequencies are, and English dominates the training text of most large models. English words therefore become single tokens while Arabic words fragment into letter-sized byte tokens. SLP3 puts it plainly: multilingual tokenizers tend to use most of their tokens for English, and other languages get poorer representations. The shared Vocabulary is a budget, and a shared budget favors whoever was most frequent when it was allocated. That is the slide's first two bullets in one sentence: shared vocabularies favor high-resource languages, so low-resource languages are over-segmented into longer sequences.

Petrov et al. (2023) turn this into a number. Take parallel sentences, tokenize both, and divide the length for language A by the length for language B. They call the ratio the tokenization premium of A relative to B.

Arabic pays 4.40 times the English token count under the GPT-2 vocabulary and 3.04 under cl100k. Shan and Burmese pay over fifteen and eleven times. The ByT5 column shows that even byte and character models, which have no merges at all, are unequal because scripts differ in bytes per character: Petrov et al. report up to fifteen times difference for subword models and over four times between the byte encodings of Burmese or Tibetan and Chinese. Ahia et al. (2023) measured the same effect on ChatGPT across 22 languages, with some needing five times as many tokens as others, and name two causes: how much of the language was in the pretraining data, and inherent properties of the language and its script. In their analysis of the BLOOMZ tokenizer they flag Arabic as an outlier with more tokens than some other mid-resourced languages.

Three bills for one over-segmented sentence

1. Compute. Every layer processes every token, so three times the tokens is at least three times the work, and self-attention, whose cost grows with the square of sequence length, makes long inputs worse still.
2. Context. A window of fixed token length holds a third as many Arabic words as English words. Ahia et al. note that fewer in-context examples fit, which directly lowers few-shot quality.
3. Money and quality. APIs bill per token, so Ahia et al. conclude that speakers of many languages are overcharged while obtaining poorer results, and SLP3 adds that fragmented tokens give poorer representations of meaning.

The slide's third bullet, that tokenizer design affects fairness and performance, is an engineering decision you can make differently.

SuperBPE: letting merges cross the space

If fences limit how much a vocabulary can compress, one way to compress more is to remove a fence late in training. Liu et al. (2025) call this SuperBPE: a pretokenization curriculum for BPE that first learns subwords under the ordinary whitespace fences, then continues with merges that are allowed to bridge whitespace and produce superwords. The figure on the slide is their example. Ordinary BPE gives By | the | way | , | I | am | a | fan | of | the | Milky | Way | ., thirteen tokens. SuperBPE gives By the way | , I am | a | fan | of the | Milky Way | ., seven.

The gain is the slide's stated goal, efficiency. At a 200K vocabulary SuperBPE beats ordinary BPE on every measure the authors report, so the extra training stage pays for itself.

SLP3 mentions BoundlessBPE (Schmidt et al. 2025) as a sibling method with the same idea.

The tradeoff is the slide's third bullet, Compositionality. Under ordinary BPE, Way in Milky Way and way in By the way share visible structure with every other use of the word. Once Milky Way is one token, the model must learn its relation to way from scratch, and a rare phrase that almost matches a superword is split in an unfamiliar way. Fewer tokens means less shared structure across tokens.

Every surprise in this part came from a switch someone set before training: whether the space belongs to the next word, whether digits are capped, whether contractions split, how many merges to make, which languages got the budget. The slide collects those switches into a checklist. Treat each line as a question with a consequence, because that is how an exam or a design review will ask it.

Three answers to the spaces question

The slide's pre-tokenization line asks one question with three real answers. Once the text is split on whitespace, what happens to the space? The choice fixes the Space marker (end-of-word marker) convention the model will live with, and it decides whether detokenization can recover the original string.

BERT throws the space away, which keeps the alphabet small but makes the token stream lossy: don't and the three tokens don, ', t cannot be told apart from a hyphenless spelling once the spaces are gone. GPT-2 glues the space onto the following piece, so ·world and world are different entries and the original spacing is recoverable. SentencePiece keeps the space as an explicit ▁ symbol, which is exactly reversible: swapping ▁ back to a space restores the input byte for byte. It also means the regex fences from the first concept apply to a text where spaces are ordinary symbols, so SentencePiece can in principle learn a merge that crosses what used to be a word boundary.

The rule that ties the list together is that all of it is frozen with the model. The Unicode normalization form, the case folding decision, the Special tokens and the Pre-tokenizer regex are recorded in the tokenizer file, and Hugging Face's pipeline documentation warns that changing the normalizer or the pre-tokenizer requires retraining the tokenizer, which in turn means the embeddings of the language model no longer match. You cannot fix an English-centric tokenizer after the fact; you choose it before the first gradient step.

Here is a passage with six periods, one question mark and one exclamation mark: Dr. Ahmad arrived at 5 p.m. on Monday. He paid 3.50 riyals. Was it enough? Yes! It contains four sentences. A splitter that cuts at every period would produce seven fragments and would glue the last two sentences together. Decide for yourself which marks end a sentence before reading the rules.

Sentence segmentation is the task of finding sentence boundaries, and the slide names its usual customers: parsing and machine translation, both of which take one sentence at a time. SLP3 states the difficulty exactly. Question marks and exclamation points are relatively unambiguous markers of sentence boundaries. The period is ambiguous between a sentence boundary and an abbreviation marker such as Dr. or Inc., and it can even do both at once: when a sentence ends in Inc., one period marks the abbreviation and the boundary. Kiss and Strunk (2006) list the period's other jobs too: initials, ordinal numbers and ellipses, and in their corpora abbreviations account for up to 30% of the candidate boundaries.

Rules, lexicons and the joint solution

The slide's rule-based heuristics combine punctuation with capitalization patterns, an abbreviation list and quote or bracket balancing. Each rule is a feature that votes on one candidate mark.

Notice the conflict in the exercise. After Dr. comes Ahmad, capitalized, so the capitalization rule votes for a boundary and only the abbreviation lexicon overrules it. After p.m. comes on, lowercase, so both rules agree. This is why the slide says segmentation is often done jointly with Tokenization and abbreviation lexicons: if the tokenizer has already kept Dr., p.m. and 3.50 as single tokens, the segmenter never has to look at those periods. Stanford CoreNLP states its rule in exactly that form: a sentence ends when a sentence-ending punctuation mark is not already grouped with other characters into a token, optionally followed by closing quotes or brackets. SLP3 notes the abbreviation dictionary can be hand-built or machine-learned.

Learning the boundary

The statistical approach on the slide replaces hand-tuned votes with a classifier that learns the probability of a boundary from annotated data. Its features are the same evidence the rules used: the token before the mark, whether it is in an abbreviation list, its length, whether the next token is capitalized, whether the next token is a frequent sentence starter, and, in neural models, the surrounding context as a whole. Kiss and Strunk (2006) went one step further and removed the annotation. Their Punkt system, the default sentence tokenizer in NLTK, is unsupervised and language independent. It treats an abbreviation as a very tight collocation of a truncated word and a final period, which can be detected from raw text by asking whether the word occurs with a period far more often than chance, helped by the facts that abbreviations are usually short and sometimes contain internal periods. It also learns frequent sentence starters and collocations across a period. Its measured results on newspaper text are in the table.