Corpora, Unicode and UTF-8

Two datasets both labelled "Arabic corpus" can be almost unrelated. One is Modern Standard Arabic newswire from 2005, written by trained journalists in a formal register. The other is Gulf-dialect tweets from 2024, full of Arabizi, emoji and code switching into English. A model trained on the first will stumble on the second, and a vocabulary learned from either will miss most of the other. Same language label, different domain, genre, time, demographics and variety.

A Corpus is a structured collection of texts, often annotated, drawn from sources such as books, news, web crawls, social media and speech transcripts. The word "structured" matters: a corpus is a deliberate sample with metadata, not a pile of files. Jurafsky and Martin describe the dimensions along which such samples vary as language and variety (including code switching), genre, the demographics of the writer or speaker, and time. The table below adds domain, which the slides list separately, and gives an Arabic example for each.

Models inherit their corpus

A language model is a compressed description of its training corpus, so it inherits the corpus's biases and blind spots. Bender and Friedman argue that systems perform best for the populations represented in their training data, so bias and exclusion arise when that data does not match the people the system is deployed for, and the fix begins with knowing what the data contains. Without that knowledge, a result on a benchmark cannot be interpreted: a 95% accuracy on 2005 newswire says nothing about 2024 tweets. To interpret any result you need the metadata below.

Data statements and datasheets

Two papers turned that metadata list into a standard document. Bender and Friedman (2018) proposed the data statement for NLP datasets, a short structured write-up with lettered sections from curation rationale through language variety, speaker and annotator demographics, speech situation and text characteristics. Gebru and colleagues (2021) proposed the datasheet for any machine learning dataset, borrowing the idea from electronics: a component datasheet lists operating ranges and test conditions, so a dataset should too. The slides merge the two into six fields.

The payoff is transparency, reproducibility and responsible use. Transparency because a reader can judge whether your corpus matches their population. Reproducibility because someone else can rebuild or extend the sample. Responsible use because a datasheet states what the data must not be used for, which is the only place such a limit can live once the files are downloaded.

Type the word café on two computers. On a Mac the é arrives as one unit, U+00E9. On a system with a decomposed input method it arrives as two, the plain letter U+0065 followed by a combining acute accent U+0301. On screen the two strings are pixel for pixel identical. In a program they are unequal, they have different lengths, and a tokenizer trained on one will split the other into pieces it never saw during training. Arabic has the same trap: a legacy keyboard emits the lam-alef ligature U+FEFB as one character, while a modern one emits U+0644 then U+0627.

To reason about this you need three separate words. Unicode assigns every character a Code point, an integer between 0 and 0x10FFFF written with a U+ prefix in hexadecimal, such as U+0061 for the letter a. That gives 1,114,112 possible positions, and the standard covers Latin, Cyrillic, Arabic, Han, emoji and over 170 other scripts within them. A Glyph is the rendered shape a font draws for a code point. The Unicode core specification is explicit that an abstract character should not be confused with a glyph: the code point is the identity, the glyph is one of many possible pictures of it.

The mapping runs many to many in both directions. One glyph can stand for several code points: the é above is one shape and two encodings, and Latin A (U+0041), Cyrillic А (U+0410) and Greek Α (U+0391) look identical in most fonts. One code point can also have several glyphs: the Arabic letter beh U+0628 is drawn as isolated, initial, medial or final form depending on its neighbours, yet the text stores a single code point and the font chooses the shape.

ASCII: the first 128 code points

The oldest slice of Unicode is ASCII, a 7-bit code with 128 positions defined in RFC 20, which occupies code points U+0000 to U+007F unchanged. The slide shows a fragment of the table in hexadecimal and decimal. The version below adds the binary column, because the binary reveals a fact you will use every time you write a case-folding rule.

Compare A at 0100 0001 with a at 0110 0001. They differ in exactly one bit, bit 5, which is worth 0x20 or 32. Every uppercase letter sits 32 positions before its lowercase partner, so Case folding in ASCII is a single OR with 0x20, which is why it is so fast and why it breaks the moment text leaves ASCII. Arabic has no case, and Greek final sigma has no simple bit rule, so Unicode defines case folding as a table lookup instead.

Normalization: one canonical spelling for each character

Back to café. Unicode calls U+00E9 the precomposed form and U+0065 U+0301 the decomposed form, and declares them canonically equivalent: they must be treated as the same character. Arabic has the same pair. Alef with hamza above, U+0623, decomposes canonically to bare alef U+0627 plus the combining hamza above U+0654, as the Unicode code chart records. A different, weaker relation is compatibility equivalence: the lam-alef ligature U+FEFB is compatible with U+0644 U+0627, and the ligature fi (U+FB01) with the two letters f and i. Compatibility variants look different or carry formatting, so folding them loses information.

Unicode normalization converts text to one of four standard forms defined in Unicode Standard Annex 15. The D forms decompose, the C forms decompose and then recompose, and the K forms also apply the lossy compatibility mappings. The table gives the result for é, for the lam-alef ligature and for the fi ligature, with the UTF-8 byte count of each result so you can see that normalization changes lengths, not only identities.

Here is the tokenizer bug normalization prevents. Suppose a BPE vocabulary is learned on NFC text, so "café" is one frequent token. Now NFD input arrives. The tokenizer sees c a f e U+0301, finds no merge that includes a lone combining mark, and emits "cafe" plus a separate accent token that no training word ever contained. Equality tests, dictionary lookups and Tokenization all fail in the same way, and all three are fixed by normalizing once, at ingestion, before anything else runs. Note that alef normalization in Arabic NLP toolkits, which folds U+0623, U+0625 and U+0622 into bare alef, is an application rule on top of Unicode, not one of the four forms.

The Arabic word كتاب has four letters, and Python's len agrees: four. Encode it and count the bytes: eight. The English word "Hi" is two and two. The difference is the whole content of this concept. Every Code point above U+007F needs more than one byte, and the rule that decides how many, and which bits go where, is small enough to learn by heart.

UTF-8 is a variable-length encoding form that maps each Unicode scalar value (a code point that is not a surrogate) to a sequence of one to four bytes. It is defined in the Unicode core specification and, for the Internet, in RFC 3629. Code points below 0x80 take one byte, and most other characters take two, three or four. The slide's mapping table is the exact rule, reproduced here with the payload capacity of each row.

Read the templates as two kinds of byte. The first byte of every sequence is the lead byte, and its run of leading ones announces the length: 0 means one byte, 110 two, 1110 three and 11110 four. Every later byte is a continuation byte and starts 10, leaving six payload bits. The payload capacities are 7, 11, 16 and 21 bits, and the last is no accident: the largest code point U+10FFFF is exactly a 21-bit number, so four bytes suffice for the whole codespace.

Now do it live. The simulator below encodes whatever you type, one character at a time, and shows the row, the payload bits, the templates and the resulting bytes. It preloads Hi كتاب 😀: nine code points, ten UTF-16 units and sixteen bytes. The € 中 preset exercises the three-byte row, the only one with a 1110 lead byte. Switch the normalization control to NFD with the café preset, or to NFKC with the ligature preset, and watch the byte count change before a single byte is encoded.

Decoding, and why the prefixes matter

The prefixes are what make UTF-8 safe to read from any position. A decoder that sees a byte starting 0 knows it is a complete ASCII character. A byte starting 110, 1110 or 11110 announces one, two or three continuation bytes to follow. A byte starting 10 can never begin a character, so a decoder dropped into the middle of a stream steps backwards at most three bytes to find a lead byte. RFC 3629 lists this as a design property (character boundaries are easily found from anywhere in an octet stream), Jurafsky and Martin call it self-synchronizing, and it is why a corrupted byte damages one character rather than the rest of the file.

Three consequences follow from the table and are worth stating explicitly. First, backward compatibility with ASCII: every code point below 0x80 becomes the single byte with the same value, so an ASCII file is already a valid UTF-8 file and a UTF-8 file that happens to contain only ASCII is byte for byte an ASCII file. Second, bytes below 0x80 never appear inside a multibyte sequence, so a program that searches for a space or a newline byte will never find one in the middle of an Arabic letter. Third, some byte values are simply impossible: C0, C1 and F5 to FF never occur in valid UTF-8, because they would encode overlong sequences or code points past U+10FFFF (RFC 3629, section 1, with the syntax that rules them out in section 4).

A tokenizer trained with a character vocabulary in 2020 meets the melting face emoji 🫠, added as U+1FAE0 in Unicode 14 the following year. It has never seen the character, so it emits the unknown token. The same happens with an Arabizi spelling like "ktaab" typed into a word-level vocabulary, or any typo, any new script, any rare Han character. Now imagine the base symbols were bytes instead. Every UTF-8 byte is one of exactly 256 values, all of them already in the vocabulary, so there is nothing left to be unknown.

That is Byte-level tokenization: treat the UTF-8 bytes of the text, not its characters, as the atoms. Radford and colleagues motivated it in the GPT-2 paper with the numbers. A Code point base vocabulary would start at over 130,000 symbols before a single merge (about 160,000 in Unicode 17), while a byte-level base needs only 256. Jurafsky and Martin draw the consequence: because there are only 256 possible byte values, there will be no unknown tokens, and the <UNK> symbol becomes unnecessary. Arbitrary scripts, emoji and typos all decompose into bytes the model has seen.

The price is structure. Bytes do not know where a character ends. The word كتاب becomes eight opaque symbols, and the letter ب is D8 A8, two units that mean nothing on their own. A sentence of Arabic is roughly twice as many base units as Latin text of the same length, which the simulator's كتاب preset shows as 4 characters against 8 bytes. Left alone, a byte model would spend its capacity relearning the UTF-8 table. So bytes are never used alone: they are combined with learned merges that glue frequent byte sequences back into characters, then into subwords. Jurafsky and Martin note that BPE over bytes rediscovers the two- and three-byte UTF-8 sequences early, and that it can also learn invalid sequences straddling a character boundary, which implementations filter out. GPT-2 adds a rule that prevents merges across character categories, because naive byte merges were sub-optimal.