Word embeddings

Open the vocabulary file of an n-gram model. cat might be entry w_412, dog entry w_977 and spreadsheet entry w_3051. Ask the model which of those three words are alike and it has nothing to say. The numbers are positions in a list, and a list position carries no meaning: 412 is not closer in sense to 977 than to 3051.

Turning each index into a vector does not help. A one-hot vector has a single 1 at the word's index and zeros everywhere else. Two distinct words never share a non-zero position, so their dot product is always 0, whichever pair you pick:

Jurafsky and Martin describe exactly this situation: in the n-gram models of Chapter 3 and in classical NLP applications, the only representation of a word is a string of letters or an index in a vocabulary list. A model trained on the cat sat learns nothing about the dog sat, because the two sentences share no symbol in the position that matters.

The logic class answer, and why it is circular

An introductory logic course offers a different answer: the meaning of dog is the predicate DOG, and the meaning of cat is CAT. Relations between meanings are then stated as axioms written by hand, for instance that every dog is a mammal:

This is more expressive than an index, because the axioms support inference. But the symbol itself still says nothing; it is the word in capital letters. The old semantics joke makes the point. Q: What is the meaning of life? A: LIFE. Jurafsky and Martin attribute it to the semanticist Barbara Partee and call capitalization a pretty unsatisfactory model of meaning. Every relation you want, from similarity to connotation, has to be typed in by a person, and nothing about DOG tells you that it should sit near CAT.

The rest of the lecture fills the third row. Vector semantics represents a word as a point in a space built from the contexts it appears in, and the resulting Embedding makes closeness a number you can compute instead of an axiom someone must write.

Consider three sentences: Ann bought a car from Bo. Bo sold Ann a car. Ann paid Bo for a car. One event, one car, one transfer of money, described from three positions. Any adequate account of word meaning has to know that buy, sell and pay are tied together this way, and none of the symbol representations from the previous concept does.

Lexical semantics is the linguistic study of word meaning. It is not the study of dictionary definitions one entry at a time; it is the study of how meanings relate to each other. Jurafsky and Martin turn its findings into a list of what a model of word meaning should deliver, and this list is the checklist every later model in the lecture is graded against.

The first three entries get their own treatment in part 02: similarity as a graded human judgment, Antonymy as opposition on a single feature, and Connotation as affective meaning. Perspective and inference are what make the list more than a thesaurus: a question answering system that is asked who bought the car must connect it to a sentence that only says who sold it.

Keep the list in view as the lecture moves on. Sparse count vectors will turn out to be good at similarity and relatedness and weak at antonymy, because hot and cold occur in the same contexts. Word2vec will improve similarity further and add analogies, yet still give one vector to every sense of a word. Each model earns or loses marks on these rows.

Type mouse into WordNet. The slide quotes two of its meanings: any of numerous small rodents, and a hand-operated device that controls a cursor. The full WordNet 3.0 entry has six: four noun senses and two verb senses. Before naming the parts of this entry, look at how much a single spelling has to carry.

from nltk.corpus import wordnet as wn

for synset in wn.synsets("mouse"):
    print(synset.name(), synset.lemma_names())

Lemma and wordform

The headword mouse is a Lemma, also called the citation form: the form under which a dictionary lists the word. The plural mice has no entry of its own; it is a wordform of the same lemma. A wordform is any inflected form a lemma takes in running text. The same split holds across verbs and languages, and it is the same lemma idea you met in tokenization and morphology.

Sense and polysemy

Each numbered meaning in the entry is a sense: a discrete aspect of the word's meaning. A lemma with several senses is polysemous, and the phenomenon is called Polysemy. WordNet groups senses into synsets, sets of near-synonymous senses that share one gloss and express one concept. That is why the black eye sense appears under the name shiner.n.01: its synset is {shiner, black_eye, mouse}, and WordNet names a synset after its first member.

How do you know two meanings really are separate senses? One practical test is the zeugma. In ?Does Air France serve breakfast and Philadelphia? the two uses of serve (providing food and flying to a city) are forced to share one verb, and the sentence sounds like a pun. That oddness is the evidence for two senses. With one sense, coordination is fine: Air France serves breakfast and lunch.

Why the split matters for systems

Jurafsky and Martin point out that a search for mouse info is ambiguous between a pet owner and a shopper. Deciding which sense a given occurrence uses is the task of word sense disambiguation. And the split sets up a limitation you will meet at the end of this lecture: a Static embedding gives each word type one vector, so the vector for mouse must blend the rodent and the device. A Contextual embedding computes a different vector for each occurrence, which is how modern models separate senses.

couch and sofa. filbert and hazelnut. car and automobile. vomit and throw up. big and large. Each pair can be swapped in some sentence without anyone noticing a change in what is claimed. That intuition has a precise form.

Two words are synonymous if they can be substituted for each other in any sentence without changing the truth conditions of the sentence, that is, the situations in which the sentence would be true. This is Synonymy in its truth-conditional definition. I sat on the couch and I sat on the sofa are true in exactly the same situations. WordNet even files car and automobile in the same synset, car.n.01.

The slide states the same idea more loosely: synonyms have the same meaning in some or all contexts. All contexts is the strict truth-conditional ideal. Some contexts is what real pairs such as big and large achieve, and that gap is exactly why synonymy is stated between senses rather than words.

The twist: the relation is between senses

Try the substitution with big and large. In Would I be flying on a large or small plane? the swap to big is harmless. In Miss Nelson became a kind of big sister to Benjamin it is not: a large sister is a different claim. The textbook draws the conclusion directly. Synonymy is a relationship between senses rather than words. WordNet makes this visible: big has 17 synsets and large has 11. They share some, such as large.a.01, above average in size, and not others, such as big.s.01, significant. A claim that two words are synonyms is really a claim about one of their senses.

water and H₂O refer to the same substance. Swap them in the glass contains water and the sentence stays true in exactly the same situations. Now imagine a hiking guide that says to carry two litres of H₂O per person. Nothing false was said, and yet the sentence is wrong for its setting. That wrongness is part of what the words mean.

The same thing happens with big and large: even setting aside the older-sibling sense, the pairs that pass the truth test still differ somewhere. Jurafsky and Martin put it carefully: while substitutions between some pairs of words like car and automobile or water and H₂O are truth preserving, the words are still not identical in meaning, and probably no two words are absolutely identical in meaning.

The principle of contrast

The generalization behind this is the Principle of contrast: a difference in linguistic form is always associated with some difference in meaning. The idea has a long history, from Girard in 1718 and Bréal in 1897 to Eve Clark in 1987, who stated it as: every two forms contrast in meaning. Clark used it to explain language acquisition: a child who already knows one word for a thing assumes a new word for the same thing must mean something different. In her words, there are no true synonyms.

Where do apparent synonyms differ, if not in truth? Clark names three dimensions, and the slide's list of politeness, slang, register and genre maps onto them.

• Dialect. autumn and fall, truck and lorry, tap and faucet: the choice tells the listener where the speaker is from.
• Register. die, pass away and pop off; attempt and try. A register is a speech style such as formal, colloquial or technical. Politeness and slang sit here, and so does genre: H₂O is the technical register of a chemistry text.
• Connotation. politician and statesman, skinny and slim: the referent can be the same while the attitude differs. This is Connotation, which part 02 measures.

The consequence for the rest of the course is terminological: when NLP papers say synonym, they mean approximate synonymy, two senses close enough to substitute in most contexts. It also has an engineering consequence. Query expansion in search, paraphrase generation and data augmentation by synonym replacement all assume substitutability, and the principle of contrast predicts they will shift register, sense or tone some of the time. A paraphraser that rewrites passed away as died has kept the truth and changed the message.

car and bicycle are not synonyms. Neither are cow and horse. Yet everyone feels they belong together: both pairs share an element of meaning, a vehicle you ride on a road, a large farm animal. Swap them, though, and the truth changes. I took my car to work describes a different morning from I took my bicycle to work.

This relation is Word similarity. Jurafsky and Martin give the reason it matters: while words don't have many synonyms, most words do have lots of similar words. A model that only knew synonymy would have almost nothing to say about most of the vocabulary; a model of similarity has something to say about every word.

There is a second, quieter shift. Synonymy was a relation between senses, which requires deciding first what the senses of every word are. Similarity is usually stated between words, which avoids committing to a sense inventory at all. That is exactly the quantity vector models compute: one number for a pair of words, later the cosine of the angle between their vectors. Part 02 shows how humans rate it, on datasets such as SimLex-999, and how it differs from relatedness.

Read these pairs and give each one a number from 0 to 10 for how alike the two meanings are: vanish and disappear, behave and obey, belief and impression, muscle and bone, modest and flexible, hole and agreement. You probably gave the first pair close to 10, the last close to 0, and found the middle harder but not impossible. Hundreds of people did exactly this, and their averages are the numbers below.

The association column comes from the University of South Florida free-association norms: how often people answer the second word when given the first as a cue. It measures connection, not shared features.

The scores fall away smoothly. There is no point where the pairs stop being similar and start being dissimilar, which is the first thing to notice. Synonymy, from part 01, is close to a yes or no question about two senses. Word similarity is a matter of degree: two words are similar when their meanings share features, and they can share many features, a few, or none. vanish and disappear share nearly all of them. muscle and bone share a few (body tissue, anatomy) and differ on the rest. hole and agreement share essentially nothing.

The second thing to notice is that similarity is a relation between words, not senses. This is the car and bicycle idea from the end of part 01. To say whether two senses are synonyms you need a sense inventory; to ask a person how similar two words feel, you do not. That makes word similarity cheap to collect and directly comparable to anything that produces one number per word pair, which is exactly what a vector model does.

Where the numbers come from

The table is SimLex-999 (Hill, Reichart and Korhonen 2015). It contains 999 pairs: 666 noun pairs, 222 verb pairs and 111 adjective pairs, mixing concrete and abstract words. About 500 Mechanical Turk workers rated them on an integer slider from 0 to 6, and the mean ratings were then linearly rescaled to 0 to 10. The slide shows the rescaled values, which match the released data file exactly. Individual raters disagree a fair amount: the average Spearman correlation between two raters is 0.67, and between one rater and the mean of the others it is 0.78. The average is what is stable.

Why should you care about these particular numbers? Because later in this lecture they become the gold standard for intrinsic evaluation. You compute the cosine between the two vectors of every SimLex pair, rank the pairs by cosine, and report the Spearman rank correlation with the human ranking. A model that agrees with people about which pairs are more alike scores high.

Compare two pairs: coffee and tea, then coffee and cup. Coffee and tea are both hot drinks made by steeping or brewing a plant, both contain caffeine, both are served in the morning. They share features, so they aresimilar. Coffee and cup share practically no features: one is a drink from a plant, the other is a manufactured container. Yet nobody would call them unconnected. They take part together in one everyday event, drinking coffee from a cup.

That second kind of connection is word relatedness, also called association. Two words are related when they are connected in any way at all: by shared features, by a shared event, by part and whole (car and wheel), by function (pencil and paper), even by opposition (hot and cold). Budanitsky and Hirst put the hierarchy plainly: similarity is a special case of relatedness. Every similar pair is related, but many related pairs are not similar. They quote Resnik's example: cars and gasoline are more closely related than cars and bicycles, but the latter pair are certainly more similar.

The SimLex file records both quantities, so you can see the split in real numbers. clothes and closet have similarity 3.27 but association 4.83: related more than they are alike. car and bicycle have similarity 3.47 and association only 0.41: alike more than they are associated.

Semantic fields

When you collect all the words that keep turning up in the same scene, you get a semantic field: a set of words that cover one domain and bear structured relations to each other. The hospital field holds surgeon, scalpel, nurse, anaesthetic and hospital. The restaurant field holds waiter, menu, plate, food and chef. The house field holds door, roof, kitchen, family and bed. Inside a field the relations are varied: a surgeon uses a scalpel, a nurse assists a surgeon, the anaesthetic is given in the hospital. What unites them is the domain, not shared features.

Fields are not only a linguist's classification. Topic models such as Latent Dirichlet Allocation read a large collection of documents with no labels and discover clusters of words that tend to occur together, and the clusters they find look very much like semantic fields. That is a first hint of the theme of this whole lecture: the company a word keeps carries information about its meaning.

Take hot and cold. Both are adjectives. Both describe temperature. Both slot into the same frames: hot coffee and cold coffee, hot weather and cold weather, it is too hot today and it is too cold today. Line up everything you know about the two words and they agree on almost every point. They disagree on exactly one: which end of the temperature scale they name.

That is the definition of antonymy. Antonyms are senses that are opposite with respect to only one feature of meaning and otherwise very similar. It sounds paradoxical that opposites are mostly alike, but you cannot be opposite to something unless you are first comparable to it. Hot is not the opposite of Tuesday.

Kinds of opposition

The single opposed feature can be of different types. In the first group the two words name the two values of a binary choice or the two ends of a scale: long and short, fast and slow, big and little. In the second group, the reversives, the two words describe change or movement in opposite directions: rise and fall, up and down. Mohammad, Dorr, Hirst and Turney refine this further (antipodals, complementaries, gradable opposites) and note that many contrasting pairs, such as warm and cold, are not strict opposites at all.

What humans say, and why models struggle

Because antonyms differ on a feature people care about, human raters call them dissimilar. Because they share everything else, they are among the most strongly associated pairs in the language. SimLex records both, and the gap is striking. Hill and colleagues conclude that antonyms are the most strongly associated word pairs among the finer-grained relations they examined.

Now look ahead. The distributional hypothesis that drives the rest of this lecture says that words in similar contexts have similar meanings. hot and cold occur in almost identical contexts, so a model built on contexts will put them close together. Opposites even co-occur in the same sentence more often than chance would predict (Charles and Miller, cited by Mohammad and colleagues), which pulls them closer still. SLP3 is blunt about the result: automatically distinguishing synonyms from antonyms can be difficult.

A museum shop sells a replica of an ancient vase. A street stall sells a knockoff. Both objects are copies of a real thing, and a description of either would read much the same. But the first word is close to praise and the second is an accusation.

The part of meaning that differs here is connotation: the aspects of a word's meaning tied to a writer's or reader's emotions, sentiment, opinions or evaluations. Some words exist mainly to evaluate: great and love are positive, terrible and hate are negative. Others, like replica and knockoff, describe the same thing while carrying different attitudes toward it. Positive or negative evaluation in language is called sentiment, and connotation is what sentiment analysis, stance detection, and NLP work on political language and consumer reviews all exploit.

Connotation can be measured. Affect lexicons give each word a score, and the one we meet in the next concept, the NRC VAD Lexicon, gives a valence (pleasantness) between 0 and 1. Here is the SLP3 example worked through with its real numbers.

Compare napping and toxic. napping is pleasant and calm, and it puts you in no particular position of control. toxic is unpleasant and agitating. A single positive or negative score would capture the first difference, but not the second, and not the question of who has the power.

Osgood and colleagues (1957) found that people's ratings of words consistently varied along three affective dimensions, now called valence, arousal and dominance:

• Valence is the pleasantness of the stimulus. napping 0.765, toxic 0.008.
• Arousal is the intensity of emotion the stimulus provokes. napping 0.046, toxic 0.885.
• Dominance is the degree of control the stimulus exerts. napping 0.306, toxic 0.492.

Three numbers per word means each word becomes a point in a three-dimensional space. SLP3 calls this the first expression of the idea behind vector semantics: on the 1 to 9 scales of Warriner and colleagues (2013), heartbreak sits at [2.45, 5.65, 3.58]. Part 03 takes that idea and scales it from three hand-chosen dimensions to hundreds learned from text.

How the NRC VAD Lexicon was built

The numbers come from the NRC VAD Lexicon (Mohammad 2018). Version 1 covers about 20,000 English words; the released file has 19,971 entries. Asking people to rate a word on a slider is unreliable, because everyone uses the slider differently. Instead Mohammad used best-worst scaling. An annotator sees four words and picks the one highest on the dimension (say, most pleasant) and the one lowest. Over many such four-word sets, each word is scored by how often it won minus how often it lost.

Comparative judgements are much more consistent than absolute ones. Splitting the annotators into two random halves and correlating the scores each half produces gives split-half reliability of r = 0.95 for valence, 0.90 for arousal and 0.91 for dominance. Version 2, released in March 2025, extends the lexicon to over 55,000 terms (about 10,000 of them multiword phrases) on a -1 to 1 scale, with split-half Spearman of 0.98, 0.97 and 0.96.

Step back and look at what we now have. One word can map to many senses: mouse is a rodent or a pointing device. One sense can map to many words: couch and sofa. The mapping between words and concepts is many to many, and on top of it sits a set of relations, some between senses and some between whole words.

A lemma groups its senses, and polysemy is the fact that it has several. Synonymy and antonymy are relations between senses. Similarity, relatedness and connotation are graded and can be asked of whole words, which is what makes them measurable with ratings and lexicons.

This table is a list of desiderata. Any representation of word meaning we build next should put similar words near each other, keep related words in recognisable neighbourhoods, encode affect in some consistent direction, and ideally tell synonyms from antonyms. Part 03 introduces vectors as that representation. A vector space captures relatedness readily and affect reasonably well; separating true similarity from relatedness is harder, and telling synonyms from antonyms is where it struggles most, as concepts 2 and 3 warned.

Take two words, "oculist" and "eye-doctor". Collect every sentence each appears in, and look at the neighbors: eye, examined, prescription, glasses, appointment. The two lists are almost the same. Now compare "oculist" with "lawyer". Some neighbors still overlap (appointment, fee, office), but far fewer. Without a dictionary, the overlap of environments already tells you which pair is closer in meaning.

That observation is the distributional hypothesis. Zellig Harris put it in 1954 using exactly this example: if two words have almost identical environments, meaning neighboring words or the grammatical frames they occur in, we call them synonyms. He then made the claim graded. The difference in meaning between two words corresponds roughly to the amount of difference in their environments. That second sentence matters more than the first, because it turns meaning into a quantity you can estimate by counting.

The idea was in the air in the 1950s. Wittgenstein had argued that, for a large class of cases, the meaning of a word is its use in the language. Joos (1950) described the meaning of a morpheme as the set of conditional probabilities of its occurrence alongside every other morpheme, which is almost a definition of a language model. Firth (1957) gave the line everyone quotes: "You shall know a word by the company it keeps." Firth and Harris are often merged, but they meant different things. Firth cared about situational and cultural context; Harris cared about the formal distribution of words inside text. NLP took Harris's version, because it can be computed from a corpus alone.

This is why the lecture moves to vector semantics. Earlier, lexical semantics gave us a list of relations a good model should respect: synonymy, similarity, relatedness, connotation. Writing those relations down by hand for every pair of words is impossible. The distributional hypothesis says you do not have to: read enough text, record each word's environments, and the relations fall out of the overlaps. Vector semantics is the standard way NLP does this today.

Suppose you have never seen the word "ongchoi", a recent borrowing into English from Cantonese, and you meet it three times: ongchoi is delicious sauteed with garlic; ongchoi is superb over rice; ongchoi leaves with salty sauces. You do not know what it is yet. But you have read plenty of other sentences: spinach sauteed with garlic over rice, chard stems and leaves are delicious, collard greens and other salty leafy greens.

Put the contexts side by side and the answer is hard to miss. Sauteed, garlic, rice, leaves, delicious and salty all show up around ongchoi and around the leafy greens you already know. Nothing about laptops, contracts or weather. So ongchoi is very likely a leafy green that people cook and eat. It is: the plant is Ipomoea aquatica, a relative of morning glory sometimes called water spinach, and it has other names in Chinese, Malay and Vietnamese.

This is the computational form of the distributional hypothesis. Define the meaning of a word by its distribution, the neighboring words or grammatical environments it appears in, and then do the obvious thing: count the words in the context of ongchoi and compare those counts with the counts for every other word. A table of such counts, one row per word and one column per context word, is the term-context matrix of part 4, and each row is a word's vector.

Here are two words with three numbers each. Heartbreak is [2.45, 5.65, 3.58] and courageous is [8.05, 5.5, 7.38]. The first number is valence (how pleasant), the second is arousal (how intense the emotion), the third is dominance (how much control is exerted). Plot both as points. They sit far apart on valence and dominance, and almost level on arousal: both words are emotionally charged, one pleasantly and with control, the other unpleasantly and with loss of control.

The idea of placing a word at a point goes back to Charles Osgood and colleagues in 1957. Osgood asked people to rate words on many bipolar scales, such as good to bad, strong to weak, active to passive, a method he called the semantic differential. Factor analysis showed that most of the variation came from three factors, which he named evaluation, potency and activity. Today the same three are usually called valence, dominance and arousal, the VAD dimensions of a word's connotation from part 2. Osgood noticed that three numbers per word make each word a point in a three-dimensional space, and he proposed that similarity of meaning is nearness in that space. That is the first appearance of vector semantics.

Put the two threads together. Idea 1 is that meaning can be defined by linguistic distribution. Idea 2, from Osgood, is that meaning can be a point in a multidimensional space. Vector semantics joins them: represent each word as a point, but let the word's distribution, not a panel of human raters, decide where the point goes.

Look at a map of word vectors squashed onto a page. Good, nice, wonderful, fantastic, amazing and very good huddle in one region. Bad, worst, worse, dislike and not good gather in another. Function words such as to, by, that, is and with sit off by themselves. Nobody placed these words by hand. Their positions came from training on text.

This is vector semantics in its working form. Each word is a vector, a list of numbers, rather than an arbitrary symbol such as the string "good" or the index w45. Similar words end up nearby in what is called semantic space. And the space is built automatically, by seeing which words are nearby in text, so the distributional hypothesis does the placing that Osgood's raters used to do.

Such a vector is called an embedding, because the word is embedded into a space. The term began in the latent semantic analysis community in the late 1990s, where it named the mapping from sparse count space into a smaller dense space, and it later shifted to mean the resulting vector itself. Embeddings are now the standard way to represent word meaning in NLP: practically every modern system, from classifiers to large language models, starts by looking up or computing them. They give a fine-grained model of similarity, a number for every pair of words instead of a yes or no.

Build a sentiment classifier the traditional way. Feature 5 is "the previous word was terrible", and training learns that it signals a negative review. At test time a review says "awful acting". If awful never appeared in the labeled training data, feature 5 does not fire, no other feature knows about awful, and the classifier has nothing to go on.

Now replace the identity feature with the previous word's embedding. During training the input was terrible's vector, say [35, 22, 17, ...], and the classifier learned weights on those coordinates. At test time awful arrives as [34, 21, 14, ...]. The weights do not care whether the word is the same string; they act on the numbers, and these numbers are nearly the same. The classifier treats awful almost exactly as it treated terrible. It has generalized to a similar but unseen word.

This matters because of what lecture 2 showed about vocabularies. Zipf's law means most word types are rare, and about half of them appear only once. A classifier trained on a few thousand labeled reviews will meet many words at test time that it never saw with a label. Embeddings, learned from billions of unlabeled words, carry knowledge about those words into the classifier. Dense vectors also need far fewer weights, around 300 per input position instead of one per vocabulary word, and they can place car and automobile close together, which separate identity features never can.

Picture two vectors for ongchoi. The first has one slot per vocabulary word, tens of thousands of them, and stores how often each word appeared near ongchoi. Garlic, rice and leaves have counts; nearly every other slot is 0. The second has about 300 real numbers, almost none of them zero, some negative, and none of them labeled with a word. Both are embeddings. This lecture covers both families.

Sparse vectors: weighted counts

A sparse vector is built from a simple function of counts. With tf-idf, the classic weighting from information retrieval, the counts come from documents. With PPMI, they come from nearby words, re-weighted so that informative co-occurrences stand out. These vectors are the workhorse of search engines and a strong, cheap baseline in almost any text task. Their dimensions are interpretable, because each one is a specific word or document.

Dense vectors: learned by prediction

A dense vector such as one from word2vec is learned instead of counted. Word2vec trains a simple classifier to predict whether a word is likely to appear near a target word, and keeps the classifier's weights as the embedding. The labels come free from running text, which is self-supervision. Both families give one fixed vector per word type, a static embedding. Later in the course, contextual embeddings such as BERT compute a different vector for each occurrence of a word.

Take four Shakespeare plays and four words, and count. As You Like It uses battle once, good 114 times, fool 36 times and wit 20 times. Twelfth Night has no battle at all but 58 fools. Julius Caesar and Henry V, the two histories, have almost no fools and plenty of battles. Put the counts in a grid with one row per word and one column per play, and you already have a usable representation of each play.

This grid is a Term-document matrix. In general it has |V| rows, one per word in the vocabulary, and |D| columns, one per document, and the cell in row w and column d counts how often w occurs in d. Each column is then a list of |V| numbers: a vector. As You Like It becomes [1, 114, 36, 20], a point in a 4-dimensional space whose axes are the words. This is the basic move of Vector semantics, applied first to documents rather than words.

Notice what was thrown away. The counts do not record where in the play a word appeared, or what came before it. A document reduced to word frequencies with order discarded is a bag of words, and it is the oldest representation in information retrieval. It sounds crude, but for the question "what is this document about?" it is surprisingly strong, because topic is carried mostly by which words occur and how often.

Seeing the vectors

Four dimensions cannot be drawn, so pick two: fool on the horizontal axis and battle on the vertical one. The comedies become long arrows lying almost flat along fool (As You Like It at [36, 1], Twelfth Night at [58, 0]), while the histories become short arrows pointing up along battle (Julius Caesar at [1, 7], Henry V at [4, 13]). Comedies point one way, histories another. That is the whole promise of the vector view: similar documents have similar columns, so they point in similar directions.

Gerard Salton turned this picture into the vector space model of information retrieval: represent every document and every query as a vector of term weights, measure how close the query vector is to each document vector, and return the closest documents first (Salton, Wong and Yang 1975). A web search for "battle" is, in this model, a vector with a single non-zero entry, and Henry V wins because its column leans hardest in that direction. The measure of closeness is the cosine you will meet in concept 4. Here is a preview of what it says about the plays, computed once on the two plotted dimensions and once on all four.

On the plane, the pattern is crisp: comedies with comedies near 1, comedy with history near 0.15. On all four dimensions every pair lands between about 0.81 and 0.999, and As You Like It looks 0.945 similar to Julius Caesar. The culprit is good: it is frequent in every play, so it dominates every vector and makes all of them point roughly the same way. A word that occurs everywhere carries no information about which document you are in, and the next part's tf-idf exists precisely to turn its weight down.

Turn the same Shakespeare table sideways and read it by rows. battle becomes [1, 0, 7, 13]: a word that shows up a little in the comedies and a lot in Julius Caesar and Henry V. fool becomes [36, 58, 1, 4], the opposite profile. Without any dictionary, the rows already say that battle is a history word and fool is a comedy word, and two words with similar rows occur in similar documents.

Counting neighbors instead of documents

Documents are a coarse unit of context. A play contains thousands of words, so two words sharing a play says little more than that they share a topic. The finer alternative is to count, for each target word, which words appear right next to it. Slide the target through a large corpus, and every time it occurs, look at the words within a fixed window on each side (SLP uses ±4) and add 1 to the cell for each of them.

The result is a Term-context matrix, also called a word-word or word-context matrix. It is square, |V| x |V|: rows are target words, columns are context words, and the cell (w, c) counts how often c appeared inside the window around w. Here are four rows from Wikipedia counts, restricted to five of the context columns.

The rows sort themselves into two families. cherry and strawberry have their mass in pie and sugar; digital and information have theirs in computer and data. Plot digital at [1683, 1670] and information at [3982, 3325] on the data and computer axes and the two arrows point almost the same way, about 5° apart, even though information is more than twice as long. That is the Distributional hypothesis made concrete: two words are similar when their context vectors are similar. Each row is an Embedding of its word, a sparse one built by counting, which later parts will reweight and then replace with short dense vectors.

Ahead of concept 4, here is what the cosine, a 0 to 1 score of how closely two rows point the same way, says about these rows.

Two practical facts follow from the shape. First, almost every cell is zero: most of the |V| words never appear within four words of cherry. These are sparse vectors, and real systems store only the non-zero entries (a compressed sparse row format in SciPy, for example). Second, |V| is not the full vocabulary of the corpus. SLP notes that it is usually the 10,000 to 50,000 most frequent words, and that keeping more than about 50,000 rarely helps (SLP 5.3).

Which word is closer to fool: good or wit? Multiply the Shakespeare rows entry by entry and add. For good and fool that gives 9162; for fool and wit only 1604. By this measure good is almost six times closer to fool than wit is, which is clearly wrong: wit and fool are both comedy words, while good is just everywhere.

The measure is the Dot product, the most natural way to compare two vectors:

It is large when both vectors have large values in the same dimensions, so it does behave like a similarity measure. Vectors whose non-zero entries sit in different dimensions get 0: they are orthogonal, which for count vectors means the two words never share a context. The trouble is that a product of two numbers grows with either number. The dot product therefore depends on how big the vectors are, and the size of a vector is its Vector length:

The general lesson is SLP's: the dot product favors long vectors, and more frequent words have longer vectors, because they co-occur with many words and do so many times. Words such as of, the and you would therefore come out as the nearest neighbor of almost everything. Information retrieval met the same problem with documents: a long document has larger counts everywhere, and a raw dot product with a query rewards it for being long rather than for being on topic (Manning, Raghavan and Schütze, "Dot products").

Run an experiment on digital. Suppose the corpus were twice as large, so every count in its row doubles, from [5, 1683, 1670] to [10, 3366, 3340] over (pie, data, computer). Its dot product with information doubles too, from 12,254,481 to 24,508,962. Nothing about the meaning of digital changed, yet the similarity score did. The cure is to measure the angle between the vectors rather than their overlap, and the angle does not move at all.

Geometry supplies the formula. For any two vectors, the dot product equals the product of their lengths times the cosine of the angle θ between them. Solve for the cosine and you get Cosine similarity:

The middle form is the useful one to remember. Dividing a vector by its own length gives a unit vector, a vector of length 1 pointing the same way: [3, 4] has length 5, so its unit vector is [0.6, 0.8], and 0.6² + 0.8² = 1. The cosine is simply the dot product of the two unit vectors. Length has been removed before the comparison, so only direction is left.

Three properties follow. The range is -1 to 1: 1 for vectors pointing the same way, 0 for orthogonal vectors, -1 for opposite directions. Scaling is invisible: for any c > 0, cos(cv, w) = cos(v, w), because the c appears once in the numerator and once in |cv| = c|v| and cancels (SLP exercise 5.2). And for raw counts the range shrinks to 0 to 1, because no count is negative.

This is also why real systems often skip the division at query time. If every stored vector is normalized to length 1 once, in advance, then a plain dot product is the cosine. scikit-learn documents exactly this: cosine_similarity is the normalized dot product, and on L2-normalized data it is equivalent to linear_kernel. Vector databases that offer an "inner product" metric rely on the same identity, and it is why the embedding vectors of many retrieval models come out already normalized.

Now do the whole computation once by hand, on three words and three context dimensions (pie, data, computer): cherry [442, 8, 2], digital [5, 1683, 1670] and information [5, 3982, 3325]. The question is which of cherry and digital is closer to information under the cosine.

The picture makes the numbers obvious. Plot the words on two of the dimensions, pie upward and computer to the right. cherry stands nearly upright, because nearly all of its mass is in pie. digital and information both lie almost flat along computer. The angle between cherry and information is close to a right angle, so its cosine is close to 0; the angle between digital and information is a sliver, so its cosine is close to 1.

Build a Term-context matrix from a large corpus and look at the row for apricot. The context sugar has a healthy count there, and that is genuinely useful: apricots are sweet things you cook with sugar, and a word whose row also has a high sugar count, such as peach, is probably similar. Now look further along the same row. The contexts the, it and they have counts many times larger than sugar. They have similarly huge counts in the row for digital, for information, for every word in the vocabulary.

That is the paradox. Frequency is clearly informative, since co-occurring often is exactly how sugar earns its place. But frequency is not proportional to information. A Dot product sums products dimension by dimension, so a dimension where both vectors carry a count in the thousands swamps a dimension where both carry a count of twenty. Even after normalising by length, Cosine similarity on raw counts mostly measures how much two words share the ubiquitous dimensions, and every word shares those. What we want instead is a weight with two pressures in it: reward a term that is frequent here, and penalise a term that is frequent everywhere.

Two classic reweightings

Which reweighting you pick depends on which matrix you have. For a Term-document matrix, where columns are documents, the standard answer is tf-idf. Jurafsky and Martin describe it as "the product of two terms, the term frequency tf and the inverse document frequency idf", and note that the "-" is a hyphen, not a minus sign. The first factor rewards local frequency; the second penalises spread across the collection.

For a term-term matrix, where both rows and columns are words, the standard answer is Pointwise mutual information. It compares how often two words actually appear together with how often they would if they were independent. Words like good and great earn a high PMI only if they meet more often than their individual frequencies predict, so a context like the that meets everything at the expected rate scores near zero. Part 06 develops PMI in full.

The idea behind the idf half is older than vector semantics. Karen Sparck Jones argued in 1972 that "matches on less frequent, more specific, terms are of greater value than matches on frequent terms", and proposed weighting each term by how few documents it appears in. Every term-weighting scheme in this lecture, PPMI included, is a variation on her insight.

Suppose a word appears 0, 1, 9, 99 or 999 times in a document. Under the slides' formula those counts become a term frequency of 0, 0.301, 1, 2 and 3. A hundredfold increase in occurrences, from 9 to 999, adds only 2 to the weight.

The simplest term frequency would be the raw count itself, tf = count(t, d). The trouble is that significance does not grow linearly with repetition. Manning, Raghavan and Schütze put it plainly: "It seems unlikely that twenty occurrences of a term in a document truly carry twenty times the significance of a single occurrence." The second mention of battle in a play tells you a lot (this play has a battle in it); the hundredth mention adds far less. So we squash the count with a logarithm.

The +1 is there because log 0 is undefined. Adding one before taking the log sends a count of zero to log10(1) = 0, which is exactly the weight an absent term should have, while barely changing large counts.

The other formula you will meet

The current SLP3 draft (now Chapter 11, on retrieval and RAG) and the IR book both use a slightly different squash. SLP3 notes in a footnote that log10(count + 1) is "this alternative formulation" used in its earlier editions, which is where the slides come from. Neither is an error; they are two members of the same sublinear family.

The two agree on the shape but not the numbers. Under the variant, a count of 1 gives 1 + log10 1 = 1 rather than 0.301, a count of 10 gives 2 rather than 1.041, and a count of 7 gives 1.845 rather than 0.903. scikit-learn offers a third version through sublinear_tf=True, which uses 1 + ln(tf) with the natural log.

Across the complete works of Shakespeare, the words Romeo and action each occur exactly 113 times. By raw volume they are identical. Yet every occurrence of Romeo sits in one play, Romeo and Juliet, while action is spread across 31 different plays. If you are handed a document and told it contains Romeo, you know which play it is. If you are told it contains action, you have learned almost nothing.

The two numbers in that story have names. The total number of times a term occurs across the whole collection is its Collection frequency, cf. The number of documents that contain the term at least once is its Document frequency, df_t. The IR book defines them exactly that way and concludes that for discriminating between documents it is "better to use a document-level statistic ... than to use a collection-wide statistic". SLP3 gives the reason: "Terms that occur in only a few documents are useful for discriminating those documents from the rest of the collection."

The Shakespeare pair is not a curiosity. The IR book finds the same pattern in the Reuters newswire collection, where try and insurance have almost the same collection frequency but very different document frequencies. Insurance clusters in the articles that are actually about insurance; try is sprinkled through everything.

Treat each of Shakespeare's 37 plays as a document and walk down a ladder of words ordered by how many plays they appear in. Romeo is in one play, salad in two, Falstaff in four, forest in twelve, battle in twenty-one, wit in thirty-four, fool in thirty-six, and good and sweet in all thirty-seven. We want a weight that is large at the top of the ladder and vanishes at the bottom.

The ratio N / df_t of collection size to Document frequency does that: it is 37 for Romeo and 1 for good. Raw ratios grow too fast, though (a word in one document out of a million would get a ratio of a million), so we take the logarithm, exactly as we did for tf. The result is the inverse document frequency.

Two boundary values are worth memorising. The largest possible idf is log10 N, reached when a term occurs in a single document: log10 37 = 1.57 here. The smallest is 0, reached when a term occurs in every document, since log10(N / N) = log10 1 = 0. SLP3 says it directly: the lowest weight, 0, is "assigned to terms that occur in every document", words "like good or sweet". Notice that idf is not linear in df. Going from df = 1 to df = 2 costs 0.30, the same as going from df = 12 to df = 24: halving the spread always buys the same amount of weight.

What counts as a document?

Nothing in the formula says a document must be a play. SLP3 defines a document as "whatever unit of text the system indexes and retrieves (web pages, scientific papers, news articles, or even shorter passages like paragraphs)". It could be a Wikipedia article, a tweet, a paragraph, or a 300-token chunk in a RAG index. The choice is a modelling decision, and it changes everything downstream: N changes, every df changes, and so every idf changes.

Switch Shakespeare from plays to paragraphs and N grows into the thousands. good is no longer in every document, because most paragraphs do not contain it, so it gets a positive idf. A word like battle that clusters in a few scenes now looks rarer relative to the collection, and its weight rises. The right unit is the one that matches what you retrieve or compare: if your system returns passages, compute df over passages.

Take battle in Julius Caesar. The play uses the word 7 times, so its term frequency is log10(7 + 1) = log10 8 = 0.903. battle appears in 21 of the 37 plays, so its idf is 0.246. Multiply: 0.903 × 0.246 = 0.222, which the slide prints as 0.22. That is one cell of the tf-idf matrix, and every other cell is computed the same way.

The rule is just the definition applied cell by cell: squash the count into Term frequency, look up the word's Inverse document frequency, multiply. The table below combines the raw counts on slide 52 with the idf column of slide 50 and shows the intermediate tf values the slide skips. All sixteen weights agree with the slide.

What the table teaches

Look at the good row. It has the largest counts in the table, from 62 to 114, and its tf values are all around 2. Yet every tf-idf cell is 0, because good occurs in all 37 plays and its idf is log10 1 = 0. The word that dominated the raw vectors has been removed from the comparison without anyone writing a stop list.

Now look at battle. In raw counts it was a minor dimension, dwarfed by good in every play and by fool in the comedies. After weighting it is the largest value in every column where it appears, and it cleanly separates Julius Caesar (0.22) and Henry V (0.28), plays full of war, from the comedies As You Like It (0.074) and Twelfth Night (0). The IR book summarises the pattern: a weight is highest when a term occurs many times in a small number of documents, lower when it occurs fewer times or in many documents, and lowest when it occurs in virtually all documents.

The resulting document vectors are still sparse vectors of vocabulary length, mostly zeros, and you compare them with Cosine similarity exactly as in part 04. The difference is that the dimensions now carry information in proportion to how distinctive they are.

tf-idf in real systems

The formula is the same everywhere, but the details are not, and the differences matter when you reproduce a paper or debug a pipeline. scikit-learn's TfidfVectorizer by default uses raw counts for tf, a smoothed natural-log idf, ln((1 + n) / (1 + df)) + 1, and then L2-normalises each document vector. Because of the +1, a term in every document gets idf 1, not 0, so good would survive. The documentation's own example turns the vector [3, 0, 1.8473] into [0.8515, 0, 0.5243] after normalisation.

BM25 is the modern member of the family and the standard keyword retriever in search engines and hybrid RAG systems. It keeps idf but replaces log tf with a saturating function controlled by a parameter k, usually between 1.2 and 2, and normalises for document length with b = 0.75. Salton and Buckley's 1988 comparison of term-weighting schemes is the classic study of why these choices matter.

Start with numbers. In the small Term-context matrix used throughout this part, the word information accounts for 7703 of the N = 11716 counted word-context pairs, and the context data for 5673 of them. If the two words had nothing to do with each other, how often would you expect to see them together? The full table, with every row and column sum, is in the counts-to-probabilities section below.

Independence answers that. If knowing one word tells you nothing about the other, the probability of the pair is just the product of the separate probabilities: P(information) × P(data) = .6575 × .4842 = .3184. So about 31.8% of all pairs should be (information, data) by chance alone. The observed share is 3982 / 11716 = .3399. The ratio of observed to expected is 1.0676: the pair occurs only 6.8% more often than chance, and log2 1.0676 = .0944 bits. Compare cherry and pie: there the ratio is 20.8, and the log is 4.38 bits. That number is pointwise mutual information.

Read the fraction one piece at a time. The numerator is what actually happened: how often the pair appeared together. The denominator is what independence predicts, because under independence P(x, y) = P(x) P(y). The ratio therefore says how many times more often the pair occurs than chance, and the base-2 logarithm turns that ratio into bits. A ratio of 1 gives 0 bits (independent), a ratio above 1 gives a positive score (attraction), and a ratio below 1 gives a negative score (avoidance). Each extra bit means the pair is twice as over-represented.

Where the measure comes from

Church and Hanks (1989, journal version 1990) brought the measure to lexicography. They estimated P(x, y) by counting how often x is followed by y within a window of w = 5 words, and they read the score exactly as above: well above zero for genuine association, near zero for no relation, well below zero for words in complementary distribution. Their rough rule of thumb was that pairs with a score above 3 tend to be interesting, and they ignored pairs seen 5 times or fewer because the ratio was unstable, an early sign of the rare-event problem later in this part. One subtlety a PhD reader should notice: because their count encoded word order (x before y), their association ratio was not symmetric. The term-context version on these slides counts a symmetric window, so PMI(w, c) depends only on which pair you pick.

Suppose two words each have probability 10^-6, which is typical of most of the vocabulary. Independence predicts that they appear together with probability 10^-12. To say with confidence that they appear together less than that, you must be able to estimate probabilities well below 10^-12, which takes a corpus on the order of trillions of pairs. With a corpus of a few million tokens you will simply never see them together, and you cannot tell "these words avoid each other" apart from "these words never happened to meet".

That is the first reason negative PMI is unreliable: the evidence needed to estimate below-chance co-occurrence grows with the rarity of the words, and for most word pairs it is never available. The second reason is about evaluation: it is not clear that people can even judge "unrelatedness" reliably, so there is no good gold standard to check negative scores against. The practical answer, used since Church and Hanks and later Dagan and colleagues (1993) and Niwa and Nitta (1994), is to keep only the positive side.

Positive PMI keeps the attraction signal and floors everything else at zero. It has a welcome side effect. A pair that never co-occurs has P(w, c) = 0, so its Pointwise mutual information is log2 0 = -∞, a value that would wreck any later arithmetic. Clipping maps it cleanly to 0. In the slide table this is exactly what happens to strawberry/computer and strawberry/data, both with count 0. The result is a sparse vector per word: long, mostly zero, with a few positive entries for the contexts that really characterise it.

Empirically the choice pays off. Levy, Goldberg and Dagan (2015) report that Bullinaria and Levy (2007) found PPMI outperforms plain PMI on semantic similarity tasks, and PPMI became the default count-based baseline against which neural embeddings were later compared.

Everything so far needs three probabilities per cell: the joint P(w, c) and the two marginals P(w) and P(c). All three come from one count matrix F. Here is the matrix the slides use, taken from SLP3, whose counts come from Wikipedia: four target words as rows, five context words as columns, with the row and column sums added.

Add every cell and you get the grand total N = 11716. Each probability is then a share of that one total. The joint probability of a cell is the cell divided by N. The marginal probability of a target word is its row sum divided by N, and the marginal probability of a context is its column sum divided by N.

Because the joint and both marginals come from the same matrix and the same N, they are consistent: each row of joint probabilities sums to its row marginal, each column to its column marginal, and everything sums to 1. SLP3 is candid that this "pretends" the five listed contexts are the only ones in the world. In a real system the matrix has tens of thousands of columns and the same recipe applies unchanged.

With the three probabilities in hand, one cell of the Pointwise mutual information matrix is a division and a log. Work it once by hand for information/data, then notice a shortcut that skips the rounding entirely.

Repeat for every cell and you get the full PMI matrix below. Every Positive PMI value on slide 58 is this matrix after clipping; SLP3 quotes PMI(cherry, computer) = -6.7 in its figure caption as an example of a large negative value that PPMI discards.

Now read the PPMI matrix as a set of word vectors. The fruit rows light up only on pie and sugar; the technology rows only on computer, data and result. The two groups share no non-zero dimension, so the cosine similarity between cherry and digital is exactly 0, while cherry and strawberry have a cosine of about 0.96. The raw counts were noisier: cherry co-occurs with computer and data a few times, and in a full-size matrix such incidental counts, scaled by frequent contexts, blur every row. PPMI vectors are still long and sparse; the second half of this lecture learns short dense vectors instead.

Add one more row and one more column to the slide table: a word w and a context c, each seen exactly once, and that once together. The shortcut gives PMI = log2(1 × 11716 / (1 × 1)) = log2 11716 = 13.5 bits. That is more than double strawberry/sugar (5.51), from a single observation that may be a typo or a coincidence. One accident beats thousands of genuine co-occurrences.

The mechanism is the denominator. When a context is rare, P(c) is tiny, so even one co-occurrence produces a large ratio. Levy, Goldberg and Dagan (2015) call this Pointwise mutual information's Achilles' heel, citing Turney and Pantel (2010): a word's highest-scoring dimensions become obscure contexts it met once or twice, and since similar words rarely share those accidents, their vectors look less alike under cosine than they should. Clipping does not help, because Positive PMI only touches negative values and this bias inflates positive ones.

Two fixes

1. Give rare contexts a little more probability. If P(c) for a rare context is nudged upward, its PMI falls. This is the alpha weighting of the next concept.
2. Add-k smoothing. Add a small constant k to every count before computing probabilities, exactly as in the n-gram language models of Lecture 03. Slide 59 names the simplest case, add-one smoothing, which is k = 1 and lowers strawberry/sugar from 5.51 to 5.41. SLP3 gives k = 0.1 to 3 as common choices, and notes that the larger the k, the more the non-zero counts are discounted. A count of 1 becomes 3 with k = 2, tripling, while a count of 442 barely moves. On the slide table, add-2 lowers strawberry/sugar from 5.51 to 5.31 and leaves the overall pattern intact.

The oldest fix is the bluntest. Church and Hanks simply discarded pairs seen 5 times or fewer. Frequency cut-offs of that kind are still common in collocation tools.

Take two contexts with P(a) = .99 and P(b) = .01. Raising both to 0.75 and renormalising roughly triples the rare one, so every Pointwise mutual information involving b falls by about 1.6 bits. The worked example below shows each step.