Vector semantics and the distributional hypothesis

Take two words, "oculist" and "eye-doctor". Collect every sentence each appears in, and look at the neighbors: eye, examined, prescription, glasses, appointment. The two lists are almost the same. Now compare "oculist" with "lawyer". Some neighbors still overlap (appointment, fee, office), but far fewer. Without a dictionary, the overlap of environments already tells you which pair is closer in meaning.

That observation is the distributional hypothesis. Zellig Harris put it in 1954 using exactly this example: if two words have almost identical environments, meaning neighboring words or the grammatical frames they occur in, we call them synonyms. He then made the claim graded. The difference in meaning between two words corresponds roughly to the amount of difference in their environments. That second sentence matters more than the first, because it turns meaning into a quantity you can estimate by counting.

The idea was in the air in the 1950s. Wittgenstein had argued that, for a large class of cases, the meaning of a word is its use in the language. Joos (1950) described the meaning of a morpheme as the set of conditional probabilities of its occurrence alongside every other morpheme, which is almost a definition of a language model. Firth (1957) gave the line everyone quotes: "You shall know a word by the company it keeps." Firth and Harris are often merged, but they meant different things. Firth cared about situational and cultural context; Harris cared about the formal distribution of words inside text. NLP took Harris's version, because it can be computed from a corpus alone.

This is why the lecture moves to vector semantics. Earlier, lexical semantics gave us a list of relations a good model should respect: synonymy, similarity, relatedness, connotation. Writing those relations down by hand for every pair of words is impossible. The distributional hypothesis says you do not have to: read enough text, record each word's environments, and the relations fall out of the overlaps. Vector semantics is the standard way NLP does this today.

Suppose you have never seen the word "ongchoi", a recent borrowing into English from Cantonese, and you meet it three times: ongchoi is delicious sauteed with garlic; ongchoi is superb over rice; ongchoi leaves with salty sauces. You do not know what it is yet. But you have read plenty of other sentences: spinach sauteed with garlic over rice, chard stems and leaves are delicious, collard greens and other salty leafy greens.

Put the contexts side by side and the answer is hard to miss. Sauteed, garlic, rice, leaves, delicious and salty all show up around ongchoi and around the leafy greens you already know. Nothing about laptops, contracts or weather. So ongchoi is very likely a leafy green that people cook and eat. It is: the plant is Ipomoea aquatica, a relative of morning glory sometimes called water spinach, and it has other names in Chinese, Malay and Vietnamese.

This is the computational form of the distributional hypothesis. Define the meaning of a word by its distribution, the neighboring words or grammatical environments it appears in, and then do the obvious thing: count the words in the context of ongchoi and compare those counts with the counts for every other word. A table of such counts, one row per word and one column per context word, is the term-context matrix of part 4, and each row is a word's vector.

Here are two words with three numbers each. Heartbreak is [2.45, 5.65, 3.58] and courageous is [8.05, 5.5, 7.38]. The first number is valence (how pleasant), the second is arousal (how intense the emotion), the third is dominance (how much control is exerted). Plot both as points. They sit far apart on valence and dominance, and almost level on arousal: both words are emotionally charged, one pleasantly and with control, the other unpleasantly and with loss of control.

The idea of placing a word at a point goes back to Charles Osgood and colleagues in 1957. Osgood asked people to rate words on many bipolar scales, such as good to bad, strong to weak, active to passive, a method he called the semantic differential. Factor analysis showed that most of the variation came from three factors, which he named evaluation, potency and activity. Today the same three are usually called valence, dominance and arousal, the VAD dimensions of a word's connotation from part 2. Osgood noticed that three numbers per word make each word a point in a three-dimensional space, and he proposed that similarity of meaning is nearness in that space. That is the first appearance of vector semantics.

Put the two threads together. Idea 1 is that meaning can be defined by linguistic distribution. Idea 2, from Osgood, is that meaning can be a point in a multidimensional space. Vector semantics joins them: represent each word as a point, but let the word's distribution, not a panel of human raters, decide where the point goes.

Look at a map of word vectors squashed onto a page. Good, nice, wonderful, fantastic, amazing and very good huddle in one region. Bad, worst, worse, dislike and not good gather in another. Function words such as to, by, that, is and with sit off by themselves. Nobody placed these words by hand. Their positions came from training on text.

This is vector semantics in its working form. Each word is a vector, a list of numbers, rather than an arbitrary symbol such as the string "good" or the index w45. Similar words end up nearby in what is called semantic space. And the space is built automatically, by seeing which words are nearby in text, so the distributional hypothesis does the placing that Osgood's raters used to do.

Such a vector is called an embedding, because the word is embedded into a space. The term began in the latent semantic analysis community in the late 1990s, where it named the mapping from sparse count space into a smaller dense space, and it later shifted to mean the resulting vector itself. Embeddings are now the standard way to represent word meaning in NLP: practically every modern system, from classifiers to large language models, starts by looking up or computing them. They give a fine-grained model of similarity, a number for every pair of words instead of a yes or no.

Build a sentiment classifier the traditional way. Feature 5 is "the previous word was terrible", and training learns that it signals a negative review. At test time a review says "awful acting". If awful never appeared in the labeled training data, feature 5 does not fire, no other feature knows about awful, and the classifier has nothing to go on.

Now replace the identity feature with the previous word's embedding. During training the input was terrible's vector, say [35, 22, 17, ...], and the classifier learned weights on those coordinates. At test time awful arrives as [34, 21, 14, ...]. The weights do not care whether the word is the same string; they act on the numbers, and these numbers are nearly the same. The classifier treats awful almost exactly as it treated terrible. It has generalized to a similar but unseen word.

This matters because of what lecture 2 showed about vocabularies. Zipf's law means most word types are rare, and about half of them appear only once. A classifier trained on a few thousand labeled reviews will meet many words at test time that it never saw with a label. Embeddings, learned from billions of unlabeled words, carry knowledge about those words into the classifier. Dense vectors also need far fewer weights, around 300 per input position instead of one per vocabulary word, and they can place car and automobile close together, which separate identity features never can.

Picture two vectors for ongchoi. The first has one slot per vocabulary word, tens of thousands of them, and stores how often each word appeared near ongchoi. Garlic, rice and leaves have counts; nearly every other slot is 0. The second has about 300 real numbers, almost none of them zero, some negative, and none of them labeled with a word. Both are embeddings. This lecture covers both families.

Sparse vectors: weighted counts

A sparse vector is built from a simple function of counts. With tf-idf, the classic weighting from information retrieval, the counts come from documents. With PPMI, they come from nearby words, re-weighted so that informative co-occurrences stand out. These vectors are the workhorse of search engines and a strong, cheap baseline in almost any text task. Their dimensions are interpretable, because each one is a specific word or document.

Dense vectors: learned by prediction

A dense vector such as one from word2vec is learned instead of counted. Word2vec trains a simple classifier to predict whether a word is likely to appear near a target word, and keeps the classifier's weights as the embedding. The labels come free from running text, which is self-supervision. Both families give one fixed vector per word type, a static embedding. Later in the course, contextual embeddings such as BERT compute a different vector for each occurrence of a word.