What words mean: lemmas, senses and synonymy

Open the vocabulary file of an n-gram model. cat might be entry w_412, dog entry w_977 and spreadsheet entry w_3051. Ask the model which of those three words are alike and it has nothing to say. The numbers are positions in a list, and a list position carries no meaning: 412 is not closer in sense to 977 than to 3051.

Turning each index into a vector does not help. A one-hot vector has a single 1 at the word's index and zeros everywhere else. Two distinct words never share a non-zero position, so their dot product is always 0, whichever pair you pick:

Jurafsky and Martin describe exactly this situation: in the n-gram models of Chapter 3 and in classical NLP applications, the only representation of a word is a string of letters or an index in a vocabulary list. A model trained on the cat sat learns nothing about the dog sat, because the two sentences share no symbol in the position that matters.

The logic class answer, and why it is circular

An introductory logic course offers a different answer: the meaning of dog is the predicate DOG, and the meaning of cat is CAT. Relations between meanings are then stated as axioms written by hand, for instance that every dog is a mammal:

This is more expressive than an index, because the axioms support inference. But the symbol itself still says nothing; it is the word in capital letters. The old semantics joke makes the point. Q: What is the meaning of life? A: LIFE. Jurafsky and Martin attribute it to the semanticist Barbara Partee and call capitalization a pretty unsatisfactory model of meaning. Every relation you want, from similarity to connotation, has to be typed in by a person, and nothing about DOG tells you that it should sit near CAT.

The rest of the lecture fills the third row. Vector semantics represents a word as a point in a space built from the contexts it appears in, and the resulting Embedding makes closeness a number you can compute instead of an axiom someone must write.

Consider three sentences: Ann bought a car from Bo. Bo sold Ann a car. Ann paid Bo for a car. One event, one car, one transfer of money, described from three positions. Any adequate account of word meaning has to know that buy, sell and pay are tied together this way, and none of the symbol representations from the previous concept does.

Lexical semantics is the linguistic study of word meaning. It is not the study of dictionary definitions one entry at a time; it is the study of how meanings relate to each other. Jurafsky and Martin turn its findings into a list of what a model of word meaning should deliver, and this list is the checklist every later model in the lecture is graded against.

The first three entries get their own treatment in part 02: similarity as a graded human judgment, Antonymy as opposition on a single feature, and Connotation as affective meaning. Perspective and inference are what make the list more than a thesaurus: a question answering system that is asked who bought the car must connect it to a sentence that only says who sold it.

Keep the list in view as the lecture moves on. Sparse count vectors will turn out to be good at similarity and relatedness and weak at antonymy, because hot and cold occur in the same contexts. Word2vec will improve similarity further and add analogies, yet still give one vector to every sense of a word. Each model earns or loses marks on these rows.

Type mouse into WordNet. The slide quotes two of its meanings: any of numerous small rodents, and a hand-operated device that controls a cursor. The full WordNet 3.0 entry has six: four noun senses and two verb senses. Before naming the parts of this entry, look at how much a single spelling has to carry.

from nltk.corpus import wordnet as wn

for synset in wn.synsets("mouse"):
    print(synset.name(), synset.lemma_names())

Lemma and wordform

The headword mouse is a Lemma, also called the citation form: the form under which a dictionary lists the word. The plural mice has no entry of its own; it is a wordform of the same lemma. A wordform is any inflected form a lemma takes in running text. The same split holds across verbs and languages, and it is the same lemma idea you met in tokenization and morphology.

Sense and polysemy

Each numbered meaning in the entry is a sense: a discrete aspect of the word's meaning. A lemma with several senses is polysemous, and the phenomenon is called Polysemy. WordNet groups senses into synsets, sets of near-synonymous senses that share one gloss and express one concept. That is why the black eye sense appears under the name shiner.n.01: its synset is {shiner, black_eye, mouse}, and WordNet names a synset after its first member.

How do you know two meanings really are separate senses? One practical test is the zeugma. In ?Does Air France serve breakfast and Philadelphia? the two uses of serve (providing food and flying to a city) are forced to share one verb, and the sentence sounds like a pun. That oddness is the evidence for two senses. With one sense, coordination is fine: Air France serves breakfast and lunch.

Why the split matters for systems

Jurafsky and Martin point out that a search for mouse info is ambiguous between a pet owner and a shopper. Deciding which sense a given occurrence uses is the task of word sense disambiguation. And the split sets up a limitation you will meet at the end of this lecture: a Static embedding gives each word type one vector, so the vector for mouse must blend the rodent and the device. A Contextual embedding computes a different vector for each occurrence, which is how modern models separate senses.

couch and sofa. filbert and hazelnut. car and automobile. vomit and throw up. big and large. Each pair can be swapped in some sentence without anyone noticing a change in what is claimed. That intuition has a precise form.

Two words are synonymous if they can be substituted for each other in any sentence without changing the truth conditions of the sentence, that is, the situations in which the sentence would be true. This is Synonymy in its truth-conditional definition. I sat on the couch and I sat on the sofa are true in exactly the same situations. WordNet even files car and automobile in the same synset, car.n.01.

The slide states the same idea more loosely: synonyms have the same meaning in some or all contexts. All contexts is the strict truth-conditional ideal. Some contexts is what real pairs such as big and large achieve, and that gap is exactly why synonymy is stated between senses rather than words.

The twist: the relation is between senses

Try the substitution with big and large. In Would I be flying on a large or small plane? the swap to big is harmless. In Miss Nelson became a kind of big sister to Benjamin it is not: a large sister is a different claim. The textbook draws the conclusion directly. Synonymy is a relationship between senses rather than words. WordNet makes this visible: big has 17 synsets and large has 11. They share some, such as large.a.01, above average in size, and not others, such as big.s.01, significant. A claim that two words are synonyms is really a claim about one of their senses.

water and H₂O refer to the same substance. Swap them in the glass contains water and the sentence stays true in exactly the same situations. Now imagine a hiking guide that says to carry two litres of H₂O per person. Nothing false was said, and yet the sentence is wrong for its setting. That wrongness is part of what the words mean.

The same thing happens with big and large: even setting aside the older-sibling sense, the pairs that pass the truth test still differ somewhere. Jurafsky and Martin put it carefully: while substitutions between some pairs of words like car and automobile or water and H₂O are truth preserving, the words are still not identical in meaning, and probably no two words are absolutely identical in meaning.

The principle of contrast

The generalization behind this is the Principle of contrast: a difference in linguistic form is always associated with some difference in meaning. The idea has a long history, from Girard in 1718 and Bréal in 1897 to Eve Clark in 1987, who stated it as: every two forms contrast in meaning. Clark used it to explain language acquisition: a child who already knows one word for a thing assumes a new word for the same thing must mean something different. In her words, there are no true synonyms.

Where do apparent synonyms differ, if not in truth? Clark names three dimensions, and the slide's list of politeness, slang, register and genre maps onto them.

• Dialect. autumn and fall, truck and lorry, tap and faucet: the choice tells the listener where the speaker is from.
• Register. die, pass away and pop off; attempt and try. A register is a speech style such as formal, colloquial or technical. Politeness and slang sit here, and so does genre: H₂O is the technical register of a chemistry text.
• Connotation. politician and statesman, skinny and slim: the referent can be the same while the attitude differs. This is Connotation, which part 02 measures.

The consequence for the rest of the course is terminological: when NLP papers say synonym, they mean approximate synonymy, two senses close enough to substitute in most contexts. It also has an engineering consequence. Query expansion in search, paraphrase generation and data augmentation by synonym replacement all assume substitutability, and the principle of contrast predicts they will shift register, sense or tone some of the time. A paraphraser that rewrites passed away as died has kept the truth and changed the message.

car and bicycle are not synonyms. Neither are cow and horse. Yet everyone feels they belong together: both pairs share an element of meaning, a vehicle you ride on a road, a large farm animal. Swap them, though, and the truth changes. I took my car to work describes a different morning from I took my bicycle to work.

This relation is Word similarity. Jurafsky and Martin give the reason it matters: while words don't have many synonyms, most words do have lots of similar words. A model that only knew synonymy would have almost nothing to say about most of the vocabulary; a model of similarity has something to say about every word.

There is a second, quieter shift. Synonymy was a relation between senses, which requires deciding first what the senses of every word are. Similarity is usually stated between words, which avoids committing to a sense inventory at all. That is exactly the quantity vector models compute: one number for a pair of words, later the cosine of the angle between their vectors. Part 02 shows how humans rate it, on datasets such as SimLex-999, and how it differs from relatedness.