Weighting with tf-idf

Build a Term-context matrix from a large corpus and look at the row for apricot. The context sugar has a healthy count there, and that is genuinely useful: apricots are sweet things you cook with sugar, and a word whose row also has a high sugar count, such as peach, is probably similar. Now look further along the same row. The contexts the, it and they have counts many times larger than sugar. They have similarly huge counts in the row for digital, for information, for every word in the vocabulary.

That is the paradox. Frequency is clearly informative, since co-occurring often is exactly how sugar earns its place. But frequency is not proportional to information. A Dot product sums products dimension by dimension, so a dimension where both vectors carry a count in the thousands swamps a dimension where both carry a count of twenty. Even after normalising by length, Cosine similarity on raw counts mostly measures how much two words share the ubiquitous dimensions, and every word shares those. What we want instead is a weight with two pressures in it: reward a term that is frequent here, and penalise a term that is frequent everywhere.

Two classic reweightings

Which reweighting you pick depends on which matrix you have. For a Term-document matrix, where columns are documents, the standard answer is tf-idf. Jurafsky and Martin describe it as "the product of two terms, the term frequency tf and the inverse document frequency idf", and note that the "-" is a hyphen, not a minus sign. The first factor rewards local frequency; the second penalises spread across the collection.

For a term-term matrix, where both rows and columns are words, the standard answer is Pointwise mutual information. It compares how often two words actually appear together with how often they would if they were independent. Words like good and great earn a high PMI only if they meet more often than their individual frequencies predict, so a context like the that meets everything at the expected rate scores near zero. Part 06 develops PMI in full.

The idea behind the idf half is older than vector semantics. Karen Sparck Jones argued in 1972 that "matches on less frequent, more specific, terms are of greater value than matches on frequent terms", and proposed weighting each term by how few documents it appears in. Every term-weighting scheme in this lecture, PPMI included, is a variation on her insight.

Suppose a word appears 0, 1, 9, 99 or 999 times in a document. Under the slides' formula those counts become a term frequency of 0, 0.301, 1, 2 and 3. A hundredfold increase in occurrences, from 9 to 999, adds only 2 to the weight.

The simplest term frequency would be the raw count itself, tf = count(t, d). The trouble is that significance does not grow linearly with repetition. Manning, Raghavan and Schütze put it plainly: "It seems unlikely that twenty occurrences of a term in a document truly carry twenty times the significance of a single occurrence." The second mention of battle in a play tells you a lot (this play has a battle in it); the hundredth mention adds far less. So we squash the count with a logarithm.

The +1 is there because log 0 is undefined. Adding one before taking the log sends a count of zero to log10(1) = 0, which is exactly the weight an absent term should have, while barely changing large counts.

The other formula you will meet

The current SLP3 draft (now Chapter 11, on retrieval and RAG) and the IR book both use a slightly different squash. SLP3 notes in a footnote that log10(count + 1) is "this alternative formulation" used in its earlier editions, which is where the slides come from. Neither is an error; they are two members of the same sublinear family.

The two agree on the shape but not the numbers. Under the variant, a count of 1 gives 1 + log10 1 = 1 rather than 0.301, a count of 10 gives 2 rather than 1.041, and a count of 7 gives 1.845 rather than 0.903. scikit-learn offers a third version through sublinear_tf=True, which uses 1 + ln(tf) with the natural log.

Across the complete works of Shakespeare, the words Romeo and action each occur exactly 113 times. By raw volume they are identical. Yet every occurrence of Romeo sits in one play, Romeo and Juliet, while action is spread across 31 different plays. If you are handed a document and told it contains Romeo, you know which play it is. If you are told it contains action, you have learned almost nothing.

The two numbers in that story have names. The total number of times a term occurs across the whole collection is its Collection frequency, cf. The number of documents that contain the term at least once is its Document frequency, df_t. The IR book defines them exactly that way and concludes that for discriminating between documents it is "better to use a document-level statistic ... than to use a collection-wide statistic". SLP3 gives the reason: "Terms that occur in only a few documents are useful for discriminating those documents from the rest of the collection."

The Shakespeare pair is not a curiosity. The IR book finds the same pattern in the Reuters newswire collection, where try and insurance have almost the same collection frequency but very different document frequencies. Insurance clusters in the articles that are actually about insurance; try is sprinkled through everything.

Treat each of Shakespeare's 37 plays as a document and walk down a ladder of words ordered by how many plays they appear in. Romeo is in one play, salad in two, Falstaff in four, forest in twelve, battle in twenty-one, wit in thirty-four, fool in thirty-six, and good and sweet in all thirty-seven. We want a weight that is large at the top of the ladder and vanishes at the bottom.

The ratio N / df_t of collection size to Document frequency does that: it is 37 for Romeo and 1 for good. Raw ratios grow too fast, though (a word in one document out of a million would get a ratio of a million), so we take the logarithm, exactly as we did for tf. The result is the inverse document frequency.

Two boundary values are worth memorising. The largest possible idf is log10 N, reached when a term occurs in a single document: log10 37 = 1.57 here. The smallest is 0, reached when a term occurs in every document, since log10(N / N) = log10 1 = 0. SLP3 says it directly: the lowest weight, 0, is "assigned to terms that occur in every document", words "like good or sweet". Notice that idf is not linear in df. Going from df = 1 to df = 2 costs 0.30, the same as going from df = 12 to df = 24: halving the spread always buys the same amount of weight.

What counts as a document?

Nothing in the formula says a document must be a play. SLP3 defines a document as "whatever unit of text the system indexes and retrieves (web pages, scientific papers, news articles, or even shorter passages like paragraphs)". It could be a Wikipedia article, a tweet, a paragraph, or a 300-token chunk in a RAG index. The choice is a modelling decision, and it changes everything downstream: N changes, every df changes, and so every idf changes.

Switch Shakespeare from plays to paragraphs and N grows into the thousands. good is no longer in every document, because most paragraphs do not contain it, so it gets a positive idf. A word like battle that clusters in a few scenes now looks rarer relative to the collection, and its weight rises. The right unit is the one that matches what you retrieve or compare: if your system returns passages, compute df over passages.

Take battle in Julius Caesar. The play uses the word 7 times, so its term frequency is log10(7 + 1) = log10 8 = 0.903. battle appears in 21 of the 37 plays, so its idf is 0.246. Multiply: 0.903 × 0.246 = 0.222, which the slide prints as 0.22. That is one cell of the tf-idf matrix, and every other cell is computed the same way.

The rule is just the definition applied cell by cell: squash the count into Term frequency, look up the word's Inverse document frequency, multiply. The table below combines the raw counts on slide 52 with the idf column of slide 50 and shows the intermediate tf values the slide skips. All sixteen weights agree with the slide.

What the table teaches

Look at the good row. It has the largest counts in the table, from 62 to 114, and its tf values are all around 2. Yet every tf-idf cell is 0, because good occurs in all 37 plays and its idf is log10 1 = 0. The word that dominated the raw vectors has been removed from the comparison without anyone writing a stop list.

Now look at battle. In raw counts it was a minor dimension, dwarfed by good in every play and by fool in the comedies. After weighting it is the largest value in every column where it appears, and it cleanly separates Julius Caesar (0.22) and Henry V (0.28), plays full of war, from the comedies As You Like It (0.074) and Twelfth Night (0). The IR book summarises the pattern: a weight is highest when a term occurs many times in a small number of documents, lower when it occurs fewer times or in many documents, and lowest when it occurs in virtually all documents.

The resulting document vectors are still sparse vectors of vocabulary length, mostly zeros, and you compare them with Cosine similarity exactly as in part 04. The difference is that the dimensions now carry information in proportion to how distinctive they are.

tf-idf in real systems

The formula is the same everywhere, but the details are not, and the differences matter when you reproduce a paper or debug a pipeline. scikit-learn's TfidfVectorizer by default uses raw counts for tf, a smoothed natural-log idf, ln((1 + n) / (1 + df)) + 1, and then L2-normalises each document vector. Because of the +1, a term in every document gets idf 1, not 0, so good would survive. The documentation's own example turns the vector [3, 0, 1.8473] into [0.8515, 0, 0.5243] after normalisation.

BM25 is the modern member of the family and the standard keyword retriever in search engines and hybrid RAG systems. It keeps idf but replaces log tf with a saturating function controlled by a parameter k, usually between 1.2 and 2, and normalises for document length with b = 0.75. Salton and Buckley's 1988 comparison of term-weighting schemes is the classic study of why these choices matter.