← All concepts

Probabilistic Reasoning

Naive Bayes

Classification with Bayes' rule under the assumption that features are conditionally independent given the class.

The marker starts at the prior. Each known word in the email pushes it by its log-likelihood ratio: red toward spam, blue toward ham. Where it ends up is P(spam | email).

Known words (tap to add to the email):

Log-odds tug of warArrows for each word move a marker toward spam or ham.← hamspam →prior −0.41urgent +1.61you +0.24winner +3.18click +1.83today +0.29free +2.30prize +3.28money +2.01P(spam) = 100.0%0.0%0.2%2%12%50%88%98%99.8%100.0%Axis: log-odds, labelled as P(spam)

Prior log-odds

−0.41

Sum of word evidence

+14.75

Posterior P(spam | email)

100.0%

Verdict

Spam

log-odds = -0.41 + (1.61 + 0.24 + 3.18 + 1.83 + 0.29 + 2.30 + 3.28 + 2.01) = 14.34 → P = 1/(1+e^−14.34) = 1.000 Ignored (not in vocabulary): are, a, here, to, claim, your

Why the steps add up

For an email with words w₁ … wₙ, Bayes’ theorem gives:

P(spam | email) ∝ P(spam) · P(w₁, …, wₙ | spam)

The “naive” step assumes the words are independent once you know the class, so the joint probability becomes a product:

P(w₁, …, wₙ | spam) = Πᵢ P(wᵢ | spam)

Divide the spam version by the ham version. The hard-to-compute P(email) cancels:

P(spam|email) / P(ham|email) = [P(spam)/P(ham)] · Πᵢ [P(wᵢ|spam) / P(wᵢ|ham)]

Take logs and the product becomes a sum, which is the chain of arrows above:

log-odds = log-prior + Σᵢ log[P(wᵢ|spam) / P(wᵢ|ham)]

The sigmoid turns it back into a probability: P = 1 / (1 + e−log-odds).

Things to try

  • Load “Mixed signals”: spammy words and work words pull against each other, and the verdict depends on which side adds up to more.
  • Move the prior slider. It only shifts the starting point; strong word evidence still outweighs it.
  • Add “the”. It’s equally common in both classes, so its ratio is 1 and its log is 0.
  • Add “click” and “free” together. They tend to appear in the same spam, but the model counts them as separate evidence, so it ends up overconfident. That’s the cost of the “naive” assumption.

Simplifications

  • The word probabilities are illustrative numbers, not estimates from real data. A real filter counts words in labelled emails and uses Laplace smoothing so nothing is exactly 0.
  • Each word counts once, and absent words count not at all. Multinomial naive Bayes adds an arrow for every repeat; Bernoulli naive Bayes also adds a small arrow for each vocabulary word missing from the email.
  • Words outside the vocabulary are ignored and listed under the formula.