Suppose in one pen there are 9 orange chickens and 1 blue chicken.
\[P(\text{orange}) = \frac{9}{10} = 0.9 \qquad P(\text{blue}) = \frac{1}{10} = 0.1\]
So surprise should be a number that behaves like this:
High probability → Low surprise · Low probability → High surprise
A first guess could be:
\[\text{Surprise} = \frac{1}{p}\]
This seems reasonable because if \(p\) is small, \(\frac{1}{p}\) is large, and if \(p\) is large, \(\frac{1}{p}\) is small.
But there is a problem.
Suppose an event is certain: \(p = 1\). Then:
\[\frac{1}{1} = 1\]
But if something is certain, the surprise should be 0, not 1. So inverse probability is not the best choice.
To fix that, we use:
\[\text{Surprise} = \log\left(\frac{1}{p}\right)\]
This is also written as:
\[\text{Surprise} = -\log(p) \qquad \text{because} \qquad \log\left(\frac{1}{p}\right) = -\log(p)\]
Both mean the same thing.
Case 1: Event is certain
If \(p = 1\),
\[\text{Surprise} = \log\left(\frac{1}{1}\right) = \log(1) = 0\]
A certain event gives zero surprise.
Case 2: Event is rare
If \(p = 0.1\),
\[\text{Surprise} = \log_2\left(\frac{1}{0.1}\right) = \log_2(10) \approx 3.32\]
That is much larger.
Common event → small surprise · Rare event → large surprise
Let us calculate surprise for our chickens using base-2 log.
Orange chicken — \(P(\text{orange}) = 0.9\)
\[\text{Surprise of orange} = -\log_2(0.9) \approx 0.15\]
So picking an orange chicken gives very little surprise.
Blue chicken — \(P(\text{blue}) = 0.1\)
\[\text{Surprise of blue} = -\log_2(0.1) \approx 3.32\]
So picking a blue chicken gives much more surprise. That is exactly what we expected.
Suppose the farmer picks 3 chickens and gets: Orange, Orange, Blue
The probabilities are: Orange = 0.9, Orange = 0.9, Blue = 0.1
So total probability of that exact sequence is \(0.9 \times 0.9 \times 0.1\)
The surprise of the full sequence is:
\[-\log_2(0.9 \times 0.9 \times 0.1) = -\log_2(0.9) - \log_2(0.9) - \log_2(0.1)\]
\[= 0.15 + 0.15 + 3.32 = 3.62\]
Important result: Total surprise for several events = sum of individual surprises.
This matters because entropy is built from this idea.
Suppose we keep picking chickens many times from the same pen.
Some picks will be orange, some blue. Each result has its own surprise:
But we do not want surprise of one single outcome anymore. We want:
“On average, how much surprise should we expect?”
That average surprise is called entropy.
Entropy = Expected Surprise
That is the most important sentence.
For each outcome, we take:
\[\text{probability of outcome} \times \text{surprise of outcome}\]
Then add for all outcomes. For our chicken example:
\[H = 0.9 \cdot 0.15 + 0.1 \cdot 3.32\]
\[H = 0.135 + 0.332\]
\[H \approx 0.467\]
On average, each pick gives about 0.47 units of surprise. That is entropy.
Since surprise is \(-\log(p(x))\) and entropy is expected surprise, we get:
\[H(X) = \sum p(x)\left(-\log p(x)\right)\]
which is commonly written as:
\[H(X) = -\sum p(x) \log p(x)\]
Entropy measures uncertainty or average unpredictability.
Low Entropy
One outcome is much more likely than the others.
\(P(\text{orange}) = 0.99\), \(P(\text{blue}) = 0.01\)
Almost every time, you know what will happen. So uncertainty is low.
High Entropy
Outcomes are balanced.
\(P(\text{orange}) = 0.5\), \(P(\text{blue}) = 0.5\)
Now either outcome is equally possible. So uncertainty is high.
Case 1: Very uneven distribution — \(P(\text{orange}) = 0.9,\ P(\text{blue}) = 0.1\)
\[H = -(0.9\log_2 0.9 + 0.1\log_2 0.1) \approx 0.47\]
Case 2: Perfectly balanced distribution — \(P(\text{orange}) = 0.5,\ P(\text{blue}) = 0.5\)
\[H = -(0.5\log_2 0.5 + 0.5\log_2 0.5)\]
Since \(\log_2(0.5) = -1\):
\[H = -(0.5(-1) + 0.5(-1)) = 1\]
So entropy is higher in Case 2. That makes sense because 50–50 is more uncertain than 90–10.
A pen is high entropy when you are not sure what chicken you will get. A pen is low entropy when one color is so common that there is hardly any uncertainty.
Now: KL Divergence
KL divergence stands for Kullback–Leibler Divergence. It does not measure average surprise within one distribution. Instead, it compares two probability distributions.
Imagine two descriptions of the same chicken pen:
| Orange | Blue | |
|---|---|---|
| True distribution \(P\) | 0.9 | 0.1 |
| Guessed model \(Q\) | 0.5 | 0.5 |
Your model thinks both colors are equally likely. But that is wrong.
So KL divergence measures: “How bad is it to use \(Q\) when the truth is \(P\)?”
It tells us how different the guessed distribution is from the true one.
\[D_{\mathrm{KL}}(P \,\|\, Q) = \sum_x P(x)\log\!\left(\frac{P(x)}{Q(x)}\right)\]
Given \(P = (0.9,\ 0.1)\) and \(Q = (0.5,\ 0.5)\):
\[D_{\mathrm{KL}}(P \| Q) = 0.9\log_2\!\left(\frac{0.9}{0.5}\right) + 0.1\log_2\!\left(\frac{0.1}{0.5}\right)\]
First term
\(\frac{0.9}{0.5} = 1.8\)
\(\log_2(1.8) \approx 0.848\)
\(0.9 \times 0.848 \approx 0.763\)
Second term
\(\frac{0.1}{0.5} = 0.2\)
\(\log_2(0.2) \approx -2.322\)
\(0.1 \times (-2.322) \approx -0.232\)
\[D_{\mathrm{KL}}(P \| Q) \approx 0.763 - 0.232 = \mathbf{0.531}\]
This means the guessed distribution \(Q\) is not a very good match for the true distribution \(P\).
Entropy measures the average information content of a distribution:
\[H(P) = -\sum P(x)\log P(x)\]
Cross-entropy measures the average information when using the wrong distribution \(Q\) to encode \(P\):
\[H(P,Q) = -\sum P(x)\log Q(x)\]
KL Divergence is simply the difference:
\[D_{KL}(P \| Q) = H(P,Q) - H(P)\]