Surprise

\begin{aligned} S u r p r i s e & =\frac{1}{P (x)} \\ =log_{2}\frac{1}{P (x)} \\ =log_{2}1-log_{2}P(x) \\ =-log_{2}P(x) \end{aligned}

Surprise = ?

$S u r p r i s e = l o g_{2} \frac{1}{P (x)} = - l o g_{2} P (x)$

Entropy is the expected surprise of a distribution; see also Cross Entropy and KL Divergence.

Related Notes