KL Divergence

KL-Divergence = Kullback-Leibler Divergence
KL divergence in it's core is the ration of two probability distribution
Very common in Reinforcement Learning from Human Feedback (RLHF) algorithms (i.e Proximal Policy Optimization (PPO), GRPO, KTO)
So,

\begin{aligned} K L & =\frac{1}{N}log\frac{\prod_{x} P (X = x)}{\prod_{x} Q (X = x)} \\ =\frac{1}{N}log\frac{P_{1} * P_{1} * P_{2} * P_{1} * P_{1}}{Q_{1} * Q_{2} * Q_{1} * Q_{2} * Q_{2}} \\ =\frac{1}{N}log\frac{P_{1}^{N_{1}} P_{2}^{N_{2}}}{Q_{1}^{N_{1}} Q_{2}^{N_{2}}} \\ =\frac{1}{N}(N_{1}logP_{1}+N_{2}logP_{2}-N_{1}logQ_{1}-N_{2}logQ_{2}) \\ =P_{1}logP_{1}+P_{2}logP_{2}-P_{1}logQ_{1}-P_{2}logQ_{2} \\ =P_{1}log\frac{P_{1}}{Q_{1}}+P_{2}log\frac{P_{2}}{Q_{2}} \\ =\sum_{i}P_{i}log\frac{P_{i}}{Q_{i}} \end{aligned}

KL Divergence = ?

$D_{K L} (P | | Q) = \sum_{i} P_{i} l o g \frac{P_{i}}{Q_{i}}$ $D_{K L} (P | | Q) = \int P (x) l o g \frac{P (x)}{Q (x)}$

Related Notes