KL divergence averages the log probability ratio under the first distribution. It is nonnegative and zero when the distributions match, but exchanging their roles generally changes the result. Here and are Bernoulli success probabilities; natural logarithms give units of nats.KL 散度按第一个分布,对两者概率之比的对数取平均。它非负,两分布相同时为零,但交换两者通常会得到不同结果。这里的 和 是伯努利分布的成功概率,使用自然对数,单位为 nat。
Keep fixed and move . Green is , cyan is ; both meet zero at . Their different heights show why KL is not a symmetric distance. Only is plotted, avoiding infinite divergences at unsupported outcomes.固定 ,改变 。绿色表示 ,青色表示 ,两者都在 时降至零。不同的高度说明 KL 不是对称距离。图中只画 ,避开对某个可能结果赋零概率而产生的无穷散度。