How to use this calculator
- Enter or choose distribution P (true / observed).
- Enter or choose distribution Q (model / reference).
- Enter or choose logarithm base.
- Enter or choose pseudo-count added to every category (optional).
- The d(P ‖ Q) updates as you type. Use Copy result or Share to keep it.
Method
D(P‖Q) = Σ p_i log(p_i ÷ q_i), with 0·log(0/q) = 0 and p·log(p/0) = ∞; H(P, Q) = H(P) + D(P‖Q); JS = ½D(P‖M) + ½D(Q‖M), M = (P + Q) ÷ 2
- p_i, q_i
- probabilities of category i under P and Q
- M
- the average of P and Q
Worked example
- Q normalises to ⅓ each.
- D(P‖Q) = 0.36 ln(1.08) + 0.48 ln(1.44) + 0.16 ln(0.48) = 0.0277 + 0.1750 − 0.1174 = 0.0853 nats.
- D(Q‖P) = ⅓[ln(⅓ ÷ 0.36) + ln(⅓ ÷ 0.48) + ln(⅓ ÷ 0.16)] = 0.0975 nats.
Approximating the binomial(2, 0.4) distribution by a uniform one loses 0.0853 nats; the reverse direction gives 0.0975, showing KL is not symmetric.
What the result means
Use D(P‖Q) with P as the data and Q as the model. Jensen–Shannon divergence is symmetric and always finite (at most ln 2 nats or 1 bit), so it is easier to compare across pairs.
Assumptions and limits
- Both lists cover the same categories in the same order.
- Zero handling: categories where P is 0 add nothing; if Q is 0 where P is not, D(P‖Q) is infinite and the calculator asks for smoothing instead of returning a number.
- Discrete distributions only.
- Smoothing changes the result; report the pseudo-count you used.
Questions
Why is KL divergence not a distance?
It is not symmetric and does not satisfy the triangle inequality. Its square-root-of-JS cousin is a true metric.
How does it relate to cross-entropy loss?
Cross-entropy H(P, Q) = H(P) + D(P‖Q). With P fixed by the data, minimising cross-entropy is the same as minimising KL divergence.