Cross entropy

In information theory and mathematical statistics, cross entropy is a measure of the quality of a model for a probability distribution .

definition

Let be a random variable with a target set that is distributed accordingly . Let it be a distribution on the same event space . ${\displaystyle X}$ ${\displaystyle \Omega }$ ${\displaystyle P}$ ${\displaystyle Q}$

Then the cross entropy is defined by:

${\displaystyle H(X;P;Q)=H(X)+D(P\Vert Q)}$

Here denote the entropy of and the Kullback-Leibler divergence of the two distributions. ${\displaystyle H(X)}$ ${\displaystyle X}$ ${\displaystyle D(P\|Q)}$

Equivalent formulation

By inserting the two definition equations, simplification results in the discrete case

${\displaystyle H(X;P;Q)=-\sum _{x\in \Omega }P(X=x)\cdot \log Q(X=x)\,.}$

and in the continuous case (with density functions and ) ${\displaystyle p}$ ${\displaystyle q}$

${\displaystyle H(X;P;Q)=-\int _{\Omega }p(x)\cdot \log q(x)\mathrm {d} x}$

estimate

Although the cross entropy has a similar significance as the pure Kullback-Leibler divergence, the former can however also be estimated without precise knowledge . In practical application, it is therefore usually an approximation of an unknown distribution . ${\displaystyle P}$ ${\displaystyle Q}$ ${\displaystyle P}$

According to the above equation:

${\displaystyle H(X;P;Q)=E(-\log Q(X))}$

Where denotes the expected value according to the distribution . ${\displaystyle E}$ ${\displaystyle P}$

Are now realizations of , i. H. an independent and identically according distributed sample , it is therefore ${\displaystyle x_{1};\dots ;x_{n}\in \Omega }$ ${\displaystyle X}$ ${\displaystyle P}$

${\displaystyle {\hat {H}}(Q;n)=-{\frac {1}{n}}\sum _{i=1}^{n}\log Q(x_{i})}$

an unbiased estimator for the cross entropy.

Derived quantities

The size or is also referred to as perplexity . It is mainly used in speech recognition . ${\displaystyle 2^{H(X;P;Q)}}$ ${\displaystyle 2^{H(X)}}$

Literature & web links

• Rubinstein, Reuven Y. / Kroese, Dirk P .: The Cross-Entropy Method - A Unified Approach to Combinatorial Optimization, Monte-Carlo Simulation and Machine Learning . Springer Verlag 2004, ISBN 978-0-387-21240-1 .
• Entropy script Heidelberg University
• Statistical Language Models University of Munich (PDF; 531 kB)