Cross entropy
In information theory and mathematical statistics, cross entropy is a measure of the quality of a model for a probability distribution .
definition
Let be a random variable with a target set that is distributed accordingly . Let it be a distribution on the same event space .
Then the cross entropy is defined by:
Here denote the entropy of and the Kullback-Leibler divergence of the two distributions.
Equivalent formulation
By inserting the two definition equations, simplification results in the discrete case
and in the continuous case (with density functions and )
estimate
Although the cross entropy has a similar significance as the pure Kullback-Leibler divergence, the former can however also be estimated without precise knowledge . In practical application, it is therefore usually an approximation of an unknown distribution .
According to the above equation:
Where denotes the expected value according to the distribution .
Are now realizations of , i. H. an independent and identically according distributed sample , it is therefore
an unbiased estimator for the cross entropy.
Derived quantities
The size or is also referred to as perplexity . It is mainly used in speech recognition .
Literature & web links
- Rubinstein, Reuven Y. / Kroese, Dirk P .: The Cross-Entropy Method - A Unified Approach to Combinatorial Optimization, Monte-Carlo Simulation and Machine Learning . Springer Verlag 2004, ISBN 978-0-387-21240-1 .
- Entropy script Heidelberg University
- Statistical Language Models University of Munich (PDF; 531 kB)