Differential entropy

The differential entropy is a term from information theory and represents a measure of the entropy of a continuous random variable , similar to the Shannon entropy for discrete random variables.

Strictly speaking, it is a measure of a probability distribution . It can be used to compare two continuous random variables, but does not have the same information as the Shannon entropy.

definition

A continuous random variable can have an infinite number of values, i.e. if one could determine its values ​​exactly, the probability for a certain value would be zero: ${\displaystyle X}$ ${\displaystyle P(\cdot )}$ ${\displaystyle x_{0}}$

${\displaystyle P(X=x_{0})=\lim _{\varepsilon \to 0}\varepsilon f_{X}(x_{0})=0}$

And thus the information content of each value is infinite:

${\displaystyle \lim _{p_{x}\to 0}I(p_{x})=-\lim _{p_{x}\to 0}\log _{2}(p_{x})=\infty }$

Let be a continuous random variable with the probability density function , then its differential entropy is defined as ${\displaystyle X}$ ${\displaystyle f_{X}}$

${\displaystyle h(X)=-\int _{-\infty }^{+\infty }f_{X}(x)\log f_{X}(x)\,{\text{d}}x=E[-\log f_{X}(x)]}$

In contrast to the Shannon entropy, the differential entropy can also be negative.

Since the differential entropy is not scaling-invariant (see below), it is advisable to normalize the random variable appropriately so that it is dimensionless .

properties

• The differential entropy is shift-invariant, ie for constant . It is therefore sufficient to consider mean-value-free random variables. ${\displaystyle h(X+c)=h(X)}$ ${\displaystyle c}$
• The following applies to the scaling: with the random vector and the amount of the determinant . ${\displaystyle h(A{\textbf {X}})=h({\textbf {X}})+\log _{2}(|A|)}$ ${\displaystyle {\textbf {X}}\in \mathbb {R} ^{n}}$ ${\displaystyle |A|}$

Differential entropy for different distributions

For a given variance , the Gaussian distribution has the maximum differential entropy, ie its “randomness” or its surprise value is - compared to all other distributions - the greatest. It is therefore also used to model interference in the channel model , since it represents a worst-case model for interference (see also additive white Gaussian noise ). ${\displaystyle \sigma _{x}^{2}}$

For a finite range of values, ie a given maximum amount, a uniformly distributed random variable has the maximum differential entropy.

Overview of different probability distributions and their differential entropy

distribution	Probability density function	Differential entropy (in bits)	carrier
equal distribution	${\displaystyle f(x)={\frac {1}{2a}}}$	${\displaystyle \log _{2}(2a)}$	${\displaystyle [-a,+a]}$
Normal distribution	${\displaystyle f(x)={\frac {1}{\sqrt {2\pi \sigma ^{2}}}}\exp \left(-{\frac {(x-\mu )^{2}}{2\sigma ^{2}}}\right)}$	${\displaystyle {\frac {1}{2}}\log _{2}(2\pi e\sigma _{x}^{2})}$	${\displaystyle (-\infty ,\infty )}$
Laplace distribution	${\displaystyle f(x)={\frac {1}{{\sqrt {2}}\sigma _{x}}}\exp \left(-{\frac {{\sqrt {2}}|x-\mu |}{\sigma _{x}}}\right)}$	${\displaystyle {\frac {1}{2}}\log _{2}(2e^{2}\sigma _{x}^{2})}$	${\displaystyle (-\infty ,\infty )}$
Symmetrical triangular distribution	${\displaystyle f(x)={\begin{cases}{\frac {1}{a}}\left(1-{\frac {|x|}{a}}\right)&\forall \;|x|\leq a\\0&\forall \;|x|>a\end{cases}}}$	${\displaystyle {\frac {1}{2}}\log _{2}(6e\sigma _{x}^{2})}$	${\displaystyle [0,1]}$

literature

• Thomas M. Cover, Joy A. Thomas: Elements of Information Theory . John Wiley & Sons, 1991, ISBN 0-471-06259-6 , pp. 224-238 .
• Martin Werner: Information and Coding. Basics and Applications, 2nd edition, Vieweg + Teubner Verlag, Wiesbaden 2008, ISBN 978-3-8348-0232-3 .
• Peter Adam Höher: Basics of digital information transfer. From theory to mobile radio applications, 1st edition, Vieweg + Teubner Verlag, Wiesbaden 2011, ISBN 978-3-8348-0880-6 .

Web links

• Differential Entropy , Wolfram Mathworld
• Information and Coding Theory (accessed February 2, 2018)
• Entropy différentielle (accessed February 2, 2018)