Rényi entropy

In information theory , the Rényi entropy (named after Alfréd Rényi ) is a generalization of the Shannon entropy . The Rényi entropy belongs to the family of functions designed a system to quantify the diversity, uncertainty or randomness.

The Rényi entropy of order α, where α> 0, is defined as:

{\ displaystyle H _ {\ alpha} (X) = {\ frac {1} {1- \ alpha}} \ log _ {2} {\ Bigg (} \ sum _ {i = 1} ^ {n} p_ { i} ^ {\ alpha} {\ Bigg)}}

Here, X is a random variable with a range of values { x ₁ , x ₂ ... x _n } and p _{i is} the probability that X = x _i . If the probabilities p _{i are} all equal, then H _α ( X ) = log ₂ n , independent of α. Otherwise the entropies are monotonically decreasing as a function of α.

Here are a few individual cases:

{\ displaystyle H_ {0} (X) = \ log _ {2} n = \ log _ {2} | X |, \,}

which the logarithm of the thickness of X is sometimes also the "Hartley-entropy" X is called.

If the limit approaches 1 ( L'Hospital ), we get: ${\ displaystyle \ alpha}$

{\ displaystyle H_ {1} (X) = - \ sum _ {i = 1} ^ {n} p_ {i} \ log _ {2} p_ {i}}

which corresponds to the "Shannon entropy / information entropy".

Further

{\ displaystyle H_ {2} (X) = - \ log _ {2} \ sum _ {i = 1} ^ {n} p_ {i} ^ {2}}

which corresponds to the "entropy of correlation". The limit of for is ${\ displaystyle H _ {\ alpha}}$ ${\ displaystyle \ alpha \ rightarrow \ infty}$

{\ displaystyle H _ {\ infty} (X) = - \ log _ {2} \ sup _ {i = 1..n} p_ {i}}

and is also called min-entropy because it is the smallest value of . ${\ displaystyle H _ {\ alpha}}$

The Rényi entropies are important in ecology and statistics as indices of diversity. They also lead to a spectrum of indices of the fractal dimension .

literature

Dieter Masak: IT alignment. IT architecture and organization, Springer Verlag, Berlin / Heidelberg 2006, ISBN 978-3-540-31153-9 .
Lienhard Pagel: Information is energy. Definition of a physically based information term, Springer Fachmedien, Wiesbaden 2013, ISBN 978-3-8348-2611-4 .

Web links

Shannon Entropy, Renyi Entropy, and Information (accessed February 23, 2018)
Rényi Entropy and the Uncertainty Relations (accessed February 23, 2018)
Calculation of information and complexity (accessed on February 23, 2018)
Characterizations of Shannon and Renyi entropy (accessed February 23, 2018)
Convexity / Concavity of Renyi Entropy and α-Mutual Information (accessed February 23, 2018)