Exponential family

In probability theory and statistics , an exponential family (or exponential family ) is a class of probability distributions of a very specific form. One chooses this special form in order to take advantage of certain computational advantages or for reasons of generalization. In a certain sense, exponential families are very natural distributions and a dominated distribution class , which brings many simplifications in handling with it. The concept of exponential families goes back to EJG Pitman , G. Darmois , and BO Koopman (1935–6).

One-parameter exponential family

definition

A family of probability measures on the measurement space with is called a one-parameter exponential family if there is a σ-finite measure such that all are a density function of the shape ${\displaystyle (P_{\vartheta })_{\vartheta \in \Theta }}$ ${\displaystyle (X,{\mathcal {A}})}$ ${\displaystyle \Theta \subset \mathbb {R} }$ ${\displaystyle \mu }$ ${\displaystyle P_{\vartheta }}$

${\displaystyle f(x,\vartheta )=h(x)A(\vartheta )\exp(\eta (\vartheta )T(x))}$

regarding own. Usually it is ${\displaystyle \mu }$ ${\displaystyle \mu }$

• around the Lebesgue measure , the density functions are then conventional probability density functions (continuous case).
• around the count , the density functions are then count densities (discrete case).

It is

${\displaystyle T:(X,{\mathcal {A}})\to (\mathbb {R} ,{\mathcal {B}}(\mathbb {R} ))}$

a measurable function , the natural sufficient statistics or canonical statistics of the exponential family. Likewise is

${\displaystyle h:(X,{\mathcal {A}})\to (\mathbb {R} ,{\mathcal {B}}(\mathbb {R} ))}$

a measurable function. The function

${\displaystyle A:\Theta \to \mathbb {R} }$

is called the normalization function or normalization constant and guarantees that the normalization required in the definition of a probability measure is given. Furthermore is

${\displaystyle \eta :\Theta \to \mathbb {R} }$

any real function of the parameter.

Examples

The binomial distributions on with are an elementary example . They have the probability function (or the density function with regard to the count) ${\displaystyle X=\{1,\dots ,n\}}$ ${\displaystyle {\mathcal {A}}={\mathcal {P}}(X)}$

${\displaystyle f(x,\vartheta )={\binom {n}{x}}\vartheta ^{x}(1-\vartheta )^{n-x}={\binom {n}{x}}(1-\vartheta )^{n}\exp \left(x\ln \left({\frac {\vartheta }{1-\vartheta }}\right)\right)}$

with . Thus the binomial distribution is part of an exponential family and is characterized by ${\displaystyle \vartheta \in (0,1)}$

${\displaystyle T(x)=x,\quad ,\eta (\vartheta )=\ln \left({\frac {\vartheta }{1-\vartheta }}\right),\quad A(\vartheta )=(1-\vartheta )^{n}{\text{ und }}h(x)={\binom {n}{x}}}$.

Another example is the exponential distributions . They are defined with and have the probability density function ${\displaystyle ([0,\infty ),{\mathcal {B}}([0,\infty ))}$ ${\displaystyle \vartheta \in (0,\infty )}$

${\displaystyle f(x,\vartheta )=\vartheta \exp \left(-\vartheta x\right)}$

So in this case

${\displaystyle T(x)=x,\quad \eta (\vartheta )=-\vartheta \,{\text{ sowie }}A(\vartheta )=\vartheta }$.

It should be noted that a one-parameter exponential family can be a multivariate distribution . Single-parameter here only means that the dimensionality of the "shape parameter" is one. Whether the defined probability distribution is univariate or multivariate depends on the dimensionality of the basic space , which is not subject to any requirements. ${\displaystyle \vartheta }$ ${\displaystyle X}$

Alternative definitions

The definitions of an exponential family usually differ in the following points:

• Not all authors write the functions and as a product in front of the exponential function, sometimes they are also used as a sum in the exponential function, sometimes with a negative sign. This is how the definitions can be found ${\displaystyle h}$ ${\displaystyle A}$

${\displaystyle f(x,\vartheta )=\exp(\eta (\vartheta )T(x)+h^{*}(x)+A^{*}(\vartheta )){\text{ oder alternativ }}f(x,\vartheta )=h(x)\exp(\eta (\vartheta )T(x)-{\tilde {A}}(\vartheta ))}$.

These differently defined functions can usually be converted into one another without any problems. However, when specifying the functions and, it is important to note how precisely they are defined. ${\displaystyle A}$ ${\displaystyle h}$

• Some authors provide the density function with a characteristic function with respect to a set . The density function is then given as ${\displaystyle M}$

${\displaystyle f(x,\vartheta )=\chi _{M}(x)h(x)A(\vartheta )\exp(\eta (\vartheta )T(x))}$.

The choice of the amount should be independent of the parameter . This definition enables certain criteria that are based on the positivity of the density function to be more general. Such criteria can be found, for example, in regular statistical models . ${\displaystyle M}$ ${\displaystyle \vartheta }$

k -parametric exponential family

definition

A family of probability measures on the measurement space with is called a k-parametric exponential family if there is a σ-finite measure such that all the density function ${\displaystyle (P_{\vartheta })_{\vartheta \in \Theta }}$ ${\displaystyle (X,{\mathcal {A}})}$ ${\displaystyle \Theta \subset \mathbb {R} ^{k}}$ ${\displaystyle \mu }$ ${\displaystyle P_{\vartheta }}$

${\displaystyle f(x,\vartheta )=h(x)A(\vartheta )\exp \left(\sum _{i=1}^{k}\eta _{i}(\vartheta )T_{i}(x)\right)}$

regarding own. The parameter is often written. Are there ${\displaystyle \mu }$ ${\displaystyle \vartheta =(\vartheta _{1},\dots ,\vartheta _{k})}$

${\displaystyle h,T_{1},\dots ,T_{k}:(X,{\mathcal {A}})\to (\mathbb {R} ,{\mathcal {B}}(\mathbb {R} ))}$

measurable functions and

${\displaystyle A,\eta _{1},\dots ,\eta _{k}:\mathbb {R} ^{k}\supset \Theta \to \mathbb {R} }$

Functions of the k-dimensional parameter . Here, as in the one-parameter case, the function is called the natural sufficient statistics or the canonical statistics. ${\displaystyle \vartheta }$ ${\displaystyle T=(T_{1},\dots ,T_{k})}$

example

A classic example of a 2-parametric exponential family is the normal distribution . It is as well . Each is then of the form . With the parameterizations and one obtains the normal distribution from the usual density function ${\displaystyle (X,{\mathcal {A}})=(\mathbb {R} ,{\mathcal {B}}(\mathbb {R} ))}$ ${\displaystyle \Theta =\mathbb {R} \times (0,\infty )}$ ${\displaystyle \vartheta \in \Theta }$ ${\displaystyle \vartheta =(\vartheta _{1},\vartheta _{2})}$ ${\displaystyle \mu =\vartheta _{1}}$ ${\displaystyle \sigma ^{2}=\vartheta _{2}^{2}}$

${\displaystyle f(x,\vartheta _{1},\vartheta _{2})={\frac {1}{\sqrt {2\pi \vartheta _{2}^{2}}}}\exp \left(-{\frac {\vartheta _{1}^{2}}{2\vartheta _{2}^{2}}}\right)\exp \left({\frac {\vartheta _{1}}{\vartheta _{2}^{2}}}x-{\frac {1}{2\vartheta _{2}^{2}}}x^{2}\right)}$.

Thus, the normal distribution is part of a two-parameter exponential family with

${\displaystyle A(\vartheta _{1},\vartheta _{2})={\frac {1}{\sqrt {2\pi \vartheta _{2}^{2}}}}\exp \left(-{\frac {\vartheta _{1}^{2}}{2\vartheta _{2}^{2}}}\right),\quad T_{1}(x)=x,\quad T_{2}(x)=x^{2},\quad \eta _{1}(\vartheta _{1},\vartheta _{2})={\frac {\vartheta _{1}}{\vartheta _{2}^{2}}},\quad \eta _{2}(\vartheta _{1},\vartheta _{2})=-{\frac {1}{2\vartheta _{2}^{2}}}}$.

Here, too, the following applies again: a k-parametric exponential family can definitely describe a probability distribution in only one dimension. The number k only indicates the number of shape parameters, not the dimensionality of the distribution. In the above example, the normal distribution is one-dimensional, but part of a 2-parametric exponential family.

Another example of a 2-parametric exponential family is the gamma distribution .

Alternative definitions

For the k-parametric exponential family, the same variants exist in the definition as were already discussed in the one-parametric case. In addition, some authors require in the definition that the following two properties apply:

1. The functions are linearly independent ${\displaystyle \eta _{1},\dots ,\eta _{k}}$
2. The functions are almost certainly linearly independent for everyone . ${\displaystyle 1,T_{1},\dots ,T_{k}}$ ${\displaystyle P_{\vartheta }}$

With these additional requirements, statements can be made about the covariance matrix of , for example . ${\displaystyle T}$

The natural parameterization

In both the one-parametric and the k-parametric case, it is said that the exponential family is present in the natural parameterization if is. ${\displaystyle \eta (\vartheta )=\vartheta }$

properties

Sufficiency

For the exponential family, canonical statistics are always sufficient statistics . This follows directly from the Neyman criterion for sufficiency. Therefore it is also referred to as natural sufficient statistics. ${\displaystyle T}$ ${\displaystyle T}$

Score function

For a one-parameter exponential family the score function is given by

${\displaystyle S_{\vartheta }(x):={\frac {\partial }{\partial \vartheta }}\ln f(x,\vartheta )=\eta '(\vartheta )T(x)+{\frac {A'(\vartheta )}{A(\vartheta )}}}$.

With natural parameterization this is simplified to

${\displaystyle S_{\vartheta }(x)=T(x)+{\frac {A'(\vartheta )}{A(\vartheta )}}}$.

Fisher information

The Fisher information can be derived from the score function . it is

${\displaystyle I(\vartheta )=\operatorname {Var} _{\vartheta }(S_{\vartheta })=\left[\eta '(\vartheta )\right]^{2}\cdot \operatorname {Var} _{\vartheta }(T(x))}$.

With natural parameterization, the Fisher information thus results

${\displaystyle I(\vartheta )=\operatorname {Var} _{\vartheta }(T(x))}$.

Role in statistics

Classic appreciation: sufficiency

According to the Pitman-Koopman-Darmois theorem , among probability families whose support does not depend on the parameters there are sufficient statistics only among the exponential families , the dimensions of which remain limited as the sample size increases. A little more detailed: Let be independently and identically distributed random variables whose probability distribution family is known. Only if this family is an exponential family is there a (possibly vectorial) sufficient statistic whose number of scalar components does not increase if the sample size is increased. ${\displaystyle X_{n},\ n=1,2,3,\dots }$ ${\displaystyle T(X_{1},\dots ,X_{n})}$ ${\displaystyle n}$

Bayesian estimation: conjugate distributions

Exponential families are also important for Bayesian statistics . In Bayesian statistics, an a priori probability distribution is multiplied by a likelihood function and then normalized in order to arrive at the a posteriori probability distribution (see Bayes' theorem ). If the likelihood belongs to an exponential family, there also exists a family of conjugate a priori distributions, which is often also an exponential family. An a priori conjugate distribution for the parameter of an exponential family is defined by ${\displaystyle \pi }$ ${\displaystyle \eta }$

${\displaystyle \pi (\eta )\propto \exp(-\eta ^{\top }\alpha -\beta \,A(\eta )),}$

where and are hyperparameters (parameters that are not estimated, but rather determined within the framework of the model). ${\displaystyle \alpha \in \mathbb {R} ^{n}}$ ${\displaystyle \beta >0}$

In general, the likelihood function does not belong to any exponential family, which is why there is generally no conjugate a priori distribution. The posterior distribution must then be calculated using numerical methods.

Hypothesis tests: equally best test

The one-parametric exponential family is one of the distribution classes with a monotonic density quotient in canonical statistics if is monotonically increasing. Therefore, for the one-sided test problem with ${\displaystyle T}$ ${\displaystyle \eta }$

${\displaystyle \Theta _{0}=\{\vartheta \in \Theta \,|\,\vartheta \leq \vartheta _{0}\}\quad {\text{und}}\quad \Theta _{1}=\{\vartheta \in \Theta \,|\,\vartheta >\vartheta _{0}\}}$

a consistently best test at a given level . An explicit description of the test with sketched derivation from the Neyman-Pearson lemma can be found here . ${\displaystyle \alpha }$

literature

• EL Lehmann, Casella, G .: Theory of Point Estimation 1998, ISBN 0-387-98502-6 , S. 2nd ed., Sec. 1.5.
• Robert W. Keener: Statistical Theory: Notes for a Course in Theoretical Statistics . Springer, 2006, pp. 27-28, 32-33.
• Hans-Otto Georgii : Stochastics . Introduction to probability theory and statistics. 4th edition. Walter de Gruyter, Berlin 2009, ISBN 978-3-11-021526-7 , doi : 10.1515 / 9783110215274 .
• Ludger Rüschendorf: Mathematical Statistics . Springer Verlag, Berlin Heidelberg 2014, ISBN 978-3-642-41996-6 , doi : 10.1007 / 978-3-642-41997-3 .
• Claudia Czado, Thorsten Schmidt: Mathematical Statistics . Springer-Verlag, Berlin Heidelberg 2011, ISBN 978-3-642-17260-1 , doi : 10.1007 / 978-3-642-17261-8 .

Individual evidence

1. ↑ Erling Andersen: Sufficiency and Exponential Families for Discrete Sample Spaces . In: Journal of the American Statistical Association . 65, No. 331, September 1970, pp. 1248-1255. doi : 10.2307 / 2284291 .
2. ^ E. Pitman: Sufficient statistics and intrinsic accuracy . In: Proc. Camb. phil. Soc. . 32, 1936, pp. 567-579.
3. ↑ G. Darmois: Sur les lois de probabilites a estimation exhaustive . In: CR Acad. sci. Paris . 200, 1935, pp. 1265-1266.
4. ^ B Koopman: On distribution admitting a sufficient statistic . In: Trans. Amer. math. Soc. . 39, 1936, pp. 399-409. doi : 10.2307 / 1989758 .