Cramér-Rao inequality

The Cramér-Rao inequality , also called information inequality or Fréchet inequality , is a central inequality in estimation theory , a branch of mathematical statistics . In regular statistical models, it provides an estimate for the variance of point estimates and thus a possibility of comparing different estimates with one another, as well as a criterion for determining uniformly best unbiased estimates .

The inequality is named after Harald Cramér and Calyampudi Radhakrishna Rao or after Maurice René Fréchet .

statement

Framework

A one-parameter standard model is given , that is, it is and each has a density function with respect to the measure . ${\displaystyle (X,{\mathcal {A}},(P_{\vartheta })_{\vartheta \in \Theta })}$ ${\displaystyle \Theta \subset \mathbb {R} }$ ${\displaystyle P_{\vartheta }}$ ${\displaystyle f(x,\vartheta )}$ ${\displaystyle \mu }$

Furthermore, the Cramér-Rao regularity conditions are fulfilled, that is, the following applies:

• ${\displaystyle \Theta }$ is an open crowd.
• The density function is actually greater than 0. ${\displaystyle f(x,\vartheta )}$ ${\displaystyle X\times \Theta }$
• The score function

${\displaystyle S_{\vartheta }(x)={\frac {\partial }{\partial \vartheta }}\ln f(x,\vartheta )}$

exists and is finite.

• The Fisher information is really positive and finite. ${\displaystyle I(\vartheta )}$
• The exchange relation applies

${\displaystyle \int {\frac {\partial }{\partial \vartheta }}f(x,\vartheta )\mathrm {d} \mu (x)={\frac {\partial }{\partial \vartheta }}\int f(x,\vartheta )\mathrm {d} \mu (x)}$.

formulation

Is then an estimator with finite variance and is ${\displaystyle T}$

${\displaystyle \operatorname {E} _{\vartheta }(T):=g(\vartheta )\quad \mathrm {f{\ddot {u}}r\;alle\;} \vartheta \in \Theta }$

so is an unbiased estimator for . Is now a regular estimator in the sense that the commutation relation ${\displaystyle T}$ ${\displaystyle g}$ ${\displaystyle T}$

${\displaystyle {\frac {\partial }{\partial \vartheta }}\int T(x)\cdot f(x,\vartheta )\,\mathrm {d} \mu (x)=\int T(x)\cdot {\frac {\partial }{\partial \vartheta }}f(x,\vartheta )\,\mathrm {d} \mu (x)}$,

is valid, then the Cramér-Rao inequality applies

${\displaystyle \operatorname {Var} _{\vartheta }(T)\geq {\frac {\left[g'(\vartheta )\right]^{2}}{I(\vartheta )}}\quad \mathrm {f{\ddot {u}}r\;alle\;} \vartheta \in \Theta }$

where is the derivative of . ${\displaystyle g'(\vartheta )}$ ${\displaystyle g(\vartheta )}$

Remarks

The definition of the function to be estimated via the expected value of guarantees the differentiability of this function. Alternatively, it can also be defined as an unbiased estimator for a differentiable function . ${\displaystyle g}$ ${\displaystyle T}$ ${\displaystyle T}$ ${\displaystyle g}$

Derived terms

Cramér-Rao barrier

If there is an unbiased estimator for the function , then the Cramér-Rao inequality simplifies to ${\displaystyle T}$ ${\displaystyle g(\vartheta )=\vartheta }$

${\displaystyle \operatorname {Var} _{\vartheta }(T)\geq {\frac {1}{I(\vartheta )}}\quad \mathrm {f{\ddot {u}}r\;alle\;} \vartheta \in \Theta }$.

This is also called the Cramér-Rao bound .

Cramér Rao efficiency and super efficiency

An estimator that satisfies the Cramér-Rao inequality with equality is called a Cramér-Rao efficient estimator . It is an equally best unbiased estimator for the class of regular estimators, i.e. those for which the above commutation relation applies. The simplest and best-known example of a Cramér-Rao efficient estimator is the arithmetic mean as an estimator for the expected value of a normal distribution. ${\displaystyle {\overline {X}}}$ ${\displaystyle \mu }$

Estimates that are even below the Cramér-Rao inequality are called super-efficient . These are necessarily non-regular or non-expected, so they do not meet the conditions of the Cramér-Rao inequality. The best-known representative of super efficient estimators is the James Stein estimator .

Regularity conditions and idea of ​​proof

The proof of the Cramér-Rao inequality is essentially based on the Cauchy-Schwarz inequality and two model assumptions that regulate the interchangeability of differentiation and integration .

On the one hand should

${\displaystyle \mathrm {E} _{\vartheta }\left[{\frac {\partial }{\partial \vartheta }}\log f_{\vartheta }(X_{i})\right]=0}$

apply and on the other hand we take

${\displaystyle \mathrm {E} _{\vartheta }\left[T(X){\frac {\partial }{\partial \vartheta }}\log f_{\vartheta }(X_{i})\right]={\frac {\partial }{\partial \vartheta }}\mathrm {E} _{\vartheta }\left[T(X)\right]}$

on. Substituting it directly into the Cauchy-Schwarz inequality then yields the claim.

Multi-dimensional formulation

Under similar regularity conditions, the Cramér-Rao inequality can also be formulated in the case of multi-dimensional parameters. The statement is then carried over to the consideration of the covariance matrix of the multidimensional estimator and provides a relation in the sense of the Löwner partial order for matrices . ${\displaystyle \leq }$

Let the vector of the unknown parameters and a multivariate random variable with the associated probability density be . ${\displaystyle {\underline {\vartheta }}:=\left[\vartheta _{1},\vartheta _{2},...,\vartheta _{n}\right]^{T}}$ ${\displaystyle \mathbf {X} }$ ${\displaystyle f(\mathbf {X} ;{\underline {\vartheta }})}$

The appraiser

${\displaystyle {\underline {T}}(\mathbf {X} ):=\left[T_{1}(\mathbf {X} ),T_{2}(\mathbf {X} ),...,T_{n}(\mathbf {X} )\right]^{T}}$

for the parameter vector has a covariance matrix ${\displaystyle {\underline {\vartheta }}}$

${\displaystyle \mathrm {Cov} [{\underline {T}}(\mathbf {X} )]=\mathrm {E} [({\underline {T}}(\mathbf {X} )-{\underline {\vartheta }})({\underline {T}}(\mathbf {X} )-{\underline {\vartheta }})^{T}]}$.

The Cramér-Rao inequality in this case reads

${\displaystyle \mathrm {Cov} [{\underline {T}}(\mathbf {X} )]\geq {\mathcal {I}}^{-1}({\underline {\vartheta }})}$

where the Fisher information matrix ${\displaystyle {\mathcal {I}}}$

${\displaystyle {\mathcal {I}}_{ij}({\underline {\vartheta }})=\mathrm {E} \left[{\frac {\partial }{\partial \vartheta _{i}}}\log \prod _{\ell =1}^{n}f(X_{\ell };{\underline {\vartheta }}){\frac {\partial }{\partial \vartheta _{j}}}\log \prod _{\ell =1}^{n}f(X_{\ell };{\underline {\vartheta }})\right]}$

is.

Applications

With the help of the Cramér-Rao inequality, the dynamic permeability number of membranes can be estimated, which is particularly popular in bio- and nanotechnology.

Generalizations

One possible generalization is the Chapman-Robbins inequality . It allows an estimation of the variance of an estimator with respect to a fixed predetermined one and is therefore used for estimations in the context of the investigation of locally minimal estimators . At the border crossing it provides a pointwise version of the Cramér-Rao inequality. ${\displaystyle P_{\vartheta _{0}}}$

The Van Trees inequality from Bayesian statistics can also be viewed as a generalization of the Cramér-Rao inequality . In contrast to this, the Van Trees inequality can also be applied to estimators that are not true to expectations.

Web links

• MS Nikulin: Rao-Cramer inequality . In: Michiel Hazewinkel (Ed.): Encyclopaedia of Mathematics . Springer-Verlag , Berlin 2002, ISBN 978-1-55608-010-4 (English, online ).

literature

• Ludger Rüschendorf: Mathematical Statistics . Springer Verlag, Berlin Heidelberg 2014, ISBN 978-3-642-41996-6 , doi : 10.1007 / 978-3-642-41997-3 .
• Claudia Czado, Thorsten Schmidt: Mathematical Statistics . Springer-Verlag, Berlin Heidelberg 2011, ISBN 978-3-642-17260-1 , doi : 10.1007 / 978-3-642-17261-8 .
• Hans-Otto Georgii : Stochastics . Introduction to probability theory and statistics. 4th edition. Walter de Gruyter, Berlin 2009, ISBN 978-3-11-021526-7 , doi : 10.1515 / 9783110215274 .
• Helmut Pruscha: Lectures on mathematical statistics. BG Teubner, Stuttgart 2000, ISBN 3-519-02393-8 , Section V.1.