Reduction principle

In mathematical statistics, reduction principles are various methods and arguments that make it easier to find good statistical procedures. The number of methods in question is reduced by a certain structural requirement, which makes it easier to find good methods. The structural requirement arises, for example, from the properties of the statistical model or pragmatic considerations. Three classic reduction principles are sufficiency , equivariance / invariance and faithfulness to expectations .

motivation

As an example, consider a statistical product model that is provided with an arbitrary family of probability distributions . So data are collected several times, each collection being a realization of an unknown probability distribution from the corresponding model. The unknown expected value of the present probability distribution is to be estimated, so the function to be estimated is of the form ${\ displaystyle (P _ {\ vartheta}) _ {\ vartheta \ in \ Theta}}$ ${\ displaystyle n}$

{\ displaystyle g (P _ {\ vartheta}) = \ operatorname {E} (P _ {\ vartheta})}

The collected data are elements of the , the expected value is a real number. Every estimator for the expected value is therefore a function of to , formal ${\ displaystyle (x_ {1}, x_ {2}, \ dots, x_ {n})}$ ${\ displaystyle \ mathbb {R} ^ {n}}$ ${\ displaystyle T}$ ${\ displaystyle \ mathbb {R} ^ {n}}$ ${\ displaystyle \ mathbb {R}}$

{\ displaystyle T \ colon \ mathbb {R} ^ {n} \ to \ mathbb {R}}

.

In order to be able to speak of good or bad estimators, a loss function is introduced with which a risk function can then be determined. The Gaussian loss is common , which is the mean square error (Means squared error, MSE) as a risk function.

{\ displaystyle \ operatorname {MSE} (T, \ operatorname {E} (P _ {\ vartheta})): = \ operatorname {E} _ {\ vartheta} \ left (\ left (T- \ operatorname {E} ( P _ {\ vartheta}) \ right) ^ {2} \ right)}

delivers (here denotes the formation of the expected value with respect to ). ${\ displaystyle \ operatorname {E} _ {\ vartheta}}$ ${\ displaystyle P _ {\ vartheta}}$

Now the best possible estimator is to be found for the expected value, i.e. an estimator whose mean square error is smaller than that of all other estimators.

The problem is that the previous question is a very open question with little structure, since the number of possible estimators is very large. Thus, on the one hand, the number of potential optimal estimators is very large; on the other hand, it is also difficult to show that a candidate for the best estimator is really better than all other estimators.

Therefore, it makes sense to look for criteria that reduce the set of estimation functions that are considered in order to simplify the search for optimal estimators. Three typical criteria are:

Faithfulness to expectations : Here one restricts oneself to estimators that are correct on average and therefore do not have any systematic errors
Sufficiency : The central question of sufficiency is whether the existing data can be compressed without loss of information. The compressed data then forms a simpler model that is easier to examine.
Equivariance and invariance: These criteria deal with the geometric properties of the model and exploit them. For example, certain estimated values should be independent of the scaling of the data, and the expected value in the above model should shift when the data is shifted by this same value.

Expectancy

The reduction through faithfulness to expectations is based on the idea that a good estimator should, on average, correctly estimate the value sought. Formulated the other way around, this corresponds to the requirement that the estimator should not show any systematic error. The characteristic of being true to expectations is also transferred to statistical tests and confidence ranges under the keyword of undistortion .

Of the reduction principles, the fidelity to expectation is the most accessible, since it is only based on dealing with the expected value and no further complex mathematical structures are required. However, unbiased estimators do not exist for every problem or can be completely nonsensical.

Reduction through sufficiency

The underlying idea behind sufficiency is that statistical models may contain information that is not needed to solve a task (estimating, testing, etc.). Therefore one tries to compress the existing data without loss of information. The compressed data can then be searched for using optimal statistical methods.

A simple example of this is tossing a coin n times. The repetitions should be independent of each other. The task is to estimate the unknown probability of the coin showing upside down. For the sake of simplicity, the head is coded with the number 1 and the number with the number 0. Performing the toss n times in a row suggests modeling the experiment as a product experiment. Either a one or a zero can be thrown in each round, so after rounds there is a base space ${\ displaystyle n}$

{\ displaystyle {\ mathcal {X}} _ {1} = \ {0.1 \} ^ {n}}

.

For each round from 1 to, this contains the information whether a 0 or 1 was thrown. One way to compress this information is to just note the number of ones thrown. This corresponds to the basic room ${\ displaystyle n}$

{\ displaystyle {\ mathcal {X}} _ {2}: = \ {0,1,2, \ dots, n \}}

,

the compression is due to the figure

{\ displaystyle (x_ {1}, x_ {2}, \ dots, x_ {n}) \ mapsto x_ {1} + x_ {2} + \ dots + x_ {n}}

conveyed. The fact that there is compression can already be seen from the power of the sets: contains elements, whereas only contains elements. The interesting question is whether all relevant information for the estimate is still available or whether information has already been lost. If all the relevant information (for the task at hand!) Is still contained in it, it is completely sufficient to look for good estimators . ${\ displaystyle {\ mathcal {X}} _ {1}}$ ${\ displaystyle 2 ^ {n}}$ ${\ displaystyle {\ mathcal {X}} _ {2}}$ ${\ displaystyle n + 1}$ ${\ displaystyle {\ mathcal {X}} _ {2}}$ ${\ displaystyle {\ mathcal {X}} _ {2}}$

The central tool when modeling compression is the conditional expected value . It not only enables the compression of data through images, but also the information content of set systems , in particular σ-algebras , to be recorded.

Reduction through invariance and equivariance

With the reduction through invariance and equivariance, one tries to identify and use geometric and algebraic structures in the model and the task. For example, the following applies to the expected value of a random variable

{\ displaystyle \ operatorname {E} (X + a) = \ operatorname {E} (X) + a}

for a number . Shifting the random variable by thus shifts the expected value by . This property is also called displacement equivariance. ${\ displaystyle a}$ ${\ displaystyle a}$ ${\ displaystyle a}$

If the expected value is to be estimated, then it makes sense to require estimators to be compatible with this property of the expected value. Is considered to be an estimator for the expected value, so should ${\ displaystyle T}$

{\ displaystyle T (X + a) = T (X) + a}

be valid. Such estimators are called equivariant. This corresponds to the intuition that a measure of position such as the expected value should change in the event of a shift in the position of the data and precisely this shift, since it is supposed to record the position of the data. An analogous example applies to the variance, since it is always

{\ displaystyle \ operatorname {Var} (X + a) = \ operatorname {Var} (X)}

Fulfills. It is therefore invariant in terms of displacement. Accordingly, an estimator for the variance should also be shift-invariant, i.e.

{\ displaystyle V (X + a) = V (X)}

fulfill. This is in line with the intuition that a measure of dispersion like variance is independent of the position of the data.

When reducing by invariance and equivariance, one tries to find such underlying algebraic and geometric requirements and then restricts the search for optimal statistical methods to those that are compatible with the underlying structure. In this case, which is group theory used as an aid, as the geometric and algebraic structures by groups be formalized. The relevant statistical methods are then those that are compatible with the group operations .

In the above example, the corresponding group would be the translation group , the compatibility of the images then corresponds to the (displacement) equivariance in the case of the expected value and the displacement invariance in the case of the variance. ${\ displaystyle \ mathbb {R} ^ {n}}$

Individual evidence

^ Ludger Rüschendorf: Mathematical Statistics . Springer Verlag, Berlin Heidelberg 2014, ISBN 978-3-642-41996-6 , p. V , doi : 10.1007 / 978-3-642-41997-3 .
↑ Ehrhard Behrends: Elementary Stochastics . A learning book - co-developed by students. Springer Spectrum, Wiesbaden 2013, ISBN 978-3-8348-1939-0 , pp. 298 , doi : 10.1007 / 978-3-8348-2331-1 .
↑ AS Kholevo: Sufficient Statistic . In: Michiel Hazewinkel (Ed.): Encyclopaedia of Mathematics . Springer-Verlag , Berlin 2002, ISBN 978-1-55608-010-4 (English, online ).
^ Francisco J. Samaniego: A Comparison of the Bayesian and Frequentist Approaches to Estimation . Springer-Verlag, New York / Dordrecht / Heidelberg 2010, ISBN 978-1-4419-5940-9 , p. 21-22 , doi : 10.1007 / 978-1-4419-5941-6 .
^ Ludger Rüschendorf: Mathematical Statistics . Springer Verlag, Berlin Heidelberg 2014, ISBN 978-3-642-41996-6 , p. 249-250 , doi : 10.1007 / 978-3-642-41997-3 .

[RüschendorfV-1] Ludger Rüschendorf: Mathematical Statistics . Springer Verlag, Berlin Heidelberg 2014, ISBN 978-3-642-41996-6 , p. V , doi : 10.1007 / 978-3-642-41997-3 .

[Behrends298-2] Ehrhard Behrends: Elementary Stochastics . A learning book - co-developed by students. Springer Spectrum, Wiesbaden 2013, ISBN 978-3-8348-1939-0 , pp. 298 , doi : 10.1007 / 978-3-8348-2331-1 .

[EOMSuff-3] AS Kholevo: Sufficient Statistic . In: Michiel Hazewinkel (Ed.): Encyclopaedia of Mathematics . Springer-Verlag , Berlin 2002, ISBN 978-1-55608-010-4 (English, online ).

[Samaniego21-4] Francisco J. Samaniego: A Comparison of the Bayesian and Frequentist Approaches to Estimation . Springer-Verlag, New York / Dordrecht / Heidelberg 2010, ISBN 978-1-4419-5940-9 , p. 21-22 , doi : 10.1007 / 978-1-4419-5941-6 .

[Rüschendorf249-5] Ludger Rüschendorf: Mathematical Statistics . Springer Verlag, Berlin Heidelberg 2014, ISBN 978-3-642-41996-6 , p. 249-250 , doi : 10.1007 / 978-3-642-41997-3 .