Estimating the variance of an estimator

This article or the following section is not adequately provided with supporting documents ( e.g. individual evidence ). Information without sufficient evidence could be removed soon. Please help Wikipedia by researching the information and including good evidence.

This article describes variance estimation in terms of estimating the variance of an estimator of an unknown population parameter. To estimate the population variance, see Estimating Population Variance .

As estimate of the variance of an estimation function in the statistics to estimate the variance of a is estimation function , an unknown parameter of the population, respectively. This estimation is a method of measuring the accuracy of estimation procedures. It allows the construction of confidence intervals ( interval estimation ).

If you have an estimate function for an unknown parameter of the population, you initially only have a point estimate for this. However, one is interested in also giving confidence intervals for the estimated parameter, i.e. H. one has to know the distribution and the variance of . ${\displaystyle {\hat {\theta }}}$ ${\displaystyle \theta }$ ${\displaystyle {\hat {\vartheta }}}$ ${\displaystyle {\hat {\theta }}}$

However, this is not always possible and therefore there are different procedures:

• direct methods based on the likelihood function ,
• linear approximation of the log-likelihood function and
• Resampling methods.

If the estimator was calculated using the maximum likelihood method , one knows about the asymptotic behavior: ${\displaystyle {\hat {\theta }}}$

• ${\displaystyle \lim _{n\to \infty }{\hat {\theta }}\longrightarrow {\mathcal {N}}(\theta ;\Sigma _{\hat {\theta }})}$ such as
• ${\displaystyle \lim _{n\to \infty }Var({\hat {\theta }})\longrightarrow {\mathcal {I}}^{-1}(\theta )=\Sigma _{\hat {\theta }}}$

with the covariance matrix of the estimator (s) and the Fisher information matrix . ${\displaystyle \Sigma _{\hat {\theta }}}$ ${\displaystyle {\hat {\theta }}}$ ${\displaystyle I(\theta )}$

Known distribution of ${\displaystyle {\hat {\theta }}}$

If the distribution of can be determined at least approximately, for example with the aid of the central limit theorem , then the variance can easily be estimated. ${\displaystyle {\hat {\theta }}}$

An example is the sample mean of a normally distributed population or, if the central limit theorem is valid, for any distribution in the population:

${\displaystyle {\bar {X}}={\frac {1}{n}}\sum _{i=1}^{n}X_{i}\approx {\mathcal {N}}\left(\mu ;{\frac {\sigma ^{2}}{n}}\right)}$.

The confidence interval can be derived from this

${\displaystyle P\left({\bar {X}}-z_{1-\alpha /2}{\frac {\sigma }{\sqrt {n}}}\leq \mu \leq {\bar {X}}+z_{1-\alpha /2}{\frac {\sigma }{\sqrt {n}}}\right)\approx 1-\alpha }$

with from the standard normal distribution . ${\displaystyle z_{1-\alpha /2}}$

Direct procedure

The representation is used for direct methods

${\displaystyle \operatorname {Var} ({\hat {\theta }})=\int ({\hat {\theta }}-\theta )^{2}L(x_{1},\ldots ,x_{n}|\theta )dx_{1}\ldots dx_{n}}$ or multivariate ${\displaystyle \sigma _{ij}({\hat {\theta }})=\int ({\hat {\theta }}_{i}-\theta _{i})({\hat {\theta }}_{j}-\theta _{j})L(x_{1},\ldots ,x_{n}|\theta )dx_{1}\ldots dx_{n}}$

Variance estimates based on this can usually only be given for simple point estimates. Here, approximation formulas are only required for sample designs with second-order inclusion probabilities . Exact methods, i.e. formulas that are easy to calculate, can be given in the case of a linear estimator.

However, neither the true parameter nor the function are known. Therefore, the estimated values ​​and the normalized likelihood function are used as the probability density for : ${\displaystyle \theta }$ ${\displaystyle L(x_{1},\ldots ,x_{n}|\theta )}$ ${\displaystyle \theta }$

${\displaystyle {\widehat {\operatorname {Var} }}({\hat {\theta }})={\frac {\int (\theta -{\hat {\vartheta }})^{2}L(x_{1},\ldots ,x_{n}|\theta )d\theta }{\int L(x_{1},\ldots ,x_{n}|\theta )d\theta }}}$ or multivariate ${\displaystyle {\hat {\sigma }}_{ij}({\hat {\theta }})={\frac {\int (\theta _{i}-{\hat {\vartheta }}_{i})(\theta _{j}-{\hat {\vartheta }}_{j})L(x_{1},\ldots ,x_{n}|\theta )d\theta _{1}\ldots d\theta _{m}}{\int L(x_{1},\ldots ,x_{n}|\theta )d\theta _{1}\ldots d\theta _{m}}}}$

The estimate is then made with the help of numerical integration .

Linear approximation

In the case of non-linear estimators (e.g. a ratio estimator ), approximate methods are used. Can the log-likelihood function be expanded around the maximum using the Taylor approximation

${\displaystyle \log(L(x_{1},\ldots ,x_{n}|\theta ))\approx \log(\underbrace {L(x_{1},\ldots ,x_{n}|{\hat {\vartheta }})} _{=L_{max}})+\underbrace {\left(\theta -{\hat {\vartheta }}\right)\left.{\frac {\partial \log(L)}{\partial \theta }}\right|_{\theta ={\hat {\vartheta }}}} _{=0}+{\tfrac {1}{2}}\left(\theta -{\hat {\vartheta }}\right)^{2}\left.{\frac {\partial ^{2}\log(L)}{\partial \theta ^{2}}}\right|_{\theta ={\hat {\vartheta }}}}$

and taking advantage of the Fisher information matrix definition

${\displaystyle \log(L(x_{1},\ldots ,x_{n}|\theta ))\approx \log(L_{max})-{\tfrac {1}{2}}\left(\theta -{\hat {\vartheta }}\right)^{2}\sigma _{\hat {\theta }}^{-1}}$

follows

${\displaystyle {\hat {\sigma }}_{\hat {\theta }}=\left(-\left.{\frac {\partial ^{2}\log(L)}{\partial \theta ^{2}}}\right|_{\theta ={\hat {\vartheta }}}\right)^{-1}}$.

Alternatively, Woodruff linearization can be used to convert non-linear estimators to linear ones.

Resampling methods

Another possibility is resampling methods such as the bootstrapping process . Here, sub-samples are drawn at random from the existing sample and an estimated value is calculated with them. These estimates are an empirical approximation to the unknown distribution of . ${\displaystyle B}$ ${\displaystyle {\hat {\vartheta }}^{(i)}}$ ${\displaystyle {\hat {\theta }}}$

Sample:	${\displaystyle x_{1},\ldots x_{n}}$	${\displaystyle \longrightarrow {\hat {\vartheta }}}$
Subsample 1:	${\displaystyle x_{1}^{(1)},\ldots x_{n}^{(1)}}$	${\displaystyle \longrightarrow {\hat {\vartheta }}^{(1)}}$
${\displaystyle \vdots }$	${\displaystyle \vdots }$	${\displaystyle \vdots }$
Sub-sample B:	${\displaystyle x_{1}^{(B)},\ldots x_{n}^{(B)}}$	${\displaystyle \longrightarrow {\hat {\vartheta }}^{(B)}}$

Hence it results

${\displaystyle {\widehat {\operatorname {Var} }}({\hat {\theta }})={\frac {1}{B-1}}\sum _{i=1}^{B}\left({\hat {\vartheta }}^{(i)}-{\bar {\vartheta }}\right)^{2}}$

with . During the estimation, the sample design can be taken into account by weighting. ${\displaystyle {\bar {\vartheta }}={\frac {1}{B}}\sum _{i=1}^{B}{\hat {\vartheta }}^{(i)}}$