Alpha error accumulation

The alpha error accumulation , often also called α-error inflation , describes in statistics the global increase in the alpha error probability ( error of the first type ) through multiple testing in the same sample .

To put it clearly: The more hypotheses you test on a data set, the higher the probability that one of them will be assumed to be correct (incorrectly).

Multiple testing

Often not only a null hypothesis is established in a study , but one wants to answer several questions using the data obtained. These can be further null hypotheses, but also confidence intervals or estimated values .

In the case of several null hypotheses , one speaks of a multiple test problem . ${\displaystyle {H_{0},H_{1},H_{2},\ldots ,H_{k}}\ }$

In such cases, the following two problems are encountered:

1) inconsistencies (ex.)

Suppose someone wants to compare the expected values . In the pairwise test , all null hypotheses are not rejected, only the hypothesis is rejected. ${\displaystyle {\mu _{1},\mu _{2},\mu _{3}}\ }$ ${\displaystyle {\mu _{1}=\mu _{2},\mu _{1}=\mu _{3},\mu _{2}=\mu _{3}}\ }$ ${\displaystyle {\mu _{1}=\mu _{2}=\mu _{3}}\ }$

2) Inflation of the α error

In the case of multiple test problems, a distinction is made between the local α level (only affecting the individual hypothesis) and the global α level (for the entire family of hypotheses). If the tests are independent, the local α for each null hypothesis can be adjusted on the basis of the global level using the following formula: with k = number of individual hypotheses . ${\displaystyle {\alpha _{\mathrm {lokal} }=1-(1-\alpha _{\mathrm {global} })^{1/k}}\ }$

Adjustment of the global α level

But how can one counteract or correct this α-error inflation?

Bonferroni correction

The Bonferroni correction is the simplest and most conservative way of adjusting the multiple α level. The global α level is divided equally between the individual tests:

${\displaystyle P(H_{i}{\mbox{ nicht ablehnen}}|H_{0})\leq {\frac {\alpha }{k}}}$

Using the Bonferroni inequality , it follows that each individual test is carried out below the level (and not ): For applies ${\displaystyle {\frac {\alpha }{k}}}$ ${\displaystyle \alpha }$ ${\displaystyle {i=1,\ldots ,k}\ }$

${\displaystyle P({\mbox{irgendeine }}H_{i}{\mbox{ nicht ablehnen}}|H_{0})\leq \alpha }$

The very conservative approach to the Bonferroni correction has the disadvantage that the result must have a very low p-value in order to be considered statistically significant. Further developments such as the Bonferroni-Holm procedure try to avoid this.

Bonferroni-Holm procedure

The Bonferroni-Holm procedure is an extension of the Bonferroni correction. The following algorithm is used:

Step 1:

Determination of the global α level ${\displaystyle \alpha _{g}}$

2nd step:

Execution of all individual tests and determination of the p-values

3rd step:

Sort the p-values from smallest to largest

4th step:

Calculation of the local α levels as the ratio of the global α level to the number of tests - i, where:

${\displaystyle {i=1,\ldots ,k}}$

${\displaystyle \alpha _{1}={\frac {\alpha _{g}}{k}}}$

${\displaystyle \alpha _{2}={\frac {\alpha _{g}}{k-1}}}$

${\displaystyle \alpha _{i}={\frac {\alpha _{g}}{k-i+1}}}$

5th step

Compare the p-values with the calculated sorted local α-levels (starting with ) and repeat this step until the p-value is greater than the associated value. ${\displaystyle {\alpha _{1}}\ }$ ${\displaystyle {\alpha _{i}}\ }$

6th step

All null hypotheses whose p were smaller than the local α value are rejected (meaning: the effect is significant, it is assumed that the alternative hypothesis applies). The procedure ends with the null hypothesis whose p is greater than the local α level. All of the following null hypotheses are not rejected (below the global α level).

The Bonferroni-Holm procedure is less conservative than the Bonferroni correction. Only the first test has to be statistically significant at the level required for the Bonferroni correction, after which the required level drops steadily. However, like the Bonferroni correction, this procedure also has the disadvantage that any logical and stochastic dependencies between the test statistics are not used.

Other methods

In addition to the adjustments described, there are further options for adapting to a global α level. These include, for example:

• Tukey T method
• Dunnett's procedure
• False Discovery Rate

supporting documents

1. ↑ A. Victor, A. Elsässer, G. Hommel, M. Blettner: How do you evaluate the p-value flood? In: Dtsch Arztebl Int. 107 (4), 2010, pp. 50–56 doi: 10.3238 / arztebl.2010.0050 .
2. ^ S. Holm: A simple sequentially rejective multiple test procedure. In: Scandinavian Journal of Statistics. Vol. 6, 1979, pp. 65-70.