Testing general linear hypotheses

In the test theory is the testing of general linear hypotheses , testing hypotheses linear , general linear hypothesis testing , multiple hypothesis testing the generalization of test problems in regression models. In comparison to the t test, this test procedure allows testing of several null hypotheses with regard to a group of parameters in linear single equation models. Under multiple hypothesis tests one understands on the one hand the F test for the multiple regression model , which is characterized by the fact that the test statistics of the hypothesis testfollows an F-distribution under the null hypothesis and the t test for the multiple regression model . An ordinary F test only tests a single equation.

Starting position

Since many variables of interest do not depend on just one independent variable, we consider one dependent variable that we want to explain by several independent variables. For example, the total production of an economy depends on its capital input, labor input and its area. Such a multiple dependency comes much closer to reality and one abandons the assumption of simple linear regression , in which the variable of interest only depends on one variable. To model such a multiple dependency, we consider a typical multiple linear regression model or, more precisely, a classic linear model that includes the assumption of normal distribution with given data for statistical units as the starting point . It should be noted that in addition to the dimension of the independent variables, we also integrate a time dimension, which results in a linear system of equations which can also be represented in a matrices. The relationship between the dependent variable and the independent variables can be shown as follows ${\displaystyle \{y_{i},x_{ik}\}_{i=1,\dots ,n,k=1,\dots ,K}}$ ${\displaystyle n}$

${\displaystyle y_{i}=x_{i1}\beta _{1}+x_{i2}\beta _{2}+\ldots +x_{ik}\beta _{K}+\varepsilon _{i}=\mathbf {x} _{i}^{\top }{\boldsymbol {\beta }}+\varepsilon _{i},\quad i=1,2,\dotsc ,n}$.

In vector matrix form too

${\displaystyle {\begin{pmatrix}y_{1}\\y_{2}\\\vdots \\y_{n}\end{pmatrix}}_{(n\times 1)}\quad =\quad {\begin{pmatrix}1&x_{11}&x_{12}&\cdots &x_{1K}\\1&x_{21}&x_{22}&\cdots &x_{2K}\\\vdots &\vdots &\vdots &\ddots &\vdots \\1&x_{n1}&x_{n2}&\cdots &x_{nK}\end{pmatrix}}_{(n\times p)}\quad \cdot \quad {\begin{pmatrix}\beta _{0}\\\beta _{1}\\\vdots \\\beta _{K}\end{pmatrix}}_{(p\times 1)}\quad +\quad {\begin{pmatrix}\varepsilon _{1}\\\varepsilon _{2}\\\vdots \\\varepsilon _{n}\end{pmatrix}}_{(n\times 1)}}$

or in compact notation

${\displaystyle \mathbf {y} =\mathbf {X} {\boldsymbol {\beta }}+{\boldsymbol {\varepsilon }}}$

Here represents a vector of unknown parameters (known as regression coefficients ) that must be estimated from the data. It is also assumed that the error terms are zero on average: which means that we can assume that our model is correct on average. The data matrix is assumed to have full (column) rank , that is, it applies or . Furthermore, one expects the error for the covariance matrix to be true. Furthermore, it is assumed that the Gauss-Markow assumptions apply so that the above model can be estimated efficiently and without distortion using the least squares method . ${\displaystyle {\boldsymbol {\beta }}}$ ${\displaystyle \operatorname {E} [{\boldsymbol {\boldsymbol {\varepsilon }}}]=\mathbf {0} }$ ${\displaystyle \mathbf {X} \in \mathbb {R} ^{n\times p}}$ ${\displaystyle {\mbox{Rang}}({\boldsymbol {X}})=p\;}$ ${\displaystyle \Pr \!{\big [}\,\operatorname {Rang} (\mathbf {X} )=p\,{\big ]}=1}$ ${\displaystyle {\mbox{Cov}}({\boldsymbol {\varepsilon }})=\sigma ^{2}\mathbf {I} _{n}}$

General linear hypothesis

The most general null hypothesis comprises a number of a linear restrictions on the coefficients. One can formulate this general linear hypothesis , i.e. the hypothesis that is interested in rejecting it, as

${\displaystyle H_{0}:{\boldsymbol {R}}={\begin{pmatrix}{\boldsymbol {R}}_{1}\\{\boldsymbol {R}}_{2}\\\vdots \\{\boldsymbol {R}}_{a}\end{pmatrix}}_{(a\times p)}\quad \cdot \quad {\begin{pmatrix}\beta _{0}\\\beta _{1}\\\vdots \\\beta _{k}\end{pmatrix}}_{(p\times 1)}\quad =\quad {\begin{pmatrix}r_{1}\\r_{2}\\\vdots \\r_{a}\end{pmatrix}}_{(a\times 1)}}$

respectively

${\displaystyle H_{0}:{\boldsymbol {R}}{\boldsymbol {\beta }}={\boldsymbol {r}}}$

with the -Hypothesenmatrix and the vector of the restrictions and the vector of the regression coefficients and the -Zeilenvektoren . The test problem is then ${\displaystyle \quad p=k+1}$ ${\displaystyle (a\times p)}$ ${\displaystyle {\boldsymbol {R}}}$ ${\displaystyle (a\times 1)}$ ${\displaystyle {\boldsymbol {r}}}$ ${\displaystyle (p\times 1)}$ ${\displaystyle (1\times p)}$ ${\displaystyle {\boldsymbol {R}}_{i}}$

${\displaystyle H_{0}:{\boldsymbol {R}}{\boldsymbol {\beta }}={\boldsymbol {r}}\quad {\mbox{gegen}}\quad H_{1}:{\boldsymbol {R}}{\boldsymbol {\beta }}\neq {\boldsymbol {r}}}$,

where it is assumed that or applies and linearly independent restrictions exist. As usual, the vector of the regression coefficients is assumed to be estimated using the least squares method . The following efficiency property applies to the least squares estimator ${\displaystyle \operatorname {Rang} ({\boldsymbol {R}})=a\leq p}$ ${\displaystyle \Pr \!{\big [}\,\operatorname {Rang} ({\boldsymbol {R}})=a\leq p\,{\big ]}=1}$ ${\displaystyle a}$

${\displaystyle \operatorname {Var} (\mathbf {R} _{1}\mathbf {b} )\;\leq \;\operatorname {Var} (\mathbf {R} _{1}{\overline {\boldsymbol {\beta }}})}$

This means that any parameter constellation of the least squares estimator is better than that of any linear unbiased estimator. This result is also known as the Gauss-Markow theorem.

T-test for the multiple regression model

Single equation model

In many cases one is only interested in testing a single hypothesis, e.g. B. a single linear combination of the regression coefficients. For example, one might be interested in testing whether the sum of the regression coefficients corresponds to a certain value; H.

${\displaystyle \sum \nolimits _{k=1}^{K}\beta _{k}=r_{1}}$

or in vector notation

${\displaystyle {\boldsymbol {R}}_{1}{\boldsymbol {\beta }}={\begin{pmatrix}1&\cdots &1&\cdots &1\end{pmatrix}}{\boldsymbol {\beta }}=r_{1}}$

For example, if one were to test whether the sum of the coefficients was 1; H. , then from an economic point of view this would represent a test for constant economies of scale . First of all, the attestation statistics for this test must be drawn up. Therefore, one is interested in finding out the parameterization of the distribution of the individual linear combinations. For the distribution results ${\displaystyle \sum \nolimits _{k=1}^{K}\beta _{k}=1}$

${\displaystyle {\boldsymbol {R}}_{1}{\boldsymbol {\hat {\beta }}}\sim {\mathcal {N}}({\boldsymbol {R}}_{1}{\boldsymbol {\beta }},\sigma ^{2}{\boldsymbol {R}}_{1}(\mathbf {X} ^{\top }\mathbf {X} )^{-1}{\boldsymbol {R}}_{1}^{\top })}$,

where represents the least squares estimator. One standardized to the standard normal distribution and obtained for the pivot size ${\displaystyle {\hat {\boldsymbol {\beta }}}=(\mathbf {X} ^{\top }\mathbf {X} )^{-1}\mathbf {X} ^{\top }\mathbf {y} }$ ${\displaystyle {\mathcal {N}}(0,1)}$

${\displaystyle {\frac {{\boldsymbol {R}}_{1}{\boldsymbol {\hat {\beta }}}-{\boldsymbol {R}}_{1}{\boldsymbol {\beta }}}{\sqrt {\sigma ^{2}{\boldsymbol {R}}_{1}(\mathbf {X} ^{\top }\mathbf {X} )^{-1}{\boldsymbol {R}}_{1}^{\top }}}}\;\;{\stackrel {H_{0}}{\sim }}\;\;{\mathcal {N}}(0,1)}$,

that the limits of the central fluctuation interval surround them with the probability , d. H. ${\displaystyle (1-\alpha )}$

${\displaystyle P\left(-z_{\left(1-{\frac {\alpha }{2}}\right)}<{\frac {R_{1}{\boldsymbol {\hat {\beta }}}-R_{1}{\boldsymbol {\beta }}}{\sqrt {\sigma ^{2}{\boldsymbol {R}}_{1}(\mathbf {X} ^{\top }\mathbf {X} )^{-1}{\boldsymbol {R}}_{1}^{\top }}}}<z_{\left(1-{\frac {\alpha }{2}}\right)}\right)=1-\alpha }$,

where the - is the quantile of the standard normal distribution . The problem with this expression is that the variance of the confounders is usually unknown. If the unknown parameter is replaced by the unbiased estimator for the disturbance variable variance, the following distribution results for the pivot variable ${\displaystyle z_{\left(1-{\frac {\alpha }{2}}\right)}}$ ${\displaystyle (1-{\tfrac {\alpha }{2}})}$ ${\displaystyle \sigma ^{2}=\sigma _{\varepsilon }^{2}}$ ${\displaystyle {\hat {\sigma }}^{2}={\tfrac {1}{n-k-1}}\sum \nolimits _{i=1}^{n}{\hat {\varepsilon }}_{i}^{2}}$

${\displaystyle {\frac {R_{1}{\boldsymbol {\hat {\beta }}}-R_{1}{\boldsymbol {\beta }}}{\sqrt {{\hat {\sigma }}^{2}{\boldsymbol {R}}_{1}(\mathbf {X} ^{\top }\mathbf {X} )^{-1}{\boldsymbol {R}}_{1}^{\top }}}}\sim {\mathcal {t}}(n-p)}$.

The pivot size is now instead of the standard normal distribution , t -distributed with straight lines of freedom. This results in the following probability for testing the individual equation ${\displaystyle (n-p)}$

${\displaystyle P\left({R_{1}{\boldsymbol {\hat {\beta }}}-t_{\left(1-{\frac {\alpha }{2}}\right)}(n-p){\sqrt {{\hat {\sigma }}^{2}{\boldsymbol {R}}_{1}(\mathbf {X} ^{\top }\mathbf {X} )^{-1}{\boldsymbol {R}}_{1}^{\top }}}}<R_{1}{\boldsymbol {\beta }}<{R_{1}{\boldsymbol {\hat {\beta }}}-t_{\left(1-{\frac {\alpha }{2}}\right)}(n-p){\sqrt {{\hat {\sigma }}^{2}{\boldsymbol {R}}_{1}(\mathbf {X} ^{\top }\mathbf {X} )^{-1}{\boldsymbol {R}}_{1}^{\top }}}}\right)=1-\alpha }$

and thus the following confidence interval

${\displaystyle KI_{1-\alpha }(R_{1}{\boldsymbol {\beta }})={R_{1}{\boldsymbol {\hat {\beta }}}\pm t_{\left(1-{\frac {\alpha }{2}}\right)}(n-p){\sqrt {{\hat {\sigma }}^{2}{\boldsymbol {R}}_{1}(\mathbf {X} ^{\top }\mathbf {X} )^{-1}{\boldsymbol {R}}_{1}^{\top }}}}}$.

Single equation models can be represented not only as an F test for the multiple regression model, but alternatively also as a t test.

F-test for the multiple regression model

Construction of the test statistic

For the construction of the test statistic , one uses the following result, which is easy to check with the help of the assumption of the faithfulness to expectations of the least squares estimator and the calculation rules for covariance matrices

${\displaystyle {\boldsymbol {R}}{\boldsymbol {\hat {\beta }}}-{\boldsymbol {r}}\sim {\mathcal {N}}({\boldsymbol {R}}{\boldsymbol {\beta }}-{\boldsymbol {r}},\sigma ^{2}{\boldsymbol {R}}(\mathbf {X} ^{\top }\mathbf {X} )^{-1}{\boldsymbol {R}}^{\top })}$,

d. H. In the present classic model, the null hypothesis follows a normal distribution with a covariance matrix and expected value . ${\displaystyle \sigma ^{2}{\boldsymbol {R}}(\mathbf {X} ^{\top }\mathbf {X} )^{-1}{\boldsymbol {R}}^{\top }}$ ${\displaystyle {\boldsymbol {R}}{\boldsymbol {\beta }}-{\boldsymbol {r}}}$

It can be shown that the weighted hypothesis sum of squares under the null hypothesis

${\displaystyle Q_{1}={\frac {1}{\sigma ^{2}}}({\boldsymbol {R}}{\boldsymbol {\hat {\beta }}}-{\boldsymbol {r}})^{\top }({\boldsymbol {R}}(\mathbf {X} ^{\top }\mathbf {X} )^{-1}{\boldsymbol {R}}^{\top })^{-1}({\boldsymbol {R}}{\boldsymbol {\hat {\beta }}}-{\boldsymbol {r}})\sim \chi ^{2}(a)}$

follows a chi-square distribution with straight lines of freedom. This measures how far the estimated value deviates from the null hypothesis . Furthermore, the corresponding sum of the squared deviations is (analogous to the residual sum of squares ). This sum of the squared deviations is weighted with the inverse covariance matrix , because deviations that are just as large are not necessarily an indicator for a large covariance . Another important result that is needed to construct the test statistic is ${\displaystyle a}$ ${\displaystyle {\boldsymbol {R}}{\boldsymbol {\hat {\beta }}}-{\boldsymbol {r}}}$ ${\displaystyle {\boldsymbol {\hat {\beta }}}}$ ${\displaystyle {\boldsymbol {R}}{\boldsymbol {\beta }}-{\boldsymbol {r}}=\mathbf {0} }$ ${\displaystyle ({\boldsymbol {R}}{\boldsymbol {\beta }}-{\boldsymbol {r}})^{\top }({\boldsymbol {R}}{\boldsymbol {\beta }}-{\boldsymbol {r}})}$ ${\displaystyle ({\boldsymbol {R}}(\mathbf {X} ^{\top }\mathbf {X} )^{-1}{\boldsymbol {R}}^{\top })^{-1}/\sigma ^{2}}$ ${\displaystyle {\boldsymbol {R}}{\boldsymbol {\hat {\beta }}}-{\boldsymbol {r}}}$ ${\displaystyle H_{0}}$

${\displaystyle Q_{2}={\frac {(n-p){\hat {\sigma }}^{2}}{\sigma ^{2}}}\sim \chi ^{2}(n-p)}$

The test statistic now results with stochastic independence from and as ${\displaystyle Q_{1}}$ ${\displaystyle Q_{2}}$

${\displaystyle {\begin{aligned}F\;\;{\stackrel {H_{0}}{=}}{\frac {Q_{1}/a}{Q_{2}/(n-p)}}={\frac {{\frac {1}{a}}({\boldsymbol {R}}{\boldsymbol {\hat {\beta }}}-{\boldsymbol {r}})^{\top }({\boldsymbol {R}}(\mathbf {X} ^{\top }\mathbf {X} )^{-1}{\boldsymbol {R}}^{\top })^{-1}({\boldsymbol {R}}{\boldsymbol {\hat {\beta }}}-{\boldsymbol {r}})}{{\hat {\sigma }}^{2}}}\;\;{\stackrel {H_{0}}{\sim }}\;\;F(a,n-p)\end{aligned}}}$.

From this result it can be seen that the test statistic can alternatively be expressed as the quotient of the "mean hypothesis square " and the " mean residual square "

${\displaystyle MQH\equiv {\frac {SQH}{\operatorname {Rang} ({\boldsymbol {R}})}}\quad }$and , ${\displaystyle \quad MQR\equiv {\frac {SQR}{\operatorname {Rang} ({\boldsymbol {Q}})}}}$

can be represented as

${\displaystyle {\begin{aligned}F\;\;{\stackrel {H_{0}}{=}}{\frac {MQH}{MQR}}={\frac {{\frac {1}{\operatorname {Rang} ({\boldsymbol {R}})}}({\boldsymbol {R}}{\boldsymbol {\hat {\beta }}}-{\boldsymbol {r}})^{\top }({\boldsymbol {R}}(\mathbf {X} ^{\top }\mathbf {X} )^{-1}{\boldsymbol {R}}^{\top })^{-1}({\boldsymbol {R}}{\boldsymbol {\hat {\beta }}}-{\boldsymbol {r}})}{{\frac {1}{\operatorname {Rang} ({\boldsymbol {Q}})}}({\boldsymbol {y}}-\mathbf {X} {\boldsymbol {\hat {\beta }}})^{\top }({\boldsymbol {y}}-\mathbf {X} {\boldsymbol {\hat {\beta }}})}}\end{aligned}}}$,

where represents the rank of the residue generating matrix and represents the rank of the hypothesis matrix. If you divide the sums of squares by (or ), you get mean squares of deviations . This makes sense, since larger deviations are to be expected for more hypotheses (observations). These test statistic provides the framework and basis for testing general linear hypotheses and interval estimators for the unknown vector . As usual, these test statistic are sensitive to the test problem; H. if the deviation is large relative to the error variance , this speaks against it ${\displaystyle \operatorname {Rang} ({\boldsymbol {Q}})}$ ${\displaystyle \operatorname {Rang} ({\boldsymbol {R}})}$ ${\displaystyle a}$ ${\displaystyle n-p}$ ${\displaystyle {\boldsymbol {R}}{\boldsymbol {\beta }}}$ ${\displaystyle {\boldsymbol {R}}{\boldsymbol {\hat {\beta }}}-{\boldsymbol {r}}}$ ${\displaystyle H_{0}}$

To finally carry out the test, one uses the corresponding quantiles of the F distribution. The null hypothesis is rejected if

${\displaystyle F>F(a,n-p)}$,

the F statistic is therefore greater than the critical value . The critical value can be read from a quantile table for the F distribution. ${\displaystyle F(a,n-p)}$

Multiple testing problem

→ Main article : Alpha error accumulation

The problem with multiple testing is that with each individual test at a significance level of 5%, you have a 5% probability of incorrectly finding out the difference you are looking for (alpha error) and these errors add up due to multiple null hypotheses. The false detection rate plays an important role in this context ; it is defined for a test procedure as the expected ratio of incorrectly rejected null hypotheses to the rejected null hypotheses.

Web links

• Multiple Hypothesis Testing: The F-test
• Quantile table of the F distribution

literature

• Ludwig Fahrmeir , Thomas Kneib , Stefan Lang, Brian Marx: Regression: models, methods and applications. Springer Science & Business Media, 2013, ISBN 978-3-642-34332-2
• George G. Judge, R. Carter Hill, W. Griffiths, Helmut Lütkepohl , TC Lee. Introduction to the Theory and Practice of Econometrics. 2nd Edition. John Wiley & Sons, New York / Chichester / Brisbane / Toronto / Singapore 1988, ISBN 0-471-62414-4

Individual evidence

1. ^ Ludwig Fahrmeir , Thomas Kneib , Stefan Lang, Brian Marx: Regression: models, methods and applications. Springer Science & Business Media, 2013, ISBN 978-3-642-34332-2 , p. 296 ff.
2. ^ Ludwig Fahrmeir , Thomas Kneib , Stefan Lang, Brian Marx: Regression: models, methods and applications. Springer Science & Business Media, 2013, ISBN 978-3-642-34332-2 , p. 285 ff.
3. ↑ George G. Judge, R. Carter Hill, W. Griffiths, Helmut Lütkepohl , TC Lee. Introduction to the Theory and Practice of Econometrics. 2nd Edition. John Wiley & Sons, New York / Chichester / Brisbane / Toronto / Singapore 1988, ISBN 0-471-62414-4 , p. 242 ff.