Variance of the mean and predicted responses

In linear regression mean response and predicted response are values of the dependent variable calculated from the regression parameters and a given value of the independent variable. The values of these two responses are the same, but their calculated variances are different.

Straight line regression

In straight line fitting the model is

$y_{i}=\alpha +\beta x_{i}+\epsilon \,$

where $y_{i}$ is a dependent variable, $x_{i}$ is an independent variable and $\alpha$ and $\beta$ are parameters. The predicted response value for a given value, x_d, of the independent variable is given by

$y_{d}={\hat {\alpha }}+{\hat {\beta }}x_{d}$

Expressions for the values and variances of ${\hat {\alpha }}{\text{ and }}{\hat {\beta }}$ are given in linear regression.

Mean response is an estimate of the mean of the y population associated with x_d, that is $E(y|x_{d})\!$. The variance of the mean response is given by error propagation as

${\text{Var}}\left({\hat {\alpha }}+{\hat {\beta }}x_{d}\right)={\text{Var}}\left({\hat {\alpha }}\right)+\left({\text{Var}}{\hat {\beta }}\right)x_{d}^{2}+2x_{d}{\text{Cov}}\left({\hat {\alpha }},{\hat {\beta }}\right)$

This expression can be simplified to

${\text{Var}}\left({\hat {\alpha }}+{\hat {\beta }}x_{d}\right)=\sigma ^{2}\left({\frac {1}{m}}+{\frac {\left(x_{d}-{\bar {x}}\right)^{2}}{\sum (x_{i}-{\bar {x}})^{2}}}\right)$

To demonstrate this simplification, one can make use of the identity

$\sum (x_{i}-{\bar {x}})^{2}=\sum x_{i}^{2}-{\frac {1}{m}}\left(\sum x_{i}\right)^{2}$

Predicted response is the expected range of values of y at some confidence level. The predicted response distribution is the predicted distribution of the residuals at the given point x_d. So the variance is given by

${\text{Var}}\left(y_{d}-\left[{\hat {\alpha }}+{\hat {\beta }}x_{d}\right]\right)={\text{Var}}\left(y_{d}\right)+{\text{Var}}\left({\hat {\alpha }}+{\hat {\beta }}x_{d}\right)$

The second part of this expression was already calculated for the mean response. Since ${\text{Var}}\left(y_{d}\right)=\sigma ^{2}$, the variance of the predicted response is given by

${\text{Var}}\left(y_{d}-\left[{\hat {\alpha }}+{\hat {\beta }}x_{d}\right]\right)=\sigma ^{2}+\sigma ^{2}\left({\frac {1}{m}}+{\frac {\left(x_{d}-{\bar {x}}\right)^{2}}{\sum (x_{i}-{\bar {x}})^{2}}}\right)=\sigma ^{2}\left(1+{\frac {1}{m}}+{\frac {\left(x_{d}-{\bar {x}}\right)^{2}}{\sum (x_{i}-{\bar {x}})^{2}}}\right)$

Confidence intervals

The $100(1-\alpha )\%$ confidence intervals are computed as $y_{d}\pm t_{{\frac {\alpha }{2}},m-n-1}{\sqrt {\text{Var}}}$. Thus, the confidence interval for predicted response is wider than the interval for mean response. This is expected intuitively – the variance population of $y$ values does not shrink when one samples from it, but the variance mean of the $y$ does shrink with increased sampling. This is analogous to the difference between the variance of a population and the variance of the mean of a population.

General linear regression

The general linear model can be written as

$y_{i}=\sum _{j=1}^{j=n}X_{ij}\beta _{j}+\epsilon \,$

Therefore since $y_{d}=\sum _{j=1}^{j=n}X_{dj}{\hat {\beta }}_{j}$ the general expression for the variance of the mean response is

${\text{Var}}\left(\sum _{j=1}^{j=n}X_{dj}{\hat {\beta }}_{j}\right)=\sum _{i=1}^{i=n}\sum _{j=1}^{j=n}X_{di}M_{ij}X_{dj}$

where M is the covariance matrix of the parameters, $\sigma ^{2}\left(\mathbf {X^{T}X} \right)^{-1}$.

References

Draper, N.R., Smith, H. (1998) Applied Regression Analysis. Wiley. ISBN 0-471-17082-8