M-estimator

M-estimators , also maximum likelihood-like estimators, represent a class of estimation functions that can be viewed as a generalization of the maximum likelihood method . Compared to other estimators such as B. the maximum likelihood estimators more robust against outliers .

This article deals with M-estimators for determining the location parameter .

Derivation by generalization of the maximum likelihood method

The principle of maximum likelihood estimators is based on the function

${\displaystyle \sum _{i=1}^{n}-\ln f_{X_{i}}(x_{i};\Theta )}$

with the corresponding density or probability function depending on . ${\displaystyle f_{X}(x)}$ ${\displaystyle \Theta }$

The idea with M-estimators is to replace the function with a function that is less sensitive to outliers. The job is the expression ${\displaystyle -\ln f_{X_{i}}(x_{i};\Theta )}$ ${\displaystyle \rho (x;\Theta )}$

${\displaystyle \sum _{i=1}^{n}\rho (x_{i};\Theta )}$

depending on to minimize or the equation ${\displaystyle \Theta }$

${\displaystyle \sum \psi (x_{i};\Theta )=0}$

With

${\displaystyle \psi (x_{i};\Theta )={\frac {\partial \rho }{\partial \Theta }}(x_{i};\Theta )}$

to solve.

Each solution to this equation is called an M-estimator.

Implicit definition

Let an arbitrary distribution function and an odd and monotonically increasing function not equal to 0. Then is defined as the solution of the equation ${\displaystyle F}$ ${\displaystyle \psi }$ ${\displaystyle \mu _{\psi }(F)}$ ${\displaystyle \mu =\mu _{\psi }(F)}$

${\displaystyle \operatorname {E} (\psi (x-\mu ))=\int \psi (x-\mu )dF(x)=0}$

It should be noted that, depending on the choice of and, there can be either none, one or more solutions. In the case of a specific sample , the solution is ${\displaystyle \psi }$ ${\displaystyle F}$ ${\displaystyle \mu =\mu _{\psi }(F_{n})}$

${\displaystyle {\frac {1}{n}}\sum _{i=1}^{n}\psi (x_{i}-\mu )=\int \psi (x-\mu )dF_{n}(x)=0}$

Called M-estimator.

Suitable functions ρ

The following are the according to ${\displaystyle x_{i}}$

${\displaystyle z_{i}={\frac {x_{i}-\Theta }{S_{n}}}}$

standardized to achieve scale invariance . represents a dispersion estimator for which the MAD (Median Absolute Deviation) is mostly used. ${\displaystyle S_{n}}$

method	${\displaystyle \rho (z)}$	${\displaystyle \psi (z)}$	${\displaystyle w(z)}$
Least Squares Method	${\displaystyle \rho _{LS}(z)={\frac {z^{2}}{2}}}$	${\displaystyle \psi _{LS}(z)=z}$	${\displaystyle w_{LS}(z)=1}$
Huber k estimator	${\displaystyle \rho _{H}(z)={\begin{cases}{\frac {1}{2}}z^{2}&|z|\leq {}k\\k|z|-{\frac {1}{2}}k^{2}&|z|>k\end{cases}}}$	${\displaystyle \psi _{H}(z)={\begin{cases}z&|z|\leq {}k\\k\operatorname {sgn} (z)&|z|>k\end{cases}}}$	${\displaystyle w_{H}(z)={\begin{cases}1&|z|\leq {}k\\{\frac {k}{|z|}}&|z|>k\end{cases}}}$
Fine art estimator	${\displaystyle \rho _{Ha}(z)={\begin{cases}{\frac {z^{2}}{2}}&|z|\leq {}a\\a|z|-{\frac {a^{2}}{2}}&a<|z|\leq b\\ab-{\frac {a^{2}}{2}}+(c-b){\frac {a}{2}}\left(1-\left({\frac {c-|z|}{c-b}}\right)^{2}\right)&b<|z|\leq c\\ab-{\frac {a^{2}}{2}}+(c-b){\frac {a}{2}}&|z|>c\end{cases}}}$	${\displaystyle \psi _{Ha}(z)={\begin{cases}z&|z|\leq {}a\\a\,\operatorname {sgn} (z)&a<|z|\leq b\\a{\frac {c-|z|}{c-b}}\operatorname {sgn} (z)&b<|z|\leq c\\0&|z|>c\end{cases}}}$	${\displaystyle w_{Ha}(z)={\begin{cases}1&|z|\leq {}a\\a{\frac {1}{|z|}}&a<|z|\leq b\\a{\frac {c-|z|}{c-b}}{\frac {1}{|z|}}&b<|z|\leq c\\0&|z|>c\end{cases}}}$
Andrews wave	${\displaystyle \rho _{Aw}(z)={\begin{cases}{\frac {a^{2}}{\pi ^{2}}}\left(1-\cos \left({\frac {\pi z}{a}}\right)\right)&|z|\leq {}a\\{\frac {2a^{2}}{\pi ^{2}}}&|z|>a\end{cases}}}$	${\displaystyle \psi _{Aw}(z)={\begin{cases}{\frac {a}{\pi }}\sin \left({\frac {\pi z}{a}}\right)&|z|\leq {}a\\0&|z|>a\end{cases}}}$	${\displaystyle w_{Aw}(z)={\begin{cases}{\frac {a}{\pi z}}\sin \left({\frac {\pi z}{a}}\right)&|z|\leq {}a\\0&|z|>a\end{cases}}}$
Tukey's biweight	${\displaystyle \rho _{Tb}(z)={\begin{cases}{\frac {a^{2}}{6}}\left(1-\left(1-{\frac {z^{2}}{a^{2}}}\right)^{3}\right)&|z|\leq {}a\\{\frac {a^{2}}{6}}&|z|>a\end{cases}}}$	${\displaystyle \psi _{Tb}(z)={\begin{cases}z\left(1-{\frac {z^{2}}{a^{2}}}\right)^{2}&|z|\leq {}a\\0&|z|>a\end{cases}}}$	${\displaystyle w_{Tb}(z)={\begin{cases}\left(1-{\frac {z^{2}}{a^{2}}}\right)^{2}&|z|\leq {}a\\0&|z|>a\end{cases}}}$

The weight functions in the following figure show the differences between the estimators: with Huber-k, even extreme observations have a low weight, with the Hampel, Andrews wave and Tukey's biweight estimators, extreme observations are assigned the weight zero.

Weight functions w (z) for different M-estimators. The parameter values ​​correspond to the standard values ​​of SPSS.

robustness

With a suitable choice of (even, bounded and monotonically increasing), M-estimators have a breakpoint of ${\displaystyle \psi }$ ${\displaystyle \epsilon ^{*}=0.5}$

Numerical solution method

For many functions no explicit solution can be given, so it has to be calculated numerically. As usual for calculating zero problems, the Newton-Raphson method is also available here , and the following iteration rule results, where again  : ${\displaystyle \rho }$ ${\displaystyle z_{i}={\frac {x_{i}-\mu }{S_{n}}}}$

${\displaystyle \mu _{k+1}=\mu _{k}+{\frac {S_{n}\sum _{i=1}^{n}\psi (z_{i})}{\sum _{i=1}^{n}\psi ^{\prime }(z_{i})}}}$

The median is usually used as a suitable starting value . This iteration method converges very quickly, usually two to three iteration steps are sufficient. ${\displaystyle \mu _{0}}$

W estimator

W-estimators are very similar to M-estimators and usually deliver the same results. The only difference is in the solution to the minimization problem. W-estimators are mostly used in robust regression .

It becomes the weighting function

${\displaystyle w(z)={\frac {\psi (z)}{z}}}$

With

${\displaystyle \psi (x_{i};\Theta )={\frac {\partial \rho }{\partial \Theta }}(x_{i};\Theta )}$

introduced, with the help of which the minimization problem can be rewritten in

${\displaystyle \sum _{i=1}^{n}z_{i}w(z_{i})=0}$

Inserting the definition of , multiplying and rearranging finally results in the fixed point equation ${\displaystyle z_{i}}$

${\displaystyle \Theta ={\frac {\sum _{i=1}^{n}x_{i}w({\frac {x_{i}-\Theta }{S_{n}}})}{\sum _{i=1}^{n}w({\frac {x_{i}-\Theta }{S_{n}}})}}}$

the iteration rule

${\displaystyle \Theta _{t+1}={\frac {\sum _{i=1}^{n}x_{i}w({\frac {x_{i}-\Theta _{t}}{S_{n}}})}{\sum _{i=1}^{n}w({\frac {x_{i}-\Theta _{t}}{S_{n}}})}}}$

See also

• So-called RANSAC algorithms

literature

• Robert G. Staudte: Robust estimation and testing . Wiley, New York 1990. ISBN 0-471-85547-2
• Rand R. Wilcox: Introduction to robust estimation and hypothesis testing . Academic Press, San Diego Cal 1997. ISBN 0-12-751545-3