Multitrait multimethod matrix

The term Multitrait Multi-Method matrix (or Multitrait Multi-Method analysis ; short MTMM analysis ; . Engl many features and many methods) refers to a group of statistical methods for the detection of the construct of a psychological tests are used. In general, a combination of several ( personality ) characteristics and measurement methods is used for this.

Theoretical background

A general scientific principle is that important facts (e.g. a physical constant, the functional reliability of technical systems, medical findings) are measured in different ways, if possible, in order to validate the results. This process is known as multiple operationalization .

The MTMM analysis can be used for the empirical verification (plausibility check) of psychological characteristics (theoretical constructs ) (to ensure construct validity). With what degree of agreement do different measurement methods measure a certain construct in contrast to other, unintended constructs?

As a rule, the construct validity of a new test procedure (e.g. a questionnaire) is determined by the connection ( correlation ) with other already validated test procedures that measure the same facts (construct). The correlation between a new intelligence test and another intelligence test should therefore be high so that the construct validity can be taken as given. This is known as convergent validity (a subtype of construct validity).

In 1959 the psychologists Donald T. Campbell and Donald W. Fiske published their article in which they introduced the concept of multitrait, multimethod analysis . With this method, they expanded the determination of construct validity through discriminant validity . This is based on the consideration that measurements of different properties should only be slightly related to each other (compared to measurements of the same properties). In order to ensure that a survey instrument (e.g. a questionnaire in a survey ) really captures what it is intended to capture, several of the target constructs (e.g. social attitudes , anomie , dogmatism and conservatism ) are collected.

Furthermore, Campbell and Fiske first considered in their article that every measurement consists of a measurement error and a systematic property-method unit. The new thing about this consideration was that it was not only assumed that the property has a systematic influence on the result, but also the measurement method. This method effect can have a decisive influence on the quality of conclusions. The sample survey above should not only use several constructs, but also several methods (e.g. telephone, personal and written questionnaires).

In this simple example, the results of this survey should agree largely or even very highly. Attempts at construct validation are more difficult if they include various methods of investigation of the target construct: self-assessments, behavioral observations , psychological tests, and questionnaires. In this case, a lower agreement can be expected.

Since several methods and constructs are compared in an MTMM analysis, at least two methods or constructs are required in each case.

Method effect

If a new intelligence test is designed, it is validated using other intelligence tests. The relationship between these different tests will probably be different if you use two questionnaires or if you use a language-free test and a questionnaire. The effect of this different relationship is referred to as the method effect.

Causes of method effects can be traced back to measuring instruments, assessors or situations (method variance):

Measuring instrument: Different measuring instruments are used to measure different properties. Used to protect against a measurement method-specific bias .
Appraiser: Different appraisers judge a person on the basis of several characteristics (protection against an appraiser-specific bias ).
context: A survey of several properties is carried out at different points in time (protection against a context-specific bias ).

The multitrait multimethod matrix

The agreement of the different methods and constructs can be determined empirically and expressed in coefficients of correlation . In a scheme called MTMM matrix, these coefficients are arranged in such a way that the agreement (convergent validity) of different methods for one property and at the same time their ability to distinguish (discriminant validity) with regard to other properties can be recognized. So z. B. the behavior of a school child with regard to social behavior, order and ability to concentrate can be assessed by the parents, the teachers or the child himself (external and self-assessments).

Correlation types

A distinction must be made between:

The monotrait-monomethod correlation can be found in the main diagonal of the MTMM matrix (highlighted in red). It indicates the relationship between the measurement of a property using the same method. This means that the reliabilities are entered in it, which should be as high and similar as possible.
The monotrait-heteromethod correlation in the secondary diagonals (highlighted in yellow) indicates the relationship between the measurement of a property with one method and the measurement of the same property with another method. It corresponds to the convergent validity.
The heterotrait monomethod correlation between measuring one property with one method and measuring the other property with the same method. It corresponds to the divergent validity (or discriminant validity). In the MTMM matrix, these correlations are in triangular matrices below the reliability diagonal (shaded gray).
The heterotrait-heteromethod correlation between the measurement of one property with the first method and the measurement of the other property with the other method. In the MTMM matrix below the secondary diagonal (highlighted in green).

Sample matrix

A multitrait, multimethod matrix
		Method a			Method B.			Method c
	feature	1	2	3	1	2	3	1	2	3
Method a	1	${\ displaystyle rel_ {A1, A1}}$
	2	${\ displaystyle r_ {A2, A1}}$	${\ displaystyle rel_ {A2, A2}}$
	3	${\ displaystyle r_ {A3, A1}}$	${\ displaystyle r_ {A3, A2}}$	${\ displaystyle rel_ {A3, A3}}$
Method B.	1	${\ displaystyle r_ {B1, A1}}$	${\ displaystyle r_ {B1, A2}}$	${\ displaystyle r_ {B1, A3}}$	${\ displaystyle rel_ {B1, B1}}$
	2	${\ displaystyle r_ {B2, A1}}$	${\ displaystyle r_ {B2, A2}}$	${\ displaystyle r_ {B2, A3}}$	${\ displaystyle r_ {B2, B1}}$	${\ displaystyle rel_ {B2, B2}}$
	3	${\ displaystyle r_ {B3, A1}}$	${\ displaystyle r_ {B3, A2}}$	${\ displaystyle r_ {B3, A3}}$	${\ displaystyle r_ {B3, B1}}$	${\ displaystyle r_ {B3, B2}}$	${\ displaystyle rel_ {B3, B3}}$
Method c	1	${\ displaystyle r_ {C1, A1}}$	${\ displaystyle r_ {C1, A2}}$	${\ displaystyle r_ {C1, A3}}$	${\ displaystyle r_ {C1, B1}}$	${\ displaystyle r_ {C1, B2}}$	${\ displaystyle r_ {C1, B3}}$	${\ displaystyle rel_ {C1, C1}}$
	2	${\ displaystyle r_ {C2, A1}}$	${\ displaystyle r_ {C2, A2}}$	${\ displaystyle r_ {C2, A3}}$	${\ displaystyle r_ {C2, B1}}$	${\ displaystyle r_ {C2, B2}}$	${\ displaystyle r_ {C2, B3}}$	${\ displaystyle r_ {C2, C1}}$	${\ displaystyle rel_ {C2, C2}}$
	3	${\ displaystyle r_ {C3, A1}}$	${\ displaystyle r_ {C3, A2}}$	${\ displaystyle r_ {C3, A3}}$	${\ displaystyle r_ {C3, B1}}$	${\ displaystyle r_ {C3, B2}}$	${\ displaystyle r_ {C3, B3}}$	${\ displaystyle r_ {C3, C1}}$	${\ displaystyle r_ {C3, C2}}$	${\ displaystyle rel_ {C3, C3}}$

Analysis of the MTMM matrix

Campbell and Fiske proposed the following evaluation of the MTMM matrix.

The convergent validity is fulfilled if:

the monotrait-heteromethod correlation should differ (statistically significant) from zero and be high. Campbell and Fiske did not give an absolute measure. If this condition is violated, different constructs are measured using different methods.

The discriminant validity is fulfilled if:

the heterotrait-monomethod correlation are lower than the monotrait-heteromethod correlations.
the heterotrait-heteromethod correlations are lower than the monotrait-heteromethod correlations.
the correlation coefficients are approximately the same both within a method and between the methods.

If not all criteria are met 100%, the construct validity does not have to be rejected. However, it is up to the assessor to decide when to reject the construct validity.

criticism

Since it is not clear what exactly high or low correlation coefficients are, rules of thumb must be applied here. Other methods for the separate determination of the trait-related and method-related variance, such as the latent structural equation models based on the latent state trait theory , also face this difficulty.

The empirical results of the MTMM analyzes were unsatisfactory, even disappointing, in many studies, because the coefficients of convergent validity became statistically significant, but often remained so low that none of the methods could replace the other. MTMM studies often led to the conclusion that instead of one property that is claimed to be uniform, several, relatively independent components are to be assumed (Fiske 1978). The results could differ even more if e.g. For example, for the school child named in the example, additional data such as observations of social behavior or tidiness in everyday life outside of school and independent behavior measurements would be included (see multimodal diagnostics ).

Further developments

MTMM matrices can also be evaluated using a confirmatory factor analysis. This method allows not only the separation of property, method and measurement error components, but also a check of the uncorrelatedness of the method and property factors, which is why this further development is the most frequently used variant of the MTMM analysis.

literature

Manfred Amelang, Lothar Schmidt-Atzert: Psychological diagnostics and intervention . 4th edition Springer, Berlin 2006, ISBN 978-3-540-28507-6 .
Donald T. Campbell, Donald W. Fiske: Convergent and discriminant validation by the multitrait-multimethod matrix. In: Psychological Bulletin , 1959, Volume 56, 81-105. doi : 10.1037 / h0046016
Michael Eid , Fridtjof W. Nussbeck, Tanja Lischetzke: Multitrait-Multimethod-Analysis . In Franz Petermann, Michael Eid (eds.). Manual of psychological diagnostics. Hogrefe, Göttingen 2006, ISBN 978-3-8017-1911-1 , pp. 332-345.
Donald W. Fiske: Strategies for Personality Research . Jossey-Bass, San Francisco 1978.
Hermann-Josef Fisseni: textbook of psychological diagnostics. Hogrefe, Göttingen 2004, ISBN 3-8017-0981-7 .
Schermelleh-Engel, K. & Schweizer, K. (2012) Multitrait-Multimethod-Analyzes. In Kelava, Augustin & Moosbrugger, Helfried (2012). Test theory and questionnaire construction , 2nd, updated and revised edition. Berlin, Heidelberg: Springer Berlin Heidelberg.
Schermelleh-Engel, K. & Schweizer, K. (2003). Discriminant validity. In Kubinger, Klaus D., (2003). Key terms in psychological diagnostics, 1st edition Weinheim; Basel; Berlin: Beltz.
Werner W. Wittmann: Fundamentals of successful research in psychology: multimodal diagnostics, multiplism, multivariate reliability and validity theory. In: Diagnostica, 1987, Volume 33, 209-226.

Individual evidence

↑ ^a ^b D. T. Campbell and DW Fiske (1959). Convergent and Discriminant Validation by the Multitrait Multimethod Matrix. In: Psychological Bulletin , Volume 56 (2): pp. 81-105, doi : 10.1037 / h0046016 .
^ ^A ^b Rainer Schnell, Paul B. Hill, Elke Esser: Methods of empirical social research . Oldenbourg, Munich, Vienna 2008, p. 158 f. ISBN 978-3-486-58708-1
↑ Podsakoff, PM, MacKenzie, SB, Lee, J.-Y. & Podsakoff, NP (2003). Common method biases in behavioral research: A critical review of the literature and recommended remedies. Journal of Applied Psychology , 88, 879-903, doi : 10.1037 / 0021-9010.88.5.879 .
^ Jürgen Bortz, Nicola Döring: Research methods and evaluation. Springer, Berlin 1995, p. 188. ISBN 3-540-59375-6
^ Michael Eid , Fridtjof W. Nussbeck, Tanja Lischetzke: Multitrait-Multimethod-Analyze . In Franz Petermann, Michael Eid (eds.). Manual of psychological diagnostics. Hogrefe, Göttingen 2006, ISBN 978-3-8017-1911-1 , pp. 332-345.

[campbell-1] D. T. Campbell and DW Fiske (1959). Convergent and Discriminant Validation by the Multitrait Multimethod Matrix. In: Psychological Bulletin , Volume 56 (2): pp. 81-105, doi : 10.1037 / h0046016 .

[MethSozialforschung-2] A ^b Rainer Schnell, Paul B. Hill, Elke Esser: Methods of empirical social research . Oldenbourg, Munich, Vienna 2008, p. 158 f. ISBN 978-3-486-58708-1

[3] Podsakoff, PM, MacKenzie, SB, Lee, J.-Y. & Podsakoff, NP (2003). Common method biases in behavioral research: A critical review of the literature and recommended remedies. Journal of Applied Psychology , 88, 879-903, doi : 10.1037 / 0021-9010.88.5.879 .

[4] Jürgen Bortz, Nicola Döring: Research methods and evaluation. Springer, Berlin 1995, p. 188. ISBN 3-540-59375-6

[5] Michael Eid , Fridtjof W. Nussbeck, Tanja Lischetzke: Multitrait-Multimethod-Analyze . In Franz Petermann, Michael Eid (eds.). Manual of psychological diagnostics. Hogrefe, Göttingen 2006, ISBN 978-3-8017-1911-1 , pp. 332-345.