MTMM - class 8

MTMM (gathering evidence of construct validity) (3)

multitrait-multimethod matrix (MTMM): a way of representing relations btwn several constructs and several methods for measuring these constructs

*way of showing relationships btwn diff constructs and diff methods measuring these constructs

MTMM is a statistical method used to evaluate the convergent and divergent (discriminant) validity of a measure.

It also helps researchers detect and account for method effects (i.e., variance due to the measurement method rather than the construct itself).

Breakdown of “Multitrait-Multimethod” (3)

Multi-Trait. Refers to measuring multiple constructs/traits (e.g., anxiety, depression, dominance).

Multi-Method. Refers to using different types of measurement methods, such as:

Self-report questionnaires, Behavioral observations, Peer ratings, Interviews, Other reporting formats

convergent and divergent validity in the MTMM (3)

both convergent and divergent evidence of validity is needed for the MTMM 

test scores are correlated with other measure of the same constuct (convergent) and uncorrelated with meaasures of constructs from which the construct intended to differ (divergent) 

Construct–Method Units (3)

Every test is a construct–method unit.

This means it combines:

Construct content (what you’re trying to measure), and

Measurement procedures (the method used to measure it), which are not unique to that construct.

(Ex. Measuring neuroticism using a self-report questionnaire)

Components of Test Score Variance (4)

Whenever we measure a psychological trait using any method, the test score variance (aka observed test score) reflects a combination of three sources of variance:

1)True Score Variance. Variance due to real differences in the construct (e.g., actual extraversion levels).

2)Method Variance (systematic error). Variance caused by the measurement method rather than the actual construct. Also referred to as systematic error.

Example: Using a self-report questionnaire to measure extraversion introduces method-specific influences like response bias or item wording.

3)Random Error Variance. Variance due to unpredictable, non-systematic influences.

*So, every test score is a mix of these three sources of variance.

Method variance (ex. Acquiescence bias) (2)

RE: Method variance = variance due to the measurement method, not the construct.

ex. Measuring extraversion using a self-report questionnaire with 3 positively worded items: “I feel energized around others,”“I often take the lead in social interactions,” “I enjoy being the center of attention.”

If some people tend to agree with any positively worded statements (regardless of their true extraversion level), this introduces: Acquiescence bias → a form of method variance.

So, scores include: Construct-related variance (true extraversion) + Method variance (response style, wording) + Random error

Since each method introduces its own bias, MTMM is needed to isolate and detect method variance *from true variance

Why Use MTMM to Address Method Variance? (2)

MTMM helps us:

Separate method variance from true construct variance. *how much variance is due what a test/measure introduces into its observed scores vs is from true construct

Compare the same trait across different methods, which makes method effects visible. *By using diff methods, we can isolate method variance that comes from each method

Show how much variance is due to the method *test itself (e.g., self-report vs peer ratings).

What MTMM (Multitrait-Multimethod) Does (3)

MTMM allows us to assess:

✅ Convergent validity (same trait, different methods)

✅ Divergent validity (different traits)

✅ Method variance (variance tied to the measurement method)

• Matrix is imp bc shows correlations in patterned fashion, and that pattern allows us to spot the validity of each scale or measure

• Matrix allows us to discern diff aspects of validity of the scale

what does MTMM look for in terms of variance? (3)

1)high ‘trait variance”: variance due to construct (true trait signal)

2)low ‘method variance”: variance due to method (method bias kept minimal)

3)low “irrelevant variance": variance shared with theoretically unrelated measures (good discriminant validity)

how does low ‘method variance” support discriminant validity? (4)

*discriminant validity: unrelated constructs should have low correlations with one another

if two different constructs are measured using the same method, they might appear correlated not because the constructs are actually related, but because of method bias *as a result of being measured by the same method.

This *method bias artificially inflates correlations, making it harder to show discriminant validity.

Low method variance supports discriminant validity because it ensures that any correlations between measures reflect the constructs themselves, not the measurement method. *by having the least amount of method bias as possible

Why is trait variance important in psychological measurement? (4)

The goal of measurement is to see how people differ on a specific attribute. We want scores to reflect a person’s true standing on the trait (e.g., true extraversion).

In practice, scores often include unwanted influences (e.g., acquiescence bias), so they become a mix of: /observed test score vairance = true trait variance + Method/bias variance + random error

This “contaminates” the score and reduces its validity.

When measuring a group, differences in scores should reflect only true differences in the trait—nothing else.

Therefore, trait variance should represent real variability in the construct, not method effects or irrelevant influences.

1. High Trait Variance (3)

➡ Variance due to the construct itself

This is the good variance—it reflects the true differences in the trait or construct being measured.

Example: If two different methods measure extroversion and the scores are strongly correlated, that signals convergent validity.

Goal: You want a high amount of variance that genuinely comes from the trait, not outside influences.

2. Low Method Variance (3)

➡ Variance caused by the measurement method

This is systematic error introduced by using a particular method (e.g., self-report bias, item wording, interviewer influence).

If two traits measured using the same method correlate highly just because of the method, that's a problem. *undermines discriminant validity

• Goal: Keep method variance low so that scores reflect the trait itself—not the quirks of the method used.

3. Low Irrelevant Variance (3)

➡ Variance shared with unrelated constructs

This is variance that comes from sources unrelated to the trait, such as random noise or overlap with unrelated constructs.

Example: If a measure of extraversion correlates with a measure of math ability, that's irrelevant variance.

Goal: You want low correlations between measures of unrelated traits → this supports discriminant validity.

minimum requirement for an MTMM (2)

to examine divergent validity, convergent validity and method variance, more than 1 constuct and more than 1 measuremnt method are needed in the validation process →MTMM

MTMM: presents all the interrelations resulting when each of several constructs in measured by each of several methods

Minimum requirement: at least 2 measures representing 2 methods (e.g. self-report, observer ratings, objective tests) for each of 2 constructs

classical MTMM design

A classical MTMM design requires:

At least 2 traits

Each measured by at least 2 different methods

structure of MTMM matrix (6 pts)

1)monomethod blocks

a. reliability diagonal

b. heterotrait-monomethod triangles

2)heteromethod blocks

a.validity digonals

b.heterotrait-hetermethod triangles

1)monomethod blocks (4)

A monomethod block contains the correlations among traits measured using the same method.

Each block shows how trait scores relate when measured with the same method

You create one monomethod block per method.

(Ex. If you measure self-esteen, affect, depression using a self-report questionnaire (method 3), the correlations among those three scores form one monomethod block.) *method 3 x method3

monomethod block contain reliability diagonals (aka monotrait–monomethod correlations) AND heterotrait-monomethod triangles

reliability diagonals (3)

The main diagonal (*uppermost diagonal) of each monomethod block contains reliability estimates.

These are monotrait–monomethod correlations, meaning:

The same trait measured using the same method (e.g., the self-report extraversion scale correlated with itself).

This reflects measures like internal consistency reliability.

*When you correlate the method/scale with itself (or examine consistency among items within that scale), you how well: you measure the same underlying construct and how well they hang together statistically

Heterotrait Monomethod Triangles (4)

*’off diagonal’ element of monothetic blocks

heterotrait–monomethod triangles—diff traits as measured by the same method

Represent discriminant validity of a scale—extent to which two measures of different constructs are unrelated.

heterotrait–monomethod correlations do—they help detect method variance.

If correlations in heterotrait-monomethod triangles are too high, it suggests: The method is inflating similarity between different traits. The correlations may reflect shared method bias, not real relationships between constructs.

(Ex: In a monomethod block, triangles off (*below the reliability diagonal) the main diagonal show correlations between different traits measured using the same method.)

summary of monomethod block (4)

A monomethod block: Contains correlations among traits measured using the same method.

There is one block per method used in the MTMM design.

The main diagonal of each block shows reliability estimates, also called: Monotrait–monomethod correlations (e.g., internal consistency for a scale)

off the main diagonal in each block there heterotrait monomethod triangles, which show correlations between different traits measured using the same method. *discriminant validity

2) Heteromethod Blocks (4)

Called heteromethod because they show correlations between traits measured by different methods.

Each block shows how trait scores relate when measured with different methods

Number of heteromethod blocks = number of methods choose 2 (all pairwise combinations of different methods).

2 pts: validity diagonal (monotrait-hetermethod) + heterotrait-monomethod

Validity diagonals (4)

aka monotrait-hetermethod diagonals—same traits as measured by diff methods *aka convergent validity coefficients

Represent convergent validity—degree to which different methods converge in measuring the same construct.

High correlation indicates strong convergent validity

 correlations should be strong and positive: different methods should still point to the same underlying trait. *so those traits should be correlated with eachother even if measured by diff methods

Each coefficient along this diagonal provides evidence for convergent validity for the corresponding trait.

Example: Correlation of 0.57 between extraversion measured via self-report and an objective test.

Heterotrait–heteromethod triangles (4)

Triangles on each side of the validity diagonal *”off diagonal” elements on heteromethod block

Represent correlations between different traits measured with different methods.

Used to assess discriminant validity:

If traits are theoretically unrelated, correlations should be low or near zero. *heterotrait-heteromethod correlations should be the lowest of all correlations

summary of heteromethod block (4)

A heteromethod block: Contains correlations among traits measured using different method.

There is one block per method used in the MTMM design.

The main diagonal—validity diagonal (aka mono trait-heteromethod correlations), should be strong and positve showing convergent validity 

off the main diagonal in each block there heterotrait heteromethod triangles, which show correlations between different traits measured using diff methods, should be lowest correlations showing discriminant validity  

5 rules of evaluation for MTMM 

1)reliability (monotrait-monomethod diagonals) *reliabilty diagonals

2)method effects check for discriminant validity (heterotrait-monomethod triangles)

3)convergent validity (monotrait-heteromethod diagonals) *validity diagonals

4)discriminant validity (heterotrait-heteromethod triangles)

5)convergent vs discriminant validity for discrimination validity (heterotrait-heteromethod vs monotrait-heteromethod) *discriminant validity should be lower/smaller than convergent validity

RE: monotrait-hetermethod diagonals (3) purpose & key ideas

validity diagonals

Purpose: Shows how well a construct is measured using different methods.

Key Idea: Correlations between different methods measuring the same construct should be high.

These correlations are called convergent validity coefficients.

1)monotrait-hetermethod diagonals (aka validity diagonals) rules (3)

rules (3): Correlations btwn diff methods of same construct should be large and statistically significant → convergent validity

High correlations indicate that different methods are capturing the same underlying construct.

This is the essence of convergent validity: measures of the same construct, even if obtained via different methods, should agree.

Rule for Evaluating MTMM Matrix: For convergent validity, focus on monotrait–heteromethod correlations (validity diagonals) and confirm they are strong and significant.

2)heterotrait monomethod triangles (3)

heterotrait monomethod triangles def: display correlations among different constructs measured using the same method.

purpose: Assess discriminant validity by detecting method bias.

correlations btwn diff constructs using the same method should be lower than convergent valdity coefficients → method bias

*Compare the correlations in the red triangle (heterotrait monomethod triangles) with the validity diagonals

Validity diagonals (monotrait–heteromethod correlations): if high correlations→ convergent validity present (different methods agree on the same trait).

heterotrait monomethod triangle (red triangle correlations): if low correlations → discriminant validity present  (different traits stay distinct).

RE: Heterotrait–Heteromethod Triangles

Def: Show correlations among different constructs measured using different methods.

Purpose: Provide additional evidence for discriminant validity.

3)rules for Heterotrait–Heteromethod Triangles (4)

Key Points:

1)Correlations should be as low as possible if there is no theoretical reason to expect a relationship.

2)These correlations should be lower than: *rule 4

• Convergent validity correlations (validity diagonals).

• heterotrait-monomethod triangle correlations (red triangles).

Reason: Different constructs measured with different methods share neither trait nor method variance, so they are expected to have the lowest correlations in the MTMM matrix.

4)correlations between different constructs measures with diff methods

correlations between different constructs measures with diff methods should be smaller than covergent validity coefficients → discriminant validity (heterotrait-heteromethod & heterotrait-monomethod)

Correlations btwn diff constructs measured by diff methods should be smaller than convergent validity coefficients--same trait measured with diff methods

• And when we see that we have evidence that convergentvalidity is present and basically confirmed both convergent and discriminant validity

5)monotrait-monomethod correlations (3)

reliability diagonal

monotrait-monomethod correlations/reliability diagonal: tell how reliably each constuct (A, B, C) can be measured with each methods

reliability should be expected

adavantages of MTMM Approach (3)

allows examination of convergent and discriminant validity simultaneously

*If construct a a matrix; we an answer about convergent and divergent validity at the same

Instead of doing multiple studies, we can collect that info all at once and answer the questions in one study, mearing diff construct at the same time

stresses the charactersitics that a good test should have

*desirable characteristics of a good test; test scores contaminated as little as possible

reflect natures of construct validation: construct validity is not a single coefficient

*Method highlights that validity is not a yes/no answer, that it's not represented by a single coefficient, and that construct evidence is an accumulated over time (an ongoing process)

disadvantages of MTMM Approach (3)

sometimes not feasible

Not easy to design a study that generates this information

*We need at least 2 methods and to measure at least 2 traits, which is a lot of info to collect in a single study (even for the least complex matrix)

Need to invest time and resources

If we were to generate info by using diff constructs, a lot more time and resources that a single study would require

A matrix would be useful method to examine validity and method variance in a scale in an efficient manner, but not easy to get there

which part of the MTMM measures discriminant validity?

1)heterotrait-monomethod triangles

2)hetertrait heteromethod triangles

which part of the MTMM measures convergent validity?

1)validity diagonals (aka monotrait-heteromethod diagonals)

see lect slides