Lecture 6.2: Convergent and Discriminant Validity

Nomological Network

Interconnections between a construct and related constructs

Convergent and Discriminant Validity

Reflects the degree to which test scores have the “correct” patterns of associations with other variables

Focused Associations

Specific correlations between test scores and criteria. Also known as predictive validity.

Validity Generalisation

Process of evaluating test validity across diverse studies

Validity Coefficient confidence

If validity coefficients are sufficiently large, we have more confidence in using the test for its intended purpose.

Questions Validity Generalisation studies addresses

1. Estimate the average level of predictive validity across studies

2. Estimate the degree of variability in validity coefficients

3. Identity sources of systematic variability in validity coefficients.

Multitrait-Multimethod Matrices

Method to distinguish trait and method variances in correlations. Researchers must administer at least 3 different methods to validate the interpretations of test scores. We hope to see larger correlations between same traits in comparison to scores based on the same methods.

Evaluating MTMM Results

No clear guidelines to evaluate differences in mean correlations.

• Monotrait-heteromethod correlations > Heterotrait-monomethod correlations

Rarity of MTMM studies

• Lack of guidelines

• Labour intensive to conduct

• Large percentage of shared variance between measures is due to shared method variance.

Average Level of Predictive Validity

Estimation of the mean validity level across studies

Variability of Validity Coefficients

Degree of differences in validity coefficients across studies

Trait Variance

Variance in scores attributed to the trait being measured

Method Variance

Variance in scores due to the method of measurement

Monotrait-heteromethod correlations

Correlations within the same trait measured by different methods

Heterotrait-monomethod correlations

Correlations between different traits measured by the same method

Quantifying construct validity

1. Procedure predicting correlations between measure of interest and their selected criteria.

2. Correlations between measure of interest and these selected criteria are estimated.

3. Finally, correlation between predicted and estimated correlations are estimated.

Limitations with QCV

No guidelines on how large the correlation must be to indicate anything

Factors affecting validity coefficient

Magnitude would be affected by number of factors, some more statistical in nature and others more measurement related.

Measurement error

Validity coefficients will be biased downwardly to the extent that the measures are associated with scores with imperfect reliability.

Range restriction

The amount of variability in one or both distributions of scores can affect the correlation between the two sets of scores.

Realistic cases for range restriction

1. The measure(s) are not sensitive enough to distinguish people

2. Sample is more homogeneous than the population of interest.

Convenience samples in restricted range

Convenience samples are often used in practice and they tend to be more homogeneous than the population. Tends to yield smaller SD’s in the data, which reduces the magnitude of the observed correlation, in comparison to the correlation in the population of interest.

Effect sizes in psychology

Many “true” effect sizes between dimensions are likely larger than the observed effects reported in published papers. Observed effect sizes are smaller due to restricted/imperfect measures and samples. Difference is probably substantial.