Week 12: Advanced Reliability and Validity Concepts

Spearman-Brown Prophecy Formula

Used to predict the reliability of a test after changing its length.

Disattenuation Formula

Corrects for measurement error in correlations.

Ways to Improve a Test

1. Increase relationships between psychological constructs and tests. 2. Remove sources of inconsistency in test administration and interpretation. 3. Increase the number of items on the test.

Key Points of Validity

Validity refers to test interpretations and uses, not the test itself. Must be grounded in relevant evidence and theory. Validity is a continuum, not a binary feature. Often requires reference to an external criterion.

Criterion Validity

Extent to which test scores can predict scores on relevant criterion variables.

Concurrent Validity

Test scores evaluated against a criterion measured at the same time.

Predictive Validity

Test scores evaluated against a criterion measured later.

Practice of Criterion Validity

Typically calculated by correlating test scores with criterion scores.

Content Validity

Extent to which test content adequately covers the full domain of the constructs it assesses.

Face Validity

Extent to which a test seems to non-experts to have content validity; generally not taken seriously by test makers but can be practically important.

Threats to Content Validity

Inclusion of construct-irrelevant content, construct underrepresentation.

Practice of Content Validity

Lawshe's (1975) Content Validity Ratio (CVR) based on expert ratings.

Construct Validity

Extent to which a test measures the constructs it is meant to measure.

Convergent Validity

Test scores should be strongly correlated with tests of related constructs.

Discriminant Validity

Test scores should not be strongly correlated with tests of unrelated constructs.

Construct Validity as 'One Validity to Rule Them All'

Some argue that content and criterion validity should be subsumed within construct validity, as the meaning of a measure must always be pursued.

Response Processes Validity

Do test-takers use intended cognitive processes? (Evidence: direct questioning, eye tracking)

Internal Structure Validity

Does the test have the intended internal structure? (Evidence: factor analysis)

Relations to Other Variables Validity

Does the test relate to other variables as expected (convergent/discriminant validity, criterion variables)?

Consequences of Testing Validity

Does testing bring about beneficial outcomes?

Sensitivity

Test's ability to correctly detect positive cases (True Positives / (True Positives + False Negatives))

Specificity

Test's ability to correctly detect negative cases (True Negatives / (True Negatives + False Positives))

Positive Predictive Power (PPP)

Probability a positive result indicates a positive case (True Positives / (True Positives + False Positives))

Negative Predictive Power (NPP)

Probability a negative result indicates a negative case (True Negatives / (True Negatives + False Negatives))

Prevalence

Probability a random case is criterion positive.

Mnemonics for Validity

Sp-P-In (highly specific test, positive result rules in diagnosis), Sn-N-Out (highly sensitive test, negative result rules out diagnosis)

Caution with PPP/NPP

Values depend on the prevalence in the study sample and may not apply directly to clinical settings with different prevalence.

Nomological Networks

An interlocking system of laws connecting observed variables, theoretical constructs to observables, and theoretical constructs to each other, forming a theory.

Multitrait-Multimethod (MTMM) Matrices

Technique to test nomological networks by obtaining multiple measures of traits and multiple applications of the same methods to different traits.

Trait Variance

Measures correlate if based on same traits.

Method Variance

Measures correlate if based on same methods.

Heterotrait-heteromethod

Different constructs, different methods.

Heterotrait-monomethod

Different constructs, same method.

Monotrait-heteromethod

Same construct, different methods (evidence for convergent validity).

Monotrait-monomethod

Same construct, same method.

Traps in Correlation

Correlations can be deflated by measurement error, restricted range, or time lags.