Test Validity – Content and Construct Validity

Reliability

The consistency of test scores; it indicates that scores can be expected to be stable over time

Validity

a test measures what it was designed to measure, evaluating its accuracy

Content Validity

test adequately represents the content or behavior domain it aims to measure.

How is content validity established

during test development through expert review

Face Validity

how valid a test appears to be to test-takers based on its content

not a formal type of validity, it can affect examinees' willingness to try their best

Face validity

Construct Validity

a test accurately measures a hypothetical construct

Construct Validity procedures

through procedures such as convergent and divergent validity.

Convergent Validity

The degree to which scores on a test correlate highly with scores on other tests that measure the same constructs

Divergent Validity (Discriminant Validity) correlation

The degree to which scores on a test do not correlate with scores on tests of unrelated constructs.

Multitrait-Multimethod Matrix

A table of correlation coefficients that helps assess a test’s reliability, convergent validity, and divergent validity.

Monotrait-Monomethod Coefficient

The reliability coefficient of the test being validated when using the same trait and method (e.g., coefficient alpha).

Monotrait-Heteromethod Coefficient

The correlation coefficient between two tests measuring the same trait but using different methods, showing evidence of convergent validity.

Heterotrait-Monomethod Coefficient

The correlation coefficient between a test measuring one trait and another test measuring a different trait using the same method.

what does a Small Coefficient indicate when using the Heterotrait-Monomethod

indicates divergent validity

Heterotrait-Heteromethod Coefficient

The correlation coefficient between tests measuring different traits using different methods.

Heterotrait-Heteromethod Small Coefficient

supports divergent validity

Factor Analysis

used to assess a test’s convergent and divergent validity by analyzing the relationships between a set of variables

Rotated Factor Matrix

The result of factor analysis, where factor loadings are adjusted to make the data easier to interpret.

What does a Rotated Factor Matrix indicate with it’s findings?

correlation between each test and the identified factors

Factor Loadings

The correlation coefficients in the rotated factor matrix, showing the relationship between each test and each identified factor.

Communality

The amount of variability in a test that is explained by the identified factors in a factor analysis.

Orthogonal Factors

Factors that are uncorrelated

What does Orthogonal Factors variability indicate?

their variability can be independently measured in factor analysis

Oblique Factors

Factors that are correlated with each other in factor analysis

Oblique Factors interpretations

their interpretation is more complex.

Factor Analysis: Step 1

Administer the test being validated alongside tests of the same or related traits and unrelated traits

Factor Analysis: Step 2

Correlate all pairs of scores and create a correlation matrix.Factor Analysis

Factor Analysis: Step 3

Use the correlation matrix to derive the initial factor matrix.

Factor Analysis: Step 4

Rotate the initial factor matrix and interpret the resulting data to name the factors.

Factor analysis supports convergent validity

the test correlates strongly with related traits

Factor analysis supports divergent validity

does not correlate with unrelated traits

Reliability is necessary for validity

A test can be consistent (reliable) but still fail to measure the right thing (not valid).

What type of validity comes first during test development?

Content validity

convergent and divergent validity provide evidence that…

the test is accurately measuring the intended construct

What type of validity does monotrait-heteromethod assess?

convergent validity

What type of validity does heterotrait-monomethod assess?

divergent validity

What does the weakest correlation support in heterotrait-heteromethod?

divergent validity

What does factor analysis identify?

factors (underlying traits) that explain variability in test scores

What are correlations between tests and factors?

Factor loadings

What are factor loadings used for?

show a test strongly relates to a specific factor

what does factor analysis provide evidence for?

construct validity

What does strong convergent validity mean?

a test loads highly onto it’s intended factors

What does strong divergent validity mean?

a test does not load highly onto unrelated factors