Psychology and Measurement: Validity, Reliability, and Test Construction

Face validity

When a measure appears to measure what it purports to measure

Test-retest reliability problem

Memory effect

Preliminary item pool size

Twice as many as the number of items you plan to include on your measure

Optimum test retest interval

2-4 weeks

Correlation of .9

A strong positive correlation between factors

Observed score components

True Score + Error

Rating scales

A grouping of words, statements, or symbols on which judgments of the strength of a particular variable are made by the test taker

Incremental validity

The amount of gain in predictive value obtained by using an additional test

Measurement error

Any fluctuation in test scores that results from factors related to the measurement process that are irrelevant to what is being measured

Taylor-Russel Tables

Calculates the validity of the measure based on correlation with successful decision making

Ways to minimize measurement error

Writing items clearly, making instructions easily understood, adhering to proper test administration, providing consistent scoring

r²

The proportion of variance in Y that can be accounted for by knowing X. Conversely, it is the proportion of variance in X that can be accounted for by knowing Y.

Predictive validity

The degree to which scores on a test predict future behavior on a criterion variable.

Orthogonal

Statistically independent.

Factor analysis

Procedure to identify the items that best 'load' on a given factor of an attribute.

Generalizability theory

Attempts to determine the sources of consistency and inconsistency.

Base rate

The extent to which a given attribute exists in the population.

Concurrent validity

The correlation between predictor (test) and criterion scores obtained at approximately the same time.

Construct under representation

When a measure is too narrow (does not fully sample the domains of the construct).

Test-retest interval and correlation

The shorter the interval the higher the correlation.

Standard error of the measure

Used for constructing confidence intervals.

Response set

Patterns of answering items unrelated to the construct (e.g., answering 'c' more often).

Construct validity

The degree to which a test measures a specified construct as determined by the interpretation of the psychological meaning of test scores and the implications of this interpretation.

Cronbach's alpha

Measures the homogeneity of the test.

Construct irrelevant variance

When a measure is too broad it samples domains outside the construct of interest.

Factors causing test score variability

1. The nature of the construct: IQ vs. personality. 2. General ability to comprehend instructions. 3. Stable response sets (e.g., answering 'C' option more frequently). 4. The element of chance of getting a question right. 5. Conditions of testing.

Test-retest paradigm

Uses the same population with the same measure, reducing individual variance.

Best method for assessing breadth of assessment

Factor analysis.

Ceiling effect

When scores tend to cluster at the upper end of a distribution.

Discriminant validity

The degree to which scores on a test do NOT correlate with variables they are NOT supposed to correlate with, given the nature of the construct.

Key to obtaining best preliminary data in pilot study

Similarity with the intended measure in terms of sample, testing conditions, administration, and data analysis.

Relationship between number of items and reliability

As the number of items increases, the value of alpha increases.

Ways of measuring test homogeneity

1. Split half. 2. Odd Even.

Coefficient of determination

.64

Alternate forms reliability assessment

The administration of two alternate forms of the same measurement device and then comparing the scores.

Limitations of alternate forms reliability

Are the two forms of the test actually measuring the same thing? More Expensive. Requires additional work to develop two measurement tools.

Inter rater reliability assessment

Computing the correlation coefficient between the scores of two raters for the set of respondents.

Content validity

A measure containing items that adequately sample the relevant items for a given domain.

Norm referenced cut score

Used to divide test scores into 2 classifications (select/don't select) based on normative data.

Types of criterion validity

Concurrent and predictive.

Most important type of validity

Construct Validity.

Primary factor inflating test-retest correlation

Retest interval.

Reliability

Consistency.

Expectancy Data

Data that analyzes the correlation between the test results and successful decision making.

Formula for predicting reliability after change in test length

Spearman Brown.

Comprehensive definition of validity

An overall judgment of the degree to which both empirical evidence and theoretical considerations support the interpretation of the score and the implications for action that this interpretation entails.

Convergent validity

The degree to which scores on a test correlate with variables they are supposed to correlate with, given the nature of the construct.