Psychological Assessment, Reliability, and Validity

Flashcards covering psychological assessment methods, reviews, reliability theory, and various forms of validity evidence according to lecture notes.

Psychological assessment

A systematic process that uses a combination of techniques and methods to evaluate various psychological and behaviour characteristics of an individual or group of individuals.

Narrative review

A review where an expert subjectively decides which studies should be included and how they should be weighted, focusing on qualitative descriptions of prior results.

Systematic review

A review that attempts to collate all empirical evidence fitting pre-specified eligibility criteria using transparent, predefined search strategies.

Meta Analysis

The statistical analysis of a large collection of results from individual studies for the purpose of integrating the findings into a summary estimate of the true effect.

Risk ratio (Relative risk)

The probability of an outcome in the exposed group divided by the risk of the outcome in the unexposed group; a risk ratio $< 1$ means lower risk in the exposed group.

The file drawer problem

A criticism of meta-analysis suggesting that the omission of unpublished studies with null results may invalidate the findings.

Reliability

The property of consistency in measurement; a test is reliable if it yields stable and consistent results.

Classical test theory formula

$x_i = \tau + \epsilon_i$, where $x_i$ is the observed score, $\tau$ is the true score, and $\epsilon_i$ is the error component.

Test-retest reliability

A method of estimating reliability by obtaining and correlating scores from an original test and a retest occasion.

Carryover effects

Problems in reliability testing where participants remember previous answers or change behavior due to having taken the test before.

Alternate-forms reliability

A reliability estimate obtained by correlating scores from two different but parallel forms of the same test.

Split-half reliability

An estimate of reliability obtained by splitting a test into two parallel subtests and calculating the correlation between them.

Cronbach’s $\alpha$

A reliability measure thought of as averaging across all possible split-half estimates after adjusting them to reflect the full length of the test.

Standard error of measurement ($SE_m$)

A value that quantifies the typical size of measurement error in test-score units, defined as $SE_m = \sigma_x\sqrt{1 - r_{xx}}$.

Diattenuation formula

A formula used to estimate the correlation between constructs if they were measured without error, correcting for the fact that measurement error makes observed correlations smaller.

Nunnally & Bernstein benchmark

The standard benchmark for reliability, defined as $r_{xx}' = 0.9$.

Validity

The degree to which evidence and theory support the interpretations of test scores for proposed uses of tests.

Criterion validity

The extent to which a test can predict scores on relevant criterion variables, calculated via correlation between test scores and criterion scores.

Concurrent validity

A type of criterion validity where test scores are evaluated against a criterion measured at the same time as the test.

Predictive validity

A type of criterion validity where test scores are evaluated against a criterion measured at a later date.

Content validity

The extent to which a test's content adequately covers the full domain of the construct it is assessing.

Face validity

The extent to which a test seems to non-experts (such as test takers) to have content validity.

Lawshe (1975)’s Content Validity Ratio (CVR)

$CVR = \frac{n_e - (\frac{N}{2})}{(\frac{N}{2})}$, where $n_e$ is the number of experts responding 'essential' and $N$ is the total number of experts.

Construct validity

The extent to which a test measures the specific construct it is intended to measure, involving convergent and discriminant validity.

Convergent validity

Evidence showing that test scores are strongly correlated with tests of related constructs.

Discriminant validity

Evidence showing that test scores are not strongly correlated with tests of unrelated constructs.

Sensitivity

A test's ability to correctly detect positive cases, calculated as $\frac{true\,positives}{true\,positives + false\,negatives}$.

Specificity

A test's ability to correctly detect negative cases, calculated as $\frac{true\,negatives}{true\,negatives + false\,positives}$.

Positive predictive power (PPP)

The probability that a positive test result indicates a true positive case, calculated as $\frac{true\,positives}{true\,positives + false\,positives}$.

Prevalence

The probability that a random case is criterion positive, calculated as the sum of true positives and false negatives divided by the total number of cases.

Nomological networks

An interlocking system of laws that relate observed variables to each other, to theoretical constructs, and theoretical constructs to each other (Cronbach & Meehl, 1995).

Multitrait-multimethod matrices

A method proposed by Campbell & Fiske (1959) to evaluate convergent and discriminant validity by assessing trait variance and method variance.

Trait variance

Score variance attributable to the underlying trait being measured.

Method variance

Score variance attributable to the specific measurement method used.

The 'hard test' of construct validity

A success pattern where monotrait-heteromethod correlations are greater than heterotrait-monomethod correlations, suggesting trait variance outweighs method variance.