Tests and Measures Exam 1

Psychological test

standardized, objective measure of a sample of behavior

Standardized

fixed instructions and procedures for administration and scoring — everyone takes the test the same way, which is what makes scores comparable across people. Scores from standardized tests can be rendered useless if those procedures aren't followed

Tests of maximal performance

require test takers to perform a particular, well-defined task (think ability and achievement tests)

Behavior observation tests

involve observing people’s behavior and how people typically respond in a particular context

Self-report tests

require test takers to report or describe their feelings, beliefs, opinions, or mental states

Standardized sample

people who are tested to obtain data to establish a frame of reference for interpreting individual test scores

Norms for standardized and unstandardized tests

indicate the average performance of a group and the distribution of scores above and below this average

Nonstandardized tests

do not have standardization samples and are more common, than standardized tests, are usually constructed by a teacher or trainer in a less formal manner for a single administration 

Objective tests

tests where test takers choose a response or provide a response and there predetermined correct answers, requiring little subjective judgment of the person scoring the test

Projective tests

those on which test takers view and are asked to respond to unstructured or ambiguous stimuli such as images or incomplete sentences

Achievement tests

measure a person’s previous learning in a specific academic area

Aptitude tests

assess a test taker’s potential for learning or ability to perform in a new job or situation

Intelligence tests

like aptitude tests and assess a test taker’s ability to cope with the environment, but at a broader level; measure behavior on a set of tasks

Ability

broader, sometimes ambiguous term that can refer to current capability

Interest inventories

assess a person’s interests in educational programs for job settings and provide information for making career decisions

Personality tests

measure human character or disposition

Assessment

more involved than a single test; integrates multiple sources if information and data (tests, direct observations, interviews, client history)

Test

just one tool

Psychological assessment

uses multiple methods, such as personal history interviews, behavior observations, and psychological tests, for gathering information about an individual

Measurement

the process of assessing the size, the amount, or the degree of an attribute using specific rules for transforming the attribute into numbers

Measurement instrument

a tool or technique for assessing the size, amount, or degree of an attribute

Surveys

like psychological tests (and psychological assessments), they are used to collect important information from individuals; they focus on group outcomes; results are often reported at the question level by providing the percentage of respondents who selected each answer alternative

User qualification: Level A

little to no specialized training required (e.g., achievement tests)

User qualification: Level B

requires some technical knowledge of test construction and use, typically a master's degree plus relevant coursework

User qualification: Level C

requires advanced training in test theory and relevant clinical experience (e.g., projective personality tests, individually administered intelligence tests)

Hypothetical construct

inferred from observable behavior rather than observed outright (ex: intelligence, self-esteem, anxiety)

Norms for tests

descriptive statistics drown from a reference sample, used to compare an individual’s score against that comparison group

Norm-referenced tests

compare an individual's performance to the performance of a group who previously took the test. Best for comparing someone's relative standing.

Criterion-referenced tests

compare performance to a fixed standard or cut score, independent of how anyone else performed. Best for determining mastery (e.g., 100% of multiplication items correct)

Nominal

categories, labels, identification (ex: jersey numbers, gender, dSM-5 diagnostic codes)

Ordinal

nominal and in rank order, but with unequal intervals (ex: military rank, race finishing order)

Interval

ordinal but with equal intervals; NO true zero (ex: IQ scores, fahrenheit, celcius)

Ratio

interval but WITH true zero (an absence of what’s being measured) (ex: height, speed, reaction time)

Ordinal 2

ordinal data with averaging (ex: likert scale, strongly agree-strongly disagree)

What data supports calculating a mean and other parametric statistics?

interval and ratio data

What data is limited to nonparametric approaches?

nominal and ordinal data

Central tendency

mean, median, mode (roughly the same value in a normal distribution)

Mean

the arithmetic average; requires at least interval data; the most powerful measure but sensitive to skew (rogue scores pull it off center)

Median

the middle score when data is ordered; unaffected by extreme scores; useful when data is skewed.

Mode

the most frequent score; a data set can be unimodal or multimodal

Standard deviation

describes how spread out scores are around the mean; foundation for interpreting standard scores and calculating the standard error of measurement

Small standard deviation

scores cluster tightly around the mean

Large standard deviation

scores are more spread out

z-score

mean: 0

SD: 1

T-score

mean: 50

SD: 10

Deviation IQ

mean: 100

SD: 15

Stanine

mean: 5

SD: 2

Normal distribution

symmetrical; the same shape mirrored on both sides of the peak, with mean, median, and mode all landing at the center

x-axis represents:

scores

y-axis represents:

frequency

Correlation coefficients

symbol: r

range from −1.0 to +1.0

positive r means both variables move in the same direction

negative r means they move in opposite directions.

Strength of r

0 to ±.3 = weak/little relationship

±.3 to ±.7 = moderate

±.7 to ±1.0 = strong

Coefficient of determination

symbol: r²

the proportion of variance in one variable explained by the other

multiply r by itself.

Restricted range

when a sample only includes a narrow slice of the full range of scores (e.g., only people who were hired, or only people who studied a lot), the resulting correlation will look weaker than the true relationship, even though nothing about the underlying relationship changed

weakens validity coefficients the same way it weakens any correlation

Classical test theory (CTT) equation

observed score = true score + error

(true score is hypothetical)

Random (unsystematic) error

unpredictable fluctuation (ex: illness, distraction, motivation, luck)

assumed to be normally distributed and to cancel out over repeated measurement

Systematic error

a consistent, repeatable flaw in the test itself (ex: a scale that always reads three pounds heavy)

test publishers try to eliminate this before publication

Rxx value: 1.0

perfect reliability

theoretical only, never happens in practice

Rxx value: 0.90 and above

very good

seen in well-developed tests like IQ tests

Rxx value: 0.80

good

generally the acceptable threshold

Rxx value: 0.70

marginal

acceptable depending on how complex the construct is

Rxx value: below 0.70

cause for concern

Rxx value: 0.0

no reliability

all error, no true score

Test-retest

method of estimating reliability

same test, same people, two occasions, correlate the scores (watch for practice/carryover effects → counterbalancing can help offset this)

Alternate forms

method of estimating reliability

two different but parallel versions of a test, correlated against each other (watch for order effects)

Internal consistency

method of estimating reliability

how well items on a test correlate with each other

includes: split-half reliability & coefficient alpha/cronbach’s alpha

best suited for unidimensional (homogeneous tests)

Interrater reliability

agreement between two or more independent scorers

essential wherever subjective judgment is involved (open-ended responses, behavioral observation)

Standard error of measurement (SEM)

estimates how much an individual's observed score might differ from their true score — it applies to a single person's score, not to a group

Confidence intervals

± 1 SEM ≈ 68% confidence

± 2 SEM ≈ 95% confidence

± 3 SEM ≈ 99.5% confidence.

Expectancy tables

a practical tool for interpreting the relationship between test scores and a criterion

test score ranges across the top, criterion outcomes down the side, with each cell showing how many people in that score range achieved that outcome

A test can be reliable…

and still be invalid

Unreliable tests…

can never be valid

Evidence based on test content

does the test adequately sample the domain it claims to measure?

includes test blueprints/specifications and expert judges

Evidence based on response processes

do test takers respond to items the way we'd expect, given the construct being measured?

Evidence based on internal structure

do the relationships among items, components, or scales match the construct's theorized structure? Assessed through factor analysis, evidence of homogeneity, and differential item functioning (DIF)

Evidence based on relations to other variables (criterion-related validity)

concurrent validity (test and criterion measured at the same time) and predictive validity (test used to estimate a future criterion)

evaluated with a validity coefficient

Evidence based on consequences of testing

the intended and unintended consequences of how a test is actually used once it's in the world

Validity coefficients

 tend to run smaller than reliability coefficients and are harder to interpret

a good coefficient is often more modest than a good reliability coefficient

Linear regression

uses one test score (x) to predict a criterion score (y)

Multiple regression

uses more than one predictor at once (X₁, X₂…) to predict a single criterion

Concrete attributes

directly observable

Construct explication

identifying the behaviors related to a construct, the other constructs that may be related to it, and whether those related constructs' behaviors also connect back to the original construct

Nomological network

mapping a construct's relationship to as many other constructs and behaviors as possible — what should correlate strongly, weakly, or negatively with it

Convergent validity

a measure correlates highly with other measures of the same construct

Discriminant validity

a measure does not correlate with measures of unrelated constructs

Multitrait-multimethod (MTMM) design

a single study design that captures reliability, convergent validity, and discriminant validity evidence at once, using a correlation matrix across multiple traits and multiple measurement methods

Factor analysis

a statistical procedure that explains why items or tests correlate with each other

Exploratory factor analysis

identifies underlying factors with no prior hypothesis

Confirmatory factor analysis

tests whether a pre-specified theoretical structure actually fits the data