PSY 343 - Psychological Measurement

reliability

the ability of research to produce the same results consistently

measures of reliability

• test-retest; consistency over time

• parallel forms; consistency across forms

• internal consistency; consistency across items

• interrater agreement; consistency across judges

validity

the ability of a test to accurately measure what it intends to measure

measures of validity

scale of measurement

• categorical or nominal (classification of type)

• ordinal (rank order + directionality + no real zero)

• interval (size of difference + directionality)

• ratio (size of difference + directionality + meaningful zero)

variables

characteristics that vary across different cases and define behavior

independent variable

the variable that is being manipulated

dependent variable

the variable that is being measured

moderator

a variable that changes the strength and/or direction of the relationship between the IV and DV

mediator

a variable that explains how or why the IV impacts DV

construct

abstract mental frameworks that we can use to describe psychological phenomenon

operational definitions

a precise, measurable description or process of how a theoretical construct or variable will be observed, quantified, or manipulated in a specific study

dimensionality

a part of construct; level of specificity or granularity

likert scale

a survey scale used to measure people’s opinions, attitudes, or level of agreement

open-ended questions

require detailed and explanatory answers

closed-ended questions

can be answered with static choices or simple yes/no

item best practices

use clear language, avoid loaded questions, avoid double-negatives, intentional about reverse-scored items, consider order

response issues

introduce error → reduce reliability and/or validity + inaccurate inferences

deliberate distortions

intentional attempts to manipulate responses or data

demand characteristics

subtle cues or clues in an experimental setting that let participants figure out the research hypothesis or guess what the researcher expects them to do

triangulation

using third-party sources to validate information

unintentional distortions

inadvertant attempts that lead to inaccurate responses or data

acquiescence and counter-acquiescence

agreeing or disagreeing with statements regardless of what the statement is asking

retesting effects

score improvements that happen when people take the same or a similar cognitive test more than once, on average

standardization

making sure each test is administered and scored consistently

face validity

subjective judgment of whether a test or research tool appears to measure what it claims to measure

dimensionality

refers to whether a psychological construct measures one variable (unidimensional) or multiple factors (multidimensionality)

norm-referenced approach

comparing an individual raw score against other individual scores (ex. percentiles)

criterion-referenced approach

comparing an individual raw score against a pre-determined standard

scale

set of questions used to measure a psychological trait or characteristic

evaluating scales

construct, dimensions, population, validity, resources, and reliability

jingle-jangle fallacy

different things have the same name, so we think they are the same, or the same thing has different names, so we think they are different

modality

the number of distinct peaks (modes) in a probability distribution or dataset

positive skew

(right-skewed): tail extends to the right; mean > median

negative skew

(left-skewed): tail extends to the left; mean < median

kurtosis

amount of outliers

measures of central tendency

mean, median, mode (describes a typical score in the distribution)

measures of variability

range, variance, standard deviation (dissimilarity in distribution)

standard deviation

data values compared to the average

standard error

measures how much the sample statistic is expected to vary from the true population

correlation coefficient

pearson’s r (sign; direction + number; strength)

standard error of measurement (SEM)

estimates the spread of random measurement errors in a test score

cohen’s d

measure of effect size that tells you how big the difference is between the means (averages) of two groups

spearman-brown formula

used to estimate the reliability of a test when its length is increased or decreased by adding or removing similar test items