psychometrics

reliability

the consistency of the scores that you generate from a test

parallel/alternative forms reliability

uses two equivalent versions of a test to check for consistency

test-retest reliability

a measure of how consistently a test produces similar results when the same individuals take it again over time

inter-rater reliability

the degree of agreement between two or more independent observers who are rating, coding or assessing the same phenomenon

cronbach’s alpha

measures the internal consistency of a test or scale, indicating how closely related a set of items are as a group.

validity

does the test measure what it is supposed to measure?

content validity

the extent to which an assessment measures all aspects of the specific subject or construct it is designed to measure

construct/factorial validity

the degree to which a test accurately measures the theoretical concept it is intended to assess

response process validity

the evidence that the thinking and actions a person uses to arrive at an answer are aligned with the intended construct the assessment aims to measure

concurrent validity

the extent to which a new test’s results agree with an existing, established gold standard test that measures the same thing at the same time 

predictive validity

the extent to which a test accurately forecasts a future outcome (a type of criterion-related validity)

convergent validity

shows how a measures correlation with other measures of the same or similar construct

divergent validity

a type of construct validity that shows a measure is not strongly correlated with measures of unrelated constructs

guidlines for alpha

beginning stages research =.70

typical research =.80

important decisions =.90

inter-correlation

excluding the total column, you want it to be closer to .15-.20

where to find corrected item-total correlation

r.drop column

how to decide which items to remove?

remove items that are higher than the original alpha value. you can find this info in the raw_alpha or std_alpha column depending on whether you are analysing raw or standardised data

classical test theory

a body of theory underpinning multiple aspects of psychometric testing and development. X = T + E

X

observed score = actual scores we obtain from tests

T

true score = hypothetical score devoid of measurement error, therefore we want these to be as close to the observed score as possible

E

error score = necessary to work out T, error scores should have a mean of zero and be uncorrelated with true scores. we can estimate error by evaluating the reliability of the scores produced by a test

limitations of cronbach’s alpha

• assumes unidimensionality

• lower bound estimate

• can be easily inflated

• sample specific