reliability

Reliability

-test is repeatable and will give the same results

-refers to the consistency and dependability of the data collection process

-within itself, overtime, over circumstances

Internal Consistency/Reliability (Within Itself)

-how consistently each item behaves relative to the overall tests

-how well a test measures what we want to measure

-harmony among different items within a measurement tool aiming to assess the same concept

-if I get a high score on a whole test, do I also get a high score on each item

-Cronbach alpha, split-half, item total correlations

Test-Retest (Overtime)

-the degree to which a test will produce the same result over repeated administration

-consistency of test response overtime: assuming there is no intervention/change to the thing being measured between tests

-shorter time period likely to have higher consistency

-the degree to which a test will produce the same results over repeated administration

-I did well last time, will I do well the next time?

-rank order correlations, intra-class correlation

Inter-Rater Reliability (Over Circumstances/Over Raters)

-the degree to which multiple raters will give the same results for a given target

-extend to which different raters produce consistent results when rating the same thing

-do mothers and fathers give their child the same score?

-Cohen kappa, intra-class correlation

Cronbach’s Alpha

-internal consistency/reliability

-how closely related a set of items are as a group

-value 0.7 is considered acceptable, with larger values indicating greater reliability

Split-Half

-internal consistency/reliability

-splits a test in 2 halves and measures the correlation between them

-higher correlation indicates greater internal consistency

-1 is considered perfect correlation

Item Total Correlations (Item Discrimination)

-internal consistency/reliability

-differentiates between a question score and the overall test scores

-0-0,19: indicate that the question is not discriminating well

-0.2-0.39: indicate good discrimination

-0.4+ indicates very good discrimination

Factors Impacting Internal Consistency

-sensitive to sample size: the bigger the sample, the more consistent it appears

-sensitive to number of items: more items the more reliable

-sensitive to difficulty: adding lots of “easy” or “difficult'“ times will decrease the reliability, but adding lots of “average” items will increase it

-gives the reliability of a test, not the reliability of the scores

Rank Order Correlations

-test-retest reliability

-do respondents stay in the same order overtime?

-less than 0.2: negligible

-0.2-0.29: weak

-0.3-0.39: moderate

-0.4-0.69: strong

-more than 0.70: very strong

Intra-Class Correlation (ICC)

-test-retest reliability

-estimates the proportion of variability in scores as a function time

-less than 0.5: poor

-0.5-0.75: moderate

-0.75-0.90: good

-more than 0.90: excellent

Factors Impacting Test-Retest Reliability

-time interval: shorter intervals tend to nil higher reliability but may introduce memory bias, while longer intervals risk chances in the measured construct

-test content: tests with ambiguous or poorly defined items are less likely to yield consist results

-participant factors: variations in participants conditions between administration

-environmental factors: difference in testing environments

Cohen’s Kappa

-inter-rater reliability

-measures how 2 raters agree with each other beyond what is expected by chance

-used to assess the reliability of comparative agreement between raters

Intra-Class Correlation (ICC)

-inter-rater reliability

-estimates the proportion of variability in test scores that are not a function of rater (1- rater effect)

Factors Impacting Inter-Rater Reliability

-subjectivity: raters may interpret criteria differently, especially for qualitative assessments

-chance agreement: methods like percent agreement may overestimate reliability by ignoring chance agreement

-rater bias: personal biases or fatigue can influence ratings, reducing reliability

-sample dependence: reliability estimates can vary based on the sample or context, limiting generalizability

Latent Variables

-provide detail on the mature of the relationship between items and constructs

-allow for scoring to be adjusted to increase the reliability of a measure

-allow for estimation go score reliability rather than scale reliability

-exploratory factor analysis, confirmatory factor analysis, item response theory

Exploratory Factor Analysis (EFA)

-data driven approach where number and structure of latent variables is unspecified

-often mistaken for principal components analysis

Confirmation Factor Analysis (CFA)

-theory driven approach

-number of latent variables and structure are specified by the researcher

Item Response Theory (IRT)

-theory driven approach similar to CFA (differences are largely technical)

-Rasch Model is commonly used

structural assumptions are not always met