Measurement, Reliability and Validity

Reliability

Does your measurement produce stable and consistent measurements?

Validity

Are you measuring what you say you are measuring?

Reliability- Stability

• Stable measure will produce identical measurement results whenever it is applied to an identical amount of the theoretical concept (ex- thermometer and ice water)

• Need some mathematical way of expressing the amount of stability

• Correlation coefficient: r

  • Ranges from 1.0 (perfect reliability) to 0.0 (no consistent pattern of relationship)

Ways to asses stability

• Test-retest reliability

  • Measure same concept with same unit of analysis on more than one occasion

• Split-Half

  • Split the patricipant’s responses in half (ideally randomly) and compare

• Multiple-sample

  • Take two random samples from the population and compare (only works for random samples)

Reliability- Consistency

• Intercoder reliability

  • Consistency with operational definitions

  • 2 or more coders look at same units and asses how much they agree with each other

• Internal consistency

  • To what degree do your items measure the same thing

  • Cronbach’s alpha

Face Validity

Is the measure reasonably representative of the universe of possible items that could represent the concept of being measured?

Criterion Validity

Use an established measure as the “criterion” against which you are measuring your new measure (ex- ACT/SAT are supposed to predict how well students will perform in college)

Discriminant Validity

The degree to which the new measure does not convey strongly with the results given by measures of different concepts (ex- athletic ability v. coaching skill)

Predictive Validity

The degree to which the new measure predicts future scored on a measure which it should predict (ex- if we used the 5 love languages, we would want this measure to predict positive relational outcomes)