Lecture 5: How do we evaluate the validity of a claim?

Construct validity

how well the researchers measured their variable of interest

• Do different levels of the variable accurately correspond to true differences in the construct

• Is the manipulation of the independent variable appropriate (causal claim)

• How well is the outcome variable measured (causal claim)

Ask:

• How did the researchers define their construct of interest?

• How did they measure the variable? Was the measurement appropriate given the definition?

External validity

= how well would the claim generalise beyond the current sample (ex. Other people/contexts/times/place) (causal claim?)

• Is the sample representative of the larger population?

• How was this sample selected?

Statistical validity

= how accurate is the frequency claim?

• What is the margin of error?

• How strong is the association? Is it statistically significant? (association claim) (causal claim)

• Does the design of study minimise probability of errors in decision making? (causal claim)

Internal validity 

given the design of the study, can you rule out all other possible explanations?

• Is there another possible explanation for the results we are seeing?

• Given the design of the study can you rule out all other explanations

3 criteria for establishing causation

1. Covariance

• The study shows that as A changes, B changes (ex. High levels of A cause high levels of B)

2. Temporal precedence

• A must come before B. the study's method ensures that A comes before B

3. Internal validity 

• There is no other plausible alternative explanations for the change in B; A is the only thing that change

*The only type of research design that can support a causal claim is an experiment. you need an experiment to claim that A causes B

The big 4 validities

Construct validity: How well the variables in a study are measured or manipulated

• the extent to which the operational variables in a study are a good approximation of the conceptual variables

External Validity: The extent to which the results of a study generalize to some larger population (ex. whether the results of this sample of teens represents all teens), as well to other times or situations

Statistical validity: How well the numbers support the claim, how strong the effect is and the precision of the estimate

Internal validity: in a relationship between one variable (A) and another (B) the extent to which one variable (A) rather than some other (C), is responsible for changes in (B)

Jack Dagger exercise: does this study support a causal claim?

Music and risk taking

• People placed in the happy music room or the sad music room

• The people who listened to the happy music rather than the sad music were more likely to accept standing in for jack daggers knife throwing

Q: Can this study support the causal claim that happy music causes people to accept 

more risk?

A: this study can not support the causal claim because it lacked the internal validity