Ch 11: Threats to internal validity/null effects

Maturation threat

A change in behavior that emerges more or less sponaneously/naturally over time.

• Preventing these threats: Include a no-treatment comparison group.

History threats

A specific event happens (unrealated to the study) between the pre and post test and affects everyone in the group.

• Comparison group can help control.

Regression threat

a threat to internal validity related to regression to the mean.

Regression to the mean

• An extreme finding is likely to be closer to its own typical/mean level the next time it is measured because the same combination of chance factors that made the finding extreme are not present the second time.

  • If peformance is extreme at pretest, performance is likely to be less extreme at posttest.

  • Prevent with comparison group.

Attrition threat

In pretest/posttest, repeated measures or quasi study; a threat to internal validity that occurs when a systematic participant drops out of study before it ends.

• Problematic when attrition is systematic / not random.

• To prevent: comapre the drop out and remove the participants who drop out from analysis.

Testing threat

in repeated-measures expiriment or quasi-experiment, a kind of order effect in which scores change over time just because participants have taken the test more than once; includes practice effects and boredom effects.

• Can prevent with using post-test only, alternate forms of tests for pre and post test, comparison group.

Instrumentation threat

occurs when a measuring instrument changes over time.

• Solutions: use post-test only design, calibrate forms to be comparable, establish reliability and validity and pre and post test, counterbalance diferent forms across pre and post test.

Selection effect

independent-groups design, the two groups have systematically different kinds of participants in them..

Observer bias

researcher expectations influence interpretation fo results.

Avoiding observer bias

• Double blind design - neither the participants nor the researchers who evaluate them know who is in the treatment group and who is in the comparison group.

• Masked design - participants know which group they are in but observers do not.

Demand characteristics

Participants guess study hypotheses and change their behavior accordingly.

• Double blind design

• Masked design

Placebo effect

people believe they will improve and they do only because of their beliefs.

Double blind plaacebo control study

neither the people treating or the patients know if they are real or placebo.

Null effect

a finding that the IV did not make a difference in the DV.

Reasons for null effects:

• Not enough between-group differences.

• Weak manipulations;

  • Insensitive measures;

• Ceiliing effect

• Floor effect

• Manipulation check

Weak manipulations

not enough of a difference between the two groups.

Insensitive measures

researchers havent used an operalization of the DV with enough sensitivity.

Ceiling effect

all the scores are squezed together at the high end.

Floor effect

all the scores cluster at the low end.

Manipulation check

a seperate DV that experimenters include a study, specifically to make sure the manipulation worked.

Noise

unsystematic variability among members in a group; caused by situation noise, individual differences, or measurement error.

Situation noise

external distractions

Measurement error

• a human or instrument factor that can randomly inflate or delate a person’s true score on the dependent variable.

  • Ex. holding ruler at angle.

Power

an aspect of statistical validity; the likelihood that a study will return an accurate result when the independent variable really has an effect.