Chapter 11: Experiments: Threats to Internal Validity Null effects

Maturation threats to internal validity

• a change in behaviour that emerges more or less spontaneously over time

• people adapt to changed environments; children get better at walking and talking; plants grow taller - not because of any outside intervention, it just happens

• improve on your own

preventing maturation threats

• include a no-treatment comparison group

history threats to internal validity

• a specific event (unrelated to the study) takes place between pre and post test affects everyone in the group

• to be a history threat, the external factor must affect most people in the group in the same direction (systematically,) not just a few people (unsystematically)

preventing history threats

• include a no treatment comparison group

regression threats to internal validity

• regression threat: refers to a statistical concept called regression to the mean

  • when group average is unusually extreme at time 1, the next time that group is measured (time 2), it is likely to be less extreme - closer to its typical or average performance

    • extreme pretest (regardless of IV level)

    • less extreme at posttest

• regression threats occur only when a group is measured twice, and only when the group has an extreme score at pretest

preventing regression threats

• include a comparison group

attrition threats to internal validity

• can happen when a pretest and posttest are administered on sperepate days and some participants are not available for the second day

• problem for internal validity when attrition is systemic; only a certain kind of (extreme cases) particular drops out

  • most shyest, depressed, worst behaviour, etc.

• A) if 2 people (noted by blue dots) drop out of a study, both of whom scored at the high end of the distribution on the pretest, the group mean changes when their scores are omitted, even if all other scores stay the same

• B) if the dropouts scores on the pretest are close the group mean, removing their scores does not change the group mean as much

preventing attrition threats

• how can attribution effects be identified: compare drop outs versus completers on the pretest measures to see whether attrition is selective or random

• how can attrition effects be accounted for: remove participants who drop out from pre and post test analyses, do you see the same pattern of results?

testing threats to internal validity

• people might have become more practiced at taking the test, leading to improved scores, or they may become bored, which could lead to worse scores over time

  • In education, students may score higher on a posttest than a pretest due to practice from the first test, not because of the intervention.

preventing testing threats

• abandon a pretest altogether and use a posttest only design

• use alternate forms of test at pre and posttest

• use comparison group to rule out effect of repeated testing

instrumentation threats to internal validity

• measuring instrument changes over time

  • people who are coding behaviours are the measuring instrument, and over a period of time, they might change their standards for judging behaviour by becoming stricter or more lenient

preventing instrumentation threats

• posttest only design

• calibrate forms to be comparable

• establish reliability and validity at both pre and post test

• counterbalance different forms across the pre and post test

threats to internal validity that can apply to ANY experiment

• observer bias

• demand characteristics’s

• placebo effects

Observer bias

• behavioural dependent variable

• researchers expectations influence their interpretation of the results

  • may expect the low sugar diet to work so he views the boys posttest behaviour more positively

demand characteristics

• problem when participants guess what the study is supposed to be about and change their behaviour in the expected direction

  • patients know they are getting therapy, if they think the researcher expects them to get better, they might change their self reports of symptoms in the expected direction

how to avoid observer bias and demand characteristics

• conduct a double blind study: neither the participants nor the researchers who evaluate them know who is in the treatment group and who is in the comparison group

  • arrange to have 2 cabins of equally lively campers and replace the sugary snacks with good tasting low sugar versions for only one group

  • they wont know which kind of snacks they were eating, and the people observing their behaviour would also be blind to which the boys were in the group

placebo effects

• occurs when people receive treatment and really improve, but only because the recipients believe they are receiving a valid treatment

designing studies to rule out the placebo effect

• double blind placebo control study

• if there is a placebo effect, the pattern of results will show that the no therapy group does not improve as much as the placebo group

null effect

• a finding that an independent variable did not make a difference in the dependent variable; there is no significant covariance between the two

• also called null result

examples of null effects

• Researchers tested if money increases happiness by giving three groups (no money, a little money, a lot of money) different amounts of cash. The next day, all groups reported similar happiness levels, with overlapping 95% confidence intervals.

• An educational psychologist studied whether online reading games improve reading skills in 5-year-olds. Children were randomly assigned to play a reading game for a week or attend regular kindergarten classes. After testing, the reading game group scored slightly higher, but the 95% CI included zero, suggesting no clear difference.

possible explanations for a null effect

• there actually isn’t a relation between the variables “in the real world”

  • perhaps money does not make people happier

• some experimental design feature obscure a true effect

design features that could lead to a null effect

• weak manipulation of the IV

• intensive measures of the DV

• ceiling or floor effects on either the IV or DV

weak manipulation of the IV (null effects)

• wondering why the study show that money had a little effect on people’s mood

  • how much money the researchers gave each group

intensive measures of the DV (null effects)

• have not used an operationalize of the DV with enough sensitivity

  • if medication reduces fever by a tenth of a degree, you would not be able to detect it with a thermometer that was calibrated in one degree increments, not sensitive enough

  • Similarly, if online reading games improve reading scores by about 2 points, you wouldn’t be able to detect the improvement with a simple pass/fail reading test (either passing or failing, nothing in between).

ceiling and floor effects (null effects)

• ceiling effect: all the scores are squeezed together at the high end

• floor effect: all the scores cluster at the low end

how does manipulation checks help detect weak manipulations, ceilings, and floors

• manipulation check: separate dependent variable that experimenters include in a study, specifically to make sure the manipulation worked (IV has intended impact)

  • For example, in the anxiety study, after telling people they were going to receive a 10-volt, 50-volt, or 100-volt shock, the researchers might have asked: How anxious are you right now, on a scale of 1 to 10? If the manipulation check showed that participants in all three groups felt nearly the same level of anxiety

• (A) These results suggest the anxiety manipulation did not work because people at all three levels of the independent variable reported being equally anxious. (B) These results suggest the manipulation did work because the anxiety of people in the three independent variable groups did vary in the expected way. The error bars depict fabricated 95%

what causes large within group variability

• measurement error

• pre existing individual differences

• situation noise

mesurement error

• human or instrument factor that can randomly inflate or deflate a person’s true score on the dependent variable. For example, a person who is 160 centimeters tall might be measured at 160.25 cm because of the angle of vision of the person using the meter stick, or they might be recorded as 159.75 cm because they slouched a bit.

• Measurement error is reduced when the reliability of the measure goes up and when precise measurement systems are used

  • Measurement error is also reduced when a large sample is used

  • Errors more likely to balance/cancel each other ou

pre existing individual differences

• impact response/sensitivity to treatment

  • normal mood for people can vary

situation noise

• Variability in external situation can create within-group variability

  • smell of the nearby coffee shop might make some participants feel cozy, seeing friends at the next table might make some feel extra happy

• Think through possible controls before study