K205 Module 7: Evaluating Quantitative Research

FHV: Define placebo, double blind and single blind studies and why researchers use them

Placebo:

One group performs the experiment using ‘natural’ variables (no intervention). But the participants don’t know they received the placebo and can’t tell the difference between the real or sugar pill

Double blind:

researcher and participant unaware of some aspect of the study

Single blind:

Participant unaware of some aspect of the study

Define reliability

Consistency of obtaining the same results given we are measuring the same, unchanged object

• An instrument that gives the same measurement each time it measures the same, unchanged object/events

• A test must be reliable for it to be valid (for it to give accurate numbers)

3 types of reliability:

1. Test-retest: IQ test stays relatively the same for the same person

2. Inter-rater: 2 or more people viewing the same event

3. Intra-rater: same person viewing event twice

Define internal and external validity

1. Internal Validity: The extent to which the changes in the DV can be attributed to manipulation of the IV

1. External Validity: The extent to which the researcher can generalize their findings to other people, situations, and times 

   • Population Validity: the extent to which the researcher can generalize their findings to broader populations

   • Ecological Validity: the extent to which the research findings emulate the real world 

Understand the trade-off between internal and external validity

Maximizing internal validity (tightly controlled lab env), to confirm that changes in DV are due only to the IV, means reducing the chance of the study emulating the real world

Identify 8 threats to the internal validity related to the design of the study

1. History: events that occur during the course of the experiment that causes changes in the DV which are not the intervention

   • eg: change in seasons

2. Maturation: processes within the participants that occur as a result of time passing

   • growing older

3. Selection bias: pre-existing differences between the non-equivalent groups which could cause changes in the DV.

   • to maximize internal validity, you need random assignment

4. Testing: Effects of one test on subsequently administrations of the same test

   1. Reactivity: exposure to pre-test changes behavior

   2. Warm-up effect: doing better at the test b/c you’re better at it

   3. Loss of naivety: catching on to what you’re testing/measuring

5. Instrumentation: changes in calibration of the instrumentation leading to unreliable/inaccurate results

   • how reliability is related to internal validity

6. Selection-maturation bias: Some changes within non-equivalent groups occur that effect one group but not the other

7. Expectancy: Researcher anticipating that certain participants will perform better

8. Experimental Mortality: Loss of participants for non-random reasons

   • those who stayed on was because they’re highly motivated so better results are not attributable to the intervention

Identify 3 ways to control threats to internal validity

• Randomization: 

  • Helps control for history

  • Random assignment: each individual in the population has an equal chance of being selected into any one of IV levels

    • Matched pairs: gender, age

    • Counterbalancing: ABC, ACB, BAC

• Placebo, single blind and double-blind studies

  • Helps control for expectancy

  • Placebo: control group that’s not given the actual intervention

  • Single blind: participant unaware to some aspects of the study

  • Double-blind: Participant and researcher are unaware of some aspects of the study

• Standardization of the experiment protocol:

  • helps control for instrumentation

  • Standardize experiments and instruments as much as possible

Identify 4 threats to external validity related to the design of the study

1. Incorrect generalizations

2. Reactive/interactive effects of testing

   • Pretest makes the participant more aware of upcoming treatment 

   • Treatment not as effective without pretest

3. Interaction of selection bias and experimental treatment

   • Group is selected based on some characteristics 

     • the treatment may only work on the groups possessing those characteristics.

4. Multiple treatment interference

   1. when participant receives multiple treatment the effects of previous treatment may influence subsequent ones

Identify 2 ways to control threats to external validity

1. Random selection: selecting participants to represent a broader population

   1. Looking at our inclusion criteria

2. Researchers might restrict claims about groups to which the results cannot be generalized

   1. cannot make broad sweeping claims