Research Validity

External Validity - Setting

• Concerns the representativeness of the research setting to other settings, especially the real world.
• Psychologists aim to understand thinking and behavior in everyday situations.
• Controlled experiments often require labs, which differ significantly from the real world.
• Research labs are designed to isolate variables and exclude real-world influences.
• Harlow's monkey experiments exemplify lab research with real-world inferences.
• Eyewitness Memory:
  • Eyewitness testimony is crucial but often flawed.
  • Faulty memories can lead to wrongful convictions.
  • Research aims to assess trustworthiness and improve accuracy.
  • It is difficult to study immediately after crimes due to unpredictability and varying circumstances, as well as gaining the approval to study these cases.
• Lab simulations:
  • Participants view staged crime videos.
  • Researchers control variables like viewing conditions and timing.
  • Allows for consistent presentation and large sample sizes, studying hundreds of participants.
• Ecological Validity:
  • Questions whether simulated crime scenarios replicate real-world experiences.
  • Real-world crime witnessing involves different cognitive and emotional states.
• Skepticism about Research Settings:
  • Some prefer personal experience due to perceived artificiality of research settings.
  • Counterpoint: Many research questions require controlled environments.
  • Labs enable necessary experimental controls.

Statistical Validity

• Addresses whether data supports the claims made.
• A claim is statistically valid if numbers back it up.
• Data should support a claim.
• Many claims lack supporting data or are contradicted by available data.
• Example: Diet Sodas
  • Claim: Diet sodas help with weight loss.
  • This claim is statistically invalid.
  • Marketing implies weight loss benefits despite evidence to the contrary.
  • Coca-Cola's Website:
    • Initially addressed concerns about weight gain, suggesting no direct link.
    • Later shifted focus to addressing obesity in general.
  • Weight Gain Study:
    • Measured belly fat gain across three groups: non-diet soda drinkers, occasional diet soda drinkers, and daily diet soda drinkers.
    • Results:
      • Non-diet soda drinkers: $0.8$ inches
      • Occasional diet soda drinkers: $1.8$ inches
      • Daily diet soda drinkers: $3.2$ inches
    • Positive correlation between diet soda consumption and weight gain shown in the study.

Statistical Significance

• If data supports a claim, the next question is whether the support is strong enough to be statistically significant.
• Statistical significance helps determine the reliability of a claim.
• Pre-Exam Meditation Program (example):
  • Goal: To determine what helps most with exam scores during the fifteen minutes before an exam starts.
  • Groups: Meditation, jumping rope, and studying.
  • The question is whether meditating helps exam scores, and of course, compared to what?
  • How much better would the meditation group have to score than the other groups, before you believe that meditation is actually effective?
• How much better does the meditation group have to do than the other two groups before you would believe that meditation really works as a pre-exam method to increase performance?
• The question becomes: what constitutes a meaningful difference to suggest effectiveness?
  • For example, improvements of $1\%$, $2\%$, and $4\%$, would you find these convincing?
  • If the meditation group averages $79\%$, compared to the control groups equalling $75\%$, would you consider this enough of a win for meditation?
• Need to determine the threshold for believing meditation is effective.

P-Value

• $p$-value is the usual standard for knowing if a result is statistically significant.
• Critical value: $0.05$
• If p < 0.05, the result is statistically significant, indicating a real effect.
• Goal: To understand the meaning and implications of the $p$-value.
  • Important to know what $p$-value really, really, means.
  • We want the $p$-value to hold more value than simply the number that SPSS spits out.

Null Hypothesis

• Null hypothesis: the starting point for all tests; means "zero" effect.

• Null hypothesis examples:

  • Zero correlation between variables.
  • Independent variable has no effect.
  • In example, meditation does not improve exam performance compared to other activities.

• Experiment: Meditation Group got $81\%$, Control Group got $74\%$. We find a $7\%$ advantage between the groups.

• The p-value now means, what are the odds of getting this $7\%$ difference between the groups if meditation really has no effect? That is to say, what are the odds of getting a $7\%$ difference just by chance?

• If $p = 0.02$, there is only a $2\%$ likelihood of seeing such a $7\%$ different by chance alone, if meditation has no effect. This very small likelihood implies a real, positive effect from meditation.

• If $p = 0.5$, there is a $50\%$ likelihood of seeing a $7\%$ difference by chance alone if meditation has no effect!

• Therefore, we want a low p-value because it means there is a very low chance that a result occurred by chance.

• A low $p$-value (e.g., < 0.05) means you have "found something" meaningful in your study:

  • A statistically significant relationship between variables, or
  • A statistically significant effect on performance or behavior.

Significance of $0.05$

• There is nothing particularly special about $0.05$; is simply an agreed-upon standard.
• Analogous to the drinking age, which varies across countries.
• Setting a standard ensures that everyone is operating on the same "law."
• If your number is below $0.05$, it is (generally) taken as strong enough evidence that the result is statistically significant. If not, then it may not be.
• There is nothing magical about $0.05$, it is just what we all use.