Science Methods & Research Designs Notes

Observation

• Science is a special way of learning about the observable world, not a belief system; it requires evidence and is guided by systematic methods.
• Three basic research designs (methods) to understand people: $3$. Observation, Experiments, and Surveys.
• Observational data come from: naturalistic settings (home), laboratories, or existing data (official statistics, social media, birth statistics). Observations generate hypotheses, not conclusions.
• Example: Kurt et al. (2018) linked county-level religious affiliation to grocery-sales per store; correlation observed, but not causation due to potential confounds like age, income, and population density.

Experiments

• Purpose: establish causality. An experiment manipulates an independent variable (IV) — the treatment — and observes the effect on a dependent variable (DV).
• Structure: typically two groups — experimental (receives IV) and comparison/control (no IV). Some studies use more groups.
• Key concepts: $IV = \text{treatment}$, $DV = \text{outcome}$; aim to determine whether IV affects DV.
• Process: plan and execute with a basic sequence; rigorous method and analysis influence publication in scientific journals.
• Bias and changes: participants may alter behavior simply because they know they are in an experiment (Hawthorne effect).
• Examples:
  • A three-group study among kindergartners: music instruction, sports instruction, control; both programs improved cognition vs. control; maturation also contributed; music showed strongest effects but is not the only factor.
  • Deb Kellerman study: 502 participants; instruction tailored to ability improved understanding of natural selection.
• Takeaway: experiments provide stronger evidence for causes than observations, but replication and rigorous design are essential; no single study proves a claim.

Surveys

• Method: collect data from many people via interviews, questionnaires, or other means; quick and scalable.
• Risks: results can be biased by how questions are framed, worded, or titled.
• Example: public opinion on harm-reduction sites for drug users varied by survey title:
  • Overdose prevention title: approval ≈ $45\%$
  • Safe consumption title: approval ≈ $29\%$
• Lesson: question wording and framing influence outcomes; surveys need careful design and interpretation.

Measuring change over time

• Developmentalists study how people change or stay the same over time using three designs: cross-sectional, longitudinal, and cross-sequential.
• Cross-sectional research
  • Quick, least expensive: compare different ages at one time (e.g., reading ability at ages 5, 8, 11).
  • Limitation: groups differ in more than age (context, education, environment).
  • Example concern: impact of remote learning vs. in-class learning confounds age-group comparisons.
• Longitudinal research
  • Data collected repeatedly from the same individuals over time.
  • Example: a large birth cohort study showing early experiences (e.g., pre-school) predict college graduation; some statements: only 4\% of a Baltimore cohort graduated by age 28; preschool experience and social context mattered more than later schooling.
  • Major problem: historical context changes (technology, culture, policy) can make older data less applicable to new generations; some exposures (new chemicals, new products) are not yet trackable long-term.
• Cross-sequential research (cohort-sequential, time-sequential)
  • Combines cross-sectional and longitudinal designs; study several age cohorts, follow over time, then synthesize results.
  • Pros: disentangles age effects from history; provides rich causal inferences; more time-consuming and complex to analyze.
  • Example: Seattle Longitudinal Study: some intellectual abilities (vocabulary) increase after age 60; others (certain processing tasks) decline around age 30; math decline linked more to education than age.
  • Value: many researchers replicate across cohorts to gain the benefits of cohort-sequential insights without waiting decades.

Meta-analysis

• Definition: combines results from many studies to weigh overall evidence and reduce reliance on a single study.
• Strengths: robust against small samples or biases in one study; especially valuable in longitudinal research.
• Process: define inclusion criteria, search databases (e.g., $16$ databases), seek unpublished studies, include only high-quality designs and data, remove duplicates; assess so-called grist for analysis.
• Example finding: loneliness increases risk of heart disease (≈ $29\%$) and stroke (≈ $31\%$); meta-analytic approach strengthens confidence that social connections affect health.
• Key point: meta-analyses explain exactly how studies were chosen and weighted; they provide standardized, less biased evidence.

What have you learned? Cautions and challenges

• Science has reduced many problems (infectious diseases, illiteracy, sexism, racism) but progress is ongoing.
• No single study is conclusive; replication and converging evidence from multiple designs are essential.
• Different designs have distinct strengths and weaknesses; combining them (and meta-analyses) helps triangulate truth.