Midterm 1 - Statistics and Experimental Design

Science

The concerted human effort to increase our understanding of the history of the natural world and how it works, using observable physical evidence as the basis of that understanding

Scientific Discovery

This is accomplished through observation of natural phenomena, and experimentation that tries to simulate natural processes under controlled conditions.

Hypothesis

Propositions that seek to explain observations

Theory

Provides a coherent set of propositions that explain a broad class of phenomena, are supported by extensive factual evidence, and may be used to predict future observations

Statistics in Science

Provides a method for analyzing and interpreting data objectively. To this end, it provides methods for separating real patterns from random noise and quantifying uncertainty.

Experimental Design in Science

Provides the logical structure that makes scientific inference possible. Ensures that data collected can answer a specific question, support causal conclusion, and be analyzed statistically.

Experimental Design

• Defines what will be manipulated (independent variable) and what will be measured (dependent variable)

• Selects appropriate controls and population (randomization)

• Identifies confounding variables before they bias results

• Converts vague questions into testable hypotheses

• Isolates cause-and-effect relationships in complex systems

• Allows scientists to distinguish correlation vs. causation

Karl Popper

Argued that all hypotheses and scientific theories should be based on the logic of falsification.

To prove a hypothesis, one should seek contradictory evidence, rather than confirmatory evidence. If you find one thing that contradicts it, one can definitively reject the hypothesis.

Inductive Reasoning

The accumulation of multiple corroborating observations causes scientists to propose a general theory/hypothesis (e.g. after observing many metals expand when heated, one infers that all metals expand when heated).

Deductive Reasoning

The critical evaluation of a general theory/hypothesis by testing logical predictions of it, or attempting to falsify it (e.g. if metals expand when heated, then all iron rods should expand when heated. If not, reject the hypothesis).

Hypothetico-Deductive Method

Step 1: Postulate a hypothesis (educated guess) or a theory that explains some phenomenon

Step 2: Search for evidence that contradicts this hypothesis/theory by testing specific predictions that flow logically from it, using controlled experiments

Prediction

a forecast or extrapolation from the current state of the system of interest. If valid, it is specific and falsifiable.

Experiment

a controlled investigation designed to evaluate the outcomes of manipulating of a system of interest

Alternative (H_A) Hypothesis

a default statement that assumes there is a relationship, effect, or difference between variables (e.g. Drug A causes heart rate to increase).

Null (H₀) Hypothesis

a default statement that assumes there is no relationship, effect, or difference between variables (e.g. Drug A does not increase heart rate)

Karl Popper argued that the goal of an experiment should be to obtain evidence that rejects this hypothesis.

Anecdotal induction

• It is based on one or a few personal observations or stories.

• From a small, unsystematic sample, one draws a broad generalization.

• The core flaw is that the sample may be biased, and there is an insufficient N.

Confirmation-based induction

• When an experimenter uses repeated confirmatory observations to

  support a hypothesis/theory.

• Relies on confirmatory evidence, not efforts to falsify the hypothesis.

• Confidence increases with replication and consistency.

• Always provisional (open to future disconfirmation)

• “I know three people who felt better after taking this

  supplement, so it works.”

Post hoc induction

• One infers causality because based on a temporal sequence (i.e., A

happened, then B happened)

• The core flaw is confusing temporal order with causation

• “I started meditating, then my blood pressure dropped—so meditation caused it.”

Increased Confidence in Inductive Inferences

Scientists rely on:

• Replication across labs and methodological controls

• Converging evidence (behavior, physiology, imaging, modeling)

• Statistical controls and preregistration of hypotheses

• Causal interventions (lesions, stimulation, optogenetics) to complement correlational data

Quack Science

• The promotion of unsubstantiated, fraudulent, or false health practices, products, or beliefs presented as scientifically valid.

• Often use weak evidence to deceive or exploit people.

• Characterized by exaggerated claims, cures for many ailments, reliance on testimonials, and a refusal to acknowledge scientific doubt.

• Exploits inductive reasoning.

Theory (in science)

It is a system of explanations that ties together a whole bunch of facts. It not only explains those facts, but predicts what you ought to find from other observations and experiments.

Best (ideal) measurement in a study

The measurement that best characterizes the response (i.e., dependent) variable

Factors that prevent ideal measuements

• They may lack accurate measurement tools, owing to prohibitive costs or technological limitations.

• Practical constraints can make it difficult.

• Ethical and medical concerns often prevent clinicians from taking the most direct (and reliable) measurements.

• Investigators are specifically studying mental constructs.

Dependent Variable

Changes as a result of the independent variable manipulation. The hypothesis is that it “depends” on the independent variable.

Independent Variable

Variable that you manipulate or vary, Not influenced by any other variables in the study.

Subject Variables

Cannot be manipulated by researchers, but can be used to group research subjects into different categories (ex: age, species, subregion)

Experimental Variable

Directly manipulated by researchers (ex: different dosages of a drug)

• Two levels of manipulation indicate whether an independent variable has any effect at all

• Multiple levels indicate how the independent variable affects the dependent variable (levels).

Negative Control Group

Ensures that there is no effect to the group when there should NOT be one. Subjects receive no manipulation, a sham manipulation, or a sugar pill.

Positive Control Group

Ensures that there is an effect on the group when there SHOULD be one.

• If the group does not respond as expected, scientists know that something is wrong.

• Establishes that the researcher can produce a positive result

• Increases the validity of a negative response to a treatment.

Situational Variable

Feature of the testing environment that may affect the subject’s response (ex: noise, temperature, time of day).

Solution: implement standardized procedures and instructions to ensure that conditions are as similar as possible for all participants.

Order Effects

When each subject is subjected to all of the treatment levels, which can influence data/results over time (fatigue, progressive learning)

Solution: use counter-balancing to control for order effects (ex: 50% get stimulus 1 then 2, 50% get stimulus 2 then 1).

Subject Variables

Individual differences across subjects (ex: tiredness, physical ability, diet)

Solution: selecting a homogenous population of subjects, using subjects as their own control, blocking, matching, randomly assigning subjects to treatment levels

Blocking

Explicitly incorporates individual differences into the design, assigning an equal number of people of various differences to a group.

Matching

Matching subjects across treatment levels on the basis of characteristics (ex: genetics, sex, anxiety) that may confound the independent variable.

Randomization

Increases the chances that any pre-existing differences across subjects are distributed equally across treatments.

Experimenter Variables

Occurs when the experimenter unintentionally influences how subjects should behave.

Solution: keep the experimenter “blind” to each subject’s treatment level.

Single-Blind Procedure

Controls for expectations the subject may have about how to respond to experimental manipulation (ex: concealing the treatment from the subject),

Double-Blind Manipulation

Controls for any expectations that the subject and experimenter may have about experimental manipulation (keeping subject and experimenter ignorant to treatment assignment).

Statistical Conclusion Validity

Asks if there is a true relationship between independent and dependent variables

• Could be threatened if experiment uses an experimental design of statistical procedure with a poor ability to reject the null, violates the assumption of the statistical test (ex: non-random assignment of subjects), based on data with a lot of noise

Construct Validity

Asks if the constructs used for the independent variable are valid

• To increase validity, investigators should use multiple complimentary measurements, use operational definitions of the constructs, and show that novel measures of data agree with established construct.

Internal Validity

Asks if there is a causal relationship between the independent and dependent variables..

Applies to a specific group of subjects only.

External Validity

Asks if the results of the study can be generalized beyond the experimental situation.

True Experiment

Random assignment of experimental units to treatment conditions, coupled with manipulation of independent variable.

• Strengths: randomization distributes unknown and known confounding variables among treatment groups

• Weaknesses: Can be expensive, impractical, artificial, or unethical

• Investigators control the levels of the independent variable, extraneous factors, group assignment

Quasi-Experiment

Lacks random assignment. Researchers may still manipulate a variable or take advantage of an intervention that occurs independently.

• Strength: allow researchers to investigate potentially causal relationships when random assignment is impractical or unethical

• Weakness: selection bias and confounding can occur. Groups differ not in terms of treatment levels, but in many unknowable ways.

• Investigator controls the levels of the independent variable and extraneous factors, not group assignment.

Descriptive Experiement

Observes variables without explicitly/experimentally manipulating them. The magnitude of one variable is correlated with another, and measured.

• Strength: ability to study the world as it actually is, relative feasibility, discovery of patterns + early hypothesis formation, examination of variables that cannot be ethically or practically manipulated'

• Investigator cannot control the independent variable (doesn’t exist) or the control group assignment (also doesn’t exist), only extraneous variables.

• Can uncover causal relationships between variables.

Correlations

Can both uncover causal relationships between variables, and lead to spurious causal claims

Confounding Variable

Systematically covaries with independent variables, providing an alternative explanation for the treatment effect.

Prospective Randomized Control Trial

Participants enroll and are randomly assigned to a treatment or control group. Follow up occurs after.

Forward-looking, controlled + blind, deliberately study cause and effect.

Retrospective Study

Retrieves data from existing records and separates it into treated and non-treated groups, before analyzing the outcomes.

Backward-looking, observational, associates only.

Between-Subject Experiment

• Each subject receives different treatments

• Comparison occurs between different groups of subjects

• Subjects are not their own control

• Individual differences contribute to error

• Major concern: differences among groups/subjects

Within-Subject Experiment

• Same subjects receive all treatments

• Comparison occurs within the same individuals

• Subjects are their own control

• Contribution of individual difference to error is greatly reduced

Single-Factor Design

Compares responses to manipulations of one independent variable (ex: the effect of drug dosage on heart rate)

Two-Factor Design

Consists of two independent variables, and allows the dependent variable to be determined by both (ex: heart rate can depend on drug dosage and age).

2×2 Factorial Design

consists or 2 independent variables and 2 levels within each variable (± exercise, ± caffiene

Can test for interactions between two independent variables, indicating that the effect of one variable on the dependent variable compounds/covaries with the level of the other.

Two-Factor Between-Subject Design

Both factors are between subject factors. The experiment can test a main effect of the first, a main effect of the second, and variable 1 x variable 2 interaction, all in one.

Mixed-Model Design

Combines at least one between-subject factor with at least one within-subject factor (ex: drug treatment could be placebo vs. drug between subjects, and time could be before and after within subjects.

Test for Association

Determining whether one variable (ex: smoking) changes with another variable (incidence of lung cancer)

Test for Causality

Determining if one behavior causes a specific outcome

Ecological Study

• Experimental unit is a group (ex: nationality) rather than an individual

• Results are correlational, so causality is only speculated

• Results are limited by the mean of the independent and dependent variables

Koch Postulates (Jacob Henle and Robert Koch)

Created first experimental guidelines for causality between disease and micro-organism in the early 1800s:

• micro-organisms must be present in all patients with the disease

• the microbe must be isolated from disease patients and grown in a pure culture

• capable of transmitting the disease when introduced to a second host

Sir Bradford Hill

Developed guidelines for causality in epidemiology as a response to the correlation between smoking and lung cancer: temporality, strength, dose-response, reversibility, consistency, specificity, biological plausibility, analogy, and coherence.

Temporality

Bradford-Hill guideline that states exposure should proceed the disease, with adequate time elapsed. You cannot establish causality if exposure does not precede the disease/health outcome.

Strength of Association (effect size)

Bradford-Hill guideline that states there should be a strong association between the incidence of the disease and exposure to the event, condition, or agent.

Larger association = larger likelihood of causality.

Dose-Response (Biological Gradient)

Bradford-Hill guideline that states disease rates should increase with exposure.

Reversibility (cessation of exposure)

Termination of exposure should decrease the rate of disease.

Law of reversibility is not necessary for causality → if the pathogenic process of lung cancer has already started, quitting smoking may not reduce the risk of mortality.

Consistency

Bradford-Hill guideline that states consistent findings observed by different persons in different places with different samples strengthens the likelihood of an effect.

Biological Plausibility

Bradford-Hill guideline that states plausible mechanism exists between cause and effect (ex: tobacco is a multipotent carcinogenic mixture which can cause cancer).

Specificity

Bradford-Hill guideline that states specific population of people (ex: smokers) are impacted at a specific site of the body (respiratory system) with no other likely explanation.

Analogy

Bradford-Hill guideline that states a cause should be analogous to another established causal relationships (ex: smoking and lung cancer is analogous to that of other inhaled environmental toxins + lung cancer)

Coherency

Bradford-Hill guideline that states coherence between epidemiological and laboratory findings increase the likelihood of causality.

Necessary Cause

If A is a ___ cause of B, condition B shouldn’t occur without condition A. However, A can occur without B.

Sufficient Cause

If A is a sufficient cause of B, then when condition A occurs, condition B should follow.

Web of Causality

model that explains health outcomes and diseases as the result of multiple interacting biological, social, environmental, and behavioral factors rather than a single isolated cause