STAT 212 Exam 3

Experiments

designed to identify causal relationships

• in experiments, we have the power to assign participants to an intervention

Observational Studies

identify assiociations that may signal causation

• in observational studies, we can only observe what participants choose to do

Reasons for completing an observational study

• it may be impossible or extremely difficult to assign participants to an intervention

• it may not be ethical to assign participants to an intervention if it increases risk of harm

• in special cases, the response being studied might be rare and difficult to reproduce without gathering a very large sample size or waiting a very large time

• experiments are generally more expensive and may require time and extensive planning

Scientific Modeling

modeling as trying to represent how two variables relate to one another in some kind of process or system

• typically when we propose a scientific model, we are trying to identify causal mechanisms

Cofounder

two variables may be correlated, but not causally linked

• explains why two events are likely to occur together

Mediating Variable

two outcomes may be causally connected through a mediating variable

• if the mediating variable is disrupted, the causality chain breaks

For a variable to be a true cofounder, it must be…

• truly causing (directly or indirectly) changes in the response variable

• linked to the explanatory variable, but not necessarily in a known causal way

Stratification

the analytical process of breaking down our comparison groups into smaller subgroups based on a potential cofounder to see if differences still show up in the more targeted comparisons

Cross-Sectional Studies

collect both the explanatory and response outcome data for a single point

• data at a cross-section of someone’s life

Cohort Studies

• differ from cross-sectional studies in having some type of longitudinal element

• we have explanatory outcomes, and then at a later point in time, we see if there are differences in response outcomes

• typically prospective in form, meaning that the response variable data is not available until a later time when we collect it

Case-Control Studies

• researchers identify people based on having or not having certain response outcomes, then data for one or more explanatory variables are collected separately either by asking the participants or through records

• typically retrospective in form, meaning that the explanatory variable data is not available until we collect it later

Occasional weakness for experiments, cross-sectional studies, and cohort studies

• when studying a rare incidence response outcome, we may need a very large sample size in order to estimate the true risk in both explanatory groups with reasonable precision

• when that isn’t feasible, we may turn to case-control studies where we directly find people with each response, rather than using a natural incidence sampling approach

Risk

the probability of an adverse event occurring

• the number out of the total

• “the probability of rolling a 1 is 1/6 or 0.167”

Odds

also assesses the likelihood of an adverse event occurring, but not as a probability

• the number with divided by the number without

• “the odds of rolling a 1 would be 1 to 5 or 0.2”

• P(outcome) / P(not outcome) = # Cases With / # Cases Without

Odds Ratio

Odds(A) / Odds(B)

• the construction of an odds ratio allows it to proportionally balance out the assymmetry in our explanatory group sizes (the numerator is affected proportional to the denominator term)

• as a result, when working with case-control data, one should use an odds ratio rather than relative risk, and it may often be used as a reasonable estimate for relative risk

Advantages and Disadvantages of Cohort Studies

• allow for extended observation

  • they provide more data in real time and sometimes help researchers better construct causality arguments

• they can take a long time and may need lots of participants in rare incidence cases

Advantages and Disadvantages of Case-Control Studies

• allow for efficient data collection

  • we can directly identify people with this rare outcome rather than waiting for it to happen

• we may rely on participants’ memory of previous activity or exposure, or find records, and we typically don’t have the same detailed accounts of timing as cohort studies

Odds Ratio versus Relative Risk

• in low incidence situations, you need very large samples to detect effects

• case-control designs are an efficient option that doesn’t require an enormous sample size, but in case-control designs, relative risk cannot be calculated accurately or validly measured

• an odds ratio will exaggerate the effect in comparison to relative risk

  • the odds ratio tends to be farther from 1 than the true relative risk will be

  • the larger the sample size, the closer the odds tends to be in approximating relative risk

• an odds ratio is still valid in other designs and occasionally reported in other designs

  • but relative risk is often preferred when appropriate because it is the simpler, more straightforward measure to report

Blinded Study

participants don’t know what the comparison is, or they just don’t know which group they’re in

Double Blind Study

the people administering the intervention also do not know who is in which group

Placebo

a non-effective substance/intervention that is designed to mimic the interventional experience of the treatment factor

What does a “good experiment” do

identifies differences in the response that can only be attributed to the treatment factor and nothing else

• does a good job eliminating possible cofounders to the causal link

Single Group Pre-Post Designs

all units complete the same intervention(s) in the same order, we then compare the pre and post measures to see if there is a systematic difference on average

• control (intervention) → pre-measure → treatment intervention → post-measure

Single group pre-post studies should be used cautiously due to confounding threats from…

• placebo effect threat

• time effects difference

• test familiarity

Multi-Group Designs

in a multi-group design, we can now separate the treatment/control factors into separate groups and potentially avoid other confounding differences, such as timing differences, test familiarity, or reactance/placebo effects

• now that there are two groups, we need to be confident that these two groups are equivalent and have no systematic differences between them

Random Controlled Experiments

use random assigment to sort units

• it may be pure random assignment or with blocking by some relevant factors

Random Assignment

using a random chance process to sort units

• works well with larger groups (n > 50)

Random Assignment with Blocking

first involves vlocking units by possible cofounding factors, and then randomly assignming from each block

Blocking

identifying individual characteristics (like age, medical condition, sex, etc.) that might interact with the treatment or affect the response

• can help ensure equivalent groups when groups are small

Non-randomized Controlled Experiment

uses a non-random assignment method to sort groups

• may create systematic bias in groups!

Randomized Controlled Experiment with Repeated Measures

• by taking mulitple measures per person at different time points, we can better track how individuals respond to the treatment and understand individual variation

• it may not always be advantageous to take multiple measurements depending on what type of response measurements you are taking

Randomized Controlled Experiment with Crossover Trials

crossover trials are a special case of repeated measures

• each group completes both experimental conditions and produce response measures from both

  • group 1: treatment intervention → response measurement → control intervention → response measurement

  • group 2: control intervention → response measurement → treatment intervention → response measurement

Lingering Effects

in crossover trials, researchers do need to be wary of lingering effects during the second round

• in some studies, as a result, researchers may add buffer time between each phase

Threats to Causality

in general, ask whether the treatment factor has clearly been isolated in the comparison (choosing an appropriate placebo or comparative intervention is important!)

• group selection

• drop out differences

• test famililiarity

• timing effects

• setting effects

• independence

Group Selection

are there any systematic differences between groups?

• if we are comparing two or more groups in an experiment, the groups should be similar

• random assignment with large groups, or random assignment with blocking for smaller groups are the best way to guard against systematic differences between groups

Drop Out Differences

did drop-out differences introduce non-equivalency at the end?

• some attrition or mortality is expected in medical studies, but drop out differences can threaten the causality argument when

• drop out rates are different between grouos

• drop out reasons are systematically different between each group

• in cases where there is some level of attrition due to non-adherence, researchers may make a comparison of all intent to treat participants to ensure balance

Attrition

when participants do not contine in the study

• loss of contact

• failure to fully adhere to their treatment plan

Mortality

when participants might pass away during the study

• perhaps as a result of the condition being treated or the treatment itself

Test Familiarity

are participants simply getting better at completing the measure?

• most pertinent when the instrumentation is a duplicated mental or physical test

  • participants are getting an opportunity to practice or learn from the test!

• exacerbated in a single group pre-post design when there is no control group to compare that test familiarity bump to

• typically balances in multi-group designs, but test familiarity could still weaken the validity of the instrument itself as a reliable post-measure

Timing Effects

do systematic differences in group timing affect outcomes?

• if there are systematic differences between the groups’ intervention times, that could lead to timing-related cofounders

  • current events, weather, time-of-day differences

• it is ok if individuals happen to complete their intervention or measure at different times - the question is whether one group is systematically earlier/later than another

Setting Effects

do any other setting or experiential differences affect the response?

• placebo effect

• researcher effects

• environment condition effects

Placebo Effect

are participants improving just because they know they are recieving something?

• a concern when we don’t have a placebo/comparison treatment for the control group, or no comparison group at all

Researcher Effects

if researchers interacting with the participants know who is in which group, they may act differently around each group

• use double blinding when this is a significant effect

Environment Condition Differences

are environmental conditions different between the treatment and control conditions beyond the treatment factor you wish to study?

• people, location, context

• the concern is group differences, not individual differences

Independence

are the units in each group providing independent response outcomes?

• in experiments where people might interact with one another in their group, group dynamics may threaten independence of our data

• in extreme cases, group dynamics could turn your group of, say 30 people, into a monolith, resulting in a functional comparison of sample sizes of one

Causality Arguments

internal validity

• ask whether a factor from an explanatory variable affects change in response variable

  • does taking this medication cause an increase in serotonin levels?

  • does this vaccine cause a decrease in the likelihood for people to contract the virus?

• we typically draw stronger causality from experiments than we can from observational studies

  • we have the power to assign people to an intervention and make a controlled comparison

Generalizability

external validity

• ask whether findings in a study extend to a broader population, setting, and time

  • is this medication effective at increasing serotonin levels for adults of all ages?

  • is this vaccination effective at decreasing viral contraction with new variants of this virus

  • did the group of people we surveyed for this presidential approval poll reflect the broader U.S. population