Marketing Science Stats Flashcards

Flashcards covering core statistical concepts for experimentation, including hypothesis testing, uncertainty metrics, power analysis, and causal inference terms.

Null Hypothesis ($H_0$)

The hypothesis that there is no treatment effect or no difference from the specified null value, usually zero.

Alternative Hypothesis ($H_1$)

The hypothesis representing an effect different from the null; it may be two-sided or directional.

p-value

Assuming $H_0$ is true, the probability of observing a result at least as extreme as the one observed.

Alpha ($\alpha$)

The pre-specified Type I error rate / significance threshold used to decide when to reject $H_0$.

Type I Error

A false positive: rejecting $H_0$ when $H_0$ is actually true.

Type II Error

A false negative: failing to reject $H_0$ when the specified alternative effect is actually true.

Beta ($\beta$)

The probability of a Type II error for a specified alternative effect.

Power

$1 - \beta$: the probability of rejecting $H_0$ when a specified true effect exists.

Statistical Significance

Indicates that the observed result crossed the pre-specified evidence threshold against $H_0$. It does not by itself mean the effect is large or commercially important.

Estimator

The statistical rule used to estimate an unknown population quantity, e.g. treatment mean - control mean.

Estimate

The specific number produced by the estimator in one sample, e.g. $+4\%$ lift.

Standard Deviation (SD)

Variation among individual observations/outcomes.

Standard Error (SE)

The standard deviation of the sampling distribution of an estimator; it measures uncertainty in the estimate.

Sampling Distribution

The distribution of the estimates you would obtain across many repeated random samples/experiments.

Central Limit Theorem (CLT)

With sufficiently large samples, the sampling distribution of many estimators such as means tends toward approximately normal, even if individual outcomes are skewed.

z-score

How many standard errors the estimate is from the null value: $z = \frac{\text{estimate} - \text{null}}{\text{SE}}$.

Critical z-value

The cutoff in standard-error units that defines the rejection region for a chosen alpha.

Confidence Interval (CI)

A procedure constructed so that across repeated samples, $95\%$ of such intervals would contain the true parameter.

CI Width

Represents precision: narrower intervals indicate more precision, while wider intervals indicate more uncertainty.

Minimum Detectable Effect (MDE)

The effect size the study is designed to detect at its chosen alpha and power.

SE and n Relationship

Roughly: $\text{SE} \propto \frac{1}{\sqrt{n}}$. Quadrupling sample size approximately halves the SE.

50/50 Allocation

Statistically efficient for fixed total sample and similar per-arm variability as it generally minimizes the SE of the treatment-control difference.

Absolute Lift

Example: a change from $10\%$ to $12\%$ is $+2$ percentage points.

Relative Lift

Example: a change from $10\%$ to $12\%$ calculated as $\frac{12 - 10}{10} = +20\%$ relative lift.

CUPED

A variance-reduction technique that uses predictive pre-treatment information to explain outcome variation and reduce residual variance, lowering SE.

Residual Variance

Outcome variation that remains after accounting for predictive covariates.

Stratification

A design-stage randomization technique that balances treatment/control within important pre-treatment groups.

Outcome Transformation

A pre-specified transformation of the measured outcome (e.g., capping, winsorizing, or log transform) to manage skew/extremes.

Randomization

A process to make treatment assignment independent of pre-treatment characteristics on average, supporting an unbiased causal comparison.

Confounding

A specific source of bias where another variable is related to both treatment assignment and the outcome, obscuring the treatment’s causal effect.

Contamination

When users receive exposure inconsistent with their assigned experimental condition.

Sample Ratio Mismatch (SRM)

When the observed treatment/control allocation differs from the planned allocation by more than expected from random fluctuation.

Bonferroni Correction

A specific multiplicity correction where the per-test $\alpha = \frac{\text{family alpha}}{\text{number of tests}}$.

Treatment-Effect Heterogeneity

When the causal treatment effect differs across subgroups.

Interaction Test

A direct test of whether the treatment effect in Group A minus the treatment effect in Group B is not equal to zero ($\text{Effect}_A - \text{Effect}_B \neq 0$).

Counterfactual

What would have happened to the treated users if they had not received treatment.

Incrementality

Asks which conversions happened because of exposure, calculated as the treatment outcome minus its estimated counterfactual outcome.