1/29
Comprehensive vocabulary flashcards covering binary decision-making, hypothesis testing (NHST), Neyman-Pearson lemma, ROC curves, multiple testing (FWER, FDR, Bonferroni, Benjamini-Hochberg), and statistical decision theory from Data 102.

Confusion Matrix
A 2×2 contingency table summarizing binary classification outcomes by mapping true reality (R∈{0,1}) against decisions (D∈{0,1}), tabulating True Negatives (TN), False Positives (FP), False Negatives (FN), and True Positives (TP).
False Positive Rate (FPR)
A row-wise error rate defined as the conditional probability of deciding positive (D=1) when the true reality is negative (R=0): FPR=P(D=1∣R=0)=FP+TNFP. It corresponds to the Type I error rate and the significance level (α).
False Negative Rate (FNR)
A row-wise error rate defined as the conditional probability of deciding negative (D=0) when the true reality is positive (R=1): FNR=P(D=0∣R=1)=TP+FNFN. It corresponds to the Type II error rate (βfn).
Sensitivity (TPR)
The probability of correctly identifying a positive condition: Sensitivity=TPR=P(D=1∣R=1)=TP+FNTP=1−FNR. Also referred to as statistical Power or Recall.
Specificity (TNR)
The probability of correctly identifying a negative condition: Specificity=TNR=P(D=0∣R=0)=TN+FPTN=1−FPR.
False Discovery Proportion (FDP)
A column-wise error metric defined as the proportion of declared discoveries that are actually false: FDP=P(R=0∣D=1)=max(D,1)FP=FP+TPFP, which evaluates to 0 if no discoveries are made (D=0).
False Omission Proportion (FOP)
A column-wise error metric defined as the proportion of negative decisions that are actually false negatives: FOP=P(R=1∣D=0)=max(W,1)FN=FN+TNFN, where W denotes the total number of negative decisions.
Base Rate Fallacy
The phenomenon where ignoring a very low prior prevalence (π1=P(R=1)≪1) results in a high False Discovery Proportion (FDP≈1), despite having high sensitivity and high specificity, because false positives outnumber true positives in the total population.
Frequentist Paradigm
A statistical framework where the state of nature or reality (θ or R) is assumed fixed and non-random, randomness enters solely via the sampling/data-generating process, and probabilities represent long-run frequencies under identical, hypothetical repetitions.
Bayesian Paradigm
A statistical framework where unknown parameters and hypotheses (θ or R) are treated as random variables possessing prior distributions representing belief or knowledge, and inference is conducted conditionally on the observed data X=x via the posterior distribution.
p-Value
The probability, calculated under the assumption that the null hypothesis (H0) is true, of observing a test statistic at least as extreme as the observed value: p(s(x))=PH0(S≥s(x)). For a continuous test statistic under H0, P∼Uniform(0,1).
Simple vs. Composite Hypotheses
A simple hypothesis completely specifies the data-generating probability distribution (e.g., H0:μ=5 with known σ2), whereas a composite hypothesis leaves parameters unspecified and represents a family of distributions (e.g., Ha:μ=5).
Permutation Test
A simulation-based, non-parametric test where the null distribution of a test statistic is estimated by repeatedly permuting the class labels across observed data, valid under the null hypothesis that data are distributed independently and identically across classes.
Neyman-Pearson Lemma
A theorem stating that for simple hypotheses H0:X∼f0 versus H1:X∼f1, the decision rule that maximizes power (TPR) for a fixed significance level α (FPR) rejects H0 if and only if the likelihood ratio LR(x)=f0(x)f1(x)≥η, where PH0(LR(X)≥η)=α.

Receiver Operating Characteristic (ROC) Curve
A curve that traces classifier performance across all decision thresholds by plotting True Positive Rate (TPR) on the vertical axis against False Positive Rate (FPR) on the horizontal axis, running from (0,0) to (1,1).
Concavity of Optimal ROC Curves
The mathematical property that an optimal likelihood ratio test produces an ROC curve whose slope dαdTPR equals the threshold η(α); since η(α) decreases monotonically as α increases, the second derivative satisfies dα2d2TPR≤0.
Convex Hull of an ROC Curve
The boundary of all achievable (FPR,TPR) pairs obtained by randomly mixing decision thresholds or classifiers, where randomizing between two operating points yields a linear combination lying along the segment connecting them.
Kullback-Leibler (KL) Divergence
The expectation of the log-likelihood ratio under a given distribution, which quantifies the distinctness of two distributions. Under the Chernoff-Stein lemma, the probability of a false negative decays exponentially in sample size at a rate equal to the KL divergence.
Family-Wise Error Rate (FWER)
The probability of committing at least one false positive error across m simultaneous hypothesis tests: FWER=P(V≥1), where V is the count of false discoveries.
False Discovery Rate (FDR)
The expected proportion of rejected hypotheses that are false positives: FDR=E[FDP]=E[max(Rdisc,1)V], where V is the number of false positives and Rdisc is total discoveries.
Bonferroni Correction
A conservative multiple testing method controlling FWER≤α across m tests by rejecting individual null hypotheses H0,i if and only if pi≤mα, justified via Boole's inequality (union bound).

Benjamini-Hochberg (BH) Procedure
A step-up multiple testing procedure controlling FDR≤α by sorting m p-values (p(1)≤⋯≤p(m)), finding the maximum index k∗ where p(k∗)≤mk∗α, and rejecting all null hypotheses with rank k≤k∗.
Goodhart's Law
The principle that once an evaluation metric becomes a target, it ceases to be a good metric; in scientific testing, prioritizing p<0.05 incentivizes publication bias, file-drawer bias, and p-hacking.
Loss Function
A function ℓ(θ,a)≥0 in decision theory that specifies the numerical cost or penalty incurred by taking an action a∈A when the true state of nature is θ∈Θ.
Frequentist Risk
The expected loss of a decision rule d(X) for a fixed, true state of nature θ, integrated over all hypothetical datasets generated under the likelihood: RF(θ,d)=EX∣θ[ℓ(θ,d(X))].
Bayesian Posterior Risk
The expected loss of an action a conditioned on a specifically observed data sample X=x, integrated over the posterior distribution of the parameter: RP(a∣x)=Eθ∣X=x[ℓ(θ,a)].
Bayes Risk
The total expected loss of a decision rule d averaged across both the prior distribution of the parameter θ and the data distribution X: RB(d)=Eθ[RF(θ,d)]=EX[RP(d(X)∣X)].
Admissibility
A decision theory criterion where a decision rule d is admissible if no other decision rule d′ dominates it (i.e., there is no rule with RF(θ,d′)≤RF(θ,d) for all θ and strict inequality for at least one θ).
Minimax Optimality
A decision criterion that minimizes the worst-case Frequentist risk across all possible parameter values: dminimax=argmindmaxθ∈ΘRF(θ,d).
Pareto Frontier
The boundary of achievable objective value pairs where no single objective (e.g., error rate or risk) can be improved without degrading another. In binary classification, the ROC curve of a likelihood ratio test represents the Pareto frontier between FPR and TPR.