Data 102: Binary Decision Making, Hypothesis Testing, and Decision Theory

0.0(0)
Studied by 0 people
call kaiCall Kai
learnLearn
examPractice Test
spaced repetitionSpaced Repetition
heart puzzleMatch
flashcardsFlashcards
GameKnowt Play
Card Sorting

1/29

flashcard set

Earn XP

Description and Tags

Comprehensive vocabulary flashcards covering binary decision-making, hypothesis testing (NHST), Neyman-Pearson lemma, ROC curves, multiple testing (FWER, FDR, Bonferroni, Benjamini-Hochberg), and statistical decision theory from Data 102.

30 Terms

1
New cards
<p>Confusion Matrix</p>

Confusion Matrix

A 2×22 \times 2 contingency table summarizing binary classification outcomes by mapping true reality (R∈{0,1}R \in \{0, 1\}) against decisions (D∈{0,1}D \in \{0, 1\}), tabulating True Negatives (TNTN), False Positives (FPFP), False Negatives (FNFN), and True Positives (TPTP).

2
New cards

False Positive Rate (FPR\text{FPR})

A row-wise error rate defined as the conditional probability of deciding positive (D=1D=1) when the true reality is negative (R=0R=0): FPR=P(D=1∣R=0)=FPFP+TN\text{FPR} = \mathbb{P}(D=1 \mid R=0) = \frac{FP}{FP + TN}. It corresponds to the Type I error rate and the significance level (α\alpha).

3
New cards

False Negative Rate (FNR\text{FNR})

A row-wise error rate defined as the conditional probability of deciding negative (D=0D=0) when the true reality is positive (R=1R=1): FNR=P(D=0∣R=1)=FNTP+FN\text{FNR} = \mathbb{P}(D=0 \mid R=1) = \frac{FN}{TP + FN}. It corresponds to the Type II error rate (βfn\beta_{\text{fn}}).

4
New cards

Sensitivity (TPR\text{TPR})

The probability of correctly identifying a positive condition: Sensitivity=TPR=P(D=1∣R=1)=TPTP+FN=1−FNR\text{Sensitivity} = \text{TPR} = \mathbb{P}(D=1 \mid R=1) = \frac{TP}{TP + FN} = 1 - \text{FNR}. Also referred to as statistical Power or Recall.

5
New cards

Specificity (TNR\text{TNR})

The probability of correctly identifying a negative condition: Specificity=TNR=P(D=0∣R=0)=TNTN+FP=1−FPR\text{Specificity} = \text{TNR} = \mathbb{P}(D=0 \mid R=0) = \frac{TN}{TN + FP} = 1 - \text{FPR}.

6
New cards

False Discovery Proportion (FDP\text{FDP})

A column-wise error metric defined as the proportion of declared discoveries that are actually false: FDP=P(R=0∣D=1)=FPmax⁡(D,1)=FPFP+TP\text{FDP} = \mathbb{P}(R=0 \mid D=1) = \frac{FP}{\max(D, 1)} = \frac{FP}{FP + TP}, which evaluates to 00 if no discoveries are made (D=0D=0).

7
New cards

False Omission Proportion (FOP\text{FOP})

A column-wise error metric defined as the proportion of negative decisions that are actually false negatives: FOP=P(R=1∣D=0)=FNmax⁡(W,1)=FNFN+TN\text{FOP} = \mathbb{P}(R=1 \mid D=0) = \frac{FN}{\max(W, 1)} = \frac{FN}{FN + TN}, where WW denotes the total number of negative decisions.

8
New cards

Base Rate Fallacy

The phenomenon where ignoring a very low prior prevalence (π1=P(R=1)≪1\pi_1 = \mathbb{P}(R=1) \ll 1) results in a high False Discovery Proportion (FDP≈1\text{FDP} \approx 1), despite having high sensitivity and high specificity, because false positives outnumber true positives in the total population.

9
New cards

Frequentist Paradigm

A statistical framework where the state of nature or reality (θ\theta or RR) is assumed fixed and non-random, randomness enters solely via the sampling/data-generating process, and probabilities represent long-run frequencies under identical, hypothetical repetitions.

10
New cards

Bayesian Paradigm

A statistical framework where unknown parameters and hypotheses (θ\theta or RR) are treated as random variables possessing prior distributions representing belief or knowledge, and inference is conducted conditionally on the observed data X=xX=x via the posterior distribution.

11
New cards

pp-Value

The probability, calculated under the assumption that the null hypothesis (H0H_0) is true, of observing a test statistic at least as extreme as the observed value: p(s(x))=PH0(S≥s(x))p(s(x)) = \mathbb{P}_{H_0}(S \ge s(x)). For a continuous test statistic under H0H_0, P∼Uniform(0,1)P \sim \text{Uniform}(0, 1).

12
New cards

Simple vs. Composite Hypotheses

A simple hypothesis completely specifies the data-generating probability distribution (e.g., H0:μ=5H_0: \mu = 5 with known σ2\sigma^2), whereas a composite hypothesis leaves parameters unspecified and represents a family of distributions (e.g., Ha:μ≠5H_a: \mu \neq 5).

13
New cards

Permutation Test

A simulation-based, non-parametric test where the null distribution of a test statistic is estimated by repeatedly permuting the class labels across observed data, valid under the null hypothesis that data are distributed independently and identically across classes.

14
New cards

Neyman-Pearson Lemma

A theorem stating that for simple hypotheses H0:X∼f0H_0: X \sim f_0 versus H1:X∼f1H_1: X \sim f_1, the decision rule that maximizes power (TPR\text{TPR}) for a fixed significance level α\alpha (FPR\text{FPR}) rejects H0H_0 if and only if the likelihood ratio LR(x)=f1(x)f0(x)≥η\text{LR}(x) = \frac{f_1(x)}{f_0(x)} \ge \eta, where PH0(LR(X)≥η)=α\mathbb{P}_{H_0}(\text{LR}(X) \ge \eta) = \alpha.

15
New cards
<p>Receiver Operating Characteristic (ROC) Curve</p>

Receiver Operating Characteristic (ROC) Curve

A curve that traces classifier performance across all decision thresholds by plotting True Positive Rate (TPR\text{TPR}) on the vertical axis against False Positive Rate (FPR\text{FPR}) on the horizontal axis, running from (0,0)(0,0) to (1,1)(1,1).

16
New cards

Concavity of Optimal ROC Curves

The mathematical property that an optimal likelihood ratio test produces an ROC curve whose slope dTPRdα\frac{d\text{TPR}}{d\alpha} equals the threshold η(α)\eta(\alpha); since η(α)\eta(\alpha) decreases monotonically as α\alpha increases, the second derivative satisfies d2TPRdα2≤0\frac{d^2\text{TPR}}{d\alpha^2} \le 0.

17
New cards

Convex Hull of an ROC Curve

The boundary of all achievable (FPR,TPR)(\text{FPR}, \text{TPR}) pairs obtained by randomly mixing decision thresholds or classifiers, where randomizing between two operating points yields a linear combination lying along the segment connecting them.

18
New cards

Kullback-Leibler (KL) Divergence

The expectation of the log-likelihood ratio under a given distribution, which quantifies the distinctness of two distributions. Under the Chernoff-Stein lemma, the probability of a false negative decays exponentially in sample size at a rate equal to the KL divergence.

19
New cards

Family-Wise Error Rate (FWER\text{FWER})

The probability of committing at least one false positive error across mm simultaneous hypothesis tests: FWER=P(V≥1)\text{FWER} = \mathbb{P}(V \ge 1), where VV is the count of false discoveries.

20
New cards

False Discovery Rate (FDR\text{FDR})

The expected proportion of rejected hypotheses that are false positives: FDR=E[FDP]=E[Vmax⁡(Rdisc,1)]\text{FDR} = \mathbb{E}[\text{FDP}] = \mathbb{E}\left[\frac{V}{\max(R_{\text{disc}}, 1)}\right], where VV is the number of false positives and RdiscR_{\text{disc}} is total discoveries.

21
New cards

Bonferroni Correction

A conservative multiple testing method controlling FWER≤α\text{FWER} \le \alpha across mm tests by rejecting individual null hypotheses H0,iH_{0,i} if and only if pi≤αmp_i \le \frac{\alpha}{m}, justified via Boole's inequality (union bound).

22
New cards
<p>Benjamini-Hochberg (BH) Procedure</p>

Benjamini-Hochberg (BH) Procedure

A step-up multiple testing procedure controlling FDR≤α\text{FDR} \le \alpha by sorting mm pp-values (p(1)≤⋯≤p(m)p_{(1)} \le \dots \le p_{(m)}), finding the maximum index k∗k^* where p(k∗)≤k∗mαp_{(k^*)} \le \frac{k^*}{m}\alpha, and rejecting all null hypotheses with rank k≤k∗k \le k^*.

23
New cards

Goodhart's Law

The principle that once an evaluation metric becomes a target, it ceases to be a good metric; in scientific testing, prioritizing p<0.05p < 0.05 incentivizes publication bias, file-drawer bias, and pp-hacking.

24
New cards

Loss Function

A function ℓ(θ,a)≥0\ell(\theta, a) \ge 0 in decision theory that specifies the numerical cost or penalty incurred by taking an action a∈Aa \in \mathcal{A} when the true state of nature is θ∈Θ\theta \in \Theta.

25
New cards

Frequentist Risk

The expected loss of a decision rule d(X)d(X) for a fixed, true state of nature θ\theta, integrated over all hypothetical datasets generated under the likelihood: RF(θ,d)=EX∣θ[ℓ(θ,d(X))]R_F(\theta, d) = \mathbb{E}_{X \mid \theta}[\ell(\theta, d(X))].

26
New cards

Bayesian Posterior Risk

The expected loss of an action aa conditioned on a specifically observed data sample X=xX=x, integrated over the posterior distribution of the parameter: RP(a∣x)=Eθ∣X=x[ℓ(θ,a)]R_P(a \mid x) = \mathbb{E}_{\theta \mid X=x}[\ell(\theta, a)].

27
New cards

Bayes Risk

The total expected loss of a decision rule dd averaged across both the prior distribution of the parameter θ\theta and the data distribution XX: RB(d)=Eθ[RF(θ,d)]=EX[RP(d(X)∣X)]R_B(d) = \mathbb{E}_\theta[R_F(\theta, d)] = \mathbb{E}_X[R_P(d(X) \mid X)].

28
New cards

Admissibility

A decision theory criterion where a decision rule dd is admissible if no other decision rule d′d' dominates it (i.e., there is no rule with RF(θ,d′)≤RF(θ,d)R_F(\theta, d') \le R_F(\theta, d) for all θ\theta and strict inequality for at least one θ\theta).

29
New cards

Minimax Optimality

A decision criterion that minimizes the worst-case Frequentist risk across all possible parameter values: dminimax=arg⁡min⁡dmax⁡θ∈ΘRF(θ,d)d^{\text{minimax}} = \arg\min_d \max_{\theta \in \Theta} R_F(\theta, d).

30
New cards

Pareto Frontier

The boundary of achievable objective value pairs where no single objective (e.g., error rate or risk) can be improved without degrading another. In binary classification, the ROC curve of a likelihood ratio test represents the Pareto frontier between FPR\text{FPR} and TPR\text{TPR}.