Data 102: Binary Decision Making, Hypothesis Testing, and Decision Theory

0.0(0)
Studied by 0 people
call kaiCall Kai
learnLearn
examPractice Test
spaced repetitionSpaced Repetition
heart puzzleMatch
flashcardsFlashcards
GameKnowt Play
Card Sorting

1/26

flashcard set

Earn XP

Description and Tags

Comprehensive practice flashcards covering binary decision making, error rates, Null Hypothesis Statistical Testing, the Neyman-Pearson lemma, ROC curves, multiple hypothesis testing procedures (Bonferroni and Benjamini-Hochberg), the replication crisis, and statistical decision theory.

27 Terms

1
New cards

How are the False Positive Rate (FPR\text{FPR}), False Negative Rate (FNR\text{FNR}), and False Discovery Proportion (FDP\text{FDP}) defined as conditional probabilities for decision D∈{0,1}D \in \{0, 1\} and reality R∈{0,1}R \in \{0, 1\}?

The conditional probability definitions are:

  • False Positive Rate: FPR=Pr⁡(D=1∣R=0)\text{FPR} = \Pr(D = 1 \mid R = 0)
  • False Negative Rate: FNR=Pr⁡(D=0∣R=1)\text{FNR} = \Pr(D = 0 \mid R = 1)
  • False Discovery Proportion: FDP=Pr⁡(R=0∣D=1)\text{FDP} = \Pr(R = 0 \mid D = 1)
2
New cards

In binary decision making, evaluate the truth value of these three statements:

  1. A decision procedure is sensitive if it has a false positive rate near zero.
  2. A decision procedure is specific if it has a true negative rate near one.
  3. A decision procedure with a small false positive rate must also have a small false discovery proportion.
  1. False: Sensitivity is the True Positive Rate (TPR=1−FNR\text{TPR} = 1 - \text{FNR}), which measures the detection of positive cases rather than the false positive rate.
  2. True: Specificity is defined as the True Negative Rate (TNR=1−FPR\text{TNR} = 1 - \text{FPR}).
  3. False: When the prevalence/base rate of the condition is very low, even a small FPR\text{FPR} can result in a very high FDP\text{FDP}.
3
New cards

How does Bayes' rule express the False Discovery Proportion (FDP\text{FDP}) in terms of prevalence (π1=Pr⁡(R=1)\pi_1 = \Pr(R = 1)), TPR\text{TPR}, and FPR\text{FPR}, and what happens as π1→0\pi_1 \rightarrow 0?

By Bayes' rule: FDP=Pr⁡(R=0∣D=1)=FPR⋅(1−π1)FPR⋅(1−π1)+TPR⋅π1\text{FDP} = \Pr(R = 0 \mid D = 1) = \frac{\text{FPR} \cdot (1 - \pi_1)}{\text{FPR} \cdot (1 - \pi_1) + \text{TPR} \cdot \pi_1} When positive cases are rare (π1→0\pi_1 \rightarrow 0 and π0→1\pi_0 \rightarrow 1), the denominator is dominated by false positives, driving FDP→1\text{FDP} \rightarrow 1.

4
New cards

If a diagnostic test has TPR=0.99\text{TPR} = 0.99 and TNR=0.95\text{TNR} = 0.95, what is the resulting FDP\text{FDP} at prevalence levels π1=0.5\pi_1 = 0.5, π1=0.05\pi_1 = 0.05, and π1=0.001\pi_1 = 0.001?

With FPR=1−TNR=0.05\text{FPR} = 1 - \text{TNR} = 0.05:

  • At π1=0.5\pi_1 = 0.5: FDP=0.048\text{FDP} = 0.048
  • At π1=0.05\pi_1 = 0.05: FDP=0.489\text{FDP} = 0.489
  • At π1=0.001\pi_1 = 0.001: FDP=0.981\text{FDP} = 0.981
5
New cards

What is the primary difference in how Frequentists and Bayesians interpret probability and randomness?

  • Frequentist: The true state of nature (or parameter) is fixed and unknown. Randomness comes strictly from the sampling or data-generating process over hypothetical i.i.d. repetitions.
  • Bayesian: Parameters and hypotheses are themselves random variables. Probability represents a subjective state of knowledge or degree of belief, and inferences are drawn by conditioning on the observed data.
6
New cards

What are the four steps of a Null Hypothesis Statistical Test (NHST), and what error rate does it mathematically guarantee?

  1. Observe data xx.
  2. Compute test statistic s(x)s(x).
  3. Calculate the pp-value: p(s(x))=Pr⁡(S≥s(x)∣H0)p(s(x)) = \Pr(S \ge s(x) \mid H_0), where S=s(X)S = s(X).
  4. Reject H0H_0 if p(s(x))≤αp(s(x)) \le \alpha; fail to reject otherwise. Guarantee: The procedure controls the False Positive Rate (FPR\text{FPR} / Type I error rate) at level α\alpha: Pr⁡(reject∣H0)≤α\Pr(\text{reject} \mid H_0) \le \alpha
7
New cards

What is the distribution of the pp-value under the null hypothesis H0H_0 for a continuous test statistic?

Under H0H_0, the pp-value is uniformly distributed over the interval [0,1][0, 1]: P∼Uniform(0,1)P \sim \text{Uniform}(0, 1) Consequently, for any significance level α∈[0,1]\alpha \in [0, 1]: Pr⁡(P≤α∣H0)=α\Pr(P \le \alpha \mid H_0) = \alpha

8
New cards

How does a permutation test estimate the pp-value for the difference in means between two groups defined by a binary column?

  1. Compute the observed statistic: tobs=∣xˉ1−xˉ0∣t_{\text{obs}} = |\bar{x}_1 - \bar{x}_0|.
  2. For k=1,…,Kk = 1, \dots, K (e.g., K=10,000K = 10{,}000):    (a) Randomly permute (shuffle without replacement) the binary group labels while holding the numerical data fixed.    (b) Recompute the test statistic tk=∣xˉ1,k−xˉ0,k∣t_k = |\bar{x}_{1,k} - \bar{x}_{0,k}| on the shuffled data.
  3. Return the empirical pp-value: p=1K∑k=1K1(tk≥tobs)p = \frac{1}{K}\sum_{k=1}^K \mathbf{1}(t_k \ge t_{\text{obs}})
9
New cards

What does the Neyman-Pearson Lemma state regarding testing two simple hypotheses H0:X∼f0H_0: X \sim f_0 versus H1:X∼f1H_1: X \sim f_1?

To maximize power (TPR\text{TPR} / sensitivity) for any fixed significance level (FPR=α\text{FPR} = \alpha), the optimal decision rule rejects H0H_0 when the likelihood ratio exceeds a threshold η\eta: LR(x)=f1(x)f0(x)≥η\text{LR}(x) = \frac{f_1(x)}{f_0(x)} \ge \eta Any injective, strictly monotonic increasing transformation of LR(x)\text{LR}(x) (such as log⁡(LR(x))\log(\text{LR}(x))) yields an equivalent, equally optimal test.

10
New cards

For nn i.i.d. observations x⃗=(x1,…,xn)\vec{x} = (x_1, \dots, x_n), how does the log-likelihood ratio log⁡(LR(x⃗))\log(\text{LR}(\vec{x})) decompose across individual data points?

Because independent samples have joint densities equal to the product of marginal densities: LR(x⃗)=∏i=1nf1(xi)f0(xi)=∏i=1nLR(xi)\text{LR}(\vec{x}) = \prod_{i=1}^n \frac{f_1(x_i)}{f_0(x_i)} = \prod_{i=1}^n \text{LR}(x_i) Taking the logarithm converts the product into an additive sum: log⁡(LR(x⃗))=∑i=1nlog⁡(LR(xi))\log(\text{LR}(\vec{x})) = \sum_{i=1}^n \log(\text{LR}(x_i))

11
New cards

What is the asymptotic distribution under H0H_0 of the sample mean log-likelihood ratio S=1n∑i=1nlog⁡(LR(Xi))S = \frac{1}{n}\sum_{i=1}^n \log(\text{LR}(X_i)) as n→∞n \rightarrow \infty, given mean μ0\mu_0 and standard deviation σ0\sigma_0?

By the Central Limit Theorem and Weak Law of Large Numbers, SS converges in probability to μ0\mu_0 with standard error σ0n\frac{\sigma_0}{\sqrt{n}}, and is asymptotically normal: S≈N(μ0,σ02n)S \approx \mathcal{N}\left(\mu_0, \frac{\sigma_0^2}{n}\right) The decision threshold at significance level α\alpha is: s∗(α)≈μ0+σ0nΦ−1(1−α)s^*(\alpha) \approx \mu_0 + \frac{\sigma_0}{\sqrt{n}}\Phi^{-1}(1 - \alpha) For α<0.5\alpha < 0.5, s∗(α)s^*(\alpha) approaches μ0\mu_0 from above as n→∞n \rightarrow \infty.

12
New cards

What are the key geometric properties of the Receiver Operating Characteristic (ROC) curve for an optimal likelihood ratio test?

  • Endpoints: Passes through (0,0)(0, 0) (never reject) and (1,1)(1, 1) (always reject).
  • Monotonicity: Strictly increasing in FPR\text{FPR}.
  • Concavity: Strictly concave for distinct distributions (f1≠f0f_1 \neq f_0), because the slope dTPRdFPR=η\frac{d\text{TPR}}{d\text{FPR}} = \eta equals the likelihood ratio threshold, which decreases as FPR\text{FPR} increases.
  • Pareto Optimality: Represents the Pareto frontier of achievable (FPR,TPR)(\text{FPR}, \text{TPR}) risk pairs.
13
New cards

For an exponential test H0:λ=cH_0: \lambda = c versus H1:λ=bcH_1: \lambda = bc with b>1b > 1, what are the likelihood ratio LR(t)\text{LR}(t), cutoff t∗(α)t^*(\alpha), and power TPR(α)\text{TPR}(\alpha)?

  • Likelihood ratio: LR(t)=bce−bctce−ct=be−(b−1)ct\text{LR}(t) = \frac{bce^{-bct}}{ce^{-ct}} = be^{-(b-1)ct}
  • Cutoff: Since b>1b > 1, LR(t)\text{LR}(t) decreases in tt, so rejecting for large LR\text{LR} is equivalent to T \le t^*$, where:\nt^(\alpha) = -\frac{1}{c}\log(1 - \alpha)\n- True Positive Rate:\n\text{TPR}(\alpha) = 1 - e^{-bct^(\alpha)} = 1 - (1 - \alpha)^b$$
14
New cards

How is the Family-Wise Error Rate (FWER) defined for mm hypotheses, and how does the Bonferroni procedure control it at level α\alpha?

  • Definition: FWER=Pr⁡(V≥1)\text{FWER} = \Pr(V \ge 1), the probability of committing at least one false positive error across all mm tests.
  • Bonferroni procedure: Rejects hypothesis H0,iH_{0,i} if and only if: pi≤αmp_i \le \frac{\alpha}{m}
  • Control proof: By Boole's inequality (union bound): FWER≤∑i∈H0Pr⁡(pi≤αm)=m0αm≤α\text{FWER} \le \sum_{i \in \mathcal{H}_0} \Pr\left(p_i \le \frac{\alpha}{m}\right) = m_0 \frac{\alpha}{m} \le \alpha
15
New cards

How is the False Discovery Rate (FDR) defined, and how does the Benjamini-Hochberg (BH) procedure operate?

  • Definition: FDR=E[FDP]=E[Vmax⁡(Rdisc,1)]\text{FDR} = \mathbb{E}[\text{FDP}] = \mathbb{E}\left[\frac{V}{\max(R_{\text{disc}}, 1)}\right], the expected proportion of declared discoveries that are false.
  • BH Procedure:
    1. Sort all pp-values in ascending order: p(1)≤p(2)≤⋯≤p(m)p_{(1)} \le p_{(2)} \le \dots \le p_{(m)}.
    2. Find the largest index k∗k^* such that:   p(k∗)≤k∗mαp_{(k^*)} \le \frac{k^*}{m}\alpha
    3. Reject all k∗k^* hypotheses corresponding to p(1),…,p(k∗)p_{(1)}, \dots, p_{(k^*)}.
16
New cards

Why can a hypothesis with p(k)>kmαp_{(k)} > \frac{k}{m}\alpha still be rejected under the Benjamini-Hochberg algorithm?

The Benjamini-Hochberg method is a step-up procedure. Once the largest index k∗k^* satisfying p(k∗)≤k∗mαp_{(k^*)} \le \frac{k^*}{m}\alpha is identified, the procedure rejects all hypotheses ranked 11 through k∗k^*, regardless of whether individual intermediate hypotheses with rank k<k∗k < k^* failed their respective comparisons.

17
New cards

How do the rejections and error guarantees of the Benjamini-Hochberg procedure compare to the Bonferroni procedure?

Benjamini-Hochberg makes at least as many discoveries as Bonferroni (BH rejections⊇Bonferroni rejections\text{BH rejections} \supseteq \text{Bonferroni rejections}). Bonferroni guards against making even one false positive (FWER≤α\text{FWER} \le \alpha), making it highly conservative. Benjamini-Hochberg tolerates false positives as long as they are balanced by sufficient true positives (FDR≤α\text{FDR} \le \alpha).

18
New cards

What empirical power statistics were reported in literature for neuroscience, psychology, and biomedical clinical trials regarding the replication crisis?

  • Neuroscience and behavior: Average power of only 8%–31%8\%\text{--}31\% (Button et al. 2013).
  • Psychology: Average power of 44%44\% despite 92%92\% of studies reporting statistically significant results (Szucs and Ioannidis 2017).
  • Cochrane Database of 20,00020{,}000 clinical trials: Median power was 13%13\%, with only 12%12\% of studies achieving power >80%> 80\%.
19
New cards

What is Goodhart's Law, and how does it explain questionable research practices like pp-hacking and HARKing?

Goodhart's Law states: "Once a metric becomes a goal, it ceases to be a valid metric." When achieving p<0.05p < 0.05 becomes the primary target for publication, researchers may engage in:

  • pp-hacking: collecting data until significance is reached, or selectively excluding outliers.
  • HARKing: Hypothesizing After Results are Known.
  • Publication/file-drawer bias: only publishing significant positive results.
20
New cards

How are Frequentist Risk, Posterior Risk, and Bayes Risk mathematically defined for a loss function ℓ(d(x),θ)\ell(d(x), \theta)?

  • Frequentist Risk (average over hypothetical data given reality): RF(θ)=EX∣Θ=θ[ℓ(d(X),θ)]R_F(\theta) = \mathbb{E}_{X \mid \Theta = \theta}[\ell(d(X), \theta)]
  • Posterior Risk (average over possible realities given observed data): Rp(x)=EΘ∣X=x[ℓ(d(x),Θ)]R_p(x) = \mathbb{E}_{\Theta \mid X = x}[\ell(d(x), \Theta)]
  • Bayes Risk (average over both data and reality): RB(d)=EΘ,X[ℓ(d(X),Θ)]=EΘ[RF(Θ)]=EX[Rp(X)]R_B(d) = \mathbb{E}_{\Theta, X}[\ell(d(X), \Theta)] = \mathbb{E}_\Theta[R_F(\Theta)] = \mathbb{E}_X[R_p(X)]
21
New cards

Under 0-1 loss in simple hypothesis testing, what are the Frequentist risks under θ=0\theta = 0 and θ=1\theta = 1?

With 0-1 loss (ℓ(d,θ)=1(d≠θ)\ell(d, \theta) = \mathbf{1}(d \neq \theta)), Frequentist risk equals the probability of error:

  • If θ=0\theta = 0: RF(0)=FPRR_F(0) = \text{FPR} (minimizing risk maximizes specificity).
  • If θ=1\theta = 1: RF(1)=FNRR_F(1) = \text{FNR} (minimizing risk maximizes sensitivity). These two objectives directly conflict because reducing FPR\text{FPR} increases FNR\text{FNR}.
22
New cards

In decision theory, what is a dominated decision rule, and what is an admissible decision rule?

  • Dominated Rule: A decision rule d1d_1 is dominated by d2d_2 if RF(θ,d2)≤RF(θ,d1)R_F(\theta, d_2) \le R_F(\theta, d_1) for all θ∈Θ\theta \in \Theta, with strict inequality for at least one θ\theta.
  • Admissible Rule: A decision rule is admissible if it is not dominated by any other decision rule.
23
New cards

What is minimax optimality in decision theory, and what is its primary limitation?

  • Definition: A minimax rule minimizes the maximum (worst-case) Frequentist risk across all parameter states: d∗=arg⁡min⁡dmax⁡θ∈ΘRF(θ,d)d^* = \arg\min_d \max_{\theta \in \Theta} R_F(\theta, d)
  • Limitation: It is often overly conservative because it optimizes against the worst-case scenario, which may be extremely rare under realistic prior distributions.
24
New cards

Why does minimizing the posterior risk pointwise for each observation X=xX = x minimize the overall Bayes risk?

Since the Bayes risk is the expectation of posterior risk over the marginal distribution of the data: RB=EX[Rp(X)]=∫XRp(d(x)∣x)p(x) dxR_B = \mathbb{E}_X[R_p(X)] = \int_{\mathcal{X}} R_p(d(x) \mid x) p(x) \, dx and the decision d(x)d(x) can be selected independently for each observed xx, choosing d∗(x)=arg⁡min⁡jRp(j∣x)d^*(x) = \arg\min_j R_p(j \mid x) minimizes the integrand at every point, globally minimizing RBR_B.

25
New cards

For disease prevalence π1=0.04\pi_1 = 0.04, FPR=α\text{FPR} = \alpha, TPR=β=1−(1−α)2\text{TPR} = \beta = 1 - (1-\alpha)^2, and losses ℓ(1∣0)=1\ell(1 \mid 0) = 1 and ℓ(0∣1)=k\ell(0 \mid 1) = k, what is the Bayes risk RB(α,k)R_B(\alpha, k)?

The Bayes risk averages loss over both states of reality: RB(α,k)=ℓ(1∣0)Pr⁡(R=0)FPR+ℓ(0∣1)Pr⁡(R=1)FNRR_B(\alpha, k) = \ell(1 \mid 0)\Pr(R = 0)\text{FPR} + \ell(0 \mid 1)\Pr(R = 1)\text{FNR}RB(α,k)=1⋅(0.96)⋅α+k⋅(0.04)⋅(1−β)R_B(\alpha, k) = 1 \cdot (0.96) \cdot \alpha + k \cdot (0.04) \cdot (1 - \beta) Substituting 1−β=(1−α)21 - \beta = (1 - \alpha)^2: RB(α,k)=0.96α+0.04k(1−α)2R_B(\alpha, k) = 0.96\alpha + 0.04k(1 - \alpha)^2

26
New cards

How does the optimal threshold α∗(k)\alpha^*(k) minimizing Bayes risk change as false negative cost kk increases, or as disease prevalence increases?

  • As false negative penalty kk increases: Missing a true case becomes costlier, so the test threshold lowers to capture more positive cases, causing α∗(k)\alpha^*(k) to increase.
  • As prevalence increases: True positive cases become more frequent, making aggressive positive classification more advantageous, which also causes α∗(k)\alpha^*(k) to increase.
27
New cards

Given a positive test result with posterior probability Pr⁡(R=1∣D=1)=117\Pr(R = 1 \mid D = 1) = \frac{1}{17}, false negative penalty kk, and false positive penalty 11, under what condition should treatment be recommended?

  • Posterior risk if untreated: k⋅Pr⁡(R=1∣D=1)=k17k \cdot \Pr(R = 1 \mid D = 1) = \frac{k}{17}
  • Posterior risk if treated: 1⋅Pr⁡(R=0∣D=1)=1−117=16171 \cdot \Pr(R = 0 \mid D = 1) = 1 - \frac{1}{17} = \frac{16}{17} Treatment should be recommended if the posterior risk of treating is smaller than not treating: k17>1617  ⟹  k>16\frac{k}{17} > \frac{16}{17} \implies k > 16 (Do not treat if k<16k < 16, and either option is acceptable if k=16k = 16.)