1/26
Comprehensive practice flashcards covering binary decision making, error rates, Null Hypothesis Statistical Testing, the Neyman-Pearson lemma, ROC curves, multiple hypothesis testing procedures (Bonferroni and Benjamini-Hochberg), the replication crisis, and statistical decision theory.
How are the False Positive Rate (FPR), False Negative Rate (FNR), and False Discovery Proportion (FDP) defined as conditional probabilities for decision D∈{0,1} and reality R∈{0,1}?
The conditional probability definitions are:
In binary decision making, evaluate the truth value of these three statements:
How does Bayes' rule express the False Discovery Proportion (FDP) in terms of prevalence (π1=Pr(R=1)), TPR, and FPR, and what happens as π1→0?
By Bayes' rule: FDP=Pr(R=0∣D=1)=FPR⋅(1−π1)+TPR⋅π1FPR⋅(1−π1) When positive cases are rare (π1→0 and π0→1), the denominator is dominated by false positives, driving FDP→1.
If a diagnostic test has TPR=0.99 and TNR=0.95, what is the resulting FDP at prevalence levels π1=0.5, π1=0.05, and π1=0.001?
With FPR=1−TNR=0.05:
What is the primary difference in how Frequentists and Bayesians interpret probability and randomness?
What are the four steps of a Null Hypothesis Statistical Test (NHST), and what error rate does it mathematically guarantee?
What is the distribution of the p-value under the null hypothesis H0 for a continuous test statistic?
Under H0, the p-value is uniformly distributed over the interval [0,1]: P∼Uniform(0,1) Consequently, for any significance level α∈[0,1]: Pr(P≤α∣H0)=α
How does a permutation test estimate the p-value for the difference in means between two groups defined by a binary column?
What does the Neyman-Pearson Lemma state regarding testing two simple hypotheses H0:X∼f0 versus H1:X∼f1?
To maximize power (TPR / sensitivity) for any fixed significance level (FPR=α), the optimal decision rule rejects H0 when the likelihood ratio exceeds a threshold η: LR(x)=f0(x)f1(x)≥η Any injective, strictly monotonic increasing transformation of LR(x) (such as log(LR(x))) yields an equivalent, equally optimal test.
For n i.i.d. observations x=(x1,…,xn), how does the log-likelihood ratio log(LR(x)) decompose across individual data points?
Because independent samples have joint densities equal to the product of marginal densities: LR(x)=∏i=1nf0(xi)f1(xi)=∏i=1nLR(xi) Taking the logarithm converts the product into an additive sum: log(LR(x))=∑i=1nlog(LR(xi))
What is the asymptotic distribution under H0 of the sample mean log-likelihood ratio S=n1∑i=1nlog(LR(Xi)) as n→∞, given mean μ0 and standard deviation σ0?
By the Central Limit Theorem and Weak Law of Large Numbers, S converges in probability to μ0 with standard error nσ0, and is asymptotically normal: S≈N(μ0,nσ02) The decision threshold at significance level α is: s∗(α)≈μ0+nσ0Φ−1(1−α) For α<0.5, s∗(α) approaches μ0 from above as n→∞.
What are the key geometric properties of the Receiver Operating Characteristic (ROC) curve for an optimal likelihood ratio test?
For an exponential test H0:λ=c versus H1:λ=bc with b>1, what are the likelihood ratio LR(t), cutoff t∗(α), and power TPR(α)?
How is the Family-Wise Error Rate (FWER) defined for m hypotheses, and how does the Bonferroni procedure control it at level α?
How is the False Discovery Rate (FDR) defined, and how does the Benjamini-Hochberg (BH) procedure operate?
Why can a hypothesis with p(k)>mkα still be rejected under the Benjamini-Hochberg algorithm?
The Benjamini-Hochberg method is a step-up procedure. Once the largest index k∗ satisfying p(k∗)≤mk∗α is identified, the procedure rejects all hypotheses ranked 1 through k∗, regardless of whether individual intermediate hypotheses with rank k<k∗ failed their respective comparisons.
How do the rejections and error guarantees of the Benjamini-Hochberg procedure compare to the Bonferroni procedure?
Benjamini-Hochberg makes at least as many discoveries as Bonferroni (BH rejections⊇Bonferroni rejections). Bonferroni guards against making even one false positive (FWER≤α), making it highly conservative. Benjamini-Hochberg tolerates false positives as long as they are balanced by sufficient true positives (FDR≤α).
What empirical power statistics were reported in literature for neuroscience, psychology, and biomedical clinical trials regarding the replication crisis?
What is Goodhart's Law, and how does it explain questionable research practices like p-hacking and HARKing?
Goodhart's Law states: "Once a metric becomes a goal, it ceases to be a valid metric." When achieving p<0.05 becomes the primary target for publication, researchers may engage in:
How are Frequentist Risk, Posterior Risk, and Bayes Risk mathematically defined for a loss function ℓ(d(x),θ)?
Under 0-1 loss in simple hypothesis testing, what are the Frequentist risks under θ=0 and θ=1?
With 0-1 loss (ℓ(d,θ)=1(d=θ)), Frequentist risk equals the probability of error:
In decision theory, what is a dominated decision rule, and what is an admissible decision rule?
What is minimax optimality in decision theory, and what is its primary limitation?
Why does minimizing the posterior risk pointwise for each observation X=x minimize the overall Bayes risk?
Since the Bayes risk is the expectation of posterior risk over the marginal distribution of the data: RB=EX[Rp(X)]=∫XRp(d(x)∣x)p(x)dx and the decision d(x) can be selected independently for each observed x, choosing d∗(x)=argminjRp(j∣x) minimizes the integrand at every point, globally minimizing RB.
For disease prevalence π1=0.04, FPR=α, TPR=β=1−(1−α)2, and losses ℓ(1∣0)=1 and ℓ(0∣1)=k, what is the Bayes risk RB(α,k)?
The Bayes risk averages loss over both states of reality: RB(α,k)=ℓ(1∣0)Pr(R=0)FPR+ℓ(0∣1)Pr(R=1)FNRRB(α,k)=1⋅(0.96)⋅α+k⋅(0.04)⋅(1−β) Substituting 1−β=(1−α)2: RB(α,k)=0.96α+0.04k(1−α)2
How does the optimal threshold α∗(k) minimizing Bayes risk change as false negative cost k increases, or as disease prevalence increases?
Given a positive test result with posterior probability Pr(R=1∣D=1)=171, false negative penalty k, and false positive penalty 1, under what condition should treatment be recommended?