7 Classification models

How does ordinary KNN classification predict a query label?

It compares the query with stored training observations using a chosen distance and selects a label from the K nearest neighbors’ votes. There is no single fitted global coefficient equation.

How do very small and very large K affect KNN?

Small K gives a more locally flexible, noise-sensitive prediction. Large K smooths more but can ignore real local structure.

Why is KNN called a lazy or instance-based learner?

Its local comparison is performed when the query arrives using retained examples. Lazy does not mean that distance selection, preprocessing or tuning require no work.

When does choosing odd K prevent a tied KNN vote?

For two classes, exactly K neighbors and equal votes. It does not prevent every multiclass or weighted-vote tie; KNN regression does not need odd K for this reason.

What is Bayes’ theorem for class prediction?

P(C given x) = P(x given C)*P(C)/P(x). P(C) is the prior, P(x given C) the feature likelihood, and P(C given x) the posterior.

What is Naive Bayes’ conditional-independence assumption?

Given the class, feature likelihoods factor into a product. It does not require features to be independent across the entire population, and it need not hold exactly in practice.

What unnormalized score does Naive Bayes use for class C?

q_C = P(C)*∏_j P(x_j given C). Numerical predictors can contribute density values. Normalize all class scores to obtain posterior probabilities.

How are unnormalized class scores converted into posterior probabilities?

P(C given x) = q_C / ∑_k q_k, when the denominator is positive. Raw-score rankings identify the predicted class, but raw scores are not normalized posterior probabilities.

Example: Naive Bayes class scores are 0.144 and 0.056. What are the posteriors?

The score sum is 0.20. Posteriors are 0.144/0.20 = 0.72 and 0.056/0.20 = 0.28.

How do the prior and a categorical feature likelihood use different denominators?

A frequency prior uses class count divided by total count. A feature likelihood divides the relevant feature-level count by the count within that class.

Why can one zero categorical likelihood dominate a Naive Bayes score?

It makes the entire product zero. Smoothing adds pseudocounts to avoid eliminating a class merely because one combination was unseen in training.

What is the Laplace-smoothed categorical-likelihood formula?

P(X=c given C=k) = (n_ck + 1)/(n_k + v). n_ck counts the level in class k, n_k counts class observations for that estimate, and v is the number of possible feature levels.

What density does Gaussian Naive Bayes use for one numerical predictor?

f(x given C=k) = exp[-(x-μ_k)²/(2σ_k²)]/[σ_k*sqrt(2π)]. Use the class-specific mean and SD; a zero SD requires special handling.

Can a numerical density value exceed 1 in a Bayesian classifier?

Yes. A density height is not the probability of an exact continuous value. Posterior class probabilities are obtained by normalizing the class scores.

Why do Naive Bayes implementations often add log-likelihoods?

Products of many small positive likelihoods can underflow numerically. Taking logs turns the product into a sum while preserving score rankings.

What is the binary logistic-regression probability formula?

p(x) = 1/[1 + exp(-η)], with η = β0 + ∑_j β_j*x_j. The model estimates a probability for the defined success class.

What are odds and log odds?

Odds = p/(1-p); log odds = ln[p/(1-p)]. Logistic regression makes these log odds linear in the included predictors, not probability itself.

How is a logistic coefficient interpreted through an odds ratio?

Holding other included predictors fixed, a one-unit increase in x_j multiplies the fitted odds by exp(β_j). It does not add a constant to probability.

Example: p = 0.8 and an effect doubles the odds. What is the new probability?

Original odds = 0.8/0.2 = 4. New odds = 8, so new probability = 8/(1+8) = 8/9 ≈ 0.889, not 1.6.

What inequality converts a required logistic probability q into a linear-predictor threshold?

p(x) ≥ q is equivalent to η(x) ≥ ln[q/(1-q)], for 0 < q < 1. At q = 0.5, the required η is 0.

For η = β0 + β1*x, how do you solve p(x) ≥ q for x?

If β1 > 0: x ≥ [ln(q/(1-q)) - β0]/β1. If β1 < 0, reverse the inequality. If β1 = 0, probability is constant in x.

Example: η = -4 + 0.08x. When is the logistic probability at least 0.5?

-4 + 0.08x ≥ 0, so x ≥ 50. The threshold comes from the fitted coefficients and the chosen probability requirement.

Why is a reference category absent from a treatment-coded logistic coefficient table?

Its indicator contributions are zero and its reference prediction is encoded through the intercept/background profile. Other category coefficients compare against that reference.

Is a categorical predictor’s reference level the same as the response success class?

No. Predictor reference coding determines coefficient comparisons; response coding determines which event the predicted probability concerns. Check both separately.

Does a significant coefficient for one factor level prove the whole factor or every contrast is significant?

No. A level coefficient assesses its specified comparison. A joint factor comparison addresses a different hypothesis.

Does a small logistic p-value establish high predictive or causal importance?

No. It concerns a conditional coefficient hypothesis and depends on sample size, correlation and specification. Predictive usefulness needs evaluation; causality needs additional justification.

What is the probit probability link?

p = Φ(η), where Φ is the standard normal cumulative distribution function. It is an alternative to the logistic sigmoid; exp(β) is not its usual odds-ratio interpretation.

How are multinomial reference-class scores converted into probabilities?

Use P(C=k given x) = exp(η_k)/∑_j exp(η_j), including the reference score η_0 = 0. The reference contributes exp(0) = 1 to the denominator.

Example: class scores are 0, ln(2) and ln(3). What are the softmax probabilities?

Exponentiate to 1, 2 and 3; divide by their sum 6. Probabilities are 1/6, 2/6 and 3/6.

What binary cross-entropy objective corresponds to maximizing logistic likelihood?

Minimize -∑_i [y_i*ln(p_i) + (1-y_i)*ln(1-p_i)]. Confidently wrong probabilities receive a large penalty; the classification threshold is separate from this fit objective.

How do Naive Bayes and logistic regression differ as generative and discriminative models?

Naive Bayes models feature distributions given class and combines them with priors. Logistic regression directly models class probability given features. Neither is universally superior.

Is ordinary logistic regression nonparametric because the target is binary?

No. It is a parametric probability model and does not require the binary target to be normally distributed.