Logistic Regression: Probabilities, Classes, and Decision Boundaries

Foundations and Motivations for Probability Estimation

• Course Framework: AI400 Advanced Data Science, AUM Fall 2026, Week 3 — Day 1, taught by Tanya Roosta, Ph.D.

• Core Focus: Understanding logistic regression through probabilities, classes, and decision boundaries.

• Motivation for Probability Estimation over Discrete Classes:

  • In engineering diagnostic scenarios, determining the likelihood of a failure is critical rather than relying solely on a hard categorical decision.

  • Example comparison of two machines:

  • Machine A: Estimated fault probability of $51.0\%$

  • Machine B: Estimated fault probability of $95.0\%$

  • At a standard classification decision threshold of $0.50$, both Machine A and Machine B receive the identical predicted label of "faulty".

  • Despite having the same predicted label, their underlying operational risk profiles differ substantially.

  • Probability estimations provide granular risk gradients that allow maintenance teams to prioritize which machines to inspect first.

  • The operational and financial cost of an undetected fault significantly affects downstream decisions and threshold selection.

• Limitations of Standard Linear Regression for Classification:

  • Linear regression fits an unconstrained line or hyperplane of the form $z = \mathbf{w}^T \mathbf{x} + b$.

  • Plain linear predictions produce outputs on the real number line $(-\infty, \infty)$, which routinely fall below $0$ or exceed $1$.

  • Probabilities must strictly obey Kolmogorov axioms and reside within the bounded range $[0, 1]$.

  • Logistic regression resolves this by passing a real-valued weighted linear score through a nonlinear mapping that squashes values into valid probabilities.

  • It serves as a simple, highly interpretable baseline statistical learning model with directly interpretable weights.

The Sigmoid Function and Mathematical Formulation

• Definition and Functional Form:

  • The sigmoid (or standard logistic) function transforms any real-valued input score $z$ into a continuous probability value $p \in (0, 1)$.

  • Mathematical equation:     $p = \frac{1}{1 + e^{-z}}$

  • $z$: Any real-valued linear score or logit ($-\infty < z < \infty$).

  • $p$: Calculated probability output ($0 < p < 1$).

  • Base of the natural logarithm: Euler's number $e \approx 2.718$.

• Step-by-Step Worked Sigmoid Calculation:

  • For a score of $z = 2$:     $e^{-2} \approx 0.135$     $p = \frac{1}{1 + 0.135} = \frac{1}{1.135} \approx 0.881$

• Reference Evaluations across Integer Scores:

  • Score $z = -2$: $p = \frac{1}{1 + e^{2}} \approx 0.119$Score $z = -1$: $p = \frac{1}{1 + e^{1}} \approx 0.269$

  • Score $z = 0$: $p = \frac{1}{1 + e^{0}} = \frac{1}{1 + 1} = 0.500$

  • Score $z = 1$: $p = \frac{1}{1 + e^{-1}} \approx 0.731$

  • Score $z = 2$: $p = \frac{1}{1 + e^{-2}} \approx 0.881$

  • Monotonicity: The sigmoid function is strictly monotonically increasing; a higher score $z$ always produces a higher probability $p$.

• Impact of Feature Weights on the Sigmoid Curve:

  • Consider a univariate linear model without bias: $z = ax$, where $x$ is the input feature and $a$ is the learned feature weight.

  • Positive Weight ($a > 0$):

  • The probability $p$ rises monotonically as $x$ increases.

  • Magnitude of Positive Weight ($a$):

  • Larger positive values of $a$ (e.g., comparing $z = 2x$ against $z = x$) create a steeper, more abrupt transition from $0$ to $1$.

  • Negative Weight ($a < 0$):

  • The probability $p$ decreases monotonically as $x$ increases (e.g., $z = -x$ reflects the curve horizontally across $x = 0$).

Mathematical Derivation of Odds and Log Odds

• Definition and Formulation of Odds:

  • Odds quantify the ratio of the probability that an event occurs to the probability that it does not occur.

  • Mathematical expression:     $\text{odds} = \frac{p}{1 - p}$

  • $p$: Probability of the primary target event (e.g., "faulty").

  • $1 - p$: Probability of the complementary alternative outcome (e.g., "non-faulty").

  • Interpretation: An odds value of $4$ indicates that four faulty cases occur for every single non-faulty case ($4:1$).

• Worked Example Calculating Odds and Recovering Probability:

  • Illustrative cohort distribution of $100$ total machines:

  • Faulty machines: $20$

  • Non-faulty machines: $80$

  • Total machines: $100$

  • Calculating probability of a fault:     $p = \frac{20}{100} = 0.20$

  • Calculating odds of a fault:     $\text{odds} = \frac{20}{80} = 0.25$

  • Recovering probability $p$ algebraically from odds:     $p = \frac{\text{odds}}{1 + \text{odds}}$     $p = \frac{0.25}{1 + 0.25} = \frac{0.25}{1.25} = 0.20$

• Log Odds (Logit Function):

  • Taking the natural logarithm of the odds maps bounded odds values in $[0, \infty)$ onto the entire real line $(-\infty, \infty)$, centered at zero.

  • Mathematical definition:     $\text{log odds} = \ln\left(\frac{p}{1 - p}\right)$

  • Correspondence Table between Probability, Odds, and Log Odds:

  • When $p = 0.20$: $\text{odds} = 0.25$, $\text{log odds} = \ln(0.25) \approx -1.386$

  • When $p = 0.50$: $\text{odds} = 1.00$, $\text{log odds} = \ln(1) = 0$

  • When $p = 0.80$: $\text{odds} = 4.00$, $\text{log odds} = \ln(4) \approx 1.386$

• Invertibility between Sigmoid and Log Odds:

  • The sigmoid function and the logit function are mathematical inverses of each other:     $p = \text{sigmoid}(z) = \frac{1}{1 + e^{-z}}$     $z = \ln\left(\frac{p}{1 - p}\right)$

  • Demonstration:

  • Given $p = 0.80$:       $z = \ln\left(\frac{0.80}{1 - 0.80}\right) = \ln(4) \approx 1.386$

  • Re-applying the sigmoid transformation to $z = 1.386$:       $p = \frac{1}{1 + e^{-1.386}} \approx \frac{1}{1 + 0.25} = \frac{1}{1.25} = 0.80$

Binary Logistic Regression Modeling and Interpretation

• Model Formulation and Probabilistic Interpretation:

  • Binary (binomial) logistic regression addresses problems with two mutually exclusive outcomes:

  • $y = 1$: Positive outcome (e.g., inspection confirms a fault)

  • $y = 0$: Negative outcome (e.g., inspection finds no fault)

  • The model directly outputs a conditional probability estimate:     $p = P(y = 1 \mid \mathbf{x})$

  • The probability of the alternative class is the complement:     $P(y = 0 \mid \mathbf{x}) = 1 - p$

  • The term "binomial" signifies the underlying two-outcome Bernoulli/binomial trial structure.

  • The probabilistic estimation always occurs prior to assigning a final categorical decision.

• Linear Predictor and Complete Model Equation:

  • The internal linear predictor (score $z$) combines the feature variables linearly:     $z = b + w_1 x_1 + w_2 x_2$

  • Predicted probability:     $p = \text{sigmoid}(z) = \frac{1}{1 + e^{-(b + w_1 x_1 + w_2 x_2)}}$

  • Variable definitions:

  • $x_1, x_2$: Measured numeric features for an individual sample.

  • $w_1, w_2$: Model weights (coefficients) estimated from training data.

  • $b$: Intercept (bias), representing the baseline log-odds score when all features equal zero ($x_1 = 0, x_2 = 0$).

• Running Industrial Case Study:

  • Feature definitions:

  • $x_1$: Scaled temperature measurement

  • $x_2$: Scaled vibration measurement

  • Target $y = 1$: Inspection confirms a mechanical fault

  • Target $y = 0$: Inspection confirms no fault (healthy)

  • Fitted Parameter Specification:     $z = -1 + 1.2 x_1 + 0.8 x_2$

  • Worked Example for a Specific Machine ($x_1 = 1$, $x_2 = 0.5$):

  • Step 1: Compute linear score $z$:       $z = -1 + 1.2(1) + 0.8(0.5) = -1 + 1.2 + 0.4 = 0.6$

  • Step 2: Compute predicted probability $p$ via sigmoid transformation:       $p = \frac{1}{1 + e^{-0.6}} \approx \frac{1}{1 + 0.5488} \approx 0.646$

  • Step 3: Classification at default threshold $t = 0.50$:

    • Decision rule: Predict class $1$ if $p \ge t$, else predict class $0$.

    • Because $0.646 \ge 0.50$, the model predicts "Faulty".

• Decision Thresholding Mechanics:

  • Classification is decoupled from probability estimation; altering threshold $t$ changes class assignments without altering calculated probabilities.

  • Scenario evaluation for $p = 0.646$:

  • Threshold $t = 0.50 \implies 0.646 \ge 0.50 \implies$ Classified as Faulty.

  • Threshold $t = 0.80 \implies 0.646 < 0.80 \implies$ Classified as Non-faulty.

  • Lowering the decision threshold flags more instances as faulty, increasing detection sensitivity at the cost of more false alarms.

• Quantitative Interpretation of Coefficients (Odds Ratios):

  • Direction of weights:

  • Positive weight ($w > 0$): Increases in the feature increase the linear score $z$ and probability $p$.

  • Negative weight ($w < 0$): Increases in the feature decrease the linear score $z$ and probability $p$.

  • Multiplicative Effect on Odds:

  • For weight $w_1 = 1.2$, a 1-unit increase in $x_1$ (holding all other features fixed) adds $1.2$ to the log odds score $z$.

  • In terms of odds, this exponentiates the odds by $e^{1.2}$:       $e^{1.2} \approx 3.32$

  • A 1-unit increase in scaled temperature multiplies the odds of a machine fault by approximately $3.32$.

  • Critical Statistical Caveat: Regression coefficients represent statistical associations within the training distribution, not proven causal mechanisms.

Model Training, Log Loss, and Regularization

• Optimization Objective:

  • Training adjusts weights $\mathbf{w}$ and bias $b$ to make the observed ground-truth labels maximally probable under the model.

  • The model should output $p \approx 1$ for confirmed faults ($y = 1$) and $p \approx 0$ for non-faults ($y = 0$).

• Log Loss (Binary Cross-Entropy Loss):

  • Used to evaluate probabilistic predictions against discrete binary outcomes.

  • Piecewise definition:

  • If true label $y = 1$:       $\text{Loss} = -\ln(p)$

  • If true label $y = 0$:       $\text{Loss} = -\ln(1 - p)$

  • Combined single-equation form:     $\text{Loss} = -[y \ln(p) + (1 - y) \ln(1 - p)]$

  • Loss characteristics:

  • Correct and confident predictions ($p \rightarrow 1$ when $y = 1$) incur minimal loss approaching $0$.

  • Incorrect and confident predictions ($p \rightarrow 0$ when $y = 1$) incur extreme, asymptotically infinite penalties.

• Worked Log Loss Comparison:

  • Consider two distinct machines that are both verified faults ($y = 1$):

  • Machine 1 Prediction: $p = 0.90$       $\text{Loss} = -\ln(0.90) \approx 0.105 \quad (\text{Small penalty})$

  • Machine 2 Prediction: $p = 0.10$       $\text{Loss} = -\ln(0.10) \approx 2.303 \quad (\text{Large penalty})$

  • Average Log Loss across the two instances:       $\text{Average Loss} = \frac{0.105 + 2.303}{2} = \frac{2.408}{2} = 1.204$

• Conceptual Iterative Training Process:

  • Step 1: Initialize model parameters (weights $\mathbf{w}$ and intercept $b$) with arbitrary baseline values.

  • Step 2: Compute predicted probabilities for every instance across the training dataset.

  • Step 3: Compute the average log loss over all training examples.

  • Step 4: Iteratively adjust parameter values to decrease the loss objective, repeating until convergence.

• L2 Regularization (Ridge Penalty):

  • Unconstrained optimization can cause weights to grow excessively large, especially when features are correlated or data is linearly separable, leading to overfitting.

  • Total training objective with L2 regularization:     $\text{Objective} = \text{Average Log Loss} + \text{Weight Penalty}$     $\text{Penalty} = \lambda (w_1^2 + w_2^2 + \dots)$

  • The hyperparameter $\lambda$ controls penalization intensity on weight magnitudes.

  • Scikit-Learn Parameterization:

  • The scikit-learn library parameterizes regularization via $C = \frac{1}{\lambda}$.

  • Smaller $C$ values correspond to larger $\lambda$ values (stronger regularization).

• Impact of Regularization Strength on Parameter Magnitudes:

  • Consider a synthetic fitted model $p = \text{sigmoid}(wx)$ optimizing Average Log Loss $+ \lambda w^2$ on identical training data:

  • When $\lambda = 0.001$: Learned weight $w = 5.17$

  • When $\lambda = 0.1$: Learned weight $w = 1.10$

  • When $\lambda = 1.0$: Learned weight $w = 0.25$

• Regularization and Probability Curve Softening:

  • Stronger penalties shrink learned weights toward zero, flattening the slope of the sigmoid curve.

  • This softening prevents extreme, overly confident probabilities near $0$ or $1$.

  • In symmetric single-feature setups, the threshold point $p = 0.50$ remains centered at $x = 0$, but the transition becomes significantly less steep.

  • Excessive regularization causes severe underfitting; optimal penalization must be selected via validation or cross-validation.

Multiclass Logistic Regression via Softmax

• Formulation of Multiclass Problems:

  • Used when an instance belongs to exactly one of $K > 2$ mutually exclusive categorical classes.

  • Running machine health diagnosis scenario ($K = 3$ classes):

  • Class 0 (Healthy): No diagnosed fault.

  • Class 1 (Bearing fault): Primary fault isolated to the bearing assembly.

  • Class 2 (Cooling fault): Primary fault isolated to the cooling subsystem.

• The Softmax Function Formulation:

  • Generalization of the logistic function to multiple classes.

  • For class $k \in \{0, 1, \dots, K-1\}$:     $p_k = \frac{\exp(z_k)}{\sum_{j=0}^{K-1} \exp(z_j)}$

  • Individual linear class score:     $z_k = b_k + w_{k1}x_1 + w_{k2}x_2$

  • Exponentiation ($\exp(z) = e^z$) guarantees that all numerator values remain strictly positive.

  • Normalization by the shared sum of exponentiated scores ensures that all class probabilities sum to exactly $1$ ($\sum_k p_k = 1.0$).

• Step-by-Step Softmax Transformation Process:

  • Step 1 (Score Calculation): Compute linear score $z_k$ for each class.

  • Step 2 (Exponentiation): Calculate $e^{z_k}$ for each class.

  • Step 3 (Normalization): Compute the denominator sum $\sum_j e^{z_j}$ and divide each $e^{z_k}$ by this sum.

• Worked Softmax Calculation Example:

  • Input class scores:

  • Healthy ($k = 0$): Score $z_0 = 2$

  • Bearing fault ($k = 1$): Score $z_1 = 1$

  • Cooling fault ($k = 2$): Score $z_2 = 0$

  • Exponentiating scores:

  • $\exp(z_0) = e^2 \approx 7.389$

  • $\exp(z_1) = e^1 \approx 2.718$

  • $\exp(z_2) = e^0 = 1.000$

  • Computing normalizer denominator:     $\text{Denominator} = 7.389 + 2.718 + 1.000 = 11.107$

  • Resulting probabilities:

  • Healthy: $p_0 = \frac{7.389}{11.107} \approx 0.665 \quad (66.5\%$

  • Bearing fault: $p_1 = \frac{2.718}{11.107} \approx 0.245 \quad (24.5\%$

  • Cooling fault: $p_2 = \frac{1.000}{11.107} \approx 0.090 \quad (9.0\%$

  • Total probability: $66.5\% + 24.5\% + 9.0\% = 100.0\%$

  • Final decision: Predict "Healthy" based on the largest probability ($66.5\%$

• Cross-Class Coupling Effect:

  • Because all classes share the identical denominator sum, altering a single score alters the probabilities of all classes.

  • Numerical example:

  • Baseline scores $[z_0=2, z_1=1, z_2=0]$ produce probabilities $[66.5\%, 24.5\%, 9.0\%]$.

  • Increasing only the Healthy score from $2$ to $3$ (scores $[3, 1, 0]$):

    • $\exp(3) \approx 20.086$

    • New denominator: $20.086 + 2.718 + 1.000 = 23.804$

    • Healthy probability rises to $\frac{20.086}{23.804} \approx 84.4\%$

    • Bearing fault probability drops to $\frac{2.718}{23.804} \approx 11.4\%$

    • Cooling fault probability drops to $\frac{1.000}{23.804} \approx 4.2\%$

• Multiclass Training Loss:

  • Evaluated using categorical cross-entropy loss:     $\text{Loss} = -\ln(p_{\text{true\_class}})$

  • Only the predicted probability assigned to the true ground-truth class directly enters the loss computation for that instance.

• Equivalence of Two-Class Softmax to the Sigmoid Function:

  • For two classes ($k=0$ and $k=1$):     $p_1 = \frac{\exp(z_1)}{\exp(z_0) + \exp(z_1)} = \frac{1}{\frac{\exp(z_0)}{\exp(z_1)} + 1} = \frac{1}{1 + \exp(-(z_1 - z_0))} = \text{sigmoid}(z_1 - z_0)$

  • In a two-class setting, classification depends strictly on the difference between the two linear scores ($z_1 - z_0$).

• Multiclass vs. Multilabel Classification:

  • Multiclass Classification:

  • Classes are mutually exclusive; each instance belongs to exactly one class.

  • Architecture: Softmax normalization across class scores.

  • Multilabel Classification:

  • Multiple target categories can simultaneously be present on a single instance (e.g., a machine experiencing both a bearing fault and a cooling fault concurrently).

  • Architecture: Independent binary logistic regression models, applying one sigmoid activation per label.

Decision Boundaries and Regional Geometries

• Decision Boundary Mechanics:

  • A decision boundary is the geometric hypersurface in feature space that separates regions assigned to different categorical classes.

  • At the default threshold of $t = 0.50$, the boundary occurs precisely where:     $p = 0.50 \iff z = 0$

  • For a two-feature linear model:     $b + w_1 x_1 + w_2 x_2 = 0$

  • Note on linearity: Although the sigmoid activation function is nonlinear and produces a curved probability surface over the input space, the resulting decision boundary in feature space is a straight line.

• Single-Feature Decision Boundary (Point):

  • Given model: $z = 2x - 4$

  • Boundary condition ($z = 0$):     $2x - 4 = 0 \implies x = 2$

  • The boundary consists of a single scalar threshold point at $x = 2$.

• Two-Feature Linear Decision Boundary (Line):

  • Given model: $z = x_1 + x_2 - 3$

  • Decision boundary at threshold $p = 0.50$ ($z = 0$):     $x_1 + x_2 - 3 = 0 \implies x_2 = 3 - x_1$

  • Regional evaluation:

  • Test coordinate $(x_1 = 1, x_2 = 1)$: Score $z = 1 + 1 - 3 = -1 \implies p = \text{sigmoid}(-1) \approx 0.269 < 0.50$ (Class 0 region)

  • Test coordinate $(x_1 = 2, x_2 = 2)$: Score $z = 2 + 2 - 3 = 1 \implies p = \text{sigmoid}(1) \approx 0.731 \ge 0.50$ (Class 1 region)

• Threshold Alterations and Boundary Translation:

  • Changing the classification threshold $t$ shifts the decision boundary parallel to itself without rotating it:

  • At $t = 0.50$:       $z = 0 \implies x_1 + x_2 = 3$

  • At $t = 0.20$:       $z = \ln\left(\frac{0.20}{0.80}\right) = \ln(0.25) \approx -1.386$       $x_1 + x_2 - 3 = -1.386 \implies x_1 + x_2 \approx 1.614$

  • Lowering the classification threshold to $0.20$ translates the line toward the origin, classifying a larger area of the feature space as Class 1.

• Multiclass Linear Decision Boundaries (Convex Polygons):

  • Multiclass softmax partitions the feature space into piecewise-linear convex polygonal decision regions based on which class score is highest.

  • Illustrative three-class scoring model:

  • Healthy: $z_0 = 0$

  • Bearing fault: $z_1 = x_1 - 1$

  • Cooling fault: $z_2 = x_2 - 1$

  • Pairwise boundaries are established by equating scores:

  • Healthy vs. Bearing fault boundary: $z_1 = z_0 \implies x_1 - 1 = 0 \implies x_1 = 1$

  • Healthy vs. Cooling fault boundary: $z_2 = z_0 \implies x_2 - 1 = 0 \implies x_2 = 1$

  • Bearing vs. Cooling fault boundary: $z_1 = z_2 \implies x_1 - 1 = x_2 - 1 \implies x_1 = x_2$

• Nonlinear Decision Boundaries via Basis Expansion:

  • Standard logistic regression is intrinsically a linear classifier with respect to its inputs.

  • To produce nonlinear or curved decision boundaries, explicit nonlinear transformations (such as polynomial basis expansions) must be applied to the original features.

  • Example: Quadratic circular boundary model:     $z = x_1^2 + x_2^2 - 4$

  • Decision boundary equation ($z = 0$):     $x_1^2 + x_2^2 = 4$

  • Represents a circle with radius $r = 2$ centered at the origin:

  • Inside circle ($x_1^2 + x_2^2 < 4 \implies z < 0$): Predicted as Class 0.

  • Outside circle ($x_1^2 + x_2^2 > 4 \implies z > 0$): Predicted as Class 1.

Comparative Analysis: Logistic Regression vs. Support Vector Classifiers

• Direct Comparison Matrix:

  • Learning Goal:

  • Logistic Regression: Optimizes probability estimates by minimizing average log loss across all training samples.

  • Support Vector Classifier (SVC): Maximizes the geometric margin separating classes while penalizing margin violations using hinge loss.

  • Typical Output:

  • Logistic Regression: Continuous, well-calibrated class probabilities.

  • Support Vector Classifier (SVC): Uncalibrated decision scores and discrete categorical class labels.

  • Basic Decision Boundary:

  • Logistic Regression: Linear boundary in the space of the supplied features.

  • Support Vector Classifier (SVC): Linear boundary in the space of the supplied features for Linear SVM.

  • Handling Nonlinear Patterns:

  • Logistic Regression: Requires manual engineering of nonlinear features (e.g., polynomials, interactions).

  • Support Vector Classifier (SVC): Can map features implicitly into high-dimensional spaces via kernel functions (e.g., Radial Basis Function / RBF kernel).

  • Probability Estimation:

  • Logistic Regression: Probabilities are mathematically built into the model formulation.

  • Support Vector Classifier (SVC): Requires an explicit secondary post-processing calibration step (e.g., Platt scaling).

  • Common Requirements: Both model families require feature scaling, regularization tuning, and empirical validation splits. Neither algorithm consistently outperforms the other across all tasks.

• Model Selection Guidelines:

  • Select Logistic Regression when:

  • Calibrated probability estimates are required for operational risk decisions.

  • Direct interpretability of individual feature weights is needed.

  • Select Linear SVM alongside Logistic Regression when:

  • The underlying class distributions appear linearly separable or mostly planar.

  • Select RBF Kernel SVM when:

  • The underlying class boundaries display complex nonlinear curvature and the dataset size is moderate.

  • Selection must be driven by empirical comparison using identical validation splits and task-aligned evaluation metrics.

Best Practices for Predictive Modeling Workflows

• Reliable End-to-End Workflow Steps:

  • Step 1 (Data Splitting): Partition the dataset into training, validation, and test subsets before fitting any preprocessing components to prevent data leakage.

  • Step 2 (Feature Preprocessing & Model Fitting): Fit preprocessing transformations (such as standard scaling or min-max normalization) strictly on training data, then transform training and validation/test subsets. Fit model parameters using training data only.

  • Step 3 (Hyperparameter Tuning): Utilize validation splits or k-fold cross-validation to select regularization strength ($\lambda$ or $C$) and feature combinations.

  • Step 4 (Threshold Selection & Final Evaluation): Select the operational classification threshold $t$ based on application-specific utility functions, and evaluate final performance once on the hold-out test set.

• Logistic Regression Strengths:

  • Produces direct, calibrated class probabilities.

  • Feature weights describe direct log-odds associations.

  • Computationally efficient, serving as an optimal baseline model.

  • L2 and L1 regularization mitigate overfitting on large feature sets.

• Logistic Regression Limitations:

  • Probability calibration can degrade when evaluated on shifted, out-of-distribution data.

  • Multicollinearity among correlated features destabilizes weight estimates and obscures individual feature interpretation.

  • Cannot capture complex nonlinear interactions natively without explicit feature engineering.

  • Coefficients indicate statistical association, not causal mechanisms.

Model Evaluation Metrics and Classification Trade-Offs

• Core Evaluation Metrics:

  • Accuracy: The proportion of overall correct classifications out of total predictions:     $\text{Accuracy} = \frac{\text{True Positives} + \text{True Negatives}}{\text{Total Instances}}$

  • Precision: The proportion of positive predictions that are truly positive:     $\text{Precision} = \frac{\text{True Positives}}{\text{True Positives} + \text{False Positives}}$

  • Recall (Sensitivity): The proportion of actual positive cases detected by the model:     $\text{Recall} = \frac{\text{True Positives}}{\text{True Positives} + \text{False Negatives}}$

  • Log Loss and Calibration: Measures assessing the accuracy of continuous predicted probabilities.

• The Accuracy Paradox in Imbalanced Regimes:

  • In datasets where positive events (such as equipment faults) are rare, high accuracy alone is severely misleading.

  • A naive classifier that systematically predicts "non-faulty" for every machine will achieve high raw accuracy while failing completely to identify any actual faults.

• Worked Confusion Matrix Evaluation:

  • Inspection evaluation cohort of $100$ total machines:

  • True Positives (Actual Faulty, Flagged Faulty): $8$

  • False Negatives (Actual Faulty, Not Flagged): $2$

  • False Positives (Actual Non-faulty, Flagged Faulty): $2$

  • True Negatives (Actual Non-faulty, Not Flagged): $88$

  • Computed Performance Metrics:

  • Precision:       $\text{Precision} = \frac{8}{8 + 2} = \frac{8}{10} = 0.80 \quad (80\%$

  • Recall:       $\text{Recall} = \frac{8}{8 + 2} = \frac{8}{10} = 0.80 \quad (80\%$

  • Accuracy:       $\text{Accuracy} = \frac{8 + 88}{100} = \frac{96}{100} = 0.96 \quad (96\%$

  • Contrast with Naive Baseline:

  • A trivial baseline that always predicts "non-faulty" achieves $90\%$ accuracy ($90$ true non-faulty machines out of $100$), but has a recall of $0\%$, detecting zero faulty machines.

Synthesis of Conceptual Foundations

• Sigmoid Transformation: Converts a single real-valued scalar score $z$ into a valid binary class probability $p$.

• Log Odds Transformation: Provides mathematical grounding and physical interpretability to the linear score $z$.

• Softmax Transformation: Extends the binary sigmoid formulation to multiclass probability distributions over $K$ mutually exclusive classes.

• Decision Boundary Geometry: Determined jointly by the feature representations, parameter weights, and chosen probability threshold $t$.

• Generalization Pipeline: Disciplined data splitting, feature scaling, regularization, and threshold tuning govern whether a model successfully generalizes to unseen real-world environments.