Support Vector Machines (Part 1) and Classification Evaluation

Supervised Learning Foundations: Regression vs. Classification

  • Supervised Learning Paradigms:

    • Supervised learning algorithms learn a mapping from input features to a target output using labeled training data.

    • The core distinction between tasks depends directly on the mathematical nature of the target variable.

  • Regression Mechanics:

    • Target Type: Continuous numeric quantity.

    • Core Objective: Predicts a continuous value along a real number continuum.

    • Evaluation Metrics: Typical error metrics quantify the geometric or numerical distance between predicted and observed numbers (e.g., Mean Squared Error, Mean Absolute Error).

    • Representative Engineering Questions: "How much energy will the machine consume?"

    • Domain Applications: Predicting operating temperature, remaining useful life (RUL) of rotating machinery, continuous power grid demand, and material tensile strength.


Regression predicts a continuous numeric target from an input feature
  • Classification Mechanics:

    • Target Type: Discrete categorical class labels.

    • Core Objective: Assigns an observation to one of two or more distinct categories based on its coordinates in feature space.

    • Class Cardinality:

    • Binary Classification: The target variable is restricted to exactly two classes (e.g., y∈{−1,+1}y \in \{-1, +1\} or y∈{0,1}y \in \{0, 1\}).

    • Multiclass Classification: The target variable comprises three or more mutually exclusive classes.

    • Evaluation Metrics: Focuses on correct class assignments, confusion matrices, and the asymmetric costs associated with distinct types of misclassifications.

    • Representative Engineering Questions: "Is this vibration window normal or faulty?"


Classification predicts a discrete categorical class label using a decision boundary
  • Feature Space Representation:

    • Observations / Samples: Represented as rows in a design matrix (e.g., individual sensor time-windows, physical specimens, or manufactured components).

    • Features / Dimensions: Represented as columns that define the orthogonal axes of a feature space (e.g., physical dimensions, statistical kurtosis, voltage readings).

    • Geometric Partitioning: In a 2D feature space, a class boundary is represented as a line; in a 3D feature space, it forms a planar surface; in dd-dimensional space, it forms a (d−1)(d-1)-dimensional hyperplane.


Feature space representation mapping measurements to coordinates and class labels

Support Vector Machines: Hyperplanes, Margins, and Support Vectors

  • Support Vector Machine (SVM) Classification Principles:

    • SVM is a family of supervised algorithms that identifies an optimal separating hyperplane between classes in a vector space.

    • A new, unobserved point is assigned a class label depending on which side of the decision boundary it falls.

  • Mathematical Definition of a Hyperplane:

    • In a dd-dimensional feature space Rd\mathbb{R}^d, a linear hyperplane is defined by the set of points x\mathbf{x} satisfying:     wTx+b=0\mathbf{w}^T \mathbf{x} + b = 0

    • w\mathbf{w} denotes the weight vector that is strictly normal (orthogonal) to the hyperplane.

    • bb represents the bias term (intercept), which determines the offset of the hyperplane from the origin.

    • The decision rule for an arbitrary input x\mathbf{x} is given by the sign function:     f(x)=sign(wTx+b)f(\mathbf{x}) = \text{sign}(\mathbf{w}^T \mathbf{x} + b)


Linear hyperplane separating two distinct classes in a two-dimensional feature space
  • The Maximum-Margin Principle:

    • While infinitely many valid separating hyperplanes can divide linearly separable classes, SVM seeks the unique hyperplane that maximizes the geometric margin.

    • Margin Definition: The perpendicular distance between the decision boundary (wTx+b=0\mathbf{w}^T \mathbf{x} + b = 0) and the closest data points from either class.

    • Boundary Placement: The decision boundary is oriented such that it remains equidistant from the closest observations of both classes, maximizing the separation gap.


Maximum margin decision boundary flanked by equidistant margin edges
  • Support Vectors and Sparsity:

    • Support Vectors: The critical subset of training points that lie directly on the margins (or violate them in soft-margin formulations).

    • Model Sparsity: The mathematical parameters w\mathbf{w} and bb of the decision boundary are determined entirely by the support vectors.

    • Invariance to Non-Support Vectors: Observations lying strictly outside the margin can be translated, perturbed, or removed entirely without altering the position or orientation of the optimal hyperplane.

    • Robustness: A wider margin provides higher geometric tolerance, ensuring that small random measurement shifts or sensor noise do not cause misclassifications.


Linear SVM maximum margin hyperplane highlighting the circled support vectors

Mathematical Optimization: Hard-Margin vs. Soft-Margin SVM

  • Hard-Margin SVM Formulation:

    • Assumes training data are strictly linearly separable without errors or class overlap.

    • Given nn training samples with labels yi∈{−1,+1}y_i \in \{-1, +1\}, the canonical margin boundaries are defined as:     wTxi+b=+1\mathbf{w}^T \mathbf{x}_i + b = +1     wTxi+b=−1\mathbf{w}^T \mathbf{x}_i + b = -1

    • The geometric width of the total margin between the two parallel hyperplanes equals:     Margin Width=2∥w∥\text{Margin Width} = \frac{2}{\|\mathbf{w}\|}

    • Maximizing 2∥w∥\frac{2}{\|\mathbf{w}\|} is mathematically equivalent to minimizing 12∥w∥2\frac{1}{2} \|\mathbf{w}\|^2.

    • Constrained Optimization Problem:     min⁡w,b12∥w∥2\min_{\mathbf{w}, b} \frac{1}{2} \|\mathbf{w}\|^2     subject to yi(wTxi+b)≥1∀i∈{1,…,n}\text{subject to } y_i(\mathbf{w}^T \mathbf{x}_i + b) \ge 1 \quad \forall i \in \{1, \dots, n\}

    • Failure Modes of Hard-Margin Optimization:

    • Real-world engineering measurements contain high-frequency noise, overlapping class distributions, and sensor artifacts.

    • A single outlier can cause the hard-margin optimization problem to have no feasible solution (infeasibility).

    • Even if separable, hard margins lead to extreme sensitivity and severe overfitting to borderline observations.


Hard-margin SVM enforcing zero boundary or margin violations
  • Soft-Margin SVM Formulation and Slack Variables:

    • Introduced to accommodate non-separable data and control sensitivity to noise by permitting controlled margin violations.

    • Slack Variables (ξi\xi_i): A set of non-negative slack variables ξi≥0\xi_i \ge 0 is introduced for each sample xi\mathbf{x}_i:

    • ξi=0\xi_i = 0: Point lies on or outside the correct margin boundary.

    • 0<ξi≤10 < \xi_i \le 1: Point lies within the margin band but on the correct side of the decision boundary.

    • ξi>1\xi_i > 1: Point crosses the decision boundary and is misclassified.

    • Soft-Margin Objective Function:     min⁡w,b,ξ12∥w∥2+C∑i=1nξi\min_{\mathbf{w}, b, \boldsymbol{\xi}} \frac{1}{2} \|\mathbf{w}\|^2 + C \sum_{i=1}^n \xi_i     subject to yi(wTxi+b)≥1−ξiandξi≥0∀i\text{subject to } y_i(\mathbf{w}^T \mathbf{x}_i + b) \ge 1 - \xi_i \quad \text{and} \quad \xi_i \ge 0 \quad \forall i

    • Structural Role of Objective Terms:

    • 12∥w∥2\frac{1}{2} \|\mathbf{w}\|^2 acts as a regularization term that favors a wide margin.

    • ∑i=1nξi\sum_{i=1}^n \xi_i represents an upper bound on the total number and magnitude of margin errors.


Soft-margin SVM tolerating limited margin violations to maintain a robust general boundary
  • The Regularization Hyperparameter CC:

    • CC governs the exact trade-off between margin maximization and training error penalty.

    • Small CC Value:

    • Places low penalty on margin violations (∑ξi\sum \xi_i).

    • Results in a wider margin band that tolerates more misclassifications and intrusions.

    • Higher regularization; prevents overfitting on noisy training samples.

    • Large CC Value:

    • Imposes severe penalties on any violation (ξi>0\xi_i > 0).

    • Forces the optimizer to narrow the margin in order to classify as many training instances correctly as possible.

    • Lower regularization; risks overfitting to training noise and outliers.

    • Selection Criterion: CC cannot be inferred directly from training data; it must be tuned using held-out validation data or cross-validation. When two values of CC yield indistinguishable validation performance, the smaller CC is preferred for greater stability and simpler boundaries.


Effect of the regularization parameter C showing wide margin on the left and narrow margin on the right

Nonlinear Decision Boundaries and the Kernel Trick

  • Limitations of Linear Separators:

    • When feature relationships exhibit nonlinear interactions (e.g., concentric rings, parabolic structures, radial groupings), no linear hyperplane can achieve acceptable separation.


Comparison between a linear straight boundary and a nonlinear curved boundary


Nonlinear boundary isolating an inner circular class from an outer ring
  • The Kernel Trick Concept:

    • Nonlinearly separable data in an input space Rd\mathbb{R}^d can become linearly separable when mapped into a higher-dimensional feature space H\mathcal{H} via a transformation mapping ϕ(x)\phi(\mathbf{x}).

    • The Kernel Function (K(xi,xj)K(\mathbf{x}_i, \mathbf{x}_j)): Evaluates the inner product of vectors in the transformed space directly from the original coordinates, without explicitly computing or storing the high-dimensional coordinates:     K(xi,xj)=⟨ϕ(xi),ϕ(xj)⟩K(\mathbf{x}_i, \mathbf{x}_j) = \langle \phi(\mathbf{x}_i), \phi(\mathbf{x}_j) \rangle

    • This enables the algorithm to fit a linear maximum-margin hyperplane in H\mathcal{H}, which corresponds to a smooth, curved boundary in the original input space.


Conceptual 3D mapping showing linear separation achieved by projecting concentric 2D rings into a higher dimension
  • Radial Basis Function (RBF) Kernel:

    • Also known as the Gaussian kernel, defined mathematically as:     K(xi,xj)=exp⁡(−γ∥xi−xj∥2)K(\mathbf{x}_i, \mathbf{x}_j) = \exp\left(-\gamma \|\mathbf{x}_i - \mathbf{x}_j\|^2\right)

    • Compares samples based on their squared Euclidean distance in input space.

    • Maps points into an infinite-dimensional Hilbert space.

    • The hyperparameter γ\gamma (gamma) defines the influence radius of individual support vectors:

    • Small γ\gamma: Broad Gaussian curvature; wide influence radius; smoother decision boundaries.

    • Large γ\gamma: Tight Gaussian peaks around support vectors; localized influence; risk of overfitting.


Performance comparison between linear kernel which fails and RBF and degree-2 polynomial kernels which succeed
  • Polynomial Kernel and Geometric Alignment:

    • Defined mathematically as:     K(xi,xj)=(γxiTxj+r)dK(\mathbf{x}_i, \mathbf{x}_j) = (\gamma \mathbf{x}_i^T \mathbf{x}_j + r)^d     where dd denotes the polynomial degree and rr is an independent coefficient.

    • Degree Comparison:

    • A degree-2 polynomial kernel (d=2d=2) aligns with quadratic circular and elliptical geometries, capturing concentric circles cleanly.

    • Increasing to degree-3 (d=3d=3) incorporates cubic interactions, providing excess flexibility that introduces unnatural boundary distortions into simple circular geometries.


Comparison of polynomial kernels showing degree-2 capturing the boundary and degree-3 introducing distortion

Feature Scaling, Pipelines, and Implementation

  • Critical Role of Feature Scaling:

    • Support Vector Machine optimization relies on Euclidean distance calculations: ∑k=1d(xik−xjk)2\sum_{k=1}^d (x_{ik} - x_{jk})^2.

    • If features have different scales, features with large absolute numerical values (e.g., millivolts spanning 10210^2 to 10310^3) will dominate the objective function over normalized features with smaller numerical scales (e.g., kurtosis spanning 00 to 11).

    • Contrast with Tree-Based Models: Unlike decision trees or random forests, which are scale-invariant because they split on single features monotonically, SVM algorithms require features to be scaled consistently.

    • StandardScaler: The standard preprocessing step for continuous features transforms each feature column to zero mean (μ=0\mu = 0) and unit variance (σ=1\sigma = 1):     z=x−μσz = \frac{x - \mu}{\sigma}

  • Data Leakage in Preprocessing:

    • Scaling parameters (μ\mu and σ\sigma) learn structural information from data distributions.

    • Incorrect Workflow (Data Leakage):

    1. Scaling entire dataset prior to splitting.

    2. Splitting scaled data into cross-validation folds.

    3. Evaluating performance.

    • Consequence: Validation statistics leak into the scaler, producing overly optimistic validation results.

    • Correct Workflow (Leakage-Safe):

    1. Split raw data into training and validation folds.

    2. Fit scaler strictly on the training fold.

    3. Apply the learned transformation parameters to both the training fold and the validation fold.

    4. Fit and score the SVM estimator.

  • Linear SVM Pipeline Implementation in Scikit-Learn:

    • The Pipeline construct binds the transformation and estimation steps into a single unified object, ensuring preprocessing is fitted solely on training partitions during each validation pass:

from sklearn.svm import LinearSVC
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import cross_val_score

pipe = Pipeline([
    ("scale", StandardScaler()),
    ("svm", LinearSVC(C=1.0, max_iter=5000, random_state=42)),
])
scores = cross_val_score(pipe, X, y, cv=5, scoring="f1")
print(f"CV F1: {scores.mean():.3f} +/- {scores.std():.3f}")
pipe.fit(X_train, y_train)
  • Implementation Differences: LinearSVC vs. SVC(kernel="linear"):

    • LinearSVC:

    • Implemented on the underlying liblinear C/C++ library.

    • Highly optimized for linear classification; scales linearly with the number of samples (roughly O(n)\mathcal{O}(n)).

    • Solves the primal problem; does not compute or expose explicit support vectors via support_vectors_.

    • Uses One-versus-Rest (OvR) multiclass formulation by default.

    • SVC:

    • Implemented on the underlying libsvm library.

    • Employs quadratic programming algorithms (Sequential Minimal Optimization / SMO).

    • Stores and exposes support vectors via support_vectors_ and counts per class via n_support_.

    • Uses One-versus-One (OvO) multiclass strategy internally.

    • Computational scaling is between O(n2)\mathcal{O}(n^2) and O(n3)\mathcal{O}(n^3), making training impractical beyond approximately 10510^5 observations.

  • Algorithmic Trade-Offs of SVM:

    • Strengths:

    • Highly effective in high-dimensional feature spaces where the number of features exceeds the number of samples.

    • Memory efficient because prediction depends only on the learned support vectors.

    • High flexibility in boundary shape via user-selected kernel functions.

    • Weaknesses:

    • Poor computational scaling (O(n2)\mathcal{O}(n^2) to O(n3)\mathcal{O}(n^3)) for large datasets (n>105n > 10^5).

    • Strict requirement for feature scaling.

    • Lack of native probabilistic outputs (requires expensive post-hoc Platt scaling calibration).

    • Hyperparameter sensitivity requiring systematic tuning over CC and γ\gamma.

    • Lower model interpretability compared to simple linear or decision tree models.

  • Real-World Engineering and Scientific Applications:

    • Text and Document Classification: High-dimensional, sparse TF-IDF vectors where linear boundaries excel.

    • Bioinformatics: Gene expression microarray profiles and protein sequence classification characterized by few samples (nn) and tens of thousands of features (dd).

    • Classical Computer Vision: Handwritten digit recognition (e.g., MNIST baselines) and facial recognition feature classification.

    • Fault Detection and Anomaly Monitoring: One-Class SVM trained on normal operational vibrations to detect impending mechanical anomalies.

    • Manufacturing Quality Control: Classifying in-specification vs. out-of-specification fabricated components using multi-axis sensor data.

    • Financial Credit Scoring and Bankruptcy: Finding robust separation hyperplanes across moderately dimensional corporate balance sheet metrics.

Classification Metrics for Imbalanced Engineering Problems

  • The Confusion Matrix:

    • A 2×22 \times 2 contingency table categorizing actual class status against model predictions.


Predicted Positive

Predicted Negative

Actual Positive

True Positive (TP)

False Negative (FN)

Actual Negative

False Positive (FP)

True Negative (TN)

  • The Accuracy Paradox in Rare Event Detection:

    • Accuracy measures the ratio of all correct predictions to total cases:     Accuracy=TP+TNTP+TN+FP+FN\text{Accuracy} = \frac{\text{TP} + \text{TN}}{\text{TP} + \text{TN} + \text{FP} + \text{FN}}

    • Failure Scenario: Consider an industrial inspection pipeline inspecting 1,0001{,}000 fabricated components where exactly 1010 are defective (1%1\% defect rate) and 990990 are normal (99%99\%).

    • A trivial, non-operational model predicting every unit as normal yields:     Accuracy=0+9901,000=99%\text{Accuracy} = \frac{0 + 990}{1{,}000} = 99\%

    • Defects Identified: 00 out of 1010.

    • Defect Recall: 0%0\%.

    • Conclusion: Overall accuracy masks catastrophic failure in imbalanced datasets. Models must be evaluated using class-specific metrics.

  • Precision: Reliability of Positive Alarms:

    • Fraction of predicted positive labels that are actually positive:     Precision=TPTP+FP\text{Precision} = \frac{\text{TP}}{\text{TP} + \text{FP}}

    • Evaluated across the predicted positive column.

    • Application: Critical when false alarms are costly, disruptive, or cause alarm fatigue (e.g., dispatching an offshore maintenance crew for a flagged fault; spam filters hiding legitimate operational emails; emergency shutdowns of chemical plants).

  • Recall (Sensitivity): Coverage of True Positive Events:

    • Fraction of actual positive occurrences captured by the model:     Recall=TPTP+FN\text{Recall} = \frac{\text{TP}}{\text{TP} + \text{FN}}

    • Evaluated across the actual positive row.

    • Application: Critical when missing a true incident leads to catastrophic damage (e.g., cancer diagnostics; thermal runaway detection in battery packs; structural crack detection; industrial fire detection).

  • Precision vs. Recall Engineering Prioritization:

Operating Scenario

Most Costly Error Type

Metric to Maximize

Emergency shutdown alarm

False Positive: Unnecessary production halt

Precision

Cancer screening

False Negative: Missed malignant pathology

Recall

Email spam filter

False Positive: Important business email lost

Precision

Industrial fire detection

False Negative: Uncontrolled fire hazard

Recall

  • F1 Score: Harmonic Balance:

    • The harmonic mean of precision and recall:     F1=2×Precision×RecallPrecision+Recall\text{F1} = 2 \times \frac{\text{Precision} \times \text{Recall}}{\text{Precision} + \text{Recall}}

    • Penalizes extreme imbalances between precision and recall; F1 is high only when both individual metrics are strong.

    • Model Comparison Benchmark:

Candidate Model

Precision

Recall

Resulting F1 Score

Model A

0.950.95

0.450.45

0.610.61

Model B

0.780.78

0.760.76

0.770.77

Model C

0.550.55

0.940.94

0.690.69

  • Observation: Model B provides the best balance between alarm reliability and fault coverage (F1=0.77\text{F1} = 0.77), despite Model A having higher precision and Model C having higher recall.

    • Threshold Manipulation and the ROC Curve:

  • Varying the decision threshold changes the trade-off: lowering the classification threshold increases the volume of positive predictions, raising recall while typically lowering precision.

  • Receiver Operating Characteristic (ROC): Plots True Positive Rate (Recall) along the y-axis against False Positive Rate (FPR=FPFP+TN\text{FPR} = \frac{\text{FP}}{\text{FP} + \text{TN}}) along the x-axis across all continuous decision thresholds.

  • Area Under the Curve (AUC):

    • AUC=1.0\text{AUC} = 1.0: Perfect diagnostic ranking (every positive case scores higher than every negative case).

    • AUC=0.5\text{AUC} = 0.5: Random guessing performance.

  • Caveat: ROC AUC summarizes ranking over all operating thresholds, many of which may be irrelevant in deployment. For highly imbalanced datasets, Precision-Recall curves and fixed-threshold confusion matrices provide a more realistic assessment.

    • Worked Example: Equipment Fault Detection Performance:

  • An equipment fault classifier yields the following contingency matrix over a production run:

    • TP=36\text{TP} = 36

    • FN=4\text{FN} = 4

    • FP=9\text{FP} = 9

    • TN=151\text{TN} = 151

  • Calculations:     Precision=3636+9=3645=0.80(80%)\text{Precision} = \frac{36}{36 + 9} = \frac{36}{45} = 0.80 \quad (80\%)
        Recall=3636+4=3640=0.90(90%)\text{Recall} = \frac{36}{36 + 4} = \frac{36}{40} = 0.90 \quad (90\%)
        F1=2×0.80×0.900.80+0.90=1.441.70≈0.847(84.7%)\text{F1} = 2 \times \frac{0.80 \times 0.90}{0.80 + 0.90} = \frac{1.44}{1.70} \approx 0.847 \quad (84.7\%)

  • Diagnostic Decision: While an 80%80\% precision and 90%90\% recall profile provides an 0.850.85 F1 score, engineering domain requirements determine whether missing 10%10\% of impending faults (FN=4\text{FN}=4) is acceptable when failure causes unexpected downtime.

Cross-Validation and Model Selection Workflows

  • The Fundamental Partitioning Paradigm:

    • Training accuracy is systematically optimistic because model parameters are directly optimized to fit training features.

    • Data must be separated into three distinct roles:

    • Train Set: Used exclusively to optimize model weights (w,b\mathbf{w}, b).

    • Validation Set: Used to evaluate hyperparameters, tune thresholds, and select candidate algorithms.

    • Test Set: Held strictly untouched until model development is complete; provides a final, unbiased estimate of generalization performance.

  • Instability of a Single Train/Validation Split:

    • Evaluating a candidate model on a single arbitrary train/validation split creates high variance:

    • Split A: Validation accuracy =92%= 92\% ("Looks excellent")

    • Split B: Validation accuracy =84%= 84\% ("Seems acceptable")

    • Split C: Validation accuracy =76%= 76\% ("Looks weak")

    • Changes in performance stem entirely from random partitioning rather than differences in the underlying model.

  • The kk-Fold Cross-Validation Framework:

    • Procedure:

    1. Split: Divide the development dataset into kk equal-sized partitions (folds).

    2. Fit: Train the model on k−1k-1 folds.

    3. Score: Evaluate performance metrics on the single held-out validation fold.

    4. Repeat: Iterate the process kk times so that every fold serves as the validation set exactly once.

    5. Aggregate: Compute the arithmetic mean and sample variation across all kk validation scores.

Iteration Run

Fold 1

Fold 2

Fold 3

Fold 4

Fold 5

Run 1

VALIDATE

Train

Train

Train

Train

Run 2

Train

VALIDATE

Train

Train

Train

Run 3

Train

Train

VALIDATE

Train

Train

Run 4

Train

Train

Train

VALIDATE

Train

Run 5

Train

Train

Train

Train

VALIDATE

  • Comparison: Single Split vs. kk-Fold Cross-Validation:

Evaluation Question

Single Train/Validation Split

kk-Fold Cross-Validation

Volume of data validated

Exactly one subset

Every sample validated once

Sensitivity to partition split

High sensitivity (high variance)

Lower sensitivity (smoothed variance)

Computational overhead

Low (11 model fit)

Higher (kk independent model fits)

Primary operational domain

Massive datasets where fitting is expensive

Standard development and course projects

  • Selection Criteria for Number of Folds (kk):

    • k=5k = 5: Good default balance; trains on 80%80\% of data per fold; computationally faster.

    • k=10k = 10: Standard industry benchmark; trains on 90%90\% of data per fold; smaller bias in performance estimation at the expense of higher computation.

    • Leave-One-Out Cross-Validation (LOOCV): Special case where k=nk = n; each validation fold contains exactly one sample. High computational cost and can exhibit high variance because training sets across folds are nearly identical.

  • Worked Example: Partitioning 1,000 Industrial Samples:

    • Total labeled observations: 1,0001{,}000 (including a 10%10\% fault distribution).

    • Final test holdout: Reserve 200200 observations untouched (20%20\%).

    • Development set: Remaining 800800 observations (80%80\%).

    • For k=5k = 5 folds: 800÷5=160800 \div 5 = 160 observations per fold.

    • Stratified fold composition: Maintain the 90%90\% normal (144144 samples) and 10%10\% faulty (1616 samples) ratio in each fold.

    • Each CV pass: Trains on 640640 samples (44 folds) and evaluates on 160160 samples (11 fold).

  • Interpreting Cross-Validation Scores and Variance:

    • Fold accuracies: 85%85\%, 89%89\%, 82%82\%, 87%87\%, 84%84\%.

    • Cross-Validation Mean:     Mean=85+89+82+87+845=4275=85.4%\text{Mean} = \frac{85 + 89 + 82 + 87 + 84}{5} = \frac{427}{5} = 85.4\%

    • Range / Spread: Span of 82%82\% to 89%89\% (77 percentage points).

    • Interpretation: An 85.4%85.4\% mean reflects expected baseline generalization; the 7%7\% spread highlights sensitivity to input variations.

  • Handling Correlated and Time-Series Data Structures:

    • Grouped Data (GroupKFold): When observations originate from shared entities (e.g., repeated runs on the same machine, recordings from the same patient, or measurements from specific production lines), random splitting causes data leakage. GroupKFold ensures all rows from a given group are placed entirely into the training partition or the validation partition, never split between both.

    • Temporal / Time-Series Data (TimeSeriesSplit): In chronological processes, future data must never be used to predict the past. Forward-chaining splits evaluate models using expanding historical windows:

    • Split 1: Train on January; Validate on February.

    • Split 2: Train on January–February; Validate on March.

    • Split 3: Train on January–March; Validate on April.

    • Split 4: Train on January–April; Validate on May.

    • Split 5: Train on January–May; Validate on June.

  • Candidate Model Selection Under Variance Constraints:

Candidate Model

Individual Fold Scores (\%)

Cross-Validation Mean

Performance Trade-Off

Model A

82,84,81,83,8582, 84, 81, 83, 85

83.0%83.0\%

Low variance (81–85%81\text{--}85\%), highly stable

Model B

86,80,88,82,8486, 80, 88, 82, 84

84.0%84.0\%

Highest mean performance, moderate variance

Model C

91,76,89,78,8591, 76, 89, 78, 85

83.8%83.8\%

High variance (76–91%76\text{--}91\%), unstable

  • Decision Rule: Model B achieves the highest mean accuracy (84.0%84.0\%). If the operational deployment penalizes worst-case drops, Model A is an attractive alternative due to its tight performance bound (81%81\% minimum vs. Model B's 80%80\% and Model C's 76%76\%).

    • Hyperparameter Optimization with GridSearchCV:

  • Hyperparameters (CC, γ\gamma, kernel type) cannot be learned directly via gradient descent or analytical quadratic programming.

  • GridSearchCV evaluates all combinations across a user-specified parameter grid using identical cross-validation folds.

  • Standard Execution Steps:

    1. Wrap preprocessing (e.g., StandardScaler) and estimator into an integrated Pipeline.

    2. Define hyperparameter search spaces using grid dictionaries.

    3. Fit GridSearchCV using the development set, optimizing for a targeted metric (e.g., scoring="f1").

    4. Select the best hyperparameter configuration based on the highest mean validation score.

    5. Refit the selected pipeline configuration on the entire development dataset (100%100\% of non-test data).

    6. Run an evaluation once against the untouched final test set.

    • Leakage-Safe Model-Selection Protocol:

  • Reserve Final Test Set: Partition raw data and hold the test set out immediately.

  • Choose Metric: Base selection on operational costs (e.g., Precision, Recall, F1).

  • Build Pipeline: Integrate transformers and estimators into a unified sequence.

  • Run Cross-Validation: Execute kk-fold CV or GridSearchCV within the development partition.

  • Select Candidate: Choose the optimal model architecture and hyperparameter settings.

  • Refit and Final Test: Refit the chosen configuration across all development data and evaluate once on the test holdout.

Multiclass Classification Strategies and Evaluation

  • Extending Binary SVM to Multiclass Problems:

    • Standard SVM formulations are binary classifiers (y∈{−1,+1}y \in \{-1, +1\}).

    • Real-world problems often involve multiple mutually exclusive classes (e.g., Healthy, Bearing Fault, Rotor Imbalance).

    • Multiclass problems are solved by decomposing the multi-label space into multiple binary subproblems.

  • One-versus-Rest (OvR / One-vs-All):

    • For KK classes, the algorithm trains KK distinct binary classifiers.

    • Each classifier kk is trained to distinguish class kk (coded as +1+1) from all remaining K−1K-1 classes combined (coded as −1-1).

    • At inference, all KK classifiers produce a continuous signed decision score: fk(x)=wkTx+bkf_k(\mathbf{x}) = \mathbf{w}_k^T \mathbf{x} + b_k.

    • The predicted class corresponds to the classifier yielding the highest decision score:     y^=arg⁡max⁡k∈{1,…,K}fk(x)\hat{y} = \arg\max_{k \in \{1, \dots, K\}} f_k(\mathbf{x})

    • Total Model Fits: Exactly KK models.

  • One-versus-One (OvO):

    • Trains a dedicated binary classifier for every unique pairwise combination of classes.

    • Total Classifiers Required:     Total Fits=K(K−1)2\text{Total Fits} = \frac{K(K - 1)}{2}

    • Example: For K=4K = 4 classes, the framework fits 4×32=6\frac{4 \times 3}{2} = 6 binary models.

    • At inference, each classifier casts a vote for its preferred class; the final prediction is determined by majority voting across all pairwise comparisons.

  • Worked Example: Three-Class OvR Decision Logic:

    • Consider three mutually exclusive states: Normal, Bearing Fault, and Imbalance (K=3K = 3).

    • A new input vector xnew\mathbf{x}_{new} is evaluated across the three fitted binary models, yielding:

    • Normal vs. Rest Score: −0.6-0.6

    • Bearing Fault vs. Rest Score: +1.2+1.2

    • Imbalance vs. Rest Score: +0.2+0.2

    • Outcome: The Bearing Fault classifier produces the largest positive decision score (+1.2+1.2), so the sample is classified as a Bearing Fault.

    • Note: These values represent signed margins, not probabilities, and are not constrained to sum to 11.

  • Multiclass Performance Metrics: Macro F1 vs. Weighted F1:

    • Per-Class Metrics: Each class kk is treated as a positive target while all other classes are treated as negative. Binary True Positives, False Positives, and False Negatives are computed to calculate class-specific Precision, Recall, and F1.

    • Support (sks_k): The number of ground-truth occurrences belonging to class kk.

    • Macro F1: The unweighted arithmetic average of per-class F1 scores, treating all classes with equal importance:     Macro F1=1K∑k=1KF1k\text{Macro F1} = \frac{1}{K} \sum_{k=1}^K \text{F1}_k

    • Weighted F1: The average of per-class F1 scores weighted by the support of each class:     Weighted F1=1∑k=1Ksk∑k=1Ksk×F1k\text{Weighted F1} = \frac{1}{\sum_{k=1}^K s_k} \sum_{k=1}^K s_k \times \text{F1}_k

  • Worked Example: Macro F1 vs. Weighted F1 on Imbalanced Diagnoses:

    • Analysis of 1,0001{,}000 industrial machine components across three diagnostic categories:

    • Normal: Support s1=900s_1 = 900, F11=0.95\text{F1}_1 = 0.95

    • Bearing Fault: Support s2=80s_2 = 80, F12=0.80\text{F1}_2 = 0.80

    • Rotor Imbalance: Support s3=20s_3 = 20, F13=0.40\text{F1}_3 = 0.40

    • Macro F1 Calculation:     Macro F1=0.95+0.80+0.403=2.153≈0.717(71.7%\text{Macro F1} = \frac{0.95 + 0.80 + 0.40}{3} = \frac{2.15}{3} \approx 0.717 \quad (71.7\%

    • Weighted F1 Calculation:     Weighted F1=(900×0.95)+(80×0.80)+(20×0.40)1,000\text{Weighted F1} = \frac{(900 \times 0.95) + (80 \times 0.80) + (20 \times 0.40)}{1{,}000}     Weighted F1=855+64+81,000=9271,000=0.927(92.7%\text{Weighted F1} = \frac{855 + 64 + 8}{1{,}000} = \frac{927}{1{,}000} = 0.927 \quad (92.7\%

    • Diagnostic Decision: The high Weighted F1 (92.7%92.7\%) masks poor detection performance on the critical Rotor Imbalance fault (F1=0.40\text{F1} = 0.40). When rare classes represent critical operational states, models should be chosen based on Macro F1 and class-specific recall.

  • Scikit-Learn Implementation Specifics for Multiclass SVM:

    • SVC: Uses an OvO framework internally during multi-target fitting. The parameter decision_function_shape="ovr" aggregates the pairwise decision outputs into an OvR-like output shape, but does not alter the underlying OvO training process.

    • OneVsRestClassifier(SVC(...)): Wraps the binary estimator to explicitly train KK independent OvR models.

    • LinearSVC: Implements OvR natively using its underlying linear coordinate descent solver.