Support Vector Machines (Part 1) and Classification Evaluation
Supervised Learning Foundations: Regression vs. Classification
Supervised Learning Paradigms:
Supervised learning algorithms learn a mapping from input features to a target output using labeled training data.
The core distinction between tasks depends directly on the mathematical nature of the target variable.
Regression Mechanics:
Target Type: Continuous numeric quantity.
Core Objective: Predicts a continuous value along a real number continuum.
Evaluation Metrics: Typical error metrics quantify the geometric or numerical distance between predicted and observed numbers (e.g., Mean Squared Error, Mean Absolute Error).
Representative Engineering Questions: "How much energy will the machine consume?"
Domain Applications: Predicting operating temperature, remaining useful life (RUL) of rotating machinery, continuous power grid demand, and material tensile strength.

Classification Mechanics:
Target Type: Discrete categorical class labels.
Core Objective: Assigns an observation to one of two or more distinct categories based on its coordinates in feature space.
Class Cardinality:
Binary Classification: The target variable is restricted to exactly two classes (e.g., or ).
Multiclass Classification: The target variable comprises three or more mutually exclusive classes.
Evaluation Metrics: Focuses on correct class assignments, confusion matrices, and the asymmetric costs associated with distinct types of misclassifications.
Representative Engineering Questions: "Is this vibration window normal or faulty?"

Feature Space Representation:
Observations / Samples: Represented as rows in a design matrix (e.g., individual sensor time-windows, physical specimens, or manufactured components).
Features / Dimensions: Represented as columns that define the orthogonal axes of a feature space (e.g., physical dimensions, statistical kurtosis, voltage readings).
Geometric Partitioning: In a 2D feature space, a class boundary is represented as a line; in a 3D feature space, it forms a planar surface; in -dimensional space, it forms a -dimensional hyperplane.

Support Vector Machines: Hyperplanes, Margins, and Support Vectors
Support Vector Machine (SVM) Classification Principles:
SVM is a family of supervised algorithms that identifies an optimal separating hyperplane between classes in a vector space.
A new, unobserved point is assigned a class label depending on which side of the decision boundary it falls.
Mathematical Definition of a Hyperplane:
In a -dimensional feature space , a linear hyperplane is defined by the set of points satisfying:
denotes the weight vector that is strictly normal (orthogonal) to the hyperplane.
represents the bias term (intercept), which determines the offset of the hyperplane from the origin.
The decision rule for an arbitrary input is given by the sign function:

The Maximum-Margin Principle:
While infinitely many valid separating hyperplanes can divide linearly separable classes, SVM seeks the unique hyperplane that maximizes the geometric margin.
Margin Definition: The perpendicular distance between the decision boundary () and the closest data points from either class.
Boundary Placement: The decision boundary is oriented such that it remains equidistant from the closest observations of both classes, maximizing the separation gap.

Support Vectors and Sparsity:
Support Vectors: The critical subset of training points that lie directly on the margins (or violate them in soft-margin formulations).
Model Sparsity: The mathematical parameters and of the decision boundary are determined entirely by the support vectors.
Invariance to Non-Support Vectors: Observations lying strictly outside the margin can be translated, perturbed, or removed entirely without altering the position or orientation of the optimal hyperplane.
Robustness: A wider margin provides higher geometric tolerance, ensuring that small random measurement shifts or sensor noise do not cause misclassifications.

Mathematical Optimization: Hard-Margin vs. Soft-Margin SVM
Hard-Margin SVM Formulation:
Assumes training data are strictly linearly separable without errors or class overlap.
Given training samples with labels , the canonical margin boundaries are defined as:
The geometric width of the total margin between the two parallel hyperplanes equals:
Maximizing is mathematically equivalent to minimizing .
Constrained Optimization Problem:
Failure Modes of Hard-Margin Optimization:
Real-world engineering measurements contain high-frequency noise, overlapping class distributions, and sensor artifacts.
A single outlier can cause the hard-margin optimization problem to have no feasible solution (infeasibility).
Even if separable, hard margins lead to extreme sensitivity and severe overfitting to borderline observations.

Soft-Margin SVM Formulation and Slack Variables:
Introduced to accommodate non-separable data and control sensitivity to noise by permitting controlled margin violations.
Slack Variables (): A set of non-negative slack variables is introduced for each sample :
: Point lies on or outside the correct margin boundary.
: Point lies within the margin band but on the correct side of the decision boundary.
: Point crosses the decision boundary and is misclassified.
Soft-Margin Objective Function:
Structural Role of Objective Terms:
acts as a regularization term that favors a wide margin.
represents an upper bound on the total number and magnitude of margin errors.

The Regularization Hyperparameter :
governs the exact trade-off between margin maximization and training error penalty.
Small Value:
Places low penalty on margin violations ().
Results in a wider margin band that tolerates more misclassifications and intrusions.
Higher regularization; prevents overfitting on noisy training samples.
Large Value:
Imposes severe penalties on any violation ().
Forces the optimizer to narrow the margin in order to classify as many training instances correctly as possible.
Lower regularization; risks overfitting to training noise and outliers.
Selection Criterion: cannot be inferred directly from training data; it must be tuned using held-out validation data or cross-validation. When two values of yield indistinguishable validation performance, the smaller is preferred for greater stability and simpler boundaries.

Nonlinear Decision Boundaries and the Kernel Trick
Limitations of Linear Separators:
When feature relationships exhibit nonlinear interactions (e.g., concentric rings, parabolic structures, radial groupings), no linear hyperplane can achieve acceptable separation.


The Kernel Trick Concept:
Nonlinearly separable data in an input space can become linearly separable when mapped into a higher-dimensional feature space via a transformation mapping .
The Kernel Function (): Evaluates the inner product of vectors in the transformed space directly from the original coordinates, without explicitly computing or storing the high-dimensional coordinates:
This enables the algorithm to fit a linear maximum-margin hyperplane in , which corresponds to a smooth, curved boundary in the original input space.

Radial Basis Function (RBF) Kernel:
Also known as the Gaussian kernel, defined mathematically as:
Compares samples based on their squared Euclidean distance in input space.
Maps points into an infinite-dimensional Hilbert space.
The hyperparameter (gamma) defines the influence radius of individual support vectors:
Small : Broad Gaussian curvature; wide influence radius; smoother decision boundaries.
Large : Tight Gaussian peaks around support vectors; localized influence; risk of overfitting.

Polynomial Kernel and Geometric Alignment:
Defined mathematically as: where denotes the polynomial degree and is an independent coefficient.
Degree Comparison:
A degree-2 polynomial kernel () aligns with quadratic circular and elliptical geometries, capturing concentric circles cleanly.
Increasing to degree-3 () incorporates cubic interactions, providing excess flexibility that introduces unnatural boundary distortions into simple circular geometries.

Feature Scaling, Pipelines, and Implementation
Critical Role of Feature Scaling:
Support Vector Machine optimization relies on Euclidean distance calculations: .
If features have different scales, features with large absolute numerical values (e.g., millivolts spanning to ) will dominate the objective function over normalized features with smaller numerical scales (e.g., kurtosis spanning to ).
Contrast with Tree-Based Models: Unlike decision trees or random forests, which are scale-invariant because they split on single features monotonically, SVM algorithms require features to be scaled consistently.
StandardScaler: The standard preprocessing step for continuous features transforms each feature column to zero mean () and unit variance ():
Data Leakage in Preprocessing:
Scaling parameters ( and ) learn structural information from data distributions.
Incorrect Workflow (Data Leakage):
Scaling entire dataset prior to splitting.
Splitting scaled data into cross-validation folds.
Evaluating performance.
Consequence: Validation statistics leak into the scaler, producing overly optimistic validation results.
Correct Workflow (Leakage-Safe):
Split raw data into training and validation folds.
Fit scaler strictly on the training fold.
Apply the learned transformation parameters to both the training fold and the validation fold.
Fit and score the SVM estimator.
Linear SVM Pipeline Implementation in Scikit-Learn:
The
Pipelineconstruct binds the transformation and estimation steps into a single unified object, ensuring preprocessing is fitted solely on training partitions during each validation pass:
from sklearn.svm import LinearSVC
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import cross_val_score
pipe = Pipeline([
("scale", StandardScaler()),
("svm", LinearSVC(C=1.0, max_iter=5000, random_state=42)),
])
scores = cross_val_score(pipe, X, y, cv=5, scoring="f1")
print(f"CV F1: {scores.mean():.3f} +/- {scores.std():.3f}")
pipe.fit(X_train, y_train)
Implementation Differences: LinearSVC vs. SVC(kernel="linear"):
LinearSVC:
Implemented on the underlying
liblinearC/C++ library.Highly optimized for linear classification; scales linearly with the number of samples (roughly ).
Solves the primal problem; does not compute or expose explicit support vectors via
support_vectors_.Uses One-versus-Rest (OvR) multiclass formulation by default.
SVC:
Implemented on the underlying
libsvmlibrary.Employs quadratic programming algorithms (Sequential Minimal Optimization / SMO).
Stores and exposes support vectors via
support_vectors_and counts per class vian_support_.Uses One-versus-One (OvO) multiclass strategy internally.
Computational scaling is between and , making training impractical beyond approximately observations.
Algorithmic Trade-Offs of SVM:
Strengths:
Highly effective in high-dimensional feature spaces where the number of features exceeds the number of samples.
Memory efficient because prediction depends only on the learned support vectors.
High flexibility in boundary shape via user-selected kernel functions.
Weaknesses:
Poor computational scaling ( to ) for large datasets ().
Strict requirement for feature scaling.
Lack of native probabilistic outputs (requires expensive post-hoc Platt scaling calibration).
Hyperparameter sensitivity requiring systematic tuning over and .
Lower model interpretability compared to simple linear or decision tree models.
Real-World Engineering and Scientific Applications:
Text and Document Classification: High-dimensional, sparse TF-IDF vectors where linear boundaries excel.
Bioinformatics: Gene expression microarray profiles and protein sequence classification characterized by few samples () and tens of thousands of features ().
Classical Computer Vision: Handwritten digit recognition (e.g., MNIST baselines) and facial recognition feature classification.
Fault Detection and Anomaly Monitoring: One-Class SVM trained on normal operational vibrations to detect impending mechanical anomalies.
Manufacturing Quality Control: Classifying in-specification vs. out-of-specification fabricated components using multi-axis sensor data.
Financial Credit Scoring and Bankruptcy: Finding robust separation hyperplanes across moderately dimensional corporate balance sheet metrics.
Classification Metrics for Imbalanced Engineering Problems
The Confusion Matrix:
A contingency table categorizing actual class status against model predictions.
Predicted Positive | Predicted Negative | |
|---|---|---|
Actual Positive | True Positive (TP) | False Negative (FN) |
Actual Negative | False Positive (FP) | True Negative (TN) |
The Accuracy Paradox in Rare Event Detection:
Accuracy measures the ratio of all correct predictions to total cases:
Failure Scenario: Consider an industrial inspection pipeline inspecting fabricated components where exactly are defective ( defect rate) and are normal ().
A trivial, non-operational model predicting every unit as normal yields:
Defects Identified: out of .
Defect Recall: .
Conclusion: Overall accuracy masks catastrophic failure in imbalanced datasets. Models must be evaluated using class-specific metrics.
Precision: Reliability of Positive Alarms:
Fraction of predicted positive labels that are actually positive:
Evaluated across the predicted positive column.
Application: Critical when false alarms are costly, disruptive, or cause alarm fatigue (e.g., dispatching an offshore maintenance crew for a flagged fault; spam filters hiding legitimate operational emails; emergency shutdowns of chemical plants).
Recall (Sensitivity): Coverage of True Positive Events:
Fraction of actual positive occurrences captured by the model:
Evaluated across the actual positive row.
Application: Critical when missing a true incident leads to catastrophic damage (e.g., cancer diagnostics; thermal runaway detection in battery packs; structural crack detection; industrial fire detection).
Precision vs. Recall Engineering Prioritization:
Operating Scenario | Most Costly Error Type | Metric to Maximize |
|---|---|---|
Emergency shutdown alarm | False Positive: Unnecessary production halt | Precision |
Cancer screening | False Negative: Missed malignant pathology | Recall |
Email spam filter | False Positive: Important business email lost | Precision |
Industrial fire detection | False Negative: Uncontrolled fire hazard | Recall |
F1 Score: Harmonic Balance:
The harmonic mean of precision and recall:
Penalizes extreme imbalances between precision and recall; F1 is high only when both individual metrics are strong.
Model Comparison Benchmark:
Candidate Model | Precision | Recall | Resulting F1 Score |
|---|---|---|---|
Model A | |||
Model B | |||
Model C |
Observation: Model B provides the best balance between alarm reliability and fault coverage (), despite Model A having higher precision and Model C having higher recall.
Threshold Manipulation and the ROC Curve:
Varying the decision threshold changes the trade-off: lowering the classification threshold increases the volume of positive predictions, raising recall while typically lowering precision.
Receiver Operating Characteristic (ROC): Plots True Positive Rate (Recall) along the y-axis against False Positive Rate () along the x-axis across all continuous decision thresholds.
Area Under the Curve (AUC):
: Perfect diagnostic ranking (every positive case scores higher than every negative case).
: Random guessing performance.
Caveat: ROC AUC summarizes ranking over all operating thresholds, many of which may be irrelevant in deployment. For highly imbalanced datasets, Precision-Recall curves and fixed-threshold confusion matrices provide a more realistic assessment.
Worked Example: Equipment Fault Detection Performance:
An equipment fault classifier yields the following contingency matrix over a production run:
Calculations:
Diagnostic Decision: While an precision and recall profile provides an F1 score, engineering domain requirements determine whether missing of impending faults () is acceptable when failure causes unexpected downtime.
Cross-Validation and Model Selection Workflows
The Fundamental Partitioning Paradigm:
Training accuracy is systematically optimistic because model parameters are directly optimized to fit training features.
Data must be separated into three distinct roles:
Train Set: Used exclusively to optimize model weights ().
Validation Set: Used to evaluate hyperparameters, tune thresholds, and select candidate algorithms.
Test Set: Held strictly untouched until model development is complete; provides a final, unbiased estimate of generalization performance.
Instability of a Single Train/Validation Split:
Evaluating a candidate model on a single arbitrary train/validation split creates high variance:
Split A: Validation accuracy ("Looks excellent")
Split B: Validation accuracy ("Seems acceptable")
Split C: Validation accuracy ("Looks weak")
Changes in performance stem entirely from random partitioning rather than differences in the underlying model.
The -Fold Cross-Validation Framework:
Procedure:
Split: Divide the development dataset into equal-sized partitions (folds).
Fit: Train the model on folds.
Score: Evaluate performance metrics on the single held-out validation fold.
Repeat: Iterate the process times so that every fold serves as the validation set exactly once.
Aggregate: Compute the arithmetic mean and sample variation across all validation scores.
Iteration Run | Fold 1 | Fold 2 | Fold 3 | Fold 4 | Fold 5 |
|---|---|---|---|---|---|
Run 1 | VALIDATE | Train | Train | Train | Train |
Run 2 | Train | VALIDATE | Train | Train | Train |
Run 3 | Train | Train | VALIDATE | Train | Train |
Run 4 | Train | Train | Train | VALIDATE | Train |
Run 5 | Train | Train | Train | Train | VALIDATE |
Comparison: Single Split vs. -Fold Cross-Validation:
Evaluation Question | Single Train/Validation Split | -Fold Cross-Validation |
|---|---|---|
Volume of data validated | Exactly one subset | Every sample validated once |
Sensitivity to partition split | High sensitivity (high variance) | Lower sensitivity (smoothed variance) |
Computational overhead | Low ( model fit) | Higher ( independent model fits) |
Primary operational domain | Massive datasets where fitting is expensive | Standard development and course projects |
Selection Criteria for Number of Folds ():
: Good default balance; trains on of data per fold; computationally faster.
: Standard industry benchmark; trains on of data per fold; smaller bias in performance estimation at the expense of higher computation.
Leave-One-Out Cross-Validation (LOOCV): Special case where ; each validation fold contains exactly one sample. High computational cost and can exhibit high variance because training sets across folds are nearly identical.
Worked Example: Partitioning 1,000 Industrial Samples:
Total labeled observations: (including a fault distribution).
Final test holdout: Reserve observations untouched ().
Development set: Remaining observations ().
For folds: observations per fold.
Stratified fold composition: Maintain the normal ( samples) and faulty ( samples) ratio in each fold.
Each CV pass: Trains on samples ( folds) and evaluates on samples ( fold).
Interpreting Cross-Validation Scores and Variance:
Fold accuracies: , , , , .
Cross-Validation Mean:
Range / Spread: Span of to ( percentage points).
Interpretation: An mean reflects expected baseline generalization; the spread highlights sensitivity to input variations.
Handling Correlated and Time-Series Data Structures:
Grouped Data (
GroupKFold): When observations originate from shared entities (e.g., repeated runs on the same machine, recordings from the same patient, or measurements from specific production lines), random splitting causes data leakage.GroupKFoldensures all rows from a given group are placed entirely into the training partition or the validation partition, never split between both.Temporal / Time-Series Data (
TimeSeriesSplit): In chronological processes, future data must never be used to predict the past. Forward-chaining splits evaluate models using expanding historical windows:Split 1: Train on January; Validate on February.
Split 2: Train on January–February; Validate on March.
Split 3: Train on January–March; Validate on April.
Split 4: Train on January–April; Validate on May.
Split 5: Train on January–May; Validate on June.
Candidate Model Selection Under Variance Constraints:
Candidate Model | Individual Fold Scores (\%) | Cross-Validation Mean | Performance Trade-Off |
|---|---|---|---|
Model A | Low variance (), highly stable | ||
Model B | Highest mean performance, moderate variance | ||
Model C | High variance (), unstable |
Decision Rule: Model B achieves the highest mean accuracy (). If the operational deployment penalizes worst-case drops, Model A is an attractive alternative due to its tight performance bound ( minimum vs. Model B's and Model C's ).
Hyperparameter Optimization with GridSearchCV:
Hyperparameters (, , kernel type) cannot be learned directly via gradient descent or analytical quadratic programming.
GridSearchCVevaluates all combinations across a user-specified parameter grid using identical cross-validation folds.Standard Execution Steps:
Wrap preprocessing (e.g.,
StandardScaler) and estimator into an integratedPipeline.Define hyperparameter search spaces using grid dictionaries.
Fit
GridSearchCVusing the development set, optimizing for a targeted metric (e.g.,scoring="f1").Select the best hyperparameter configuration based on the highest mean validation score.
Refit the selected pipeline configuration on the entire development dataset ( of non-test data).
Run an evaluation once against the untouched final test set.
Leakage-Safe Model-Selection Protocol:
Reserve Final Test Set: Partition raw data and hold the test set out immediately.
Choose Metric: Base selection on operational costs (e.g., Precision, Recall, F1).
Build Pipeline: Integrate transformers and estimators into a unified sequence.
Run Cross-Validation: Execute -fold CV or GridSearchCV within the development partition.
Select Candidate: Choose the optimal model architecture and hyperparameter settings.
Refit and Final Test: Refit the chosen configuration across all development data and evaluate once on the test holdout.
Multiclass Classification Strategies and Evaluation
Extending Binary SVM to Multiclass Problems:
Standard SVM formulations are binary classifiers ().
Real-world problems often involve multiple mutually exclusive classes (e.g., Healthy, Bearing Fault, Rotor Imbalance).
Multiclass problems are solved by decomposing the multi-label space into multiple binary subproblems.
One-versus-Rest (OvR / One-vs-All):
For classes, the algorithm trains distinct binary classifiers.
Each classifier is trained to distinguish class (coded as ) from all remaining classes combined (coded as ).
At inference, all classifiers produce a continuous signed decision score: .
The predicted class corresponds to the classifier yielding the highest decision score:
Total Model Fits: Exactly models.
One-versus-One (OvO):
Trains a dedicated binary classifier for every unique pairwise combination of classes.
Total Classifiers Required:
Example: For classes, the framework fits binary models.
At inference, each classifier casts a vote for its preferred class; the final prediction is determined by majority voting across all pairwise comparisons.
Worked Example: Three-Class OvR Decision Logic:
Consider three mutually exclusive states: Normal, Bearing Fault, and Imbalance ().
A new input vector is evaluated across the three fitted binary models, yielding:
Normal vs. Rest Score:
Bearing Fault vs. Rest Score:
Imbalance vs. Rest Score:
Outcome: The Bearing Fault classifier produces the largest positive decision score (), so the sample is classified as a Bearing Fault.
Note: These values represent signed margins, not probabilities, and are not constrained to sum to .
Multiclass Performance Metrics: Macro F1 vs. Weighted F1:
Per-Class Metrics: Each class is treated as a positive target while all other classes are treated as negative. Binary True Positives, False Positives, and False Negatives are computed to calculate class-specific Precision, Recall, and F1.
Support (): The number of ground-truth occurrences belonging to class .
Macro F1: The unweighted arithmetic average of per-class F1 scores, treating all classes with equal importance:
Weighted F1: The average of per-class F1 scores weighted by the support of each class:
Worked Example: Macro F1 vs. Weighted F1 on Imbalanced Diagnoses:
Analysis of industrial machine components across three diagnostic categories:
Normal: Support ,
Bearing Fault: Support ,
Rotor Imbalance: Support ,
Macro F1 Calculation:
Weighted F1 Calculation:
Diagnostic Decision: The high Weighted F1 () masks poor detection performance on the critical Rotor Imbalance fault (). When rare classes represent critical operational states, models should be chosen based on Macro F1 and class-specific recall.
Scikit-Learn Implementation Specifics for Multiclass SVM:
SVC: Uses an OvO framework internally during multi-target fitting. The parameterdecision_function_shape="ovr"aggregates the pairwise decision outputs into an OvR-like output shape, but does not alter the underlying OvO training process.OneVsRestClassifier(SVC(...)): Wraps the binary estimator to explicitly train independent OvR models.LinearSVC: Implements OvR natively using its underlying linear coordinate descent solver.