Supervised Learning in Machine Learning (M1.3)
Classification and Regression
Overview
Instructor: Paniz Abedin, Assistant Professor of Computer Science, Florida Polytechnic University
What Is Supervised Learning?
Definition: Supervised learning involves using labeled data to train models.
Input–output pairs: Represented as , where:
x: Input features.
y: Desired output (label).
Goal: Predict the correct output for unseen data.
Examples:
Image + label: Associating an image (input) with its label (output).
Features + price: Using feature data such as size and location to predict the price of an item or property.
Classification vs Regression
Classification
Output: Discrete labels (e.g. class categories).
Examples:
Classifying emails into 'spam/not spam'.
Classifying handwritten digits.
Regression
Output: Continuous values.
Examples:
Predicting house prices.
Forecasting temperature.
Key Terminology
Data and Model Basics
Sample (Input):
Definition: A single data point input into the model.
Examples: An image, a text review, or a record of a house.
Prediction (Output):
Definition: The result produced by the model.
Examples: A class label or a numerical value.
Target and Ground Truth
Target:
Definition: The correct answer or output the model should predict.
Ground Truth / Annotations:
Definition: The complete set of true targets in the dataset, often generated by human labeling.
Types of Classification
Binary Classification
Definition: Involves two exclusive classes.
Examples:
Spam vs Not Spam.
Disease vs No Disease.
Multiclass Classification
Definition: Involves more than two possible classes, with each sample belonging to exactly one class.
Example:
Handwritten digits ranging from 0 to 9.
Multilabel Classification
Definition: Each sample can possess multiple labels, with the number of labels varying by sample.
Example:
An image that contains both a 'cat' and a 'dog'.
Classification Overview
Types of Classification Samples
Binary Classification: One label, two classes.
Example scenarios: Labeling as either 'Cat' or 'Dog'.
Multiclass Classification: One label, multiple unique classes.
Multilabel Classification: Multiple labels across multiple classes.
Regression Overview
Regression Basics
Definition: Regression techniques predict continuous values as outputs.
Output Types:
A real number or vector.
Examples:
Price prediction.
Temperature forecasting.
Types of Regression
Scalar Regression
Definition: The output is a single continuous value.
Examples:
House price.
Student GPA.
Vector Regression
Definition: The output consists of multiple continuous values, represented as a vector.
Examples:
Bounding box coordinates in computer vision.
3D position predictions for objects.
Neural Networks for Classification
Network Structure
Layers:
Input layer → hidden layers → output layer.
Output Layer: The size of the output layer equals the number of classes in the classification task.
Activation Function: Utilizes softmax activation to convert outputs into probabilistic scores.
Softmax Activation Function
Formula:
Functionality: Converts raw model scores into probabilities.
Property: The probabilities generated across classes sum to 1.
Classification Loss Functions
Key Loss Functions
Binary Cross-Entropy: Used for models with 2 classes.
Categorical Cross-Entropy: Used for models with multiple classes.
Formulas:
Binary Cross-Entropy Loss:
Categorical Cross-Entropy Loss:
Example: Digit Classification
Model Architecture
Input: 28x28 pixel images of handwritten digits.
Output: 10 probabilities corresponding to digits 0–9.
Layer Configuration:
Dense Layer 1: 128 units, activation function = "relu".
Dense Layer 2: 10 units, activation function = "softmax".
Relu Activation Function
Function: A rectified linear unit (ReLu) function intended to eliminate negative values.
Graph Representation: The function outputs 0 for negative inputs, and for positive inputs, it outputs the same value.
Neural Networks for Regression
Model Structure
Output: A single real number predicted by the model.
Output Layer: No activation functions applied, allowing for linear outputs.
Common Loss Function: Mean Squared Error (MSE).
Example: House Price Prediction
Model Configuration
Inputs: Features such as size, location, number of rooms.
Output: Predicted house price.
Layer Structure:
Dense Layer with 64 units, activation = "relu".
Final Dense Layer with 1 unit for output.
Evaluation Metrics
Classification Metrics
Accuracy: The percentage of correctly predicted samples.
Precision and Recall: Metrics evaluating the quality of positive predictions.
Regression Metrics
MSE (Mean Squared Error): Measures the average squared difference between predicted and actual values.
MAE (Mean Absolute Error): Evaluates the average absolute differences between predictions and actual outcomes.
Decision Boundaries
Definition: The regions in the feature space separating different classes, often learned by neural networks.
Characteristics: Decision boundaries can represent non-linear separations.
Complexity of your Model
Layer Interaction and Information Passing
Each layer in a Dense network accesses only the outputs from the previous layer.
If a layer loses crucial information, it cannot be recovered in subsequent layers; this can lead to information bottlenecks.
Example: Dataset Complexity
Reuters Dataset: Consists of 46 classes in a 16-dimensional space.
This space may not be sufficient for effective classification of 46 distinct classes, risking information loss during processing.
Modeling Considerations
Input Feature Dimensions
Let d be the number of input features.
Layer Operation: First Dense layer computes:
If the number of neurons k in the layer is:
k < d: Compression of information occurs.
k > d: Expansion of the representation space takes place.
Hidden Layer Sizes
Hidden layers should not be designated arbitrarily; they define representation capacity.
Factors influencing size decisions include:
Number of input features.
Number of output classes.
Complexity of the mapping function.
Constructing Model Size Strategically
Example: Starting with 10000 input features:
Bad: A drastic size reduction to a small number like 16 can lead to significant information loss.
Good: Gradual compression such as 10000 → 512 → 256 → 128 supports effective representation without abrupt information loss.
Overfitting and Validation
Training and Validation Loss
Monitoring loss during training across epochs helps identify overfitting.
Graph: Training vs. validation loss with epochs indicated; ideally, they should converge.
Strategies to Combat Overfitting
Utilize a training + validation data split.
Monitor validation loss throughout training phases.
Implement early stopping techniques if validation loss begins to rise post-plateau.
K-Fold Cross-Validation
Cross-Validation Technique
Involves partitioning available data into K subsets;
Typically K = 4 or 5.
Each partition is used for validation while training the model on the remaining K - 1 partitions.
Final model validation score is averaged across all K iterations, aiding reliability in validation estimates.
Visualization: Diagram representing 3 folds with distinct training and validation allocations.
Summary of Learning Insights
Preprocessing: Raw data typically requires preprocessing before training in neural networks.
Feature Scaling: Requires independent scaling of features, especially when they have different ranges.
Overfitting: Neural networks are prone to overfitting; measures to mitigate this include model size reduction.
K-Fold Validation: Employ this method to ensure reliable evaluation especially in data-scarce scenarios.