Supervised Learning in Machine Learning (M1.3)

Classification and Regression

Overview

  • Instructor: Paniz Abedin, Assistant Professor of Computer Science, Florida Polytechnic University

What Is Supervised Learning?

  • Definition: Supervised learning involves using labeled data to train models.

  • Input–output pairs: Represented as (x,y)(x, y), where:

    • x: Input features.

    • y: Desired output (label).

  • Goal: Predict the correct output for unseen data.

  • Examples:

    • Image + label: Associating an image (input) with its label (output).

    • Features + price: Using feature data such as size and location to predict the price of an item or property.

Classification vs Regression

Classification
  • Output: Discrete labels (e.g. class categories).

  • Examples:

    • Classifying emails into 'spam/not spam'.

    • Classifying handwritten digits.

Regression
  • Output: Continuous values.

  • Examples:

    • Predicting house prices.

    • Forecasting temperature.

Key Terminology

Data and Model Basics
  • Sample (Input):

    • Definition: A single data point input into the model.

    • Examples: An image, a text review, or a record of a house.

  • Prediction (Output):

    • Definition: The result produced by the model.

    • Examples: A class label or a numerical value.

Target and Ground Truth
  • Target:

    • Definition: The correct answer or output the model should predict.

  • Ground Truth / Annotations:

    • Definition: The complete set of true targets in the dataset, often generated by human labeling.

Types of Classification

Binary Classification
  • Definition: Involves two exclusive classes.

  • Examples:

    • Spam vs Not Spam.

    • Disease vs No Disease.

Multiclass Classification
  • Definition: Involves more than two possible classes, with each sample belonging to exactly one class.

  • Example:

    • Handwritten digits ranging from 0 to 9.

Multilabel Classification
  • Definition: Each sample can possess multiple labels, with the number of labels varying by sample.

  • Example:

    • An image that contains both a 'cat' and a 'dog'.

Classification Overview

Types of Classification Samples
  • Binary Classification: One label, two classes.

    • Example scenarios: Labeling as either 'Cat' or 'Dog'.

  • Multiclass Classification: One label, multiple unique classes.

  • Multilabel Classification: Multiple labels across multiple classes.

Regression Overview

Regression Basics
  • Definition: Regression techniques predict continuous values as outputs.

  • Output Types:

    • A real number or vector.

  • Examples:

    • Price prediction.

    • Temperature forecasting.

Types of Regression
Scalar Regression
  • Definition: The output is a single continuous value.

  • Examples:

    • House price.

    • Student GPA.

Vector Regression
  • Definition: The output consists of multiple continuous values, represented as a vector.

  • Examples:

    • Bounding box coordinates in computer vision.

    • 3D position predictions for objects.

Neural Networks for Classification

Network Structure
  • Layers:

    • Input layer → hidden layers → output layer.

  • Output Layer: The size of the output layer equals the number of classes in the classification task.

  • Activation Function: Utilizes softmax activation to convert outputs into probabilistic scores.

Softmax Activation Function

  • Formula:
    softmax(z<em>i)=ez</em>i∑<em>j=1Jez</em>j\text{softmax}(z<em>i) = \frac{e^{z</em>i}}{\sum<em>{j=1}^{J} e^{z</em>j}}

  • Functionality: Converts raw model scores into probabilities.

    • Property: The probabilities generated across classes sum to 1.

Classification Loss Functions

Key Loss Functions
  • Binary Cross-Entropy: Used for models with 2 classes.

  • Categorical Cross-Entropy: Used for models with multiple classes.

  • Formulas:

    • Binary Cross-Entropy Loss:
      Lbinary=−[ylog⁡(y^)+(1−y)log⁡(1−y^)]L_{binary} = - [y \log(\hat{y}) + (1 - y) \log(1 - \hat{y})]

    • Categorical Cross-Entropy Loss:
      L<em>categorical=−∑</em>i=1CY<em>ilog⁡(y^</em>i)L<em>{categorical} = - \sum</em>{i=1}^{C} Y<em>i \log(\hat{y}</em>i)

Example: Digit Classification

Model Architecture
  • Input: 28x28 pixel images of handwritten digits.

  • Output: 10 probabilities corresponding to digits 0–9.

  • Layer Configuration:

    • Dense Layer 1: 128 units, activation function = "relu".

    • Dense Layer 2: 10 units, activation function = "softmax".

Relu Activation Function

  • Function: A rectified linear unit (ReLu) function intended to eliminate negative values.

  • Graph Representation: The function outputs 0 for negative inputs, and for positive inputs, it outputs the same value.

Neural Networks for Regression

Model Structure
  • Output: A single real number predicted by the model.

  • Output Layer: No activation functions applied, allowing for linear outputs.

  • Common Loss Function: Mean Squared Error (MSE).

Example: House Price Prediction

Model Configuration
  • Inputs: Features such as size, location, number of rooms.

  • Output: Predicted house price.

  • Layer Structure:

    • Dense Layer with 64 units, activation = "relu".

    • Final Dense Layer with 1 unit for output.

Evaluation Metrics

Classification Metrics
  • Accuracy: The percentage of correctly predicted samples.

  • Precision and Recall: Metrics evaluating the quality of positive predictions.

Regression Metrics
  • MSE (Mean Squared Error): Measures the average squared difference between predicted and actual values.

  • MAE (Mean Absolute Error): Evaluates the average absolute differences between predictions and actual outcomes.

Decision Boundaries

  • Definition: The regions in the feature space separating different classes, often learned by neural networks.

  • Characteristics: Decision boundaries can represent non-linear separations.

Complexity of your Model

Layer Interaction and Information Passing
  • Each layer in a Dense network accesses only the outputs from the previous layer.

  • If a layer loses crucial information, it cannot be recovered in subsequent layers; this can lead to information bottlenecks.

Example: Dataset Complexity
  • Reuters Dataset: Consists of 46 classes in a 16-dimensional space.

    • This space may not be sufficient for effective classification of 46 distinct classes, risking information loss during processing.

Modeling Considerations

Input Feature Dimensions
  • Let d be the number of input features.

    • Layer Operation: First Dense layer computes:
      h=f(Wx+b)h = f(Wx + b)

  • If the number of neurons k in the layer is:

    • k < d: Compression of information occurs.

    • k > d: Expansion of the representation space takes place.

Hidden Layer Sizes
  • Hidden layers should not be designated arbitrarily; they define representation capacity.

  • Factors influencing size decisions include:

    • Number of input features.

    • Number of output classes.

    • Complexity of the mapping function.

Constructing Model Size Strategically
  • Example: Starting with 10000 input features:

    • Bad: A drastic size reduction to a small number like 16 can lead to significant information loss.

    • Good: Gradual compression such as 10000 → 512 → 256 → 128 supports effective representation without abrupt information loss.

Overfitting and Validation

Training and Validation Loss
  • Monitoring loss during training across epochs helps identify overfitting.

    • Graph: Training vs. validation loss with epochs indicated; ideally, they should converge.

Strategies to Combat Overfitting
  • Utilize a training + validation data split.

  • Monitor validation loss throughout training phases.

  • Implement early stopping techniques if validation loss begins to rise post-plateau.

K-Fold Cross-Validation

Cross-Validation Technique
  • Involves partitioning available data into K subsets;

    • Typically K = 4 or 5.

  • Each partition is used for validation while training the model on the remaining K - 1 partitions.

  • Final model validation score is averaged across all K iterations, aiding reliability in validation estimates.

  • Visualization: Diagram representing 3 folds with distinct training and validation allocations.

Summary of Learning Insights

  • Preprocessing: Raw data typically requires preprocessing before training in neural networks.

  • Feature Scaling: Requires independent scaling of features, especially when they have different ranges.

  • Overfitting: Neural networks are prone to overfitting; measures to mitigate this include model size reduction.

  • K-Fold Validation: Employ this method to ensure reliable evaluation especially in data-scarce scenarios.