AAI CH2 - Regression vs classification

Regression vs Classification

1. Overview

  • Categorization of Machine Learning Tasks

    • Regression: Predicting continuous values.

    • Classification: Predicting discrete labels.

2. Learning Types

2.1 Passive Learning

  • Involves acquiring information from an expert.

  • Memorization and application of given characteristics.

2.2 Active Learning

  • The expert provides partial information (only the label), and one deduces characteristics through observation.

  • More data-driven approach where features are identified by the learner.

3. Domain-Specific Classification in ML

  • Classification ≈ Regression: Both are forms of predictive analytics.

  • Key Differences:

    • Classification predicts categorical labels (e.g., spam vs. not spam).

    • Regression predicts continuous values (e.g., predicting customer spend).

4. Examples of Tasks

4.1 Classification Tasks

  • Tumor cell classification as benign or malignant.

  • Credit card transaction classification (legitimate or fraudulent).

  • Classifying news articles into categories (finance, weather, etc.).

4.2 Regression Tasks

  • Predicting stock market prices.

  • Estimating viewer age on YouTube.

  • Forecasting temperature based on various factors.

5. Steps in Classification and Regression

  • Data Preparation: Cleaning and organizing data.

  • Model Construction: Building the predictive model.

    • Usually builds on training data with labeled outputs.

  • Model Usage and Evaluation:

    • Assessing the model’s performance on unseen data.

6. Classifier Characteristics

6.1 Supervised Classification

  • Training set includes labeled data.

  • Objective is to create an accurate model to predict labels of unseen data.

6.2 Two-Step Process in Classification

  1. Model Construction: Define a set of classes and attributes.

  2. Model Usage: Classify new data and evaluate model accuracy.

7. Classification Types

7.1 Binary-Class Classification

  • Only two classes (e.g., Salmon vs. Not Salmon).

7.2 Multi-Class Classification

  • More than two classes (e.g., Salmon, Tuna, Shark).

8. Regression Process

  • Similar to classification, but applicable for continuous outcomes.

  • Focus on predicting values like loan amounts instead of class labels.

9. Distance-Based Classifiers (k-NN)

  • Instance-Based Learning: Stores training examples; classifies based on proximity.

  • Distance Metrics: Typically uses Euclidean distance.

  • Majority voting among k-nearest neighbors.

10. Challenges of k-NN

  • High computational costs in large datasets.

  • Needs all training data stored, which can be memory-intensive.

  • Performance may degrade with higher dimensions (curse of dimensionality).

11. Model Evaluation Metrics

  • Accuracy: Proportion of correctly classified instances.

  • Misclassification Rate: Proportion of incorrect classifications.

  • Confusion Matrix: Provides breakdown of true positives, negatives, and false classifications.

  • Precision and Recall: Measures focusing on positive class performance.

12. Techniques for Performance Estimation

  • Holdout Method: Split data into training and testing sets.

  • Cross-Validation: More robust approach with k-fold validation.

  • Leave-One-Out: Each sample is used for testing once, which is effective for small datasets.

13. Cost Function and Optimization

  • In regression, minimizing error between predicted and actual values.

  • Typically using techniques like gradient descent to optimize model parameters.

14. Feature Scaling in Gradient Descent

  • Importance: Ensures features on similar scale to improve optimization efficiency.

  • Techniques include normalization and standardization.

15. Bias-Variance Tradeoff

  • Bias: Error due to overly simplistic assumptions in the learning algorithm.

  • Variance: Error due to excessive complexity in the learning model.

  • Tradeoff: Balancing between bias and variance is crucial for minimizing overall prediction error.

16. Ensemble Methods in Learning

  • Bagging: Aggregate predictions from multiple independent classifiers.

  • Boosting: Sequentially training weak learners, focusing on previously misclassified instances.

17. Key Algorithms in Machine Learning

  • k-Nearest Neighbors: Instance-based learning algorithm.

  • Linear Regression: Predicts continuous outcomes based on linear relationships.

  • Logistic Regression: For binary classification tasks.

  • Decision Trees: Tree-structured model for classification tasks.