Unit2

Machine Learning Overview

Sources

  • Textbooks:

    • Introduction to Data Mining by Tan, Steinbach, Karpatne, Kumar

    • Grokking Artificial Intelligence Algorithms by Rishal Hurbans

    • Hands-on Machine Learning with Scikit-Learn, Keras, and TensorFlow by Aurélien Géron

  • Online Courses:

    • Machine Learning by Andrew Ng on Coursera

    • FA18 Machine Learning by Charles Isbell on edX

  • Various Other Sources

Supervised Learning

Classification

  • Definition: Task of assigning objects to one of several predefined categories (classes).

  • Goal: To find a model that can accurately map each attribute to a predefined class.

  • Applications:

    • Predicting stock prices = Regression

    • Approving a mortgage = Classification

    • Identifying the education level of a student from a picture = Classification

    • Determining the age of a student from a picture = Regression

Regression

  • Definition: Task of predicting continuous values rather than categories.

Key Terminology

Dataset

  • A collection of data objects (also called instances, records, points, cases, samples, entities).

Attributes

  • Properties or characteristics of data objects.

    • Also known as variables, fields, characteristics, dimensions, or features.

Target Concepts

  • A function to be learned from the training instances to map inputs to outputs.

    • Training Set: The set of instances used to determine the target concept (contains input-output pairs).

    • Inductive Learning: Providing numerous examples with labels to generalize from.

Candidate Concepts

  • Potential concepts being considered as the target concept.

Test Set

  • A separate set of instances used for evaluating the candidate concept to ensure unbiased results.

Classification Task

Defining Classification

  • Learn a model that maps each set of attributes (x) into class labels (y).

    • Instance: Characterized by a tuple (x, y), where x is the attribute set and y is the class label.

    • x: Attributes (predictors, independent variables), y: Class (response, dependent variable).

Examples of Classification Tasks

  • Categorizing Emails:

    • Features: Email headers, content → Class labels: Spam or Non-spam.

  • Identifying Tumor Cells:

    • Features: Medical images → Class labels: Malignant or Benign.

Classification Techniques

  • Decision Trees

  • Neural Networks

  • Ensemble Learning (Boosting, Bagging)

  • Support Vector Machines

  • Bayesian Learning

  • Rule-based Methods

  • Nearest-neighbor

General Approach for Building Classification Model

  1. Model Induction: Apply learning algorithms to create a model from the training data.

  2. Learn from the model using a test set to evaluate its efficacy.

Decision Trees Example

  • Splitting Attributes:

    • Home Ownership → Marital Status → Income.

Decision Tree Induction

  • Algorithms:

    • Hunt’s Algorithm, ID3, C4.5, CART, SLIQ, SPRINT.

  • General Procedure:

    • If instances belong to the same class, the node is a leaf.

    • If instances belong to multiple classes, split using an attribute.

    • Apply this recursively.

Design Issues with Decision Tree Induction

Splitting Procedures

  • How to split?:

    • Test conditions should express how to split data (multi-way or binary).

  • Stopping Criteria:

    • Instances of the same class

    • Instances sharing similar attributes

    • If instances fall below a minimum threshold.

Measures of Node Impurity

  • Common Measures:

    • Gini Index

    • Entropy

    • Classification Error

Entropy Calculation

  • Entropy is higher when instances are evenly distributed among classes.

    • Formula:[ Entropy = - \sum_{i=0}^{c-1} p_i(t) \log_2 p_i(t) ]

Finding the Best Split

  1. Compute impurity measure before splitting.

  2. Compute impurity measure after splitting.

  3. Choose the attribute condition that produces the highest gain or lowest impurity after splitting.

Issues with Multiple Partitions

  • High node impurity can lead to preference for overly complex classifications.

Gain Ratio

  • Adjusts Information Gain by the uncertainty of the split.

    • Reduces penalty for larger number of partitions.

Advantages and Disadvantages of Decision Trees

Advantages

  • Inexpensive to construct

  • Fast classification of unknown records

  • Good interpretability for small trees

Disadvantages

  • Greedy nature may miss important attribute interactions

  • Each decision boundary only involves a single attribute

Conclusion

  • Decision trees provide a clear framework for classification models with several inherent design considerations that affect their performance.