L

11-Naive Bayes

Classification Model

  • Objective: To find P(Y=1 | X1, X2, ..., Xm)

  • Examples:

    • P(Y=Default | Income < 30K, Education = College, ...)

    • P(Y=Spam Email | X1 = "lottery", X2 = "win", ...)

  • Characteristics:

    • Naïve Bayes models do not have trainable parameters like neural networks.

    • It utilizes probability theory to determine the likelihood of the event of interest without iterative processes.

Probability and Conditional Probability

  • Definitions:

    • Probability P(A): Likelihood of event A occurring

      • Example: 23% of days are rainy, P(rain) = 0.23

    • Joint Probability P(A, B): Likelihood of events A and B occurring together

    • Conditional Probability P(A|B): Likelihood of event A given event B.

      • Example: P(rain|cloudy) = 0.62, cloudy days have 62% chance of rain.

Example of Probability Calculation

  • Survey of 26 Persons:

    • Own a Pet Distribution:

      • Female: 8 own, 6 don’t

      • Male: 5 own, 7 don’t

    • Calculations:

      • P(own a pet) = 13/26 = 0.50

      • P(own a pet | female) = 8/14 = 0.57

Probability Chain Rule

  • Chain Rule Formula: P(A, B) = P(B|A)P(A)

  • Example of Calculations:

    • P(own a pet, female) can be calculated using P(female) and P(own a pet | female).

    • Formula: P(own a pet, female) = P(female) x P(own a pet | female) = (14/26) x (8/14) = 8/26

Bayes’ Theorem

  • Formula:

    • From chain rule, P(A, B) = P(B|A)P(A) = P(A|B)P(B)

  • Also known as Bayes’ Rule

Bayes’ Theorem Example

  • Disease Pre-screening:

    • Accuracy: 95% when the patient has the disease (P(positive|disease) = 0.95).

    • Probability of Disease: P(disease) = 0.001 (1 in 1000).

  • Calculating Probability of Disease Given Positive Result:

    • Formula: P(disease|positive) = P(positive|disease) * P(disease) / P(positive)

    • Computation example leads to P(disease|positive) approximately = 0.0509 (1.87%).

Use of Bayes’ Theorem in Machine Learning

  • Components:

    • Prior Probability of Labels

    • Posterior Probability of Predictions based on Evidence

  • Evidence: Observed features (attribute values)

  • Relationship between Likelihood and Prior

    • P(Label | Data) = P(Data | Label) * P(Label) / P(Data)

Bayesian Classification

  • Objective: Predict value of Y using m features

  • Apply Bayes’ Rule:

    • P(Y=1 | X1 = a1, X2 = a2, ..., Xm = am)

    • The denominator disregarded shows focus on maximizing the numerator.

Naïve Bayes Model

  • Naïve Assumption: All features are conditionally independent given the class.

    • Formula: P(X1 = a1 | Y = 1) * P(X2 = a2 | Y = 1) * ... * P(Y = 1)

  • Characteristics of Naïve Bayes:

    • Classifies based on input feature probabilities

    • Pros:

      • Simplicity

      • Performance in real-world applications

      • Handles multi-class classification effectively

    • Cons:

      • Struggles with numerical features without adjustments

      • Strong independence assumption may not always hold

Learning Objective

  • Understand the Naive Bayes model thoroughly