Untitled

Bayesian Classification

Definition and Importance

  • Bayesian Classification is a statistical method for classifying data points based on Bayes' theorem. It utilizes prior knowledge of class distributions to make predictions about new data.

A Simple Species Classification Problem

  • Example: Measure the length of a fish to classify it as either Hilsa or Tuna.

Data Collection

  • Statistics Collection: Gather data on the populations for each class (Hilsa, Tuna).

Fish Length Distribution

  • Distribution of Fish Lengths: Length in feet represented as counts or probabilities.

    • Example data: 1ft, 4ft for both classes.

  • Class Conditional Distributions: Probability density functions for given lengths in each class:

    • P(L=1.75extft∣extHILSA)P(L = 1.75 ext{ ft} | ext{HILSA})

    • P(L=4.25extft∣extTUNA)P(L = 4.25 ext{ ft} | ext{TUNA})

Decision Rule

  • Rule: If fish length L is less than or equal to boundary B, classify as HILSA; otherwise, classify as TUNA.

  • Determining B: Based on the population statistics of both species.

Error Analysis

  • Types of Errors:

    • Type 1 Error: Classifying a Tuna as Hilsa.

    • Area under the pink curve to the left of B.

    • Type 2 Error: Classifying a Hilsa as Tuna.

    • Area under the blue curve to the right of B.

Optimal Decision Rule

  • Optimal B Value: Set B* to minimize errors.

  • If costs associated with Type 1 and Type 2 errors differ, the optimal boundary shifts.

  • Minimum Possible Error Condition:

    • P(B∗∣extHILSA)=P(B∗∣extTUNA)P(B^* | ext{HILSA}) = P(B^* | ext{TUNA})

Species Identification Steps

  1. Measure lengths of a sizable population of both Hilsa and Tuna fishes.

  2. Estimate class conditional distributions for both classes.

  3. Determine optimal decision boundary B* from these distributions.

  4. Apply the decision rule to classify a new fish with minimum error probability.

Location-Dependent Class Probabilities

  • Location of Experiment: Factors influencing classification; for example:

    • Calcutta in Monsoon: More Hilsa and fewer Tuna.

    • California in Winter: More Tuna and fewer Hilsa.

  • An example of classification based on length:

    • A 2ft fish in Calcutta likely Hilsa (high value), while a 1.5ft fish may be a Tuna in California.

Apriori Probability

  • Defining Apriori Probability: Not based on measurements but statistical likelihood based on location/time.

    • For Calcutta:

    • P(extHILSA)=0.90,extP(extTUNA)=0.10P( ext{HILSA}) = 0.90, ext{ } P( ext{TUNA}) = 0.10

    • For California:

    • P(extTUNA)=0.95,extP(extHILSA)=0.05P( ext{TUNA}) = 0.95, ext{ } P( ext{HILSA}) = 0.05

    • For London:

    • P(extTUNA)=0.50,extP(extHILSA)=0.50P( ext{TUNA}) = 0.50, ext{ } P( ext{HILSA}) = 0.50

Classification Decision Making

  • Posterior Probability Calculation:

    • Using Bayes' rule for classification:

    • P(extHILSA∣L=2ft)=racP(extHILSA)imesP(L=2ft∣extHILSA)P(L=2ft)P( ext{HILSA} | L = 2ft) = rac{P( ext{HILSA}) imes P(L=2ft | ext{HILSA})}{P(L=2ft)}

  • Components:

    • Apriori: Assumed based on location/time.

    • Class Conditional: Given fish belongs to a particular class, determine length's probability.

    • Posteriori: Given the measured length, evaluate probability of class membership.

Bayes Classification Rule (Bayes Classifier)

  • Using posteriori probability to define classification boundaries:

    • Decision Boundary B: Set the optimal B such that:

    • P(HILSA∣L=B<em>)=P(TUNA∣L=B</em>)P(HILSA | L = B^<em>) = P(TUNA | L = B^</em>)

  • Posterior Distributions: Used to calculate minimum error probability: Bayes Error.

MAP Classification Strategy

  • Maximum A Posteriori (MAP): Classify to the class with the highest posterior probability for a given length L.

Multiclass Classifications

  • Extend MAP to multiple classes, finding the class with the maximum probability among all classes.

Multivariate Classifier

  • Feature Space (Length, Weight): Aim for class separability based on weight and length.

Decision Boundary Dynamics

  • Normal Distribution Assumptions:

    • Changes in decision boundaries with different variances:

    • Spherical classes with equal variance.

    • Classes with different variances, forming parabolic boundaries.

Distance Metrics in Classification

  • Various metrics used to determine proximity:

    • Euclidean, Minkowski distances for vector separation.

    • Mahalanobis, Bhattacharya distances when comparing a vector and a distribution.

    • Kullback-Liebler Divergence used for between two distributions.

Class Distribution Estimation Challenges

  • Sample Size Requirement: Challenges arise especially in high-dimensional spaces; smaller sample size leads to difficulty.

  • Independence Assumption: Length and weight can be treated as independent attributes when estimating joint probability distributions.

Naïve Bayes Classifier

  • Assumes conditional independence among attributes given a class:

    • P(A1,A2,…,An∣C)=P(A1∣C)P(A2∣C)…P(An∣C)P(A1, A2, …, An |C) = P(A1|C) P(A2|C) … P(An|C)

  • Classify new points based on the maximum probability criteria.

Estimation Techniques in Naïve Bayes

  • Probability Estimation Factors:

    • Conditional probabilities and their associated prior probabilities.

  • Handling Zero Probabilities: If a conditional probability remains zero, it nullifies classification output.

Summary Evaluation of Bayes Classifier

  • Advantages: Robust to isolated noise, missing values, and irrelevant attributes, though the independence assumption may not hold.

  • Conditional Independence Concept: Events are conditionally independent given another event/attribute.

  • Explaining that independence may not be very stable throughout datasets can bring about local properties for optimization.

Final Note

  • Understanding these principles allows for effectively applying Bayesian classification models across various datasets and domains, emphasizing the importance of prior knowledge in classification tasks.