Untitled
Bayesian Classification
Definition and Importance
Bayesian Classification is a statistical method for classifying data points based on Bayes' theorem. It utilizes prior knowledge of class distributions to make predictions about new data.
A Simple Species Classification Problem
Example: Measure the length of a fish to classify it as either Hilsa or Tuna.
Data Collection
Statistics Collection: Gather data on the populations for each class (
Hilsa,Tuna).
Fish Length Distribution
Distribution of Fish Lengths: Length in feet represented as counts or probabilities.
Example data: 1ft, 4ft for both classes.
Class Conditional Distributions: Probability density functions for given lengths in each class:
Decision Rule
Rule: If fish length L is less than or equal to boundary B, classify as HILSA; otherwise, classify as TUNA.
Determining B: Based on the population statistics of both species.
Error Analysis
Types of Errors:
Type 1 Error: Classifying a Tuna as Hilsa.
Area under the pink curve to the left of B.
Type 2 Error: Classifying a Hilsa as Tuna.
Area under the blue curve to the right of B.
Optimal Decision Rule
Optimal B Value: Set B* to minimize errors.
If costs associated with Type 1 and Type 2 errors differ, the optimal boundary shifts.
Minimum Possible Error Condition:
Species Identification Steps
Measure lengths of a sizable population of both Hilsa and Tuna fishes.
Estimate class conditional distributions for both classes.
Determine optimal decision boundary B* from these distributions.
Apply the decision rule to classify a new fish with minimum error probability.
Location-Dependent Class Probabilities
Location of Experiment: Factors influencing classification; for example:
Calcutta in Monsoon: More Hilsa and fewer Tuna.
California in Winter: More Tuna and fewer Hilsa.
An example of classification based on length:
A 2ft fish in Calcutta likely Hilsa (high value), while a 1.5ft fish may be a Tuna in California.
Apriori Probability
Defining Apriori Probability: Not based on measurements but statistical likelihood based on location/time.
For Calcutta:
For California:
For London:
Classification Decision Making
Posterior Probability Calculation:
Using Bayes' rule for classification:
Components:
Apriori: Assumed based on location/time.
Class Conditional: Given fish belongs to a particular class, determine length's probability.
Posteriori: Given the measured length, evaluate probability of class membership.
Bayes Classification Rule (Bayes Classifier)
Using posteriori probability to define classification boundaries:
Decision Boundary B: Set the optimal B such that:
Posterior Distributions: Used to calculate minimum error probability: Bayes Error.
MAP Classification Strategy
Maximum A Posteriori (MAP): Classify to the class with the highest posterior probability for a given length L.
Multiclass Classifications
Extend MAP to multiple classes, finding the class with the maximum probability among all classes.
Multivariate Classifier
Feature Space (Length, Weight): Aim for class separability based on weight and length.
Decision Boundary Dynamics
Normal Distribution Assumptions:
Changes in decision boundaries with different variances:
Spherical classes with equal variance.
Classes with different variances, forming parabolic boundaries.
Distance Metrics in Classification
Various metrics used to determine proximity:
Euclidean, Minkowski distances for vector separation.
Mahalanobis, Bhattacharya distances when comparing a vector and a distribution.
Kullback-Liebler Divergence used for between two distributions.
Class Distribution Estimation Challenges
Sample Size Requirement: Challenges arise especially in high-dimensional spaces; smaller sample size leads to difficulty.
Independence Assumption: Length and weight can be treated as independent attributes when estimating joint probability distributions.
Naïve Bayes Classifier
Assumes conditional independence among attributes given a class:
Classify new points based on the maximum probability criteria.
Estimation Techniques in Naïve Bayes
Probability Estimation Factors:
Conditional probabilities and their associated prior probabilities.
Handling Zero Probabilities: If a conditional probability remains zero, it nullifies classification output.
Summary Evaluation of Bayes Classifier
Advantages: Robust to isolated noise, missing values, and irrelevant attributes, though the independence assumption may not hold.
Conditional Independence Concept: Events are conditionally independent given another event/attribute.
Explaining that independence may not be very stable throughout datasets can bring about local properties for optimization.
Final Note
Understanding these principles allows for effectively applying Bayesian classification models across various datasets and domains, emphasizing the importance of prior knowledge in classification tasks.