Data Mining (DATS 2103): Classification

Classification

Predicts a qualitative response

Logistic Regression

Models the probability that Y belongs to a particular category

Why not Linear Regression?

Linear regression may produce values outside the [0

Logistic Function

Function used as the basis for logistic regression

Prediction Threshold

Often predict Y=1 when p(X) = Pr(Y=1|X) > 0.5

Odds

p/(1-p)

Log Odds (Logit)

The logarithm of the odds; has a linear relationship with the predictors

Logistic Regression Coefficients

Estimated using Maximum Likelihood Estimation (MLE)

Maximum Likelihood Estimation (MLE)

Selects coefficients so the predicted probability is as close as possible to the individual's observed outcome

Multiple Logistic Regression

Logistic regression using multiple predictors

Multinomial Logistic Regression

Logistic regression that allows more than two response classes (K > 2)

Multinomial Logistic Regression Methods

Run K−1 independent binary models or use softmax coding to run one model

Multinomial Logistic Regression Assumption

Odds of preferring Class A vs Class B are not dependent on the presence of other classes

Predictor Independence in Multinomial Logistic Regression

Predictors do not necessarily need to be independent

Baseline Class

A class selected as the reference when running K−1 independent binary models

Multinomial Coefficient Interpretation

Depends on the two classes being compared

Softmax Coding

Runs one multinomial model with no baseline; all K classes are treated as symmetric

Generative Models for Classification

Models the distribution of the predictors X separately within each response class using Bayes' Theorem

Why Use Generative Models?

They can be more accurate when classes are far apart

Prior Probability (πk)

Probability that a randomly chosen observation comes from the kth class

fk(x)

Density function of X for an observation that comes from the kth class

Bayes' Theorem

Used to calculate posterior probabilities for classification

Linear Discriminant Analysis (LDA)

Generative classification method that assumes predictors follow a Gaussian distribution and produces a linear decision boundary

LDA for p=1

Assumes X is normally or Gaussian distributed

LDA for Multiple Predictors

Assumes X follows a multivariate Gaussian distribution

LDA Parameters

Estimates prior probability

LDA Classification

Calculate the discriminant for each class and assign the observation to the class with the highest discriminant

LDA Decision Boundary

The point where the discriminants for two classes are equivalent

LDA Boundary

Linear

LDA Sample Size

Performs best with fewer observations because reducing variance is important

Quadratic Discriminant Analysis (QDA)

Similar to LDA but uses class-specific variance instead of a common variance

QDA Boundary

Quadratic

QDA Parameters

Estimates prior probability

QDA Sample Size

Performs best with large amounts of observations because reducing variance is less important

LDA vs QDA

LDA uses a common variance and linear boundary; QDA uses class-specific variance and quadratic boundary

Naive Bayes Classifier

Classification method that assumes predictors are independent within classes

Naive Bayes Assumption

Predictors are independent within classes

Quantitative Predictors in Naive Bayes

Can assume each predictor comes from a normal distribution within each class or use a non-parametric estimate

Histogram Estimation

Estimates the density as the fraction of training observations in a class that fall into the same histogram bin as the predictor value

Kernel Density Estimation

A smoothed version of histogram-based density estimation

Qualitative Predictors in Naive Bayes

Use the proportion of training observations corresponding to each predictor value within each class

K-Nearest Neighbors (KNN)

Non-parametric classification method that assigns a prediction based on the K nearest training observations

KNN Process

Choose K

KNN Sample Size

Requires many more observations than the other methods

KNN Coefficients

Does not report coefficients

Linear Decision Boundary

LDA and Logistic Regression perform well

Moderately Nonlinear Decision Boundary

QDA or Naive Bayes may perform better

Highly Complicated Decision Boundary

A non-parametric method such as KNN may be superior

KNN Smoothness

The level of smoothness for a non-parametric approach must be chosen carefully