Data Mining Quiz 2

Classification

Assigns class info each sample (qualitative, categorical)

Logistic Regression

Models the probability that Y belongs to a particular category

Maximum Likelihood Estimation

Selecting 𝛽0 and 𝛽1 such that the predicted probability of (default = yes) is as close as possible the individual’s observed default status

Multinomial Logistic Regression

Allows for K > 2

Assumes that the odds for one class does not depend on the other

Predictors do not necessarily need to be explicitly independent, however low correlation between variables is preferred

K - 1

Runs idependent binary models

• Banana vs Apple → binary model

• Cherry vs Apple → binary model

Generative Models

If classes are too far apart

If X is approximately normal in each class and the sample size is small, generative models can be more accurate

Easier to extend to more than two response classes

Bayes Theorem

πk​=P(Y=k) = probability that a randomly chosen observation belongs to class K.

• Example:

  • πApple=05→ 50% of all fruits are Apples.

  • πBanana=0.3→ 30% Bananas.

  • πCherry​=0.2 → 20% Cherries.

Linear Discriminant Analysis

• Prior probability = your initial belief about a class.

• Likelihood = how well the observed data fits that class.

• Posterior probability = updated belief after seeing the data.

• For each observation, calculate the discriminant for each class, assign to the class with the highest discriminant

• To find the LDA decision boundary, find the point where these two discriminants are equivalent

• Performs best with fewer observations, reducing the variance is important

Quadratic Discriminant Analysis

Like LDA, with the difference of a class specific variance rather an a common variance (how far they differ from the mean), allows for quadratic shaped decision boundaries

Performs best with large amounts of observations

Naive Bayes Classifier

Does not combine predictors in each class, but assumes they are independent

If Xj is quantitative:

○ Can assume within each class, the j th predictor comes from a normal distribution

In a line up of apples count every 5th apple as data

K-Nearest Neighbors

• Non parametric

• Many observations

• Select a value of K, plot all training observations, for each test observation find the K nearest training observations, assign predicted value to test observation based on average known response value for the K nearest training observations

Generalized Linear Models

Useful when data is neither qualitative nor quantitative

○ Value more closely represents counts of a unit

○ Ex: CaBi ride share data, predicting number of riders

LDA & Logistic Regression

• When it can be linear

• Yes/No situation

QDA and Naive Bayes

• Moderately non-linear boundaries

• QDA allows each class to have its own covariance (curved boundaries).

• Naive Bayes can capture more complex shapes depending on predictor distributions.

Non-parametric approach like KNN

• Very non-linear boundaries

• KNN does not assume any formula for the boundary. It decides based on local neighborhoods.