Reviewer for Data Mining (MIDTERM)

Regression

A supervised learning technique used to predict continuous numerical values based on one or more input variables.

Dependent Variable (Response)

The value you want to predict in a regression model.

Independent Variable (Predictor/Features)

The variable(s) used to make predictions in regression.

Goal of Regression

To find the best

Regression Applications

Finance, healthcare, marketing, manufacturing, retail.

Continuous Target Variable

Regression predicts a continuous value like sales price or height.

Mean Squared Error (MSE)

The average of the squares of the errors, a common regression metric.

Root Mean Squared Error (RMSE)

The square root of MSE, measures average prediction error in same units as target.

Overfitting

When a model is too complex and learns noise from training data (poor generalization).

Underfitting

When a model is too simple and misses key data patterns.

Interpretability

Regression coefficients show how much each predictor affects the target.

Predictor Variable (Feature)

Input used for prediction in regression.

Response Variable

Output to be predicted.

Coefficient

Represents the change in the response variable for a one

Residuals

The differences between observed and predicted values.

Multicollinearity

Situation where predictors are highly correlated, which may affect coefficient stability.

Outliers

Data points that deviate substantially and may distort the regression model.

Simple Regression

Regression with one predictor and one response variable.

Multiple Regression

Regression using two or more predictors for a single response variable.

Nonlinear Regression

Regression capturing nonlinear relationships, e.g. plant growth over time.

Simple Linear Regression Formula

y = β0 + β1 x

Multiple Linear Regression Formula

y = β0 + β1X1 + β2X2 + … + βpXp + ε

Linear Regression

Algorithm fitting a straight line to predict outcomes.

Polynomial Regression

Algorithm fitting a curve, capturing nonlinear relationships.

Ridge Regression

Regularization method preventing overfitting by shrinking coefficients.

Lasso Regression

Regularization method that can force some coefficients to exactly zero for feature selection.

Decision Tree Regression

Uses tree structures, can handle nonlinear relationships.

Random Forest Regression

Uses ensemble of trees for robust, less overfitted predictions.

Support Vector Regression (SVR)

Uses hyperplanes in high

Advantages of Regression

Interpretable, good for forecasting, reveals feature importance, flexible.

Disadvantages of Regression

Often assumes linearity, can overfit, sensitive to outliers.

Mean Absolute Error (MAE)

Average of absolute errors between predicted and actual, lower is better.

Mean Squared Error (MSE)

Average squared error, penalizes large errors, lower is better.

Root Mean Squared Error (RMSE)

Sqrt of MSE, interpretable in target units, lower is better.

R

squared (R2)

Adjusted R-squared

squared

Classification

Categorizes data points into defined classes based on features.

Binary Classification

Classification with two possible outcomes (e.g., spam/not spam).

Multiclass Classification

Classification with more than two possible labels.

Classification Applications

Credit risk analysis, shopping prediction, medical diagnosis, sentiment analysis.

Supervised Classification

Trained using labeled data (target classes known).

Unsupervised Classification

Discovers classes from unlabeled data (e.g., clustering).

Training Phase (Classification)

Model learns from labeled data.

Testing Phase (Classification)

Model is validated on new/unseen data for accuracy.

Dataset Split (Classification)

Split into training (60

Decision Tree Classifier

Uses tree structure; splits data on features to classify.

Decision Tree Algorithms

ID3 and C4.5 use information gain and Gini index for splits.

Advantage of Decision Trees

Easy to interpret and use.

Disadvantage of Decision Trees

Can be sensitive to small changes, may be inaccurate or complex.

Overfitting in Trees

Complex trees may overfit to training data.

Pruning

Removes unnecessary branches to improve prediction on unseen data.

Information Theory

Quantifies information and measures uncertainty, crucial in machine learning.

Entropy (H)

Measures randomness/uncertainty or impurity in a dataset.

High Entropy

More uncertainty; labels are mixed.

Low Entropy

More certainty; labels are pure.

Entropy Formula

H(S) =

Information Gain (IG)

How much a feature reduces entropy when splitting data.

Purpose of Information Gain

Select the best attribute for decision

Information Gain Formula

IG(S, A) = H(S)

Confusion Matrix

Shows count of actual vs. predicted classes in classification.

True Positive (TP)

Correctly predicted as positive.

True Negative (TN)

Correctly predicted as negative.

False Positive (FP)

Incorrectly predicted as positive.

False Negative (FN)

Incorrectly predicted as negative.

Precision

TP/(TP + FP): Fraction of predicted positives that are actual positives.

Recall

TP/(TP + FN): Fraction of actual positives correctly found.

F1 Score

Harmonic mean of Precision & Recall: 2(PrecisionRecall)/(Precision+Recall)