Lecture 9: Linear Regression - In-Depth Notes

Key Concepts of Regression

• Definition of Regression: Predicts a quantitative variable through relationships between independent variables and a dependent variable, useful for analyzing trends and making forecasts. Example: Predicting home prices based on various factors.

Comparison of Models

Classification vs. Regression

• Classification: Sorts data into categories; predicts categorical outcomes (binary/multiclass). Examples: Party affiliation, health status, spam detection. Models used include logistic regression and decision trees.

• Regression: Predicts continuous numerical values. Example: House pricing based on square footage and location.

K-Nearest Neighbors (KNN) Regression

• Mechanism: Averages outputs of the K-nearest neighbors based on distance metrics (e.g., Euclidean).

• Visualization: Yields a flexible, non-linear curve.

• Disadvantages:

  • Computationally Intensive: Slower with larger datasets.

  • Overfitting Risks: Poor performance with many predictors.

  • Extrapolation Issues: Less reliable beyond training ranges.

  • Trend Interpretation: Difficult due to non-parametric nature.

Linear Regression

• Definition: Establishes a linear relationship between dependent and independent variables; known as the "line of best-fit".

• When to Use:

  • Linear relationship evident.

  • Small datasets.

  • Multiple predictors present.

• Mechanism: Minimizes Root Mean Square Error (RMSE) to achieve the best fit.

• Equation: Y = B₀ + B₁X, where B₀ is the y-intercept and B₁ is the slope, indicating changes in Y with X.

Choosing the Best Fitting Line

• Criteria: Fit line minimizes RMSE; assess closeness to actual data with visualizations.

Advantages of Linear Regression vs. KNN

• Interpretable: Coefficients provide direct meaning.

• Efficient: Faster computations; fewer data requirements.

• Disadvantages: Cannot capture nonlinearities unless adjusted (e.g., polynomial terms).

Using R for Linear Regression

• Necessary Packages: Use tidyverse for data manipulation, tidymodels for modeling.

• Steps:

  1. Specify the formula (e.g., price ~ sqft).

  2. Define the model with linear_reg() using "lm" engine.

  3. Fit model to data using a combined workflow.

Multiple Linear Regression

• Definition: Uses multiple predictors to fit a hyperplane; model: Y = B₀ + B₁X₁ + B₂X₂ + …

• Interpreting Coefficients: Each coefficient shows impact on the dependent variable, holding others constant.

Model Evaluation

• RMSPE: Key metric; lower RMSPE means better model fit and prediction accuracy.

Common Issues in Linear Regression

Outliers

• Significant deviations affecting results; need visualization to assess impact.

Multicollinearity

• High correlation among predictors; leads to unreliable coefficient estimates. Use variance inflation factor (VIF) for detection.

Feature Engineering

• Process of creating or transforming predictors to enhance model fit. Avoid test data in feature creation; validate using cross-validation.

Summary of Regression

• Beyond prediction, regression analyzes variable relationships; knowing regression types deepens data analysis skills.