01 Nonlinear Extensions

Assumptions for Linear Regression

We assume that each variable correlates linearly with the output, and we also assume there is no interaction between the variables, meaning their effects on the output are independent and additive (i.e., they combine linearly without influencing each other).

Linearity “along the axes”

  • The output variable y depends linearly on the input variables individually.

  • If you focus on just one variable (ignoring the other), the relationship between y and that variable will still be a straight line.

  • In a 3D visualization, removing one variable results in a linear slice.

  • when violated: use polynomial regression

2. Additivity

  • The relationship between y and the inputs is assumed to be additive—it’s the sum of the effects.

  • This assumption breaks if there’s a multiplicative interaction between inputs​, because the effect of one variable depends on the value of the other.

  • Example: A function like z=0.3x+0.2x⋅y violates additivity because of the x⋅y.

  • when violated: use interaction term

Modeling Nonlinear Data

Approaches to Extend Linear Models

  • When linearity assumptions are violated, consider two main approaches:

    • Polynomial Regression:

      • Replace linear terms with polynomial functions if data is nonlinear along an axis.

      • Example: horsepower vs. mileage can be modeled using a quadratic function if scatterplots suggest such a relationship.

    • Interaction Terms:

      • Used to address violations of additivity by examining how the effect of one variable depends on another.

      • Interaction indicates that changes in one variable affect the slope of another variable.

Examples of Polynomial Regression

  • Example of polynomial function: z = 0.3x² + 0.6y² shows quadratic behavior.

  • Visualize using intersections to analyze how output behaves along x and y axes.

Advanced Nonlinear Functions and Techniques

Limitations of Polynomial Regression

  • Higher-degree polynomials can result in poor predictive performance outside the range of sample data.

  • Predictions can become erratic at the data extremes due to wild behavior in higher-degree models.

  • This issue necessitates careful consideration in model structure and choosing appropriate methodologies.