4 Regression

[Regression fundamentals] What is regression in supervised learning?

Predicting a numerical target from input features.

[Regression fundamentals] How does regression differ from classification?

Regression predicts a numerical value; classification predicts a discrete class label.

[Regression fundamentals] What is a dependent or response variable?

The target y whose value is to be predicted or explained.

[Regression fundamentals] What is an explanatory or predictor variable?

An input feature x used to predict the response.

[Regression fundamentals] Why is regression a supervised learning task?

Training examples include both input features and the corresponding numerical target values.

[Regression fundamentals] How does prediction differ from simply measuring correlation?

Prediction defines a target and a direction of mapping from inputs to that target; correlation summarizes an association between variables.

[Regression fundamentals] Does a useful regression predictor establish a causal relationship?

No. Predictive association alone does not show that changing an input will cause the target to change.

[Regression fundamentals] What is a valid interpretation of a predictive association?

A feature helps predict the target in the observed setting, without automatically implying causation.

[Regression fundamentals] How can a numerical target be converted into a classification problem?

Discretize it into bins, such as age <=20, 20<age<=30, and age>30, then predict the bin.

[Regression fundamentals] What is lost when the target is coarsely discretized?

Numerical resolution: different values in the same bin become indistinguishable.

[Regression fundamentals] Why does very fine target discretization need more data?

Many narrow classes must each have enough examples for reliable learning.

[Regression fundamentals] Why is linear regression an important starting point?

It is relatively interpretable, straightforward to fit, and a building block for more complex methods.

[Linear regression model] What is the one-feature linear regression model with an intercept?

y_hat = w*x + b, where w is the slope and b is the intercept.

[Linear regression model] What does the coefficient w represent in y_hat=w*x+b?

The change in the predicted target for a one-unit increase in x.

[Linear regression model] What does a positive regression slope imply?

The predicted target increases as the input increases.

[Linear regression model] What does a negative regression slope imply?

The predicted target decreases as the input increases.

[Linear regression model] What does the intercept b represent?

The model's predicted target at x=0.

[Linear regression model] Does the intercept always have a useful real-world interpretation?

No. x=0 may lie outside the meaningful or observed input range.

[Linear regression model] What restriction is imposed by the model y_hat=w*x without an intercept?

The fitted line must pass through the origin.

[Linear regression model] For y_hat=3*x+2, what is the prediction at x=4?

3*4+2 = 14.

[Linear regression model] What is a d-feature linear regression model with an intercept?

y_hat_i = w_0 + sum_{j=1}^d w_j*x_ij.

[Linear regression model] How can the intercept be included in a dot product?

Add a constant feature x_i0=1, then write y_hat_i = w^T*x_i with w=(w_0,...,w_d)^T.

[Linear regression model] What is the shape of the weight vector with d ordinary features and an intercept?

(d+1) by 1.

[Linear regression model] What is the shape of the augmented design matrix for n examples and d features?

X has n rows and d+1 columns, including the constant intercept column.

[Linear regression model] What is the matrix form of predictions for all examples?

y_hat = X*w.

[Linear regression model] Why is w^T*x_i a scalar?

It is the inner product of two column vectors of the same length.

[Linear regression model] How are the individual input vectors arranged in X?

Each row of X is x_i^T, the transpose of an example's column feature vector.

[Linear regression model] What does a coefficient w_j mean in multiple linear regression?

A one-unit increase in feature j changes the predicted target by w_j, holding the other input coordinates fixed.

[Linear regression model] What geometric object is fitted by a linear model with two input features?

A plane in the three-dimensional space (x_1,x_2,y), or a plane through the origin if there is no intercept.

[Linear regression model] What is the prediction of y_hat=10*x_1+25*x_2 at x_1=2 and x_2=3?

10*2+25*3 = 95.

[Least-squares loss] Why is exact-match classification error unsuitable for ordinary regression?

Continuous predictions rarely equal targets exactly; the magnitude of a numerical error matters.

[Least-squares loss] What is a prediction residual?

The difference between the observed target and the prediction, often written r_i=y_i-y_hat_i.

[Least-squares loss] What is the least-squares SSE objective for y_hat=w*x?

SSE(w) = sum_i (w*x_i-y_i)^2.

[Least-squares loss] What is the SSE objective when an intercept is included?

SSE(w,b) = sum_i (w*x_i+b-y_i)^2.

[Least-squares loss] Do squared residuals measure vertical or perpendicular distances in ordinary least squares?

Vertical differences in the target coordinate, not the shortest geometric distances to the line.

[Least-squares loss] Why does squared loss penalize large errors strongly?

Squaring magnifies the influence of large residual magnitudes.

[Least-squares loss] Is ordinary least squares robust to extreme target outliers?

Generally no. Large residuals can dominate the squared-error objective and pull the fitted model.

[Least-squares loss] Why is squared error convenient for optimization?

It is differentiable and produces a convex quadratic objective for a linear predictor.

[Least-squares loss] What optimization difficulty does absolute-error loss have at residual zero?

The absolute-value function is not differentiable at zero, so ordinary derivatives alone are insufficient there.

[Least-squares loss] Does replacing SSE with half-SSE change the unregularized minimizer?

No. Multiplication by the positive constant 1/2 preserves all minimizers.

[Least-squares loss] Does adding a constant to an objective change its minimizer?

No. A constant independent of the model parameters shifts all objective values equally.

[Least-squares loss] Can an objective be multiplied by any negative constant without changing its minimizers?

No. A negative multiplier reverses the ordering, exchanging minimization and maximization.

[Least-squares loss] For predictions {2,5} and targets {1,7}, what is SSE?

(2-1)^2+(5-7)^2 = 1+4 = 5.

[Least-squares loss] How are MSE and SSE related for a fixed sample?

MSE = SSE/n. They have the same unregularized minimizers because n is a fixed positive constant.

[Least-squares loss] What general stationary-point method is illustrated for one-dimensional differentiable objectives?

Differentiate, solve f'(w)=0, and check curvature or compare relevant candidates to identify a minimum.

[Least-squares loss] Why does f'(w)=0 alone not prove that w is a minimum for an arbitrary function?

A stationary point could be a maximum or another nonminimum point; additional analysis is needed.

[Least-squares loss] For f(w)=0.5*sum_i(w*x_i-y_i)^2, what is f'(w)?

f'(w) = sum_i x_i*(w*x_i-y_i) = w*sum_i x_i^2 - sum_i x_i*y_i.

[Least-squares loss] What are a, b, and c in the quadratic expansion f(w)=0.5*a*w^2-b*w+c?

a=sum_i x_i^2, b=sum_i x_i*y_i, and c=0.5*sum_i y_i^2.

[Least-squares loss] What is the least-squares solution for one feature and no intercept?

w_star = [sum_i x_i*y_i]/[sum_i x_i^2], provided the denominator is positive.

[Least-squares loss] How is the no-intercept least-squares formula derived?

Differentiate half-SSE, set w*sum_i x_i^2-sum_i x_i*y_i=0, and solve for w.

[Least-squares loss] What is the second derivative of half-SSE for one feature and no intercept?

f''(w) = sum_i x_i^2.

[Least-squares loss] When is the no-intercept one-feature least-squares minimum unique?

When at least one input x_i is nonzero, so sum_i x_i^2 > 0.

[Least-squares loss] What happens if every x_i is zero in the no-intercept model?

Every prediction is zero regardless of w, so all values of w have the same loss and the usual ratio formula is undefined.

[Least-squares loss] For no-intercept data (x,y)={(1,2),(2,3)}, what is the fitted slope?

w_star = (1*2+2*3)/(1^2+2^2) = 8/5 = 1.6.

[Least-squares loss] For no-intercept data {(1,2),(2,3)} and w=1.6, what is SSE?

Predictions are 1.6 and 3.2; SSE = (-0.4)^2+(0.2)^2 = 0.20.

[Multivariable optimization] What is the d-dimensional least-squares objective?

f(w) = sum_i (w^T*x_i-y_i)^2, or a positive constant multiple such as half that sum.

[Multivariable optimization] What is a gradient?

The column vector of partial derivatives of an objective with respect to all its parameters.

[Multivariable optimization] What is the jth partial derivative of half-SSE?

partial f/partial w_j = sum_i x_ij*(w^T*x_i-y_i).

[Multivariable optimization] What stationary-point condition replaces f'(w)=0 for several parameters?

Set the gradient vector equal to the zero vector: grad f(w)=0.

[Multivariable optimization] What is the matrix gradient of f(w)=0.5*||X*w-y||^2?

grad f(w) = X^T*(X*w-y).

[Multivariable optimization] What equations result from setting the least-squares matrix gradient to zero?

X^T*X*w = X^T*y, called the normal equations.

[Multivariable optimization] When can the normal-equation solution be written using an inverse?

If X has full column rank, w_star=(X^T*X)^(-1)*X^T*y. If not, a unique inverse-based solution is not available.

[Multivariable optimization] Why is linear least squares convex?

Its Hessian X^T*X is positive semidefinite, so every stationary point is a global minimum.

[Multivariable optimization] Does convexity always imply a unique least-squares parameter vector?

No. If the columns of X are linearly dependent, multiple parameter vectors can give the same minimum predictions/loss.

[Multivariable optimization] What is the gradient-descent update rule?

w <- w - eta*grad f(w), where eta > 0 is the learning rate.

[Multivariable optimization] Why does gradient descent subtract the gradient?

The gradient points toward local steepest increase, so its negative gives a direction of local decrease.

[Multivariable optimization] What does the learning rate control?

The step size of each parameter update.

[Multivariable optimization] What can happen if the learning rate is too large?

Updates can overshoot the minimum, oscillate, or diverge.

[Multivariable optimization] What can happen if the learning rate is very small?

Progress toward the minimum can be very slow.

[Multivariable optimization] If w=2, grad f(w)=3, and eta=0.1, what is the next gradient-descent value?

2-0.1*3 = 1.7.

[Multivariable optimization] How do an analytical solution and gradient descent differ?

An analytical approach solves the stationary equations directly; gradient descent approaches a solution through repeated numerical updates.

[Regularization] What regression behavior illustrates overfitting in the slides?

As models become more flexible and weights grow, training error falls while test error can rise.

[Regularization] What is regularization?

Adding a penalty or constraint that discourages overly complex parameter settings and can improve generalization.

[Regularization] What is the ridge objective using the half-SSE convention shown in the slides?

J_ridge(w) = 0.5*sum_i(y_i-w^T*x_i)^2 + alpha*||w||_2^2, with alpha >= 0.

[Regularization] What is the ridge penalty ||w||_2^2?

The sum of squared penalized coefficients: sum_j w_j^2.

[Regularization] What is the lasso objective using the half-SSE convention shown in the slides?

J_lasso(w) = 0.5*sum_i(y_i-w^T*x_i)^2 + alpha*||w||_1, with alpha >= 0.

[Regularization] What is the lasso penalty ||w||_1?

The sum of absolute values of the penalized coefficients: sum_j |w_j|.

[Regularization] What does alpha control in ridge and lasso?

The relative strength of the coefficient penalty compared with fitting the training targets.

[Regularization] What happens at alpha=0 in the regularized regression objectives?

The penalty disappears, leaving ordinary least squares.

[Regularization] What generally happens as regularization strength increases?

Coefficients are pushed more strongly toward smaller values; too much regularization can underfit.

[Regularization] What is a typical difference between ridge and lasso coefficients?

Ridge shrinks coefficients continuously and usually retains them; lasso can set some coefficients exactly to zero.

[Regularization] Why can lasso act as feature selection?

A zero coefficient removes that feature's direct contribution to the linear prediction.

[Regularization] What geometry represents a two-coefficient L2 constraint?

A disk or circular boundary centered at the origin.

[Regularization] What geometry represents a two-coefficient L1 constraint?

A diamond with corners on the coefficient axes.

[Regularization] Why do the corners of the L1 constraint help explain sparse lasso solutions?

A loss contour can first meet a corner where one coefficient is exactly zero.

[Regularization] Why can regularization improve test error while worsening training fit?

It sacrifices some fit to the observed sample to reduce unstable, sample-specific parameter choices.

[Regularization] How does regularization relate to bias and variance?

Stronger regularization often increases bias while reducing variance; the useful strength balances these effects.

[Regularization] How should alpha be chosen?

Use validation or cross-validation on development data rather than optimizing final test-set performance.

[Regularization] Why should coefficient scale be considered before applying L1 or L2 penalties?

Rescaling a feature changes the coefficient needed for the same prediction and therefore changes its penalty; comparable feature scales often make penalties more meaningful.

[Regularization] For w=(3,-4), what are ||w||_1 and ||w||_2^2?

||w||_1=|3|+|-4|=7; ||w||_2^2=3^2+(-4)^2=25.

[Regularization] If alpha=0.1 and w=(3,-4), what are the L1 and squared-L2 penalty contributions?

L1 contribution = 0.1*7 = 0.7; squared-L2 contribution = 0.1*25 = 2.5.

[Nonlinear features] Why might a straight-line regression model underfit a data set?

The relationship between inputs and target may be nonlinear in the original features.

[Nonlinear features] How can linear regression fit a quadratic relationship?

Transform x into features z_1=x and z_2=x^2, then fit y_hat=w_0+w_1*z_1+w_2*z_2.

[Nonlinear features] Why is polynomial regression still linear regression in this formulation?

The prediction is linear in the unknown coefficients, even though the input features are nonlinear functions of x.

[Nonlinear features] What does linear mean in a linear model with transformed features?

Linear in the parameters being learned, not necessarily a straight line as a function of the original input.

[Nonlinear features] What feature vector represents a one-variable quadratic model with an intercept?

phi(x)=(1,x,x^2)^T.

[Nonlinear features] For y_hat=1+2*x+3*x^2, what is the prediction at x=2?

1+2*2+3*4 = 17.

[Nonlinear features] What difficulty remains after allowing nonlinear feature transformations?

Choosing suitable transformations or complexity for the actual relationship, without overfitting.

[Nonlinear features] Can nonlinear feature transformations automatically guarantee good generalization?

No. Flexible transformed models still require validation and may need regularization.

[Regression trees] How does a regression tree partition input space?

It recursively splits on one feature and a threshold, creating axis-aligned regions.