Prelim 3

When is LSE bad?

When p>n, due to large variance

How does Best Subset Selection work?

For every k, find the best model with k predictors, then compare them using a CV/AIC/BIC criterion.

How does Forward Subset Work

Add 1 predictor at a time, then compare the models using CV

How does backward selection work?

Remove 1 predictor at a time, starting from the full model, and compare the models using CV.

How many models are compared in

1. Best

2. Backward

3. Forward

Selection

1. 2^p

2. 1+p(p+1)/2

3. 1+p(p+1)/2

When can forward subset be used

When p>n.

What is the formula for Mallows Cp

1/n * (RSS + 2d(variance)) where d is the number of parameters

Whats the formula for AIC

-2logL + 2d where L is the maximum likelihood & d is number of parameters

What is the formula for BIC

BIC = -2logL + logn * d where L is the maximum likelihood, n is the sample size, and d is the number of parameters.

What is the formula for AdjR²

1 - (RSS/(n-d-1))/(TSS/(n-1))

What is shrinkage

Trying to limit the coefficients to reduce variance

What is Ridge Regression?

estimates the parameters using values that minimize RSS + delta * (sum of coefficients)

What does Lasso minimize

Rss + delta * (sum of absolute value of coefficiente

What ability does lasso have that ridge doesnt

Lasso has the ability to set some coefficients to exactly zero, effectively performing variable selection, while Ridge regression shrinks coefficients but does not eliminate any.

When is lasso expected to perform better

Smaller p

With both Lasso & ridge what happesn t variance and bias

variance drops bias increases

For dimesnion reduction from p to m, what is Zm equal to

Zm = the summation of the first m principal components of Z.

How can dimension reduction outperform OLS Regression

With good choice of constants.

How does variance work with the Zs?

The first component has the most variance, then it keeps decreasing.

What is true about the multicollinearity of deimsion reduction

Dimension reduction reduces multicollinearity by transforming correlated features into a set of uncorrelated principal components, improving model stability and interpretability.

What type of structure is used in Polynomial Regression

Global Structure

How do step functions work?

Step functions partition the input space into intervals. Each interval has an indicator function.

C0(x) = I(X<C1) C1(x) = I(c1<=x<c2).. to the end

What does the coefficeinet Bj represent in step functions

The additional value to the baseline being in that interval gets you.

When si step functions good?

When there are natural breakpoints.

What are basis functions?

Y = B0 + B1(b1(x)) + … Bk(bk(x)) + Ei where the b’s are fucntions

More knots means more or elss flexible

more

what is the biggest constraint needed on the splines?

continuity

What is a degree d spline

a piecewise degree d polynomial with continuty in derivatives d-1 at each knot

In a degree d spline, what do the B’s represent

B0 = intercept

B1 = slope of first segent

B2,B3.. change in slope at it’s knot.

What is a natural spline?

Beyonf the boundary knots, eveyrhting is linear

How to determine number of knots and from there choose where they go?

Use CV on equal spaced knots.

What are smoothing splines?

minimzing RSS + delta * integral of 2nd derivative sqaured, using Cv to find delta.

What is Local Regression

compute the fit at a target point, only using nearby observations

What is s in local regression

s = k/n, the fraction of training points whose xi are closest to x0

For a given Ki what does the further point weight have, and closest point?

furthest = 0

closest = highest

How do you fit Local Regression

minimze sum of (RSS*weight) for all i

What is the problem with Local Regression

Curse of dimensionality makes it prefrom bad with high p

What are GAMS

Generalized Additive Models - Y = B0 + B1f1(x) + B2f2(x) + … + Bnfn(x), where each fi is a smooth function.

what are the ebenfits of gams

1. allows non-linear models

2. potentially more accurate

3. you can see the affect of all variables

4. solves dimensionality curse

What is the drawback of GAMS?

It doesn’t account for interaction of variables.