10 MARS regularization

What does MARS stand for, and what is its central idea?

Multivariate Adaptive Regression Splines. It combines data-adapted hinge basis functions, possibly with interactions, to describe numerical relationships that change across regions.

What are right-facing and left-facing hinge functions?

Right hinge: (x-c)_+ = max(0,x-c). Left hinge: (c-x)_+ = max(0,c-x). c is a knot where the piecewise behavior changes.

When is a right-facing hinge active, and when is a left-facing hinge active?

max(0,x-c) is positive for x > c; max(0,c-x) is positive for x < c. Both are zero at the knot.

What is MARS’s basis-function expansion?

fhat(x) = a0 + ∑_m a_m*B_m(x). Each B_m can be a hinge or a product of hinge factors; coefficients combine the selected basis terms.

What does a product of hinge functions mean?

It creates a region-dependent interaction. The product contributes only where all factors are nonzero; one predictor’s effect can depend on the other predictor’s region.

Example: B(x,z) = max(0,x-5)*max(0,3-z). Find B(7,1) and B(4,1).

B(7,1) = 2*2 = 4. B(4,1) = 0*2 = 0. The interaction is inactive when the first hinge is zero.

Example: yhat = 5 + 2(x-10)_+ - 3(10-x)_+ + 0.5(x-10)_+(z-2)_+. Find yhat at x=12, z=4.

The hinge values are 2, 0 and 2. Prediction = 5 + 2*2 - 3*0 + 0.5*2*2 = 11.

What does the degree argument control in the course’s earth MARS implementation?

The maximum permitted interaction degree, not the degree of one global polynomial. The fitted, pruned model need not retain a term of the highest allowed degree.

Can a degree-1 MARS model be nonlinear in the raw predictors?

Yes. Additive univariate hinges can change slopes at knots even without interactions.

How does a typical MARS search control complexity?

It grows useful candidate basis terms and then prunes. The search does not guarantee exhaustive evaluation of every possible model or a global optimum.

What does nonparametric mean in the MARS discussion?

The model does not impose one fixed simple functional shape in advance. It does not mean parameter-free, assumption-free or automatically best.

What is the schematic MARS GCV formula?

GCV = (RSS/n)/[1-C(M)/n]². RSS is residual sum of squares and C(M) an effective-complexity quantity under the fitting convention. Smaller model GCV is preferred.

Example: n=100. Model A has RSS=100, C=10; B has RSS=90, C=20. Which has lower GCV?

A: 1/0.9² ≈ 1.235. B: 0.9/0.8² ≈ 1.406. A has the lower penalized score despite B’s smaller training RSS.

How does model GCV differ from GCV-based variable importance?

A smaller GCV is preferred when comparing models. A larger reported relative importance can mean greater predictor contribution under that ranking criterion.

Do normalized MARS importance values necessarily sum to 100 or represent causal effects?

No. A 0-100 scale need not be percentage shares summing to 100. Importance depends on the fitted model and correlated substitutes, and is not causal by itself.

Does earth evimp automatically refit an independent model after deleting each original predictor?

Not as a general description. Its importance summaries use implementation-specific model-subset, GCV and RSS contributions; the leave-one-worker analogy is only an intuition.

What is the ridge-regression penalized objective in the guide?

Minimize ∑_i (y_i-β0-x_i^T β)² + λ*∑_j β_j². The usual intercept is excluded from the penalty; scaling conventions can change the numerical meaning of λ.

What is the lasso-regression penalized objective in the guide?

Minimize ∑_i (y_i-β0-x_i^T β)² + λ*∑_j |β_j|. The L1 penalty can set fitted coefficients exactly to zero.

How do ridge and lasso differ in coefficient behavior?

Ridge uses an L2 squared-coefficient penalty and usually keeps shrunken nonzero coefficients. Lasso uses L1 and can produce exact zeros, performing selection.

What happens as the penalty strength lambda increases?

Stronger shrinkage is favored. It may reduce variance but can eventually underfit; held-out error need not improve monotonically.

Why can deliberately biased regression coefficients improve prediction?

Shrinkage introduces bias but can reduce instability/variance enough to improve prediction on new observations. This is different from sampling bias.

What does alpha control in glmnet’s elastic-net convention?

It mixes penalty types: alpha=0 gives ridge and alpha=1 gives lasso. Lambda controls penalty strength. This alpha is unrelated to smoothing alpha or test significance.

Why is scaling relevant to coefficient penalties?

Changing predictor units changes the coefficient magnitude needed for the same prediction, and therefore changes the penalty unless scaling is handled deliberately.

Does ridge or lasso on raw linear predictors automatically discover MARS-like hinges?

No. Penalizing coefficients and constructing nonlinear hinge bases are different modeling operations. Any needed feature transformation must be specified or learned appropriately.