3.0 Statistical Learning Prerequisites

Section 3.0 of Exam MAS-II

Define:

• Response

• Explanatory

• Count

• Continuous

• Nominal

• Ordinal

• A variable of primary interest

• A variable used to study the response variable

• A quantitative variable usually valid on non-negative integers

• A real-valued quantitative variable

• A categorical/qualitative variable having categories without a meaningful or logical order

• A categorical/qualitative variable having categories with a meaningful or logical order

Say what it is:

• y, Y

• x, X

• Subscript i

• n

• Subscript j

• p

• f(x)

• ε

• ŷ, Ŷ, f̂(x)

• Response variable

• Explanatory variable

• Index for observations

• Number of observations

• Index for variables except response

• Number of variables except response

• E[Y]

• Error term

• Estimate/Estimator of f(x)

Statistical Learning:

• Supervised

• Unsupervised

• Has response variable

• No response variable

Statistical Learning:

• Regression

• Classification

• Quantitative response variable

• Categorical response variable

Statistical Learning:

• Parametric

• Non-Parametric

• Flexibility

• Functional form of f̂ specified

• Functional form of f̂ not specified

• f̂’s ability to follow the data

Statistical Learning:

• Training Data

• Test Data

• Observations used to train or obtain f̂

• Observations not used to train or obtain f̂

Regression Equation

Regression Model Accuracy Equation

Estimate for Regression Test MSE Equation

For fixed inputs x_1, ..., x_p, the test MSE can be written as

Bias-Variance Trade-Off:

• What happens to test and training MSE with increasing flexibility

• What low and high flexibility mean for bias and variance

• As flexibility increases, the training MSE decreases, but the test MSE follows a u-shaped pattern.

• Low flexibility leads to a method with low variance and high bias; high flexibility leads to a method with high variance and low bias.

Classification Equation

Classification Model Accuracy Equation

Estimate for Classification Test MSE Equation

Bayes Error Rate

Basically find the highest joint probability at each x value and take the complement of the sum

Cross-Validation: What does it do and what are the types?

• Estimates the test MSE with available data.

• Types:

  • Validation set approach

  • -fold cross-validation approach

  • LOOCV approach

Cross Validation: Issues with Validation set approach, and rank each method with respect to bias and variance

• The validation set approach has unstable results and will tend to overestimate the test MSE. The two other approaches mitigate these issues.

• With respect to bias, the validation set error has the most, followed by the -fold CV error, then the LOOCV error.

• With respect to variance, the LOOCV error has the most, followed by the -fold CV error, then the validation set error.