STAT 3400 - Regression

Exam 1

RMSE

Flexible Model

adapts to changes better, non-parametric, needs more data, variance

Inflexible Model

less adaptable, parametric, needs less data, bias

Bias

error from approximating a real world system with a model, worse in inflexible models

Variance

error from fitting the model using training data that only represents a small subset of the population, too much leads to overfitting, worse in flexible models

CER

(sum of I(yi =/= yi_hat)) / n

RSS

E^T(E) = (y-XB)^T(y-XB), least squares seeks to minimize RSS, total squared error of the training model

Minimized RSS

B_hat = (X^TX)^-1(X^Ty)

RSE

sqrt(RSS/(n-2)), measure of how much the training data differs from the line of best fit

Overfitting

RSE is small but RMSE is bad, model too specific to training data to be useful

Multiple R²

1-RSS/TSS, the proportion of variance explained by the model, single linear regression

Adjusted R²

1-(RSS/TSS)((n-1)/(n-1-p)), the proportion of variance explained by the model, multiple linear regression

TSS

sum of (yi - y_bar)², sum of the squared differences between the training data and the mean (null)

MSE

RSE² = RSS/(n-1-p), estimate of the variance of the model

MSG

(TSS - RSS)/p, compares the performance of the null and fitted models, larger MSG values indicate that the model outperforms the null

F

MSG/MSE, used in ANOVA tests, values near 1 indicate the model does not improve the null, values greater than 1 indicate that it did, if the ratio is big, the model is more accurate, the larger the ratio the more likely one of the predictors is useful.

Explain one of the uses of an unsupervised statistical learning model.

To find the association between predictor variables. For example, cluster analysis groups data into groups to determine association. (add some more stuff for the exam)

Multiple linear regression with 5 predictors, test set of 250 observations. If yi is the actual response for the i-th observation and yi-hat is the predicted response, which equation models RMSE

• sqrt ( sum from 1 to 250 of (yi–yi-hat)^2/250 )

Model to predict fish weight with RMSE of 0.75 lbs. In order to determine benefit we should compare 0.75 lbs to the error obtained from the model that always predicts _____.

• The mean fish weight (null model)

Explain the benefit of using RMSE rather than MAE (mean absolute error)

• Both have the correct units. Squared error penalizes really big errors more than absolute error, and really tries to avoid models with massive errors. RMSE doesn’t have the kink where MAE isn’t differentiable.

Which of the following is NOT a component of testing error? Bias error, variance error, flexibility error, irreducible error.

• Flexibility error (does not exist)

Tow numerical variables and observe a strong, increasing, nonlinear relationship. If we train a simple linear regression model we can expect the ______ component of testing error to be relatively large. Bias or variance.

• Bias

Two variables with a strong, decreasing, linear relationship. Train and test a 5th-order polynomial function and discover we have overfit the data. The ___ error was small but the ____ error was large.

• Training, testing

Explain the bias-variance trade-off inherent in the flexibility of a statistical model.

• Three types of error: bias, variance, and irreducible. If a model is too inflexible there will be more bias. If a model is too flexible there will be more variance. If bias decreases, variance increases. One has to try to balance the two types of error.

Training set of 1000 emails and a testing set of 100. Binary response spam/not spam. 600 labeled as spam in training, 55 labeled as spam in testing. What is the upper bound on the CER against a testing set for any new model to be useful? 40% 45% 55% 60%

• 45% - null model accuracy

After training a multiple logistic model with 6 predictors we want to assess accuracy against a test set of 125 observations. Which is CER

• Sum from 1 to 125 ( I(yi =/= yi-hat) /125 )

After estimating a simple logistic model you obtain a logarithmic function. For a given input the output of the function is ____.

• P(y=1)

Explain how to commit CER of the given dataset

• Decide on a threshold of success/failure, convert probabilities (p-hats) to y-hat values, compare to the true y response, determine what proportion of the time the model is wrong, and take the mean.

A linear regression model is considered simple if it has only one ______.

• Predictor

Suppose yi is the true response and yi-hat is the predicted based on the model B0+ B1xi. Assuming there are n observations, the least squares algorithm seeks to minimize which quantity (what is RSS)?

• Sum from i=1 to n ( (yi - yi-hat)^2 )

Estimate a simple linear regression. Training set with 250 observations and you do least squares method using the matrix notation below. What are the dimensions of B-hat?

• 2 rows by 1 column (B0 and B1 column vector)

Describe the shape, direction, and strength of association between these two variables

• Linear, negative/decreasing, strong/moderate

Fit a linear regression and test the hypothesis H0: B1=0, HA: B1=/=0 based on a significance level of alpha = 0.1, What is the interpretation of alpha?

• We accept a 10% chance of finding an association between the predictor and response when in fact there is none

Inferential analysis of a simple linear regression mode we might constrict the confidence interval below B1 +- z_a/2 * SE_B1. In the formula z_a/2 represents the number of standard errors on the ______ distribution.

• Standard normal

Null distribution with a test statistic inside the 0.05 significance level bounds. What is the appropriate conclusion?

• No evidence to suggest that B1 =/= 0, fails to reject the null distribution, not beyond the rejection threshold

Given a sampling distribution for birth weight (x) and father’s age (y) with a 95% confidence interval

• 95% confident that the true slope parameter is between [] and []. 95% confident that for every one-year increase in a father's age, the baby’s birth rate will increase by between [] and []. Say something about zero being in the interval, possible that there is no relationship.

Summary output with RSS 11.79. What does RSS mean

• On average, the margins of victory in the training data differ from the model by about 11.79 points

The RSS measures error when using the fitted linear regression mode. The TSS measures the error when using the ______.

• Mean response value (the null)

Explain how R^2 is related to the variance in the response variable

• R^2 is the proportion of variance eliminated/explained in the fitted model as compared to the null model

Using 300 observations to train an MLR model to predict penguin body weight using 3 predictors. What are the values of n and p?

• n=300, p=3

What formula would compute RSS? Given RSE is 11.79 and there are 999 observations and 4 predictors

• RSE = sqrt(RSS/n-(1+p)), so RSS = 11.79^2*995

Which of the following is false regarding adjusted (multiple) R^2?

R^2 represents the proportion of variability in the response explained by its relationship with the predictors.

R^2 cannot be less than adjusted R^2.

R^2 will always decrease when we add predictors to the model.

• R^2 will always decrease when we add predictors to the model - always increases

Estimate two different LR models using a training set with 400 observations. First has 3 predictors, second has 14. Given Adjusted R^2 = 1 - (RSS/TSS)((n-1)/(n-1-p))

• TSS and number of observations are the same. p and RSS will be different.

Conduct an ANOVA with 3 predictor variables. One type of variance is MSG, the formula of which is (TSS-RSS)/p. The TSS measures the error in a model that employs ______ predictor variables

• 0 (null model)

Using ANOVA, get the graph. F-statistic for the test statistic is to the left of the F-distribution for the null distribution (threshold). Meaning

• Not rejecting the null, so none of the predictor variables are associated with the response

To explain the margin of victory perform ANOVA. What are the hypotheses associated?

•  H_0: B1=B2=B3=0, H_A: at least one Bj=/=0

Explain the test statistic on the F distribution (F= MSG/MSE)

• MSE is the amount of error in the fitted model (want it to be small). MSG is how much error is in the model relative to the null (want it to be big). If the ratio is big, the model is more accurate. The larger the ratio the more likely one of the predictors is useful.