Lab 2: R^2, F statistic, ANOVA

What is the F table for?

The F table is used to determine whether or not our F value is small, large, or medium.

This is because the F value itself has nothing to be compared against, and so the F table is our reference table.

What does the F table give us?

The F table gives us F critical, which tells us the minimum F value needed to reject the null hypothesis. If the table says F critical is: 10.128, then F > 10.128 in order for us to reject the null hypothesis. Otherwise, if F <= 10.128, then we fail to reject the null hypothesis.

How can we read an F table?

The values on the side of the F table are the Error DF, and the values at the top are the Model Df.

What do we need to get the critical value on an F table?

a, Model DF, and Error DF.

What does SST = 100 represent? What does SSR = 80 represent?

SST = 100 represents the fact that there are 100 units of total variation in the data. If SSR = 80, it means the model explained 80 of them.

What does an R² value of 0.75 mean?

It means that 75% of the variation was explained by the model, only 25% was left unexplained.

Given:

SSR = 45

SST = 60

Calculate R² and interpret the result.

R² = SSR/SST

= 45/60 = 0.75 = 75% of the variation was explained by the model, and only 25% was left unexplained. This means that the model explains most of the variation in the target variable which suggests a strong relationship/correlation.

What does RMSE tell us?

It tells us on average, how far our predictions are from the actual values.

For example, if we’re talking about housing prices, and our RMSE is 2,000, this means on average, our predictions are off by $2,000.

Why isn’t SSE good enough? Why do we still need RMSE?

SSE isn’t good enough because it adds all the squared errors, and the problem is that SSE gets larger as we get more observations. So SSE alone can be difficult to interpret.

What does RMSE stand for?

Root Mean Squared Error

How can we interpret a correlation heatmap?

Look at the Correlation values shown on the side. If the correlation value is +1, this means it is a perfect positive correlation, if the correlation value is -1, this means it is a perfect negative correlation. 0 means no correlation.

Which shows positive, negative, and no correlation on this scatterplot?

The first is positive, the second is negative, the third is no correlation.

You can tell how strong the correlation is based on how tightly packed/grouped the dots are.

What does correlation measure?

It measures whether two variables tend to change together. For example, as height increases weight increases, or as study time decreases, score decreases.

In a positive correlation, both x and y increase.

In a negative correlation, as x increases, y decreases.

What does an OLS Summary contain information about?

An OLS Summary contains a lot of information, but all we’re really interested in are the const and coefficient values.

Do we always only have one predictor variable?

Not at all, only in very simple cases do we only have one predictor variable like in the y=mx+b equation, but real models often contain multiple predictor variables, like in the equation: y = b + c1×1 + c2×2 + c3×3…

What do each of these values in our OLS Summary represent?

const 2.7939

volatile acidity -1.2395

chlorides -1.7142

sulfates 0.8517

alcohol 0.2974

The const value represents b in our equation: y = b + c1×1 + c2×2 + c3×3…

The other values all represent the constants c1, c2, etc.

Write the OLS equation for this, which represents factors that impact the quality of wine.

const 2.7939

volatile acidity -1.2395

chlorides -1.7142

sulfates 0.8517

alcohol 0.2974

y = b + c1×1 + c2×2 + c3×3…

y = 2.7939 -1.2395(volatile acidity) -1.7142(chlorides) + 0.8517(sulfates) + 0.2974(alcohol)

This represents factors that impact the quality of wine.

Here, what does volatile acidity = -1.2395 mean?

const 2.7939

volatile acidity -1.2395

chlorides -1.7142

sulfates 0.8517

alcohol 0.2974

-1.2395 indicates a negative correlation. It means that as the volatile acidity increases, the predicted y value, or in other words, the wine quality, tends to decrease.

What is imputation?

Imputation is the process of replacing missing values with estimated values.

Why is missing data a problem?

 

Age	Weight
20	150
25	170
?	180
30	200

Missing data is a problem because if Age is missing, many models cannot directly use that row. So we have either one of two choices:

• Delete the row

• Or Impute the missing value in the row

However, we generally prefer imputing the row, because deleting a bunch of rows can lead to a loss of information. If a table only has 100 rows, and you delete 60 of them because of missing values, suddenly, you only have 40 left, which is much fewer than the original dataset.

How can we use mean imputation to impute the following value:

 

Age	Weight
20	150
30	170
40	180
?	200

20+30+40 = 90/3 = 30

Missing value = 30

How can we use median imputation to impute the following value:

Income	Age
20	25
25	40
30	33
1000	27
?	54

Recall: Median is the middle value in an ordered list.

Here, they’re already sorted (lucky for us), so all we have to do is take the middle value of:

20, 25, 30, 1000

which is: 25+30/2 = 27.5

What are the advantages and disadvantages of mean imputation?

Mean imputation is good because its very fast, very easy, and works well in simple situations.

The problem is that it reduces variation in the data, it can introduce bias, and it may not reflect the true value.

What are the advantages of median imputation?

Mean imputation is not prone to outliers.

How can we conduct mode imputation on the following table?

 

Color	Age
Red	25
Blue	40
Red	33
Red	27
?	54

The most frequent value is Red, therefore, using mode imputation, we can say that the missing value should be Red.

When is Mode imputation useful?

Mode imputation is useful for categorical data.

How can we conduct regression imputation on the following table if we know that the regression equation is: weight = height -100?

 

Height	Weight
160	60
170	70
180	?

weight = 180-100 = 80

Missing value = 80

Why is regression imputation attractive? Why might it be unattractive?

It is attractive because unlike mean imputation, it preserves the relationship between the variables. The problem is that its more complicated, so as soon as more variables go missing, things tend to get messy.

How can we do nearest neighbor imputation here?

 

Height	Weight
160	60
170	70
180	?

Nearest neighbor imputation just estimates a value based on similar neighboring values. Here, we can guess that weight will be 65.

What are the advantages/disadvantages of nearest neighbor?

Nearest neighbor preserves the realistic combination of data values, but it may be difficult to use because it requires finding appropriate neighbors, and the computation becomes difficult for large datasets.

Why do we need dummy variables?

We need dummy variables in order to be able to convert categorical data expressed in words into numerical values so that we can perform regression on them.

What is a dummy variable?

A variable that uses either a 0 or a 1 to indicate whether a category is present.

How come we don’t just use numbers like 1 and 2 to represent categories?

For example:

Male = 1

Female = 2?

Because the model may incorrectly assume that 1 and 2 carry significance, and that 2 is twice as large as 1, or that 2-1 is meaningful.

What is on-hot encoding?

Automatically creating dummy variables for every category.

It is called one-hot because each row only contains a single 1.

For example:  Instead of: Color: Red Blue Green Red, we have:

Color_Red	Color_Blue	Color_Green
1	0	0
0	1	0
0	0	1
1	0	0

Create On-Hot columns for:

 

Row	Fruit
1	Apple
2	Banana
3	Orange
4	Apple

Start out with an empty table:

 

Apple	Banana	Orange

Apple = corresponds to row 1, so we can place a 1 in the first row of the column belonging to Apple.

Banana = corresponds to row 2, so we place a 1 in the second row of the column belonging to Banana.

Orange = corresponds to row 3, so we place a 1 in the third row of the column belonging to Orange.

Apple = corresponds to row 4, corresponds to row 4, so we place a second 1 in the fourth row of the column belonging to Apple.

Then:

 

Apple	Banana	Orange
1	0	0
0	1	0
0	0	1
1	0	0

What is binning?

Binning is when we group numerical data into ranges/groups. So instead of using individual values like 18,19,20, so on, we use ranges.

What are some of the benefits of binning, and what kind of problems does it solve?

Binning also solves the problem where the regression model treats every value as distinct, and will try to find differences between all of these ages: 18 ≠ 19 ≠ 20..etc. This is a good thing because usually, the differences between these values aren’t important. For example, ages 21, 22, 23, 24, might all behave pretty similarly in the real world, so it might make more sense to treat them like a group of 18-25.

Benefits:

• It simplifies the data, which may make it easier to interpret and analyze.

• It can be used to identify patterns that may otherwise not be immediately obvious when looking at individual values like 18, 19, 20, 21, but when grouped, becomes much clearer: 18-25, 26-35, etc. The model might struggle to find a relationship when the data is not grouped

• It is also useful for ordinal data.

Why do we need cross fold validation?

We need cross fold validation because if we have 100 samples, and we perform only a single test split, we might get 80 for training and 20 for testing.

But those 20 values for testing might make the model look unusually bad or good. On the first split, we might get an RMSE value of 0.75, then on the second run, we might get an RMSE of 0.55.

The evaluation might change because different data points are being used for testing.

So a single train/test split may not necessarily be a reliable picture of a model’s performance.

What is cross fold validation?

It’s when we repeatedly train and test a model using different train/test splits so that all data gets a chance to be used for testing. So instead of one split, we perform multiple splits. This allows us to observe a model’s performance over multiple train/test splits more accurately, which produces a more reliable estimate of a model’s performance and stability.

What is a fold?

A fold is simply a section of the data.

If we have 12 observations, we can divide the data up into 3 folds, each containing 4 values.

For example:

If we have the data: 1 2 3 4 5 6 and we want 3 folds:

Run 1 might use 1,2 for testing and 3,4,5,6 for training. Avg RMSE = 0.66

Run 2 might use 3,4 for testing and 1,2,5,6 for training. Avg RMSE = 0.64

Run 3 might use 5,6 for testing, and 1,2,3,4 for training. Avg RMSE = 0.66

The idea is that every sample becomes part of the set eventually.

Why is cross fold validation better than not using cross fold validation?

It’s better because with a single train/test split, some samples never get tested. But with cross fold validation, all samples get tested. This tends to produce a more reliable estimate of the mode’s performance and stability.

What becomes our overall measure of performance with 3 cross fold validation?

The average RMSE between all 3 splits:

0.66+0.64+0.66/3 = 0.653

How do we know if a model is stable?

We can tell if a model is stable based on the RMSE values over the 3 or more splits.

For example, if the RMSE values appear to be very similar:
0.66, 0.64, 0.66, then it’s likely very stable.

If the RMSE values appear to be very different, like 0.10, 0.20, 0.50, then this is considered to be an unstable model.

A university wants to predict student GPA.

The dataset contains: study_hours, attendance, assignments_completed, gpa

Create a linear regression model using: study_hours, attendance, assignments_completed as predictors.

Use: gpa as the target variable.

Make sure to:

1. Implement 4-fold cross validation.

2. Calculate RMSE for each fold.

3. Store each RMSE value in a list.

4. Calculate the average RMSE across all folds.

5. Print all RMSE values and the final average RMSE.

*Use the cheat sheet

The only thing that changes is:

X = dataset[[“study_hours”, “attendance”, “assignments_completed]]

y = dataset[[“gpa”]]

kfold = KFold(n_splits = 4, shuffle = True)

rmses = [ ]

….stays the same

print(rmses) // in addition to the other print statement

What is a classification model?

A classification model is a model which predicts categories not numbers.

For example:

Admitted/Not Admitted

Spam/Not Spam

The model predicts either a 0 or a 1 instead of a decimal value like 0.756 in regression.

How do we know if a classification model is good?

We compare actual results with predicted results using a confusion matrix. The confusion matrix tells us how many predictions were correct, how many predictions were wrong, and what type of mistakes occurred.

Let’s say we have the following data collected, and the predicted values by the model. How can we create a confusion matrix based on this?

 

Actual	Predicted
1	1
0	1
1	0
1	1
0	1

Actual	Predicted	Thought Process
1	1	The model predicted positive, actual was positive: CORRECT, this is a TRUE POSITIVE: TP The model predicted positive, actual was negative:WRONG, this is a FALSE POSITIVE: FP The model predicted negative, actual was positive: WRONG, this is a FALSE NEGATIVE: FN The model predicted positive, actual was positive: CORRECT, this is a TRUE POSITIVE: TP The model predicted positive, actual was negative: WRONG, this is a FALSE POSITIVE: FP Count Everything: True Positives (TP): 2 False Positives (FP): 2 False Negatives (FN): 1 True Negatives (TN): 0
0	1
1	0
1	1
0	1

Then fill in the confusion matrix as follows:

	Predicted
Actual 0	TN = 0	FP = 2
1	FN = 1	TP = 2
	0	1

What is Accuracy, Precision and Recall?

Accuracy measures the percentage of predictions that were correct.

Recall measures how many of the actual positive cases the model identified.

Precision measures how often the model is correct when it predicts a positive result.

Calculate Accuracy, Precision, and Recall for the following confusion matrix:

	Predicted
Actual 0	TN = 0	FP = 2
1	FN = 1	TP = 2
	0	1

Correct: 2/5 (We can also tell by the True Positives)

Accuracy: correct predictions/total predictions OR (TP+TN) / (TP+TN+FP+FN) = 2/5

Precision: TP/TP+FP = 5/7 = 0.714 or 71.4%.

Recall: TP/TP+FN = 5/5+0 = 5/5 = 1 or 100%, this means the model found 100% of all positive cases.

What is Scaling?

Scaling is when we transform features so that they are roughly the same numerical scales/magnitude.

This is necessary because sometimes we have 2 predictor variables where one contains much larger numerical values than the other, for example: Salary and Age. Sometimes machine learning algorithms will place too much importance on the second column, simply because its values are bigger. So scaling fixes this problem altogether.

What does MinMaxScaler, RobustScaler, and StandardScaler do?

MinMaxScaler transforms all values so they lie between 0 and 1.

Standard Scaler transforms data so mean = 0 and standard deviation = 1.

Robust scaler uses the median and the IQR to scale values. Because it ignores extreme ends of the data (since it doesn’t use the min/max to calculate), it is less influenced by outliers.