F-Tests and Model Comparison

0.0(0)
Studied by 0 people
call kaiCall Kai
learnLearn
examPractice Test
spaced repetitionSpaced Repetition
heart puzzleMatch
flashcardsFlashcards
GameKnowt Play
Card Sorting

1/30

flashcard set

Earn XP

Description and Tags

Use of F and incremental F-tests, running and interpreting F-tests in R, using model comparisons to test different types of questions, difference between nested and non-nested models (and the appropriate statistics to use for comparison in each case)

31 Terms

1
New cards

What are some indications that the model might be significant (combination of xs significantly improve prediction of y, compared to not having any predictors)? What is the drawback of these indicators?

  • Slopes for individual predictors/betas associated with significant p-values

  • High R²


BUT don’t directly show model significance

2
New cards

What can be used to directly test the significance of the model as a whole?

F-test (use in combo with betas + R²)

3
New cards

What is an F-test?

Involves testing the stats significance of a test-statistic → F-ratio

4
New cards

What is the F-ratio?

  • Tests H0 that all the regression slopes in a model are 0

  • Ratio of explained to unexplained variance → how much of the variance is explained by the model (per parameter) VS how much of the variation is unexplained (per remaining df)

  • Compares 2 models → specified VS an intercept-only model (no IVs where the model’s predictions = mean of the outcome)

    • Predictors tells us nothing about our outcome → explain no variance

    • More variance our predictors explain compared to residual variance = bigger F-ratio = better fitting models

  • Always positive


5
New cards

What is the F-ratio compared to? What does this obtain?

F-distribution → p-value

6
New cards

What is the formula for the F-ratio?

  • Sums of squares of model = how much variability is explained by our model by having these lines/planes that explain the variance

  • Sum of squares residual = how much variability there is that's random/uncontrolled (how much each data point differs from the predicted value based on the model)

  • Mean squares (MS) = sums of squares calculations divided by associated df


<ul><li><p>Sums of squares of model = how much variability is explained by our model by having these lines/planes that explain the variance </p></li><li><p>Sum of squares residual = how much variability there is that's random/uncontrolled (how much each data point differs from the predicted value based on the model) </p></li><li><p>Mean squares (MS) = sums of squares calculations divided by associated df</p></li></ul><p></p>
7
New cards

How can F-ratio be interpreted?

  • High value (>1) = more variant being explained by the model than by the random variation in the residual

    • Used to show that model is explaining variability

  • Low value (<1) = more random variation than is being explained by the model


<ul><li><p>High value (&gt;1) = more variant being explained by the model than by the random variation in the residual</p><ul><li><p>Used to show that model is explaining variability </p></li></ul></li><li><p>Low value (&lt;1) = more random variation than is being explained by the model </p></li></ul><p></p>
8
New cards

What are degrees of freedom?

  • The number of independent values associated with the different calculations

  • How many values in the calculation can vary, if we keep the outcome of the calculation fixed

  • Typically linked to:

    • Amount of data (e.g. n)

    • Number of things you need to calculate/estimate based on that data (e.g. betas)


<ul><li><p>The number of independent values associated with the different calculations</p></li><li><p>How many values in the calculation can vary, if we keep the outcome of the calculation fixed</p></li><li><p>Typically linked to:</p><ul><li><p>Amount of data (e.g. n)</p></li><li><p>Number of things you need to calculate/estimate based on that data (e.g. betas)</p></li></ul></li></ul><p></p>
9
New cards

How is the significance of F tested?

  1. Select alpha level

  2. Identify proper null F-distribution + calculate critical value of F associated with chosen level of alpha

  3. Compare F-stat to critical value

  4. More extreme F-stat than critical value = significant


10
New cards

What needs to be considered for the F-statistic?

  • DFmodel

    • In (), this is shown before DF residual

  • DFresidual


<ul><li><p>DFmodel</p><ul><li><p>In (), this is shown before DF residual</p></li></ul></li><li><p>DFresidual</p></li></ul><p></p>
11
New cards

What does the F-distribution change shape based on?

df

<p>df</p>
12
New cards

What is the code for analysing the significance of F in R?

knowt flashcard image
13
New cards

What is the difference between questions being asked for F-tests, t-tests of beta estimates, and model comparisons?

  • F-test → is our overall model better than nothing?

  • T-tests of beta estimates → which variables, specifically, are good predictors of the outcome variable?

  • Model comparison → when I make a change to my model (e.g. add predictors), does it improve or not?

    • OR is a model with X1, X2 + X3 as predictors better than the model with just X1?


14
New cards

How do model comparisons determine whether models improve when predictors are added?

We can look at the combined performance of a subset of predictors

15
New cards

In what circumstances are model comparisons not useful? Why?

  • When comparing 2 models that only differ in 1 parameter → significance test for that parameter tells you the same info

  • When choosing what predictors to include in a model

    • Should include predictors that are theoretically motivated by your RQ + required to control for confounds

    • Don’t remove predictors just because they’re non-significant or don’t significantly improve the fit of the model


16
New cards

When is model comparison useful?

When you want to test the combined impact of multiple predictors

17
New cards

What is an incremental F-test?

  • Way of thinking about the F-test + F-ratio as an incremental test against an empty/null model

  • Comparing 2 models that contain fewer/more predictors → difference in F-values between said models

    • Evaluates the stat significance of the improvement in variance explained in an outcome with the addition of further predictor(s)


18
New cards

What is an empty/null model?

  • Linear model with only the intercept

  • Predicted value of the outcome for every case in the data set = mean of the outcome (all beta = 0 → H0 of the F-test)

    • No predictors → no info that may help us predict the outcome → will be “least wrong” by guessing the mean of the outcome


19
New cards

What is a full model in an incremental F-test?

Model with additional predictors (little F)

20
New cards

What is a restricted model in an incremental F-test?

Model without additional predictors (little R)

21
New cards

What is the formula for an incremental F-test (model comparison)?

knowt flashcard image
22
New cards

What are the steps involved in an incremental F-test in R?

  1. Fit + run (summarise) models

  2. Compare models based on an incremental F-test → anova(m1, m2)


<ol><li><p>Fit + run (summarise) models</p></li><li><p>Compare models based on an incremental F-test → anova(m1, m2)</p></li></ol><p></p>
23
New cards

In an incremental F-test, what does the F-ratio depend on?

The comparison models/variables being nested → the predictors in 1 model are a subset of the predictors in the other

<p>The comparison models/variables being nested → the predictors in 1 model are a subset of the predictors in the other</p>
24
New cards

What is a further condition of conducting incremental F-tests?

  • Require the models to be computed on the same data

    • Be careful of NAs → lm() excludes the whole row of data if NAs for x or y are present

    • Data sets used by the models could end up different if the additional variables you add to the full model have NAs


25
New cards

What alternatives are used to compare non-nested models?

  • AIC

  • BIC

    • Neither require 2 models to be nested (can be used for both)

    • Using either for a single model isn’t meaningful → only make sense for model comparisons

    • Smaller (more negative) values = better fitting models → compare values + choose the model with the smaller AIC or BIC value


<ul><li><p>AIC</p></li><li><p>BIC</p><ul><li><p>Neither require 2 models to be nested (can be used for both)</p></li><li><p>Using either for a single model isn’t meaningful → only make sense for model comparisons</p></li><li><p>Smaller (more negative) values = better fitting models → compare values + choose the model with the smaller AIC or BIC value</p></li></ul></li></ul><p></p>
26
New cards

What do both AIC and BIC contain?

Parsimony correction → penalise models for being complex → helps us avoid overfitting (adding predictors arbitrarily to improve fit)

<p>Parsimony correction → penalise models for being complex → helps us avoid overfitting (adding predictors arbitrarily to improve fit)</p>
27
New cards

Does AIC or BIC have a harsher parsimony penalty for typical samply sizes when applying linear models?

BIC

<p>BIC</p>
28
New cards

What is the code for AIC and BIC in R (nested model)?

knowt flashcard image
29
New cards

What is the code for AIC and BIC in R (non-nested model)?

knowt flashcard image
30
New cards

What are the thresholds for △AIC?

No specific thresholds

31
New cards

What are the thresholds for △BIC?

knowt flashcard image