Prediction Models

Difference between association and prediction models

Association	Prediction
Interested in the effect of a variable	Interested in predicted outcome
Usually focuses on a particular exposure	Uses multiple covariates
Confounding is important	Predictive performance is important

In association studies, we often ask whether a coefficient is statistically significant.

In prediction studies, a statistically significant coefficient does not automatically mean the model predicts patients accurately.

What is the adjusted effect

Each coefficient is interpreted while keeping the other variables constant.

What is R squared

Imagine 100 patients have different cholesterol levels.

Some have cholesterol of 4, some 5, some 6, and others 8.

There is variation between patients.

The better the model explains the variation, the higher its R-squared.

R2 always between 0 and 1

R² value	Interpretation
0	Model explains none of the variation
0.25	Explains 25% of variation
0.50	Explains 50% of variation
0.75	Explains 75% of variation
1	Explains 100% of variation

Eg: 75% of the variation in cholesterol is explained by sex, age and BMI.

This means 25% of the variation is not explained by our model.

What is overfitting

The model has memorised the training data instead of learning patterns that generalise to unseen data.

A prediction model achieves 98% accuracy on training data but only 65% accuracy on test data. What is the most likely explanation?

Ovefitting

What is the purpose of F test in nested models

Two models are nested when the smaller model is contained within the larger model.

For example:

Model 1:

\[ Cholesterol=b_0+b_1(sex) \]

Model 2:

\[ Cholesterol=b_0+b_1(sex)+b_2(age)+b_3(BMI) \]

Model 1 is nested within Model 2 because Model 2 contains all the variables from Model 1, plus additional variables.

The question is:

Does adding age and BMI significantly improve the model?

The F-test evaluates whether the additional variables improve the fit enough to justify the extra complexity.

Multiple linear regression assumptions

Assumption 1: Residuals are normally distributed with mean zero.

Residual=Observed value-Predicted value.

in a QQplot, If the points follow the diagonal line: Residuals are approximately normally distributed.

If the points deviate substantially from the line: There may be a violation of normality.

Assumption 2: Equal variance of residuals/ homoscedasticity

The variance of the errors should be approximately equal for all subjects.

Assumption 3: Linear contributions of covariates.

The relationship between each continuous covariate and the outcome should be approximately linear, while keeping the other covariates constant.

Which test compares nested linear regression models?

F test

What tool you use to measure variation in the outcome is explained by the logistic regression model? (In linear regression, it is r squared)

1. Area Under the ROC Curve (AUC).

2. Nagelkerke's R²- It is a pseudo-R² measure

What does the ROC curve plot?

Sensitivity against 1 − specificity

ROC and AUC help us:

1. Evaluate how well a prediction model distinguishes between groups.

2. Compare different prediction models.

3. Understand the trade-off between sensitivity and false positive rate at different cut-offs.

AUC

Measures the model's ability to discriminate between cases and controls

AUC value	Interpretation
1.0	Perfect discrimination
0.9–1.0	Excellent
0.8–0.9	Good
0.7–0.8	Acceptable
0.5	No better than random guessing
Below 0.5	Worse than random

Important: An AUC of 0.85 means that if you randomly select one patient with cancer and one without cancer, the model has an 85% probability of assigning a higher predicted risk to the patient with cancer.

What is a Good logistic regression model:

•High Nagelkerke’s R2 and AUC (i.e. closer to 1)

Difference between discimination and calibration

Discrimination	Calibration
Correct ranking of patients	Accuracy of predicted probabilities
Evaluated using AUC	Evaluated by comparing predicted and observed risks
Who is at higher risk?	Is the predicted risk accurate?

A high AUC does not guarantee good calibration.

Regression model	Comparing nested models
Linear regression	F-test
Logistic regression	Likelihood Ratio test
Cox regression	Likelihood Ratio test

Methods for variable selection/stepwiseselection. What are the limitations

Method	Starting point	What happens?
Forward	Empty model	Adds variables
Backward	Full model	Removes variables
Stepwise	Usually empty model	Adds and can remove variables
Best subset	Candidate combinations	Compares subsets

Problem 1: It does not prevent overfitting.

Problem 2: It biases standard errors, confidence intervals and p-values.

What is Akaike Information Criterion (AIC).

AIC helps us choose between competing statistical models.

AIC balances:

• How well the model fits the data.

• How complex the model is.

Low AIC is preferred

How do we make sure our model does not have overfitting

Internal & external validation

Internal validation	External validation
Uses a hold-out sample from the original dataset	Uses independent external data
Same original source population	Different external population or setting
Evaluates performance beyond the development sample	Evaluates transportability to another setting

Lower MSE = Better predictions. Explain

Patient	Actual value	Predicted value
A	10	8
B	20	18
C	30	25

What is a Brier score

For binary outcome models, the mean squared prediction error is called the Brier score.

• Continuous outcomes → MSE.

• Binary outcomes → Brier score.

How many patients do we need to develop a prediction model?/ How to calculate the sample size

1. Linear regression

The rule is: 𝑛≥10x𝑘

Where:

• \(n\) = Total sample size.

• \(k\) = Number of covariates.

• 
  

2. Logistic regression and Cox regression

take 𝑛 such that you have at least 10⋅𝑘

events and at least 10⋅𝑘non-events

A researcher wants to develop a prediction model for tumour volume (cm³) in patients diagnosed with colorectal cancer.

The following variables are available:

• Age (years)

• BMI

• Sex (male/female)

• Smoking status (yes/no)

The researcher fits a multiple linear regression model using all available predictors.

a) Which type of regression model is appropriate for this outcome? Explain your answer. (1 point)

b) The estimated regression coefficient for age is 0.45. Interpret this coefficient in the context of the study. (1 point)

c) Name two assumptions of multiple linear regression and explain how you would assess them. (2 points)

A researcher investigates whether age, BMI, smoking status and tumour stage can predict the change in tumour volume after six months of treatment.

The dataset is called cancer.

The researcher wants to develop a prediction model using all available predictors, followed by stepwise variable selection.

a) R-question. Write the R code to fit a full multiple linear regression model predicting tumour_change using all available variables. (1 point)

b) R-question. Write the R code to perform stepwise variable selection using AIC, allowing both forward and backward selection. (1 point)

c) The full model has an AIC of 245, whereas the stepwise model has an AIC of 218. Which model would you prefer based on AIC? Explain why. (1 point)

d) Explain why stepwise selection does not guarantee that the final model will perform well in new patients. (1 point)

A research team investigates whether age, smoking status and tumour stage can predict whether a patient develops cancer recurrence within two years (yes/no).

The following logistic regression output is obtained:

Variable	Estimate	p-value
Intercept	-3.00	0.001
Age	0.04	0.020
Smoking	0.80	0.005
Tumour stage	1.20	0.001

a) Which regression model is appropriate for this outcome? (0.5 points)

b) Calculate the odds ratio associated with smoking. Interpret your answer. (1 point)

c) Calculate the predicted probability of cancer recurrence for a patient who is:

• 60 years old

• Smoker (smoking = 1)

• Tumour stage = 2

Use the following equation:

\[ \begin{aligned} \text{Log-odds}={}&-3.00\\ &+0.04(\text{age})\\ &+0.80(\text{smoking})\\ &+1.20(\text{stage}) \end{aligned} \]

Convert your result into a probability. (2 points)

d) Why can we not directly interpret the regression coefficient of smoking as an increase in probability? (0.5 points)

A hospital develops two logistic regression models to predict whether patients develop sepsis.

The models are evaluated using the same independent validation dataset.

Performance measure	Model A	Model B
AUC	0.85	0.72
Brier score	0.18	0.12
Nagelkerke's R²	0.40	0.32

a) Which model has better discriminatory ability? Explain your answer. (1 point)

b) Which model has better predictive accuracy according to the Brier score? Explain your answer. (1 point)

c) Explain the difference between discrimination and calibration. (1 point)

d) Can you conclude that Model A is better overall than Model B based on these results? Explain your reasoning. (1 point)

A researcher develops a linear regression model to predict survival-related biomarker levels using 50 candidate predictors in 100 patients.

The model produces the following results:

Performance measure	Training data	Independent test data
R²	0.78	-0.15
MSE	4.2	18.5

a) What is overfitting? (1 point)

b) Explain why the model has a high R² in the training data but a negative R² in the test data. (1 point)

c) What does a negative out-of-sample R² indicate? (1 point)

d) Explain why external validation is important before implementing a prediction model in clinical practice. (1 point)