Lecture 1 - Introduction & Linear Regression

Explain the aim of both supervised and unsupervised machine learning

Supervised Learning: aims to discover the relationship between an output variable y (e.g. the price of a house) and an input variable x (e.g. the size of the house) from a set of examples.

Unsupervised learning: there is no output variable y, we aim to find some structure in the x variable (e.g. we want to cluster customers in different groups depending on their characteristics).

State both the model and real-world supervised learning equations, and explain the objective of supervised learning in relation to these equations.

In general in supervised learning, how do we find the model function f-hat?

We first define a loss function that measures the error in predicted vs. realised outcome across the examples in our data set. We then select the f-hat function that minimises this loss function. Different supervised machine learning algorithms use different loss functions and strategies to find f-hat.

When people refer to ‘training’ a supervised machine learning model, what do they mean?

Training a supervised machine learning model refers to the finding of the f-hat function that results in the y-hat predicted outcomes that are closest to the true outcomes, using a set of examples.

What data must be used to test the performance of a supervised machine learning model?

The performance of the model must be evaluated on new data that have not been used to train the model.

State and explain the two types of supervised machine learning problems

What is a linear regression problem?

What is the goal of linear regression?

The goal of linear regression is to find the best linear relationship between the input and the output.

In what form does the function f-hat take in a linear regression model?

State and write the f-hat form for the measure of the error used in linear regression.

State and write the intermediate beta-hat form for the measure of the error used in linear regression.

State and write the final beta-hat form for the measure of the error used in linear regression.

What do we we want to do with the measure of the error in linear regression?

State the steps to compute an estimate beta-hat for the parameter beta.

1. Derive the RSS with respect to the parameters.

2. Set it equal to zero.

3. Re-arrange so that everything is equal to y

4. Convert to matrix form.

5. Isolate beta by inverting the matrix.

Compute an estimate beta-hat for this linear model in matrix form.

Name and give the formula for a measure of how well the linear regression model performs on the training data.

Give another name for the coefficient of determination, explain what it measures and state the formula to calculate it.

Another name for the coefficient of determination is R-squared, which measures the fraction of variance of the training data that is explained by the model.

Give the formula for the total sum of squares (TSS)

If we have a “perfect” model, what will the RSS and R² measurements be.

The RSS will be equal to 0, and the R² will be equal to 1.

If we have the worst possible model, what will the RSS and R² measurements be.

The RSS will be equal to the TSS, and the R² will be equal to 0.

How do you interpret the R² number?

The better the model, the closer R² is to 1.

Name and give a formula for a good metric to measure the test (out-of-sample) performance of the linear model with.

What is the vectoral expression for y in a linear regression model

Does ordinary least squares provide an unbiased estimate of the beta parameters?

Compute the expected value of the parameters / prove that E[B^hat] provides an unbiased estimate of B.

State the cov[B^hat] in term of B^hat

State the cov[B^hat] in matrix form

Derive the cov[B^hat] in matrix form from the cov[B^hat) in term of B^hat form.

State the formula for an unbiased estimator of σ²

State the full expanded matrix formula for cov[B^hat]

What is the form of a linear estimator of a parameter?

When is an estimator beta^hat unbiased?

State the Gauss-Markov theorem.

The Gauss-Markov theorem states that among all linear unbiased estimators, the least square estimator has the smallest variance.

Explain how gradient descent approaches optimisation

• We want to find the minimum of the function (the red point in the figure).

• We start from an initial guess, which can be any point (the black point in the figure).

• We take a small step in the direction along which the function decreases the most and find our new guess (the orange point in the figure).

• We start from the new guess and take a further step in the direction along which the function decreases the most.

• We keep going until we reach convergence (e.g. the change in the objective function between two subsequent steps is smaller than a given threshold).

Explain what the gradient of a function is.

The gradient of a function is a vector whose components are the partial derivatives of the function along the coordinates. The gradient identifies the direction along which the function increases the most.

State the formula for the gradient of a linear regression.

For linear regression, state the formula for the k-th component of the gradient.

In relation to gradient descent, what does the parameter α do

α is the learning rate: a real positive parameter that we use to tune the length of the step that we take along the direction identified by the gradient.

For linear regression gradient descent, state the formula for the update to the ‘best guess’ beta^hat_k