Wk5 - Notes on Linear Predictive Regression

Linear (Predictive) Regression Model

Linear regression is a fundamental statistical tool used in econometrics, broadly establishing the relationship between a dependent variable (also known as the regressand) and one or more independent variables (often referred to as regressors). The hallmark of linear regression lies in its assumption of a linear relationship between these variables.

Understanding Linear Regression

The linear regression model can be represented mathematically as:

[y = a1 x1 + a2 x2 + … + ak xk + ε]

Here, y is the dependent variable, x1, x2, …, xk are independent variables, a1, a2, …, ak are coefficients to be estimated, and ε signifies the random disturbance term. This disturbance accounts for any deviation from the perfect linear relationship, allowing for variation due to measurement errors, unobserved influences, and other random effects.

Example of Linear Regression

To illustrate, let’s consider data on the excess returns of a fund manager’s portfolio compared to a market index over several years. For example, the following data points can represent excess returns:

By exploring the linear relationship between these excess returns (y) and the market index (x), one generally anticipates a positive correlation. The scatter diagram can depict this relationship visually, though real-world data often deviates from a perfect line, thus necessitating the inclusion of a disturbance term in the model.

Finding the Best Line of Fit

The goal in simple linear regression is to determine the coefficients α and β in the equation:

[yt = α + β xt + u_t]

To achieve this, we seek to minimize the sum of squared errors given by:

[min{α,β} \sum{t=1}^{T} (yt - α - β xt)^2 (1)]

This method of minimizing the residuals is known as Ordinary Least Squares (OLS).

Ordinary Least Squares (OLS) Estimator

The OLS estimator attempts to minimize the residuals, with the resulting coefficients given by:

1. Fitted value (ŷt): This is the predicted value based on the estimated model, given by [\hat{yt} = \hat{α} + \hat{β}x_t].

2. Residuals (ût): Represent the difference between observed and predicted values, calculated as [ut = yt - \hat{yt}].

3. The OLS guarantees that the sum of the squared differences between observed values and fitted values is minimized, effectively making [ (\hat{α}, \hat{β}) ] the best linear unbiased estimators under certain assumptions.

Statistical Properties of OLS

Under the classical linear regression assumptions:

1. Unbiased: OLS estimators are unbiased estimates for α and β.

2. Efficiency: Among the class of linear unbiased estimators, OLS attain the minimum variance, thus providing the most precise estimates.

3. Consistency: As the sample size increases, OLS estimators converge to their true values.

The assumptions required for these properties include:

• The error term has a mean of zero: E[ut] = 0

• The variance of error terms is constant: Var(ut) = σ² > 0

• The errors are not correlated: cov(ut, us) = 0 for t ≠ s

Interpretation of Coefficients

Using the OLS estimates, one can predict outcomes. For instance, if the market return increases by a certain amount, the expected increase in excess return can be quantified using the estimated β coefficient:

[Δyt ≈ \hat{β} × Δxt]

This relationship is vital for interpreting the effect of an independent variable on the dependent variable.

Limitations and Considerations

It's critical to interpret the intercept term (α) correctly. In practical situations, few observations close to the y-axis could lead to inaccuracies in estimating α, which can lead to erroneous conclusions.

In conclusion, linear regression, through the use of OLS, serves as a foundational methodology in econometrics for exploring and quantifying relationships between variables, while understanding its properties and limitations is paramount for sound statistical analysis.