OLS Properties and Goodness of Fit

These flashcards cover the key mathematical properties of Ordinary Least Squares (OLS) estimators and the core concepts of Goodness of Fit, including TSS, ESS, RSS, and the Coefficient of Determination ($$R^2$$).

Correlation

The statistical dependence of two random variables which is determined by pairwise comparison of values.

Uncorrelatedness of observed predictors and residuals

A mathematical property of OLS estimators where there is no linear relationship between the predictors ($X$) and the residuals ($\hat{e}$), expressed as $X'\hat{e} = 0$.

Sum and Mean of Residuals

In OLS, the sum of the residuals is zero ($\sum \hat{e}_i = 0$), which also implies that the sample mean of the residuals ($\bar{\hat{e}}$) is zero.

Point of Means

The property stating that the regression line (or hyperplane) always passes through the means of the observed values ($\bar{x}$ and $\bar{y}$), such that $\bar{y} = \hat{\beta}_0 + \hat{\beta}_1 \bar{x}$.

Uncorrelatedness of predicted values and residuals

A mathematical property where the predicted values of $y$ ($\hat{y}$) are uncorrelated with the residuals ($\hat{e}$), expressed as $\hat{y}' \hat{e} = 0$.

Equality of Observed and Predicted Means

The mathematical property stating that the mean of the predicted values ($\bar{\hat{y}}$) for the sample will equal the mean of the observed values ($\bar{y}$).

Total Sum of Squares (TSS)

The total variation in the dependent variable $y$ around its mean $\bar{y}$, defined mathematically as $TSS = \sum(y_i - \bar{y})^2$.

Explained Sum of Squares (ESS)

The variation in the dependent variable that can be explained by the regression model, defined as $ESS = \sum(\hat{y}_i - \bar{y})^2$.

Residual Sum of Squares (RSS)

The variation in the dependent variable that cannot be explained by the regression, defined as $RSS = \sum(y_i - \hat{y}_i)^2$, representing the sum of squared residuals.

Coefficient of Determination ($R^2$)

A measure indicating the proportion of the variation in the dependent variable explained through the independent variable, calculated as $R^2 = \frac{ESS}{TSS} = 1 - \frac{RSS}{TSS}$. It ranges between $0$ and $1$.

Baseline (Null Model)

A horizontal line at $\bar{y}$ used when the independent variable has no influence on the dependent variable, predicting $\bar{y}$ for every value of $x_i$.