STATS 100C - UCLA Midterm 1

Simple linear regression model (with intercept)

y_i = β₀ + β₁x_i + ε_i. x fixed (non-random), E(ε_i)=0, Var(ε_i)=σ², ε_i independent (Gauss-Markov). E(y_i|x_i)=β₀+β₁x_i is the population regression function.

Simple linear regression model (no intercept)

y_i = β₁x_i + ε_i. Same Gauss-Markov assumptions. Line forced through origin. Residuals do NOT sum to zero.

OLS estimators (with intercept)

β̂₁ = Σ(x_i − x̄)(y_i − ȳ) / Σ(x_i − x̄)² = S_xy / S_xx. β̂₀ = ȳ − β̂₁x̄. Derived by minimizing Σ(y_i − β₀ − β₁x_i)² and solving the normal equations.

OLS estimators as linear combinations

β̂₁ = Σ k_i y_i where k_i = (x_i − x̄)/S_xx. β̂₀ = Σ l_i y_i where l_i = 1/n − x̄ k_i. Useful for finding means, variances, and proving BLUE.

Key properties of the k_i weights

Σ k_i = 0, Σ k_i x_i = 1, Σ k_i² = 1/S_xx. These make β̂₁ unbiased and give its variance instantly.

Mean and variance of β̂₁

E(β̂₁)=β₁ (unbiased). Var(β̂₁)=σ² / S_xx. Holds under Gauss-Markov only (no normality needed).

Mean and variance of β̂₀

E(β̂₀)=β₀ (unbiased). Var(β̂₀)=σ² (1/n + x̄²/S_xx).

Covariance of β̂₀ and β̂₁

Cov(β̂₀,β̂₁)= −σ² x̄ / S_xx. Zero only when x̄=0 (centered x).

Residuals and their three key properties (with intercept)

e_i = y_i − β̂₀ − β̂₁ x_i. (a) Σ e_i = 0, (b) Σ e_i x_i = 0, (c) Σ e_i ŷ_i = 0. These come directly from the normal equations. Use them to prove many later results.

Do residual properties hold without intercept?

Σ e_i ≠ 0 (does not hold). Σ e_i x_i = 0 and Σ e_i ŷ_i = 0 still hold.

Why minimize SSE? (geometric intuition)

OLS finds the unique line that makes the residual vector orthogonal to the column space of X (the space spanned by the constant vector and the x vector). That is why Σ e_i=0 and Σ e_i x_i=0.

Gauss-Markov Theorem (BLUE) – statement

Among all linear unbiased estimators of β₁ (or β₀), the OLS estimator has the smallest variance. “Best” = minimum variance, “Linear” = weighted sum of the y’s, “Unbiased” = expectation equals the true parameter.

Gauss-Markov proof sketch for β₁

Any other linear unbiased estimator is b₁=Σ a_i y_i with Σ a_i=0 and Σ a_i x_i=1. Write a_i = k_i + d_i. Then Σ d_i=0, Σ d_i x_i=0, and Var(b₁)=σ² Σ(k_i+d_i)² = Var(β̂₁) + σ² Σ d_i² ≥ Var(β̂₁). Equality only when all d_i=0.

When is OLS also the MLE?

When the errors are i.i.d. normal. Then the likelihood is maximized at exactly the same point as the least-squares criterion. Under normality, MLE = OLS.

Unbiased estimator of σ²

s² = Se² = SSE/(n−2) = Σ e_i² /(n−2). The MLE Σ e_i² /n is biased downward; dividing by n−2 corrects the two degrees of freedom used to estimate β₀ and β₁.

Distribution of SSE under normality

SSE/σ² ∼ χ²_{n−2} and is independent of (β̂₀, β̂₁). This is the key fact that lets us replace σ by Se and obtain t and F distributions.

t-statistic for β₁

(β̂₁ − β₁) / (Se / √S_xx) ∼ t_{n−2}. Use for CI or test of H₀: β₁ = c.

Confidence interval for β₁ (σ unknown)

β̂₁ ± t_{n−2,1−α/2} × (Se / √S_xx).

Confidence interval for the mean response E(Y₀|x₀)

ŷ₀ ± t_{n−2,1−α/2} × Se √(1/n + (x₀−x̄)²/S_xx). This is a CI for a fixed point on the regression line.

Prediction interval for a new observation Y₀

ŷ₀ ± t_{n−2,1−α/2} × Se √(1 + 1/n + (x₀−x̄)²/S_xx). Wider than the mean-response CI because it must also cover the extra random error ε₀.

When to use a prediction interval vs a CI for the mean

CI for the mean → “What is the average y for this x?” Prediction interval → “Where will the next individual observation fall?”

Leverage / hat-matrix diagonal

h_ii = 1/n + (x_i − x̄)² / S_xx. Var(ŷ_i)=σ² h_ii, Var(e_i)=σ²(1−h_ii). High leverage points pull the line strongly.

R² definition and interpretation

R² = SSR/SST = 1 − SSE/SST. Proportion of total variation in y explained by the linear relationship with x. Only meaningful for models with intercept.

Decomposition of total sum of squares

SST = SSR + SSE (i.e. Σ(y_i−ȳ)² = Σ(ŷ_i−ȳ)² + Σ e_i²). Holds because residuals are orthogonal to fitted values.

Centered model (useful trick)

Rewrite y_i = γ₀ + β₁(x_i − x̄) + ε_i where γ₀=β₀+β₁x̄. Then γ̂₀=ȳ, the slope estimator is unchanged, and Cov(ȳ,β̂₁)=0. Makes many variance calculations cleaner.

Key summation identities you must know cold

Σ(x_i−x̄)=0, Σ(x_i−x̄)² = Σ x_i² − n x̄², Σ(x_i−x̄)y_i = Σ(x_i−x̄)(y_i−ȳ), Σ(x_i−x̄)x_i = S_xx.

How to prove Σ e_i =0 algebraically

Σ e_i = Σ y_i − n β̂₀ − β̂₁ Σ x_i = nȳ − n(ȳ − β̂₁ x̄) − β̂₁ (n x̄) =0.

How to prove Σ e_i x_i =0 algebraically

Σ e_i x_i = Σ x_i y_i − β̂₀ Σ x_i − β̂₁ Σ x_i². Substitute β̂₀ and β̂₁ and the definition of S_xy and S_xx; everything cancels.

F-test for overall significance (H₀: β₁=0)

F = (SSR/1) / (SSE/(n−2)) = β̂₁² S_xx / Se² ∼ F_{1,n−2} under H₀. Equivalent to the two-sided t-test for β₁.

Likelihood-ratio test intuition

Compare maximized likelihood under the restricted model (H₀) to the unrestricted model. For simple regression the LR statistic is a monotone function of the usual F (or t²) statistic.

When the no-intercept model is appropriate

Only when theory forces the line through the origin (e.g., some physical laws). Otherwise the intercept model is almost always safer; forcing the intercept to zero can bias the slope badly if the true intercept is not zero.

Intuitive reason OLS is BLUE

Any other linear unbiased estimator is OLS plus some extra noise that is orthogonal to the column space of X. That extra noise can only increase variance.

Quick checklist before using any formula

1. Does the model have an intercept? 2. Are the x’s fixed or random? 3. Do we assume normality? 4. Is σ known or estimated? Answers determine whether you use z, t, or F, and which variance formula.

What is σ² in the simple regression model?

σ² = Var(ε_i) = E(ε_i²). It is the common variance of the random errors. Under Gauss-Markov it is constant across all observations (homoscedasticity). It measures the inherent noise around the true regression line. We never observe it; we estimate it from the residuals.

Two common estimators of σ² and their properties

1. MLE: σ̂²_MLE = SSE / n. Biased downward (E[σ̂²_MLE] = ((n-2)/n)σ²).

2. Unbiased estimator: Se² = s² = SSE / (n-2). E[Se²] = σ².

We almost always use Se² for inference because it is unbiased and gives exact t and F distributions under normality.

Why divide by n-2 for the unbiased estimator of σ²?

We used two degrees of freedom to estimate β₀ and β₁. The residuals are constrained by the two normal equations (Σe_i=0 and Σe_i x_i=0), so only n-2 independent pieces of information remain about the variance.

Distribution of SSE under normality

If εi ∼ i.i.d. N(0,σ²), then SSE / σ² ∼ χ²{n-2}. Moreover, SSE is independent of (β̂₀, β̂₁). This independence is crucial: it lets us replace the unknown σ with Se and still obtain exact t and F distributions.

Why is SSE independent of the OLS estimators under normality?

Because the OLS estimators are linear combinations of the y_i and the residuals are the projection onto the orthogonal complement of the column space of X. Under joint normality, a linear functional and the residual vector from an orthogonal projection are independent.

Distribution of β̂₁ under normality

β̂₁ ∼ N(β₁, σ² / S_xx). Proof sketch: β̂₁ = Σ k_i y_i is a linear combination of independent normals, so it is normal. We already know E(β̂₁)=β₁ and Var(β̂₁)=σ²/S_xx from Gauss-Markov.

Distribution of β̂₀ under normality

β̂₀ ∼ N(β₀, σ²(1/n + x̄²/S_xx)).

Joint distribution of (β̂₀, β̂₁)

They are jointly bivariate normal. Their covariance is Cov(β̂₀,β̂₁) = −σ² x̄ / S_xx. They are independent only if x̄ = 0 (i.e., when the x’s are centered).

t-distribution for a single coefficient

(β̂j − β_j) / se(β̂_j) ∼ t{n-2}, where se(β̂₁) = Se / √S_xx and se(β̂₀) = Se √(1/n + x̄²/S_xx). This follows because a standard normal divided by the square root of an independent χ²{n-2}/(n-2) is t{n-2}.

What is a “one-restriction” hypothesis test?

A test of a single linear restriction on the parameters, most commonly: H₀: β₁ = c (or β₀ = c, or β₀ + β₁ x₀ = c, etc.). Under H₀ the test statistic is a t with n-2 df, or equivalently an F with 1 and n-2 df (since t² = F_{1,ν}).

How to perform a one-restriction t-test for H₀: β₁ = 0

1. Compute t = β̂₁ / (Se / √S_xx).

2. Reject H₀ at level α if |t| > t_{n-2, 1−α/2}. This is exactly the usual “is the slope significant?” test. It is also equivalent to the overall F-test for the simple regression model.

One-restriction F-test version

F = (SSR / 1) / (SSE / (n-2)) = β̂₁² S_xx / Se² ∼ F_{1,n-2} under H₀: β₁=0. Same rejection decision as the two-sided t-test.

What is a “two-restriction” hypothesis test?

A test of two simultaneous linear restrictions, e.g. H₀: β₀ = c₀ and β₁ = c₁ or H₀: β₀ = 0 and β₁ = 0 or H₀: β₀ + β₁ x₀ = μ₀ and β₁ = 0, etc. Under normality the exact test is an F-test with 2 and n-2 degrees of freedom.

General form of the F-test for q linear restrictions

F = [(SSE_R − SSE_U) / q] / (SSE_U / (n − k − 1)) ∼ F_{q, n−k−1} under H₀, where SSE_R = residual sum of squares under the restricted model, SSE_U = residual sum of squares under the unrestricted model, k = number of regressors in the unrestricted model (here k=1 for simple regression).

Two-restriction test H₀: β₀=0 and β₁=0 —> what is F under H₀?

Unrestricted model: y = β₀ + β₁x + ε → SSE_U = Σ e_i² (df = n-2).

Restricted model under H₀: y = ε → SSE_R = Σ y_i² (df = n).

Then F = [(Σ y_i² − Σ e_i²)/2] / (Σ e_i² /(n-2)) ∼ F_{2,n-2} under H₀.

Likelihood-ratio test connection

Under normality the likelihood-ratio statistic Λ = (SSE_R / SSE_U)^{n/2}. −2 ln Λ is a monotone function of the F statistic above, so the LR test, the F-test, and (for one restriction) the t-test are all equivalent.

When do we use the restricted vs unrestricted SSE approach?

Any time the null hypothesis imposes linear constraints on the parameters. Fit both models (or compute the extra sum of squares analytically), form the F ratio, and compare to the critical value from F_{q, n−p}.

Efficiency of an estimator

An unbiased estimator θ̂ is efficient if it has the smallest possible variance among all unbiased estimators of θ. In the Gauss-Markov world we only compare linear unbiased estimators; under normality we can compare all unbiased estimators.

Cramér-Rao Lower Bound (CRLB) – statement

Under regularity conditions, for any unbiased estimator θ̂ of a parameter θ, Var(θ̂) ≥ 1 / I(θ), where I(θ) = E[−∂²ℓ/∂θ²] is the Fisher information (or equivalently E[(∂ℓ/∂θ)²]). The bound is a theoretical minimum; an estimator that attains it is called efficient (or MVUE).

Does OLS attain the Cramér-Rao bound?

Yes, under the normal-error assumption. Because OLS coincides with the MLE, and the MLE is known to attain the CRLB (at least asymptotically, and exactly in the normal linear model). Therefore, under normality, OLS is fully efficient among all unbiased estimators, not merely among linear ones.

Gauss-Markov vs Cramér-Rao – comparison

Gauss-Markov: “OLS is best among linear unbiased estimators” – needs only E(ε)=0, constant variance, independence (no normality).

Cramér-Rao: “OLS is best among all unbiased estimators” – needs normality (so that OLS = MLE and the bound is attained). If the errors are not normal, OLS is still BLUE but may not be fully efficient.

Intuitive meaning of the Cramér-Rao bound for regression

The more spread out the x’s are (larger S_xx) and the smaller the noise σ², the more information we have about the slope, and therefore the smaller the lower bound on the variance of any unbiased estimator.

Distribution theory summary under normality

• β̂₀, β̂₁ jointly normal

• SSE / σ² ∼ χ²_{n-2} independent of (β̂₀, β̂₁)

• (β̂j − β_j)/se(β̂_j) ∼ t{n-2}

• F statistics for linear restrictions ∼ F_{q,n-2}

These four facts are the entire foundation of classical inference in simple linear regression.

When normality is not assumed

We still have unbiasedness and the Gauss-Markov variance formulas. For large n we can appeal to the central limit theorem: β̂₁ is approximately normal, and we can still form approximate z or t intervals, but the exact χ², t, and F distributions no longer hold.

Prediction interval derivation (key distribution steps)

Y₀ − ŷ₀ is normal with mean 0 and variance σ²(1 + 1/n + (x₀−x̄)²/S_xx).

Standardize, replace σ by Se, and obtain a t_{n-2} pivot.

The extra “1” inside the square root comes from Var(ε₀); that is why prediction intervals are wider than confidence intervals for the mean response.

Leverage and the distribution of residuals

e_i ∼ N(0, σ²(1−h_ii)), where h_ii = 1/n + (x_i−x̄)²/S_xx. High-leverage points have smaller residual variance (they pull the line toward themselves).

Quick decision tree for hypothesis tests

• One linear restriction → t-test or F(1,n-2)

• q linear restrictions → F(q,n-2) using (SSE_R − SSE_U)/q over MSE_U

• No normality → rely on asymptotics or bootstrap