Econometrics Exam 1

Lecture 1: Introduction to Econometrics


I. What Econometrics Is and Why It Matters

Econometrics combines economic theory, data, and statistical tools to answer “how much” questions about relationships in economics, business, and the social sciences.

It is not a branch of economics or statistics on its own—it is a tool that allows us to quantify relationships between variables. It uses theoretical understanding from fields like economics and business, along with statistical methods, to produce empirical, measurable results.

Econometrics is essential because businesses, policymakers, and researchers constantly face “how much” questionsthat require quantitative answers. Learning econometrics provides the skills to answer those questions and makes you valuable in today’s job market, where data-driven decision-making is key.


II. The Importance of “How Much” Questions

A. Real-World Decision Making

Every day, firms and institutions must make decisions that depend on quantifying effects. Examples illustrate how econometrics is used to guide such decisions.

  1. Victoria’s Secret Example

    The company must determine the effect of advertising on revenues.

    • Too little advertising → lost potential profit.

    • Too much advertising → wasted resources.
      Using econometric methods allows them to estimate how much advertising affects sales and find the optimal level of spending.

  2. UTD Tuition Example

    Suppose UTD (University of Texas at Dallas) wants to raise tuition to increase revenue.

    • When tuition rises by 10%, enrollment will fall—that’s known intuitively.

    • But the crucial question is by how much will enrollment fall?

      • If enrollment drops by 20%, revenue decreases (bad idea).

      • If enrollment drops by only 5%, revenue increases (good idea).
        To make this decision intelligently, UTD must estimate the price elasticity of demand for education—the percentage change in enrollment given a percentage change in tuition.
        This “how much” question is exactly what econometrics helps answer.

  3. Toyota and Tesla Example

    Toyota needs to estimate the effect of a new Tesla model on sales of its Prius hybrid.

    • If Tesla’s new car reduces Prius sales by only 5%, Toyota can wait before investing heavily in a new electric car.

    • But if Prius sales drop by 40%, Toyota must invest immediately.
      Econometric analysis quantifies the competitive impact and informs long-term strategic investment decisions.

  4. Personal Example – Education and Wages

    Even individuals make “how much” decisions. For instance, when choosing to pursue a Master’s in Business Analytics, students implicitly estimate how much higher their future wages will be.

    • If the expected wage increase is only 5%, it might not be worth it.

    • If the expected increase is around 25%, it likely is.
      Most people rely on previous econometric research that estimates the return to education—for example, that a master’s degree raises wages by about 25%.

In this course, you will learn how to estimate these kinds of relationships using data and econometric tools.


III. From Economic Models to Econometric Models

To estimate relationships such as between price and demand or advertising and sales, we must first formulate an econometric model.

Every econometric model begins with an economic model—a theory that explains how variables relate to one another. Without theory, econometric work is meaningless.

A. Economic Models

Economic theory often expresses relationships between variables using functions.
For example, the demand for Coke can be expressed as a function of several factors:

[Quantity Demanded=f(Price of Coke,Price of Pepsi,Price of Chips,Income)][\text{Quantity Demanded} = f(\text{Price of Coke}, \text{Price of Pepsi}, \text{Price of Chips}, \text{Income})]

Each factor (price of Coke, substitutes, complements, and income) affects demand in a predictable way based on economic reasoning.


IV. Illustrating Causality – The Deer Crossing Example

To show why it’s difficult to determine causal relationships, consider the “Deer Crossing” story:

A caller once complained that deer crossing signs should be moved to safer areas—like school zones or small-town roads—because “too many deer get hit on highways.”
She believed deer would see the signs and cross there, misunderstanding that the signs are for drivers, not deer.

The story highlights a crucial lesson: causality is not always obvious. People often confuse correlation (two things happening together) with causation (one causing the other).
Econometrics provides the tools to correctly identify causal effects—distinguishing whether advertising increases sales, or if high sales simply lead to more advertising.


V. The Structure of an Econometric Model

An econometric model consists of two parts:

  1. The systematic (deterministic) part

  2. The random (stochastic) part (the error term)

A. Systematic Part

This is derived from the economic theory and includes:

  1. Dependent variable (Y) – the variable being explained (e.g., quantity demanded).

  2. Explanatory variables (X’s) – the factors that explain Y (e.g., price, income).

  3. Coefficients (β’s) – the parameters showing how much each X affects Y.

  4. Functional form – the assumed mathematical shape of the relationship (often linear).

For example, a simple linear demand model might look like:

[Qd=β0+β1PCoke+β2PPepsi+β3PChips+β4Income+ε][Q_{d}=\beta_0+\beta_1P_{\text{Coke}}+\beta_2P_{\text{Pepsi}}+\beta_3P_{\text{Chips}}+\beta_4\text{Income}+\varepsilon]

The β’s represent true effects that exist in nature (e.g., β2=77,000,β3=–10,000β₂ = 77,000, β₃ = –10,000 , etc.), but we don’t know their values.
The goal of econometrics is to estimate these β’s using data.

B. Random (Error) Component – εε

The random error term(ε)(ε) represents unpredictable factors or omitted influences that affect Y but are not included in the systematic part.

For instance, in the Coke demand model:

  • Some localities may have a big wedding (more Coke sold).

  • Others may have anti-soda campaigns (less Coke sold).

  • Some may experience a heatwave (more Coke consumed).
    These unpredictable differences are captured in ε.

If an important variable (like temperature) is left out of the model, it still affects demand in reality. Since it’s not in the systematic part, it ends up inside the error term, making εε no longer “pure noise.”
This omission creates systematic bias—a problem that complicates econometric estimation.

For now, we assumeεε is pure random noise, but later in the course we’ll explore what happens when it isn’t.


VI. Uses of Econometric Models

Once an econometric model is built and estimated, it serves three key purposes:

  1. Estimation: Quantify the strength and direction of relationships between variables.

  2. Prediction/Forecasting: Predict future values of Y given hypothetical changes in X’s.

    • Example: A store manager can forecast how Coke demand changes if price drops by $0.20 and chips rise by $0.05.

  3. Hypothesis Testing: Test beliefs or theories using data.

Example: UTD’s Tuition Belief

UTD believes that demand for schooling is inelastic (students are not sensitive to tuition changes).
If that’s true, increasing tuition should raise total revenue (since enrollment would fall by less than the tuition increase).
But econometric testing can check this belief:

  • If data reject it, UTD must revise its policy—perhaps realizing that raising tuition would actually reduce revenue.


VII. Data and Where It Comes From

Econometric analysis requires data to estimate models and test hypotheses.

A. Controlled Experiments

In a controlled experiment, researchers vary one variable while holding all others constant.
This allows clear identification of cause and effect.

Example: Testing a new weight-loss pill

  • Treatment group: receives the real pill.

  • Control group: receives a placebo.
    If the treatment group loses 5 pounds more, that’s the pill’s estimated effect.

However, controlled experiments are rare in business or social sciences because they are costly or infeasible.
For example:

  • A company studying advertising effects would need to fix everything except advertising for months—impractical.

  • In studying race discrimination in wages, you cannot “change” someone’s race for experimental purposes.

Thus, in business and economics, we often rely on non-experimental (observational) data.

B. Non-Experimental or Observational Data

These are retrospective data already collected, not designed as experiments.
Economists must extract causal relationships even though many variables change simultaneously.

A common source is survey data, such as information collected from 1,000 households.

Characteristics of survey data:

  1. All variables are collected simultaneously (e.g., race, income, family size, education).

  2. Samples are random and non-replicable: Every survey yields slightly different samples and results.

Because every sample is different, the estimates from each sample vary.
This raises a key question: How close are our sample-based estimates to the true parameters in the population?

This leads to one of the central goals of econometrics—making inferences about the population from a sample.


VIII. Types of Data in Econometrics

There are three major types of data used in econometric analysis:

A. Cross-Sectional Data

  • Collected at one point in time across many individuals, firms, or entities.

  • Each entity is observed once.

  • Example: A dataset of 1,000 individuals with information on education, wage, age, and gender—all recorded in the same year.

B. Time Series Data

  • Collected over time for a single entity (a country, firm, or person).

  • Data are recorded at regular intervals (daily, monthly, quarterly, or yearly).

  • Example: U.S. GDP from 2001–2008 recorded annually.
    Regular time spacing is essential for time series analysis.

C. Panel (Longitudinal) Data

  • Combines cross-sectional and time series features.

  • Observes many entities over several time periods.

  • Example: 1,000 firms tracked for 5–7 years, recording production, employment, and area cultivated.
    In panel data, the number of entities (N) is typically much larger than the number of time periods (T).


IX. The Research Process in Econometrics

A proper research process always begins with economic theory, not with econometrics.

Steps:

  1. Start with economic theory – identify relevant variables and expected relationships.

  2. Formulate the econometric model – specify the dependent and explanatory variables and the functional form.

  3. Collect data – obtain an appropriate sample (cross-sectional, time series, or panel).

  4. Select statistical tools – choose estimation methods consistent with your model and data type.

  5. Estimate the parameters (β’s) – using software and econometric techniques.

  6. Use results for:

    • Prediction and forecasting

    • Hypothesis testing

    • Policy or business recommendations

Econometrics is the tool for the inference stage, not the starting point of research.


X. Preparation for Next Lecture

Before proceeding to the next class, review fundamental probability and statistics concepts, since econometrics builds on them.

Key topics to review:

  • Random variables

  • Mean and variance

  • Covariance and correlation

  • Normal distribution

These are summarized in the Probability Review PowerPoint and the introduction chapter of your textbook (before Chapter 1).

Complete Homework #1, which checks your understanding of these probability concepts.


XI. Looking Ahead

In the next lecture, the course will introduce the simple linear regression model, explore the least squares estimator, and examine the properties of estimators.

This concludes the overview of what econometrics is, why it’s important, how models are structured, where data come from, and how theory connects to estimation and inference.

Lecture 2 (Part 1): The Simple Linear Regression Model

I. Introduction to the Model

The goal is to estimate the relationship that exists in nature—meaning the real world or overall population—between household income and spending on food.
We want to determine, in the real world, the expected spending on food for every additional $100 of household income.

In nature, both real-world expenditures and income are random variables. Therefore, we need an econometric modelthat allows us to study the relationship between these two random variables.

Although the objective is to learn about the relationship that exists in the population, in practice we only use a sample. The relationship estimated in the sample is then used to extrapolate to the real world.


II. Building the Econometric Model

Every econometric model begins with an economic theory, which determines the systematic part of the model.

Economic theory tells us that expenditure on food depends on income, not the other way around.
Therefore:

  • Dependent variable (Y): Expenditure on food

  • Independent (explanatory) variable (X): Income

To the systematic part that comes from theory, we add a random error term that represents all non-systematic influences on the dependent variable.
The combination of the systematic component and the random component forms the econometric model.


III. The Simple Linear Regression Model

The simple linear regression model is the most widely used econometric model.
It is called “simple” because it contains only one explanatory variable. However, despite the name, it is not necessarily simple in practice.

The model consists of two parts:

  1. Systematic part (deterministic): Derived from theory

  2. Random part: Represents unobserved factors or pure noise

Mathematically, the model is expressed as:

Yi=β1+β2Xi+εiY_i =β_1 +β_2 X_i +ε_i

  • YiY_{i} Spending on food

  • XiX_{i} : Income

  • β1\beta_1 : Intercept of the regression line

  • β2\beta_2 Slope (change in expected spending for each unit increase in income)

  • εi\varepsilon_{i} : Random error term


IV. Assumptions About Reality (Nature)

We assume that, in nature, there is only one variable—income—that systematically explains spending on food, and that the relationship is linear.

That means the true relationship between income and spending on food is perfectly described by:

Y=β1+β2XY=\beta_1+\beta_2X

However, this assumption is not true in reality.

A. Missing Variables

Other important variables can explain food spending:

  • Household size: Larger families consume more.

  • Location: Food costs vary by city (e.g., New York vs. Dallas).

B. Non-Linearity

The relationship is likely non-linear.
Example:

  • A household earning $2,000 per week might spend 10% ($200) on food.

  • A household earning $10,000 per week will not spend $1,000 (also 10%)—there is a limit to consumption.

Thus, the proportion of income spent on food decreases as income increases. Spending represents a large share for poorer households and a smaller share for richer ones.

Even though this model is wrong as a description of reality, we use it as a starting point.
Later, more advanced models will relax these simplifying assumptions.


V. Estimation Problem

Our task is to estimate the unknown parameters β1\beta_1 and β2\beta_2.
These parameters exist in nature (the population) but are unobserved.

We use our sample data to obtain estimates:

  • B1B_1estimates β1\beta_1

  • B2B_2estimates β2\beta_2

Notation difference:

  • Greek letters (ββ): True, unobserved parameters in the population

  • Latin letters (B): Sample-based estimates

Example:

  • True value (in nature): β2=15\beta_2=15

  • Estimated value (from sample): B2=11.5B_2=11.5

The estimate may be close or far from the true parameter, but since the true β is unobserved, we can never know the distance between them.
The best we can do is develop an estimation procedure that increases the probability that our estimate B2B_2is close to the true β2\beta_2.

Even with the best tools, we can never know for sure if our estimate is close to the true parameter.


VI. Graphical Representation

If the model is correct and the error term represents pure noise, then:

E(ε)=0E(ε)=0

Thus, the expected value of Y (spending on food) is:

E(Y)=β1+β2XE(Y)=\beta_1+\beta_2X

  • The blue line represents the true relationship in the population between income and expected spending.

  • β1\beta_1 (intercept): Expected food spending for a household with zero income.

  • β2\beta_2 (slope): Expected increase in food spending for each additional $100 of income.

However, because of random error, this blue line cannot be observed directly.

A. Role of the Error Term

Each household experiences a random disturbance:

  • A positive error (ε>0)(ε > 0) : Spends more than expected (e.g., preference for organic food)

  • A negative error (ε<0)(ε < 0) : Spends less (e.g., dieting)

Therefore, actual observed spending (black points) deviates from the expected value (blue line).

If there were no random noise, all observations would lie perfectly on the blue line, and we would know the true parameters exactly.
Because of noise, we must estimate the blue line by fitting a red line using statistical tools.

The goal of econometrics is to estimate the true (blue) line as closely as possible with the estimated (red) line.
We can only increase the probability that the red line is close to the blue one—we can never know for sure.


VII. Assumptions of the Simple Linear Regression Model

To estimate the model, we rely on several key assumptions.

Assumption 1: The explanatory variable (X) is non-random

Although income is clearly random in reality, this assumption is made for methodological simplicity.
It allows controlled experimentation where the researcher can choose the values of X.

In practice, we only observe incomes; we do not control them.
However, the results of the model do not depend on this assumption—it only simplifies the mathematical proofs.


Assumption 2: The expected value of the error term is zero

E(εi)=0E(\varepsilon_{i})=0

If the error term is pure noise, its average should be zero.
However, this assumption implicitly requires that the model be correctly specified.

Example:
If we omit an important variable (like household size), it will end up inside the error term.
Then, the error term will contain systematic information, and E(εi)≠0E(\varepsilon_{i})\ne0 .
In that case, our model becomes misspecified, and many results will no longer hold.

For now, we assume this condition is true.


Assumption 3: Zero covariance between errors

Cov(εi,εj)=0Cov(\varepsilon_{i},\varepsilon_{j})=0     ∀i≠j\forall i\ne j

If X is non-random, then Y is random only because of ε.
Thus, zero covariance between ε’s also means:

Cov(Yi,Yj)=0Cov(Y_{i},Y_{j})=0

Interpretation: Knowing one household’s food spending tells us nothing about another household’s spending.
This implies that the sample is random and households are independent.

Independence is stronger than zero covariance.
Zero covariance means there is no linear association, but not necessarily full statistical independence.
The model only requires the weaker assumption of zero linear association.


Assumption 4: Constant variance of the error term (Homoscedasticity)

Var(εi)=σ2Var(\varepsilon_{i})=\sigma^2

The variance of ε is constant across observations.
Since X is non-random, Y has the same variance as ε.

However, in real data this is often violated.

Example (scatter plot):
In a sample of 40 households:

  • For low-income households, the range of food expenditure is small.

  • For high-income households, the range is larger.

This pattern indicates increasing variance—the variance of εε (and Y) grows with income, a phenomenon called heteroskedasticity.

Reason:

  • Low-income individuals mostly buy basic items, so spending varies little.

  • High-income individuals have more diverse choices (eating out, organics, imports), leading to greater variability.

Although these assumptions are unrealistic, they are necessary to begin with. Later in the course, we will analyze models that relax them.


Assumption 5 (Optional): Normal distribution of errors

εi∼N(0,σ2)\varepsilon_{i}\thicksim N(0,\sigma^2)

That is, the errors are normally distributed around a mean of zero with constant variance.

This assumption is not crucial because the Central Limit Theorem allows us to derive similar results even when errors are not perfectly normal.

Under this assumption, Y is also normally distributed:

Yi∼N(β1+β2Xi,σ2)Y_{i}\thicksim N(\beta_1+\beta_2X_{i},\sigma^2)

Graphically, the distribution of Y is identical in shape to that of ε, only shifted by the mean β1+β2X\beta_1+\beta_2X


Lecture 2 (Part 2): OLS Estimation

I. What We Are Estimating and Why

A. The Two Population Parameters and the True Line

There are two parameters that exist in the population in nature: β₁ (the intercept) and β₂ (the slope). These two parameters define a true regression line—visualized as a blue line—that captures the true relationship between income and spending on food. The intercept of this true line is β₁, and the slope is β₂.

B. The Fundamental Problem: We Cannot See the True Line

Nobody can observe this true regression line. The reason is the presence of errors—the unobserved disturbances that hide the true relationship between the variables. Because of these errors, all we can actually see are the observations (the points in our sample). Therefore, we must estimate the true line and produce an estimated regression line, also called the fitted line. The blue line is the true line (unobservable); the fitted line is the red line we compute from the data. The goal is to make the fitted line as close as possible to the true line (which we will never see).


II. Data and Sampling

A. Need for a Random Sample

To estimate the model, a random sample from the population of interest is required. If the sample is not random, it will not be representative, and we will not be able to generalize from the sample to the population.

B. The Sample Used

The sample consists of 40 households. For each household we have:

  • Weekly income (measured in units of $100), and

  • Weekly food expenditure (measured in dollars).

Examples (as described):

  • Household #1 earns $369 per week.

  • Household #39 earns $29,140 per week.

  • Household #2 spends $135.98 per week on food.

Summary statistics (as presented):

  • The average household in the sample makes $19,160 per week and spends $283.57 per week on food.

  • Also shown are the median, maximum, and minimum values for income and spending.
    (Clarification of units: in subsequent estimation work, income (X) is handled in $100 units. For example, (X = 20) means $2,000 of weekly income. The later numerical work uses this convention, and the means used there are (Xˉ=19.16)(\bar{X} = 19.16 ) and (Yˉ=283.57)( \bar{Y}= 283.57).)

C. Initial Visualization

The scatter plot of the sample suggests a positive relationship between income and spending on food. Our objective is to draw the fitted regression line—our estimate of the true line.


III. Why Intuition Alone Is Not Enough

A. Intuitive Lines Can Differ Widely

One might try to draw a line that appears to pass “close” to the cloud of points. But there are many lines that could look reasonable by eye and yet be very different.

  • Line 1 yields an estimate that for every $100 increase in income, spending on food increases by $21.

  • Line 2 yields an estimate that for every $100 increase in income, spending on food increases by $12.

These are very different results. Therefore, we need a systematic and accurate procedure for estimating the line.


IV. Ordinary Least Squares (OLS): The Core Idea

A. The Fitted Line and Notation

Assume we draw a fitted regression line:
Y^i=b1+b2Xi\hat{Y}_{i}=b_1+b_2X_{i}
where (b1)( b_1 ) and (b2)( b_2 ) are estimates of the two unknown population parameters β1β₁ and β2β₂ . Any equation with hats(Y^)(\hat{Y}) and (b)’s refers to the fitted (estimated) line, not the true line.

Given (b1)( b_1 ) and (b2)( b_2 ) , the prediction for observation (i) is (Y^i)(\hat{Y}_i) .

B. Residuals vs. True Errors

From the fitted regression line, we compute residuals:
[ε^i=Yi−Y^i.][\hat{\varepsilon}_i = Y_i - \hat{Y}_i .]
Residuals are the differences between the observed values (Yi)(Y_i) and the fitted values(Y^i)(\hat{Y}_i). Residuals are not the true errors. They are our estimates of the unobservable true errors.

  • The true errors (ε’s)\left(\varepsilon’s\right) are the differences between the observations and the unobservable true line (blue).

  • The residuals are the differences between the observations and the fitted line (red).

  • For each fitted line we draw, we obtain a different set of residuals.

C. The OLS Objective: Minimize the Sum of Squared Residuals

The Least Squares idea is to choose the fitted line that yields the smallest possible sum of squared residuals:
SSE=∑i=1nε^i2=∑i=1n(Yi−Y^i)2=∑i=1n(Yi−b1−b2Xi)2.SSE=\sum_{i=1}^{n}\hat{\varepsilon}_{i}^2=\sum_{i=1}^{n}\left(Y_{i}-\hat{Y}_{i}\right)^2=\sum_{i=1}^{n}(Y_{i}-b_1-b_2X_{i})^2.

Why squared residuals?

  • We want the line to be as close as possible to the observations we can see.

  • If we summed raw residuals, positive and negative values would cancel.

  • Squaring makes every term nonnegative, so nothing cancels.

  • We then sum these squares to form the SSE.

(Alternative considered and set aside for now: minimizing the sum of absolute residuals. While it also avoids cancellation, the least squares estimator has extremely good statistical properties. This motivates using squared residuals. These properties will be shown later.)

D. The Best Line in the Least Squares Sense

Let (Y^)( \hat{Y} ) denote the fitted line obtained by OLS. If (Y^∗)( \hat{Y}^* ) denotes any other fitted line (obtained by any other method), then:

SSE(Y^)≤SSE(Y^∗)SSE(\hat{Y})\le SSE(\hat{Y}^{*}) .

That is, the OLS fitted line has a smaller sum of squared residuals than any other fitted line.


V. Solving the Minimization Problem (How the Estimators Are Derived)

(This derivation is to build intuition. It is not required on the exam. The point is to show where the formulas come from rather than to present them as given.)

We minimize

SSE(b1,b2)=∑i=1n(Yi−b1−b2Xi)2SSE(b_1,b_2)=\sum_{i=1}^{n}(Y_{i}-b_1-b_2X_{i})^2

with respect to (b1)(b_1) and (b2)(b_2).

A. First-Order Condition with Respect to (b1)( b_1 )

Differentiate and set to zero:

∂SSE∂b1=−2∑i=1n(Yi−b1−b2Xi)=0.\frac{\partial SSE}{\partial b_1}= -2 \sum_{i=1}^{n} \bigl(Y_i - b_1 - b_2 X_i\bigr) = 0 .

Rearrange:

∑i=1nYi−nb1−b2∑i=1nXi=0.\sum_{i=1}^{n}Y_{i}-nb_1-b_2\sum_{i=1}^{n}X_{i}=0.

Using (∑Yi=nYˉ)( \sum Y_i = n\bar{Y} ) and (∑Xi=nXˉ)( \sum X_i = n\bar{X} ) :

nYˉ−nb1−nb2Xˉ=0⇒b1=Yˉ−b2Xˉ.n\bar{Y}-nb_1-nb_2\bar{X}=0\Rightarrow b_1=\bar{Y}-b_2\bar{X}.

B. First-Order Condition with Respect to (b2)( b_2 )

Differentiate and set to zero:

∂SSE∂b2=−2∑i=1nXi(Yi−b1−b2Xi)=0.\frac{\partial SSE}{\partial b_2}= -2 \sum_{i=1}^{n} X_i \bigl(Y_i - b_1 - b_2 X_i\bigr) = 0 .

Substitute (b1=Yˉ−b2Xˉ)( b_1 = \bar{Y} - b_2 \bar{X} ) into this condition and simplify to obtain the slope estimator (b2)(b_2). The algebra yields:

b2=∑i=1n(Xi−Xˉ)(Yi−Yˉ)∑i=1n(Xi−Xˉ)2.b_2=\frac{\sum_{i=1}^{n} (X_i - \bar{X})(Y_i - \bar{Y})}{\sum_{i=1}^{n} (X_i - \bar{X})^2}.

C. Two Equivalent Presentations and a Useful Identity

There are two interchangeable ways to write the algebra in these derivations:

  1. Deviation-from-the-mean form: use (Xi−Xˉ)(X_i - \bar{X}) and (Yi−Yˉ)(Y_i - \bar{Y}).

  2. Raw form: use (Xi)(X_i) and (Yi)(Y_i) directly.

The bridge between them is the identity:

∑i=1n(Yi−Yˉ)=0and∑i=1n(Xi−Xˉ)=0.\sum_{i=1}^{n} (Y_i - \bar{Y}) = 0\quad\text{and}\quad\sum_{i=1}^{n} (X_i - \bar{X}) = 0.

Because (Xˉ)(\bar{X}) and (Yˉ)(\bar{Y}) are constants (no index ii), they can be factored out of summations, which makes it straightforward to show terms involving sums of deviations from the mean evaluate to zero when appropriate. We will use this trick repeatedly to switch between the two presentations whenever convenient.


VI. What the Estimators Mean

A. Estimator vs. Estimate

  • Estimators (b1,b2)(b_1,b_2) are formulas—they are functions of random sample data and hence are random variables.

  • Estimates are the numerical results obtained when we plug our specific sample into those formulas. Different samples produce different estimates.

B. Two Key Takeaways from the Formulas

  1. The fitted line passes through the point of sample means (Xˉ,Yˉ)(\bar{X}, \bar{Y}).
    This follows immediately from (b1=Yˉ−b2Xˉ)( b_1 = \bar{Y} - b_2 \bar{X} ) : substituting (X=Xˉ)( X = \bar{X} ) gives (Y^=Yˉ)( \hat{Y} = \bar{Y} ) .

  2. Interpretation of (b2)( b_2 ) in covariance–variance form:

    b2=∑(Xi−Xˉ)(Yi−Yˉ)∑(Xi−Xˉ)2=sample covariance(X,Y)sample variance(X)b_2= \frac{\sum (X_i - \bar{X})(Y_i - \bar{Y})}{\sum (X_i - \bar{X})^2}= \frac{\text{sample covariance}(X,Y)}{\text{sample variance}(X)} .

    This is intuitive: the slope measures how strongly (X) and (Y) move together, so it should naturally depend on their covariance, scaled by the variation of (X).


VII. Computing the Least Squares Estimates for the Sample

A. Plugging the Sample into the Estimators

Applying the estimator formulas to the sample yields the fitted regression line:

Y^=b^1+b^2X=83.42+10.21X,\hat{Y}=\hat{b}_1+\hat{b}_2X=83.42+10.21X,

with (X) measured in $100.

  • (b^2=10.21)(\hat{b}_2 = 10.21) : if income increases by $100, the expected weekly spending on food increases by approximately $10.21.

  • (b^1=83.42)(\hat{b}_1 = 83.42) : for a household with zero income (X=0)(X=0), the predicted weekly food spending is $83.42.

B. The Fitted Line Goes Through the Means

The line passes through (Xˉ,Yˉ)(\bar{X}, \bar{Y}). Using the sample means displayed:

  • (Xˉ=19.16)( \bar{X} = 19.16 ) (i.e., $1,916 when income is in $100 units),

  • (Yˉ=283.57)( \bar{Y} = 283.57 ) (i.e., $283.57 per week).
    Graphically, the fitted line goes through this point.


VIII. Interpreting the Intercept Carefully

The intercept is the predicted spending when (X=0)(X = 0) (no income). Here, the estimate implies 83.42perweek</strong>forahouseholdwithzeroincome.Whetherthisisreasonableisanempiricalandmodelingquestion:</p><ul><li><p>It<strong>could</strong>bereasonableifsuchhouseholdsreceiveassistance(e.g.,foodstamps)orhelpfromfamily/friends.</p></li><li><p>It<strong>might</strong>beinaccurateif(b^1)isimpreciselyestimatedorifitiscapturingtheinfluenceofomittedvariablesorotherissues.</p></li></ul><p>Thiswillbeexaminedlater;fornow,wetaketheestimateasgivenandproceed.</p><divdata−type="horizontalRule"><hr></div><h3id="497aef29−4c6f−4cf0−a4d7−29061ac6070c"data−toc−id="497aef29−4c6f−4cf0−a4d7−29061ac6070c"collapsed="false"seolevelmigrated="true">IX.UsingtheEstimatedModelforPrediction</h3><h4id="829c44cc−b42d−4617−900c−0a8e048136e9"data−toc−id="829c44cc−b42d−4617−900c−0a8e048136e9"collapsed="false"seolevelmigrated="true">A.ExamplePredictionataSpecificIncome</h4><p>Predictweeklyfoodexpenditureforahouseholdwithaweeklyincomeof<strong>83.42 per week</strong> for a household with zero income. Whether this is reasonable is an empirical and modeling question:</p><ul><li><p>It <strong>could</strong> be reasonable if such households receive assistance (e.g., food stamps) or help from family/friends.</p></li><li><p>It <strong>might</strong> be inaccurate if ( \hat{b}_1 ) is imprecisely estimated or if it is capturing the influence of omitted variables or other issues.</p></li></ul><p>This will be examined later; for now, we take the estimate as given and proceed.</p><div data-type="horizontalRule"><hr></div><h3 id="497aef29-4c6f-4cf0-a4d7-29061ac6070c" data-toc-id="497aef29-4c6f-4cf0-a4d7-29061ac6070c" collapsed="false" seolevelmigrated="true">IX. Using the Estimated Model for Prediction</h3><h4 id="829c44cc-b42d-4617-900c-0a8e048136e9" data-toc-id="829c44cc-b42d-4617-900c-0a8e048136e9" collapsed="false" seolevelmigrated="true">A. Example Prediction at a Specific Income</h4><p>Predict weekly food expenditure for a household with a weekly income of <strong>2,000. Since income is measured in $100, this corresponds to (X = 20). Substitute into the fitted line:

Y^=83.42+10.21×20=287.61\hat{Y} = 83.42 + 10.21 \times 20 = 287.61 .

Prediction: $287.61 per week on food.


X. Income Elasticity

A. Definition and Formula

Income elasticity measures the responsiveness of spending to income in percentage terms. It answers: if income increases by 1%, by how many percent does spending change?

For the linear fitted model, the elasticity of (Y) with respect to (X) at a point (X,Y) is:

E=b2⋅XYE=b_2\cdot\frac{X}{Y}

Elasticity changes with the point (X,Y); it is not constant in the linear model.

B. Elasticity at the Means (A Common Practice)

Because elasticity depends on the specific (X,Y), it is typical to evaluate it at the sample means to characterize an “average” household:

E=b2⋅XˉYˉE=b_2\cdot\frac{\bar{X}}{\bar{Y}}

Using the estimates and means:

E=10.21×19.16283.57≈0.71E=10.21\times\frac{19.16}{283.57}\approx0.71

Interpretation: A 1% increase in weekly household income leads, on average, to a 0.71% increase in weekly household food expenditure.


XI. Summary of the OLS Outcome and Its Logic

  1. Because the true line is hidden by errors, we estimate a fitted line using a random, representative sample.

  2. The OLS method chooses the line that makes the sum of squared residuals as small as possible.

  3. The OLS estimators are obtained by minimizing the SSE with respect to (b1)(b_1) and (b2)(b_2).

    • (b1=Yˉ−b2Xˉ)b_1 = \bar{Y} - b_2 \bar{X} )

    • (b2=∑(Xi−Xˉ)(Yi−Yˉ)∑(Xi−Xˉ)2)( b_2 = \dfrac{\sum (X_i - \bar{X})(Y_i - \bar{Y})}{\sum (X_i - \bar{X})^2} )

  4. The fitted line passes through (Xˉ,Yˉ).\bar{X}, \bar{Y}).

  5. The slope (b2)(b_2) equals sample covariance of (X,Y) divided by the sample variance of (X).

  6. With the estimated line (Y^=83.42+10.21X)(\hat{Y} = 83.42 + 10.21 X) (with (X) in $100), predictions and elasticity calculations follow directly.

  7. The intercept’s literal interpretation at (X=0) may or may not be substantively meaningful and will be revisited later.

Lecture 2 (Part 3): Properties of Estimators and Unbiasedness of OLS

I. Estimators as Random Variables

A. An estimator is a formula you can apply to any sample.
When you plug your sample’s values into estimators such as (β^1)( \hat{\beta}_1 ) and (β^2)( \hat{\beta}_2 ), you get point estimates for that specific sample.

B. Because the estimator is built from a random sample, the estimator itself is a random variable.
Therefore, each estimator has a mean and a variance.

C. By examining the mean and variance of an estimator, we can judge whether it is a good estimator or not such a good one.


II. Three Key Properties We Want

A. Unbiasedness — You want the estimator to be unbiased.
B. Efficiency — You want the estimator to be efficient.
C. Consistency — You want the estimator to be consistent.

The plan is to define these properties precisely, then check whether the least squares (LS/OLS) estimators have them. This will let us compare least squares estimators to alternatives and decide whether they are good or whether better ones exist.


III. Unbiasedness: What It Means

A. Definition

Unbiasedness means that if you took estimates from many samples and averaged them, that average would approach the true parameter. Formally, if (E[β^]=β)(E[\hat{\beta}] = \beta), the estimator is unbiased.

B. What unbiasedness does not mean

Unbiasedness does not mean that the estimate from any one sample is close to the true parameter. It makes no promise about a single estimate; it is a statement about averages across many samples.

C. Illustration with multiple samples

Consider 10 samples, each with 40 observations, all drawn from the same population (e.g., U.S. households). Look at estimates of (β^2)( \hat{\beta}_2 ), interpreted here as “by how much expected food spending goes up when income increases by $100.” You might see estimates like $6.10, $8, or $13.

  • What is the true parameter—6, 13, or 27? We cannot know, and in fact we will never be able to know the true parameter exactly.

  • But if the estimator is unbiased, there is still something very important you can say: with many, many samples, if you compute the estimate for each sample and then take the average of those estimates, on average you will get the true parameter (β2).( \beta_2 ).

  • In other words: on average, we are right. This says nothing about whether your particular sample’s estimate is close or far from the truth.


IV. Goal: Show Least Squares Is Unbiased

A. Objective

We want to prove that the least squares estimator is unbiased. Mathematically, we want to show:

E[β^2]=β2.E[\hat{\beta}_2] = \beta_2.

B. About the Algebra

The proof involves some mathematics. It is presented to enhance understanding—you will not be asked to derive or prove formulas on the exam. The payoff is seeing exactly why the assumptions matter and what happens if they do not hold.


V. Setting Up the Proof

A. Two forms of the OLS estimator

The least squares estimator for (β2)( \beta_2 ) can be written in two ways. One is the form in terms of deviations from the mean. That form is more convenient here.

B. Define Weights

Denote by (wi)( w_i ) the weight that comes from that deviations form. Then the estimator can be written as a linear combination:

β^2=∑i=1NwiYi,\hat{\beta}_2=\sum_{i=1}^{N}w_{i}Y_{i},

where (N) is the number of observations. This shows directly that the least squares estimator is a linear estimator: (w1Y1+w2Y2+⋯+wNYN).(w_1Y_1 + w_2Y_2 + \cdots + w_NY_N).

C. Substitute the model for (Yi).(Y_i).

Use the (assumed correct) model:

Yi=β1+β2Xi+εi.Y_i = \beta_1 + \beta_2 X_i + \varepsilon_i.

Substituting into the estimator gives:

β^2=∑wi(β1+β2Xi+εi)=β1∑wi+β2∑wiXi+∑wiεi\hat{\beta}_2=\sum w_{i}(\beta_1+\beta_2X_{i}+\varepsilon_{i})=\beta_1\sum w_{i}+\beta_2\sum w_{i}X_{i}+\sum w_{i}\varepsilon_{i}

D. What we need to show

To isolate (β2)\beta_2 ), we will show:

  1. (∑wi=0)( \displaystyle \sum w_i = 0 )

  2. (∑wiXi=1)( \displaystyle \sum w_i X_i = 1 )

If both hold, then:

β^2=β2+∑wiεi.\hat{\beta}_2 = \beta_2 + \sum w_i \varepsilon_i.


VI. Establishing the Two Key Identities

A. Showing (∑wi=0).(\sum w_i = 0).

  1. Recall how (wi)( w_i ) is defined from deviations from the mean.
    In this deviations form, the numerator is (Xi−Xˉ)(X_{i}-\bar{X}) and the denominator is the sum of squared deviations (∑(Xi−Xˉ)2)( \sum (X_i - \bar{X})^2 ) . Thus:

    ∑wi=∑(Xi−Xˉ)∑(Xi−Xˉ)2.\sum w_i= \frac{\sum (X_i - \bar{X})}{\sum (X_i - \bar{X})^2}.

  2. The sum of deviations from the mean is zero:

    ∑(Xi−Xˉ)=0.\sum (X_i - \bar{X}) = 0.

  3. Therefore,

    ∑wi=0,\sum w_i = 0,

    so the term (β1∑wi)( \beta_1 \sum w_i ) is zero.

B. Showing(∑wiXi=1).(\sum w_i X_i = 1).

  1. Using the same deviations form, the numerator becomes the same sum that appears in the denominator:

    ∑wiXi=∑(Xi−Xˉ)Xi∑(Xi−Xˉ)2.\sum w_i X_i= \frac{\sum (X_i - \bar{X}) X_i}{\sum (X_i - \bar{X})^2}.

  2. This ratio equals 1 because the denominator is exactly the same sum of squared deviations that the numerator reduces to in this setup. Hence:

    ∑wiXi=1.\sum w_i X_i = 1.

C. Result after the two identities

With(∑wi=0)( \sum w_i = 0 ) and (∑wiXi=1)( \sum w_i X_i = 1 ) , we have:

β^2=β2+∑wiεi.\hat{\beta}_2 = \beta_2 + \sum w_i \varepsilon_i.


VII. Taking Expectations

A. Expectation of the estimator.

Take expectations of both sides:

E[β^2]=E[β2+∑wiεi]=β2+∑wi,E[εi],E[\hat{\beta}_2]=E\left[\beta_2+\sum w_{i}\varepsilon_{i}\right]=\beta_2+\sum w_{i},E[\varepsilon_{i}],

since (β2)( \beta_2 ) is a constant (a parameter), not a random variable.

B. The convenience assumption about (X).

For this demonstration, adopt the simplifying assumption that (X) is not treated as random. (This is only for convenience; the proof can be done without it, but it would require conditional expectations and would be more involved.) With this convenience assumption, the weights (wi)( w_i ) can be taken outside the expectation.

C. Zero mean of the error term.

By the model assumption, (E[εi]=0).( E[\varepsilon_i] = 0 ). Therefore:

∑wiE[εi]=0.\sum w_{i}E[\varepsilon_{i}]=0.

D. Conclusion for (β^2).(\hat{\beta}_2).

Thus,

E[β^2]=β2.E[\hat{\beta}_2] = \beta_2.

This proves that the least squares estimator (β^2)( \hat{\beta}_2 ) is unbiased under the stated assumptions.


VIII. The Assumptions Used and Their Role

A. Correct model specification

We assumed we wrote the correct model that represents reality precisely, so substituting (Yi=β1+β2Xi+εi)( Y_i = \beta_1 + \beta_2 X_i + \varepsilon_i ) is valid.

B. Zero-mean error

We assumed (E[εi]=0).( E[\varepsilon_i] = 0 ).

C. Convenience treatment of (X)

For simplicity in the algebra, (X) was treated as non-random so that the weights (wi)( w_i ) could be taken outside the expectation. This is a convenience, not a necessity for the truth of the result.

D. What if assumptions fail?

If these assumptions do not hold, then we are in trouble—the estimator will generally not be unbiased. This issue will be addressed later.


IX. Unbiasedness of (β^1)( \hat{\beta}_1 )

By analogous reasoning, the expected value of the estimator (β^1)( \hat{\beta}_1 ) equals the true parameter (β1)( \beta_1 ) under the same assumptions:

E[β^1]=β1.E[\hat{\beta}_1] = \beta_1.



Lecture 2 (Part 4): Variance and Efficiency

I. Concept of Variance and Efficiency

A. Comparing Unbiased Estimators

  • We look at two distributions of unbiased estimators.

  • Both are unbiased because the mean of each estimator equals the true parameter β.

  • The estimator with the smaller variance is preferable, since it is more likely to produce an estimate closer to β.

  • Therefore, among unbiased estimators, we prefer the one with the smaller variance.

B. Definition of Efficiency

  • Efficiency compares two candidate unbiased estimators:

    • If one estimator has a lower variance than another, it is said to be more efficient.

  • When comparing unbiased estimators, we always choose the one with the smaller variance because:

    • It gives an estimate that, with higher probability, lies closer to the true parameter.

  • This is always about probabilities:

    • It is possible to draw an estimate from a large-variance distribution that happens to fall close to β.

    • It is also possible to draw an estimate from a small-variance distribution that falls far from β.

    • However, it is more likely to get a closer estimate when variance is smaller.

  • Thus, when possible, we always prefer an estimator with a smaller variance.


II. Variance of Least Squares Estimators

A. General Setup

  • We want to study what affects the variance of the least-squares estimators (b1)( b_1 ) and (b2).( b_2 ).

  • We already know the least-squares estimators are unbiased, meaning

    E[bi]=βiE[b_i] = \beta_i for each (i=1,2)( i = 1, 2 ).

  • From this foundation, we can derive expressions for the variances of these estimators and identify the key factors that influence them.


III. Determinants of the Variance of Least Squares Estimators

A. Effect of Error Variance (σ2)( σ² )

  • Both variances of (b1)( b_1 ) and (b2)( b_2 ) include the term σ2σ², the variance of the random error.

  • As σ2σ²increases, the variance of the estimator also increases.

Explanation
  • A larger σ² means more noise and uncertainty in the model, making it harder to estimate the true relationship precisely.

  • Example:

    • Suppose X = income and Y = spending on food.

    • The true line shows that for every $100 of income, households spend $15 on food.

    • If σ² is large, households with similar income show very different spending levels—some on diets, some shopping more, some hosting guests.

    • The wide dispersion makes it difficult to estimate the line accurately, increasing estimator variance.

    • If σ² is small, points cluster tightly around the line, making estimation more precise and reducing variance.


B. Effect of Variance of X (Var(X))

  • Both formulas for the variances of (b1)( b_1 ) and (b2)( b_2 ) contain Var(X).

  • As Var(X) increases, the variance of the estimator decreases.

Intuition
  • When X values vary a lot (large range of X), it becomes easier to fit the line precisely.

  • When X values are concentrated (small range), it becomes harder to estimate the slope accurately.

Graphical Illustration
  • Figure A: Small range of X → small Var(X) → hard to see the true relationship.

    • Many possible lines could fit the data (steep or flat).

    • Hence, estimator variance is high.

  • Figure B: Large range of X → large Var(X) → easier to see the true relationship.

    • Line is more clearly defined.

    • Estimator variance is lower.


C. Effect of ΣXi2Σ Xᵢ² (Sum of Squared X Values)

  • This affects only the variance of (b1)( b_1 ) (the intercept), not (b2)( b_2 ) (the slope).

  • ΣXi2Σ Xᵢ² is large when the sample includes many high-income observations;
    it is small when most observations have low income.

Reasoning
  • (b1)( b_1 ) estimates the intercept, i.e., predicted spending when income=0.income = 0.

  • It is easier to estimate this intercept precisely if we have many low-income households.

  • If we mainly observe high-income households, estimating the intercept becomes difficult.

  • Therefore:

    • Many high-income values → large ΣXi2Σ Xᵢ² → variance of (b1)( b_1 ) is high.

    • Many low-income values → smaller ΣXi2Σ Xᵢ² → variance of (b1)( b_1 ) is lower.

Graphical Explanation
  • If there are many data points near income = 0, the intercept (spending at zero income) can be estimated accurately.

  • If all data points are far from income = 0, estimating that intercept precisely becomes difficult, raising variance.

  • Example: In the model, the intercept was about $83 for households with zero income.

    • This might be inaccurate because we have few such households in the data.

    • The imprecision could explain why the estimate seems implausible (e.g., “households with no income spending money on food”).


D. Effect of Sample Size ( N )

  • As N increases, both variances Var(b1)Var(b_1) and Var(b2)Var(b_2) decrease.

  • The reasoning:

    • Larger N means the denominator in variance expressions grows (because we sum over N), reducing overall variance.

    • More data → more precision.


IV. Consistency of Least Squares Estimators

A. Definition of Consistency

  • An estimator is consistent if, as N → ∞, the estimates produced converge to the true parameter value.

  • Formally:

    • As sample size increases, the sampling distribution of the estimator becomes more concentrated around the true parameter.

  • Graphically:

    • As N increases, the estimator’s distribution tightens around the mean.

    • There is a higher probability that the estimate will be closer to the true parameter.

  • This property is also called convergence in probability.

B. Relation between Consistency and Variance

  • As sample size increases:

    • Variance of the estimator decreases, meaning estimates cluster more tightly around β.

    • Probability of being close to the true parameter increases.

  • A “bad draw” remains possible, but increasingly unlikely.


V. Unbiasedness vs. Consistency

A. Key Difference

  • These two concepts are very different.

Concept

What It Says

Focus

Unbiasedness

The expected value of the estimator equals the true parameter.

Center of the sampling distribution.

Consistency

As N increases, the estimator’s distribution becomes concentrated around the true parameter.

Shape of the sampling distribution as sample size grows.

B. Interpretation

  • Unbiasedness:

    • Imagine drawing many samples and computing an estimate for each.

    • If you average all these estimates, you will get the true parameter on average.

    • This does not mean any one sample gives a good estimate—only that the mean of all possible estimates equals β.

  • Consistency:

    • Focuses on what happens as you increase your sample size.

    • It says that with a larger N, there is a higher probability that the estimate from your own sample will be close to β.


VI. Example — Unbiased but Not Consistent

A. Setup

  • Suppose ( X ) is a random variable distributed as N(μ,σ2).N(\mu,\sigma^2).

  • We draw a sample (X1,X2,…,XN)( X_1, X_2, …, X_N ) and want to estimate μμ.

B. Candidate Estimator

  • Typical estimator: the sample mean,

    Xˉ=1N∑i=1NXi\bar{X} = \frac{1}{N}\sum_{i=1}^{N} X_i

  • But now define a different estimator:

    μ^=X1\hat{\mu} = X_1

    (i.e., use only the first observation).

C. Properties

  • Unbiasedness:

    • E[X1]=μ,E[X_1] = \mu, so the estimator is unbiased.

    • Over many samples, if we always take the first draw and average all those X₁’s, we recover μ.

  • Inconsistency:

    • The variance of X₁ is σ2σ², which does not shrink as N increases.

    • Increasing sample size does not help, because we always use only one observation.

    • Hence, this estimator is not consistent: it does not converge to μ as N → ∞.


VII. Summary

  • Efficiency: Among unbiased estimators, choose the one with the smaller variance.

  • Variance determinants:

    • Largerσ2σ² → larger estimator variance.

    • Larger Var(X)Var(X) → smaller estimator variance.

    • Larger ΣXi2Σ Xᵢ² → affects only b₁ (intercept) variance.

    • Larger N → smaller variance for both b₁ and b₂.

  • Consistency: As N increases, estimator variance decreases and estimates converge to the true parameter.

  • Unbiasedness vs Consistency: Unbiasedness concerns expected values; consistency concerns sample-size behavior.

  • Example: (X1)( X_1 ) is unbiased but not consistent.


Lecture 2 (Part 5): Gauss-Markov Theorem and Variance of the Least Squares Estimators

I. The Gauss-Markov Theorem

A. Background and Purpose

We already know that the least squares estimators are unbiased and consistent.
The Gauss-Markov theorem gives an additional and very important property of these estimators.


Although the proof of the theorem is mathematically complex and not covered here, its result is essential to understand why the least squares estimators are so widely used in econometrics.

B. Statement of the Theorem

The Gauss-Markov theorem applies under Assumptions 2–4 of the classical linear regression model — the assumptions that deal with the behavior of the random error term (ui)(u_i):

  1. The expected value of the random error is zero:

    E(ui)=0E(u_i) = 0

  2. The variance of the random error is constant across all observations (homoskedasticity):

    Var(ui)=σ2\text{Var}(u_i) = \sigma^2

  3. The covariance between any two random errors is zero (no autocorrelation):

    Cov(ui,uj)=0for i≠j\text{Cov}(u_i, u_j) = 0 \quad \text{for } i \neq j

Under these three assumptions, the theorem states that the least squares estimators (B1)(B_1) and (B2)(B_2) have the smallest possible variance among all estimators that are both linear and unbiased estimators of the true parameters (β1)(\beta_1) and (β2).(\beta_2). Because of this property, these estimators are called the Best Linear Unbiased Estimators, or BLUE.


II. Understanding What “Best” Means

A. The Meaning of “Best”

The word “best” in this context does not mean perfect or always correct.
It means that within the specific family of estimators that are linear and unbiased, the least squares estimators have the smallest variance.

Having the smallest variance means they are most efficient — they fluctuate the least around the true parameter values in repeated samples.

So, if we were to repeatedly draw samples and estimate the parameters each time using least squares, the distribution of those estimates would be more tightly centered around the true parameter than any other linear and unbiased estimator.

B. Why This Matters

This is an important reason why the least squares estimator has become the dominant method in econometrics.
If one insists on using estimators that are linear and unbiased — and this is typically the goal when working with data — then the problem of choosing an estimator is already solved.
There is no need to search for another method, because no other linear and unbiased estimator will yield lower variance.

C. Summary of the Key Idea

Within the class of linear and unbiased estimators, least squares estimators are the most precise estimators possible.
This is what gives them their “best” property in the Gauss-Markov sense.


III. Limitations of the Gauss-Markov Theorem

A. Best Only Among Linear and Unbiased Estimators

The Gauss-Markov theorem compares the least squares estimators only to other estimators that are both linear and unbiased.
It does not claim that they are best among all possible estimators.

If for some reason an estimator that is nonlinear or biased is used, it is theoretically possible that such an estimator could have a smaller variance than the least squares estimators.

Thus, the “best” property of the least squares estimator is conditional — it applies only within the class of estimators that are linear and unbiased.

B. Dependence on Assumptions 2–4

For the Gauss-Markov theorem to hold, all assumptions about the random error (Assumptions 2–4) must hold true.
If any of these assumptions are violated — for example, if the variance of the error is not constant or if there is correlation among the error terms — then the least squares estimators will no longer be BLUE.

This point becomes crucial later in the course when these assumptions are relaxed.
Once those assumptions no longer hold, the least squares estimators lose their “best” property and can become inefficient or even biased.

C. Independence from Other Assumptions

It is also important to recognize what the Gauss-Markov theorem does not depend on.
It does not rely on:

  1. The assumption of normality of the errors (Assumption 5), or

  2. The assumption that (X)( X ) is a random variable (Assumption 1).

The theorem depends only on the assumptions about the random error (2–4).

D. Estimator vs. Estimate

The Gauss-Markov theorem applies to the estimator, which is the formula, not to the estimate, which is the numerical value obtained from a specific sample.

An individual sample might produce an estimate that is far from the true parameter, but the theorem concerns the general behavior of the estimator across repeated samples.
It means that the least squares estimator, as a mathematical rule, has desirable properties — specifically, it tends to give estimates closer to the true parameter than other linear and unbiased estimators.


IV. The Central Limit Theorem (CLT) and Normality

A. Connection Between Least Squares and the CLT

The Central Limit Theorem provides a connection between large samples and normality.
It says that if Assumptions 2–4 hold and the sample size is sufficiently large, the sampling distribution of the least squares estimators can be approximated by a normal distribution.

In other words, even if the random errors themselves are not normally distributed, the estimators (B1)(B_1) and (B2)(B_2) will be approximately normal when the sample is large enough.

B. Practical Importance

In modern applications, we usually work with large samples — typically 100 or several hundred observations.
Therefore, the normal approximation is very good.
For this reason, assuming normality of the estimators is often treated as a practical simplification that does not significantly affect results.

C. Distribution of the Estimators

Under these conditions, the least squares estimators can be described as normally distributed:

B1∼N(β1,Var(B1))B_1 \sim N(\beta_1, \text{Var}(B_1))

B2∼N(β2,Var(B2))B_2 \sim N(\beta_2, \text{Var}(B_2))

This means that each estimator has a mean equal to its true parameter value and a variance equal to the theoretical variance we have already derived.


V. Estimating the Variance of the Estimators

A. Goal

So far, we have calculated the point estimates (B1)(B_1) and (B2)(B_2) for the coefficients.
Now we want to estimate the variance of these estimators and, from that, obtain the standard errors.

B. The Unknown Parameter (σ2)(\sigma^2)

The formulas for the variances of (B1)(B_1) and (B2)(B_2) include a parameter (σ2),(\sigma^2), which represents the variance of the random error.
However, this quantity is unknown — it is a property of the population and not directly observable in our sample.

Therefore, before we can calculate the variances of (B1)(B_1) and (B2)(B_2) , we must first estimate (σ2).(\sigma^2).

C. Using Residuals as a Substitute

We cannot observe the true random errors, but we can calculate the residuals from our sample regression.
Residuals are our estimates of the unobservable errors.

To estimate the variance of the random errors, we take the following intuitive steps:

  1. Square each residual.

  2. Sum all the squared residuals.

  3. Divide by the sample size to get their average.

This gives an intuitive estimator for the variance of the random error.

D. Problem with the Naive Estimator

Although this intuitive estimator is consistent, meaning it converges to the true value as the sample grows, it is still biased.
That is, on average, it underestimates the true variance.


VI. The Unbiased Estimator for the Variance of the Random Error

A. Correcting for Bias

The bias can be corrected easily by adjusting the denominator.
Instead of dividing by (n), we divide by (n−2):(n - 2):

σ^2=∑ei2n−2\hat{\sigma}^2 = \frac{\sum e_i^2}{n - 2}

Here:

  • (n) is the number of observations.

  • (2) corresponds to the number of parameters estimated in the model (the intercept and the slope).

The term (n−2)(n - 2) represents the degrees of freedom of the model.

B. Generalization to Multiple Regression

In a multiple regression model with (k) parameters, the formula becomes:

σ^2=∑ei2n−k\hat{\sigma}^2 = \frac{\sum e_i^2}{n - k}

C. Interpretation

This corrected estimator is both consistent and unbiased, making it a reliable estimate of the true variance of the random error.
Once we have this estimate, we can plug it into the formulas for the variances of (B1)(B_1) and (B2)(B_2) to compute the estimated variances and standard errors.

The standard error is simply the square root of the estimated variance and gives us a measure of how much the estimator fluctuates around the true parameter value.


VII. Understanding Regression Output Conceptually

When looking at regression results, two quantities are essential:

  1. The coefficient (point estimate) — the estimated value of the parameter (e.g., (B1)(B_1) or (B2))(B_2)) .

  2. The standard error — an estimate of the variability of that coefficient, calculated using the formulas above.

The standard error provides the basis for inference — it indicates how precisely the coefficient has been estimated.
A smaller standard error means a more precise estimate.


VIII. Summary of Key Concepts from Lecture 2

A. Model Assumptions

  • The classical linear regression model relies on assumptions about the random error term.

  • These assumptions (expected error = 0, constant variance, no correlation among errors) are critical for the Gauss-Markov theorem to hold.

  • Though these assumptions may seem simple or “naive,” they are not always true in practice. Later in the course, these will be relaxed.

B. Properties of Least Squares Estimators

  1. Unbiasedness — On average, the estimators equal the true parameters.

  2. Consistency — As the sample size increases, estimates get closer to the true parameters.

  3. Efficiency (BLUE property) — Among all linear and unbiased estimators, least squares estimators have the smallest variance.

C. Variance and Standard Error Estimation

  • The variance of the random error (σ2)(\sigma^2) must be estimated using residuals.

  • To make the estimate unbiased, divide the sum of squared residuals by (n−2).(n - 2).

  • Substitute this estimated variance into the formulas for the variances of (B1)(B_1) and (B2)(B_2), then take square roots to obtain their standard errors.

D. Why Standard Errors Matter

Point estimates alone are not enough for meaningful interpretation.
We must also consider the standard errors to understand the distribution and precision of the estimators.
These two quantities together — the coefficient estimate and its standard error — allow us to make inferences about the true parameters in the population.

E. Transition to Next Topic

The next step will be to use these point estimates and their standard errors to make statistical inferences about the true parameters — for example, testing hypotheses and constructing confidence intervals to learn about the real relationship between variables such as income and food expenditure.

Lecture 3 (Part 1): Confidence Intervals 

I. The Motivation for Confidence Intervals

A. Point Estimates and Their Limitations

By now, we know how to obtain the point estimate (b2),( b_2 ), which is our estimate of the unknown population parameter (β2).( \beta_2 ).


However, a single point estimate alone is not very informative. The estimator itself is a random variable, and the point estimate we obtain from our particular sample is just one of many possible realizations of that random variable.

It is possible to be lucky and have a point estimate close to the true value of (β2),( \beta_2 ), or to be unlucky and have one far away from it. Since the true parameter (β2)( \beta_2 ) is unknown, we can never be certain whether our estimate is close to it or not.

Despite this uncertainty, we can still learn valuable information about (β2)( \beta_2 ) using two statistical techniques:

  1. Confidence intervals

  2. Hypothesis testing


II. Conceptual Idea of the Confidence Interval

A. Illustration Using Multiple Samples

Different samples yield different point estimates.
For example:

  • Sample 1: Increasing income by $100 increases mean food spending by $6.48.

  • Sample 6: Increasing income by $100 increases mean food spending by $13.55.

From one sample alone, we cannot conclude much about the true effect of income on spending.


However, if we observe estimates from 10 or even 100 samples, we start to see a pattern. It becomes unlikely that (β2)( \beta_2 ) equals 25, for instance. Instead, we might reason that the true (β2)( \beta_2 ) lies roughly between 5 and 15.

This reasoning builds a range of likely values for the true parameter — the essence of the confidence interval.


III. Building a Confidence Interval from One Sample

A. The Challenge

In practice, we usually only have one sample. The task is to construct a range (confidence interval) for the unknown (β2)( \beta_2 ) using only information from that single sample.

B. The Solution: Using the Standard Error

The sample provides not only a point estimate but also an estimate of the standard error of the estimator.
The standard error measures how much the estimator varies around its mean — and the mean, in theory, equals the true parameter (β2).( \beta_2 ).

Using both the point estimate and the estimated standard error, we can construct the confidence interval.


IV. Mathematical Derivation of the Confidence Interval

A. Starting Point: The OLS Estimator

The least squares estimator (b2)( b_2 ) is unbiased with mean (β2)( \beta_2 ) and known variance. However, it is not yet standardized.

B. Standardizing the Estimator

To standardize, we subtract the mean and divide by the standard error:

Z=b2−β2se(b2)Z = \frac{b_2 - \beta_2}{\text{se}(b_2)}

This statistic ( Z ) follows a standard normal distribution with mean 0 and variance 1.

C. Problem: Unknown Population Variance

We do not know the true variance (σ2).( \sigma^2 ). Instead, we estimate it using the sample residuals, denoted (σ^2).( \hat{\sigma}^2 ).

Replacing (σ2)( \sigma^2 ) with (σ^2)( \hat{\sigma}^2 ) introduces sampling noise, so the resulting statistic does not follow a standard normal distribution.

D. Solution: The t-Statistic

The statistic now follows a t-distribution with (n−2)( n - 2 ) degrees of freedom (since two parameters, (β1)( \beta_1 ) and (β2),( \beta_2 ), are estimated).

t=b2−β2se(b2)∼tn−2t = \frac{b_2 - \beta_2}{\text{se}(b_2)} \sim t_{n-2}


V. Properties of the t-Distribution

A. Shape and Characteristics

  • The t-distribution is bell-shaped and centered at zero.

  • It resembles the standard normal distribution but has thicker tails and a larger variance.

  • The difference arises because we use an estimated variance (σ^2)( \hat{\sigma}^2 ) rather than the true (σ2),( \sigma^2 ), which adds noise and increases uncertainty.

B. Degrees of Freedom and Convergence

The shape of the t-distribution depends on the degrees of freedom (df). As df increases — that is, as the sample size (n)( n ) increases — the t-distribution approaches the standard normal distribution.

For instance, with 40 observations and 2 estimated parameters, we have 38 degrees of freedom.
At large (n)( n ) (roughly (n>100)),( n > 100 )), the t and z distributions are nearly identical.


VI. Reading the t-Table and Finding Critical Values

A. Example

For 38 degrees of freedom and a 95% confidence level (α=0.05):( \alpha = 0.05 ):

  • The two tails together contain 5% of the probability, so each tail has 2.5%.

  • The t-critical value for 97.5% (the upper tail cutoff) is (t0.975,38=2.024).( t_{0.975, 38} = 2.024 ).

This means there is a 97.5% probability that a t-distributed variable will be less than 2.024 and a 2.5% probability that it exceeds it.


VII. Constructing the Confidence Interval

A. Example

b2±tα/2,n−2×se(b2)b_2 \pm t_{\alpha/2, n-2} \times \text{se}(b_2)

This gives the lower and upper bounds for (β2).( \beta_2 ).

B. Example: Food Expenditure Model

Given:

  • b2=10.21b_2=10.21

  • se(b2)=2.04\text{se}(b_2)=2.04

  • t0.975,38=2.024 t_{0.975, 38} = 2.024 

Then:
CI: (5.97,14.45)\text{CI: } (5.97, 14.45)

Interpretation:
With 95% confidence, the true parameter (β2)( \beta_2 ) lies between 5.97 and 14.45.
In words: increasing income by $100 increases food spending by between $5.97 and $14.45 per week, with a 95% probability.


VIII. Correct Interpretation of a Confidence Interval

A. What It Does NOT Mean

It does not mean the true parameter definitely lies within the interval.
We will never know for sure whether our particular interval contains the true (β2).( \beta_2 ).

B. What It Actually Means

If we repeatedly drew samples from the population and constructed confidence intervals using this same procedure, 95% of those intervals would contain the true (β2).( \beta_2 ).
Thus, the method itself is correct 95% of the time.

C. Intuition

Even though we cannot know if our single interval contains the true parameter, it would be quite unlucky if it did not—only 5% of properly constructed intervals miss it.


IX. Confidence Intervals in Regression Output

Regression software (like Stata) automatically provides the 95% confidence intervals for each parameter.

Example:

  • 95% CI for (β2):( \beta_2 ): (5.97, 14.45)

  • 95% CI for (β1):( \beta_1 ): (-4, 171)

The intercept’s interval includes zero, which means we cannot rule out that households with zero income may spend nothing on food. The interval also allows for negative values, which are not meaningful economically but remind us to apply common sense in interpretation.


X. Adjusting the Confidence Level

A. Changing α

If α = 1% (99% confidence interval), the critical value becomes larger (t0.995,38=2.712).( t_{0.995, 38} = 2.712 ).

The point estimate and standard error stay the same, but the interval widens:

CI: b2±2.712×se(b2)\text{CI: } b_2 \pm 2.712 \times \text{se}(b_2)

The result is a wider range, representing greater confidence but less precision — the classic “no free lunch” tradeoff.


XI. Confidence Interval for a Linear Combination of Parameters

A. Motivation

Sometimes we need a confidence interval not for a single coefficient but for a predicted mean value — a linear combination of parameters.

Example:
Predicted weekly food spending for a household with $2000 income: $287.61.
This is only a point estimate. We want a 95% confidence interval for the mean expected spending.

B. Computation

Y^=b1+b2X\hat{Y} = b_1 + b_2X

We use the estimated coefficients and their covariance matrix.

The variance of this linear combination is computed as:

Var(Y^)=Var(b1)+X2Var(b2)+2XCov(b1,b2)\text{Var}(\hat{Y}) = \text{Var}(b_1) + X^2\text{Var}(b_2) + 2X\text{Cov}(b_1, b_2)

The standard error is the square root of this variance.

C. Result

Substituting into the confidence interval formula:

Y^±tα/2,n−2×se(Y^)\hat{Y} \pm t_{\alpha/2, n-2} \times \text{se}(\hat{Y})

This yields:

95%CI: (258.91,316.31)\text{CI: }(258.91,316.31)

Interpretation:
With 95% confidence, the average weekly food expenditure for households earning $2000 per week lies between $258.91 and $316.31.






Lecture 3 (Part 2): Hypothesis Testing

I. What We’re Doing and Why

A. The Aim of Hypothesis Testing

The aim is to say something about the true parameter (β).(\beta). Confidence intervals give a likely range for (β).(\beta).

B. Connection to Population Conjectures

Hypothesis testing tackles the same goal by comparing a conjecture about the population with what the sample shows.

C. The Null and Alternative Hypotheses

Begin with a belief about the population (e.g., “for every $100 increase in income, mean food spending increases by $15”). Call this belief the null hypothesis. Keep it unless the sample provides convincing evidence against it; then reject it and adopt an alternative hypothesis.


II. How We State Hypotheses

A. The Null Hypothesis

Null hypothesis: (H0: β=C).(H_0:\ \beta = C).
β\beta : population parameter (what is happening in the real world).
CC : constant appropriate to the specific regression context.

B. The Alternative Hypothesis

Alternative hypothesis (H1)(H_1) — three logical forms:
Right-tailed: (H1: β>C)(H_1:\ \beta > C) (it is actually more tha C).
Left-tailed: (H1: β<C)(H_1:\ \beta < C) (it is actually less than C).
Two-tailed: (H1: β≠C)(H_1:\ \beta \neq C) (it is not actually C, could be more or less).


III. The Test Statistic and Its Distribution

A. Definition of the Test Statistic


t=β^−βSE⁡(β^)t = \frac{\hat{\beta}-\beta}{\operatorname{SE}(\hat{\beta})}

B. Distribution Under the Null Hypothesis

If H0H_0 is true, β=C\beta=C :

t=β^−CSE⁡(β^)∼tn−2.t = \frac{\hat{\beta}-C}{\operatorname{SE}(\hat{\beta})} \sim t_{n-2}.

C. When the Null Is False

If H0H_0 is false, this ratio does not follow tn−2.t_{n-2}.


IV. Rejection Regions and Significance Level (α)(\alpha)

A. Definition of the Rejection Region

The rejection region collects values that are very unlikely when (H0)(H_0) is true.

B. Shape of the Rejection Region

Its shape depends on (H1)(H_1) : right tail, left tail, or two tails.

C. Choosing the Significance Level (α)(\alpha)

Its threshold depends on the chosen significance level (α).(\alpha). Typical choices: (α=5%)\left(\alpha=5\%\right) ; (α=1%)\left(\alpha=1\%\right) (\alpha=1%) (more stringent); (α=10%)\left(\alpha=10\%\right) (less common).

D. Example of Rejection Thresholds

Example intuition (with (df=38)): large values like 2.7, 3.2, 4.7 are possible but very unlikely if (H0)(H_0) is true.


V. Language and Errors

A. Proper Hypothesis Testing Terminology

Say “reject (H0)”(H_0)” or “do not reject (H0)”(H_0)” — never “accept (H0).”(H_0).”

B. Type I Error

Type I error: reject (H0)(H_0) although it is true. Probability = α\alpha .

C. Type II Error

Type II error: do not reject (H0)(H_0) although it is false. In practice here, it is not computed because there are infinitely many possible true alternatives.

D. The Trade-Off Between Errors

Trade-off: lowering α\alpha makes rejection harder.


VI. Five-Step Procedure (Use This Every Time)

  1. State H0_0 and H1H_1

  2. Specify the test statistic and its distribution if H0H_0 is true.

  3. Choose α\alpha and write down the rejection region.

  4. Compute the sample (t) from your regression output.

  5. Make the decision (reject / do not reject) and state the conclusion.


VII. Example 1: Does Food Expenditure Increase with Income? (Right-Tail)

Goal: Show that the effect of income on food spending is positive.

Step 1 — Hypotheses

H0: β2=0H_0:\ \beta_2=0 (no relationship)
H1: β2>0H_1:\ \beta_2>0 (positive effect)

Step 2 — Statistic and Distribution (under H0H_0 )

t=β^2−0SE⁡(β^2)∼t38t = \frac{\hat{\beta}_2-0}{\operatorname{SE}(\hat{\beta}2)} \sim t{38}

(with 40 observations, so df=n−2=38).df=n-2=38).

Step 3 — Significance Level and Rejection Region

α=5\alpha=5%
Right-tail critical value (df 38): t0.95,38≈1.686.t_{0.95,38}\approx 1.686.
Reject for (t>1.686).(t>1.686).

Step 4 — Compute Sample (t)

From the regression: (t=4.88).(t=4.88).

Step 5 — Decision and Conclusion

4.88>1.686⇒4.88>1.686\Rightarrow reject H0.H_0.
Conclusion: there is a statistically significant positive relationship between income and food expenditure at 5% (also at 1% and far smaller α\alpha given how large 4.88 is).


VIII. Example 2: Is Mean Spending < $15 per Additional $100? (Left-Tail)

Goal: Show that the effect is less than $15 per $100.

Step 1 — Hypotheses

H0: β2≥15H_0:\ \beta_2\ge 15 (or (=15))\left(=15)\right) 
H1: β2<15H_1:\ \beta_2<15

Step 2 — Statistic and Distribution (under (H0))(H_0))
t=β^2−15SE⁡(β^2)∼t38.t=\frac{\hat{\beta}_2-15}{\operatorname{SE}(\hat{\beta}2)}\sim t{_{38}}.

Step 3 — Significance Level and Rejection Region

α=5\alpha=5%.
Left-tail critical value (df 38): (-1.686).
Reject for (t<−1.686).(t<-1.686).

Step 4 — Compute Sample (t)

From the regression: (t=-2.29).

Step 5 — Decision and Conclusion (5%)

−2.29<−1.686⇒-2.29<-1.686\Rightarrow reject H0H_0at 5%.
Conclusion: households spend less than $15 per $100 of income at the 5% level.

If α=1\alpha=1%: :
Critical value ≈−2.429.\approx -2.429.
(-2.29) does not cross it ⇒ Do not reject H0H_0 at 1%.
This does not mean ““accept H0”H_0” ; it means there is not enough evidence at 1%.


IX. Example 3: Is There Any Relationship? (Two-Tail “Test of Significance”)

Goal: Show that income affects spending in either direction.

Step 1 — Hypotheses

H0: β2=0H_0:\ \beta_2=0
H1: β2≠0H_1:\ \beta_2\neq 0

Step 2 — Statistic and Distribution (under H0)H_0)


t=β^2−0SE⁡(β^2)∼t38.t=\frac{\hat{\beta}_2-0}{\operatorname{SE}(\hat{\beta}2)}\sim t_{38}.

Step 3 — Significance Level and Rejection Region

α=5\alpha=5%.
Two-tail: split (2.5%) in each tail.
Critical values (df 38): (±2.024).\left(\pm2.024\right).
Reject for (t>2.024)(t>2.024) or (t<−2.024).(t<-2.024).

Step 4 — Compute Sample (t)

From the regression, the same (t) we computed previously is well beyond the thresholds.

Step 5 — Decision and Conclusion

Reject (H0)(H_0) . There is a statistically significant relationship between income and food expenditure at 5%.

Link to Confidence Intervals

If the 5% confidence interval for ((β2))((\beta_2)) excludes 0 (e.g., lower bound (5.97>0)(5.97>0) ), then you would also reject (H0:β2=0)(H_0:\beta_2=0) at 5%.
If the 5% interval for ((β1))((\beta_1)) includes 0, then you would not reject (H0:β1=0)(H_0:\beta_1=0) at 5%.
Same information, different format.


X. Testing Linear Combinations (Same Logic)

A. Replace the Parameter with a Linear Combination

Replace the single parameter by a linear combination ((L)).((L)).

B. Test Statistic for a Linear Combination

Under (H0: L=C0)(H_0:\ L=C_0) , use

t=L^−C0SE⁡(L^)∼tn−2.t = \frac{\widehat{L}-C_0}{\operatorname{SE}(\widehat{L})} \sim t_{n-2}.

C. Tail Direction and Interpretation

Alternatives can be right-tailed, left-tailed, or two-tailed; rejection regions and interpretations are exactly as before.


XI. Linear Combination Example (Weekly Income = $2000)

Question: Is mean food spending for households with weekly income of $2000 greater than $250 per week?

Step 1 — Hypotheses

H0:mean spending at $2000≤250(or =250)H_0: \text{mean spending at } \$2000 \le 250 \quad (\text{or } =250)

H1:mean spending at $2000>250H_1: \text{mean spending at } \$2000 > 250

Step 2 — Statistic and Distribution (under (H0H_0 ))

Linear combination’s point estimate (computed earlier): (L^=287).(\widehat{L}=287).
Use the linear combination’s standard error (computed earlier).
Test statistic:

t=287−250SE⁡(L^)=2.65∼t38 (if H0 is true).t = \frac{287-250}{\operatorname{SE}(\widehat{L})} = 2.65 \sim t_{38}\ (\text{if }H_0\text{ is true}).

Step 3 — Significance Level and Rejection Region

(α=5)\left(\alpha=5\right)
Right-tail critical value (df 38): ((≈1.686)).((\approx 1.686)).
Reject for (t>1.686).(t>1.686).

Step 4 — Compute Sample (t)

(t=2.65).(t=2.65).

Step 5 — Decision and Conclusion

(2.65>1.686⇒(2.65>1.686\Rightarrow reject (H0).(H_0).
Conclusion: expected spending at $2000 weekly income is greater than $250 at the 5% level.


XII. How This Appears in Software Output (What to Look For)

A. Post-Estimation Testing

Always run the regression first (hypothesis tests and CIs are post-estimation).

B. The Reported (t)-Statistic

For (H0:β2=0)(H_0:\beta_2=0) , the reported (t)(t) is (β^2/SE⁡(β^2))(\widehat{\beta}_2/\operatorname{SE}(\widehat{\beta}_2)) ; e.g., (t=4.88).t=4.88).

C. Comparing with Critical Values

To compare with the correct critical value, use a (t)(t) table or a function that returns the (t)(t) critical value for your α\alpha , tail(s), and (df).

D. Tail Matching and Critical Values

For left- or right-tailed tests, match the tail to the alternative; for two-tailed tests at α=5\alpha=5 , use (±2.024)\left(\pm2.024\right) with (df=38).(df=38).

E. Linear Combinations in Output

For linear combinations, compute the linear combination’s point estimate and standard error, form the same (t), and compare to the appropriate critical value.


XIII. Bottom Line

A. Aligning Hypothesis and Goal

Formulate (H1)(H_1) to match the conclusion you want (greater than, less than, or not equal).

B. Choosing the Significance Level

Choose α\alpha and the appropriate rejection region.

C. Reporting the Results

Report the test type, (df), α,\alpha, test statistic, critical value(s), and your decision: reject or do not reject (H0).(H_0).

D. Connecting to Confidence Intervals

Connect to confidence intervals when helpful (same information, different presentation).

E. Remembering the Trade-Off

Remember the trade-off: smaller α\alpha makes rejection harder.

Lecture 3 (Part 3): The P-Value

I. Introduction to the P-Value

The P-value is a central concept in hypothesis testing.
When reporting the result of a statistical hypothesis test, it is standard practice to include the P-value, or probability value, of the test.

The P-value tells us the probability of committing a Type I error, given the T-statistic obtained from the sample and the form of the alternative hypothesis.
In other words, it answers the question: If the null hypothesis is true, what is the probability of obtaining a test statistic as extreme as, or more extreme than, the one observed?


II. Understanding What the P-Value Represents

A. Setting Up the Test

Suppose we are testing a hypothesis and have calculated a test statistic from our data, denoted as T.
Under the null hypothesis, this statistic follows a t-distribution with (n–2)(n – 2) degrees of freedom.

The P-value measures the probability, assuming the null hypothesis is true, of observing a T-statistic that lies in the rejection region—that is, in the direction specified by the alternative hypothesis.


B. Right-Tailed Test

If the alternative hypothesis is of the form:


H1:βk>cH_1: \beta_k > c

then we are performing a right-tailed test.

The P-value is the probability that the test statistic falls to the right of the observed T value.
Graphically, it corresponds to the area under the t-distribution curve to the right of T.

This area represents how likely it is, under the null hypothesis, to obtain such a large or larger T-statistic purely by chance.


C. Left-Tailed Test

If the alternative hypothesis is:
H1:βk<cH_1: \beta_k < c

then we are performing a left-tailed test.

Here, the P-value is the probability that the test statistic falls to the left of the observed T value.
It is the area under the t-distribution curve to the left of T.

This tells us how likely it is, assuming the null is true, to get a T-statistic this small or smaller.


D. Two-Tailed Test

For a two-tailed test, the alternative hypothesis is:
H1:βk≠cH_1: \beta_k \neq c

This means we are testing for deviations in either direction from the null hypothesis.

In this case, the P-value is the sum of the two tail probabilities—
the probability of obtaining a value greater than ∣T∣|T| (the positive tail) plus the probability of obtaining a value less than −∣T∣−|T| (the negative tail):
P=P(T>∣t∣)+P(T<−∣t∣)P = P(T > |t|) + P(T < -|t|)

Graphically, this corresponds to the total area in both tails beyond the magnitude of the observed statistic.


III. Why the P-Value is Useful

The P-value provides an alternative way to conduct hypothesis testing without referring to the critical value directly.

It allows us to decide whether to reject or not reject the null hypothesis by comparing the P-value to the significance level (α) of the test.


A. Rejection Rule Based on the P-Value

The decision rule is straightforward:

  • IfP≤αP ≤ α , reject the null hypothesis (H₀).

  • If P>αP > α , fail to reject the null hypothesis (H₀).

This rule gives us the same outcome as the traditional comparison between the test statistic and the critical value, but it is often more intuitive, since the P-value represents the exact probability of observing the result under the null.


B. Relation to the Critical Value Method

Earlier, we compared the T-statistic with the critical value from the t-distribution.
Now, instead of transforming α into a critical value, we can transform the T-statistic into a P-value.

Both approaches are equivalent:

  • Comparing T-statistic ↔ critical value

  • Comparing P-value ↔ significance level α

In both cases, the same conclusion will be reached.


IV. Example 1 – Left-Tailed Test

A. Hypothesis Setup

We are testing whether mean spending on food is less than $15 for every additional $100 of income.
H0:β2≥15H_0: \beta_2 \geq 15

H1:β2<15H_1: \beta_2 < 15

This is a left-tailed test, since the alternative hypothesis states “less than.”


B. Test Statistic

From the sample data, we computed:


T=−2.29T = -2.29

Under the null hypothesis, the T-statistic follows a t-distribution with (n–2)(n – 2) degrees of freedom.


C. Meaning of the P-Value

The P-value is the probability, assuming the null hypothesis is true, of obtaining a T-statistic less than –2.29.
That is, the probability of being in the left tail of the distribution beyond T=–2.29.T = –2.29.

This P-value equals 0.0139, or 1.39%.


D. Decision and Interpretation

We compare the P-value to our chosen significance levels:

  • At α = 5%:
    P = 1.39% < 5% P=1.39%<5%P=1.39\%<5\% → Reject H₀.
    There is sufficient evidence to conclude that the spending is less than $15.

  • At α = 1%:
    P=1.39%>1%P=1.39\%>1\% → Fail to reject H₀.
    There is not enough evidence at this stricter level to reject the null.


E. Visual Explanation

Graphically:

  • The P-value region is the shaded left tail of the t-distribution beyond T=–2.29.T = –2.29.

  • The critical value for α = 5% is T=–1.686,T = –1.686, so our T-statistic (–2.29)(–2.29) falls in the rejection region.

  • The critical value for α = 1% is T=–2.429,T = –2.429, so our T-statistic does not fall in that rejection region.

Thus, the P-value method and the critical value method give the same result:
Reject H₀ at 5%, do not reject at 1%.


V. Example 2 – Right-Tailed Test

A. Hypothesis Setup


H0:β2=0H_0: \beta_2 = 0

H1:β2>0H_1: \beta_2 > 0

This is a right-tailed test, because the alternative hypothesis states “greater than.”


B. Test Statistic

The computed T-statistic is:


T=4.88T = 4.88

Under the null hypothesis, this follows a t-distribution with(n–2)(n – 2) degrees of freedom.


C. Meaning of the P-Value

The P-value is the probability, assuming the null hypothesis is true, of obtaining a T-statistic greater than or equal to 4.88.

This probability is extremely small, essentially close to zero.


D. Decision and Interpretation

Because the P-value is so low, we can reject the null hypothesis for:

  • α = 5%

  • α = 1%

  • and even for much smaller α values.

Therefore, the evidence is overwhelming that β₂ is greater than zero.
This shows that the relationship between income and spending is statistically significant.


VI. Example 3 – Two-Tailed Test (Significance of a Parameter)

A. Hypothesis Setup


H0:β2=0H_0: \beta_2 = 0

H1:β2≠0H_1: \beta_2 \neq 0

This is a two-tailed test, also called the test of the significance of a parameter.

It checks whether β₂ is significantly different from zero in either direction.


B. Test Statistic and P-Value

The same T-statistic applies here:


T=4.88T = 4.88

For a two-tailed test, the P-value represents the probability of obtaining a value greater than 4.88 plus the probability of obtaining a value less than –4.88.

This total P-value is extremely small, practically zero.


C. Decision and Interpretation

Since the P-value is far below any typical α (5%, 1%, etc.), we can reject the null hypothesis for all these significance levels.

This means the parameter β₂ is highly statistically significant, and we can conclude that it differs from zero.


D. Reading P-Values from Regression Output

In regression output, each coefficient’s P-value corresponds to this two-tailed significance test.

The interpretation is:

  • If P ≤ α, the coefficient is statistically significant.

  • If P > α, the coefficient is not significant.

For example:

  • For β₂ (income): P < 0.01 → reject H₀ for any α → significant.

  • For β₁ (intercept): P = 0.062 → greater than both 0.05 and 0.01 → not significant.

Interpretation:
The intercept represents expected spending for individuals with zero income.
Since we cannot reject H₀ that β₁ = 0, we conclude that for people with no income, the expected spending on food is 0.


E. Clarification on Two-Tailed P-Values

The P-values displayed in regression outputs are already for two-tailed tests.
They represent the total probability of observing a value greater than |T| or less than −|T|.

Therefore, when using these reported P-values, no additional multiplication by 2 is needed.
A common mistake is doubling the reported value unnecessarily—this should be avoided.


VII. Summary of Key Points

  1. The P-value measures the probability of obtaining a test statistic as extreme as the observed one, assuming the null hypothesis is true.

  2. The P-value depends on the direction of the alternative hypothesis:

    • Right-tailed: P(T > t)

    • Left-tailed: P(T < t)

    • Two-tailed: P(T > |t|) + P(T < –|t|)

  3. The rejection rule:

    • Reject H₀ if P ≤ α.

    • Fail to reject H₀ if P > α.

  4. The P-value method is equivalent to the critical value method but gives a direct probability interpretation.

  5. Regression outputs provide two-tailed P-values for the test of significance of each coefficient.

  6. When the P-value is very low, the result is highly significant, meaning strong evidence against the null hypothesis.


VIII. Transition to Next Topic

This concludes the discussion on the P-value and the complete analysis of the simple linear regression model.
The next step is to extend these ideas to the multiple linear regression model, which accounts for multiple explanatory variables and reflects more realistic situations.

Lecture 4 (Part 1): Multiple Linear Regression 

I. Introduction to Multiple Linear Regression

A simple linear regression model is limited because it includes only one explanatory variable. In reality, most outcomes are influenced by several factors. For example, it is not reasonable to assume that only income explains spending on food. Other crucial factors—such as the number of household members and location (city or village)—also affect spending. Rural households may have access to cheaper products, influencing overall spending patterns.

To analyze these more realistic scenarios, the model must be extended to include multiple explanatory variables. The extension is conceptually simple.


II. Structure of the Multiple Linear Regression Model

The general form of the model includes:
[Yi=β1+β2X2i+β3X3i+⋯+ui][Y_i = \beta_1 + \beta_2 X_{2i} + \beta_3 X_{3i} + \dots + u_i]

  • (β1)(\beta_1) : intercept

  • (βk):(\beta_k): slope coefficients for each explanatory variable (X_k)

  • (ui):(u_i): random error term

Each additional explanatory variable receives its own (\beta) coefficient.
In this framework, a single parameter (\beta_k) measures the effect of a change in (X_k) on the expected value of (Y), holding all other variables constant.

This “holding constant” is formalized mathematically by taking the partial derivative of (E(Y|X)) with respect to (X_k):
[∂E(Y∣X)∂Xk][\frac{\partial E(Y|X)}{\partial X_k}]
This isolates the effect of (X_k) while keeping other explanatory variables fixed.


III. Example Data: Sales of “Big Andy’s Bean Burger”

A dataset is used to illustrate the model.

  • Dependent variable (Y): Monthly sales (in thousands of dollars)

  • Observations: 75 branches

  • Explanatory variables:

    • Price (in dollars per burger)

    • Advertising expenditure (in thousands of dollars)

Data Characteristics

  • Sales range: $62,000 to $91,000

  • Price range: $4.80 to $6.50

  • Advertising expenditure range: $500 to $3,100

This indicates a small business, not a large company like Nike or Victoria’s Secret.

Before running any regression, it is important to understand the dataset—its scale, units, and variation—since these affect statistical inference.


IV. Model Interpretation

[Yi=β1+β2(Pricei)+β3(Advertisingi)+ui][Y_i = \beta_1 + \beta_2 (\text{Price}_i) + \beta_3 (\text{Advertising}_i) + u_i]

  • (\beta_2): Change in monthly sales (thousands of dollars) when price increases by 1</strong>,holding<strong>advertisingconstant</strong>.</p></li><li><p>(β3):Changein<strong>monthlysales(thousandsofdollars)</strong>when<strong>advertisingexpenditureincreasesby1</strong>, holding <strong>advertising constant</strong>.</p></li><li><p>(\beta_3): Change in <strong>monthly sales (thousands of dollars)</strong> when <strong>advertising expenditure increases by1,000, holding price constant.

Geometrically, instead of a regression line, the model estimates a regression plane:

  • The intercept is where the plane crosses the Y-axis.

  • The price slope is one directional vector.

  • The advertising slope is another directional vector.

Each added explanatory variable adds one dimension to this geometric representation.


V. Assumptions for Multiple Linear Regression

The key assumptions from simple linear regression still apply.

A. Assumption 1: Expected Error = 0

[
E(u_i) = 0
]
This holds if the model is correctly specified.
If an important explanatory variable is omitted, its effect is absorbed into the error term, making (E(u_i) \neq 0).

B. Assumption 2: Constant Variance of Errors (Homoscedasticity)

[
Var(u_i) = \sigma^2
]
Often, this does not hold—variance may increase with (X).
For now, this assumption is maintained. Later, heteroskedasticity and its implications will be examined.

C. Assumption 3: Zero Covariance Between Errors

[
Cov(u_i, u_j) = 0 \quad \text{for all } i \neq j
]
This means that one observation’s error does not provide information about another’s.
This assumption will later be relaxed when studying serial correlation.

D. Assumption 4: Normality (for Small Samples)

[
u_i \sim N(0, \sigma^2)
]
For large samples, this follows from the Central Limit Theorem; for smaller ones, it must be explicitly assumed.

E. New Assumption for Multiple Regression: No Perfect Multicollinearity

No explanatory variable is an exact linear function of any other:
[
X_k \neq a_1 X_1 + a_2 X_2 + \dots
]

Violations (examples):

  1. (X_3 = 2 X_2) → violates assumption

  2. (X_3 = 0.1X_2 + 0.7X_4) → violates assumption

  3. (X_3 = X_2^2) → does NOT violate (nonlinear relationship)

If exact multicollinearity exists, least squares estimation fails, and software will return an error message.
This is called exact collinearity, to be discussed further in a later lecture.


VI. Estimating the Model

The least squares estimators minimize the sum of squared errors (SSE):
[
SSE = \sum (Y_i - \hat{Y}_i)^2
]
As before, we take derivatives with respect to all (\beta)s—(\beta_1, \beta_2, \beta_3)—to obtain their estimators.

A. Interpretation of the Estimators

  • (\hat{\beta}_1): Ensures the regression plane passes through the sample means of (Y), (X_2), and (X_3).

  • (\hat{\beta}_2, \hat{\beta}_3): Derived using deviations from means. They adjust for covariance among explanatory variables.

B. Covariance-Based Formula (Intuitive Explanation)

If (Cov(X_2, X_3) = 0), the model simplifies:
[
\hat{\beta}_2 = \frac{Cov(X_2, Y)}{Var(X_2)}
]
which is identical to the simple regression estimator.

When (X_2) and (X_3) are not correlated, including or excluding (X_3) produces the same estimate for (\beta_2).

However, when (X_2) and (X_3) are correlated, omitting one leads to biased estimates.


VII. Omitted Variable Bias

If (X_3) is omitted and it is correlated with (X_2), the estimate of (\beta_2) becomes biased.
This bias occurs because part of the variation in (Y) attributed to (X_2) is actually due to (X_3).

This issue is extremely important because:

  • In survey data, explanatory variables often correlate.

  • In experimental data, variables can be independently manipulated, so correlation is less problematic.

Examples:

  1. Negative correlation:
    During promotional campaigns, businesses may both reduce prices and increase advertising → price and advertising negatively correlated.

  2. Positive correlation:
    Companies like Nike with large ad budgets charge higher prices → price and advertising positively correlated.

To obtain meaningful and unbiased estimates, the full model including all relevant explanatory variables must be used.


VIII. Understanding the Bias Mechanism

Suppose (X_2) and (X_3) are positively correlated.

When (X_2) increases:

  • (Y) increases directly (because of (X_2)’s positive effect).

  • (X_3) also tends to increase, causing (Y) to rise further through (X_3)’s effect.

If only (Cov(X_2, Y)) is used, all of this combined effect is wrongly attributed to (X_2).
The true effect of (X_2) alone requires subtracting the portion of variation in (Y) explained through (X_3).

This is what least squares estimation achieves—it isolates the unique effect of (X_2) by removing the shared influence through (X_3).


IX. Connection to Multicollinearity

The same formula reveals why perfect collinearity makes estimation impossible.

If (X_2) and (X_3) move exactly together (for example, when (X_3 = 2X_2)), there is no way to distinguish how much of (Y)’s variation comes from (X_2) versus (X_3).
Therefore, the model cannot separate their effects, and the least squares method fails.


X. Summary of Key Takeaways

  • Multiple linear regression allows analysis of more realistic relationships with several explanatory variables.

  • Coefficients measure partial effects, holding all other variables constant.

  • The geometric interpretation expands from a line to a plane.

  • All previous assumptions remain, with one new requirement: no perfect linear relationship among explanatory variables.

  • Omitted variable bias arises when a correlated regressor is left out, leading to incorrect inference.

  • Exact collinearity prevents estimation altogether.

Lecture 4 (Part 2): Interpreting Multiple Regression Results 


I. Introduction

After understanding the intuition behind the sales model, we now focus on interpreting the actual regression estimates using two explanatory variables: price and advertising.

It is crucial to be able to interpret regression output correctly — both for the exam and in professional applications.


II. Interpreting the Coefficient of Price

A. Initial Observation

The coefficient on price is negative, which suggests that demand is price elastic.
Holding advertising constant, an increase in price by $1 leads to a fall in monthly revenue of $7,980.
The p-value is approximately 0, indicating that the coefficient is highly significant.

However, even though this interpretation is technically correct, it is not the best way to interpret the result — and would lose points on the exam.


B. Correct Interpretation

It is better to interpret the coefficient in a way that reflects the realistic variation in the data.
The correct phrasing would be:

An increase in price by $0.10 leads to a fall in monthly revenues of $719.90, holding advertising constant.

This is preferable because the observed price range in the data is between $4.90 and $6.50 — less than $2 total variation.

Predicting what would happen with a $1.00 change is beyond the realistic range of observed data.


C. Why Large Changes Are Problematic

If the estimated relationship were taken literally, a $1 decrease in price would imply an increase in sales of more than $7,900.
If the price were decreased from $5 to $0, the model would predict sales increasing by nearly $40,000, which is nonsensical.

This occurs because regression models describe relationships only within the range of observed data.
Extrapolating beyond that range (e.g., predicting sales at prices of $2 or $10) can lead to meaningless results since consumer behavior may differ drastically at extreme prices.

Therefore, with a sample price range of less than $2, it is unreasonable to interpret results for a $1 change.
A $0.10 change provides a much more meaningful interpretation.


III. Interpreting the Coefficient of Advertising

A. Observation

The advertising coefficient is positive and highly significant (p-value ≈ 0).
Thus, at α = 1% (or even lower), it is statistically significant.

Holding price constant, an increase in advertising expenditure of $1,000 leads to an increase in monthly revenues of $1,863.


B. Adjusted Interpretation

However, the observed advertising expenditures in the data range from $500 to $3,000.
Therefore, a more appropriate interpretation is:

An increase in advertising expenditure by $100 leads to an increase in monthly revenues of $183.60.

This expresses the relationship on a scale that aligns with the data range.


IV. Interpreting the Intercept

A. Direct Interpretation

The estimated intercept suggests that if both price and advertising expenditure were zero, sales revenues would be approximately $119,000.
Even though the coefficient is statistically significant (p-value ≈ 0) and its 95% confidence interval ranges from $106K to $131K, this result is not meaningful.

A price of $0 would clearly imply revenue of $0, not $119K.
Therefore, the intercept has no real-world interpretability.


B. Why We Still Include an Intercept

Even though the intercept is not interpretable, it is almost always included in the model because it improves the fit of the regression.

Without an intercept, the regression line is forced to pass through the origin, which restricts the slope and limits the model’s flexibility to fit the data where observations actually occur.

Including an intercept allows the line to begin at any point on the Y-axis, improving the model’s fit in the region where data exist — even if predictions near the origin are meaningless.


V. The Gauss-Markov Theorem

The Gauss-Markov theorem also applies to the multiple linear regression model.

If the classical assumptions hold, then the least squares estimators are:

  • Best Linear Unbiased Estimators (BLUE) — meaning:

    • They are linear in the dependent variable.

    • They are unbiased (their expected value equals the true parameter).

    • They have the lowest variance among all linear unbiased estimators.

All the results from simple regression extend naturally to multiple regression, except that the degrees of freedombecome:

[
n - k
]

where:

  • ( n ) = number of observations

  • ( k ) = number of parameters in the model

In simple regression, this was ( n - 2 ).


VI. Confidence Intervals in Multiple Regression

The formula for the confidence interval remains the same as before, except we now use t-critical values with n – k degrees of freedom.

For example, to construct a 95% confidence interval for the effect of advertising (( \beta_3 )):

  • ( n = 75 )

  • ( k = 3 )

  • Degrees of freedom = ( 75 - 3 = 72 )

  • ( t_{0.025, 72} = 1.99 )

Given the point estimate and standard error from the regression output:

[
\text{CI}_{95%} = \hat{\beta_3} \pm 1.99 \times SE(\hat{\beta_3})
]

This yields a confidence interval such that:

Increasing advertising spending by $1,000 increases revenue by between $501 and $3,225 at α = 5%.

However, since the data range is limited, a more realistic interpretation is:

Increasing advertising spending by $100 increases revenue by between $50.10 and $322 at α = 5%.


Interpretation of Interval

The interval is relatively wide, implying uncertainty about profitability:

  • Additional advertising may barely cover its cost (if the effect is near $50).

  • Or it may be highly profitable (if the effect is near $322).
    Thus, the true effect could mean breaking even or gaining more than three times the cost.


VII. Hypothesis Testing Example

A. Question

Suppose your boss asks whether increasing the advertising budget is a good idea.

We can test the hypothesis:

[
H_0: \beta_3 = 1 \quad \text{(no gain; break-even)} \
H_1: \beta_3 > 1 \quad \text{(positive gain; revenue exceeds cost)}
]

B. Test Statistic

Under ( H_0 ), replace ( \beta_3 ) with 1 in the formula.
The test statistic follows a t-distribution with 72 degrees of freedom.

[
t = \frac{\hat{\beta_3} - 1}{SE(\hat{\beta_3})}
]

If ( \alpha = 1% ), this is a right-tailed test.

The critical value is 2.38.
We reject ( H_0 ) if ( t > 2.38 ).


C. Result

The computed ( t = 1.263 ), which is much lower than 2.38.
The p-value for this statistic is 0.105 (10.5%), so we cannot reject the null, even at α = 10%.

Hence, we conclude that increasing advertising may not necessarily cover its additional cost.


D. Connection to Confidence Intervals

This result is consistent with the earlier confidence interval:
The lower bound allows for the possibility that revenue does not cover advertising cost.

If we cannot reject ( H_0 ) for α = 2.5% (right-tailed), we certainly cannot reject it for α = 1%.
Thus, hypothesis testing and confidence intervals provide equivalent information.


VIII. Predicting Expected Sales

A. Example: Linear Combination of Parameters

We can use the model to predict expected sales for any price and advertising level within the data range.

For example:

  • Price = $5.50

  • Advertising expenditure = $1,200

The point estimate of expected sales is ≈ 77,000</strong>.</p><p>Thesevaluesarewithintheobserveddatarange,sothepredictionis<strong>meaningfulandreliable</strong>.</p><divdata−type="horizontalRule"><hr></div><h4id="1b365861−69fe−448f−861c−28311c86540f"data−toc−id="1b365861−69fe−448f−861c−28311c86540f"collapsed="false"seolevelmigrated="true"><strong>B.ConfidenceIntervalforExpectedSales</strong></h4><p>Wecancomputea<strong>confidenceinterval</strong>forthepredictedmeansalesusinga<strong>linearcombination</strong>ofparameters((β1,β2,β3)).</p><p>Althoughthiscalculationistediousbyhand,itcanbedoneeasilyusingthe<strong>linearcombinationcommand</strong>instatisticalsoftware.</p><p>Theresult:</p><figuredata−type="blockquoteFigure"><div><blockquote><p>Whenprice=77,000</strong>.</p><p>These values are within the observed data range, so the prediction is <strong>meaningful and reliable</strong>.</p><div data-type="horizontalRule"><hr></div><h4 id="1b365861-69fe-448f-861c-28311c86540f" data-toc-id="1b365861-69fe-448f-861c-28311c86540f" collapsed="false" seolevelmigrated="true"><strong>B. Confidence Interval for Expected Sales</strong></h4><p>We can compute a <strong>confidence interval</strong> for the predicted mean sales using a <strong>linear combination</strong> of parameters (( \beta_1, \beta_2, \beta_3 )).</p><p>Although this calculation is tedious by hand, it can be done easily using the <strong>linear combination command</strong> in statistical software.</p><p>The result:</p><figure data-type="blockquoteFigure"><div><blockquote><p>When price =5.50 and advertising = $1,200, the expected monthly sales lie between $76,000 and 79,000</strong>atα=579,000</strong>at α = 5%.</p></blockquote><figcaption></figcaption></div></figure><p></p><h2 id="13396e20-8a18-4854-b156-5e75d1f0652e" data-toc-id="13396e20-8a18-4854-b156-5e75d1f0652e" collapsed="true" seolevelmigrated="true"><strong>Lecture 4 (Part 3): Goodness of Fit and R² in the Simple Linear Regression Model</strong></h2><h3 id="0ba6590e-f8e5-4bc9-a0c9-24e3ca2a0a3c" data-toc-id="0ba6590e-f8e5-4bc9-a0c9-24e3ca2a0a3c" collapsed="false" seolevelmigrated="true">I. Introduction</h3><p>This lecture concludes the discussion on the <strong>simple linear regression model</strong> and focuses on the <strong>goodness of fit</strong>, measured by the <strong>R² (R-squared)</strong> statistic.<br>After understanding R², the discussion transitions to multiple linear regression, which extends the model to include several explanatory variables.</p><div data-type="horizontalRule"><hr></div><h3 id="bd7937c2-8cfa-4e0d-ab85-75fd8969f283" data-toc-id="bd7937c2-8cfa-4e0d-ab85-75fd8969f283" collapsed="false" seolevelmigrated="true">II. Purpose of Regression and the Question of Goodness of Fit</h3><p>Regression models are built to <strong>use explanatory variables (X)</strong> to explain variation in a <strong>dependent variable (Y)</strong>.<br>We expect that variation in X helps explain variation in Y.<br>The key question is: <strong>how well does the model actually explain this variation?</strong></p><div data-type="horizontalRule"><hr></div><h3 id="64acd535-f546-4ca6-a75f-d1a6ab70f5fe" data-toc-id="64acd535-f546-4ca6-a75f-d1a6ab70f5fe" collapsed="false" seolevelmigrated="true">III. Understanding Variation and Residuals</h3><p>Let:</p><ul><li><p>( \bar{X} ) and ( \bar{Y} ) = sample means of X and Y.</p></li><li><p>( \hat{Y} ) = fitted (predicted) value of Y.</p></li><li><p>( \hat{\varepsilon}_i = Y_i - \hat{Y}_i ) = residual (the unexplained part).</p></li></ul><p>The <strong>total variation</strong> of Y around its mean is ( (Y_i - \bar{Y}) ).<br>The <strong>explained variation</strong> (due to the regression model) is ( (\hat{Y}_i - \bar{Y}) ).<br>The <strong>residual variation</strong> (unexplained) is ( (Y_i - \hat{Y}_i) ).</p><p>So, total variation = explained variation + residual variation.</p><p>Variation in X around its mean (( X_i - \bar{X} )) explains only <strong>part</strong> of the variation in Y around its mean (( Y_i - \bar{Y} )).<br>The remaining part—the residual—is the variation that cannot be explained by the model.</p><div data-type="horizontalRule"><hr></div><h3 id="af2d555d-33e0-4bf2-85fd-9bfc1f10b995" data-toc-id="af2d555d-33e0-4bf2-85fd-9bfc1f10b995" collapsed="false" seolevelmigrated="true">IV. Building a Quantitative Measure of Goodness of Fit</h3><p>We want a <strong>measure</strong> that expresses the <strong>proportion of total variation in Y</strong> that is explained by the regression model.</p><p>For each observation:<br>[<br>Y_i = \hat{Y}_i + \hat{\varepsilon}_i<br>]</p><p>Subtracting the sample mean ( \bar{Y} ) from both sides gives:<br>[<br>Y_i - \bar{Y} = (\hat{Y}_i - \bar{Y}) + \hat{\varepsilon}_i<br>]</p><p>Squaring both sides and summing across all observations (to prevent positive and negative deviations from canceling out) gives:<br>[<br>\sum (Y_i - \bar{Y})^2 = \sum (\hat{Y}_i - \bar{Y})^2 + \sum \hat{\varepsilon}_i^2<br>]</p><p>This expresses that <strong>total sample variation</strong> is divided into:</p><ol><li><p><strong>Explained variation</strong> – due to regression</p></li><li><p><strong>Unexplained variation</strong> – due to error</p></li></ol><div data-type="horizontalRule"><hr></div><h3 id="67872020-26cd-4788-a605-3ef531b79490" data-toc-id="67872020-26cd-4788-a605-3ef531b79490" collapsed="false" seolevelmigrated="true">V. Decomposition of Variation</h3><p>We use the following terms:</p><ul><li><p><strong>SST (Total Sum of Squares)</strong> = total variation = ( \sum (Y_i - \bar{Y})^2 )</p></li><li><p><strong>SSR (Sum of Squares due to Regression)</strong> = explained variation = ( \sum (\hat{Y}_i - \bar{Y})^2 )</p></li><li><p><strong>SSE (Sum of Squares due to Error)</strong> = unexplained variation = ( \sum \hat{\varepsilon}_i^2 )</p></li></ul><p>These satisfy the fundamental relationship:<br>[<br>SST = SSR + SSE<br>]</p><div data-type="horizontalRule"><hr></div><h3 id="8d955b4a-dda4-465a-8d1f-f2e988f9e85b" data-toc-id="8d955b4a-dda4-465a-8d1f-f2e988f9e85b" collapsed="false" seolevelmigrated="true">VI. Definition of R²</h3><p>To find the <strong>proportion of total variation explained by the model</strong>, divide both sides by SST:</p><p>[<br>R^2 = \frac{SSR}{SST} = 1 - \frac{SSE}{SST}<br>]</p><p><strong>Interpretation:</strong><br>R² measures the proportion of variation in Y around its mean that is explained by variation in X within the regression model.</p><div data-type="horizontalRule"><hr></div><h3 id="51a7af5c-2145-4781-8a66-2964e7b07b45" data-toc-id="51a7af5c-2145-4781-8a66-2964e7b07b45" collapsed="false" seolevelmigrated="true">VII. Properties and Interpretation of R²</h3><ul><li><p>R² is a <strong>proportion</strong>, so its range is:<br>[<br>0 \leq R^2 \leq 1<br>]</p></li></ul><h4 id="76b493ca-47a2-4d8c-a18f-f4528ea62dd9" data-toc-id="76b493ca-47a2-4d8c-a18f-f4528ea62dd9" collapsed="false" seolevelmigrated="true">A. Extreme Cases</h4><ol><li><p><strong>R² = 1 (Perfect Fit):</strong></p><ul><li><p>All sample data points fall exactly on the regression line.</p></li><li><p>SSR = SST and SSE = 0.</p></li><li><p>All variation in Y is explained by X.</p></li><li><p>The model fits the data perfectly.</p></li></ul></li><li><p><strong>R² = 0 (No Fit):</strong></p><ul><li><p>Variation in X does not explain variation in Y at all.</p></li><li><p>SSE = SST and SSR = 0.</p></li><li><p>The fitted line is horizontal (no relationship).</p></li><li><p>Observations are randomly scattered, showing no linear association.</p></li></ul></li></ol><h4 id="672a6587-2f62-41d8-bc0b-07652f4d2a3c" data-toc-id="672a6587-2f62-41d8-bc0b-07652f4d2a3c" collapsed="false" seolevelmigrated="true">B. Typical Case</h4><p>In practice, ( 0 < R^2 < 1 ).<br>The closer R² is to 1, the more variation in Y is explained by the model.</p><div data-type="horizontalRule"><hr></div><h3 id="6224d997-cb9c-464c-b36e-4cfbd7cbad77" data-toc-id="6224d997-cb9c-464c-b36e-4cfbd7cbad77" collapsed="false" seolevelmigrated="true">VIII. Example: Food Expenditure and Income</h3><p>In a regression model explaining <strong>food expenditure</strong> as a function of <strong>income</strong>, the output shows:</p><p>[<br>R^2 = 0.385<br>]</p><p><strong>Interpretation:</strong><br>38.5% of the variation in food expenditure around its mean is explained by income.</p><p>Software automatically calculates R² and reports it in the regression output, along with:</p><ul><li><p>SSR (Sum of Squares due to Regression)</p></li><li><p>SSE (Sum of Squares due to Error)</p></li><li><p>SST (Total Sum of Squares)</p></li></ul><p>These values satisfy ( SSR + SSE = SST ).</p><div data-type="horizontalRule"><hr></div><h3 id="6ffbbd75-7c9b-431c-b9d6-1a744fb80acf" data-toc-id="6ffbbd75-7c9b-431c-b9d6-1a744fb80acf" collapsed="false" seolevelmigrated="true">IX. Understanding the ANOVA (Upper Table of Output)</h3><p>The regression output’s upper table shows:</p><ul><li><p><strong>Sum of Squares (SS)</strong> for Model, Error, and Total.</p></li><li><p><strong>Degrees of Freedom (df):</strong></p><ul><li><p>Model: always 1 in simple regression.</p></li><li><p>Error: ( n - k ).</p></li></ul></li><li><p><strong>Mean Squares (MS):</strong><br>[<br>MS = \frac{SS}{df}<br>]</p></li></ul><p>The important value is:<br>[<br>MSE = \frac{SSE}{n - k}<br>]<br>This is the <strong>estimate of ( \sigma^2 )</strong>, the variance of the error term.</p><p>The <strong>Root MSE</strong> is:<br>[<br>\sqrt{MSE} = \hat{\sigma}<br>]<br>which is the <strong>estimated standard deviation</strong> of the error term.</p><div data-type="horizontalRule"><hr></div><h3 id="9fb8740d-08e4-4a8b-9cea-648daec30ba6" data-toc-id="9fb8740d-08e4-4a8b-9cea-648daec30ba6" collapsed="false" seolevelmigrated="true">X. How to Interpret R² Values in Practice</h3><p>Students often ask whether an R² of 38.5% is “good enough.”<br>There is <strong>no universal rule</strong>—it depends on the <strong>type of data</strong>.</p><h4 id="3846868e-c4db-496a-9662-472d04ce8b72" data-toc-id="3846868e-c4db-496a-9662-472d04ce8b72" collapsed="false" seolevelmigrated="true">A. Cross-Sectional Data</h4><ul><li><p>Observes each entity only once.</p></li><li><p>R² is typically <strong>low</strong> (10–40% is common).</p></li><li><p>Example: household or individual data on spending behavior.</p></li><li><p>Reason: people have <strong>different preferences</strong>, creating large randomness that cannot be explained by income or demographics.</p></li></ul><p>Thus, a low R² does not mean the model is bad—it reflects real-world variability.</p><h4 id="22cc3f84-a3f6-4051-9087-81241e6382c6" data-toc-id="22cc3f84-a3f6-4051-9087-81241e6382c6" collapsed="false" seolevelmigrated="true">B. Time-Series Data</h4><ul><li><p>Observes the same variable over time.</p></li><li><p>R² is typically <strong>high</strong> because of <strong>serial correlation</strong>—past values are highly correlated with present ones.</p></li><li><p>Example: today’s price depends heavily on prices from previous days.</p></li><li><p>Hence, a large portion of today’s variation is explained by past prices, resulting in high R² values.</p></li></ul><h4 id="6558988e-7f4d-441e-87aa-9b779c0421ad" data-toc-id="6558988e-7f4d-441e-87aa-9b779c0421ad" collapsed="false" seolevelmigrated="true">C. Summary Rule</h4><p>Always interpret R² relative to the <strong>type of data</strong> and <strong>context</strong>.<br>Do <strong>not</strong> judge model quality based solely on R².</p><div data-type="horizontalRule"><hr></div><h3 id="4cf7790c-81d1-400b-a4f4-d37301681923" data-toc-id="4cf7790c-81d1-400b-a4f4-d37301681923" collapsed="false" seolevelmigrated="true">XI. Evaluating Model Quality</h3><p>Never rely only on R² when assessing a model.<br>Evaluate the following first:</p><ol><li><p><strong>Signs of coefficients</strong> – Are they consistent with theory?</p></li><li><p><strong>Magnitudes</strong> – Are the effects reasonable?</p></li><li><p><strong>Precision</strong> – Based on standard errors.</p></li><li><p><strong>Significance</strong> – Check p-values or t-tests.</p></li></ol><p>After verifying these, look at R² as an additional descriptive measure.</p><div data-type="horizontalRule"><hr></div><h3 id="5b2c3c8e-572a-4143-9fcd-3836bec2204c" data-toc-id="5b2c3c8e-572a-4143-9fcd-3836bec2204c" collapsed="false" seolevelmigrated="true">XII. Reporting Regression Results</h3><p>When reporting regression results, follow this format:</p><ul><li><p>Report <strong>coefficient estimates</strong>.</p></li><li><p>Report <strong>standard errors</strong> in parentheses below estimates.</p></li><li><p>Indicate <strong>significance levels</strong>:</p><ul><li><p>One asterisk (*) for 5% significance.</p></li><li><p>Two asterisks (**) for 1% significance.</p></li></ul></li></ul><p>Example reporting conventions:</p><ul><li><p>Intercept not significant at 1% or 5%.</p></li><li><p>Slope coefficient significant at both 1% and 5%.</p></li><li><p>Report the <strong>R²</strong> value.</p></li></ul><p>Such a report includes all critical regression information—estimates, standard errors, significance, and goodness of fit.</p><div data-type="horizontalRule"><hr></div><p>Would you like me to format this as a clean, printable exam-ready document (PDF or Word) with equations aligned and section headings styled like your other econometrics notes?</p><h2 id="868472ce-7191-4114-88d8-e5f5c6bb7f2e" data-toc-id="868472ce-7191-4114-88d8-e5f5c6bb7f2e" collapsed="true" seolevelmigrated="true"><strong>Lecture 5 (Part 1): Modeling Nonlinear Relationships in the Linear Regression Framework</strong></h2><div data-type="horizontalRule"><hr></div><h3 id="1375f345-d63a-49f6-8d7d-f944cb6c934d" data-toc-id="1375f345-d63a-49f6-8d7d-f944cb6c934d" collapsed="false" seolevelmigrated="true">I. The Flexibility of the Linear Regression Model</h3><p>The linear regression framework, despite its name, is much more flexible than it appears. Although it is called the <em>linear regression model</em>, it can also be used to model and estimate <strong>nonlinear relationships</strong> between variables.</p><p>The term “linear” does not refer to whether the relationship between the variables is a straight line; rather, it refers to the fact that the model is <em>linear in parameters</em>. This means that even nonlinear shapes can be represented within this framework, as long as the model remains linear in the coefficients.</p><div data-type="horizontalRule"><hr></div><h3 id="cdca7f87-5d3f-47f2-99d4-6235dc4bda27" data-toc-id="cdca7f87-5d3f-47f2-99d4-6235dc4bda27" collapsed="false" seolevelmigrated="true">II. The Baseline Example: Advertising and Sales</h3><p>The model under discussion assumes that there is a <strong>linear relationship</strong> between advertising spending and sales.</p><p>The model is called “linear” because it assumes that the <strong>return from advertising</strong>—that is, the increase in expected sales when advertising increases—is <strong>constant</strong>.</p><p>If advertising spending increases by one unit, the expected increase in sales is always the same, regardless of the current advertising level. This constant effect is represented by the coefficient (parameter) on the advertising variable.</p><p>Although this parameter is unknown and must be estimated, it is still treated as a <strong>fixed constant</strong>.</p><p>For example:</p><ul><li><p>If the current advertising level is1,200 and spending increases by $100, expected sales increase by $186.

  • If the current level is $2,500 and spending increases by $100, expected sales still increase by $186.

  • Thus, the model implies that the return from advertising is independent of the current level of advertising.


    III. Theoretical Critique: Diminishing Returns from Advertising

    To evaluate whether this constant-return assumption makes sense, economic theory should guide the reasoning.

    From theory and everyday experience, it is clear that a linear relationship between advertising and sales is unrealistic.

    Consider an everyday example:

    • The first time you receive a flyer about a new vegan restaurant, it captures your attention and perhaps influences your behavior.

    • By the fifth flyer, the information feels redundant.

    • By the tenth flyer, it becomes annoying, and you are less likely to visit that restaurant at all.

    This illustrates diminishing returns to advertising: at low levels, advertising strongly increases sales, but as advertising intensity grows, each additional dollar produces a smaller effect, and beyond a certain point, the effect can even turn negative when consumers become irritated.

    Formally:

    • At $500 per month in advertising, an additional $100 might raise expected sales by $1,000.

    • At $1,200 per month, the same $100 increase may raise sales by only $300.

    • At $10,000 per month, another $100 might actually reduce sales.

    Thus, from theory, the assumption of constant returns is not reasonable: the marginal effect of advertising depends on the current level of spending.


    IV. Evidence from Data and the Need for Nonlinearity

    When plotting sales (Y-axis) against advertising spending (X-axis), the data form a curved, inverted-U shape, not a straight line.

    A simple fitted line does not capture this pattern. The linear model fails to represent the observed diminishing marginal returns.

    However, fitting a curved line, such as a quadratic curve, captures the pattern much better:

    • At low advertising levels, sales increase steeply.

    • As advertising increases, the slope flattens.

    • Beyond a certain point, the slope becomes negative.

    This motivates extending the model to include nonlinear terms.


    V. Modeling Diminishing Returns: The Quadratic Model

    To represent diminishing returns mathematically, a polynomial function is introduced.

    The simplest extension adds a quadratic term—the square of advertising spending. The model becomes:

    [
    E(\text{Sales}) = \beta_0 + \beta_1 \text{Advertising} + \beta_2 (\text{Advertising})^2
    ]

    Here:

    • (\beta_1) measures the linear (first-order) effect of advertising.

    • (\beta_2) measures the curvature of the relationship.

    Although advertising and its square are correlated, this is not a violation of the model assumptions because the relationship between them is not perfectly linear.


    VI. The Marginal Effect and Interpretation

    To see how the model captures diminishing returns, we compute the marginal return from advertising, i.e., the change in expected sales for a small change in advertising:

    [∂E(Sales)∂Advertising=β1+2β2Advertising][\frac{\partial E(\text{Sales})}{\partial \text{Advertising}} = \beta_1 + 2\beta_2 \text{Advertising}]

    This derivative shows that the return from advertising depends on the current level of advertising.
    If (\beta_2 < 0), the marginal return decreases as advertising increases — consistent with diminishing returns.

    Example (using estimated values):

    • At $1,500 in advertising, each additional dollar increases expected sales by $3.84.

    • At $1,750, each additional dollar increases expected sales by $2.46.

    • Around $2,000, the increase is about $1.00 — effectively break-even.

    • Beyond $2,500, additional spending may reduce sales by about $1.60.

    This demonstrates that as advertising spending rises, returns shrink, reach zero, and eventually turn negative — matching economic theory and real-world intuition.

    Graphically, the slope of the curve becomes flatter as advertising increases, crosses zero, and then declines.


    VII. Evaluating Model Fit

    When comparing the linear model and the quadratic model, the improvement is substantial.

    • Linear model: Explains 44.8% of the variation in sales (R² = 0.448).

    • Quadratic model: Explains about 51% (R² ≈ 0.51).

    Thus, by simply allowing the model to be nonlinear, about 6% more variation in sales is explained.

    Importantly, this improvement comes purely from changing the functional form—the variables are the same, but the relationship is now allowed to curve.

    A 6% gain from the functional form alone is considered dramatic, and it confirms that the quadratic model provides a much better fit to the data.


    VIII. Economic Meaning of Marginal Effects

    In a nonlinear model, the marginal effect is not constant, so we cannot summarize the effect of advertising with a single average number.

    For example:

    • A linear model might estimate that every additional $1 of advertising increases sales by $1.80.

    • But this average is misleading.

    If a firm is currently spending $400, this average underestimates the benefit — they should spend more.
    If a firm is spending $2,400, the same average overestimates the benefit — they should spend less.

    Hence, decisions based on the average effect can lead to poor choices.
    The appropriate decision depends on the current advertising level, because the marginal return changes with it.


    IX. Increasing Returns: The Housing Example

    The quadratic model can also capture increasing returns, where the marginal effect grows with the level of the variable.

    Model

    House price is explained by the size of the house (square footage):

    [E(Price)=β0+β1Size+β2(Size)2]E(\text{Price}) = \beta_0 + \beta_1 \text{Size} + \beta_2 (\text{Size})^2]

    If (\beta_2 > 0), the marginal effect increases with size—indicating increasing returns.

    Economic reasoning

    For every additional 100 square feet, a linear model assumes the same increase in price (e.g., $20,000).
    However, this is unrealistic because:

    • Larger houses are usually located in more expensive neighborhoods.

    • They tend to use higher-quality materials and luxury features.

    • Therefore, the return per 100 square feet should rise as house size increases.

    Data illustration

    In a wealthy area like Boca Raton, Florida, most houses range between 1,000 and 3,000 square feet.
    A linear model cannot accurately fit the relationship for large houses.
    By including a quadratic term with a positive coefficient, the model captures that larger houses yield higher returns for each additional square foot.


    X. Capturing Both Increasing and Decreasing Returns: The Cubic Model

    Sometimes theory suggests that returns are first increasing and then decreasing (or the reverse).
    In such cases, a quadratic model is not sufficient; we need a cubic polynomial.

    Example: Total Cost and Marginal Cost Functions

    A typical cost function for a firm relates total cost (C) to quantity produced (Q).
    Economic theory states that the relationship between total cost and output is not linear:

    • When production is low, costs rise slowly — due to efficient use of resources.

    • As production grows, costs begin to rise more quickly — due to diminishing productivity.

    Thus, total cost has an S-shaped (nonlinear) form.

    Marginal Cost

    The marginal cost (MC) is the change in total cost from producing one more unit:
    [
    MC(Q) = \frac{dC(Q)}{dQ}
    ]
    The marginal cost curve is typically U-shaped:

    • Initially decreases (as specialization increases efficiency).

    • Then increases (as overcrowding, bureaucracy, or inefficiency set in).

    Economic Intuition Example

    Imagine a software company:

    • With only one worker, that person must code, handle accounting, do sales, and clean the office. Productivity is low, so the marginal cost is high.

    • When the firm hires specialized workers (a programmer, an accountant, a salesperson, a cleaner), productivity increases, and marginal cost falls.

    • As the firm grows large (like Microsoft or Yahoo), coordination problems appear; employees become less efficient, and marginal cost rises again.

    This leads to the U-shaped marginal cost curve.


    XI. Representing This Econometrically: The Cubic Polynomial

    The total cost function can be expressed as a third-order polynomial:

    [
    C(Q) = \beta_0 + \beta_1 Q + \beta_2 Q^2 + \beta_3 Q^3
    ]

    Differentiating with respect to (Q) gives the marginal cost function:

    [
    MC(Q) = \beta_1 + 2\beta_2 Q + 3\beta_3 Q^2
    ]

    The marginal cost is therefore quadratic in (Q), which naturally allows for the U-shape predicted by theory.

    Expected signs

    • For a U-shaped marginal cost, the coefficient on the cubic term in the total cost function should be positive((\beta_3 > 0)).

    • Typically, for the total cost function, we expect:

      • (\beta_2 > 0) (cost increases with Q²),

      • (\beta_3 < 0) (ensures the initial flattening then acceleration of total cost).

    This combination yields a marginal cost curve that first decreases and then increases, perfectly matching the theoretical behavior.


    XII. Summary of Key Insights

    1. The linear regression framework can represent nonlinear economic relationships by including polynomial terms while staying linear in parameters.

    2. A quadratic term captures diminishing or increasing returns depending on the sign of its coefficient.

    3. The marginal effect is not constant—it depends on the level of the explanatory variable.

    4. Diminishing returns occur when the quadratic coefficient is negative ((\beta_2 < 0)); increasing returns occur when it is positive ((\beta_2 > 0)).

    5. Cubic terms allow modeling relationships that change direction (e.g., first increasing then decreasing).

    6. Economic interpretation should always guide the model specification—statistical fit must reflect theoretical reasoning.


    Would you like me to continue with Lecture 5 Part 2 (if you have it) so the notes continue seamlessly from this point?


    Lecture 5 (Part 2): Log Transformations, Interactions, and Nonlinear Models

    I. Big Picture: Three Ways to Handle Nonlinearity

    A. Overview

    1. Polynomials: capture diminishing/increasing returns.

    2. Log transformations: improve fit with skewed data and enable elasticity interpretation.

    3. Interaction variables: let the return to one variable depend on the level of another (often paired with a quadratic to capture diminishing returns).

    B. Focus of This Lecture

    • Use log transformations (log-linear, linear-log, log-log).

    • Add interaction effects (Education × Experience).

    • Add a quadratic term in Experience to reflect diminishing returns.

    • Compare models and extract takeaways.


    II. Log Transformations

    A. Log-Linear Model (log on left, levels on right)

    • Form: (ln⁡(Y)=β0+β2,Education+β3,Experience+u).(\ln(Y)=\beta_0+\beta_2,\text{Education}+\beta_3,\text{Experience}+u).

    • “Log” is natural log.

    • The regression is linear in parameters, but (Y=\exp(\cdot)) makes the relationship in levels nonlinear.

    B. Why Logs Help

    1. Monetary outcomes (wages, income, sales, expenditure) are often right-skewed.

    2. The log compresses large values a lot and small values only a little.

    3. Taking logs of a skewed income distribution produces something that looks approximately normal (“log-normal”).

    4. With fewer extreme outliers, the model fits better and estimates are more reliable.

    5. Using a transformed model is typically a good idea with very skewed data.

    C. Coefficient Interpretation in Log-Linear

    • One unit increase in a regressor changes expected (Y) by approximately (100×β)(100\times\beta) percent.

    • Example: (β2=0.04⇒)(\beta_2=0.04\Rightarrow) +1 year of education ≈ +4% wages; (β3=0.01⇒)(\beta_3=0.01\Rightarrow) +1 year of experience ≈ +1% wages.

    • Interpretation is about wages (levels), not about (\ln(\text{wage})).


    III. Other Log-Based Forms

    A. Linear-Log (levels on left, log regressor(s) on right)

    • Form: (Y=\beta_0+\beta_2 X_2+\beta_3\ln X_3+\beta_4 X_4+u).

    • Interpretation for (\ln X_3): 1% increase in (X_3) → approximately (0.01\times\beta_3) units of (Y).

    • Example: if (X_3) is income and (\beta_3=4), then 1% income increase → ≈ 0.04 more cars sold per week.

    B. Log-Log (elasticities)

    • Form: (\ln Y=\beta_0+\beta_2\ln X_2+\beta_3\ln X_3+u).

    • (\beta) is an elasticity: 1% change in (X) → (\beta)% change in (Y).

    • Price elasticity is negative; e.g., (\beta_3=-2.0) means a 1% price decrease increases quantity demanded by ≈ 2%.


    IV. Interactions: Returns That Depend on Another Variable

    A. Economic Motivation (Education ↔ Experience)

    1. Return to schooling depends on experience.

      • Two programmers, same education (12 years): Alice (20 years experience) vs Bob (1 year). An extra year of schooling benefits Bob more.

    2. Return to experience depends on schooling.

      • Two workers, same experience (5 years): Alice (PhD) vs Bob (high school). An extra year of experience benefits Bob more.

    B. Interaction Model and Marginal Effects

    • Model: (\ln(\text{Wage})=\beta_0+\beta_2,\text{Educ}+\beta_3,\text{Exp}+\beta_4(\text{Educ}\times\text{Exp})+u).

    • Returns (approx. % changes ×100):

      1. To Experience: (\dfrac{\partial \ln(\text{Wage})}{\partial \text{Exp}}=\beta_3+\beta_4,\text{Educ}) (expect (\beta_4<0)).

      2. To Education: (\dfrac{\partial \ln(\text{Wage})}{\partial \text{Educ}}=\beta_2+\beta_4,\text{Exp}) (expect (\beta_4<0)).

    C. Initial Estimation with Interaction Only

    • Implementation example: (C(\text{Education})#C(\text{Experience})).

    • Result: (\beta_4) negative but essentially zero; p-value ≈ 0.94 (highly insignificant).

    • (R^2) barely changes (≈ 19% before and after).

    • Conclusion at this step: interaction alone did not improve fit.


    V. Diminishing Returns to Experience: Add a Quadratic

    A. Theory

    • Returns to experience diminish as experience rises (1→2 years matters more than 20→21 years).

    B. Model with Interaction + Quadratic in Experience

    • Form: (\ln(\text{Wage})=\beta_0+\beta_2,\text{Educ}+\beta_3,\text{Exp}+\beta_4(\text{Educ}\times\text{Exp})+\beta_5,\text{Exp}^2+u).

    • Marginal return to Experience:
      [
      \frac{\partial \ln(\text{Wage})}{\partial \text{Exp}}
      =\beta_3+\beta_4,\text{Educ}+2\beta_5,\text{Exp}.
      ]
      • Interaction: depends on Education.
      • Quadratic: depends on Experience itself.

    C. Empirical Results with Quadratic Included

    • (\beta_4<0) and now large in magnitude and highly significant (significant at (\alpha=1%) and even lower).

    • (\beta_5<0) and highly significant (p-value ≈ 0), supporting diminishing returns.

    • (R^2) increases from ≈ 19% to ≈ 24% (≈ +5 percentage points) purely from functional-form change.

    • This allows returns from schooling to vary with experience and returns from experience to vary with education, improving explanatory power.


    VI. Worked Implications (Plug-In Illustration Mentioned)

    A. Using the Interaction Model

    • For two education levels (8 and 16 years), plugging in shows:
      • +1 year of experience raises expected wages by ≈ 0.6% at 8 years of education.
      • At 16 years of education, the return is slightly lower (≈ 0.57%); the difference is marginal.


    VII. Model Comparison and Guidance

    A. Do Not Drop Variables Based on One Misspecified Run

    • An insignificant interaction in the simpler model does not imply you should exclude it.

    • The earlier insignificance appears due to omitting an important term (the quadratic in experience).

    • Omitting important variables can bias results and cause issues (to be discussed under omitted variables).

    B. Try Multiple Reasonable Specifications

    • Compare models before concluding.

    • If an interaction is insignificant across well-specified variants, then dropping it is more defensible.

    • Among the three models examined, the interaction + quadratic model is preferred (theory-consistent signs, high significance, higher (R^2)).


    VIII. Why No Quadratic in Education Here

    A. Theory as the Starting Point

    • A quadratic in education is not supported by the economic theory used here (no belief in increasing returns of this quadratic form).

    B. Credential (Completion) Effects

    • The key pattern is discrete jumps at completion years (e.g., 12th → high school diploma; 16th → bachelor’s).

    • Within “dropout” status, the exact year matters much less than the credential.

    • This suggests the simple education-squared term is not the right way to capture education effects; completion matters most.


    IX. Summary of Interpretations (One-Line References)

    A. Log-Linear:

    +1 unit in (X_k) → ≈ (100\times\beta_k)% change in expected (Y).

    B. Linear-Log:

    +1% in (X) → ≈ (0.01\times\beta) units of (Y).

    C. Log-Log:

    (\beta) is elasticity (1% in (X) → (\beta)% in (Y)).

    D. Interactions:

    Returns depend on another variable’s level.

    E. Quadratic:

    Returns depend on the same variable’s level.

    F. Best Model Here:

    Interaction + quadratic in experience (higher (R^2), theory-consistent signs, high significance).

    Lecture 5 (Part 3): Joint Hypothesis Testing 

    Sometimes, a null hypothesis contains more than one equality — this is called a joint hypothesis (or joint conjecture).
    Each equality inside the null is one “restriction.” The number of restrictions equals the number of equal signs in the null.

    Example Scenario

    Consider a model with diminishing returns from advertising, where the question of interest is:

    “Does advertising have any effect on sales?”

    To answer this, we set up a hypothesis test.

    • Null hypothesis (H_0): Advertising has no effect on sales.

    • Alternative hypothesis (H_1): Advertising does affect sales.

    Mathematically:

    • For advertising to have no effect at all, both coefficients on the advertising terms must be zero:
      [
      H_0: \beta_3 = 0 \text{ and } \beta_4 = 0
      ]

    • If at least one of these coefficients is not zero, then advertising does have an effect:
      [
      H_1: \text{At least one of } \beta_3, \beta_4 \ne 0
      ]

    This means the null contains two equalities, so we are testing two restrictions simultaneously.
    Because the null includes more than one equality, this is a joint hypothesis test.


    II. Difference Between t-Tests and F-Tests

    A. t-Tests

    Previously, we used the t-test when testing one conjecture.
    It could involve:

    • A single parameter (e.g., (H_0: \beta_2 = 0)); or

    • A linear combination of parameters (e.g., (H_0: \beta_1 + 10\beta_2 = 1000)).

    Even if multiple parameters appear, as long as there’s one equality, it’s still one conjecture, so a t-test applies.

    B. Why the F-Test is Needed

    When the null hypothesis contains more than one equality (like (\beta_3 = 0) and (\beta_4 = 0)), we cannot use a t-test anymore.
    Instead, we use an F-test, which is designed for joint hypotheses.


    III. Restricted and Unrestricted Models

    To perform an F-test, we compare two models:

    1. Unrestricted Model

    The unrestricted model is the original, full model.
    It has no restrictions imposed. It includes all explanatory variables.

    2. Restricted Model

    The restricted model is formed by imposing the null hypothesis restrictions.
    In our example, this means setting:
    [
    \beta_3 = 0 \text{ and } \beta_4 = 0
    ]
    That is, removing the advertising variables.

    So:

    • The unrestricted model includes the advertising terms.

    • The restricted model removes them.


    IV. Formula for the F Statistic

    After estimating both models, we obtain their Sum of Squared Errors (SSE):

    • (\mathrm{SSE}_R): from the restricted model.

    • (\mathrm{SSE}_U): from the unrestricted model.

    The F-statistic measures how much worse the restricted model fits compared to the unrestricted one:

    [
    F = \frac{(\mathrm{SSE}_R - \mathrm{SSE}_U) / J}{\mathrm{SSE}_U / (n - K)}
    ]

    Where:

    • (J): number of restrictions in the null hypothesis (number of equalities).

    • (n): number of observations.

    • (K): number of parameters in the unrestricted model (including the intercept).

    Under the null hypothesis, (F) follows an F distribution with:

    • (J) numerator degrees of freedom, and

    • (n - K) denominator degrees of freedom.


    V. Decision Rule

    To decide whether to reject the null:

    1. Choose a significance level (α).
      Example: α = 0.05.

    2. Find the critical value from the F table using:

      • Row = numerator df = (J).

      • Column = denominator df = (n - K).

    3. Compare your sample’s F-statistic to the table’s critical value:

      • If (F > F_{\text{critical}}): Reject (H_0).

      • If (F < F_{\text{critical}}): Do not reject (H_0).


    VI. Intuition Behind the F-Test

    A. Why the Ratio Works

    The F-test compares two models’ errors.
    When we impose restrictions (like setting coefficients to zero), we are reducing the model’s ability to explain Y.

    • Removing valid explanatory variables causes SSE to increase.

    • This increase tells us the restrictions might not be valid.

    Therefore:
    [
    \mathrm{SSE}_R \ge \mathrm{SSE}_U
    ]

    If the null hypothesis were true, removing the variables wouldn’t matter much — both SSE values would be similar.
    But if removing them causes a large jump in SSE, it means those variables were important, and the null should be rejected.

    B. Interpretation

    • A large F value means the restricted model fits much worse, implying the null restrictions are false.

    • A small F value means both models fit similarly, so the restrictions may be valid.


    VII. Walkthrough Example 1 — Does Advertising Affect Sales?

    Step 1: Hypotheses

    [
    H_0: \beta_3 = 0 \text{ and } \beta_4 = 0
    ]
    [
    H_1: \text{At least one of } \beta_3, \beta_4 \ne 0
    ]

    This means the null states that advertising has no effect on sales, while the alternative says that advertising does affect sales.

    Step 2: Information Given

    • (n = 75) (observations)

    • (K = 4) (parameters in full model)

    • (J = 2) (restrictions)

    • Denominator df = (n - K = 71)

    Step 3: Estimate Both Models

    • Run the unrestricted model (with advertising terms) → get (\mathrm{SSE}_U).

    • Run the restricted model (without advertising terms) → get (\mathrm{SSE}_R).

    Step 4: Compute the F Statistic

    [
    F = \frac{(\mathrm{SSE}_R - \mathrm{SSE}_U) / 2}{\mathrm{SSE}_U / 71}
    ]

    Step 5: Decision

    • α = 0.05.

    • Critical value (F_{2,71;0.05} = 3.126.)

    • From the data: (F = 8.44.)

    Since (8.44 > 3.126): Reject (H_0).

    Conclusion:
    At least one advertising coefficient differs from zero. Advertising does have a significant effect on sales.

    Step 6: p-Value

    The p-value = (P(F_{2,71} \ge 8.44)).
    It is essentially 0, meaning the result is highly significant.
    This implies we would reject the null for practically any reasonable α.


    VIII. The Overall Significance of the Regression Model

    A. Concept

    The F-test can also be used to test the overall significance of a regression model — that is, whether the entire model explains the dependent variable at all.

    The null hypothesis is that none of the explanatory variables have any effect on Y:
    [
    H_0: \beta_2 = \beta_3 = \beta_4 = 0
    ]
    The alternative hypothesis is that at least one coefficient is not zero.

    B. Why It Matters

    This test checks whether your entire regression model is meaningful.
    If you fail to reject (H_0), it means that none of your explanatory variables help explain Y.

    C. Special Case Formula

    When all slope coefficients are zero under (H_0), the restricted model includes only an intercept.
    In that case:
    [
    \text{SSE}_R = \text{SST}
    ]
    because there are no explanatory variables, and all variation in Y comes from the error.

    The total sum of squares (SST) depends only on Y (the sample), so SST is the same for both models.

    Hence:
    [
    F = \frac{(\text{SST} - \text{SSE}_U)/(K-1)}{\text{SSE}_U/(n-K)} = \frac{\text{SSR}_U/(K-1)}{\text{SSE}_U/(n-K)}
    ]

    D. Example Walkthrough — Overall Model Significance

    For the sales example:

    • (n = 75)

    • (K = 4)

    • So (K - 1 = 3)

    • Denominator df = (n - K = 71)

    Null hypothesis:
    [
    H_0: \beta_2 = \beta_3 = \beta_4 = 0
    ]
    Alternative:
    At least one of them ≠ 0.

    Compute:
    From the unrestricted regression:
    [
    F = 24.459
    ]
    Critical value: (F_{3,71;0.05} = 2.73).

    Since (24.459 > 2.73): Reject (H_0).

    Conclusion:
    At least one explanatory variable explains sales — the model is meaningful.
    The p-value is essentially 0, which is typical, because it is very unlikely that all regressors would truly have no effect.


    IX. Relationship Between the F-Test and the t-Test

    A. Mathematical Connection

    For a single equality, the two tests are equivalent:
    [
    t^2_{(n-K)} = F_{(1, n-K)}
    ]
    So, squaring a t-statistic gives the corresponding F-statistic for one numerator df.

    B. Implications

    • For one equality, you can use either test (they give the same result).

    • For multiple equalities, you must use the F-test.

    • For inequality hypotheses (e.g., “>” or “<”), use t-tests, because F is one-tailed and cannot distinguish between sides.


    X. Why the F-Test Often Matters More

    When you have multiple coefficients to test together (like advertising terms), separate t-tests might fail to reject each one individually — but the joint F-test can still reject the null.

    This happens because the F-test captures the combined effect of multiple variables that might be individually weak but jointly important.


    XI. Summary of Key Ideas

    Concept

    Meaning

    Joint hypothesis

    Null has multiple equalities; test them together.

    t-test

    Tests one conjecture (one equality).

    F-test

    Used when the null has multiple equalities.

    Restricted model

    Model after imposing null restrictions.

    Unrestricted model

    Full model without restrictions.

    F-statistic

    Compares fit between restricted and unrestricted models.

    Decision rule

    Reject (H_0) if (F > F_{\text{critical}}).

    Overall significance

    Tests whether all slopes = 0.

    Relationship to t-test

    (t^2 = F) when (J=1).

    Inequalities

    Use t-test, not F-test.


    Final Takeaway

    The F-test generalizes the t-test.
    When testing more than one equality, or when testing whether the entire model is significant, use an F-test.
    If the F-statistic is large, the restrictions in the null (like setting advertising coefficients to zero) cause a large loss in explanatory power — meaning those restrictions are false, and the variables do matter.


    Would you like me to include the mathematical intuition and derivation (why dividing by (J) and (n-K) creates the F ratio and what it measures about variance ratios), written step-by-step in the same style?

    Lecture 6 (Part 1): Choosing the Best Econometric Model


    I. Importance of Model Choice

    In any econometric investigation, choosing the correct model is the most important decision. The quality of your results, and therefore the usefulness of your conclusions, depends entirely on whether the model specification is correct.

    To choose the best model, three questions must be addressed:

    1. What are the important considerations when choosing a model?

    2. What are the consequences of choosing the wrong model?

    3. How can we assess whether our model is adequate or not?


    II. Review: Correct Functional Form

    Previously, the importance of choosing the correct functional form was discussed. If the chosen form is wrong, the results are misleading.

    Example:
    Suppose a model of revenue as a function of advertising assumes constant returns when the true relationship has diminishing returns.

    • If advertising is low, an additional $100 investment increases revenue.

    • If advertising is already high, an additional $100 may reduce profits or even lead to losses.

    Even if the average effect shows a return of $2.00 per $1.00 spent, this average is meaningless for managerial decision-making.

    Conclusion:
    Choosing the wrong functional form leads to incorrect conclusions and potentially harmful business decisions.


    III. Two Additional Model Specification Issues

    Today’s focus is on two additional sources of model error:

    1. Omitted Variables – leaving out an important explanatory variable.

    2. Irrelevant Variables – including variables that do not belong in the model.


    IV. Omitted Variable Bias

    A. Definition and Causes

    An omitted variable is a relevant explanatory variable that is incorrectly left out of the regression model.

    Common causes include:

    • Negligence or lack of experience (overlooking relevant factors).

    • Data limitations, which are common in practice.

    When data are limited, we must understand the consequences of omission.


    B. Example: Family Income Model

    Suppose the true model explaining family income (( Y )) includes two variables:

    [
    Y_i = \beta_1 + \beta_2(\text{Husband Education}) + \beta_3(\text{Wife Education}) + \varepsilon_i
    ]

    Assume this is the correct model—only these two variables affect income.

    • Each additional year of husband’s education increases expected income by $3,100.

    • Each additional year of wife’s education increases expected income by $4,500.

    • Both coefficients are highly significant (( p \approx 0 )).

    Now, if we omit wife’s education and estimate:

    [
    Y_i = \beta_1 + \beta_2(\text{Husband Education}) + v_i
    ]

    we obtain a new estimate for husband’s education: $5,100 per year—still significant but overstated by $2,000.

    This means that omitting a relevant variable (wife’s education) biases the estimated coefficient of husband’s education.

    This phenomenon is known as Omitted Variable Bias (OVB).


    C. Mathematical Intuition for the Bias

    Let the true model be:

    [
    Y_i = \beta_1 + \beta_2 X_{2i} + \beta_3 X_{3i} + \varepsilon_i
    ]

    But the estimated model omits ( X_3 ):

    [
    Y_i = \beta_1 + \beta_2 X_{2i} + v_i
    ]

    The omitted variable’s effect becomes part of the new error term:

    [
    v_i = \beta_3 X_{3i} + \varepsilon_i
    ]

    From class 2, the OLS estimator for ( \beta_2 ) in the simple regression is:

    [
    b_2 = \frac{\sum (x_{2i} - \bar{x}2)(y_i - \bar{y})}{\sum (x{2i} - \bar{x}_2)^2}
    ]

    Substituting the true relationship and taking expectations gives:

    [
    E[b_2] = \beta_2 + \beta_3 \frac{\text{Cov}(X_2, X_3)}{\text{Var}(X_2)}
    ]

    Thus, the bias is:

    [
    \text{Bias}(b_2) = \beta_3 \frac{\text{Cov}(X_2, X_3)}{\text{Var}(X_2)}
    ]


    D. Interpretation of the Bias

    1. Existence of Bias

      • The estimator is unbiased only if ( \text{Cov}(X_2, X_3) = 0 ).

      • If ( X_2 ) and ( X_3 ) are uncorrelated, omitting ( X_3 ) has no effect.

      • But in most real-world cases, explanatory variables are correlated, creating bias.

    2. Magnitude of Bias

      • The bias is large when either:

        • ( |\beta_3| ) is large (the omitted variable has a strong effect on ( Y )), or

        • ( |\text{Cov}(X_2, X_3)| ) is large (strong correlation between regressors).

      • If both are large, the bias is substantial.

    3. Direction of Bias

      • The sign of the bias depends on the signs of ( \beta_3 ) and ( \text{Cov}(X_2, X_3) ):

        • Same signs → positive (upward) bias.

        • Opposite signs → negative (downward) bias.

    4. Persistence of Bias

      • The bias does not disappear as the sample size increases.

      • Therefore, the estimator is not only biased but also inconsistent.

    Even with millions of observations, if an important variable is omitted, the estimator will never converge to the true parameter.

    Thus, the OLS estimator under omission is biased and inconsistent, rendering it useless.


    E. Application to the Family Income Example

    1. Correlation Structure

      • Correlation between husband and wife education ≈ 0.6 (positive and significant).

      • People often marry partners with similar education levels.

    2. Coefficients and Signs

      • ( \beta_3 > 0 ): Wife’s education positively affects income.

      • ( \text{Cov}(X_2, X_3) > 0 ): Husband’s and wife’s education are positively correlated.

      • Therefore, bias is positive → upward bias.

      The estimated coefficient for husband’s education increases from $3,100 (true) to $5,100 (biased).
      The bias (2,000)islargebecause:</p><ul><li><p>Theomittedvariablehasa<strong>largeeffect</strong>((β3=4,500)).</p></li><li><p>Thecorrelationis<strong>high(0.6)</strong>.</p></li></ul><p>Together,theyproduceasubstantialpositivebias.</p></li></ol><divdata−type="horizontalRule"><hr></div><h4id="f8068601−b8ba−4f0a−a028−70b77aec9a1d"data−toc−id="f8068601−b8ba−4f0a−a028−70b77aec9a1d"collapsed="false"seolevelmigrated="true">F.SecondExample:AddingNumberofYoungChildren</h4><p>Supposethetruemodelactuallyincludesathirdvariable—the<strong>numberofchildrenunderagesix</strong>:</p><p>[<br>Yi=β1+β2(Husband Education)+β3(Wife Education)+β4(Young Children)+εi<br>]</p><p>Eachadditionalyoungchildreducesexpectedfamilyincomeby<strong>2,000) is large because:</p><ul><li><p>The omitted variable has a <strong>large effect</strong> (( \beta_3 = 4,500 )).</p></li><li><p>The correlation is <strong>high (0.6)</strong>.</p></li></ul><p>Together, they produce a substantial positive bias.</p></li></ol><div data-type="horizontalRule"><hr></div><h4 id="f8068601-b8ba-4f0a-a028-70b77aec9a1d" data-toc-id="f8068601-b8ba-4f0a-a028-70b77aec9a1d" collapsed="false" seolevelmigrated="true">F. Second Example: Adding Number of Young Children</h4><p>Suppose the true model actually includes a third variable — the <strong>number of children under age six</strong>:</p><p>[<br>Y_i = \beta_1 + \beta_2(\text{Husband Education}) + \beta_3(\text{Wife Education}) + \beta_4(\text{Young Children}) + \varepsilon_i<br>]</p><p>Each additional young child reduces expected family income by <strong>14,000, significant at the 1% level.

      If we omit this variable and estimate only with parents’ education, the coefficients on education change very little.

      Why? Because the correlation between young children and parents’ education is very small:

      • ( \text{Corr}(\text{Husband Education}, \text{Young Kids}) ≈ 0.1 )

      • ( \text{Corr}(\text{Wife Education}, \text{Young Kids}) ≈ 0.13 )

      Thus, the bias is negligible.

      • Since ( \beta_4 < 0 ) (negative) and covariance is slightly positive, the bias is negative (downward), but very small.

      • The estimated return to education changes from $3,200 to $3,100 — practically insignificant.

      Conclusion:
      We only worry about omitted variables when:

      • The omitted variable has a large effect, and/or

      • It is highly correlated with variables in the model.

      Every model omits something, but small or uncorrelated omissions are not a serious problem.


      G. Violation of OLS Assumptions

      The OLS estimator becomes biased because one of its classical assumptions no longer holds.

      In particular, the assumption that:

      [
      E(\varepsilon_i | X_i) = 0
      ]

      is violated.
      When a relevant variable is omitted, the error term contains systematic variation correlated with regressors, so the expected value of the error is not zero.

      As a result:

      • The estimator is biased.

      • It is also inconsistent.

      OLS is no longer BLUE (Best Linear Unbiased Estimator).


      V. Irrelevant Variables

      A. Definition and Problem Setup

      The opposite problem occurs when we add variables that should not be in the model.

      Although omitting variables is dangerous, including too many variables is also undesirable.


      B. Theoretical Problems

      A good model explains complex reality with relatively few explanatory variables.
      Adding unnecessary ones causes several problems:

      1. Overcomplication: The model becomes cluttered and hard to interpret.

      2. Overfitting:

        • The model fits the current sample well but fails to generalize.

        • Results lose meaning for the population.

      Example:
      Estimating wages based on:

      • Education

      • Type of degree (e.g., business vs. humanities)

      • Type of school (private/public, Ivy League or not)

      • Student rank, etc.

      If we keep adding details endlessly, the model becomes as complex as reality itself—useless for insight.


      C. Empirical Effects: Variance Inflation

      Adding irrelevant variables inflates the variance of the estimators.

      Example:
      Two artificial variables ( X_5 ) and ( X_6 ) were added to the family income model.
      These were correlated with education variables but had no real relationship with family income.

      • ( X_5, X_6 ): High correlation with husband and wife education.

      • As expected, their ( p )-values are large (insignificant).

      • However, after adding them, the ( p )-values of the real variables (education) increased.

      For instance, wife’s education was highly significant (( p \approx 0 )) before; after adding irrelevant variables, it became significant only at ( \alpha = 1% ).

      In other datasets, this inflation could even make an important variable appear insignificant at ( \alpha = 5% ).


      D. Consequences and Comparison with Omitted Variables

      Problem Type

      Bias

      Consistency

      Efficiency

      Practical Effect

      Omitted Variable

      Biased

      Inconsistent

      —

      Misleading estimates

      Irrelevant Variable

      Unbiased

      Consistent

      Inefficient

      Larger standard errors, higher p-values

      Including irrelevant variables:

      • Keeps OLS unbiased and consistent.

      • But increases standard errors, leading to smaller t-values and larger p-values.

      • This can make truly important variables appear statistically insignificant.

      Thus, adding “junk” variables can mask real effects.


      VI. Summary

      1. Omitted Variables:

        • Bias and inconsistency arise.

        • Bias depends on the size and correlation of omitted variables.

        • Leads to incorrect, misleading estimates.

      2. Irrelevant Variables:

        • OLS remains unbiased and consistent.

        • But inefficiency increases (larger standard errors).

        • Risk of concluding “no effect” when an effect exists.

      Both cases show why careful model specification is crucial.
      The goal is to balance simplicity and completeness—include all important variables, but only those that truly belong.


      Would you like me to continue with Lecture 6 Part 2 in the same structured format when you send it?

      Lecture 6 (Part 2): Model Construction, Testing, and Selection

      Model Construction, Testing, and Selection


      I. The Foundation of Model Building

      A. The Role of Economic Theory

      Every econometric model must begin with economic theory.
      Theory determines which variables belong in the model and what relationships to expect between them.
      It also guides the functional form of the equation — how variables are mathematically related to one another.

      Starting with the correct theoretical structure is essential.
      If the model is built on a poor theoretical foundation, no amount of statistical refinement will make it a good model.
      A weak starting point leads inevitably to a weak result, even with perfect data and advanced techniques.

      Before estimating any equation, it is crucial to research what others have done on similar topics.
      Most problems in economics have been studied before.
      Reviewing previous studies helps identify reasonable variables, functional forms, and expectations.
      Existing models can then be adapted to fit the specific context or dataset being analyzed.

      Economic reasoning is therefore the cornerstone — everything else builds on it.


      II. Evaluating an Estimated Model

      Once the model is estimated, it must be examined carefully to determine whether it makes sense in both statistical and economic terms.

      A. Checking Coefficient Signs

      The first step is to check whether the signs of the estimated coefficients align with theoretical expectations.
      If a variable that should increase the outcome shows a negative coefficient, or vice versa, this suggests potential misspecification.

      B. Checking Magnitudes

      The next step is to evaluate whether the magnitudes of coefficients are reasonable.
      For example, in wage equations estimated with data from developed countries, the expected return to an additional year of schooling is around 3–4%.
      If the estimate shows a return of 12%, this suggests something is wrong — possibly an omitted variable, which would bias the education coefficient upward.

      However, a 12% return might make sense in a developing country, where education is relatively scarce and its impact larger.
      Thus, magnitude assessment always depends on the economic environment and context of the data.


      III. Constructing the Wage Equation

      A typical example of model construction is the wage equation, where the goal is to explain variations in wages using individual characteristics.

      [WAGE=β0+β2(EDUC)+β3(EDUC2)+β4(EXP)+β5(EXP2)+β6(EDUC×EXP)+β7(HOURS)+ε][\text{WAGE} = \beta_0 + \beta_2(\text{EDUC}) + \beta_3(\text{EDUC}^2) + \beta_4(\text{EXP}) + \beta_5(\text{EXP}^2) + \beta_6(\text{EDUC} \times \text{EXP}) + \beta_7(\text{HOURS}) + \varepsilon]

      This model includes education, experience, and hours worked, along with squared and interaction terms to capture nonlinear effects.

      A. Expected Signs of Coefficients

      Variable

      Expected Sign

      Economic Reasoning

      Education (β₂)

      Positive

      More education increases productivity and wages.

      Education² (β₃)

      Positive

      Returns to education rise at higher levels (e.g., graduate degrees).

      Experience (β₄)

      Positive

      More experience generally raises wages.

      Experience² (β₅)

      Negative

      Diminishing returns to experience; later years add less.

      Education × Experience (β₆)

      Negative

      The benefit of an extra year of education falls for highly experienced workers.

      Hours Worked (β₇)

      Positive

      More hours lead to higher total pay.

      B. Capturing Education More Precisely

      Although an education-squared term allows for nonlinear returns, it is not the most accurate way to model education.
      Not all years of education matter equally.
      Completing a degree (such as a bachelor’s, master’s, or PhD) often causes a larger jump in wages than an additional year of schooling within a degree.
      A more precise approach would use dummy variables indicating the highest degree attained.
      These can capture discrete jumps in education levels, such as “has a bachelor’s degree” or “has a master’s degree.”


      IV. Evaluating the Estimated Wage Model

      When this model is estimated, the signs of the coefficients typically match expectations:

      • The coefficient on experience is positive.

      • The coefficient on experience² is negative, confirming diminishing returns.

      • Education is positive.

      • Education² is positive, showing increasing returns.

      • The interaction term between education and experience is negative.

      • The coefficient on hours worked is positive.

      However, a striking issue arises: none of the education variables are statistically significant.
      Their p-values are large, meaning that statistically, one cannot reject the idea that education has no effect on wages.

      This seems implausible — it is unlikely that education truly has no effect.
      Therefore, further testing is required.


      V. Testing the Significance of Education Variables

      A. Purpose of the Test

      To determine whether education as a group (education, education², and education × experience) has any joint effect on wages.

      B. Hypotheses

      [
      H_0: \beta_2 = \beta_3 = \beta_6 = 0
      ]
      [
      H_1: \text{At least one of } \beta_2, \beta_3, \beta_6 \neq 0
      ]

      The null hypothesis states that none of the education terms affect wages.
      The alternative states that at least one does.

      C. Interpretation of the Test

      Rejecting the null means education contributes to explaining wages.
      Failing to reject it would suggest that education has no role, which would contradict established economic reasoning.

      D. Conceptual Result

      The calculated F-statistic for this test is approximately 86.45, with a p-value near zero.
      This is an extremely strong result, meaning the null hypothesis is decisively rejected.
      Thus, education clearly affects wages.


      VI. Manual Calculation of the F-Test

      On an exam, the F-test must be computed manually from two regressions.

      A. Two Models

      1. Unrestricted model: includes all variables, including education terms.

      2. Restricted model: excludes the education variables entirely.

      B. Required Components

      • ( SSR_r ): Sum of squared residuals for the restricted model.

      • ( SSR_u ): Sum of squared residuals for the unrestricted model.

      • ( J ): Number of restrictions (here, 3: β₂, β₃, β₆).

      • ( n ): Number of observations (here, 1000).

      • ( K ): Number of parameters in the unrestricted model (6 explanatory variables + intercept = 7).

      C. Formula

      [
      F = \frac{(SSR_r - SSR_u)/J}{SSR_u/(n - K)}
      ]

      D. Decision

      In this case, ( F = 86.44 ) and the critical value at 5% significance is ( 2.6 ).
      Because ( 86.44 > 2.6 ), the null hypothesis is rejected.
      Education collectively affects wages.


      VII. Investigating Large Standard Errors

      Even though education matters, its estimated coefficients are individually insignificant.
      This may occur because of redundant or irrelevant variables in the model.
      Including unnecessary terms can inflate standard errors, making all coefficients appear less precise.
      To address this, smaller subsets of education variables should be tested to identify which are truly needed.


      VIII. Subset F-Test: Testing Education² and Interaction

      A. Hypotheses

      [
      H_0: \beta_3 = 0, ; \beta_6 = 0
      ]
      [
      H_1: \text{At least one of } \beta_3, \beta_6 \neq 0
      ]

      B. Interpretation of the Null

      If this null is true, the relationship between education and wages is linear.
      The return to each additional year of education is constant.
      For example, if ( \beta_2 = 0.04 ), every additional year of education increases wages by approximately 4%.

      C. Rejection of the Null

      Rejecting the null would mean the return to education is not constant — it varies depending on education or experience.

      D. Results

      The calculated F-statistic is approximately 2.8, with a p-value of 6.1%.
      At the 5% significance level, the null cannot be rejected.
      Thus, the relationship is linear, and the education² and interaction terms can be dropped.


      IX. The Simplified Wage Model

      After removing the redundant terms, the model becomes:

      [
      \text{WAGE} = \beta_0 + \beta_2(\text{EDUC}) + \beta_4(\text{EXP}) + \beta_5(\text{EXP}^2) + \beta_7(\text{HOURS}) + \varepsilon
      ]

      A. Interpretation

      • The relationship between education and wages is linear.

      • Each additional year of education increases wages by about 10%.

      • All remaining variables are statistically significant and have the correct signs.

      B. Comparison of Fit

      Model

      R² (%)

      Full model

      31.7

      Simplified model

      31.35

      Dropping the extra education terms barely changes explanatory power, confirming they were redundant.


      X. Criteria for Model Comparison

      To decide which model is best among several valid options, use model selection criteria.

      A. Adjusted R²

      R² always rises when variables are added, even if they are irrelevant.
      Adjusted R² corrects this by penalizing complexity:

      [
      \text{Adjusted } R^2 = 1 - \frac{(SSE/(n - K))}{(SST/(n - 1))}
      ]

      When a variable is added:

      • SSE decreases → adjusted R² increases.

      • K increases → adjusted R² decreases.

      Adjusted R² increases only if the improvement in fit outweighs the penalty for adding a variable.
      The model with the higher adjusted R² is preferred.

      B. Information Criteria: AIC and BIC

      Both the Akaike Information Criterion (AIC) and the Bayesian Information Criterion (BIC) include:

      1. A measure of fit (based on SSE).

      2. A penalty for model complexity.

      The formulas differ in the size of the penalty:

      • AIC uses ( 2K ).

      • BIC uses ( K \ln(n) ).

      Since ( \ln(n) > 2 ) for most samples, BIC penalizes extra variables more strongly.
      Therefore, BIC tends to prefer simpler models.

      Lower values of AIC and BIC indicate better models.


      XI. Example: Comparing Models

      A. Family Income Example

      A model including husband’s education, wife’s education, and the number of young children is considered best because:

      • It has the highest adjusted R².

      • It has the lowest AIC and BIC.

      • Plain R² is not informative because it always increases with more variables.

      B. Education Model Comparison

      Model

      Adjusted R²

      AIC

      BIC

      Preferred

      Full model

      31.73%

      1348

      1383

      AIC, Adj. R²

      Constant-return (linear only)

      31.07%

      1350

      1374

      BIC

      No education

      Very low

      Very high

      Very high

      —

      Although adjusted R² and AIC slightly favor the full model, the differences are minimal.
      The BIC provides a stronger preference for the simpler model.
      Therefore, the constant-return model is selected.


      XII. Rules for Model Comparison

      Information criteria are valid only when comparing models with the same dependent variable.
      For example, a model explaining wages cannot be compared directly to one explaining log(wages).
      The dependent variable must remain the same; explanatory variables may differ.


      XIII. Importance of Economic Reasoning

      Model selection methods — F-tests, adjusted R², AIC, and BIC — are refinements, not substitutes for theory.
      These tools help compare and improve models, but they cannot define the model in the first place.
      Only theory can determine which variables belong and what form the relationships should take.

      A strong wage model should include not just education, experience, and hours, but also key demographic factors such as gender and race, which significantly influence wages.
      Without these, even statistically sound models may give biased estimates.

      After additional learning, variables like gender and race can be introduced through dummy variables, improving the model’s explanatory power and realism.


      XIV. Summary of Key Steps

      1. Start with theory — it defines relevant variables and their expected signs.

      2. Estimate the model and check whether coefficients make sense.

      3. Test for significance of groups of variables with F-tests.

      4. Simplify intelligently by removing redundant terms.

      5. Compare models using adjusted R², AIC, and BIC.

      6. Prefer simpler models when fit differences are minimal.

      7. Maintain theoretical consistency — statistical tests do not replace sound economic reasoning.


      Would you like me to add a separate section after this that breaks down every test and model selection criterion (F-test, adjusted R², AIC, BIC) with short “exam reasoning walkthroughs,” showing how to state hypotheses, interpret outcomes, and decide between models step by step?

      Lecture 6, Part 3: Collinearity

      I. Understanding Collinearity

      Collinearity is another challenge in econometric modeling, but unlike model specification issues, it is not related to choosing the best model, the right explanatory variables, or the correct functional form. When a collinearity problem occurs, the issue lies not with the model itself but with the data. Collinearity is fundamentally a data quality problem.

      Collinearity arises when variables in non-experimental (observational) data move together in systematic ways. This typically happens in datasets that are the result of uncontrolled experiments, where economic variables may co-vary naturally. For instance, in survey data, husband’s education and wife’s education tend to be strongly correlated because individuals often choose partners with similar education levels. Such relationships appear in real-life data but would not exist in controlled experiments, where one variable is intentionally varied while all others are held constant. Controlled experiments allow isolation of the causal effect of one variable on another; observational data do not.


      II. Collinearity vs. Multicollinearity

      Collinearity does not violate the assumptions of the linear regression model. If it did, we would not be able to estimate models at all—yet we do so frequently, even when some explanatory variables are correlated. Recall from earlier lectures that the least squares estimator is designed to isolate the effect of each explanatory variable even in the presence of correlation among regressors.

      However, when the correlation among explanatory variables is very high, it can lead to a collinearity problem (or multicollinearity problem). Both terms are used interchangeably in the literature. The key distinction is that collinear variables simply refer to correlated explanatory variables, whereas a collinearity problem arises when this correlation is so strong that it interferes with estimation precision.


      III. Variance of the Estimator and the Role of Correlation

      Consider a multiple linear regression model with two explanatory variables, ( X_2 ) and ( X_3 ). The variance of the estimator ( \hat{\beta}_2 ) in this model depends not only on the usual factors—such as the variance of the error term and the sample size—but also on the correlation between ( X_2 ) and ( X_3 ).

      In the simple linear regression model, the variance of an estimator ( \hat{\beta}_2 ) is:

      [
      Var(\hat{\beta}_2) = \frac{\sigma^2}{\sum (X_2 - \bar{X}_2)^2}
      ]

      where:

      • ( \sigma^2 ) = variance of the error term

      • ( \sum (X_2 - \bar{X}_2)^2 ) = sample variance of the explanatory variable ( X_2 )

      This shows that ( Var(\hat{\beta}_2) ) is a function of three factors:

      1. Error variance (( \sigma^2 )) — Larger ( \sigma^2 ) increases the estimator’s variance.

      2. Sample size (( n )) — A larger ( n ) decreases variance.

      3. Variance of the explanatory variable (( X_2 )) — Higher variance in ( X_2 ) reduces the variance of ( \hat{\beta}_2 ).

      In the multiple linear regression model, variance also depends on the correlation coefficient ( \rho ) between explanatory variables ( X_2 ) and ( X_3 ). The formula includes a term that captures this correlation:

      [
      Var(\hat{\beta}2) = \frac{\sigma^2}{(1 - \rho{23}^2) \sum (X_2 - \bar{X}_2)^2}
      ]

      Here, ( \rho_{23} ) is the sample correlation coefficient:

      [
      \rho_{23} = \frac{Cov(X_2, X_3)}{s_{X_2} s_{X_3}}
      ]

      where ( s_{X_2} ) and ( s_{X_3} ) are the standard deviations of ( X_2 ) and ( X_3 ).
      The correlation coefficient ranges from −1 to +1, measuring the strength of the linear relationship between ( X_2 ) and ( X_3 ).

      As ( |\rho_{23}| ) approaches 1, the variance of ( \hat{\beta}2 ) increases dramatically. When ( \rho{23} = 1 ) or ( \rho_{23} = -1 ), the variance becomes infinite, meaning the estimator cannot be computed. Thus, high correlation leads to unstable and imprecise estimates.


      IV. Intuition Behind Collinearity Problems

      When two explanatory variables move closely together, it becomes difficult to distinguish their individual effects on the dependent variable ( Y ). Suppose ( X_2 ) and ( X_3 ) are positively correlated and both coefficients ( \beta_2 ) and ( \beta_3 ) are positive. When ( X_2 ) increases, ( Y ) increases; but since ( X_3 ) also tends to rise simultaneously, ( Y ) rises for both reasons. Consequently, it is impossible to tell how much of the increase in ( Y ) is due to ( X_2 ) and how much is due to ( X_3 ). This uncertainty increases the standard errors of ( \hat{\beta}_2 ) and ( \hat{\beta}_3 ).


      V. Exact Collinearity

      There are two cases of exact collinearity where the variance of the estimator becomes infinite, and the least squares estimator cannot be computed.

      A. Perfect Correlation Between Explanatory Variables

      When ( \rho_{23} = 1 ) or ( \rho_{23} = -1 ), ( X_2 ) and ( X_3 ) move exactly together, making it impossible to isolate their effects. The least squares procedure fails, and the estimator is not defined.

      B. Zero Variance in an Explanatory Variable

      When ( Var(X_2) = 0 ), the variable is constant across observations. This means ( X_2 ) is perfectly collinear with the intercept term. For example, if everyone in the sample has exactly 12 years of education, then education has no variation and its effect on wages cannot be estimated. Since the least squares method relies on variation in ( X ) to explain variation in ( Y ), the estimator cannot be defined in this case.


      VI. Near-Exact Collinearity

      In practice, exact collinearity is rare. More often, we encounter near-exact collinearity, where correlation is very high (close to ±1) or variance of a regressor is very low (close to 0). In such cases:

      • The OLS estimator is still defined, so regression results can be obtained.

      • However, these results may be unreliable because standard errors become large.


      VII. Consequences and Symptoms of Collinearity

      When multicollinearity is high but not perfect:

      1. Large Standard Errors
        High correlation inflates the standard errors of estimators. As a result, t-tests often fail to reject the null hypothesis that a coefficient equals zero, even when the variable is important. This can lead to incorrect conclusions about insignificance.

      2. Wide Confidence Intervals
        Large standard errors translate into wide confidence intervals, indicating that the estimates are imprecise.

      3. Sensitivity to Model Changes
        Estimates may change dramatically if a few observations are added or removed or if a correlated variable is included or excluded. Dropping a correlated variable can make another variable suddenly significant because the omitted variable’s effect becomes captured by the remaining one. Similarly, reducing sample variation (e.g., removing outliers) can increase collinearity.

      There is no fixed correlation threshold that determines when collinearity becomes a problem. For instance, even a correlation of 0.8 or 0.9 might be tolerable if the variance of explanatory variables is large enough. Thus, collinearity is not defined by a single rule—it depends on both correlation and variance.


      VIII. Collinearity as a Data Problem

      Collinearity is primarily a data problem, not a violation of the OLS assumptions. As long as estimates are significant and have the expected signs and magnitudes, there is no issue. However, when results are unexpectedly poor, collinearity may be the reason.


      IX. Example: Car Mileage Regression

      Consider a model estimating the effect of car characteristics on mileage (miles per gallon):

      [
      \text{Mileage} = \beta_0 + \beta_1(\text{Cylinders}) + \beta_2(\text{Engine Size}) + \beta_3(\text{Weight}) + u
      ]

      We expect all coefficients to be negative: more cylinders, larger engines, and heavier cars reduce mileage.

      The data, however, exhibit very high correlations:

      • Corr(Engine, Cylinders) = 0.95

      • Corr(Cylinders, Weight) = 0.89

      • Corr(Engine, Weight) = 0.93

      This occurs because in real life, manufacturers produce heavy cars with large engines and many cylinders.

      A. Step 1: Regression with Cylinders Only

      When mileage is regressed only on cylinders, the coefficient is significant with a near-zero p-value.

      B. Step 2: Adding Engine

      Including engine still yields significant results, despite the high correlation.

      C. Step 3: Adding Weight

      When weight is added, both engine and cylinders become insignificant, and their p-values increase dramatically. This suggests collinearity, as high correlations among explanatory variables make it difficult to isolate their individual effects.


      X. Testing for Collinearity

      To confirm collinearity, an F-test is used:

      [
      H_0: \beta_2 = \beta_3 = 0 \quad \text{vs.} \quad H_1: \text{At least one } \beta_i \neq 0
      ]

      This joint hypothesis test distinguishes between two cases:

      1. Both variables truly have no effect (irrelevant).

      2. Variables are relevant, but their effects cannot be separated due to collinearity.

      If ( H_0 ) is rejected, it means the variables jointly affect ( Y ) (mileage), confirming that the problem is collinearity rather than irrelevance.
      In the example, the F-statistic = 4.3 with p-value = 0.014 (< 0.05). Thus, ( H_0 ) is rejected, confirming collinearity.


      XI. Addressing Collinearity

      A. Option 1: Obtain More Data

      The best solution is to collect additional data. Larger sample sizes reduce variance and may increase variability among explanatory variables, which decreases standard errors. However, this is not always feasible.

      B. Option 2: Keep the Model Despite Collinearity

      If variables are relevant (as confirmed by the F-test), retain them even if they are individually insignificant. The model can still be used for prediction even if individual effects cannot be precisely identified. Predictions remain valid as long as the same relationships hold in future data.

      C. Option 3: Drop a Variable (Last Resort)

      Dropping one variable can reduce collinearity but introduces omitted variable bias if the omitted regressor is relevant. This option is sometimes used in fields like marketing, where Principal Component Analysis (PCA) is applied to remove redundant variables with similar meaning. For example, survey questions like “satisfied with product,” “would recommend to a friend,” and “would purchase again” are all highly correlated and can be represented by a single “customer satisfaction” factor. However, this approach is inappropriate when variables measure distinct concepts (e.g., engine size vs. cylinders).


      XII. Detecting Collinearity

      A. Pairwise Correlations

      Compute simple pairwise correlations among explanatory variables. High pairwise correlations (above 0.8) may indicate a problem.

      B. Auxiliary Regression (Variance Inflation)

      When collinearity involves multiple variables, pairwise correlations may not capture it. In that case:

      • Regress each explanatory variable on all the others.

      • Examine the ( R^2 ) of this auxiliary regression.
        If ( R^2 > 0.8 ), it suggests strong collinearity, as most of the variation in one regressor can be explained by the others.


      XIII. Summary

      • Collinearity is a data problem, not an OLS violation.

      • Exact collinearity (perfect correlation or zero variance) makes estimation impossible.

      • Near-exact collinearity causes inflated standard errors, wide confidence intervals, and unstable estimates.

      • If estimates are reasonable and significant, there is no problem.

      • Only check for collinearity when results appear unreasonably poor.


      Would you like me to export this Markdown-formatted version into a PDF or Word file next (keeping all the formatting and math clean for printing)?