Econometrics Exam 1
Lecture 1: Introduction to Econometrics
I. What Econometrics Is and Why It Matters
Econometrics combines economic theory, data, and statistical tools to answer “how much” questions about relationships in economics, business, and the social sciences.
It is not a branch of economics or statistics on its own—it is a tool that allows us to quantify relationships between variables. It uses theoretical understanding from fields like economics and business, along with statistical methods, to produce empirical, measurable results.
Econometrics is essential because businesses, policymakers, and researchers constantly face “how much” questionsthat require quantitative answers. Learning econometrics provides the skills to answer those questions and makes you valuable in today’s job market, where data-driven decision-making is key.
II. The Importance of “How Much” Questions
A. Real-World Decision Making
Every day, firms and institutions must make decisions that depend on quantifying effects. Examples illustrate how econometrics is used to guide such decisions.
Victoria’s Secret Example
The company must determine the effect of advertising on revenues.
Too little advertising → lost potential profit.
Too much advertising → wasted resources.
Using econometric methods allows them to estimate how much advertising affects sales and find the optimal level of spending.
UTD Tuition Example
Suppose UTD (University of Texas at Dallas) wants to raise tuition to increase revenue.
When tuition rises by 10%, enrollment will fall—that’s known intuitively.
But the crucial question is by how much will enrollment fall?
If enrollment drops by 20%, revenue decreases (bad idea).
If enrollment drops by only 5%, revenue increases (good idea).
To make this decision intelligently, UTD must estimate the price elasticity of demand for education—the percentage change in enrollment given a percentage change in tuition.
This “how much” question is exactly what econometrics helps answer.
Toyota and Tesla Example
Toyota needs to estimate the effect of a new Tesla model on sales of its Prius hybrid.
If Tesla’s new car reduces Prius sales by only 5%, Toyota can wait before investing heavily in a new electric car.
But if Prius sales drop by 40%, Toyota must invest immediately.
Econometric analysis quantifies the competitive impact and informs long-term strategic investment decisions.
Personal Example – Education and Wages
Even individuals make “how much” decisions. For instance, when choosing to pursue a Master’s in Business Analytics, students implicitly estimate how much higher their future wages will be.
If the expected wage increase is only 5%, it might not be worth it.
If the expected increase is around 25%, it likely is.
Most people rely on previous econometric research that estimates the return to education—for example, that a master’s degree raises wages by about 25%.
In this course, you will learn how to estimate these kinds of relationships using data and econometric tools.
III. From Economic Models to Econometric Models
To estimate relationships such as between price and demand or advertising and sales, we must first formulate an econometric model.
Every econometric model begins with an economic model—a theory that explains how variables relate to one another. Without theory, econometric work is meaningless.
A. Economic Models
Economic theory often expresses relationships between variables using functions.
For example, the demand for Coke can be expressed as a function of several factors:
Each factor (price of Coke, substitutes, complements, and income) affects demand in a predictable way based on economic reasoning.
IV. Illustrating Causality – The Deer Crossing Example
To show why it’s difficult to determine causal relationships, consider the “Deer Crossing” story:
A caller once complained that deer crossing signs should be moved to safer areas—like school zones or small-town roads—because “too many deer get hit on highways.”
She believed deer would see the signs and cross there, misunderstanding that the signs are for drivers, not deer.
The story highlights a crucial lesson: causality is not always obvious. People often confuse correlation (two things happening together) with causation (one causing the other).
Econometrics provides the tools to correctly identify causal effects—distinguishing whether advertising increases sales, or if high sales simply lead to more advertising.
V. The Structure of an Econometric Model
An econometric model consists of two parts:
The systematic (deterministic) part
The random (stochastic) part (the error term)
A. Systematic Part
This is derived from the economic theory and includes:
Dependent variable (Y) – the variable being explained (e.g., quantity demanded).
Explanatory variables (X’s) – the factors that explain Y (e.g., price, income).
Coefficients (β’s) – the parameters showing how much each X affects Y.
Functional form – the assumed mathematical shape of the relationship (often linear).
For example, a simple linear demand model might look like:
The β’s represent true effects that exist in nature (e.g., , etc.), but we don’t know their values.
The goal of econometrics is to estimate these β’s using data.
B. Random (Error) Component –
The random error term represents unpredictable factors or omitted influences that affect Y but are not included in the systematic part.
For instance, in the Coke demand model:
Some localities may have a big wedding (more Coke sold).
Others may have anti-soda campaigns (less Coke sold).
Some may experience a heatwave (more Coke consumed).
These unpredictable differences are captured in ε.
If an important variable (like temperature) is left out of the model, it still affects demand in reality. Since it’s not in the systematic part, it ends up inside the error term, making no longer “pure noise.”
This omission creates systematic bias—a problem that complicates econometric estimation.
For now, we assume is pure random noise, but later in the course we’ll explore what happens when it isn’t.
VI. Uses of Econometric Models
Once an econometric model is built and estimated, it serves three key purposes:
Estimation: Quantify the strength and direction of relationships between variables.
Prediction/Forecasting: Predict future values of Y given hypothetical changes in X’s.
Example: A store manager can forecast how Coke demand changes if price drops by $0.20 and chips rise by $0.05.
Hypothesis Testing: Test beliefs or theories using data.
Example: UTD’s Tuition Belief
UTD believes that demand for schooling is inelastic (students are not sensitive to tuition changes).
If that’s true, increasing tuition should raise total revenue (since enrollment would fall by less than the tuition increase).
But econometric testing can check this belief:
If data reject it, UTD must revise its policy—perhaps realizing that raising tuition would actually reduce revenue.
VII. Data and Where It Comes From
Econometric analysis requires data to estimate models and test hypotheses.
A. Controlled Experiments
In a controlled experiment, researchers vary one variable while holding all others constant.
This allows clear identification of cause and effect.
Example: Testing a new weight-loss pill
Treatment group: receives the real pill.
Control group: receives a placebo.
If the treatment group loses 5 pounds more, that’s the pill’s estimated effect.
However, controlled experiments are rare in business or social sciences because they are costly or infeasible.
For example:
A company studying advertising effects would need to fix everything except advertising for months—impractical.
In studying race discrimination in wages, you cannot “change” someone’s race for experimental purposes.
Thus, in business and economics, we often rely on non-experimental (observational) data.
B. Non-Experimental or Observational Data
These are retrospective data already collected, not designed as experiments.
Economists must extract causal relationships even though many variables change simultaneously.
A common source is survey data, such as information collected from 1,000 households.
Characteristics of survey data:
All variables are collected simultaneously (e.g., race, income, family size, education).
Samples are random and non-replicable: Every survey yields slightly different samples and results.
Because every sample is different, the estimates from each sample vary.
This raises a key question: How close are our sample-based estimates to the true parameters in the population?
This leads to one of the central goals of econometrics—making inferences about the population from a sample.
VIII. Types of Data in Econometrics
There are three major types of data used in econometric analysis:
A. Cross-Sectional Data
Collected at one point in time across many individuals, firms, or entities.
Each entity is observed once.
Example: A dataset of 1,000 individuals with information on education, wage, age, and gender—all recorded in the same year.
B. Time Series Data
Collected over time for a single entity (a country, firm, or person).
Data are recorded at regular intervals (daily, monthly, quarterly, or yearly).
Example: U.S. GDP from 2001–2008 recorded annually.
Regular time spacing is essential for time series analysis.
C. Panel (Longitudinal) Data
Combines cross-sectional and time series features.
Observes many entities over several time periods.
Example: 1,000 firms tracked for 5–7 years, recording production, employment, and area cultivated.
In panel data, the number of entities (N) is typically much larger than the number of time periods (T).
IX. The Research Process in Econometrics
A proper research process always begins with economic theory, not with econometrics.
Steps:
Start with economic theory – identify relevant variables and expected relationships.
Formulate the econometric model – specify the dependent and explanatory variables and the functional form.
Collect data – obtain an appropriate sample (cross-sectional, time series, or panel).
Select statistical tools – choose estimation methods consistent with your model and data type.
Estimate the parameters (β’s) – using software and econometric techniques.
Use results for:
Prediction and forecasting
Hypothesis testing
Policy or business recommendations
Econometrics is the tool for the inference stage, not the starting point of research.
X. Preparation for Next Lecture
Before proceeding to the next class, review fundamental probability and statistics concepts, since econometrics builds on them.
Key topics to review:
Random variables
Mean and variance
Covariance and correlation
Normal distribution
These are summarized in the Probability Review PowerPoint and the introduction chapter of your textbook (before Chapter 1).
Complete Homework #1, which checks your understanding of these probability concepts.
XI. Looking Ahead
In the next lecture, the course will introduce the simple linear regression model, explore the least squares estimator, and examine the properties of estimators.
This concludes the overview of what econometrics is, why it’s important, how models are structured, where data come from, and how theory connects to estimation and inference.
Lecture 2 (Part 1): The Simple Linear Regression Model
I. Introduction to the Model
The goal is to estimate the relationship that exists in nature—meaning the real world or overall population—between household income and spending on food.
We want to determine, in the real world, the expected spending on food for every additional $100 of household income.
In nature, both real-world expenditures and income are random variables. Therefore, we need an econometric modelthat allows us to study the relationship between these two random variables.
Although the objective is to learn about the relationship that exists in the population, in practice we only use a sample. The relationship estimated in the sample is then used to extrapolate to the real world.
II. Building the Econometric Model
Every econometric model begins with an economic theory, which determines the systematic part of the model.
Economic theory tells us that expenditure on food depends on income, not the other way around.
Therefore:
Dependent variable (Y): Expenditure on food
Independent (explanatory) variable (X): Income
To the systematic part that comes from theory, we add a random error term that represents all non-systematic influences on the dependent variable.
The combination of the systematic component and the random component forms the econometric model.
III. The Simple Linear Regression Model
The simple linear regression model is the most widely used econometric model.
It is called “simple” because it contains only one explanatory variable. However, despite the name, it is not necessarily simple in practice.
The model consists of two parts:
Systematic part (deterministic): Derived from theory
Random part: Represents unobserved factors or pure noise
Mathematically, the model is expressed as:
Spending on food
: Income
: Intercept of the regression line
Slope (change in expected spending for each unit increase in income)
: Random error term
IV. Assumptions About Reality (Nature)
We assume that, in nature, there is only one variable—income—that systematically explains spending on food, and that the relationship is linear.
That means the true relationship between income and spending on food is perfectly described by:
However, this assumption is not true in reality.
A. Missing Variables
Other important variables can explain food spending:
Household size: Larger families consume more.
Location: Food costs vary by city (e.g., New York vs. Dallas).
B. Non-Linearity
The relationship is likely non-linear.
Example:
A household earning $2,000 per week might spend 10% ($200) on food.
A household earning $10,000 per week will not spend $1,000 (also 10%)—there is a limit to consumption.
Thus, the proportion of income spent on food decreases as income increases. Spending represents a large share for poorer households and a smaller share for richer ones.
Even though this model is wrong as a description of reality, we use it as a starting point.
Later, more advanced models will relax these simplifying assumptions.
V. Estimation Problem
Our task is to estimate the unknown parameters and .
These parameters exist in nature (the population) but are unobserved.
We use our sample data to obtain estimates:
estimates
estimates
Notation difference:
Greek letters (): True, unobserved parameters in the population
Latin letters (B): Sample-based estimates
Example:
True value (in nature):
Estimated value (from sample):
The estimate may be close or far from the true parameter, but since the true β is unobserved, we can never know the distance between them.
The best we can do is develop an estimation procedure that increases the probability that our estimate is close to the true .
Even with the best tools, we can never know for sure if our estimate is close to the true parameter.
VI. Graphical Representation
If the model is correct and the error term represents pure noise, then:
Thus, the expected value of Y (spending on food) is:
The blue line represents the true relationship in the population between income and expected spending.
(intercept): Expected food spending for a household with zero income.
(slope): Expected increase in food spending for each additional $100 of income.
However, because of random error, this blue line cannot be observed directly.
A. Role of the Error Term
Each household experiences a random disturbance:
A positive error : Spends more than expected (e.g., preference for organic food)
A negative error : Spends less (e.g., dieting)
Therefore, actual observed spending (black points) deviates from the expected value (blue line).
If there were no random noise, all observations would lie perfectly on the blue line, and we would know the true parameters exactly.
Because of noise, we must estimate the blue line by fitting a red line using statistical tools.
The goal of econometrics is to estimate the true (blue) line as closely as possible with the estimated (red) line.
We can only increase the probability that the red line is close to the blue one—we can never know for sure.
VII. Assumptions of the Simple Linear Regression Model
To estimate the model, we rely on several key assumptions.
Assumption 1: The explanatory variable (X) is non-random
Although income is clearly random in reality, this assumption is made for methodological simplicity.
It allows controlled experimentation where the researcher can choose the values of X.
In practice, we only observe incomes; we do not control them.
However, the results of the model do not depend on this assumption—it only simplifies the mathematical proofs.
Assumption 2: The expected value of the error term is zero
If the error term is pure noise, its average should be zero.
However, this assumption implicitly requires that the model be correctly specified.
Example:
If we omit an important variable (like household size), it will end up inside the error term.
Then, the error term will contain systematic information, and .
In that case, our model becomes misspecified, and many results will no longer hold.
For now, we assume this condition is true.
Assumption 3: Zero covariance between errors
If X is non-random, then Y is random only because of ε.
Thus, zero covariance between ε’s also means:
Interpretation: Knowing one household’s food spending tells us nothing about another household’s spending.
This implies that the sample is random and households are independent.
Independence is stronger than zero covariance.
Zero covariance means there is no linear association, but not necessarily full statistical independence.
The model only requires the weaker assumption of zero linear association.
Assumption 4: Constant variance of the error term (Homoscedasticity)
The variance of ε is constant across observations.
Since X is non-random, Y has the same variance as ε.
However, in real data this is often violated.
Example (scatter plot):
In a sample of 40 households:
For low-income households, the range of food expenditure is small.
For high-income households, the range is larger.
This pattern indicates increasing variance—the variance of (and Y) grows with income, a phenomenon called heteroskedasticity.
Reason:
Low-income individuals mostly buy basic items, so spending varies little.
High-income individuals have more diverse choices (eating out, organics, imports), leading to greater variability.
Although these assumptions are unrealistic, they are necessary to begin with. Later in the course, we will analyze models that relax them.
Assumption 5 (Optional): Normal distribution of errors
That is, the errors are normally distributed around a mean of zero with constant variance.
This assumption is not crucial because the Central Limit Theorem allows us to derive similar results even when errors are not perfectly normal.
Under this assumption, Y is also normally distributed:
Graphically, the distribution of Y is identical in shape to that of ε, only shifted by the mean
Lecture 2 (Part 2): OLS Estimation
I. What We Are Estimating and Why
A. The Two Population Parameters and the True Line
There are two parameters that exist in the population in nature: β₁ (the intercept) and β₂ (the slope). These two parameters define a true regression line—visualized as a blue line—that captures the true relationship between income and spending on food. The intercept of this true line is β₁, and the slope is β₂.
B. The Fundamental Problem: We Cannot See the True Line
Nobody can observe this true regression line. The reason is the presence of errors—the unobserved disturbances that hide the true relationship between the variables. Because of these errors, all we can actually see are the observations (the points in our sample). Therefore, we must estimate the true line and produce an estimated regression line, also called the fitted line. The blue line is the true line (unobservable); the fitted line is the red line we compute from the data. The goal is to make the fitted line as close as possible to the true line (which we will never see).
II. Data and Sampling
A. Need for a Random Sample
To estimate the model, a random sample from the population of interest is required. If the sample is not random, it will not be representative, and we will not be able to generalize from the sample to the population.
B. The Sample Used
The sample consists of 40 households. For each household we have:
Weekly income (measured in units of $100), and
Weekly food expenditure (measured in dollars).
Examples (as described):
Household #1 earns $369 per week.
Household #39 earns $29,140 per week.
Household #2 spends $135.98 per week on food.
Summary statistics (as presented):
The average household in the sample makes $19,160 per week and spends $283.57 per week on food.
Also shown are the median, maximum, and minimum values for income and spending.
(Clarification of units: in subsequent estimation work, income (X) is handled in $100 units. For example, (X = 20) means $2,000 of weekly income. The later numerical work uses this convention, and the means used there are and .)
C. Initial Visualization
The scatter plot of the sample suggests a positive relationship between income and spending on food. Our objective is to draw the fitted regression line—our estimate of the true line.
III. Why Intuition Alone Is Not Enough
A. Intuitive Lines Can Differ Widely
One might try to draw a line that appears to pass “close” to the cloud of points. But there are many lines that could look reasonable by eye and yet be very different.
Line 1 yields an estimate that for every $100 increase in income, spending on food increases by $21.
Line 2 yields an estimate that for every $100 increase in income, spending on food increases by $12.
These are very different results. Therefore, we need a systematic and accurate procedure for estimating the line.
IV. Ordinary Least Squares (OLS): The Core Idea
A. The Fitted Line and Notation
Assume we draw a fitted regression line:
where and are estimates of the two unknown population parameters and . Any equation with hats and (b)’s refers to the fitted (estimated) line, not the true line.
Given and , the prediction for observation (i) is .
B. Residuals vs. True Errors
From the fitted regression line, we compute residuals:
Residuals are the differences between the observed values and the fitted values. Residuals are not the true errors. They are our estimates of the unobservable true errors.
The true errors are the differences between the observations and the unobservable true line (blue).
The residuals are the differences between the observations and the fitted line (red).
For each fitted line we draw, we obtain a different set of residuals.
C. The OLS Objective: Minimize the Sum of Squared Residuals
The Least Squares idea is to choose the fitted line that yields the smallest possible sum of squared residuals:
Why squared residuals?
We want the line to be as close as possible to the observations we can see.
If we summed raw residuals, positive and negative values would cancel.
Squaring makes every term nonnegative, so nothing cancels.
We then sum these squares to form the SSE.
(Alternative considered and set aside for now: minimizing the sum of absolute residuals. While it also avoids cancellation, the least squares estimator has extremely good statistical properties. This motivates using squared residuals. These properties will be shown later.)
D. The Best Line in the Least Squares Sense
Let denote the fitted line obtained by OLS. If denotes any other fitted line (obtained by any other method), then:
.
That is, the OLS fitted line has a smaller sum of squared residuals than any other fitted line.
V. Solving the Minimization Problem (How the Estimators Are Derived)
(This derivation is to build intuition. It is not required on the exam. The point is to show where the formulas come from rather than to present them as given.)
We minimize
with respect to and .
A. First-Order Condition with Respect to
Differentiate and set to zero:
Rearrange:
Using and :
B. First-Order Condition with Respect to
Differentiate and set to zero:
Substitute into this condition and simplify to obtain the slope estimator . The algebra yields:
C. Two Equivalent Presentations and a Useful Identity
There are two interchangeable ways to write the algebra in these derivations:
Deviation-from-the-mean form: use and .
Raw form: use and directly.
The bridge between them is the identity:
Because and are constants (no index ), they can be factored out of summations, which makes it straightforward to show terms involving sums of deviations from the mean evaluate to zero when appropriate. We will use this trick repeatedly to switch between the two presentations whenever convenient.
VI. What the Estimators Mean
A. Estimator vs. Estimate
Estimators are formulas—they are functions of random sample data and hence are random variables.
Estimates are the numerical results obtained when we plug our specific sample into those formulas. Different samples produce different estimates.
B. Two Key Takeaways from the Formulas
The fitted line passes through the point of sample means .
This follows immediately from : substituting gives .Interpretation of in covariance–variance form:
.
This is intuitive: the slope measures how strongly (X) and (Y) move together, so it should naturally depend on their covariance, scaled by the variation of (X).
VII. Computing the Least Squares Estimates for the Sample
A. Plugging the Sample into the Estimators
Applying the estimator formulas to the sample yields the fitted regression line:
with (X) measured in $100.
: if income increases by $100, the expected weekly spending on food increases by approximately $10.21.
: for a household with zero income , the predicted weekly food spending is $83.42.
B. The Fitted Line Goes Through the Means
The line passes through . Using the sample means displayed:
(i.e., $1,916 when income is in $100 units),
(i.e., $283.57 per week).
Graphically, the fitted line goes through this point.
VIII. Interpreting the Intercept Carefully
The intercept is the predicted spending when (no income). Here, the estimate implies 2,000. Since income is measured in $100, this corresponds to (X = 20). Substitute into the fitted line:
.
Prediction: $287.61 per week on food.
X. Income Elasticity
A. Definition and Formula
Income elasticity measures the responsiveness of spending to income in percentage terms. It answers: if income increases by 1%, by how many percent does spending change?
For the linear fitted model, the elasticity of (Y) with respect to (X) at a point (X,Y) is:
Elasticity changes with the point (X,Y); it is not constant in the linear model.
B. Elasticity at the Means (A Common Practice)
Because elasticity depends on the specific (X,Y), it is typical to evaluate it at the sample means to characterize an “average” household:
Using the estimates and means:
Interpretation: A 1% increase in weekly household income leads, on average, to a 0.71% increase in weekly household food expenditure.
XI. Summary of the OLS Outcome and Its Logic
Because the true line is hidden by errors, we estimate a fitted line using a random, representative sample.
The OLS method chooses the line that makes the sum of squared residuals as small as possible.
The OLS estimators are obtained by minimizing the SSE with respect to and .
(
The fitted line passes through (
The slope equals sample covariance of (X,Y) divided by the sample variance of (X).
With the estimated line (with (X) in $100), predictions and elasticity calculations follow directly.
The intercept’s literal interpretation at (X=0) may or may not be substantively meaningful and will be revisited later.
Lecture 2 (Part 3): Properties of Estimators and Unbiasedness of OLS
I. Estimators as Random Variables
A. An estimator is a formula you can apply to any sample.
When you plug your sample’s values into estimators such as and , you get point estimates for that specific sample.
B. Because the estimator is built from a random sample, the estimator itself is a random variable.
Therefore, each estimator has a mean and a variance.
C. By examining the mean and variance of an estimator, we can judge whether it is a good estimator or not such a good one.
II. Three Key Properties We Want
A. Unbiasedness — You want the estimator to be unbiased.
B. Efficiency — You want the estimator to be efficient.
C. Consistency — You want the estimator to be consistent.
The plan is to define these properties precisely, then check whether the least squares (LS/OLS) estimators have them. This will let us compare least squares estimators to alternatives and decide whether they are good or whether better ones exist.
III. Unbiasedness: What It Means
A. Definition
Unbiasedness means that if you took estimates from many samples and averaged them, that average would approach the true parameter. Formally, if , the estimator is unbiased.
B. What unbiasedness does not mean
Unbiasedness does not mean that the estimate from any one sample is close to the true parameter. It makes no promise about a single estimate; it is a statement about averages across many samples.
C. Illustration with multiple samples
Consider 10 samples, each with 40 observations, all drawn from the same population (e.g., U.S. households). Look at estimates of , interpreted here as “by how much expected food spending goes up when income increases by $100.” You might see estimates like $6.10, $8, or $13.
What is the true parameter—6, 13, or 27? We cannot know, and in fact we will never be able to know the true parameter exactly.
But if the estimator is unbiased, there is still something very important you can say: with many, many samples, if you compute the estimate for each sample and then take the average of those estimates, on average you will get the true parameter
In other words: on average, we are right. This says nothing about whether your particular sample’s estimate is close or far from the truth.
IV. Goal: Show Least Squares Is Unbiased
A. Objective
We want to prove that the least squares estimator is unbiased. Mathematically, we want to show:
B. About the Algebra
The proof involves some mathematics. It is presented to enhance understanding—you will not be asked to derive or prove formulas on the exam. The payoff is seeing exactly why the assumptions matter and what happens if they do not hold.
V. Setting Up the Proof
A. Two forms of the OLS estimator
The least squares estimator for can be written in two ways. One is the form in terms of deviations from the mean. That form is more convenient here.
B. Define Weights
Denote by the weight that comes from that deviations form. Then the estimator can be written as a linear combination:
where (N) is the number of observations. This shows directly that the least squares estimator is a linear estimator:
C. Substitute the model for
Use the (assumed correct) model:
Substituting into the estimator gives:
D. What we need to show
To isolate (, we will show:
If both hold, then:
VI. Establishing the Two Key Identities
A. Showing
Recall how is defined from deviations from the mean.
In this deviations form, the numerator is and the denominator is the sum of squared deviations . Thus:
The sum of deviations from the mean is zero:
Therefore,
so the term is zero.
B. Showing
Using the same deviations form, the numerator becomes the same sum that appears in the denominator:
This ratio equals 1 because the denominator is exactly the same sum of squared deviations that the numerator reduces to in this setup. Hence:
C. Result after the two identities
With and , we have:
VII. Taking Expectations
A. Expectation of the estimator.
Take expectations of both sides:
since is a constant (a parameter), not a random variable.
B. The convenience assumption about (X).
For this demonstration, adopt the simplifying assumption that (X) is not treated as random. (This is only for convenience; the proof can be done without it, but it would require conditional expectations and would be more involved.) With this convenience assumption, the weights can be taken outside the expectation.
C. Zero mean of the error term.
By the model assumption, Therefore:
D. Conclusion for
Thus,
This proves that the least squares estimator is unbiased under the stated assumptions.
VIII. The Assumptions Used and Their Role
A. Correct model specification
We assumed we wrote the correct model that represents reality precisely, so substituting is valid.
B. Zero-mean error
We assumed
C. Convenience treatment of (X)
For simplicity in the algebra, (X) was treated as non-random so that the weights could be taken outside the expectation. This is a convenience, not a necessity for the truth of the result.
D. What if assumptions fail?
If these assumptions do not hold, then we are in trouble—the estimator will generally not be unbiased. This issue will be addressed later.
IX. Unbiasedness of
By analogous reasoning, the expected value of the estimator equals the true parameter under the same assumptions:
Lecture 2 (Part 4): Variance and Efficiency
I. Concept of Variance and Efficiency
A. Comparing Unbiased Estimators
We look at two distributions of unbiased estimators.
Both are unbiased because the mean of each estimator equals the true parameter β.
The estimator with the smaller variance is preferable, since it is more likely to produce an estimate closer to β.
Therefore, among unbiased estimators, we prefer the one with the smaller variance.
B. Definition of Efficiency
Efficiency compares two candidate unbiased estimators:
If one estimator has a lower variance than another, it is said to be more efficient.
When comparing unbiased estimators, we always choose the one with the smaller variance because:
It gives an estimate that, with higher probability, lies closer to the true parameter.
This is always about probabilities:
It is possible to draw an estimate from a large-variance distribution that happens to fall close to β.
It is also possible to draw an estimate from a small-variance distribution that falls far from β.
However, it is more likely to get a closer estimate when variance is smaller.
Thus, when possible, we always prefer an estimator with a smaller variance.
II. Variance of Least Squares Estimators
A. General Setup
We want to study what affects the variance of the least-squares estimators and
We already know the least-squares estimators are unbiased, meaning
for each .From this foundation, we can derive expressions for the variances of these estimators and identify the key factors that influence them.
III. Determinants of the Variance of Least Squares Estimators
A. Effect of Error Variance
Both variances of and include the term , the variance of the random error.
As increases, the variance of the estimator also increases.
Explanation
A larger σ² means more noise and uncertainty in the model, making it harder to estimate the true relationship precisely.
Example:
Suppose X = income and Y = spending on food.
The true line shows that for every $100 of income, households spend $15 on food.
If σ² is large, households with similar income show very different spending levels—some on diets, some shopping more, some hosting guests.
The wide dispersion makes it difficult to estimate the line accurately, increasing estimator variance.
If σ² is small, points cluster tightly around the line, making estimation more precise and reducing variance.
B. Effect of Variance of X (Var(X))
Both formulas for the variances of and contain Var(X).
As Var(X) increases, the variance of the estimator decreases.
Intuition
When X values vary a lot (large range of X), it becomes easier to fit the line precisely.
When X values are concentrated (small range), it becomes harder to estimate the slope accurately.
Graphical Illustration
Figure A: Small range of X → small Var(X) → hard to see the true relationship.
Many possible lines could fit the data (steep or flat).
Hence, estimator variance is high.
Figure B: Large range of X → large Var(X) → easier to see the true relationship.
Line is more clearly defined.
Estimator variance is lower.
C. Effect of (Sum of Squared X Values)
This affects only the variance of (the intercept), not (the slope).
is large when the sample includes many high-income observations;
it is small when most observations have low income.
Reasoning
estimates the intercept, i.e., predicted spending when
It is easier to estimate this intercept precisely if we have many low-income households.
If we mainly observe high-income households, estimating the intercept becomes difficult.
Therefore:
Many high-income values → large → variance of is high.
Many low-income values → smaller → variance of is lower.
Graphical Explanation
If there are many data points near income = 0, the intercept (spending at zero income) can be estimated accurately.
If all data points are far from income = 0, estimating that intercept precisely becomes difficult, raising variance.
Example: In the model, the intercept was about $83 for households with zero income.
This might be inaccurate because we have few such households in the data.
The imprecision could explain why the estimate seems implausible (e.g., “households with no income spending money on food”).
D. Effect of Sample Size ( N )
As N increases, both variances and decrease.
The reasoning:
Larger N means the denominator in variance expressions grows (because we sum over N), reducing overall variance.
More data → more precision.
IV. Consistency of Least Squares Estimators
A. Definition of Consistency
An estimator is consistent if, as N → ∞, the estimates produced converge to the true parameter value.
Formally:
As sample size increases, the sampling distribution of the estimator becomes more concentrated around the true parameter.
Graphically:
As N increases, the estimator’s distribution tightens around the mean.
There is a higher probability that the estimate will be closer to the true parameter.
This property is also called convergence in probability.
B. Relation between Consistency and Variance
As sample size increases:
Variance of the estimator decreases, meaning estimates cluster more tightly around β.
Probability of being close to the true parameter increases.
A “bad draw” remains possible, but increasingly unlikely.
V. Unbiasedness vs. Consistency
A. Key Difference
These two concepts are very different.
Concept | What It Says | Focus |
|---|---|---|
Unbiasedness | The expected value of the estimator equals the true parameter. | Center of the sampling distribution. |
Consistency | As N increases, the estimator’s distribution becomes concentrated around the true parameter. | Shape of the sampling distribution as sample size grows. |
B. Interpretation
Unbiasedness:
Imagine drawing many samples and computing an estimate for each.
If you average all these estimates, you will get the true parameter on average.
This does not mean any one sample gives a good estimate—only that the mean of all possible estimates equals β.
Consistency:
Focuses on what happens as you increase your sample size.
It says that with a larger N, there is a higher probability that the estimate from your own sample will be close to β.
VI. Example — Unbiased but Not Consistent
A. Setup
Suppose ( X ) is a random variable distributed as
We draw a sample and want to estimate .
B. Candidate Estimator
Typical estimator: the sample mean,
But now define a different estimator:
(i.e., use only the first observation).
C. Properties
Unbiasedness:
so the estimator is unbiased.
Over many samples, if we always take the first draw and average all those X₁’s, we recover μ.
Inconsistency:
The variance of X₁ is , which does not shrink as N increases.
Increasing sample size does not help, because we always use only one observation.
Hence, this estimator is not consistent: it does not converge to μ as N → ∞.
VII. Summary
Efficiency: Among unbiased estimators, choose the one with the smaller variance.
Variance determinants:
Larger → larger estimator variance.
Larger → smaller estimator variance.
Larger → affects only b₁ (intercept) variance.
Larger N → smaller variance for both b₁ and b₂.
Consistency: As N increases, estimator variance decreases and estimates converge to the true parameter.
Unbiasedness vs Consistency: Unbiasedness concerns expected values; consistency concerns sample-size behavior.
Example: is unbiased but not consistent.
Lecture 2 (Part 5): Gauss-Markov Theorem and Variance of the Least Squares Estimators
I. The Gauss-Markov Theorem
A. Background and Purpose
We already know that the least squares estimators are unbiased and consistent.
The Gauss-Markov theorem gives an additional and very important property of these estimators.
Although the proof of the theorem is mathematically complex and not covered here, its result is essential to understand why the least squares estimators are so widely used in econometrics.
B. Statement of the Theorem
The Gauss-Markov theorem applies under Assumptions 2–4 of the classical linear regression model — the assumptions that deal with the behavior of the random error term :
The expected value of the random error is zero:
The variance of the random error is constant across all observations (homoskedasticity):
The covariance between any two random errors is zero (no autocorrelation):
Under these three assumptions, the theorem states that the least squares estimators and have the smallest possible variance among all estimators that are both linear and unbiased estimators of the true parameters and Because of this property, these estimators are called the Best Linear Unbiased Estimators, or BLUE.
II. Understanding What “Best” Means
A. The Meaning of “Best”
The word “best” in this context does not mean perfect or always correct.
It means that within the specific family of estimators that are linear and unbiased, the least squares estimators have the smallest variance.
Having the smallest variance means they are most efficient — they fluctuate the least around the true parameter values in repeated samples.
So, if we were to repeatedly draw samples and estimate the parameters each time using least squares, the distribution of those estimates would be more tightly centered around the true parameter than any other linear and unbiased estimator.
B. Why This Matters
This is an important reason why the least squares estimator has become the dominant method in econometrics.
If one insists on using estimators that are linear and unbiased — and this is typically the goal when working with data — then the problem of choosing an estimator is already solved.
There is no need to search for another method, because no other linear and unbiased estimator will yield lower variance.
C. Summary of the Key Idea
Within the class of linear and unbiased estimators, least squares estimators are the most precise estimators possible.
This is what gives them their “best” property in the Gauss-Markov sense.
III. Limitations of the Gauss-Markov Theorem
A. Best Only Among Linear and Unbiased Estimators
The Gauss-Markov theorem compares the least squares estimators only to other estimators that are both linear and unbiased.
It does not claim that they are best among all possible estimators.
If for some reason an estimator that is nonlinear or biased is used, it is theoretically possible that such an estimator could have a smaller variance than the least squares estimators.
Thus, the “best” property of the least squares estimator is conditional — it applies only within the class of estimators that are linear and unbiased.
B. Dependence on Assumptions 2–4
For the Gauss-Markov theorem to hold, all assumptions about the random error (Assumptions 2–4) must hold true.
If any of these assumptions are violated — for example, if the variance of the error is not constant or if there is correlation among the error terms — then the least squares estimators will no longer be BLUE.
This point becomes crucial later in the course when these assumptions are relaxed.
Once those assumptions no longer hold, the least squares estimators lose their “best” property and can become inefficient or even biased.
C. Independence from Other Assumptions
It is also important to recognize what the Gauss-Markov theorem does not depend on.
It does not rely on:
The assumption of normality of the errors (Assumption 5), or
The assumption that is a random variable (Assumption 1).
The theorem depends only on the assumptions about the random error (2–4).
D. Estimator vs. Estimate
The Gauss-Markov theorem applies to the estimator, which is the formula, not to the estimate, which is the numerical value obtained from a specific sample.
An individual sample might produce an estimate that is far from the true parameter, but the theorem concerns the general behavior of the estimator across repeated samples.
It means that the least squares estimator, as a mathematical rule, has desirable properties — specifically, it tends to give estimates closer to the true parameter than other linear and unbiased estimators.
IV. The Central Limit Theorem (CLT) and Normality
A. Connection Between Least Squares and the CLT
The Central Limit Theorem provides a connection between large samples and normality.
It says that if Assumptions 2–4 hold and the sample size is sufficiently large, the sampling distribution of the least squares estimators can be approximated by a normal distribution.
In other words, even if the random errors themselves are not normally distributed, the estimators and will be approximately normal when the sample is large enough.
B. Practical Importance
In modern applications, we usually work with large samples — typically 100 or several hundred observations.
Therefore, the normal approximation is very good.
For this reason, assuming normality of the estimators is often treated as a practical simplification that does not significantly affect results.
C. Distribution of the Estimators
Under these conditions, the least squares estimators can be described as normally distributed:
This means that each estimator has a mean equal to its true parameter value and a variance equal to the theoretical variance we have already derived.
V. Estimating the Variance of the Estimators
A. Goal
So far, we have calculated the point estimates and for the coefficients.
Now we want to estimate the variance of these estimators and, from that, obtain the standard errors.
B. The Unknown Parameter
The formulas for the variances of and include a parameter which represents the variance of the random error.
However, this quantity is unknown — it is a property of the population and not directly observable in our sample.
Therefore, before we can calculate the variances of and , we must first estimate
C. Using Residuals as a Substitute
We cannot observe the true random errors, but we can calculate the residuals from our sample regression.
Residuals are our estimates of the unobservable errors.
To estimate the variance of the random errors, we take the following intuitive steps:
Square each residual.
Sum all the squared residuals.
Divide by the sample size to get their average.
This gives an intuitive estimator for the variance of the random error.
D. Problem with the Naive Estimator
Although this intuitive estimator is consistent, meaning it converges to the true value as the sample grows, it is still biased.
That is, on average, it underestimates the true variance.
VI. The Unbiased Estimator for the Variance of the Random Error
A. Correcting for Bias
The bias can be corrected easily by adjusting the denominator.
Instead of dividing by (n), we divide by
Here:
(n) is the number of observations.
(2) corresponds to the number of parameters estimated in the model (the intercept and the slope).
The term represents the degrees of freedom of the model.
B. Generalization to Multiple Regression
In a multiple regression model with (k) parameters, the formula becomes:
C. Interpretation
This corrected estimator is both consistent and unbiased, making it a reliable estimate of the true variance of the random error.
Once we have this estimate, we can plug it into the formulas for the variances of and to compute the estimated variances and standard errors.
The standard error is simply the square root of the estimated variance and gives us a measure of how much the estimator fluctuates around the true parameter value.
VII. Understanding Regression Output Conceptually
When looking at regression results, two quantities are essential:
The coefficient (point estimate) — the estimated value of the parameter (e.g., or .
The standard error — an estimate of the variability of that coefficient, calculated using the formulas above.
The standard error provides the basis for inference — it indicates how precisely the coefficient has been estimated.
A smaller standard error means a more precise estimate.
VIII. Summary of Key Concepts from Lecture 2
A. Model Assumptions
The classical linear regression model relies on assumptions about the random error term.
These assumptions (expected error = 0, constant variance, no correlation among errors) are critical for the Gauss-Markov theorem to hold.
Though these assumptions may seem simple or “naive,” they are not always true in practice. Later in the course, these will be relaxed.
B. Properties of Least Squares Estimators
Unbiasedness — On average, the estimators equal the true parameters.
Consistency — As the sample size increases, estimates get closer to the true parameters.
Efficiency (BLUE property) — Among all linear and unbiased estimators, least squares estimators have the smallest variance.
C. Variance and Standard Error Estimation
The variance of the random error must be estimated using residuals.
To make the estimate unbiased, divide the sum of squared residuals by
Substitute this estimated variance into the formulas for the variances of and , then take square roots to obtain their standard errors.
D. Why Standard Errors Matter
Point estimates alone are not enough for meaningful interpretation.
We must also consider the standard errors to understand the distribution and precision of the estimators.
These two quantities together — the coefficient estimate and its standard error — allow us to make inferences about the true parameters in the population.
E. Transition to Next Topic
The next step will be to use these point estimates and their standard errors to make statistical inferences about the true parameters — for example, testing hypotheses and constructing confidence intervals to learn about the real relationship between variables such as income and food expenditure.
Lecture 3 (Part 1): Confidence Intervals
I. The Motivation for Confidence Intervals
A. Point Estimates and Their Limitations
By now, we know how to obtain the point estimate which is our estimate of the unknown population parameter
However, a single point estimate alone is not very informative. The estimator itself is a random variable, and the point estimate we obtain from our particular sample is just one of many possible realizations of that random variable.
It is possible to be lucky and have a point estimate close to the true value of or to be unlucky and have one far away from it. Since the true parameter is unknown, we can never be certain whether our estimate is close to it or not.
Despite this uncertainty, we can still learn valuable information about using two statistical techniques:
Confidence intervals
Hypothesis testing
II. Conceptual Idea of the Confidence Interval
A. Illustration Using Multiple Samples
Different samples yield different point estimates.
For example:
Sample 1: Increasing income by $100 increases mean food spending by $6.48.
Sample 6: Increasing income by $100 increases mean food spending by $13.55.
From one sample alone, we cannot conclude much about the true effect of income on spending.
However, if we observe estimates from 10 or even 100 samples, we start to see a pattern. It becomes unlikely that equals 25, for instance. Instead, we might reason that the true lies roughly between 5 and 15.
This reasoning builds a range of likely values for the true parameter — the essence of the confidence interval.
III. Building a Confidence Interval from One Sample
A. The Challenge
In practice, we usually only have one sample. The task is to construct a range (confidence interval) for the unknown using only information from that single sample.
B. The Solution: Using the Standard Error
The sample provides not only a point estimate but also an estimate of the standard error of the estimator.
The standard error measures how much the estimator varies around its mean — and the mean, in theory, equals the true parameter
Using both the point estimate and the estimated standard error, we can construct the confidence interval.
IV. Mathematical Derivation of the Confidence Interval
A. Starting Point: The OLS Estimator
The least squares estimator is unbiased with mean and known variance. However, it is not yet standardized.
B. Standardizing the Estimator
To standardize, we subtract the mean and divide by the standard error:
This statistic ( Z ) follows a standard normal distribution with mean 0 and variance 1.
C. Problem: Unknown Population Variance
We do not know the true variance Instead, we estimate it using the sample residuals, denoted
Replacing with introduces sampling noise, so the resulting statistic does not follow a standard normal distribution.
D. Solution: The t-Statistic
The statistic now follows a t-distribution with degrees of freedom (since two parameters, and are estimated).
V. Properties of the t-Distribution
A. Shape and Characteristics
The t-distribution is bell-shaped and centered at zero.
It resembles the standard normal distribution but has thicker tails and a larger variance.
The difference arises because we use an estimated variance rather than the true which adds noise and increases uncertainty.
B. Degrees of Freedom and Convergence
The shape of the t-distribution depends on the degrees of freedom (df). As df increases — that is, as the sample size increases — the t-distribution approaches the standard normal distribution.
For instance, with 40 observations and 2 estimated parameters, we have 38 degrees of freedom.
At large (roughly the t and z distributions are nearly identical.
VI. Reading the t-Table and Finding Critical Values
A. Example
For 38 degrees of freedom and a 95% confidence level
The two tails together contain 5% of the probability, so each tail has 2.5%.
The t-critical value for 97.5% (the upper tail cutoff) is
This means there is a 97.5% probability that a t-distributed variable will be less than 2.024 and a 2.5% probability that it exceeds it.
VII. Constructing the Confidence Interval
A. Example
This gives the lower and upper bounds for
B. Example: Food Expenditure Model
Given:
Then:
Interpretation:
With 95% confidence, the true parameter lies between 5.97 and 14.45.
In words: increasing income by $100 increases food spending by between $5.97 and $14.45 per week, with a 95% probability.
VIII. Correct Interpretation of a Confidence Interval
A. What It Does NOT Mean
It does not mean the true parameter definitely lies within the interval.
We will never know for sure whether our particular interval contains the true
B. What It Actually Means
If we repeatedly drew samples from the population and constructed confidence intervals using this same procedure, 95% of those intervals would contain the true
Thus, the method itself is correct 95% of the time.
C. Intuition
Even though we cannot know if our single interval contains the true parameter, it would be quite unlucky if it did not—only 5% of properly constructed intervals miss it.
IX. Confidence Intervals in Regression Output
Regression software (like Stata) automatically provides the 95% confidence intervals for each parameter.
Example:
95% CI for (5.97, 14.45)
95% CI for (-4, 171)
The intercept’s interval includes zero, which means we cannot rule out that households with zero income may spend nothing on food. The interval also allows for negative values, which are not meaningful economically but remind us to apply common sense in interpretation.
X. Adjusting the Confidence Level
A. Changing α
If α = 1% (99% confidence interval), the critical value becomes larger
The point estimate and standard error stay the same, but the interval widens:
The result is a wider range, representing greater confidence but less precision — the classic “no free lunch” tradeoff.
XI. Confidence Interval for a Linear Combination of Parameters
A. Motivation
Sometimes we need a confidence interval not for a single coefficient but for a predicted mean value — a linear combination of parameters.
Example:
Predicted weekly food spending for a household with $2000 income: $287.61.
This is only a point estimate. We want a 95% confidence interval for the mean expected spending.
B. Computation
We use the estimated coefficients and their covariance matrix.
The variance of this linear combination is computed as:
The standard error is the square root of this variance.
C. Result
Substituting into the confidence interval formula:
This yields:
95%
Interpretation:
With 95% confidence, the average weekly food expenditure for households earning $2000 per week lies between $258.91 and $316.31.
Lecture 3 (Part 2): Hypothesis Testing
I. What We’re Doing and Why
A. The Aim of Hypothesis Testing
The aim is to say something about the true parameter Confidence intervals give a likely range for
B. Connection to Population Conjectures
Hypothesis testing tackles the same goal by comparing a conjecture about the population with what the sample shows.
C. The Null and Alternative Hypotheses
Begin with a belief about the population (e.g., “for every $100 increase in income, mean food spending increases by $15”). Call this belief the null hypothesis. Keep it unless the sample provides convincing evidence against it; then reject it and adopt an alternative hypothesis.
II. How We State Hypotheses
A. The Null Hypothesis
Null hypothesis:
: population parameter (what is happening in the real world).
: constant appropriate to the specific regression context.
B. The Alternative Hypothesis
Alternative hypothesis — three logical forms:
Right-tailed: (it is actually more tha C).
Left-tailed: (it is actually less than C).
Two-tailed: (it is not actually C, could be more or less).
III. The Test Statistic and Its Distribution
A. Definition of the Test Statistic
B. Distribution Under the Null Hypothesis
If is true, :
C. When the Null Is False
If is false, this ratio does not follow
IV. Rejection Regions and Significance Level
A. Definition of the Rejection Region
The rejection region collects values that are very unlikely when is true.
B. Shape of the Rejection Region
Its shape depends on : right tail, left tail, or two tails.
C. Choosing the Significance Level
Its threshold depends on the chosen significance level Typical choices: ; (\alpha=1%) (more stringent); (less common).
D. Example of Rejection Thresholds
Example intuition (with (df=38)): large values like 2.7, 3.2, 4.7 are possible but very unlikely if is true.
V. Language and Errors
A. Proper Hypothesis Testing Terminology
Say “reject or “do not reject — never “accept
B. Type I Error
Type I error: reject although it is true. Probability = .
C. Type II Error
Type II error: do not reject although it is false. In practice here, it is not computed because there are infinitely many possible true alternatives.
D. The Trade-Off Between Errors
Trade-off: lowering makes rejection harder.
VI. Five-Step Procedure (Use This Every Time)
State H and
Specify the test statistic and its distribution if is true.
Choose and write down the rejection region.
Compute the sample (t) from your regression output.
Make the decision (reject / do not reject) and state the conclusion.
VII. Example 1: Does Food Expenditure Increase with Income? (Right-Tail)
Goal: Show that the effect of income on food spending is positive.
Step 1 — Hypotheses
(no relationship)
(positive effect)
Step 2 — Statistic and Distribution (under )
(with 40 observations, so
Step 3 — Significance Level and Rejection Region
Right-tail critical value (df 38):
Reject for
Step 4 — Compute Sample (t)
From the regression:
Step 5 — Decision and Conclusion
reject
Conclusion: there is a statistically significant positive relationship between income and food expenditure at 5% (also at 1% and far smaller given how large 4.88 is).
VIII. Example 2: Is Mean Spending < $15 per Additional $100? (Left-Tail)
Goal: Show that the effect is less than $15 per $100.
Step 1 — Hypotheses
(or
Step 2 — Statistic and Distribution (under
Step 3 — Significance Level and Rejection Region
Left-tail critical value (df 38): (-1.686).
Reject for
Step 4 — Compute Sample (t)
From the regression: (t=-2.29).
Step 5 — Decision and Conclusion (5%)
reject at 5%.
Conclusion: households spend less than $15 per $100 of income at the 5% level.
If :
Critical value
(-2.29) does not cross it ⇒ Do not reject at 1%.
This does not mean accept ; it means there is not enough evidence at 1%.
IX. Example 3: Is There Any Relationship? (Two-Tail “Test of Significance”)
Goal: Show that income affects spending in either direction.
Step 1 — Hypotheses
Step 2 — Statistic and Distribution (under
Step 3 — Significance Level and Rejection Region
Two-tail: split (2.5%) in each tail.
Critical values (df 38):
Reject for or
Step 4 — Compute Sample (t)
From the regression, the same (t) we computed previously is well beyond the thresholds.
Step 5 — Decision and Conclusion
Reject . There is a statistically significant relationship between income and food expenditure at 5%.
Link to Confidence Intervals
If the 5% confidence interval for excludes 0 (e.g., lower bound ), then you would also reject at 5%.
If the 5% interval for includes 0, then you would not reject at 5%.
Same information, different format.
X. Testing Linear Combinations (Same Logic)
A. Replace the Parameter with a Linear Combination
Replace the single parameter by a linear combination
B. Test Statistic for a Linear Combination
Under , use
C. Tail Direction and Interpretation
Alternatives can be right-tailed, left-tailed, or two-tailed; rejection regions and interpretations are exactly as before.
XI. Linear Combination Example (Weekly Income = $2000)
Question: Is mean food spending for households with weekly income of $2000 greater than $250 per week?
Step 1 — Hypotheses
Step 2 — Statistic and Distribution (under ( ))
Linear combination’s point estimate (computed earlier):
Use the linear combination’s standard error (computed earlier).
Test statistic:
Step 3 — Significance Level and Rejection Region
Right-tail critical value (df 38):
Reject for
Step 4 — Compute Sample (t)
Step 5 — Decision and Conclusion
reject
Conclusion: expected spending at $2000 weekly income is greater than $250 at the 5% level.
XII. How This Appears in Software Output (What to Look For)
A. Post-Estimation Testing
Always run the regression first (hypothesis tests and CIs are post-estimation).
B. The Reported (t)-Statistic
For , the reported is ; e.g., (
C. Comparing with Critical Values
To compare with the correct critical value, use a table or a function that returns the critical value for your , tail(s), and (df).
D. Tail Matching and Critical Values
For left- or right-tailed tests, match the tail to the alternative; for two-tailed tests at , use with
E. Linear Combinations in Output
For linear combinations, compute the linear combination’s point estimate and standard error, form the same (t), and compare to the appropriate critical value.
XIII. Bottom Line
A. Aligning Hypothesis and Goal
Formulate to match the conclusion you want (greater than, less than, or not equal).
B. Choosing the Significance Level
Choose and the appropriate rejection region.
C. Reporting the Results
Report the test type, (df), test statistic, critical value(s), and your decision: reject or do not reject
D. Connecting to Confidence Intervals
Connect to confidence intervals when helpful (same information, different presentation).
E. Remembering the Trade-Off
Remember the trade-off: smaller makes rejection harder.
Lecture 3 (Part 3): The P-Value
I. Introduction to the P-Value
The P-value is a central concept in hypothesis testing.
When reporting the result of a statistical hypothesis test, it is standard practice to include the P-value, or probability value, of the test.
The P-value tells us the probability of committing a Type I error, given the T-statistic obtained from the sample and the form of the alternative hypothesis.
In other words, it answers the question: If the null hypothesis is true, what is the probability of obtaining a test statistic as extreme as, or more extreme than, the one observed?
II. Understanding What the P-Value Represents
A. Setting Up the Test
Suppose we are testing a hypothesis and have calculated a test statistic from our data, denoted as T.
Under the null hypothesis, this statistic follows a t-distribution with degrees of freedom.
The P-value measures the probability, assuming the null hypothesis is true, of observing a T-statistic that lies in the rejection region—that is, in the direction specified by the alternative hypothesis.
B. Right-Tailed Test
If the alternative hypothesis is of the form:
then we are performing a right-tailed test.
The P-value is the probability that the test statistic falls to the right of the observed T value.
Graphically, it corresponds to the area under the t-distribution curve to the right of T.
This area represents how likely it is, under the null hypothesis, to obtain such a large or larger T-statistic purely by chance.
C. Left-Tailed Test
If the alternative hypothesis is:
then we are performing a left-tailed test.
Here, the P-value is the probability that the test statistic falls to the left of the observed T value.
It is the area under the t-distribution curve to the left of T.
This tells us how likely it is, assuming the null is true, to get a T-statistic this small or smaller.
D. Two-Tailed Test
For a two-tailed test, the alternative hypothesis is:
This means we are testing for deviations in either direction from the null hypothesis.
In this case, the P-value is the sum of the two tail probabilities—
the probability of obtaining a value greater than (the positive tail) plus the probability of obtaining a value less than (the negative tail):
Graphically, this corresponds to the total area in both tails beyond the magnitude of the observed statistic.
III. Why the P-Value is Useful
The P-value provides an alternative way to conduct hypothesis testing without referring to the critical value directly.
It allows us to decide whether to reject or not reject the null hypothesis by comparing the P-value to the significance level (α) of the test.
A. Rejection Rule Based on the P-Value
The decision rule is straightforward:
If , reject the null hypothesis (H₀).
If , fail to reject the null hypothesis (H₀).
This rule gives us the same outcome as the traditional comparison between the test statistic and the critical value, but it is often more intuitive, since the P-value represents the exact probability of observing the result under the null.
B. Relation to the Critical Value Method
Earlier, we compared the T-statistic with the critical value from the t-distribution.
Now, instead of transforming α into a critical value, we can transform the T-statistic into a P-value.
Both approaches are equivalent:
Comparing T-statistic ↔ critical value
Comparing P-value ↔ significance level α
In both cases, the same conclusion will be reached.
IV. Example 1 – Left-Tailed Test
A. Hypothesis Setup
We are testing whether mean spending on food is less than $15 for every additional $100 of income.
This is a left-tailed test, since the alternative hypothesis states “less than.”
B. Test Statistic
From the sample data, we computed:
Under the null hypothesis, the T-statistic follows a t-distribution with degrees of freedom.
C. Meaning of the P-Value
The P-value is the probability, assuming the null hypothesis is true, of obtaining a T-statistic less than –2.29.
That is, the probability of being in the left tail of the distribution beyond
This P-value equals 0.0139, or 1.39%.
D. Decision and Interpretation
We compare the P-value to our chosen significance levels:
At α = 5%:
P = 1.39% < 5% → Reject H₀.
There is sufficient evidence to conclude that the spending is less than $15.At α = 1%:
→ Fail to reject H₀.
There is not enough evidence at this stricter level to reject the null.
E. Visual Explanation
Graphically:
The P-value region is the shaded left tail of the t-distribution beyond
The critical value for α = 5% is so our T-statistic falls in the rejection region.
The critical value for α = 1% is so our T-statistic does not fall in that rejection region.
Thus, the P-value method and the critical value method give the same result:
Reject H₀ at 5%, do not reject at 1%.
V. Example 2 – Right-Tailed Test
A. Hypothesis Setup
This is a right-tailed test, because the alternative hypothesis states “greater than.”
B. Test Statistic
The computed T-statistic is:
Under the null hypothesis, this follows a t-distribution with degrees of freedom.
C. Meaning of the P-Value
The P-value is the probability, assuming the null hypothesis is true, of obtaining a T-statistic greater than or equal to 4.88.
This probability is extremely small, essentially close to zero.
D. Decision and Interpretation
Because the P-value is so low, we can reject the null hypothesis for:
α = 5%
α = 1%
and even for much smaller α values.
Therefore, the evidence is overwhelming that β₂ is greater than zero.
This shows that the relationship between income and spending is statistically significant.
VI. Example 3 – Two-Tailed Test (Significance of a Parameter)
A. Hypothesis Setup
This is a two-tailed test, also called the test of the significance of a parameter.
It checks whether β₂ is significantly different from zero in either direction.
B. Test Statistic and P-Value
The same T-statistic applies here:
For a two-tailed test, the P-value represents the probability of obtaining a value greater than 4.88 plus the probability of obtaining a value less than –4.88.
This total P-value is extremely small, practically zero.
C. Decision and Interpretation
Since the P-value is far below any typical α (5%, 1%, etc.), we can reject the null hypothesis for all these significance levels.
This means the parameter β₂ is highly statistically significant, and we can conclude that it differs from zero.
D. Reading P-Values from Regression Output
In regression output, each coefficient’s P-value corresponds to this two-tailed significance test.
The interpretation is:
If P ≤ α, the coefficient is statistically significant.
If P > α, the coefficient is not significant.
For example:
For β₂ (income): P < 0.01 → reject H₀ for any α → significant.
For β₁ (intercept): P = 0.062 → greater than both 0.05 and 0.01 → not significant.
Interpretation:
The intercept represents expected spending for individuals with zero income.
Since we cannot reject H₀ that β₁ = 0, we conclude that for people with no income, the expected spending on food is 0.
E. Clarification on Two-Tailed P-Values
The P-values displayed in regression outputs are already for two-tailed tests.
They represent the total probability of observing a value greater than |T| or less than −|T|.
Therefore, when using these reported P-values, no additional multiplication by 2 is needed.
A common mistake is doubling the reported value unnecessarily—this should be avoided.
VII. Summary of Key Points
The P-value measures the probability of obtaining a test statistic as extreme as the observed one, assuming the null hypothesis is true.
The P-value depends on the direction of the alternative hypothesis:
Right-tailed: P(T > t)
Left-tailed: P(T < t)
Two-tailed: P(T > |t|) + P(T < –|t|)
The rejection rule:
Reject H₀ if P ≤ α.
Fail to reject H₀ if P > α.
The P-value method is equivalent to the critical value method but gives a direct probability interpretation.
Regression outputs provide two-tailed P-values for the test of significance of each coefficient.
When the P-value is very low, the result is highly significant, meaning strong evidence against the null hypothesis.
VIII. Transition to Next Topic
This concludes the discussion on the P-value and the complete analysis of the simple linear regression model.
The next step is to extend these ideas to the multiple linear regression model, which accounts for multiple explanatory variables and reflects more realistic situations.
Lecture 4 (Part 1): Multiple Linear Regression
I. Introduction to Multiple Linear Regression
A simple linear regression model is limited because it includes only one explanatory variable. In reality, most outcomes are influenced by several factors. For example, it is not reasonable to assume that only income explains spending on food. Other crucial factors—such as the number of household members and location (city or village)—also affect spending. Rural households may have access to cheaper products, influencing overall spending patterns.
To analyze these more realistic scenarios, the model must be extended to include multiple explanatory variables. The extension is conceptually simple.
II. Structure of the Multiple Linear Regression Model
The general form of the model includes:
: intercept
slope coefficients for each explanatory variable (X_k)
random error term
Each additional explanatory variable receives its own (\beta) coefficient.
In this framework, a single parameter (\beta_k) measures the effect of a change in (X_k) on the expected value of (Y), holding all other variables constant.
This “holding constant” is formalized mathematically by taking the partial derivative of (E(Y|X)) with respect to (X_k):
This isolates the effect of (X_k) while keeping other explanatory variables fixed.
III. Example Data: Sales of “Big Andy’s Bean Burger”
A dataset is used to illustrate the model.
Dependent variable (Y): Monthly sales (in thousands of dollars)
Observations: 75 branches
Explanatory variables:
Price (in dollars per burger)
Advertising expenditure (in thousands of dollars)
Data Characteristics
Sales range: $62,000 to $91,000
Price range: $4.80 to $6.50
Advertising expenditure range: $500 to $3,100
This indicates a small business, not a large company like Nike or Victoria’s Secret.
Before running any regression, it is important to understand the dataset—its scale, units, and variation—since these affect statistical inference.
IV. Model Interpretation
(\beta_2): Change in monthly sales (thousands of dollars) when price increases by 1,000, holding price constant.
Geometrically, instead of a regression line, the model estimates a regression plane:
The intercept is where the plane crosses the Y-axis.
The price slope is one directional vector.
The advertising slope is another directional vector.
Each added explanatory variable adds one dimension to this geometric representation.
V. Assumptions for Multiple Linear Regression
The key assumptions from simple linear regression still apply.
A. Assumption 1: Expected Error = 0
[
E(u_i) = 0
]
This holds if the model is correctly specified.
If an important explanatory variable is omitted, its effect is absorbed into the error term, making (E(u_i) \neq 0).
B. Assumption 2: Constant Variance of Errors (Homoscedasticity)
[
Var(u_i) = \sigma^2
]
Often, this does not hold—variance may increase with (X).
For now, this assumption is maintained. Later, heteroskedasticity and its implications will be examined.
C. Assumption 3: Zero Covariance Between Errors
[
Cov(u_i, u_j) = 0 \quad \text{for all } i \neq j
]
This means that one observation’s error does not provide information about another’s.
This assumption will later be relaxed when studying serial correlation.
D. Assumption 4: Normality (for Small Samples)
[
u_i \sim N(0, \sigma^2)
]
For large samples, this follows from the Central Limit Theorem; for smaller ones, it must be explicitly assumed.
E. New Assumption for Multiple Regression: No Perfect Multicollinearity
No explanatory variable is an exact linear function of any other:
[
X_k \neq a_1 X_1 + a_2 X_2 + \dots
]
Violations (examples):
(X_3 = 2 X_2) → violates assumption
(X_3 = 0.1X_2 + 0.7X_4) → violates assumption
(X_3 = X_2^2) → does NOT violate (nonlinear relationship)
If exact multicollinearity exists, least squares estimation fails, and software will return an error message.
This is called exact collinearity, to be discussed further in a later lecture.
VI. Estimating the Model
The least squares estimators minimize the sum of squared errors (SSE):
[
SSE = \sum (Y_i - \hat{Y}_i)^2
]
As before, we take derivatives with respect to all (\beta)s—(\beta_1, \beta_2, \beta_3)—to obtain their estimators.
A. Interpretation of the Estimators
(\hat{\beta}_1): Ensures the regression plane passes through the sample means of (Y), (X_2), and (X_3).
(\hat{\beta}_2, \hat{\beta}_3): Derived using deviations from means. They adjust for covariance among explanatory variables.
B. Covariance-Based Formula (Intuitive Explanation)
If (Cov(X_2, X_3) = 0), the model simplifies:
[
\hat{\beta}_2 = \frac{Cov(X_2, Y)}{Var(X_2)}
]
which is identical to the simple regression estimator.
When (X_2) and (X_3) are not correlated, including or excluding (X_3) produces the same estimate for (\beta_2).
However, when (X_2) and (X_3) are correlated, omitting one leads to biased estimates.
VII. Omitted Variable Bias
If (X_3) is omitted and it is correlated with (X_2), the estimate of (\beta_2) becomes biased.
This bias occurs because part of the variation in (Y) attributed to (X_2) is actually due to (X_3).
This issue is extremely important because:
In survey data, explanatory variables often correlate.
In experimental data, variables can be independently manipulated, so correlation is less problematic.
Examples:
Negative correlation:
During promotional campaigns, businesses may both reduce prices and increase advertising → price and advertising negatively correlated.Positive correlation:
Companies like Nike with large ad budgets charge higher prices → price and advertising positively correlated.
To obtain meaningful and unbiased estimates, the full model including all relevant explanatory variables must be used.
VIII. Understanding the Bias Mechanism
Suppose (X_2) and (X_3) are positively correlated.
When (X_2) increases:
(Y) increases directly (because of (X_2)’s positive effect).
(X_3) also tends to increase, causing (Y) to rise further through (X_3)’s effect.
If only (Cov(X_2, Y)) is used, all of this combined effect is wrongly attributed to (X_2).
The true effect of (X_2) alone requires subtracting the portion of variation in (Y) explained through (X_3).
This is what least squares estimation achieves—it isolates the unique effect of (X_2) by removing the shared influence through (X_3).
IX. Connection to Multicollinearity
The same formula reveals why perfect collinearity makes estimation impossible.
If (X_2) and (X_3) move exactly together (for example, when (X_3 = 2X_2)), there is no way to distinguish how much of (Y)’s variation comes from (X_2) versus (X_3).
Therefore, the model cannot separate their effects, and the least squares method fails.
X. Summary of Key Takeaways
Multiple linear regression allows analysis of more realistic relationships with several explanatory variables.
Coefficients measure partial effects, holding all other variables constant.
The geometric interpretation expands from a line to a plane.
All previous assumptions remain, with one new requirement: no perfect linear relationship among explanatory variables.
Omitted variable bias arises when a correlated regressor is left out, leading to incorrect inference.
Exact collinearity prevents estimation altogether.
Lecture 4 (Part 2): Interpreting Multiple Regression Results
I. Introduction
After understanding the intuition behind the sales model, we now focus on interpreting the actual regression estimates using two explanatory variables: price and advertising.
It is crucial to be able to interpret regression output correctly — both for the exam and in professional applications.
II. Interpreting the Coefficient of Price
A. Initial Observation
The coefficient on price is negative, which suggests that demand is price elastic.
Holding advertising constant, an increase in price by $1 leads to a fall in monthly revenue of $7,980.
The p-value is approximately 0, indicating that the coefficient is highly significant.
However, even though this interpretation is technically correct, it is not the best way to interpret the result — and would lose points on the exam.
B. Correct Interpretation
It is better to interpret the coefficient in a way that reflects the realistic variation in the data.
The correct phrasing would be:
An increase in price by $0.10 leads to a fall in monthly revenues of $719.90, holding advertising constant.
This is preferable because the observed price range in the data is between $4.90 and $6.50 — less than $2 total variation.
Predicting what would happen with a $1.00 change is beyond the realistic range of observed data.
C. Why Large Changes Are Problematic
If the estimated relationship were taken literally, a $1 decrease in price would imply an increase in sales of more than $7,900.
If the price were decreased from $5 to $0, the model would predict sales increasing by nearly $40,000, which is nonsensical.
This occurs because regression models describe relationships only within the range of observed data.
Extrapolating beyond that range (e.g., predicting sales at prices of $2 or $10) can lead to meaningless results since consumer behavior may differ drastically at extreme prices.
Therefore, with a sample price range of less than $2, it is unreasonable to interpret results for a $1 change.
A $0.10 change provides a much more meaningful interpretation.
III. Interpreting the Coefficient of Advertising
A. Observation
The advertising coefficient is positive and highly significant (p-value ≈ 0).
Thus, at α = 1% (or even lower), it is statistically significant.
Holding price constant, an increase in advertising expenditure of $1,000 leads to an increase in monthly revenues of $1,863.
B. Adjusted Interpretation
However, the observed advertising expenditures in the data range from $500 to $3,000.
Therefore, a more appropriate interpretation is:
An increase in advertising expenditure by $100 leads to an increase in monthly revenues of $183.60.
This expresses the relationship on a scale that aligns with the data range.
IV. Interpreting the Intercept
A. Direct Interpretation
The estimated intercept suggests that if both price and advertising expenditure were zero, sales revenues would be approximately $119,000.
Even though the coefficient is statistically significant (p-value ≈ 0) and its 95% confidence interval ranges from $106K to $131K, this result is not meaningful.
A price of $0 would clearly imply revenue of $0, not $119K.
Therefore, the intercept has no real-world interpretability.
B. Why We Still Include an Intercept
Even though the intercept is not interpretable, it is almost always included in the model because it improves the fit of the regression.
Without an intercept, the regression line is forced to pass through the origin, which restricts the slope and limits the model’s flexibility to fit the data where observations actually occur.
Including an intercept allows the line to begin at any point on the Y-axis, improving the model’s fit in the region where data exist — even if predictions near the origin are meaningless.
V. The Gauss-Markov Theorem
The Gauss-Markov theorem also applies to the multiple linear regression model.
If the classical assumptions hold, then the least squares estimators are:
Best Linear Unbiased Estimators (BLUE) — meaning:
They are linear in the dependent variable.
They are unbiased (their expected value equals the true parameter).
They have the lowest variance among all linear unbiased estimators.
All the results from simple regression extend naturally to multiple regression, except that the degrees of freedombecome:
[
n - k
]
where:
( n ) = number of observations
( k ) = number of parameters in the model
In simple regression, this was ( n - 2 ).
VI. Confidence Intervals in Multiple Regression
The formula for the confidence interval remains the same as before, except we now use t-critical values with n – k degrees of freedom.
For example, to construct a 95% confidence interval for the effect of advertising (( \beta_3 )):
( n = 75 )
( k = 3 )
Degrees of freedom = ( 75 - 3 = 72 )
( t_{0.025, 72} = 1.99 )
Given the point estimate and standard error from the regression output:
[
\text{CI}_{95%} = \hat{\beta_3} \pm 1.99 \times SE(\hat{\beta_3})
]
This yields a confidence interval such that:
Increasing advertising spending by $1,000 increases revenue by between $501 and $3,225 at α = 5%.
However, since the data range is limited, a more realistic interpretation is:
Increasing advertising spending by $100 increases revenue by between $50.10 and $322 at α = 5%.
Interpretation of Interval
The interval is relatively wide, implying uncertainty about profitability:
Additional advertising may barely cover its cost (if the effect is near $50).
Or it may be highly profitable (if the effect is near $322).
Thus, the true effect could mean breaking even or gaining more than three times the cost.
VII. Hypothesis Testing Example
A. Question
Suppose your boss asks whether increasing the advertising budget is a good idea.
We can test the hypothesis:
[
H_0: \beta_3 = 1 \quad \text{(no gain; break-even)} \
H_1: \beta_3 > 1 \quad \text{(positive gain; revenue exceeds cost)}
]
B. Test Statistic
Under ( H_0 ), replace ( \beta_3 ) with 1 in the formula.
The test statistic follows a t-distribution with 72 degrees of freedom.
[
t = \frac{\hat{\beta_3} - 1}{SE(\hat{\beta_3})}
]
If ( \alpha = 1% ), this is a right-tailed test.
The critical value is 2.38.
We reject ( H_0 ) if ( t > 2.38 ).
C. Result
The computed ( t = 1.263 ), which is much lower than 2.38.
The p-value for this statistic is 0.105 (10.5%), so we cannot reject the null, even at α = 10%.
Hence, we conclude that increasing advertising may not necessarily cover its additional cost.
D. Connection to Confidence Intervals
This result is consistent with the earlier confidence interval:
The lower bound allows for the possibility that revenue does not cover advertising cost.
If we cannot reject ( H_0 ) for α = 2.5% (right-tailed), we certainly cannot reject it for α = 1%.
Thus, hypothesis testing and confidence intervals provide equivalent information.
VIII. Predicting Expected Sales
A. Example: Linear Combination of Parameters
We can use the model to predict expected sales for any price and advertising level within the data range.
For example:
Price = $5.50
Advertising expenditure = $1,200
The point estimate of expected sales is ≈ 5.50 and advertising = $1,200, the expected monthly sales lie between $76,000 and 1,200 and spending increases by $100, expected sales increase by $186.
If the current level is $2,500 and spending increases by $100, expected sales still increase by $186.
Thus, the model implies that the return from advertising is independent of the current level of advertising.
III. Theoretical Critique: Diminishing Returns from Advertising
To evaluate whether this constant-return assumption makes sense, economic theory should guide the reasoning.
From theory and everyday experience, it is clear that a linear relationship between advertising and sales is unrealistic.
Consider an everyday example:
The first time you receive a flyer about a new vegan restaurant, it captures your attention and perhaps influences your behavior.
By the fifth flyer, the information feels redundant.
By the tenth flyer, it becomes annoying, and you are less likely to visit that restaurant at all.
This illustrates diminishing returns to advertising: at low levels, advertising strongly increases sales, but as advertising intensity grows, each additional dollar produces a smaller effect, and beyond a certain point, the effect can even turn negative when consumers become irritated.
Formally:
At $500 per month in advertising, an additional $100 might raise expected sales by $1,000.
At $1,200 per month, the same $100 increase may raise sales by only $300.
At $10,000 per month, another $100 might actually reduce sales.
Thus, from theory, the assumption of constant returns is not reasonable: the marginal effect of advertising depends on the current level of spending.
IV. Evidence from Data and the Need for Nonlinearity
When plotting sales (Y-axis) against advertising spending (X-axis), the data form a curved, inverted-U shape, not a straight line.
A simple fitted line does not capture this pattern. The linear model fails to represent the observed diminishing marginal returns.
However, fitting a curved line, such as a quadratic curve, captures the pattern much better:
At low advertising levels, sales increase steeply.
As advertising increases, the slope flattens.
Beyond a certain point, the slope becomes negative.
This motivates extending the model to include nonlinear terms.
V. Modeling Diminishing Returns: The Quadratic Model
To represent diminishing returns mathematically, a polynomial function is introduced.
The simplest extension adds a quadratic term—the square of advertising spending. The model becomes:
[
E(\text{Sales}) = \beta_0 + \beta_1 \text{Advertising} + \beta_2 (\text{Advertising})^2
]
Here:
(\beta_1) measures the linear (first-order) effect of advertising.
(\beta_2) measures the curvature of the relationship.
Although advertising and its square are correlated, this is not a violation of the model assumptions because the relationship between them is not perfectly linear.
VI. The Marginal Effect and Interpretation
To see how the model captures diminishing returns, we compute the marginal return from advertising, i.e., the change in expected sales for a small change in advertising:
This derivative shows that the return from advertising depends on the current level of advertising.
If (\beta_2 < 0), the marginal return decreases as advertising increases — consistent with diminishing returns.
Example (using estimated values):
At $1,500 in advertising, each additional dollar increases expected sales by $3.84.
At $1,750, each additional dollar increases expected sales by $2.46.
Around $2,000, the increase is about $1.00 — effectively break-even.
Beyond $2,500, additional spending may reduce sales by about $1.60.
This demonstrates that as advertising spending rises, returns shrink, reach zero, and eventually turn negative — matching economic theory and real-world intuition.
Graphically, the slope of the curve becomes flatter as advertising increases, crosses zero, and then declines.
VII. Evaluating Model Fit
When comparing the linear model and the quadratic model, the improvement is substantial.
Linear model: Explains 44.8% of the variation in sales (R² = 0.448).
Quadratic model: Explains about 51% (R² ≈ 0.51).
Thus, by simply allowing the model to be nonlinear, about 6% more variation in sales is explained.
Importantly, this improvement comes purely from changing the functional form—the variables are the same, but the relationship is now allowed to curve.
A 6% gain from the functional form alone is considered dramatic, and it confirms that the quadratic model provides a much better fit to the data.
VIII. Economic Meaning of Marginal Effects
In a nonlinear model, the marginal effect is not constant, so we cannot summarize the effect of advertising with a single average number.
For example:
A linear model might estimate that every additional $1 of advertising increases sales by $1.80.
But this average is misleading.
If a firm is currently spending $400, this average underestimates the benefit — they should spend more.
If a firm is spending $2,400, the same average overestimates the benefit — they should spend less.
Hence, decisions based on the average effect can lead to poor choices.
The appropriate decision depends on the current advertising level, because the marginal return changes with it.
IX. Increasing Returns: The Housing Example
The quadratic model can also capture increasing returns, where the marginal effect grows with the level of the variable.
Model
House price is explained by the size of the house (square footage):
[
If (\beta_2 > 0), the marginal effect increases with size—indicating increasing returns.
Economic reasoning
For every additional 100 square feet, a linear model assumes the same increase in price (e.g., $20,000).
However, this is unrealistic because:
Larger houses are usually located in more expensive neighborhoods.
They tend to use higher-quality materials and luxury features.
Therefore, the return per 100 square feet should rise as house size increases.
Data illustration
In a wealthy area like Boca Raton, Florida, most houses range between 1,000 and 3,000 square feet.
A linear model cannot accurately fit the relationship for large houses.
By including a quadratic term with a positive coefficient, the model captures that larger houses yield higher returns for each additional square foot.
X. Capturing Both Increasing and Decreasing Returns: The Cubic Model
Sometimes theory suggests that returns are first increasing and then decreasing (or the reverse).
In such cases, a quadratic model is not sufficient; we need a cubic polynomial.
Example: Total Cost and Marginal Cost Functions
A typical cost function for a firm relates total cost (C) to quantity produced (Q).
Economic theory states that the relationship between total cost and output is not linear:
When production is low, costs rise slowly — due to efficient use of resources.
As production grows, costs begin to rise more quickly — due to diminishing productivity.
Thus, total cost has an S-shaped (nonlinear) form.
Marginal Cost
The marginal cost (MC) is the change in total cost from producing one more unit:
[
MC(Q) = \frac{dC(Q)}{dQ}
]
The marginal cost curve is typically U-shaped:
Initially decreases (as specialization increases efficiency).
Then increases (as overcrowding, bureaucracy, or inefficiency set in).
Economic Intuition Example
Imagine a software company:
With only one worker, that person must code, handle accounting, do sales, and clean the office. Productivity is low, so the marginal cost is high.
When the firm hires specialized workers (a programmer, an accountant, a salesperson, a cleaner), productivity increases, and marginal cost falls.
As the firm grows large (like Microsoft or Yahoo), coordination problems appear; employees become less efficient, and marginal cost rises again.
This leads to the U-shaped marginal cost curve.
XI. Representing This Econometrically: The Cubic Polynomial
The total cost function can be expressed as a third-order polynomial:
[
C(Q) = \beta_0 + \beta_1 Q + \beta_2 Q^2 + \beta_3 Q^3
]
Differentiating with respect to (Q) gives the marginal cost function:
[
MC(Q) = \beta_1 + 2\beta_2 Q + 3\beta_3 Q^2
]
The marginal cost is therefore quadratic in (Q), which naturally allows for the U-shape predicted by theory.
Expected signs
For a U-shaped marginal cost, the coefficient on the cubic term in the total cost function should be positive((\beta_3 > 0)).
Typically, for the total cost function, we expect:
(\beta_2 > 0) (cost increases with Q²),
(\beta_3 < 0) (ensures the initial flattening then acceleration of total cost).
This combination yields a marginal cost curve that first decreases and then increases, perfectly matching the theoretical behavior.
XII. Summary of Key Insights
The linear regression framework can represent nonlinear economic relationships by including polynomial terms while staying linear in parameters.
A quadratic term captures diminishing or increasing returns depending on the sign of its coefficient.
The marginal effect is not constant—it depends on the level of the explanatory variable.
Diminishing returns occur when the quadratic coefficient is negative ((\beta_2 < 0)); increasing returns occur when it is positive ((\beta_2 > 0)).
Cubic terms allow modeling relationships that change direction (e.g., first increasing then decreasing).
Economic interpretation should always guide the model specification—statistical fit must reflect theoretical reasoning.
Would you like me to continue with Lecture 5 Part 2 (if you have it) so the notes continue seamlessly from this point?
Lecture 5 (Part 2): Log Transformations, Interactions, and Nonlinear Models
I. Big Picture: Three Ways to Handle Nonlinearity
A. Overview
Polynomials: capture diminishing/increasing returns.
Log transformations: improve fit with skewed data and enable elasticity interpretation.
Interaction variables: let the return to one variable depend on the level of another (often paired with a quadratic to capture diminishing returns).
B. Focus of This Lecture
Use log transformations (log-linear, linear-log, log-log).
Add interaction effects (Education × Experience).
Add a quadratic term in Experience to reflect diminishing returns.
Compare models and extract takeaways.
II. Log Transformations
A. Log-Linear Model (log on left, levels on right)
Form:
“Log” is natural log.
The regression is linear in parameters, but (Y=\exp(\cdot)) makes the relationship in levels nonlinear.
B. Why Logs Help
Monetary outcomes (wages, income, sales, expenditure) are often right-skewed.
The log compresses large values a lot and small values only a little.
Taking logs of a skewed income distribution produces something that looks approximately normal (“log-normal”).
With fewer extreme outliers, the model fits better and estimates are more reliable.
Using a transformed model is typically a good idea with very skewed data.
C. Coefficient Interpretation in Log-Linear
One unit increase in a regressor changes expected (Y) by approximately percent.
Example: +1 year of education ≈ +4% wages; +1 year of experience ≈ +1% wages.
Interpretation is about wages (levels), not about (\ln(\text{wage})).
III. Other Log-Based Forms
A. Linear-Log (levels on left, log regressor(s) on right)
Form: (Y=\beta_0+\beta_2 X_2+\beta_3\ln X_3+\beta_4 X_4+u).
Interpretation for (\ln X_3): 1% increase in (X_3) → approximately (0.01\times\beta_3) units of (Y).
Example: if (X_3) is income and (\beta_3=4), then 1% income increase → ≈ 0.04 more cars sold per week.
B. Log-Log (elasticities)
Form: (\ln Y=\beta_0+\beta_2\ln X_2+\beta_3\ln X_3+u).
(\beta) is an elasticity: 1% change in (X) → (\beta)% change in (Y).
Price elasticity is negative; e.g., (\beta_3=-2.0) means a 1% price decrease increases quantity demanded by ≈ 2%.
IV. Interactions: Returns That Depend on Another Variable
A. Economic Motivation (Education ↔ Experience)
Return to schooling depends on experience.
Two programmers, same education (12 years): Alice (20 years experience) vs Bob (1 year). An extra year of schooling benefits Bob more.
Return to experience depends on schooling.
Two workers, same experience (5 years): Alice (PhD) vs Bob (high school). An extra year of experience benefits Bob more.
B. Interaction Model and Marginal Effects
Model: (\ln(\text{Wage})=\beta_0+\beta_2,\text{Educ}+\beta_3,\text{Exp}+\beta_4(\text{Educ}\times\text{Exp})+u).
Returns (approx. % changes ×100):
To Experience: (\dfrac{\partial \ln(\text{Wage})}{\partial \text{Exp}}=\beta_3+\beta_4,\text{Educ}) (expect (\beta_4<0)).
To Education: (\dfrac{\partial \ln(\text{Wage})}{\partial \text{Educ}}=\beta_2+\beta_4,\text{Exp}) (expect (\beta_4<0)).
C. Initial Estimation with Interaction Only
Implementation example: (C(\text{Education})#C(\text{Experience})).
Result: (\beta_4) negative but essentially zero; p-value ≈ 0.94 (highly insignificant).
(R^2) barely changes (≈ 19% before and after).
Conclusion at this step: interaction alone did not improve fit.
V. Diminishing Returns to Experience: Add a Quadratic
A. Theory
Returns to experience diminish as experience rises (1→2 years matters more than 20→21 years).
B. Model with Interaction + Quadratic in Experience
Form: (\ln(\text{Wage})=\beta_0+\beta_2,\text{Educ}+\beta_3,\text{Exp}+\beta_4(\text{Educ}\times\text{Exp})+\beta_5,\text{Exp}^2+u).
Marginal return to Experience:
[
\frac{\partial \ln(\text{Wage})}{\partial \text{Exp}}
=\beta_3+\beta_4,\text{Educ}+2\beta_5,\text{Exp}.
]
• Interaction: depends on Education.
• Quadratic: depends on Experience itself.
C. Empirical Results with Quadratic Included
(\beta_4<0) and now large in magnitude and highly significant (significant at (\alpha=1%) and even lower).
(\beta_5<0) and highly significant (p-value ≈ 0), supporting diminishing returns.
(R^2) increases from ≈ 19% to ≈ 24% (≈ +5 percentage points) purely from functional-form change.
This allows returns from schooling to vary with experience and returns from experience to vary with education, improving explanatory power.
VI. Worked Implications (Plug-In Illustration Mentioned)
A. Using the Interaction Model
For two education levels (8 and 16 years), plugging in shows:
• +1 year of experience raises expected wages by ≈ 0.6% at 8 years of education.
• At 16 years of education, the return is slightly lower (≈ 0.57%); the difference is marginal.
VII. Model Comparison and Guidance
A. Do Not Drop Variables Based on One Misspecified Run
An insignificant interaction in the simpler model does not imply you should exclude it.
The earlier insignificance appears due to omitting an important term (the quadratic in experience).
Omitting important variables can bias results and cause issues (to be discussed under omitted variables).
B. Try Multiple Reasonable Specifications
Compare models before concluding.
If an interaction is insignificant across well-specified variants, then dropping it is more defensible.
Among the three models examined, the interaction + quadratic model is preferred (theory-consistent signs, high significance, higher (R^2)).
VIII. Why No Quadratic in Education Here
A. Theory as the Starting Point
A quadratic in education is not supported by the economic theory used here (no belief in increasing returns of this quadratic form).
B. Credential (Completion) Effects
The key pattern is discrete jumps at completion years (e.g., 12th → high school diploma; 16th → bachelor’s).
Within “dropout” status, the exact year matters much less than the credential.
This suggests the simple education-squared term is not the right way to capture education effects; completion matters most.
IX. Summary of Interpretations (One-Line References)
A. Log-Linear:
+1 unit in (X_k) → ≈ (100\times\beta_k)% change in expected (Y).
B. Linear-Log:
+1% in (X) → ≈ (0.01\times\beta) units of (Y).
C. Log-Log:
(\beta) is elasticity (1% in (X) → (\beta)% in (Y)).
D. Interactions:
Returns depend on another variable’s level.
E. Quadratic:
Returns depend on the same variable’s level.
F. Best Model Here:
Interaction + quadratic in experience (higher (R^2), theory-consistent signs, high significance).
Lecture 5 (Part 3): Joint Hypothesis Testing
Sometimes, a null hypothesis contains more than one equality — this is called a joint hypothesis (or joint conjecture).
Each equality inside the null is one “restriction.” The number of restrictions equals the number of equal signs in the null.
Example Scenario
Consider a model with diminishing returns from advertising, where the question of interest is:
“Does advertising have any effect on sales?”
To answer this, we set up a hypothesis test.
Null hypothesis (H_0): Advertising has no effect on sales.
Alternative hypothesis (H_1): Advertising does affect sales.
Mathematically:
For advertising to have no effect at all, both coefficients on the advertising terms must be zero:
[
H_0: \beta_3 = 0 \text{ and } \beta_4 = 0
]If at least one of these coefficients is not zero, then advertising does have an effect:
[
H_1: \text{At least one of } \beta_3, \beta_4 \ne 0
]
This means the null contains two equalities, so we are testing two restrictions simultaneously.
Because the null includes more than one equality, this is a joint hypothesis test.
II. Difference Between t-Tests and F-Tests
A. t-Tests
Previously, we used the t-test when testing one conjecture.
It could involve:
A single parameter (e.g., (H_0: \beta_2 = 0)); or
A linear combination of parameters (e.g., (H_0: \beta_1 + 10\beta_2 = 1000)).
Even if multiple parameters appear, as long as there’s one equality, it’s still one conjecture, so a t-test applies.
B. Why the F-Test is Needed
When the null hypothesis contains more than one equality (like (\beta_3 = 0) and (\beta_4 = 0)), we cannot use a t-test anymore.
Instead, we use an F-test, which is designed for joint hypotheses.
III. Restricted and Unrestricted Models
To perform an F-test, we compare two models:
1. Unrestricted Model
The unrestricted model is the original, full model.
It has no restrictions imposed. It includes all explanatory variables.
2. Restricted Model
The restricted model is formed by imposing the null hypothesis restrictions.
In our example, this means setting:
[
\beta_3 = 0 \text{ and } \beta_4 = 0
]
That is, removing the advertising variables.
So:
The unrestricted model includes the advertising terms.
The restricted model removes them.
IV. Formula for the F Statistic
After estimating both models, we obtain their Sum of Squared Errors (SSE):
(\mathrm{SSE}_R): from the restricted model.
(\mathrm{SSE}_U): from the unrestricted model.
The F-statistic measures how much worse the restricted model fits compared to the unrestricted one:
[
F = \frac{(\mathrm{SSE}_R - \mathrm{SSE}_U) / J}{\mathrm{SSE}_U / (n - K)}
]
Where:
(J): number of restrictions in the null hypothesis (number of equalities).
(n): number of observations.
(K): number of parameters in the unrestricted model (including the intercept).
Under the null hypothesis, (F) follows an F distribution with:
(J) numerator degrees of freedom, and
(n - K) denominator degrees of freedom.
V. Decision Rule
To decide whether to reject the null:
Choose a significance level (α).
Example: α = 0.05.Find the critical value from the F table using:
Row = numerator df = (J).
Column = denominator df = (n - K).
Compare your sample’s F-statistic to the table’s critical value:
If (F > F_{\text{critical}}): Reject (H_0).
If (F < F_{\text{critical}}): Do not reject (H_0).
VI. Intuition Behind the F-Test
A. Why the Ratio Works
The F-test compares two models’ errors.
When we impose restrictions (like setting coefficients to zero), we are reducing the model’s ability to explain Y.
Removing valid explanatory variables causes SSE to increase.
This increase tells us the restrictions might not be valid.
Therefore:
[
\mathrm{SSE}_R \ge \mathrm{SSE}_U
]
If the null hypothesis were true, removing the variables wouldn’t matter much — both SSE values would be similar.
But if removing them causes a large jump in SSE, it means those variables were important, and the null should be rejected.
B. Interpretation
A large F value means the restricted model fits much worse, implying the null restrictions are false.
A small F value means both models fit similarly, so the restrictions may be valid.
VII. Walkthrough Example 1 — Does Advertising Affect Sales?
Step 1: Hypotheses
[
H_0: \beta_3 = 0 \text{ and } \beta_4 = 0
]
[
H_1: \text{At least one of } \beta_3, \beta_4 \ne 0
]
This means the null states that advertising has no effect on sales, while the alternative says that advertising does affect sales.
Step 2: Information Given
(n = 75) (observations)
(K = 4) (parameters in full model)
(J = 2) (restrictions)
Denominator df = (n - K = 71)
Step 3: Estimate Both Models
Run the unrestricted model (with advertising terms) → get (\mathrm{SSE}_U).
Run the restricted model (without advertising terms) → get (\mathrm{SSE}_R).
Step 4: Compute the F Statistic
[
F = \frac{(\mathrm{SSE}_R - \mathrm{SSE}_U) / 2}{\mathrm{SSE}_U / 71}
]
Step 5: Decision
α = 0.05.
Critical value (F_{2,71;0.05} = 3.126.)
From the data: (F = 8.44.)
Since (8.44 > 3.126): Reject (H_0).
Conclusion:
At least one advertising coefficient differs from zero. Advertising does have a significant effect on sales.
Step 6: p-Value
The p-value = (P(F_{2,71} \ge 8.44)).
It is essentially 0, meaning the result is highly significant.
This implies we would reject the null for practically any reasonable α.
VIII. The Overall Significance of the Regression Model
A. Concept
The F-test can also be used to test the overall significance of a regression model — that is, whether the entire model explains the dependent variable at all.
The null hypothesis is that none of the explanatory variables have any effect on Y:
[
H_0: \beta_2 = \beta_3 = \beta_4 = 0
]
The alternative hypothesis is that at least one coefficient is not zero.
B. Why It Matters
This test checks whether your entire regression model is meaningful.
If you fail to reject (H_0), it means that none of your explanatory variables help explain Y.
C. Special Case Formula
When all slope coefficients are zero under (H_0), the restricted model includes only an intercept.
In that case:
[
\text{SSE}_R = \text{SST}
]
because there are no explanatory variables, and all variation in Y comes from the error.
The total sum of squares (SST) depends only on Y (the sample), so SST is the same for both models.
Hence:
[
F = \frac{(\text{SST} - \text{SSE}_U)/(K-1)}{\text{SSE}_U/(n-K)} = \frac{\text{SSR}_U/(K-1)}{\text{SSE}_U/(n-K)}
]
D. Example Walkthrough — Overall Model Significance
For the sales example:
(n = 75)
(K = 4)
So (K - 1 = 3)
Denominator df = (n - K = 71)
Null hypothesis:
[
H_0: \beta_2 = \beta_3 = \beta_4 = 0
]
Alternative:
At least one of them ≠ 0.
Compute:
From the unrestricted regression:
[
F = 24.459
]
Critical value: (F_{3,71;0.05} = 2.73).
Since (24.459 > 2.73): Reject (H_0).
Conclusion:
At least one explanatory variable explains sales — the model is meaningful.
The p-value is essentially 0, which is typical, because it is very unlikely that all regressors would truly have no effect.
IX. Relationship Between the F-Test and the t-Test
A. Mathematical Connection
For a single equality, the two tests are equivalent:
[
t^2_{(n-K)} = F_{(1, n-K)}
]
So, squaring a t-statistic gives the corresponding F-statistic for one numerator df.
B. Implications
For one equality, you can use either test (they give the same result).
For multiple equalities, you must use the F-test.
For inequality hypotheses (e.g., “>” or “<”), use t-tests, because F is one-tailed and cannot distinguish between sides.
X. Why the F-Test Often Matters More
When you have multiple coefficients to test together (like advertising terms), separate t-tests might fail to reject each one individually — but the joint F-test can still reject the null.
This happens because the F-test captures the combined effect of multiple variables that might be individually weak but jointly important.
XI. Summary of Key Ideas
Concept | Meaning |
|---|---|
Joint hypothesis | Null has multiple equalities; test them together. |
t-test | Tests one conjecture (one equality). |
F-test | Used when the null has multiple equalities. |
Restricted model | Model after imposing null restrictions. |
Unrestricted model | Full model without restrictions. |
F-statistic | Compares fit between restricted and unrestricted models. |
Decision rule | Reject (H_0) if (F > F_{\text{critical}}). |
Overall significance | Tests whether all slopes = 0. |
Relationship to t-test | (t^2 = F) when (J=1). |
Inequalities | Use t-test, not F-test. |
Final Takeaway
The F-test generalizes the t-test.
When testing more than one equality, or when testing whether the entire model is significant, use an F-test.
If the F-statistic is large, the restrictions in the null (like setting advertising coefficients to zero) cause a large loss in explanatory power — meaning those restrictions are false, and the variables do matter.
Would you like me to include the mathematical intuition and derivation (why dividing by (J) and (n-K) creates the F ratio and what it measures about variance ratios), written step-by-step in the same style?
Lecture 6 (Part 1): Choosing the Best Econometric Model
I. Importance of Model Choice
In any econometric investigation, choosing the correct model is the most important decision. The quality of your results, and therefore the usefulness of your conclusions, depends entirely on whether the model specification is correct.
To choose the best model, three questions must be addressed:
What are the important considerations when choosing a model?
What are the consequences of choosing the wrong model?
How can we assess whether our model is adequate or not?
II. Review: Correct Functional Form
Previously, the importance of choosing the correct functional form was discussed. If the chosen form is wrong, the results are misleading.
Example:
Suppose a model of revenue as a function of advertising assumes constant returns when the true relationship has diminishing returns.
If advertising is low, an additional $100 investment increases revenue.
If advertising is already high, an additional $100 may reduce profits or even lead to losses.
Even if the average effect shows a return of $2.00 per $1.00 spent, this average is meaningless for managerial decision-making.
Conclusion:
Choosing the wrong functional form leads to incorrect conclusions and potentially harmful business decisions.
III. Two Additional Model Specification Issues
Today’s focus is on two additional sources of model error:
Omitted Variables – leaving out an important explanatory variable.
Irrelevant Variables – including variables that do not belong in the model.
IV. Omitted Variable Bias
A. Definition and Causes
An omitted variable is a relevant explanatory variable that is incorrectly left out of the regression model.
Common causes include:
Negligence or lack of experience (overlooking relevant factors).
Data limitations, which are common in practice.
When data are limited, we must understand the consequences of omission.
B. Example: Family Income Model
Suppose the true model explaining family income (( Y )) includes two variables:
[
Y_i = \beta_1 + \beta_2(\text{Husband Education}) + \beta_3(\text{Wife Education}) + \varepsilon_i
]
Assume this is the correct model—only these two variables affect income.
Each additional year of husband’s education increases expected income by $3,100.
Each additional year of wife’s education increases expected income by $4,500.
Both coefficients are highly significant (( p \approx 0 )).
Now, if we omit wife’s education and estimate:
[
Y_i = \beta_1 + \beta_2(\text{Husband Education}) + v_i
]
we obtain a new estimate for husband’s education: $5,100 per year—still significant but overstated by $2,000.
This means that omitting a relevant variable (wife’s education) biases the estimated coefficient of husband’s education.
This phenomenon is known as Omitted Variable Bias (OVB).
C. Mathematical Intuition for the Bias
Let the true model be:
[
Y_i = \beta_1 + \beta_2 X_{2i} + \beta_3 X_{3i} + \varepsilon_i
]
But the estimated model omits ( X_3 ):
[
Y_i = \beta_1 + \beta_2 X_{2i} + v_i
]
The omitted variable’s effect becomes part of the new error term:
[
v_i = \beta_3 X_{3i} + \varepsilon_i
]
From class 2, the OLS estimator for ( \beta_2 ) in the simple regression is:
[
b_2 = \frac{\sum (x_{2i} - \bar{x}2)(y_i - \bar{y})}{\sum (x{2i} - \bar{x}_2)^2}
]
Substituting the true relationship and taking expectations gives:
[
E[b_2] = \beta_2 + \beta_3 \frac{\text{Cov}(X_2, X_3)}{\text{Var}(X_2)}
]
Thus, the bias is:
[
\text{Bias}(b_2) = \beta_3 \frac{\text{Cov}(X_2, X_3)}{\text{Var}(X_2)}
]
D. Interpretation of the Bias
Existence of Bias
The estimator is unbiased only if ( \text{Cov}(X_2, X_3) = 0 ).
If ( X_2 ) and ( X_3 ) are uncorrelated, omitting ( X_3 ) has no effect.
But in most real-world cases, explanatory variables are correlated, creating bias.
Magnitude of Bias
The bias is large when either:
( |\beta_3| ) is large (the omitted variable has a strong effect on ( Y )), or
( |\text{Cov}(X_2, X_3)| ) is large (strong correlation between regressors).
If both are large, the bias is substantial.
Direction of Bias
The sign of the bias depends on the signs of ( \beta_3 ) and ( \text{Cov}(X_2, X_3) ):
Same signs → positive (upward) bias.
Opposite signs → negative (downward) bias.
Persistence of Bias
The bias does not disappear as the sample size increases.
Therefore, the estimator is not only biased but also inconsistent.
Even with millions of observations, if an important variable is omitted, the estimator will never converge to the true parameter.
Thus, the OLS estimator under omission is biased and inconsistent, rendering it useless.
E. Application to the Family Income Example
Correlation Structure
Correlation between husband and wife education ≈ 0.6 (positive and significant).
People often marry partners with similar education levels.
Coefficients and Signs
( \beta_3 > 0 ): Wife’s education positively affects income.
( \text{Cov}(X_2, X_3) > 0 ): Husband’s and wife’s education are positively correlated.
Therefore, bias is positive → upward bias.
The estimated coefficient for husband’s education increases from $3,100 (true) to $5,100 (biased).
The bias (14,000, significant at the 1% level.If we omit this variable and estimate only with parents’ education, the coefficients on education change very little.
Why? Because the correlation between young children and parents’ education is very small:
( \text{Corr}(\text{Husband Education}, \text{Young Kids}) ≈ 0.1 )
( \text{Corr}(\text{Wife Education}, \text{Young Kids}) ≈ 0.13 )
Thus, the bias is negligible.
Since ( \beta_4 < 0 ) (negative) and covariance is slightly positive, the bias is negative (downward), but very small.
The estimated return to education changes from $3,200 to $3,100 — practically insignificant.
Conclusion:
We only worry about omitted variables when:The omitted variable has a large effect, and/or
It is highly correlated with variables in the model.
Every model omits something, but small or uncorrelated omissions are not a serious problem.
G. Violation of OLS Assumptions
The OLS estimator becomes biased because one of its classical assumptions no longer holds.
In particular, the assumption that:
[
E(\varepsilon_i | X_i) = 0
]is violated.
When a relevant variable is omitted, the error term contains systematic variation correlated with regressors, so the expected value of the error is not zero.As a result:
The estimator is biased.
It is also inconsistent.
OLS is no longer BLUE (Best Linear Unbiased Estimator).
V. Irrelevant Variables
A. Definition and Problem Setup
The opposite problem occurs when we add variables that should not be in the model.
Although omitting variables is dangerous, including too many variables is also undesirable.
B. Theoretical Problems
A good model explains complex reality with relatively few explanatory variables.
Adding unnecessary ones causes several problems:Overcomplication: The model becomes cluttered and hard to interpret.
Overfitting:
The model fits the current sample well but fails to generalize.
Results lose meaning for the population.
Example:
Estimating wages based on:Education
Type of degree (e.g., business vs. humanities)
Type of school (private/public, Ivy League or not)
Student rank, etc.
If we keep adding details endlessly, the model becomes as complex as reality itself—useless for insight.
C. Empirical Effects: Variance Inflation
Adding irrelevant variables inflates the variance of the estimators.
Example:
Two artificial variables ( X_5 ) and ( X_6 ) were added to the family income model.
These were correlated with education variables but had no real relationship with family income.( X_5, X_6 ): High correlation with husband and wife education.
As expected, their ( p )-values are large (insignificant).
However, after adding them, the ( p )-values of the real variables (education) increased.
For instance, wife’s education was highly significant (( p \approx 0 )) before; after adding irrelevant variables, it became significant only at ( \alpha = 1% ).
In other datasets, this inflation could even make an important variable appear insignificant at ( \alpha = 5% ).
D. Consequences and Comparison with Omitted Variables
Problem Type
Bias
Consistency
Efficiency
Practical Effect
Omitted Variable
Biased
Inconsistent
—
Misleading estimates
Irrelevant Variable
Unbiased
Consistent
Inefficient
Larger standard errors, higher p-values
Including irrelevant variables:
Keeps OLS unbiased and consistent.
But increases standard errors, leading to smaller t-values and larger p-values.
This can make truly important variables appear statistically insignificant.
Thus, adding “junk” variables can mask real effects.
VI. Summary
Omitted Variables:
Bias and inconsistency arise.
Bias depends on the size and correlation of omitted variables.
Leads to incorrect, misleading estimates.
Irrelevant Variables:
OLS remains unbiased and consistent.
But inefficiency increases (larger standard errors).
Risk of concluding “no effect” when an effect exists.
Both cases show why careful model specification is crucial.
The goal is to balance simplicity and completeness—include all important variables, but only those that truly belong.Would you like me to continue with Lecture 6 Part 2 in the same structured format when you send it?
Lecture 6 (Part 2): Model Construction, Testing, and Selection
Model Construction, Testing, and Selection
I. The Foundation of Model Building
A. The Role of Economic Theory
Every econometric model must begin with economic theory.
Theory determines which variables belong in the model and what relationships to expect between them.
It also guides the functional form of the equation — how variables are mathematically related to one another.Starting with the correct theoretical structure is essential.
If the model is built on a poor theoretical foundation, no amount of statistical refinement will make it a good model.
A weak starting point leads inevitably to a weak result, even with perfect data and advanced techniques.Before estimating any equation, it is crucial to research what others have done on similar topics.
Most problems in economics have been studied before.
Reviewing previous studies helps identify reasonable variables, functional forms, and expectations.
Existing models can then be adapted to fit the specific context or dataset being analyzed.Economic reasoning is therefore the cornerstone — everything else builds on it.
II. Evaluating an Estimated Model
Once the model is estimated, it must be examined carefully to determine whether it makes sense in both statistical and economic terms.
A. Checking Coefficient Signs
The first step is to check whether the signs of the estimated coefficients align with theoretical expectations.
If a variable that should increase the outcome shows a negative coefficient, or vice versa, this suggests potential misspecification.B. Checking Magnitudes
The next step is to evaluate whether the magnitudes of coefficients are reasonable.
For example, in wage equations estimated with data from developed countries, the expected return to an additional year of schooling is around 3–4%.
If the estimate shows a return of 12%, this suggests something is wrong — possibly an omitted variable, which would bias the education coefficient upward.However, a 12% return might make sense in a developing country, where education is relatively scarce and its impact larger.
Thus, magnitude assessment always depends on the economic environment and context of the data.III. Constructing the Wage Equation
A typical example of model construction is the wage equation, where the goal is to explain variations in wages using individual characteristics.
This model includes education, experience, and hours worked, along with squared and interaction terms to capture nonlinear effects.
A. Expected Signs of Coefficients
Variable
Expected Sign
Economic Reasoning
Education (β₂)
Positive
More education increases productivity and wages.
Education² (β₃)
Positive
Returns to education rise at higher levels (e.g., graduate degrees).
Experience (β₄)
Positive
More experience generally raises wages.
Experience² (β₅)
Negative
Diminishing returns to experience; later years add less.
Education × Experience (β₆)
Negative
The benefit of an extra year of education falls for highly experienced workers.
Hours Worked (β₇)
Positive
More hours lead to higher total pay.
B. Capturing Education More Precisely
Although an education-squared term allows for nonlinear returns, it is not the most accurate way to model education.
Not all years of education matter equally.
Completing a degree (such as a bachelor’s, master’s, or PhD) often causes a larger jump in wages than an additional year of schooling within a degree.
A more precise approach would use dummy variables indicating the highest degree attained.
These can capture discrete jumps in education levels, such as “has a bachelor’s degree” or “has a master’s degree.”IV. Evaluating the Estimated Wage Model
When this model is estimated, the signs of the coefficients typically match expectations:
The coefficient on experience is positive.
The coefficient on experience² is negative, confirming diminishing returns.
Education is positive.
Education² is positive, showing increasing returns.
The interaction term between education and experience is negative.
The coefficient on hours worked is positive.
However, a striking issue arises: none of the education variables are statistically significant.
Their p-values are large, meaning that statistically, one cannot reject the idea that education has no effect on wages.This seems implausible — it is unlikely that education truly has no effect.
Therefore, further testing is required.V. Testing the Significance of Education Variables
A. Purpose of the Test
To determine whether education as a group (education, education², and education × experience) has any joint effect on wages.
B. Hypotheses
[
H_0: \beta_2 = \beta_3 = \beta_6 = 0
]
[
H_1: \text{At least one of } \beta_2, \beta_3, \beta_6 \neq 0
]The null hypothesis states that none of the education terms affect wages.
The alternative states that at least one does.C. Interpretation of the Test
Rejecting the null means education contributes to explaining wages.
Failing to reject it would suggest that education has no role, which would contradict established economic reasoning.D. Conceptual Result
The calculated F-statistic for this test is approximately 86.45, with a p-value near zero.
This is an extremely strong result, meaning the null hypothesis is decisively rejected.
Thus, education clearly affects wages.VI. Manual Calculation of the F-Test
On an exam, the F-test must be computed manually from two regressions.
A. Two Models
Unrestricted model: includes all variables, including education terms.
Restricted model: excludes the education variables entirely.
B. Required Components
( SSR_r ): Sum of squared residuals for the restricted model.
( SSR_u ): Sum of squared residuals for the unrestricted model.
( J ): Number of restrictions (here, 3: β₂, β₃, β₆).
( n ): Number of observations (here, 1000).
( K ): Number of parameters in the unrestricted model (6 explanatory variables + intercept = 7).
C. Formula
[
F = \frac{(SSR_r - SSR_u)/J}{SSR_u/(n - K)}
]D. Decision
In this case, ( F = 86.44 ) and the critical value at 5% significance is ( 2.6 ).
Because ( 86.44 > 2.6 ), the null hypothesis is rejected.
Education collectively affects wages.VII. Investigating Large Standard Errors
Even though education matters, its estimated coefficients are individually insignificant.
This may occur because of redundant or irrelevant variables in the model.
Including unnecessary terms can inflate standard errors, making all coefficients appear less precise.
To address this, smaller subsets of education variables should be tested to identify which are truly needed.VIII. Subset F-Test: Testing Education² and Interaction
A. Hypotheses
[
H_0: \beta_3 = 0, ; \beta_6 = 0
]
[
H_1: \text{At least one of } \beta_3, \beta_6 \neq 0
]B. Interpretation of the Null
If this null is true, the relationship between education and wages is linear.
The return to each additional year of education is constant.
For example, if ( \beta_2 = 0.04 ), every additional year of education increases wages by approximately 4%.C. Rejection of the Null
Rejecting the null would mean the return to education is not constant — it varies depending on education or experience.
D. Results
The calculated F-statistic is approximately 2.8, with a p-value of 6.1%.
At the 5% significance level, the null cannot be rejected.
Thus, the relationship is linear, and the education² and interaction terms can be dropped.IX. The Simplified Wage Model
After removing the redundant terms, the model becomes:
[
\text{WAGE} = \beta_0 + \beta_2(\text{EDUC}) + \beta_4(\text{EXP}) + \beta_5(\text{EXP}^2) + \beta_7(\text{HOURS}) + \varepsilon
]A. Interpretation
The relationship between education and wages is linear.
Each additional year of education increases wages by about 10%.
All remaining variables are statistically significant and have the correct signs.
B. Comparison of Fit
Model
R² (%)
Full model
31.7
Simplified model
31.35
Dropping the extra education terms barely changes explanatory power, confirming they were redundant.
X. Criteria for Model Comparison
To decide which model is best among several valid options, use model selection criteria.
A. Adjusted R²
R² always rises when variables are added, even if they are irrelevant.
Adjusted R² corrects this by penalizing complexity:[
\text{Adjusted } R^2 = 1 - \frac{(SSE/(n - K))}{(SST/(n - 1))}
]When a variable is added:
SSE decreases → adjusted R² increases.
K increases → adjusted R² decreases.
Adjusted R² increases only if the improvement in fit outweighs the penalty for adding a variable.
The model with the higher adjusted R² is preferred.B. Information Criteria: AIC and BIC
Both the Akaike Information Criterion (AIC) and the Bayesian Information Criterion (BIC) include:
A measure of fit (based on SSE).
A penalty for model complexity.
The formulas differ in the size of the penalty:
AIC uses ( 2K ).
BIC uses ( K \ln(n) ).
Since ( \ln(n) > 2 ) for most samples, BIC penalizes extra variables more strongly.
Therefore, BIC tends to prefer simpler models.Lower values of AIC and BIC indicate better models.
XI. Example: Comparing Models
A. Family Income Example
A model including husband’s education, wife’s education, and the number of young children is considered best because:
It has the highest adjusted R².
It has the lowest AIC and BIC.
Plain R² is not informative because it always increases with more variables.
B. Education Model Comparison
Model
Adjusted R²
AIC
BIC
Preferred
Full model
31.73%
1348
1383
AIC, Adj. R²
Constant-return (linear only)
31.07%
1350
1374
BIC
No education
Very low
Very high
Very high
—
Although adjusted R² and AIC slightly favor the full model, the differences are minimal.
The BIC provides a stronger preference for the simpler model.
Therefore, the constant-return model is selected.XII. Rules for Model Comparison
Information criteria are valid only when comparing models with the same dependent variable.
For example, a model explaining wages cannot be compared directly to one explaining log(wages).
The dependent variable must remain the same; explanatory variables may differ.XIII. Importance of Economic Reasoning
Model selection methods — F-tests, adjusted R², AIC, and BIC — are refinements, not substitutes for theory.
These tools help compare and improve models, but they cannot define the model in the first place.
Only theory can determine which variables belong and what form the relationships should take.A strong wage model should include not just education, experience, and hours, but also key demographic factors such as gender and race, which significantly influence wages.
Without these, even statistically sound models may give biased estimates.After additional learning, variables like gender and race can be introduced through dummy variables, improving the model’s explanatory power and realism.
XIV. Summary of Key Steps
Start with theory — it defines relevant variables and their expected signs.
Estimate the model and check whether coefficients make sense.
Test for significance of groups of variables with F-tests.
Simplify intelligently by removing redundant terms.
Compare models using adjusted R², AIC, and BIC.
Prefer simpler models when fit differences are minimal.
Maintain theoretical consistency — statistical tests do not replace sound economic reasoning.
Would you like me to add a separate section after this that breaks down every test and model selection criterion (F-test, adjusted R², AIC, BIC) with short “exam reasoning walkthroughs,” showing how to state hypotheses, interpret outcomes, and decide between models step by step?
Lecture 6, Part 3: Collinearity
I. Understanding Collinearity
Collinearity is another challenge in econometric modeling, but unlike model specification issues, it is not related to choosing the best model, the right explanatory variables, or the correct functional form. When a collinearity problem occurs, the issue lies not with the model itself but with the data. Collinearity is fundamentally a data quality problem.
Collinearity arises when variables in non-experimental (observational) data move together in systematic ways. This typically happens in datasets that are the result of uncontrolled experiments, where economic variables may co-vary naturally. For instance, in survey data, husband’s education and wife’s education tend to be strongly correlated because individuals often choose partners with similar education levels. Such relationships appear in real-life data but would not exist in controlled experiments, where one variable is intentionally varied while all others are held constant. Controlled experiments allow isolation of the causal effect of one variable on another; observational data do not.
II. Collinearity vs. Multicollinearity
Collinearity does not violate the assumptions of the linear regression model. If it did, we would not be able to estimate models at all—yet we do so frequently, even when some explanatory variables are correlated. Recall from earlier lectures that the least squares estimator is designed to isolate the effect of each explanatory variable even in the presence of correlation among regressors.
However, when the correlation among explanatory variables is very high, it can lead to a collinearity problem (or multicollinearity problem). Both terms are used interchangeably in the literature. The key distinction is that collinear variables simply refer to correlated explanatory variables, whereas a collinearity problem arises when this correlation is so strong that it interferes with estimation precision.
III. Variance of the Estimator and the Role of Correlation
Consider a multiple linear regression model with two explanatory variables, ( X_2 ) and ( X_3 ). The variance of the estimator ( \hat{\beta}_2 ) in this model depends not only on the usual factors—such as the variance of the error term and the sample size—but also on the correlation between ( X_2 ) and ( X_3 ).
In the simple linear regression model, the variance of an estimator ( \hat{\beta}_2 ) is:
[
Var(\hat{\beta}_2) = \frac{\sigma^2}{\sum (X_2 - \bar{X}_2)^2}
]where:
( \sigma^2 ) = variance of the error term
( \sum (X_2 - \bar{X}_2)^2 ) = sample variance of the explanatory variable ( X_2 )
This shows that ( Var(\hat{\beta}_2) ) is a function of three factors:
Error variance (( \sigma^2 )) — Larger ( \sigma^2 ) increases the estimator’s variance.
Sample size (( n )) — A larger ( n ) decreases variance.
Variance of the explanatory variable (( X_2 )) — Higher variance in ( X_2 ) reduces the variance of ( \hat{\beta}_2 ).
In the multiple linear regression model, variance also depends on the correlation coefficient ( \rho ) between explanatory variables ( X_2 ) and ( X_3 ). The formula includes a term that captures this correlation:
[
Var(\hat{\beta}2) = \frac{\sigma^2}{(1 - \rho{23}^2) \sum (X_2 - \bar{X}_2)^2}
]Here, ( \rho_{23} ) is the sample correlation coefficient:
[
\rho_{23} = \frac{Cov(X_2, X_3)}{s_{X_2} s_{X_3}}
]where ( s_{X_2} ) and ( s_{X_3} ) are the standard deviations of ( X_2 ) and ( X_3 ).
The correlation coefficient ranges from −1 to +1, measuring the strength of the linear relationship between ( X_2 ) and ( X_3 ).As ( |\rho_{23}| ) approaches 1, the variance of ( \hat{\beta}2 ) increases dramatically. When ( \rho{23} = 1 ) or ( \rho_{23} = -1 ), the variance becomes infinite, meaning the estimator cannot be computed. Thus, high correlation leads to unstable and imprecise estimates.
IV. Intuition Behind Collinearity Problems
When two explanatory variables move closely together, it becomes difficult to distinguish their individual effects on the dependent variable ( Y ). Suppose ( X_2 ) and ( X_3 ) are positively correlated and both coefficients ( \beta_2 ) and ( \beta_3 ) are positive. When ( X_2 ) increases, ( Y ) increases; but since ( X_3 ) also tends to rise simultaneously, ( Y ) rises for both reasons. Consequently, it is impossible to tell how much of the increase in ( Y ) is due to ( X_2 ) and how much is due to ( X_3 ). This uncertainty increases the standard errors of ( \hat{\beta}_2 ) and ( \hat{\beta}_3 ).
V. Exact Collinearity
There are two cases of exact collinearity where the variance of the estimator becomes infinite, and the least squares estimator cannot be computed.
A. Perfect Correlation Between Explanatory Variables
When ( \rho_{23} = 1 ) or ( \rho_{23} = -1 ), ( X_2 ) and ( X_3 ) move exactly together, making it impossible to isolate their effects. The least squares procedure fails, and the estimator is not defined.
B. Zero Variance in an Explanatory Variable
When ( Var(X_2) = 0 ), the variable is constant across observations. This means ( X_2 ) is perfectly collinear with the intercept term. For example, if everyone in the sample has exactly 12 years of education, then education has no variation and its effect on wages cannot be estimated. Since the least squares method relies on variation in ( X ) to explain variation in ( Y ), the estimator cannot be defined in this case.
VI. Near-Exact Collinearity
In practice, exact collinearity is rare. More often, we encounter near-exact collinearity, where correlation is very high (close to ±1) or variance of a regressor is very low (close to 0). In such cases:
The OLS estimator is still defined, so regression results can be obtained.
However, these results may be unreliable because standard errors become large.
VII. Consequences and Symptoms of Collinearity
When multicollinearity is high but not perfect:
Large Standard Errors
High correlation inflates the standard errors of estimators. As a result, t-tests often fail to reject the null hypothesis that a coefficient equals zero, even when the variable is important. This can lead to incorrect conclusions about insignificance.Wide Confidence Intervals
Large standard errors translate into wide confidence intervals, indicating that the estimates are imprecise.Sensitivity to Model Changes
Estimates may change dramatically if a few observations are added or removed or if a correlated variable is included or excluded. Dropping a correlated variable can make another variable suddenly significant because the omitted variable’s effect becomes captured by the remaining one. Similarly, reducing sample variation (e.g., removing outliers) can increase collinearity.
There is no fixed correlation threshold that determines when collinearity becomes a problem. For instance, even a correlation of 0.8 or 0.9 might be tolerable if the variance of explanatory variables is large enough. Thus, collinearity is not defined by a single rule—it depends on both correlation and variance.
VIII. Collinearity as a Data Problem
Collinearity is primarily a data problem, not a violation of the OLS assumptions. As long as estimates are significant and have the expected signs and magnitudes, there is no issue. However, when results are unexpectedly poor, collinearity may be the reason.
IX. Example: Car Mileage Regression
Consider a model estimating the effect of car characteristics on mileage (miles per gallon):
[
\text{Mileage} = \beta_0 + \beta_1(\text{Cylinders}) + \beta_2(\text{Engine Size}) + \beta_3(\text{Weight}) + u
]We expect all coefficients to be negative: more cylinders, larger engines, and heavier cars reduce mileage.
The data, however, exhibit very high correlations:
Corr(Engine, Cylinders) = 0.95
Corr(Cylinders, Weight) = 0.89
Corr(Engine, Weight) = 0.93
This occurs because in real life, manufacturers produce heavy cars with large engines and many cylinders.
A. Step 1: Regression with Cylinders Only
When mileage is regressed only on cylinders, the coefficient is significant with a near-zero p-value.
B. Step 2: Adding Engine
Including engine still yields significant results, despite the high correlation.
C. Step 3: Adding Weight
When weight is added, both engine and cylinders become insignificant, and their p-values increase dramatically. This suggests collinearity, as high correlations among explanatory variables make it difficult to isolate their individual effects.
X. Testing for Collinearity
To confirm collinearity, an F-test is used:
[
H_0: \beta_2 = \beta_3 = 0 \quad \text{vs.} \quad H_1: \text{At least one } \beta_i \neq 0
]This joint hypothesis test distinguishes between two cases:
Both variables truly have no effect (irrelevant).
Variables are relevant, but their effects cannot be separated due to collinearity.
If ( H_0 ) is rejected, it means the variables jointly affect ( Y ) (mileage), confirming that the problem is collinearity rather than irrelevance.
In the example, the F-statistic = 4.3 with p-value = 0.014 (< 0.05). Thus, ( H_0 ) is rejected, confirming collinearity.XI. Addressing Collinearity
A. Option 1: Obtain More Data
The best solution is to collect additional data. Larger sample sizes reduce variance and may increase variability among explanatory variables, which decreases standard errors. However, this is not always feasible.
B. Option 2: Keep the Model Despite Collinearity
If variables are relevant (as confirmed by the F-test), retain them even if they are individually insignificant. The model can still be used for prediction even if individual effects cannot be precisely identified. Predictions remain valid as long as the same relationships hold in future data.
C. Option 3: Drop a Variable (Last Resort)
Dropping one variable can reduce collinearity but introduces omitted variable bias if the omitted regressor is relevant. This option is sometimes used in fields like marketing, where Principal Component Analysis (PCA) is applied to remove redundant variables with similar meaning. For example, survey questions like “satisfied with product,” “would recommend to a friend,” and “would purchase again” are all highly correlated and can be represented by a single “customer satisfaction” factor. However, this approach is inappropriate when variables measure distinct concepts (e.g., engine size vs. cylinders).
XII. Detecting Collinearity
A. Pairwise Correlations
Compute simple pairwise correlations among explanatory variables. High pairwise correlations (above 0.8) may indicate a problem.
B. Auxiliary Regression (Variance Inflation)
When collinearity involves multiple variables, pairwise correlations may not capture it. In that case:
Regress each explanatory variable on all the others.
Examine the ( R^2 ) of this auxiliary regression.
If ( R^2 > 0.8 ), it suggests strong collinearity, as most of the variation in one regressor can be explained by the others.
XIII. Summary
Collinearity is a data problem, not an OLS violation.
Exact collinearity (perfect correlation or zero variance) makes estimation impossible.
Near-exact collinearity causes inflated standard errors, wide confidence intervals, and unstable estimates.
If estimates are reasonable and significant, there is no problem.
Only check for collinearity when results appear unreasonably poor.
Would you like me to export this Markdown-formatted version into a PDF or Word file next (keeping all the formatting and math clean for printing)?