Math and Probability Background for Econometrics
Summation Notations and Mathematical Rules
Basic Summation Formula: The summation symbol represents the addition of a sequence of numbers. Formally, for a sequence of observations from to , the sum is defined as:
Constants in Summation: If a variable or value in the summation does not have a subscript (meaning it is a constant relative to the index ), it can be pulled out of the summation:
Variables with Subscripts: If a variable in the summation has a subscript, it cannot be pulled out of the summation. For example, . This cannot be simplified as a general matter.
Non-Linear Functions in Sums: As a general rule, a non-linear function applied to a sum is not the same as the sum of the non-linear functions of individual observations. For instance, will not equal except in very specific circumstances, such as when for all observations.
Mathematical Expectation: Definitions and Properties
Definition of Expectation: The expectation is the value we expect a random variable to be. It essentially represents the average of the random variable if we were to sample from the variable's distribution an infinite number of times.
Example: Six-Sided Die: The expected value of a single roll of a six-sided die is . This is calculated because each side is expected to appear in an equal proportion of times over many trials. Formally, the expected value is calculated as: where is the set of outcomes and is the probability of each outcome ( for each value).
Expectation of a Constant Times a Function: The expectation of a constant multiplied by a function is equal to the constant multiplied by the expectation of the function:
Application: If wanting to know the expectation of times the number on a die, it is simply multiplied by the expectation of the die roll ().
Variance: Measures of Distribution Spread
Definition: The variance of a random variable is a formal measure of how spread out the distribution is.
Large Sample Variance Formula: In large samples, variance is estimated as:
Small Sample Variance Formula: In small samples, a degrees of freedom correction is applied, dividing by instead of . For large , the difference is negligible, and software programs typically handle this correction automatically.
De-construction of the Variance Equation: Calculating variance involves three simple math steps:
- Take the deviation from the mean for each observation ().
- Square each deviation to ensure the values are positive.
- Calculate the average of these squared deviations.
Property 1 (Constant Addition): The variance of a constant plus a random variable is simply the variance of the random variable:
Property 2 (Constant Multiplication): The variance of a random variable multiplied by a constant is the constant squared times the variance of the variable:
Property 3 (Correlated Random Variables): When random variables and are correlated, the variance of their sum or difference involves their covariance:
Property 4 (Uncorrelated Random Variables): If two variables are uncorrelated, their covariance equals zero (), and the variance of their sum or difference is simply the sum of their individual variances:
Covariance: Measuring Relationship Between Variables
Definition: Covariance measures how much two random variables vary together.
Large Sample Covariance Formula:
Property 1: The covariance of a random variable and a constant is always zero:
Property 2: The covariance of a random variable with itself is simply the variance of that variable:
Property 3: The covariance of and (where are constants and are variables) is:
Property 4: Adding a constant to a random variable does not change its covariance with another variable:
Correlation and Its Relation to Regression
Correlation Formula: Correlation is the covariance of two variables divided by the product of their standard deviations:
Correlation Bounds:
- Perfect positive correlation () results in .
- Perfect negative correlation () results in .
- Variables are also perfectly correlated if they are identical once normalized (e.g., or ).
Bivariate Regression Connection: The bivariate regression coefficient is a re-standardized correlation:
Probability Density Functions (PDFs)
Definition: A PDF is a mathematical function describing the relative probability for a continuous random variable to take on a specific probability.
Fundamental Features:
- The values of a PDF () are greater than or equal to zero for all possible values of .
- The total area under the PDF curve equals one:
Continuous Nature: For a continuous variable, the probability of the variable equaling an exact decimal (e.g., ) is effectively zero because there are infinite possible values near it. Consequently, probabilities must be calculated for regions/intervals of values using calculus.
Interval Calculation: The probability that a random variable is between and is the integral of the PDF from to : , where is the integral (Cumulative Distribution Function).
Normal and Standard Normal Distributions
Normal Distribution: A specific and famous type of PDF.
Standard Normal Distribution: A normal distribution with a mean () of zero and a variance () of one. The standard deviation () is also one.
Cumulative Distribution Function (CDF): Denoted as . It indicates the probability of seeing a random variable less than the value , representing the area under the PDF curve to the left of .
Symmetry: The normal distribution is symmetric. Half of the area lies to the left of the mean (), and half to the right.
Standard Normal Probabilities (Examples from Figure A.2 and A.3):
- ( chance of observed value being below )
- ( chance of observed value being below )
- ( chance of observed value being below )
- ( chance of observed value being below )
Calculating Probabilities Greater Than Z: Using the complement rule (), the probability that a standard normal variable is greater than is calculated as . Examples:
Standard Deviations as Units: In a standard normal distribution, values can be interpreted as the number of standard deviations away from the mean. For instance, is standard deviations below the mean. The probability of observing a value less than is .
Standardizing Normally Distributed Random Variables
The Standardization Process: Any normally distributed random variable can be converted to the standard normal distribution using its mean and standard deviation.
Standardization Formula: To find how many standard deviations a value is away from the mean () with standard error :
Standardized Values (Examples from Table A.1):
- Example 1: (Value is SD above the mean).
- Example 2: (Value is SD above the mean).
- Example 3: (Value is SD above the mean).
- Example 4: (Value is SD below the mean).
- Example 5: (Value is SD below the mean).
- Example 6: (Value is SD below the mean).
Questions & Discussion
- Question 1: What is the probability a standard normal random variable is less than or equal to ?
- Question 2: What is the probability a standard normal random variable is less than or equal to ?
- Question 3: What is the probability a standard normal random variable is greater than ?
- Question 4: What is the probability a normal random variable with a mean of zero and a standard deviation of is less than ?
- Question 5: What is the probability a normal random variable with a mean of zero and a variance of is less than ?
- Question 6: Approximately what is the probability a normal random variable with a mean of and a variance of is less than ?
Supplemental Statistical Distributions
- The (Chi-squared) Distribution: Describes the distribution of squared normal variables. Specifically, the distribution of a squared standard normal random variable follows a distribution.
- Other Workhorse Distributions: Other common distributions derived from the normal distribution used in econometric practice include the and distributions.