Math and Probability Background for Econometrics

Summation Notations and Mathematical Rules

  • Basic Summation Formula: The summation symbol ∑\sum represents the addition of a sequence of numbers. Formally, for a sequence of observations XiX_i from i=1i=1 to NN, the sum is defined as: ∑i=1NXi=X1+X2+X3+...+XN\sum_{i=1}^N X_i = X_1 + X_2 + X_3 + ... + X_N

  • Constants in Summation: If a variable or value in the summation does not have a subscript (meaning it is a constant relative to the index ii), it can be pulled out of the summation: ∑i=1NaXi=aX1+aX2+aX3+...+aXN=a(X1+X2+X3+...+XN)=a∑i=1NXi\sum_{i=1}^N aX_i = aX_1 + aX_2 + aX_3 + ... + aX_N = a(X_1 + X_2 + X_3 + ... + X_N) = a\sum_{i=1}^N X_i

  • Variables with Subscripts: If a variable in the summation has a subscript, it cannot be pulled out of the summation. For example, ∑i=1NXiYi=X1Y1+X2Y2+X3Y3+...+XNYN\sum_{i=1}^N X_i Y_i = X_1Y_1 + X_2Y_2 + X_3Y_3 + ... + X_N Y_N. This cannot be simplified as a general matter.

  • Non-Linear Functions in Sums: As a general rule, a non-linear function applied to a sum is not the same as the sum of the non-linear functions of individual observations. For instance, ∑i=1NXi2\sum_{i=1}^N X_i^2 will not equal (∑i=1NXi)2(\sum_{i=1}^N X_i)^2 except in very specific circumstances, such as when Xi=1X_i = 1 for all observations.

Mathematical Expectation: Definitions and Properties

  • Definition of Expectation: The expectation is the value we expect a random variable to be. It essentially represents the average of the random variable if we were to sample from the variable's distribution an infinite number of times.

  • Example: Six-Sided Die: The expected value of a single roll of a six-sided die is 3.53.5. This is calculated because each side is expected to appear in an equal proportion of times over many trials. Formally, the expected value is calculated as: E[X]=∑i=16p(Xi)XiE[X] = \sum_{i=1}^6 p(X_i)X_i where XX is the set of outcomes {1,2,3,4,5,6}\{1, 2, 3, 4, 5, 6\} and p(Xi)p(X_i) is the probability of each outcome (p(Xi)=16p(X_i) = \frac{1}{6} for each value).

  • Expectation of a Constant Times a Function: The expectation of a constant kk multiplied by a function g(X)g(X) is equal to the constant multiplied by the expectation of the function: E[kg(X)]=kE[g(X)]E[kg(X)] = kE[g(X)]

  • Application: If wanting to know the expectation of 1010 times the number on a die, it is simply 1010 multiplied by the expectation of the die roll (10×3.5=3510 \times 3.5 = 35).

Variance: Measures of Distribution Spread

  • Definition: The variance of a random variable is a formal measure of how spread out the distribution is.

  • Large Sample Variance Formula: In large samples, variance is estimated as: var(X)=1N∑i=1N(Xi−Xˉ)2var(X) = \frac{1}{N} \sum_{i=1}^N (X_i - \bar{X})^2

  • Small Sample Variance Formula: In small samples, a degrees of freedom correction is applied, dividing by N−1N - 1 instead of NN. For large NN, the difference is negligible, and software programs typically handle this correction automatically.

  • De-construction of the Variance Equation: Calculating variance involves three simple math steps:

    1. Take the deviation from the mean for each observation (Xi−XˉX_i - \bar{X}).
    2. Square each deviation to ensure the values are positive.
    3. Calculate the average of these squared deviations.
  • Property 1 (Constant Addition): The variance of a constant kk plus a random variable ϵ\epsilon is simply the variance of the random variable: var(k+ϵ)=var(k)+var(ϵ)=0+var(ϵ)=σ2var(k + \epsilon) = var(k) + var(\epsilon) = 0 + var(\epsilon) = \sigma^2

  • Property 2 (Constant Multiplication): The variance of a random variable multiplied by a constant is the constant squared times the variance of the variable: var(kϵ)=k2var(ϵ)=k2σ2var(k\epsilon) = k^2 var(\epsilon) = k^2 \sigma^2

  • Property 3 (Correlated Random Variables): When random variables ϵ\epsilon and η\eta are correlated, the variance of their sum or difference involves their covariance:

    • var(ϵ+η)=var(ϵ)+var(η)+2cov(ϵ,η)var(\epsilon + \eta) = var(\epsilon) + var(\eta) + 2cov(\epsilon, \eta)
    • var(ϵ−η)=var(ϵ)+var(η)−2cov(ϵ,η)var(\epsilon - \eta) = var(\epsilon) + var(\eta) - 2cov(\epsilon, \eta)
  • Property 4 (Uncorrelated Random Variables): If two variables are uncorrelated, their covariance equals zero (cov(ϵ,η)=0cov(\epsilon, \eta) = 0), and the variance of their sum or difference is simply the sum of their individual variances: var(ϵ±η)=var(ϵ)+var(η)var(\epsilon \pm \eta) = var(\epsilon) + var(\eta)

Covariance: Measuring Relationship Between Variables

  • Definition: Covariance measures how much two random variables vary together.

  • Large Sample Covariance Formula: cov(X1,X2)=∑i=1N(X1i−Xˉ1)(X2i−Xˉ2)Ncov(X_1, X_2) = \frac{\sum_{i=1}^N (X_{1i} - \bar{X}_1)(X_{2i} - \bar{X}_2)}{N}

  • Property 1: The covariance of a random variable ϵ\epsilon and a constant kk is always zero: cov(ϵ,k)=0cov(\epsilon, k) = 0

  • Property 2: The covariance of a random variable ϵ\epsilon with itself is simply the variance of that variable: cov(ϵ,ϵ)=σϵ2cov(\epsilon, \epsilon) = \sigma^2_{\epsilon}

  • Property 3: The covariance of k1ϵk_1 \epsilon and k2ηk_2 \eta (where k1,k2k_1, k_2 are constants and ϵ,η\epsilon, \eta are variables) is: k1k2cov(ϵ,η)k_1 k_2 cov(\epsilon, \eta)

  • Property 4: Adding a constant to a random variable does not change its covariance with another variable: cov(ϵ,η+k)=cov(ϵ,η)cov(\epsilon, \eta + k) = cov(\epsilon, \eta)

Correlation and Its Relation to Regression

  • Correlation Formula: Correlation is the covariance of two variables divided by the product of their standard deviations: corr(X,Y)=cov(X,Y)σXσYcorr(X, Y) = \frac{cov(X, Y)}{\sigma_X \sigma_Y}

  • Correlation Bounds:

    • Perfect positive correlation (X=YX = Y) results in corr(X,Y)=+1corr(X, Y) = +1.
    • Perfect negative correlation (X=−YX = -Y) results in corr(X,Y)=−1corr(X, Y) = -1.
    • Variables are also perfectly correlated if they are identical once normalized (e.g., X=10YX = 10Y or X=5+3YX = 5 + 3Y).
  • Bivariate Regression Connection: The bivariate regression coefficient β^1,BivariateOLS\hat{\beta}_{1, \text{BivariateOLS}} is a re-standardized correlation: β^1=corr(X,Y)×σYσX\hat{\beta}_1 = corr(X, Y) \times \frac{\sigma_Y}{\sigma_X}

Probability Density Functions (PDFs)

  • Definition: A PDF is a mathematical function describing the relative probability for a continuous random variable to take on a specific probability.

  • Fundamental Features:

    • The values of a PDF (f(x)f(x)) are greater than or equal to zero for all possible values of xx.
    • The total area under the PDF curve equals one: ∫−∞∞f(x)dx=1\int_{-\infty}^{\infty} f(x)dx = 1
  • Continuous Nature: For a continuous variable, the probability of the variable equaling an exact decimal (e.g., 2.22.2) is effectively zero because there are infinite possible values near it. Consequently, probabilities must be calculated for regions/intervals of values using calculus.

  • Interval Calculation: The probability that a random variable xx is between aa and bb is the integral of the PDF from aa to bb: P(a<x<b)=∫abf(x)dx=F(b)−F(a)P(a < x < b) = \int_a^b f(x)dx = F(b) - F(a), where F()F() is the integral (Cumulative Distribution Function).

Normal and Standard Normal Distributions

  • Normal Distribution: A specific and famous type of PDF.

  • Standard Normal Distribution: A normal distribution with a mean (μ\mu) of zero and a variance (σ2\sigma^2) of one. The standard deviation (σ\sigma) is also one.

  • Cumulative Distribution Function (CDF): Denoted as Φ(z)=Prob(X<Z)\Phi(z) = Prob(X < Z). It indicates the probability of seeing a random variable less than the value ZZ, representing the area under the PDF curve to the left of ZZ.

  • Symmetry: The normal distribution is symmetric. Half of the area lies to the left of the mean (00), and half to the right.

  • Standard Normal Probabilities (Examples from Figure A.2 and A.3):

    • Φ(0)=0.500\Phi(0) = 0.500 (50%50\% chance of observed value being below 00)
    • Φ(−2)=0.023\Phi(-2) = 0.023 (2.3%2.3\% chance of observed value being below −2-2)
    • Φ(1.96)=0.975\Phi(1.96) = 0.975 (97.5%97.5\% chance of observed value being below 1.961.96)
    • Φ(1.0)=0.841\Phi(1.0) = 0.841 (84.1%84.1\% chance of observed value being below 1.01.0)
  • Calculating Probabilities Greater Than Z: Using the complement rule (1−P1 - P), the probability that a standard normal variable is greater than ZZ is calculated as 1−Φ(Z)1 - \Phi(Z). Examples:

    • 1−Φ(0)=0.5001 - \Phi(0) = 0.500
    • 1−Φ(−2)=0.9771 - \Phi(-2) = 0.977
    • 1−Φ(1.96)=0.0251 - \Phi(1.96) = 0.025
    • 1−Φ(1.0)=0.1591 - \Phi(1.0) = 0.159
  • Standard Deviations as Units: In a standard normal distribution, values can be interpreted as the number of standard deviations away from the mean. For instance, −2.32-2.32 is 2.322.32 standard deviations below the mean. The probability of observing a value less than −2.32-2.32 is 0.0100.010.

Standardizing Normally Distributed Random Variables

  • The Standardization Process: Any normally distributed random variable can be converted to the standard normal distribution using its mean and standard deviation.

  • Standardization Formula: To find how many standard deviations a value BB is away from the mean (μ1\mu_1) with standard error se(β^1)se(\hat{\beta}_1): Standard deviations from mean=B−μ1se(β^1)\text{Standard deviations from mean} = \frac{B - \mu_1}{se(\hat{\beta}_1)}

  • Standardized Values (Examples from Table A.1):

    • Example 1: B=3,μ=0,se=3→3−03=1B=3, \mu=0, se=3 \rightarrow \frac{3-0}{3} = 1 (Value is 11 SD above the mean).
    • Example 2: B=1,μ=0,se=3→1−03=0.33B=1, \mu=0, se=3 \rightarrow \frac{1-0}{3} = 0.33 (Value is 0.330.33 SD above the mean).
    • Example 3: B=7,μ=4,se=3→7−43=1B=7, \mu=4, se=3 \rightarrow \frac{7-4}{3} = 1 (Value is 11 SD above the mean).
    • Example 4: B=1,μ=4,se=3→1−43=−1B=1, \mu=4, se=3 \rightarrow \frac{1-4}{3} = -1 (Value is 11 SD below the mean).
    • Example 5: B=6,μ=8,se=2→6−82=−1B=6, \mu=8, se=2 \rightarrow \frac{6-8}{2} = -1 (Value is 11 SD below the mean).
    • Example 6: B=1,μ=8,se=2→1−82=−3.5B=1, \mu=8, se=2 \rightarrow \frac{1-8}{2} = -3.5 (Value is 3.53.5 SD below the mean).

Questions & Discussion

  • Question 1: What is the probability a standard normal random variable is less than or equal to 1.641.64?
  • Question 2: What is the probability a standard normal random variable is less than or equal to −1.28-1.28?
  • Question 3: What is the probability a standard normal random variable is greater than 1.281.28?
  • Question 4: What is the probability a normal random variable with a mean of zero and a standard deviation of 22 is less than −4-4?
  • Question 5: What is the probability a normal random variable with a mean of zero and a variance of 99 is less than −3-3?
  • Question 6: Approximately what is the probability a normal random variable with a mean of 7.27.2 and a variance of 44 is less than 99?

Supplemental Statistical Distributions

  • The χ2\chi^2 (Chi-squared) Distribution: Describes the distribution of squared normal variables. Specifically, the distribution of a squared standard normal random variable follows a χ2\chi^2 distribution.
  • Other Workhorse Distributions: Other common distributions derived from the normal distribution used in econometric practice include the tt and FF distributions.