STAT511 — Fall 2025 Lecture Notes

Chapter Overview

• Point Estimator and Point Estimate

  • Define point estimator and estimate
  • Unbiased estimators
  • Minimum variance estimators

• Finding the Standard Error of an Estimator

  • Deriving directly
  • Bootstrapping

• Methods of Point Estimation

  • The method of moments
  • The method of maximum likelihood

Motivation for Point Estimation

• Objective: To find the ratio $p$ of FIV (Feline Immunodeficiency Virus) infected cats in a specific area.
• Challenge: It's impossible to check all feral cats in the area.
• Proposed Method:
  1. Let $X = 1$ if a cat has FIV, and $X = 0$ if it does not.
  2. Define $p$ as the proportion of FIV-infected cats.
  3. The distribution of $X$: Bernoulli with unknown parameter $p$.
• Goal: Estimate the value of the parameter $p$ of a Bernoulli distribution.

Point Estimator and Point Estimate

• Definition:
  • A point estimate of a parameter $\theta$ is a single number that can be regarded as a sensible value for $\theta$.
  • A point estimate is obtained by selecting a suitable statistic and computing its value from given sample data.
• Key Distinctions:
  • A point estimate is a value; a point estimator is a statistic.
  • Denote point estimators of parameter $\theta$ usually by $\hat{\theta}$.
• Multiplicity:
  • Different statistics can be used to estimate the same parameter, meaning a parameter may have multiple point estimators.

Example: Breakdown Voltage for Epoxy Resin

• Scenario: Breakdown voltage follows a normal distribution.
• Task: Estimate the mean $\mu$ of the breakdown voltage using a sample.
• Data: Randomly check the breakdown voltages:
  {24.46, 25.61, 26.25, 26.42, 26.66, 27.15, 27.31, 27.54, 27.74, 27.94, 27.98, 28.04, 28.28, 28.49, 28.50, 28.87, 29.11, 29.13, 29.50, 30.88}.
• Possible Point Estimators:
  1. Sample Mean: $\hat{\mu} = \bar{X}$.
  2. Sample Median: $\hat{\mu} = \tilde{X}$.
  3. Average of Extremes: $\hat{\mu} = \frac{\min(X<em>i) + \max(X</em>i)}{2}$.
  4. Trimmed Mean: $\hat{\mu} = \bar{X}_{tr}(10)$ (10% trimmed mean).
  5. Other methods can also be used.

Example: Variance of Elastic Modulus of AZ91D Alloy

• Scenario: Estimate variance $\sigma^2$ but population distribution is unknown.
• Data: Sample of 8 elastic modulus observations:
  {44.2, 43.9, 44.7, 44.2, 44.0, 43.8, 44.6, 43.1}.
• Possible Point Estimators:
  1. Sample Variance: $\hat{\sigma}^2 = S^2 = \frac{\sum (X_i - \bar{X})^2}{n-1}$.
  2. Alternative estimators like $\hat{\sigma}^2 = \frac{\sum (X_i - \bar{X})^2}{n}$.

Choosing an Optimal Estimator

• A point estimator $\hat{\theta}$ is a random variable, with value varying based on sample drawn.
• Error of Estimation:
  • $\text{Error of Estimation} = \hat{\theta} - \theta$
  • Can be expressed as: $\hat{\theta} = \theta + \text{Error of Estimation}$.
• Criteria for Optimal Estimation:
  • Unbiased Estimator: the one that is closest to the true value of parameter $\theta$ on average.
  • Minimum Variance Estimator: an estimation error to be small.

Unbiased Estimators and Minimum Variance Estimators

• Unbiased Estimator:
  • Definition: A point estimator $\hat{\theta}$ is unbiased if $E(\hat{\theta}) = \theta$.
  • If $\hat{\theta}$ is biased, the difference $E(\hat{\theta}) - \theta$ is called the bias of $\hat{\theta}$.

Examples of Unbiased Estimators

• Sample Mean:
  • $\bar{X} = \frac{X<em>1 + X</em>2 + \cdots + X_n}{n}$ is an unbiased estimator of population mean $\mu$.
• Sample Median and Trimmed Mean:
  • Unbiased for continuous, symmetric distributions.
• Sample Variance:
  • $S^2 = \frac{\sum (X_i - \bar{X})^2}{n-1}$ is an unbiased estimator of population variance $\sigma^2$.

Exercises

1. For $X$ is a binomial$(n, p)$ random variable with $p$ unknown, is the estimator $\hat{p} = \frac{X}{n}$ unbiased?
2. For normal distribution with mean $\mu$ and variance $\sigma^2$, given a random sample, is sample mean $\bar{X} = \frac{X<em>1 + \cdots + X</em>n}{n}$ an unbiased estimator of $\mu$?
3. For any distribution, is sample mean $\bar{X}$ an unbiased estimator of population mean $\mu$?
4. Given a random sample $X<em>1, X</em>2, \cdots, X<em>n$ from a continuous uniform distribution with $f(x) = \frac{1}{\theta}$ defined on $[0, \theta]$, is $\hat{\theta} = 2\bar{X} = 2\left(\frac{X</em>1 + X<em>2 + \cdots + X</em>n}{n}\right)$ an unbiased estimator of $\theta$?

Answers to Exercises

1. Yes, $\hat{p} = \frac{X}{n}$ is an unbiased estimator.
   (Hint: Use the definition of an unbiased estimator to verify.)
2. Yes, $\bar{X}$ is an unbiased estimator of $\mu$.
   (Hint: Reference the Proposition in the text.)
3. Yes, the sample mean $\bar{X}$ is an unbiased estimator of the population mean.
   (Hint: Reference the Proposition in the text.)
4. Yes, check bias $= E(\hat{\theta}) - \theta$.

Principle of Unbiased Estimation

• Among different estimators for $\theta$, prefer the one that is unbiased.
• Example:
  • For a sample $X<em>1, X</em>2, X<em>3, \cdots, X</em>4$ from a normal distribution with unknown variance $\sigma^2$, two estimators can be considered to estimate $\sigma^2$:
  1. $\hat{\sigma}^1 = S^2 = \frac{\sum (X_i - \bar{X})^2}{4-1}$
  2. $\hat{\sigma}^2 = \frac{\sum (X_i - \bar{X})^2}{4}$
• Calculate:
  1. $E(\hat{\sigma}^1)$ and $E(\hat{\sigma}^2)$ to determine unbiasedness.
  2. Choose the estimator with minimum variance if both are unbiased.

Principle of Minimum Variance Unbiased Estimator (MVUE)

• Among all unbiased estimators of a parameter $\theta$, choose the one that has the minimum variance.
• Thus, the resulting estimator $\hat{\theta}$ is called the MVUE of $\theta$.
• Why Minimum Variance?
• Example:
  1. Estimator $\hat{p}<em>1 = \frac{X</em>1}{10}$ (sample size of 1).
  2. Estimator $\hat{p}<em>2 = \frac{X</em>1+X<em>2+\cdots+X</em>m}{10m}$ (sample size of $m$).
  • Both are unbiased but find preference based on variance.

MVUE for the Mean of Normal Distribution

• Theorem 2:
  • Let $X<em>1, X</em>2, \cdots, X_n$ be a random sample from a normal distribution with parameters $\mu$ and $\sigma$.
  • Then, the estimator $\hat{\mu} = \bar{X}$ is the MVUE for $\mu$.

Standard Error of an Estimator

• Definition:
  • The standard error of an estimator $\hat{\theta}$ is defined as its standard deviation $\sigma_{\hat{\theta}} = \sqrt{\text{Var}(\hat{\theta})}$.
• If the standard error involves unknown parameters, substituting estimators into $\sigma_{\hat{\theta}}$ gives the estimated standard error.
• Notation for Estimated Standard Error: Either $\hat{\sigma}<em>{\hat{\theta}}$ or $s</em>{\hat{\theta}}$.

How to Compute Standard Error

• Methods:
  • Derivative method to compute the standard deviation directly.
  • Employ computer-intensive methods such as bootstrapping.

Bootstrapping Example

• Given breakdown voltage for epoxy resin random sample
• Compute:
  1. Point estimate of mean $\mu$.
  2. Given known $\sigma = 1.5$, compute standard error.
  3. If $\sigma$ is unknown, estimate standard error.

Bootstrapping Method

1. Suppose population pdf is $f(x; \theta)$ and data is $x<em>1, x</em>2, \cdots, x_n$ yielding $\hat{\theta}$.
2. Generate bootstrap samples from $f(x, \hat{\theta})$.
3. For each sample, calculate bootstrap estimate $\hat{\theta}^*$.
4. Repeat process $B$ times, obtaining $\hat{\theta}^<em><em>1, \hat{\theta}^</em></em>2, \cdots, \hat{\theta}^*_B$.
5. Compute: Sample means and estimate standard error.

Example: Bootstrapping with Exponential Distribution

• Given Data: Sample times from breakdown X follows $Exponential(\lambda)$.
• Observed Sample: {41.53, 18.73, 2.99, 30.34, 12.33, 117.52, 73.02, 223.63, 4.00, 26.87}.
• Use $\hat{\lambda} = \frac{1}{\bar{X}} = 0.018153$ as an estimator for sample mean $\bar{X}$.
• Generate bootstrap samples data with distribution.

Methods of Point Estimation

• Method of Moments:
  • Definition of Moments:
  • Population Moment (Definition 3): For a distribution $X$, $k$-th moment: $E(X^k)$.
  • Sample Moment (Definition 4): Given random samples $X<em>1, X</em>2, \cdots, X<em>n$: $\frac{\sum</em>{i=1}^n X_i^k}{n}$.
  • Application: Sample moments can be used to estimate population moments.

Examples of Moments

• For Normal Distribution:
  • First moment: $E(X) = \mu$; Sample mean $\bar{X}$.
  • Second moment: $E(X^2) = \sigma^2 + \mu^2$; Sample $\frac{\sum X_i^2}{n}$.

The Method of Moments (Definition 5)

• For samples from distribution with parameters $\theta<em>1, \theta</em>2, …, \theta_m$, method involves solving equations from moments.

Example: Moment Estimator for Uniform Distribution

1. Given random samples of uniform distribution with known observed values.
2. Use sample means to set up equations for estimating parameters.

Maximum Likelihood Estimation (MLE) - Example

1. Sample of new bike helmets: $X<em>i = 1$ if flawed, $X</em>i = 0$ otherwise.
2. Use MLE to estimate $p = P(flawed helmet)$ given observed values of flaws.

Likelihood Function

• Definition 6: Likelihood function for observed sample values given parameters $θ<em>1, θ</em>2, …θ_m$.
• Objective: Find estimates $\hat{θ}<em>1, \hat{θ}</em>2, …$ that maximize the likelihood function.

Example: MLE with Normal Distribution

• Task: Estimate $\mu$ and $\sigma^2$ from normal distribution using observed values from the sample.
• Procedure: Set up likelihood function, derive logarithm, and find maximizing values through partial derivatives.

Estimating Functions of Parameters

• Invariance Principle: The MLE of a function of parameters is the function of MLEs of those parameters.

Large Sample Behavior of MLE

• Proposition: Under general conditions, as sample size increases, MLE is approximately unbiased and has minimal variance.

Summary

• Key Points:
  • Understanding point estimators and estimates, unbiasedness, MVUE, and standard error calculations are critical.
  • Two main methods of point estimation: Method of Moments and Maximum Likelihood Estimation.
  • MLE exhibits good asymptotic properties, making it a robust choice in statistical estimation.

Note: These lecture notes capture pivotal concepts from statistical estimation theory, including practical examples and exercises for enhanced comprehension.