Rao-Blackwell Theorem and Minimum-Variance Unbiased Estimator

Let $\hat{\theta}$ be an estimator of a parameter $\theta$ .
Desirable properties of $\hat{\theta}$ include:
1. Unbiasedness: The estimator should satisfy
 $E(\hat{\theta}) = \theta$ .
2. Consistency: As the sample size $n$ approaches infinity, the probability that the estimator deviates from the true parameter should diminish, formally,
 \lim_{n \to \infty} P(|\hat{\theta} - \theta| > \epsilon) = 0, \; \forall \epsilon > 0.
3. Efficiency: The efficiency of $\hat{\theta}$ relative to another unbiased estimator $\hat{\theta}2$ should be defined as $\text{eff}(\hat{\theta}, \hat{\theta}2) = \frac{V(\hat{\theta}2)}{V(\hat{\theta})} \geq 1 \; \forall \text{ unbiased estimator } \hat{\theta}2.$
An estimator satisfying these properties is known as the Minimum Variance Unbiased Estimator (MVUE).

Question: How to find an MVUE of $\theta$ ?
Answer: Let $X1, …, Xn$ be independent and identically distributed (i.i.d.) random variables from a probability density function (pdf) given by,
$f(x;\theta) = \exp{{g(\theta) + h(x)}}.$ If $\hat{\gamma} = G(\hat{\theta})$ and $\gamma(\theta)$ , then $\hat{\gamma}$ is the MVUE of $\gamma$ by Theorem 9.5 (The Rao-Blackwell Theorem).

Let $\hat{\theta}$ be an unbiased estimator of $\theta$ such that the variance V(\hat{\theta}) < \infty.
If $U$ is a sufficient statistic for $\theta$ , then define:
$\hat{\theta}^{*} = E(\hat{\theta}|U).$
The properties of this estimator are:
- For all $\theta$ ,
  $E(\hat{\theta}^{*}) = \theta$
- The variance is guaranteed to be less than or equal to that of any unbiased estimator:
  $V(\hat{\theta}^{*}) \leq V(\hat{\theta}).$

Definition 9.3: A statistic $U = g(Y1, …, Yn)$ is sufficient for $\theta$ if the conditional distribution of $Y1, …, Yn$ given $U$ does not depend on $\theta$ . This is formalized as:
$f{Y1,…,Yn|U}(y1, …, y_n | u)$ does not depend on $\theta$ .
Advantages of Sufficient Statistics:
1. Simplifies data for making inferences about $\theta$ .
2. Leads to the MVUE of $\theta$ or a function $W(\theta)$ .

Definition 9.4: For sample observations $y1, …, yn$ taken on corresponding random variables $Y1, …, Yn$ whose distribution depends on parameter $\theta$ , the likelihood of the sample is defined as:
$L(y1, …, yn | \theta) \equiv \begin{cases} \prod \text{ the joint probability of } y1, …, yn & \text{ if } Y \text{ is discrete} \ \text{the joint density of } y1, …, yn & \text{ if } Y \text{ is continuous random variable.} \end{cases}$
For simplicity, we may write:
$L(\theta) = L(y1, …, yn | \theta) = L(\hat{y} | \theta).$
Theorem 9.4: A statistic $U$ based on the random sample $Y1, …, Yn$ is sufficient for estimating $\theta$ if the likelihood function can be expressed in a factored form:
$L(\theta) = g(u, \theta) h(y1, …, yn)$ where $g(u, \theta)$ is a function only of $u$ and $\theta$ and $h()$ does not depend on $\theta$ .

Let $Y \sim Bin(m, p)$ . We check if $\hat{p} = \frac{Y}{m}$ is an MVUE of $p$ :
1. $Y$ is sufficient.
2. Compute:
- $E(\hat{p}) = E\left(\frac{Y}{m}\right) = \frac{1}{m}E(Y) = p$ .
Thus, $\hat{p}$ is the MVUE of $p$ .

Suppose $Y1, …, Yn$ i.i.d. from the distribution given by:
f = 2y \theta e^{-y^2 / \theta}, \, y > 0.
To find the MVUE of $\theta$ :
- Use a similar approach as in this section. For instance, let $n = 1$ and show that $Y^2$ is a sufficient statistic.
Calculate expectation as necessary.

Suppose $X1, …, Xn$ i.i.d. from $N(\mu, \sigma^2)$ :
- Formulate the likelihood as:
 $f(x) = \exp\left(T(x) \psi(p) + g(p) + h(x)\right)$ :
 $f = \frac{1}{\sqrt{2\pi \sigma^2}} e^{-\frac{(x-\mu)^2}{2\sigma^2}}.$
- Identify sufficient statistics and check for MVUEs by finding expectations.

The MVUE of $\sigma^2$ may involve estimates derived from sufficient statistics.
- There is a relationship between sample variance and unbiased estimators.

Understanding the Rao-Blackwell theorem greatly aids in the identification and calculation of MVUEs for various statistical models and distributions. The properties of sufficient statistics are pivotal in simplifying the process of estimator verification and validation.