Comprehensive Study Notes on Statistical Inference, Estimation, and Hypothesis Testing

Foundations of Statistical Inference, Parameter Space, and Sample Space

• Statistical Inference Overview (Dated 1/6/26):

  • Statistical inference is defined as the scientific process of drawing valid conclusions about the unknown probability distribution of a random variable $X$ on the basis of an observed sample.
  • Statistical inference is broadly bifurcated into two primary branches:
  • Testing of Hypotheses: Deals with the formulation and decision-making framework of either accepting or rejecting a stated statistical hypothesis regarding population characteristics based on sample evidence.
  • Estimation: Operates under the premise that the probability density function (pdf) or probability mass function (pmf) of the random variable $X$ possesses a known functional mathematical form, yet contains one or more unknown constants known as parameters. Estimation deals with determining suitable values or functions of sample observations to approximate these unknown parameters.

• Parameter Space (Dated 6/26):

  • If the parameters governing the distribution of a random variable $X$ are collectively denoted by $\theta$, the underlying probability density function is expressed functionally as $f(x; \theta)$.
  • Definition: The set of all admissible, mathematically valid values that the parameter $\theta$ can assume is formally designated as the parameter space.
  • The parameter space is universally symbolized by the capital Greek letter Omega ($\Omega$).
  • Any specific parameter value is an element of this space, written mathematically as $\theta \in \Omega$.
  • Canonical Examples of Parameter Spaces:
  • Exponential Distribution: For an exponential random variable $X$ defined by parameter $\theta$ with pdf $f(x; \theta) = \theta e^{-\theta x}$ for $x > 0$, the parameter must be strictly positive. Hence, the parameter space is $\Omega = \{\theta : \theta > 0\} = (0, \infty)$.
  • Normal Distribution: For a normally distributed random variable $X \sim N(\mu, \sigma^2)$ governed by mean $\mu$ and variance $\sigma^2$, the parameter vector is $\theta = (\mu, \sigma^2)$. The mean can take any real value while the variance must be strictly positive, defining the parameter space as $\Omega = \{(\mu, \sigma^2) : -\infty < \mu < \infty, \sigma^2 > 0\}$.
  • Poisson Distribution: For a Poisson random variable $X \sim P(\lambda)$, the rate parameter $\lambda$ must be strictly positive, yielding the parameter space $\Omega = \{\lambda : \lambda > 0\}$.

• Sample Space and Random Samples:

  • If $X$ represents a random variable and $x_1$ denotes an observed realization of $X$, theoretically $x_1$ is itself a random variable distributed identically to $X$.
  • If a sequence of $n$ observations $(x_1, x_2, \dots, x_n)$ is collected independently on $X$, then $(x_1, x_2, \dots, x_n)$ constitutes a collection of $n$ independent and identically distributed (IID) random variables.
  • In the formal discipline of statistical inference, the technical term random sample strictly denotes a collection of $n$ IID random variables.
  • Definition: The set encompassing all possible realization vectors of $\mathbf{x} = (x_1, x_2, \dots, x_n)$ across the $n$-dimensional Euclidean product space is termed the sample space, formally denoted by $S$ or $\mathcal{X}$ ($n$-cross product space).

Point Estimation and Criteria of a Good Estimator

• Point Estimation Framework (Dated 5/6/26):

  • Consider a random variable $X$ characterized by a probability density function $f(x; \theta)$, where $\theta \in \Omega$.
  • Let $g(\theta)$ represent a real-valued function of the parameter $\theta$ defined over the domain $\Omega$, whose exact numerical value is targeted for estimation (in many standard cases, $g(\theta)$ simplifies to $\theta$ itself).
  • In point estimation, a single calculated numerical value is proposed as the optimal approximation of the unknown parameter using sample observations drawn randomly from the parent population.

• Distinction Between Estimator, Statistic, and Estimate:

  • Estimator: Let $(x_1, x_2, \dots, x_n)$ denote $n$ independent observations on $X$. Any mathematical function $t(x_1, x_2, \dots, x_n) = t(\mathbf{x})$ that systematically maps the sample space ($S$) into the parameter space ($\Omega$) is defined as an estimator. An estimator is a rule, formula, or random variable.
  • Statistic: Any function of the sample observations $(x_1, x_2, \dots, x_n)$ that contains no unknown parameters is designated as a statistic (e.g., sample mean $\bar{x}$, sample variance $s^2$).
  • Estimate: The specific numerical evaluation produced by the estimator function $t(\mathbf{x})$ when calculated for a concrete, realized numerical sample vector $\mathbf{x} = (x_1, x_2, \dots, x_n)$ is called the estimate.
  • Exemplary Case: When estimating the unknown population mean $\mu$, the sample mean formula $\bar{x} = \frac{1}{n}\sum_{i=1}^n x_i$ functions as the estimator ($\hat{\mu} = \bar{x}$). When evaluated on a concrete sample dataset, the resulting numerical figure is the estimate of $\mu$.

• Criteria of a Good Estimator (R. A. Fisher's Criteria):

  • For any given parameter $\theta$, an arbitrary number of distinct candidate estimators can be constructed. For instance, in a Poisson population with parameter $\lambda$, candidate estimators include the sample mean, the sample median, and the sample variance.
  • To systematically evaluate and rank estimator performance, Sir R. A. Fisher formulated four fundamental criteria that an optimal estimator should fulfill:
  1. Unbiasedness
  2. Consistency
  3. Efficiency
  4. Sufficiency
  • Complementary mathematical criteria utilized in advanced estimation include Minimum Variance, Minimum Mean Square Error (MSE), and Robustness.

Unbiasedness and Estimable Functions

• Mathematical Formulation of Unbiasedness:

  • Let $X$ be a random variable with density $f(x; \theta)$, where $\theta \in \Omega$, and let $(x_1, x_2, \dots, x_n)$ be an IID random sample of size $n$.
  • An estimator $t = t(\mathbf{x})$ is defined to be an unbiased estimator of the parameter $\theta$ if its mathematical expectation is identically equal to the true parameter value across the entire parameter space:     $E(t(\mathbf{x})) = \theta, \quad \forall \theta \in \Omega$
  • Similarly, an estimator $t$ is an unbiased estimator of a parametric function $g(\theta)$ if and only if:     $E(t) = g(\theta), \quad \forall \theta \in \Omega$
  • Estimable Function: Any parametric function $g(\theta)$ for which there exists at least one estimator $t$ satisfying $E(t) = g(\theta)$ is formally designated as an estimable function.

• Bias and Asymptotic Unbiasedness:

  • Any estimator that fails to satisfy the condition of unbiasedness is termed a biased estimator.
  • If $E(t) = g(\theta) + b(\theta)$, the non-zero quantity $b(\theta) = E(t) - g(\theta)$ is explicitly defined as the bias of the estimator $t$.
  • Asymptotic Unbiasedness: If an estimator $t$ is biased for finite sample sizes $n$, but satisfies the expectation condition in the asymptotic limit as $n$ approaches infinity:     $\lim_{n \to \infty} E(t) = g(\theta)$     then $t$ is designated as an asymptotically unbiased estimator.

• Non-Uniqueness of Unbiased Estimators:

  • An unbiased estimator for an unknown population parameter is not necessarily unique.
  • Proof by Construction: Let $x_1, x_2$ be two observations from a population with mean $\mu$. Consider the estimator $t = \frac{x_1 + x_2}{2}$. Its expected value is:     $E(t) = E\left(\frac{x_1 + x_2}{2}\right) = \frac{1}{2}[E(x_1) + E(x_2)] = \frac{1}{2}(\mu + \mu) = \frac{1}{2}(2\mu) = \mu$
  • Simultaneously, the full sample mean $\bar{x} = \frac{1}{n}\sum_{i=1}^n x_i$ also satisfies $E(\bar{x}) = \mu$, establishing that multiple distinct unbiased estimators can coexist for the identical parameter.

• Proofs and Solved Problems on Unbiasedness:

  • Problem 1: Unbiasedness of Sample Mean for Population Mean:

  • Proposition: Let $x_1, x_2, \dots, x_n$ be a random sample of size $n$ drawn from a population with mean $\mu$. Prove that the sample mean $\bar{x}$ is an unbiased estimator of $\mu$.

  • Proof: Given that $E(x_i) = \mu$ for all $i = 1, 2, \dots, n$, the sample mean is defined as:       $\bar{x} = \frac{1}{n}\sum_{i=1}^n x_i = \frac{x_1 + x_2 + \dots + x_n}{n}$

  • Taking mathematical expectations on both sides:       $E(\bar{x}) = E\left(\frac{x_1 + x_2 + \dots + x_n}{n}\right) = \frac{1}{n}[E(x_1) + E(x_2) + \dots + E(x_n)]$$E(\bar{x}) = \frac{1}{n}[\mu + \mu + \dots + \mu] = \frac{1}{n}(n\mu) = \mu$

  • Because $E(\bar{x}) = \mu$, $\bar{x}$ is rigorously an unbiased estimator of $\mu$.

  • Problem 2: Bias of Sample Variance in Normal Populations (Dated 8/6/26):

  • Proposition: For a random sample taken from $N(\mu, \sigma^2)$, demonstrate that the sample variance $s^2 = \frac{1}{n}\sum_{i=1}^n (x_i - \bar{x})^2$ is a biased estimator of the population variance $\sigma^2$, and deduce an unbiased estimator.

  • Proof: By distribution theory, the normalized sample variance statistic follows a chi-square distribution with $n-1$ degrees of freedom:       $\frac{n s^2}{\sigma^2} \sim \chi^2_{(n-1)}$

  • For a standard chi-square variable $W \sim \chi^2_{(k)}$, the mean is $E(W) = k$ and variance is $V(W) = 2k$. Setting $k = n - 1$:       $E\left(\frac{n s^2}{\sigma^2}\right) = n - 1$

  • Utilizing linearity of expectation:       $\frac{n}{\sigma^2} E(s^2) = n - 1 \implies E(s^2) = \frac{n-1}{n}\sigma^2$

  • Because $E(s^2) = \frac{n-1}{n}\sigma^2 \neq \sigma^2$, $s^2$ is a biased estimator of $\sigma^2$, with bias $b(\sigma^2) = -\frac{\sigma^2}{n}$.

  • To construct an unbiased estimator, multiply $s^2$ by the reciprocal factor $\frac{n}{n-1}$, yielding the sample variance definition $S^2$:     $E\left(\frac{n}{n-1}s^2\right) = \frac{n}{n-1}E(s^2) = \frac{n}{n-1}\left(\frac{n-1}{n}\sigma^2\right) = \sigma^2$

  • Thus, $S^2 = \frac{n}{n-1}s^2 = \frac{1}{n-1}\sum_{i=1}^n (x_i - \bar{x})^2$ is an unbiased estimator of $\sigma^2$.

• Problem 3: Unbiased Estimator of $\mu^2 + 1$ in $N(\mu, 1)$:

  • Proposition: Let $x_1, x_2, \dots, x_n$ be a random sample from $N(\mu, 1)$. Show that $t = \frac{1}{n}\sum_{i=1}^n x_i^2$ is an unbiased estimator of $\mu^2 + 1$.
  • Proof: For each observation, $E(x_i) = \mu$ and $V(x_i) = 1$. By the variance expansion identity $V(x_i) = E(x_i^2) - [E(x_i)]^2$:     $E(x_i^2) = V(x_i) + [E(x_i)]^2 = 1 + \mu^2$
  • Taking expectation of $t$:     $E(t) = E\left(\frac{1}{n}\sum_{i=1}^n x_i^2\right) = \frac{1}{n}\sum_{i=1}^n E(x_i^2) = \frac{1}{n}[n(1 + \mu^2)] = \mu^2 + 1$
  • Hence, $t$ is an unbiased estimator of $\mu^2 + 1$.

• Problem 4: Unbiased Estimator of $p^2$ in Bernoulli Populations:

  • Proposition: Let $x_1, x_2, \dots, x_n$ be drawn from a Bernoulli distribution with parameter $p$, where $x_i \sim B(1, p)$. Show that $\frac{T(T-1)}{n(n-1)}$ is an unbiased estimator of $p^2$, where $T = \sum_{i=1}^n x_i$.
  • Proof: Since the sum of independent Bernoulli variables is binomial, $T = \sum_{i=1}^n x_i \sim B(n, p)$. The moments are:     $E(T) = np, \quad V(T) = np(1-p)$$E(T^2) = V(T) + [E(T)]^2 = np(1-p) + n^2 p^2$
  • Consider the expectation of the product $T(T-1) = T^2 - T$:     $E[T(T-1)] = E(T^2) - E(T) = [np(1-p) + n^2 p^2] - np$$E[T(T-1)] = np - np^2 + n^2 p^2 - np = n^2 p^2 - np^2 = np^2(n-1)$
  • Dividing by the constant denominator $n(n-1)$:     $E\left[\frac{T(T-1)}{n(n-1)}\right] = \frac{np^2(n-1)}{n(n-1)} = p^2$
  • Consequently, $\frac{T(T-1)}{n(n-1)}$ is an unbiased estimator of $p^2$.

• Problem 5: Non-linear Transformations of Unbiased Estimators:

  • Proposition: If $T$ is an unbiased estimator of $\theta$ ($E(T) = \theta$), prove that $T^2$ and $\sqrt{T}$ are biased estimators of $\theta^2$ and $\sqrt{\theta}$, respectively.
  • Proof for $T^2$: Since $T$ is a non-degenerate random variable, its variance is strictly positive:     $V(T) = E\left([T - E(T)]^2\right) = E(T^2) - [E(T)]^2 > 0$$E(T^2) - \theta^2 > 0 \implies E(T^2) > \theta^2$
  • Because $E(T^2) \neq \theta^2$, $T^2$ is positively biased for $\theta^2$.

• Proof for $\sqrt{T}$: Similarly, the variance of $\sqrt{T}$ is strictly positive:   $V(\sqrt{T}) = E\left([\sqrt{T} - E(\sqrt{T})]^2\right) = E\left[(\sqrt{T})^2\right] - [E(\sqrt{T})]^2 > 0$$E(T) - [E(\sqrt{T})]^2 > 0 \implies \theta - [E(\sqrt{T})]^2 > 0$$[E(\sqrt{T})]^2 < \theta \implies E(\sqrt{T}) < \sqrt{\theta}$

• Because $E(\sqrt{T}) \neq \sqrt{\theta}$, $\sqrt{T}$ is a biased estimator of $\sqrt{\theta}$.

• Problem 6: Estimators for Exponential Distribution:

  • Let $x_1, x_2, \dots, x_n$ be an IID random sample from an exponential distribution with density $f(x; \theta) = \frac{1}{\theta}e^{-x/\theta}$ for $x > 0$, where $E(x) = \theta$ and $V(x) = \theta^2$.
  • For $\theta$: The sample mean is $\bar{x} = \frac{1}{n}\sum x_i$. Evaluating expectation:     $E(\bar{x}) = \frac{1}{n}\sum_{i=1}^n E(x_i) = \frac{1}{n}(n\theta) = \theta$     Thus, $\bar{x}$ is an unbiased estimator of $\theta$.
  • For $\theta^2$: The variance of $\bar{x}$ is $V(\bar{x}) = \frac{V(x)}{n} = \frac{\theta^2}{n}$. By variance expansion:     $E(\bar{x}^2) = V(\bar{x}) + [E(\bar{x})]^2 = \frac{\theta^2}{n} + \theta^2 = \theta^2\left(\frac{n+1}{n}\right)$     Multiplying both sides by $\frac{n}{n+1}$:     $E\left(\frac{n}{n+1}\bar{x}^2\right) = \theta^2$     Thus, $\frac{n}{n+1}\bar{x}^2$ is an unbiased estimator of $\theta^2$.

• Problem 7: Geometric Distribution:

  • Proposition: Prove that the sample mean $\bar{x}$ is an unbiased estimator of $\frac{1}{\theta}$ for a geometric distribution with pmf $f(x; \theta) = \theta(1-\theta)^{x-1}$ for $x = 1, 2, \dots$ ($0 < \theta < 1$).
  • Proof: The theoretical population mean of this geometric distribution is $E(x) = \frac{1}{\theta}$.     $E(\bar{x}) = E\left(\frac{1}{n}\sum_{i=1}^n x_i\right) = \frac{1}{n}\sum_{i=1}^n E(x_i) = \frac{1}{n}\left(n \cdot \frac{1}{\theta}\right) = \frac{1}{\theta}$     The expected value matches the target parameter identically, confirming $\bar{x}$ is an unbiased estimator of $\frac{1}{\theta}$.

• Problem 8: Three Unbiased Estimators of Poisson Parameter $\lambda$:

  • For a random sample $x_1, x_2, \dots, x_n$ drawn from $P(\lambda)$, where $E(x_i) = \lambda$ and $V(x_i) = \lambda$:
  1. Sample Mean: $t_1 = \bar{x} = \frac{1}{n}\sum_{i=1}^n x_i$, with $E(t_1) = \lambda$.
  2. Single Observation: $t_2 = x_1$ (or any single observation, or linear pairs such as $t = \frac{x_1 + x_2}{2}$), with $E(t_2) = E(x_1) = \lambda$.
  3. Sample Variance: $t_3 = S^2 = \frac{1}{n-1}\sum_{i=1}^n (x_i - \bar{x})^2$. Since for any distribution $E(S^2) = \sigma^2$, and in Poisson $\sigma^2 = \lambda$, it follows directly that $E(S^2) = \lambda$.

• Problem 9: Uniform Distribution over $(0, \theta)$:

  • For a continuous rectangular distribution over $(0, \theta)$, the $n$-th order statistic is $X_{(n)} = \max(x_1, x_2, \dots, x_n)$.
  • The expected value of the maximum order statistic is $E(X_{(n)}) = \frac{n}{n+1}\theta$.
  • Adjusting for bias yields the uniformly minimum-variance unbiased estimator:     $\hat{\theta} = \frac{n+1}{n}X_{(n)}$

• Problem 10: UMVUE of $e^{-\lambda}$ in Poisson Populations:

  • Let $x_1, x_2, \dots, x_n$ be IID from $P(\lambda)$. Find an unbiased estimator for $g(\lambda) = e^{-\lambda}$.
  • Step 1 (Sufficient Statistic): By the additive property of independent Poisson variables, the sum $Y = \sum_{i=1}^n x_i$ is sufficient for $\lambda$ and follows a Poisson distribution with parameter $n\lambda$:     $P(Y = y) = \frac{e^{-n\lambda}(n\lambda)^y}{y!}, \quad y = 0, 1, 2, \dots$
  • Step 2 (Unbiased Condition): Let $g(Y)$ be the required estimator satisfying $E[g(Y)] = e^{-\lambda}$:     $\sum_{y=0}^\infty g(y) \frac{e^{-n\lambda}(n\lambda)^y}{y!} = e^{-\lambda}$
  • Step 3 (Solving the Functional Equation): Factor out $e^{-n\lambda}$:     $\sum_{y=0}^\infty g(y) \frac{n^y \lambda^y}{y!} = \frac{e^{-\lambda}}{e^{-n\lambda}} = e^{(n-1)\lambda}$
  • Expanding the right-hand side via its Maclaurin Taylor series:     $e^{(n-1)\lambda} = \sum_{y=0}^\infty \frac{[(n-1)\lambda]^y}{y!} = \sum_{y=0}^\infty \frac{(n-1)^y \lambda^y}{y!}$
  • Equating the coefficients of $\lambda^y$ term-by-term on both sides:     $g(y)\frac{n^y}{y!} = \frac{(n-1)^y}{y!} \implies g(y) = \frac{(n-1)^y}{n^y} = \left(1 - \frac{1}{n}\right)^y$
  • Substituting $Y = \sum_{i=1}^n x_i$, the unique Uniformly Minimum Variance Unbiased Estimator (UMVUE) is:     $g(Y) = \left(1 - \frac{1}{n}\right)^{\sum_{i=1}^n x_i} = \left(\frac{n-1}{n}\right)^{\sum_{i=1}^n x_i}$

Consistency of Estimators

• Mathematical Formulation of Consistency (Dated 10/6):

  • Let $x_1, x_2, \dots, x_n$ represent an IID random sample taken from a population characterized by an unknown parameter $\theta$.
  • Let $t_n = t(x_1, x_2, \dots, x_n)$ be an estimator constructed from a sample of size $n$.
  • Definition: The estimator sequence $t_n$ is defined to be a consistent estimator of $\theta$ if for every arbitrary positive threshold $\varepsilon > 0$:     $\lim_{n \to \infty} P(|t_n - \theta| < \varepsilon) = 1$     or equivalently expressed via the probability of deviation:     $\lim_{n \to \infty} P(|t_n - \theta| \ge \varepsilon) = 0$
  • Consistency signifies convergence in probability, symbolized formally as $t_n \xrightarrow{P} \theta$.
  • Consistency is strictly a large-sample asymptotic property. It guarantees that as the sample size $n$ expands indefinitely, the probability distribution of $t_n$ concentrates entirely in an arbitrarily tight neighborhood around the true parameter $\theta$.
  • If at least one consistent estimator exists for a parameter $\theta$, an infinite family of consistent estimators can be constructed.

• Sufficient Condition for Consistency:

  • A general and widely applicable sufficient condition states that an estimator $t_n$ is consistent for $\theta$ if both of the following asymptotic limits hold simultaneously as $n \to \infty$:
  1. $\lim_{n \to \infty} E(t_n) = \theta$
  2. $\lim_{n \to \infty} V(t_n) = 0$

• Solved Problems on Consistency:

  • Problem 1: Consistency of Sample Mean for Normal Mean:

  • Proposition: For a random sample $x_1, x_2, \dots, x_n$ from $N(\mu, \sigma^2)$, prove that $\bar{x}$ is a consistent estimator of $\mu$.

  • Proof: The sampling moments of $\bar{x}$ are:       $E(\bar{x}) = \mu$$V(\bar{x}) = V\left(\frac{1}{n}\sum_{i=1}^n x_i\right) = \frac{1}{n^2}\sum_{i=1}^n V(x_i) = \frac{1}{n^2}(n\sigma^2) = \frac{\sigma^2}{n}$

  • Taking limits as $n \to \infty$:       $\lim_{n \to \infty} E(\bar{x}) = \mu$$\lim_{n \to \infty} V(\bar{x}) = \lim_{n \to \infty} \frac{\sigma^2}{n} = 0$

  • Both criteria of the sufficient condition are satisfied; hence, $\bar{x}$ is a consistent estimator of $\mu$.

  • Problem 2: Consistency of Adjusted Estimator in Poisson Populations:

  • Proposition: For a random sample from $P(\lambda)$, show that $t_n = \frac{n\bar{x}}{n+1}$ is a consistent estimator of $\lambda$.

  • Proof: The expectation of the estimator is:       $E(t_n) = E\left(\frac{n\bar{x}}{n+1}\right) = \frac{n}{n+1}E(\bar{x}) = \frac{n}{n+1}\lambda = \frac{1}{1 + \frac{1}{n}}\lambda$$\lim_{n \to \infty} E(t_n) = \lim_{n \to \infty} \frac{1}{1 + \frac{1}{n}}\lambda = \frac{1}{1 + 0}\lambda = \lambda$

  • The variance of the estimator is:       $V(t_n) = V\left(\frac{n\bar{x}}{n+1}\right) = \left(\frac{n}{n+1}\right)^2 V(\bar{x}) = \frac{n^2}{(n+1)^2}\left(\frac{\lambda}{n}\right) = \frac{n\lambda}{(n+1)^2}$$V(t_n) = \frac{n\lambda}{n^2\left(1 + \frac{1}{n}\right)^2} = \frac{\lambda}{n\left(1 + \frac{1}{n}\right)^2}$$\lim_{n \to \infty} V(t_n) = \lim_{n \to \infty} \frac{\lambda}{n\left(1 + \frac{1}{n}\right)^2} = 0$

  • Since $E(t_n) \to \lambda$ and $V(t_n) \to 0$, $t_n = \frac{n\bar{x}}{n+1}$ is a consistent estimator of $\lambda$.

  • Problem 3: Consistency of Sample Variance for Normal Variance:

  • Proposition: Prove that the sample variance $s^2 = \frac{1}{n}\sum_{i=1}^n (x_i - \bar{x})^2$ is a consistent estimator of $\sigma^2$ in $N(\mu, \sigma^2)$.

  • Proof: Recall that $\frac{n s^2}{\sigma^2} \sim \chi^2_{(n-1)}$. Therefore:       $E\left(\frac{n s^2}{\sigma^2}\right) = n - 1 \implies E(s^2) = \frac{n-1}{n}\sigma^2 = \left(1 - \frac{1}{n}\right)\sigma^2$$\lim_{n \to \infty} E(s^2) = \lim_{n \to \infty} \left(1 - \frac{1}{n}\right)\sigma^2 = \sigma^2$

  • Applying the property $V(aX) = a^2 V(X)$, the variance of the chi-square statistic is:       $V\left(\frac{n}{\sigma^2} s^2\right) = \frac{n^2}{\sigma^4} V(s^2) = 2(n-1)$$V(s^2) = \frac{2(n-1)\sigma^4}{n^2} = \frac{2\left(1 - \frac{1}{n}\right)\sigma^4}{n}$$\lim_{n \to \infty} V(s^2) = \lim_{n \to \infty} \frac{2\left(1 - \frac{1}{n}\right)\sigma^4}{n} = 0$

  • Because $E(s^2) \to \sigma^2$ and $V(s^2) \to 0$, $s^2$ is a consistent estimator of $\sigma^2$.

Efficiency and Relative Efficiency

• Concept and Definition of Efficiency:

  • Within the class of all possible unbiased estimators for a given population parameter $\theta$, efficiency serves as the metric to rank estimators based on their dispersion (variance).
  • Let $t_1$ and $t_2$ be two unbiased estimators of the parameter $\theta$ such that $E(t_1) = E(t_2) = \theta$.
  • Definition: Estimator $t_1$ is defined to be more efficient than $t_2$ if and only if:     $V(t_1) < V(t_2)$
  • As the sampling variance of an unbiased estimator decreases, its statistical efficiency monotonically increases. The estimator that achieves the theoretical minimum variance within the unbiased class is designated as the most efficient estimator.

• Relative Efficiency (RE):

  • Let $t$ represent the most efficient estimator for the parameter $\theta$. The relative efficiency of any other competing unbiased estimator $t_1$ with respect to $t$ is defined as the ratio of their precisions (inverse variances):     $\text{RE}(t_1) = \frac{1/V(t_1)}{1/V(t)} = \frac{V(t)}{V(t_1)}$

• Comparative Problem on Estimator Efficiencies:

  • Problem: For a random sample $x_1, x_2, \dots, x_n$ drawn from $N(\mu, \sigma^2)$, evaluate the unbiasedness and compare the relative efficiencies of the following proposed estimators for $\mu$:     $t_1 = x_1, \quad t_2 = \frac{x_1 + x_2}{2}, \quad t_n = \bar{x} = \frac{x_1 + x_2 + \dots + x_n}{n}$
  • Evaluation of Unbiasedness:$E(t_1) = E(x_1) = \mu$$E(t_2) = E\left(\frac{x_1 + x_2}{2}\right) = \frac{1}{2}[E(x_1) + E(x_2)] = \frac{1}{2}(\mu + \mu) = \mu$$E(t_n) = E\left(\frac{1}{n}\sum_{i=1}^n x_i\right) = \frac{1}{n}(n\mu) = \mu$     All three estimators are strictly unbiased for $\mu$.
  • Evaluation of Variances:$V(t_1) = V(x_1) = \sigma^2$$V(t_2) = V\left(\frac{x_1 + x_2}{2}\right) = \frac{1}{4}[V(x_1) + V(x_2)] = \frac{1}{4}[\sigma^2 + \sigma^2] = \frac{2\sigma^2}{4} = \frac{\sigma^2}{2}$$V(t_n) = V\left(\frac{1}{n}\sum_{i=1}^n x_i\right) = \frac{1}{n^2}[n\sigma^2] = \frac{\sigma^2}{n}$
  • Comparison:
  • Because $n \ge 2$, the variances satisfy the strict inequality chain:       $\sigma^2 > \frac{\sigma^2}{2} > \dots > \frac{\sigma^2}{n} \iff V(t_1) > V(t_2) > \dots > V(t_n)$
  • Consequently, $t_n = \bar{x}$ has the smallest variance and is strictly more efficient than $t_1, t_2, \dots, t_{n-1}$.

Sufficiency and the Fisher-Neyman Factorization Theorem

• Concept of Sufficiency (Dated 17/6):

  • Formulated by Sir R. A. Fisher, the concept of sufficiency holds central importance in statistical inference.
  • Definition: An estimator (or statistic) $t$ is defined to be sufficient for a parameter $\theta$ if it captures all the information regarding $\theta$ that is contained within the sample.
  • Once the numerical value of a sufficient statistic is known, no other statistic calculated from the same sample can provide any additional information concerning the parameter $\theta$.

• The Likelihood Function:

  • Let $\mathbf{x} = (x_1, x_2, \dots, x_n)$ be an IID random sample from a population with pdf or pmf $f(x; \theta)$, where $\theta \in \Omega$.
  • The joint density of the sample, viewed functionally as a function of the parameter $\theta$ for fixed sample data $\mathbf{x}$, is defined as the likelihood function $L(\mathbf{x}; \theta)$:     $L(\mathbf{x}; \theta) = \prod_{i=1}^n f(x_i; \theta) = f(x_1; \theta) \times f(x_2; \theta) \times \dots \times f(x_n; \theta)$

• Fisher-Neyman Factorization Theorem:

  • Statement: A statistic $t = t(\mathbf{x})$ is sufficient for the parameter $\theta$ if and only if the joint likelihood function $L(\mathbf{x}; \theta)$ can be factored into a product of two distinct non-negative functions:     $L(\mathbf{x}; \theta) = g(t(\mathbf{x}); \theta) \times h(\mathbf{x})$     where:
  • $g(t(\mathbf{x}); \theta)$ is a non-negative function that depends on the sample data only through the statistic $t(\mathbf{x})$, and explicitly involves the parameter $\theta$.
  • $h(\mathbf{x})$ is a non-negative function of the sample observations alone, and is completely independent of the parameter $\theta$.
  • The theorem applies directly when both $t$ and $\theta$ are multidimensional vectors.

• Solved Factorization Problems:

  • Problem 1: Exponential Distribution:

  • Let $x_1, x_2, \dots, x_n$ be drawn from $f(x; \theta) = \theta e^{-\theta x}$ for $x > 0, \theta > 0$.

  • The likelihood function is:       $L(\mathbf{x}; \theta) = \prod_{i=1}^n \left(\theta e^{-\theta x_i}\right) = \theta^n e^{-\theta \sum_{i=1}^n x_i}$

  • Factoring into the Fisher-Neyman structure:       $L(\mathbf{x}; \theta) = \left[\theta^n e^{-\theta \sum_{i=1}^n x_i}\right] \times [1]$

  • Here, $g(t; \theta) = \theta^n e^{-\theta t}$ with $t = \sum_{i=1}^n x_i$, and $h(\mathbf{x}) = 1$.

  • Thus, $t = \sum_{i=1}^n x_i$ is a sufficient statistic for $\theta$.

  • Problem 2: Poisson Distribution:

  • Let $x_1, x_2, \dots, x_n$ be drawn from $f(x; \lambda) = \frac{e^{-\lambda}\lambda^x}{x!}$ for $x = 0, 1, 2, \dots$.

  • The likelihood function is:       $L(\mathbf{x}; \lambda) = \prod_{i=1}^n \frac{e^{-\lambda}\lambda^{x_i}}{x_i!} = \frac{e^{-n\lambda}\lambda^{\sum_{i=1}^n x_i}}{\prod_{i=1}^n x_i!}$

  • Factoring:       $g(t; \lambda) = e^{-n\lambda}\lambda^{\sum_{i=1}^n x_i}, \quad h(\mathbf{x}) = \frac{1}{\prod_{i=1}^n x_i!}$

  • Therefore, $t = \sum_{i=1}^n x_i$ is a sufficient statistic for $\lambda$.

  • Problem 3: Power Function Distribution (Dated 22/6):

  • Let $f(x; \theta) = \theta x^{\theta - 1}$ for $0 < x < 1$ and $\theta > 0$.

  • The likelihood function is:       $L(\mathbf{x}; \theta) = \prod_{i=1}^n \left(\theta x_i^{\theta - 1}\right) = \theta^n \left(\prod_{i=1}^n x_i\right)^{\theta - 1} = \theta^n \frac{\left(\prod_{i=1}^n x_i\right)^\theta}{\prod_{i=1}^n x_i}$

  • Decomposing the terms:       $g(t; \theta) = \theta^n \left(\prod_{i=1}^n x_i\right)^\theta, \quad h(\mathbf{x}) = \frac{1}{\prod_{i=1}^n x_i}$

  • By the factorization theorem, $t = \prod_{i=1}^n x_i$ is a sufficient statistic for $\theta$.

  • Problem 4: Normal Distribution (Comprehensive Case Study):

  • Let $x_1, x_2, \dots, x_n$ be drawn from $N(\mu, \sigma^2)$. The density is:       $f(x_i) = \frac{1}{\sqrt{2\pi}\sigma} e^{-\frac{1}{2\sigma^2}(x_i - \mu)^2}$

  • The likelihood function is:       $L(\mathbf{x}; \mu, \sigma^2) = \left(\frac{1}{\sqrt{2\pi}\sigma}\right)^n e^{-\frac{1}{2\sigma^2}\sum_{i=1}^n (x_i - \mu)^2}$

  • Algebraic Identity for Exponent: Decompose the sum of squared deviations by adding and subtracting $\bar{x}$:       $\sum_{i=1}^n (x_i - \mu)^2 = \sum_{i=1}^n [(x_i - \bar{x}) + (\bar{x} - \mu)]^2$$\sum_{i=1}^n (x_i - \mu)^2 = \sum_{i=1}^n (x_i - \bar{x})^2 + n(\bar{x} - \mu)^2 + 2(\bar{x} - \mu)\sum_{i=1}^n (x_i - \bar{x})$

  • Because the sum of deviations from the sample mean is identically zero ($\sum_{i=1}^n (x_i - \bar{x}) = 0$):       $\sum_{i=1}^n (x_i - \mu)^2 = \sum_{i=1}^n (x_i - \bar{x})^2 + n(\bar{x} - \mu)^2 = n s^2 + n(\bar{x} - \mu)^2$

  • Case A: $\mu$ Unknown, $\sigma^2$ Known:$L(\mathbf{x}; \mu) = e^{-\frac{n}{2\sigma^2}(\bar{x} - \mu)^2} \times \left[\left(\frac{1}{\sqrt{2\pi}\sigma}\right)^n e^{-\frac{1}{2\sigma^2}\sum_{i=1}^n (x_i - \bar{x})^2}\right]$       Setting $g(t; \mu) = e^{-\frac{n}{2\sigma^2}(\bar{x} - \mu)^2}$ and $h(\mathbf{x}) = \left(\frac{1}{\sqrt{2\pi}\sigma}\right)^n e^{-\frac{1}{2\sigma^2}\sum_{i=1}^n (x_i - \bar{x})^2}$, it follows that $t = \bar{x}$ (or $\sum_{i=1}^n x_i$) is sufficient for $\mu$.

  • Case B: $\sigma^2$ Unknown, $\mu$ Known:$L(\mathbf{x}; \sigma^2) = \left[\left(\frac{1}{\sqrt{2\pi}\sigma}\right)^n e^{-\frac{1}{2\sigma^2}\sum_{i=1}^n (x_i - \mu)^2}\right] \times [1]$       Setting $g(t; \sigma^2) = L(\mathbf{x}; \sigma^2)$ and $h(\mathbf{x}) = 1$, $t = \sum_{i=1}^n (x_i - \mu)^2$ is sufficient for $\sigma^2$.

  • Case C: Both $\mu$ and $\sigma^2$ Unknown:$L(\mathbf{x}; \mu, \sigma^2) = \left[\left(\frac{1}{\sqrt{2\pi}\sigma}\right)^n e^{-\frac{1}{2\sigma^2}[n s^2 + n(\bar{x} - \mu)^2]}\right] \times [1]$       The entire likelihood depends on the sample exclusively through the pair $(\bar{x}, s^2)$, or equivalently $\left(\sum_{i=1}^n x_i, \sum_{i=1}^n x_i^2\right)$. Thus, $(\bar{x}, s^2)$ is a joint sufficient statistic for the parameter vector $(\mu, \sigma^2)$.

  • Problem 5: Beta Distribution of the First Kind:

  • Let $f(x; p, q) = \frac{1}{B(p, q)} x^{p-1}(1-x)^{q-1}$ for $0 < x < 1$, where $p > 0, q > 0$.

  • The likelihood function is:       $L(\mathbf{x}; p, q) = \left(\frac{1}{B(p, q)}\right)^n \left(\prod_{i=1}^n x_i\right)^{p-1} \left(\prod_{i=1}^n (1 - x_i)\right)^{q-1}$

  • Factoring with $h(\mathbf{x}) = 1$, the joint sufficient statistic for the vector $(p, q)$ is:       $\mathbf{t} = \left(\prod_{i=1}^n x_i, \prod_{i=1}^n (1 - x_i)\right)$

  • Problem 6: Zero-Mean Normal with Parameter $\theta = \sigma^2$:

  • For $f(x; \theta) = \frac{1}{\sqrt{2\pi\theta}} e^{-\frac{x^2}{2\theta}}$, the likelihood is:       $L(\mathbf{x}; \theta) = (2\pi\theta)^{-n/2} e^{-\frac{1}{2\theta}\sum_{i=1}^n x_i^2} \times [1]$

  • The sample data enters solely through $\sum_{i=1}^n x_i^2$, which confirms that $t = \sum_{i=1}^n x_i^2$ is the sufficient statistic for $\theta$.

  • Problem 7: Gamma Distribution:

  • For $f(x; m, p) = \frac{m^p}{\Gamma(p)} e^{-mx} x^{p-1}$ for $x > 0$, where $m > 0, p > 0$.

  • The likelihood function is:       $L(\mathbf{x}; m, p) = \left(\frac{m^p}{\Gamma(p)}\right)^n e^{-m\sum_{i=1}^n x_i} \left(\prod_{i=1}^n x_i\right)^{p-1} \times [1]$

  • The joint sufficient statistic for $(m, p)$ is $\left(\sum_{i=1}^n x_i, \prod_{i=1}^n x_i\right)$.