STAT 3202 Midterm 2 Review

What is the idea of consistency?

Convergence to the truth AND low variability

We say an estimator theta-hat for theta is consistent if it converges to the true parameter.

converges to the true parameter meaning

Theta hat is getting arbitrarily close to theta as we collect more data

Mathematical definition of consistency

As n increases, theta hat gets arbitrarily close to theta. Difference between theta hat and theta gets small.

lim as n approaches infinity of P(|theta hat - theta| < epsilon) = 1.

"Theta hat is consistent for theta"

What is epsilon in the mathematical definition of consistency?

Epsilon refers to some arbitrary positive number

How do we prove theta hat is consistent for theta?

(How do we prove consistency)?

1.) Show that theta-hat is unbiased for theta

2.) Show that lim as n approaches infinity of V(theta-hat) = 0.

NOTE: If 1 & 2 are true, theta-hat is consistent for theta.

Likelihood

Joint density of the observed data, which encompasses the collective probabilities of observing each individual data point.

Product of the individual pmfs / pdfs

Likelihood function

L(theta | X) = Product from i = 1 to n of f(xi | theta)

NOTE: Depending on discrete vs. continuous case, f(xi | theta) could be PDF / PMF

(joint probability of the observed data, shows how likely a particular population will produce an observed sample)

Joint density

Summarizes the probabilities of multiple random variables together

Goal is to split likelihood into two components g and h which each contain:

g can contain parameter (lambda, theta, p, etc), constants, and sufficient statistic

h cannot contain the parameter (lambda, theta, p, etc), but can contain constants, and the data in some other form (NO PARAMETER)!!!

Sufficiency

A statistic T(x1,...,xn) is sufficient for theta if T contains all information about theta that our data can provide.

NOTE: If a sample statistic keeps all information in a sample relevant to the estimation of the parameter, it is a so-called sufficient statistic

A sufficient statistic

Utilizes information from the data as efficiently as possible

Implication of sufficiency

Individual values of the sample do not matter, only the value of the sum (the sufficient statistic)

Population moment

Every outcome appears once, weighted by its probability (these are theoretical moments)

E[X], E[X^2], E[xk]

Sample moment

More probable outcomes appear more frequently, weight equally

We assume that sample moments are

good / reasonable estimates of the population moments

To compute a method of moments estimator, we:

1.) Equate the population (theoretical) & sample (empirical) moments

2.) Solve for theta-hat (the estimator)

Basic idea about method of moments estimation to estimate parameters

Sampling moments should provide reasonable / good estimates of the corresponding population moments.

Method of moments is a mathematical framework for proposing formulas for estimates

An estimator for the first moment is always the ___

sample mean

An estimator for the second moment is always the ____

average of the squared observations

maximum likelihood estimation

A method that determines values of the parameters and these values maximize the likelihood and best fit the data

A method that helps us determine the optimum / most likely value for our parameter, theta (for example). We maximize the likelihood function with respect to the parameter, theta. We refer to the value of theta that maximizes the likelihood function as the maximum likelihood estimate of theta.

Maximum likelihood estimate

The point in which the parameter value that maximizes the likelihood function