Probability Theory

UW Madison Stat Qual Prep Option B

Theorem 1: Let X \subset R^d. Suppose that f: X \rightarrow R is non-negative and integrable. Then, …

There is a distribution P over X such that if x \sim P, for every A \subset X, we have \mathbb{P}(X \in A) = \int_{A} p(\textbf{x})d\textbf{x} where p(\textbf{x}) = \frac{f(\textbf{x})}{\int_Xf(\textbf{x})d\textbf{x}}

Bernoulli Density

p(x) = p^x(1-p)^{1-x} for x \in {0,1}

Bernoulli Expected Value

p

Bernoulli Variance

p(1-p)

Poisson Distribution Density

p(x) = \frac{\lambda^xe^{-\lambda}}{x!} for x \in \{0,1,\cdots\}

Poisson Expected Value

\lambda

Poisson Variance

\lambda

Gamma Function

\Gamma(x) = \int_{0}^{\infty}t^{x-1}e^{-t}dt

Gamma Function Properties

\Gamma(0) = \Gamma(1) = 1, \Gamma(x+1) = x\Gamma(x), \Gamma(n) = (n-1)! , \Gamma(1/2) = \sqrt{\pi}

Gamma Distribution Density

s,r > 0, p(x) = \frac{r^s}{\Gamma(s)}x^{s-1}e^{-rx} for x>0

Gamma Distribution Expected Value

s/r

Gamma Distribution Variance

s/r²

Beta Function

B(a,b) = \int_0^1 t^{a-1}(1-t)^{b-1}dt for a,b>0

Alternative Beta Function

B(a,b) = \frac{\Gamma(a) \Gamma(b)}{\Gamma(a+b)}

Standard Normal Density

p(x) = (2\pi)^{-1/2}e^{-x²/2}

Standard Normal Expected Value

0

Standard Normal Variance

1

Basis of Monte Carlo Approximations

Approximation of probability based on random samples

Relation between marginal and joint distribution

Can determine marginal distributions exactly and uniquely from the joint density, but cannotdetermine unique and exact joint density from the marginals

Marginal Density

Integrate out all variables that we are not interested in

Negative Binomial Density

p(x) = {{x-1}\choose{r-1}} p^r(1-p)^{x-r} , x = # of trials until rth success with probability of success p in each trial

Negative Binomial Expected Value

r/p

Negative Binomial Variance

r(1-p)/p²

Negative Binomial MGF

\left [ \frac{pe^t}{1-(1-p)e^t} \right ]^r

Poisson MGF

\exp[\lambda(e^t-1)]

Normal MGF

\exp \left [ \mu t + \frac{t²\sigma²}{2} \right ]

Gamma MGF

(1-\beta t)^{-\alpha}

Normal Density

p(x) = (2\pi\sigma²)^{-1/2} \exp\left [ -(2\sigma²)^{-1}(x-\mu)² \right ]

Binomial Density

p(x) = {n\choose x} p^x (1-p)^x

Binomial Expected Value

np

Binomial Variance

np(1-p)

Binomial MGF

[pe^t + (1-p)]^n

Chi-Square Distribution

p(x) = \frac{x^{\nu/2 - 1}e^{-x/2}}{2^{\nu/2}\Gamma(\nu/2)}

Chi Squared Mean

\nu

Chi Squared Variance

2\nu

Chi Squared MGF

(1-2t)^{-\nu/2}

P(A|B) =

P(A|B) = \frac{P(A\cap B)}{P(B)}

Bayes Rule: P(A|B) =

P(A|B) = P(B|A)\frac{P(A)}{P(B)}

Law of Total Probability

P(A_i|B) = \frac{P(B|A_i)P(A_i)}{\sum_{j=1}^{\infty} P(B|A_j)P(A_j)}

Independence

P(A \cap B) = P(A)P(B) AND/OR P(A|B) = P(A)

Random Variable

Function from a sample space into the real numbers

Cumulative Distribution Function

F(x) = P(X \leq x)

Transformations: MGF method

1. Define U to be a function of n random variables.

2. m_U(t) = E[e^{tu}]

3. Match m_U(t) to mgfs of known distributions

4. If match, then U follows known distribution by uniqueness property of MGFs

Transformations: Method of Distribution Functions

1. We know Y is random var with CDF F_Y(y), and we are interested in distribution of U = h(y)

2. Find F_U(u) using F_Y(y), remember to adjust bounds!

3. Take derivative with respect to U to get f_U(u)

Transformations: Method of Transformations

1. NEED U = h(y) to be either increasing or decreasing

2. Find y = h^{-1}(u)

3. Substitute y = h^{-1}(u) into f_Y(y)

4. Multiply by \frac{dh^{-1}}{du} to find f_U(u)

5. IF U decreasing, take absolute value

Expected Value

weighted average

Moment Generating Function

M_X(t) = E[e^{tX}] , exists if there exists a constant b such that M_X(t) is finite for |t| \leq b

kth moment from MGF

\mu’_k = \frac{d^k m(t)}{dt^k}|_{t=0}

Exponential Family definition

f(x|\theta) = h(x)c(\theta)\exp\left( \sum_{i=1}^k w_i(\theta)t_i(x) \right) where h(x) \geq 0, c(\theta) \geq 0, t_1(x), \ldots, t_k(x), w_1(\theta),\ldots, w_k(\theta) real valued.

Location-Scale Family general idea

specify a single pdf as the “standard”, then generate any other pdf by transforming the standard in a prescribed way

Location-Scale Family definition

Let f(x) be any pdf. Then, the family of pdfs \frac{1}{\sigma}f\left(\frac{x-\mu}{\sigma}\right) is the location-scale family with standard pdf f(x), location parameter \mu, and scale parameter \sigma

Chebychev’s Inequality

Let X be a random variable and let g(x) be a nonnegative function. Then, for any r>0, P(g(X) \geq r) \leq E(g(X))/r

If X \sim Pois(\lambda) then P(X = x+1) =

P(X = x+1) = \frac{\lambda}{x+1}P(X=x)

Integration by Parts

\int udv = uv - \int vdu

Marginal Distribution of X

f(x) = \int_y f(x,y)dy

MGF of X+Y when X,Y independent

m_{X+Y}(t) = m_X(t)m_Y(t)

Covariance definition

Cov(X,Y) = E[(X-\mu_X)(Y-\mu_Y)]

Covariance shortcut

Cov(X,Y) = E(XY) - E(X)E(Y)

Cov(X+Y, Z) =

Cov(X+Y, Z) = Cov(X,Z) + Cov(Y,Z)

Correlation definition

Cor(X,Y) = \frac{Cov(X,Y)}{\sigma_X\sigma_Y}

Var(\sum a_ix_i) =

Var(\sum a_ix_i) = \sum a_i² Var(x_i) + 2\sum\sum_{i < j} a_ia_jCov(X_i, X_j)

X_1, X_2, \ldots converges in probability to a random variable X if

for every \epsilon > 0, \lim_{n\rightarrow\infty} P(|X_n - X| < \epsilon) = 1

Weak Law of Large Numbers

Let X_1, X_2, \ldots be iid random variables with E(X_i) = \mu, Var(X_i) = \sigma² < \infty . Then, for every \epsilon > 0, \lim_{n \rightarrow\infty}P(|\bar{X} - \mu| < \epsilon) = 1

Consistent estimator

estimator is consistent for a quantity if it converges in probability to the quantity

If X_1, X_2, \ldots converges in probability to RV X and h(x) is a continuous function, then

If X_1, X_2, \ldots converges in probability to RV X and h(x) is a continuous function, then h(X_1), h(X_2), \ldots converges in probability to RV h(X)

Convergence almost surely

A sequence of random variables X_1, X_2, \ldots such that for every \epsilon > 0, P(\lim_{n\rightarrow\infty}|X_n - X| < \epsilon) = 1

Strong Law of Large Numbers

Let X_1, X_2, \ldots be iid random variables with E(X_i) = \mu and Var(X_i) = \sigma²<\infty. Then for every \epsilon > 0, P(\lim_{n \rightarrow\infty}|\bar{X} - \mu| < \epsilon) = 1

Convergence in Distribution

A sequence of random variables X_1, X_2, \ldots, such that \lim_{n\rightarrow\infty} F_{X_n}(x) = F_X(x) at all points x where F_X(x) is continuous

Relationship between types of convergence

a.s. \rightarrow prob \rightarrow dist

Central Limit Theorem

Let X_1, X_2, \ldots be a sequence of iid random variables with E[X_i] = \mu and 0< Var(X_i) = \sigma² < \infty. Then, \sqrt{n}(\bar{X} - \mu)/\sigma converges in distribution to the N(0,1) distribution

Slutsky’s Theorem

If X_n \rightarrow X in distribution and Y_n \rightarrow a in probability where a is a constant, then

1. Y_nX_n \rightarrow aX in distribution

2. X_n + Y_n \rightarrow X + a in distribution

Delta Method general use

When we are interested in the distribution of some function of a random variable

rth order Taylor Expansion about a

T_r(x) = \sum_{i=1}^r \frac{g^{(i)}(a)}{i!}(x-a)^i

Delta Method

Let Y_n be a sequence of ranomd variables that satisfies \sqrt{n}(Y_n - \theta) \rightarrow N(0,\sigma²) in distribution. For a given function g and a specific value of \theta, suppose that g’(\theta) exists and is nonzero. Then,

\sqrt{n}|g(Y_n) - g(\theta)| \rightarrow N(0,\sigma²[g’(\theta)]²) in distribution