Expectation, Covariance and Variance of Sums

Define the expectation of a discrete random variable $X$ with range $R_X$, and of $g(X)$.

$\mathbb{E}[X] = \sum_{x \in R_X} x\,\mathbb{P}(X = x)$ and $\mathbb{E}[g(X)] = \sum_{x \in R_X} g(x)\,\mathbb{P}(X = x)$, provided the series converges absolutely (e.g. $\sum_{x \in R_X} |x|\,\mathbb{P}(X = x) < \infty$).

Define the expectation of a continuous random variable $X$, and of $g(X)$.

$\mathbb{E}[X] = \int_{-\infty}^{\infty} x f_X(x)\,dx$ and $\mathbb{E}[g(X)] = \int_{-\infty}^{\infty} g(x) f_X(x)\,dx$, provided the integral converges absolutely (e.g. $\int_{-\infty}^{\infty} |x| f_X(x)\,dx < \infty$).

What condition is needed for $\mathbb{E}[X]$ to exist?

Absolute convergence: $\sum_{x \in R_X} |x|\,\mathbb{P}(X = x) < \infty$ in the discrete case, or $\int_{-\infty}^{\infty} |x| f_X(x)\,dx < \infty$ in the continuous case.

Define $\mathbb{E}[g(X,Y)]$ for a function $g : \mathbb{R}^2 \to \mathbb{R}$.

Discrete: $\mathbb{E}[g(X,Y)] = \sum_{(x,y) \in R_{X,Y}} g(x,y)\,p_{X,Y}(x,y)$.

Continuous: $\mathbb{E}[g(X,Y)] = \int_{-\infty}^{\infty}\int_{-\infty}^{\infty} g(x,y)\,f_{X,Y}(x,y)\,dx\,dy$.

Both need absolute convergence.

If $g$ depends only on $x$, why does the bivariate definition of $\mathbb{E}[g(X)]$ agree with the univariate one?

$\int_{-\infty}^{\infty}\int_{-\infty}^{\infty} g(x) f_{X,Y}(x,y)\,dx\,dy = \int_{-\infty}^{\infty} g(x)\left(\int_{-\infty}^{\infty} f_{X,Y}(x,y)\,dy\right)dx = \int_{-\infty}^{\infty} g(x) f_X(x)\,dx$. The discrete case is similar, so $\mathbb{E}[g(X)]$ is unambiguous.

Does $\mathbb{E}[\alpha X + \beta Y] = \alpha\mathbb{E}[X] + \beta\mathbb{E}[Y]$ need $X$ and $Y$ to be independent?

No. Linearity of expectation holds even when $X$ and $Y$ are dependent.

If $X$ and $Y$ are independent, what is $\mathbb{E}[g(X)h(Y)]$ for $g, h : \mathbb{R} \to \mathbb{R}$?

$\mathbb{E}[g(X)h(Y)] = \mathbb{E}[g(X)]\,\mathbb{E}[h(Y)]$. In particular $\mathbb{E}[XY] = \mathbb{E}[X]\,\mathbb{E}[Y]$.

If $X$ and $Y$ are independent and $\alpha, \beta \in \mathbb{R}$, what is $\mathrm{Var}(\alpha X + \beta Y)$?

$\mathrm{Var}(\alpha X + \beta Y) = \alpha^2\,\mathrm{Var}(X) + \beta^2\,\mathrm{Var}(Y)$.

Define the covariance $\mathrm{Cov}(X,Y)$.

$\mathrm{Cov}(X,Y) = \mathbb{E}[(X - \mathbb{E}[X])(Y - \mathbb{E}[Y])]$, provided the expectations exist.

What is the alternative (computational) formula for $\mathrm{Cov}(X,Y)$?

$\mathrm{Cov}(X,Y) = \mathbb{E}[XY] - \mathbb{E}[X]\,\mathbb{E}[Y]$.

What are $\mathrm{Cov}(X,X)$ and $\mathrm{Cov}(Y,X)$ in terms of other quantities?

$\mathrm{Cov}(X,X) = \mathrm{Var}(X)$ and $\mathrm{Cov}(Y,X) = \mathrm{Cov}(X,Y)$ (symmetry).

What is $\mathrm{Cov}(X,Y)$ when $X$ and $Y$ are independent, and why?

$\mathrm{Cov}(X,Y) = 0$, because independence gives $\mathbb{E}[XY] = \mathbb{E}[X]\mathbb{E}[Y]$, so $\mathbb{E}[XY] - \mathbb{E}[X]\mathbb{E}[Y] = 0$.

What is $\mathrm{Var}(\alpha X + \beta Y)$ for general (possibly dependent) $X$ and $Y$?

$\mathrm{Var}(\alpha X + \beta Y) = \alpha^2\,\mathrm{Var}(X) + \beta^2\,\mathrm{Var}(Y) + 2\alpha\beta\,\mathrm{Cov}(X,Y)$.

State the formula for $\mathrm{Var}\left(\sum_{i=1}^n a_i X_i\right)$ for random variables $X_1,\dots,X_n$ and $a_1,\dots,a_n \in \mathbb{R}$.

$\mathrm{Var}\left(\sum_{i=1}^n a_i X_i\right) = \sum_{i=1}^n a_i^2\,\mathrm{Var}(X_i) + \sum_{1 \leq i < j \leq n} 2a_i a_j\,\mathrm{Cov}(X_i, X_j)$.

Prove the formula for $\mathrm{Var}\left(\sum_{i=1}^n a_i X_i\right)$.

1. 1. $\mathrm{Var}\left(\sum_i a_i X_i\right) = \mathbb{E}\left[\left(\sum_i a_i X_i\right)^2\right] - \left(\sum_i a_i\mathbb{E}[X_i]\right)^2$, using linearity.

2. Expand both squares: $\left(\sum_i c_i\right)^2 = \sum_i c_i^2 + 2\sum_{i \lt j} c_i c_j$.

3. Use linearity: $\sum_i a_i^2\mathbb{E}[X_i^2] + 2\sum_{i \lt j} a_i a_j\mathbb{E}[X_iX_j] - \sum_i a_i^2\mathbb{E}[X_i]^2 - 2\sum_{i \lt j} a_i a_j\mathbb{E}[X_i]\mathbb{E}[X_j]$.

4. Group: $\sum_i a_i^2(\mathbb{E}[X_i^2] - \mathbb{E}[X_i]^2) + 2\sum_{i \lt j} a_i a_j(\mathbb{E}[X_iX_j] - \mathbb{E}[X_i]\mathbb{E}[X_j])$.

5. Recognise $\mathrm{Var}(X_i)$ and $\mathrm{Cov}(X_i,X_j)$.

If $X_1,\dots,X_n$ are independent, what is $\mathrm{Var}\left(\sum_{i=1}^n a_i X_i\right)$, and why?

$\mathrm{Var}\left(\sum_{i=1}^n a_i X_i\right) = \sum_{i=1}^n a_i^2\,\mathrm{Var}(X_i)$, because every $\mathrm{Cov}(X_i, X_j) = 0$ for $i \neq j$ in the general variance-of-a-sum formula.