Lecture L11: Bayesian Inference and Learning

What is the Bayesian interpretation of the probability $P(X=x)$?

It is our degree of belief that the random variable $X$ assumes the value $x$.

What does Bayesian probability quantify?

Our uncertainty about a phenomenon.

How are model parameters treated in the context of Bayesian learning?

They are treated as random variables.

What does the prior distribution $P(\Theta)$ model in Bayesian learning?

Our uncertainty about the parameters $\Theta$ prior to observing any data.

Into which distribution do we update our prior beliefs after observing data $x$?

The posterior distribution $P(\Theta|x)$.

What is the mathematical formulation of the Bayesian update rule (Bayes theorem)?

$$P(\Theta|x) = \frac{P(x|\Theta)}{P(x)} \cdot P(\Theta)$$

What is the analytical expression for the normalising marginal likelihood $P(x)$ for continuous parameters?

$$P(x) = \int_{\Theta} P(x|\Theta)P(\Theta)d\Theta$$

What is the purpose of the marginal likelihood $P(x)$ in the context of Bayes theorem?

It acts as a normalising constant that ensures the posterior distribution integrates to one.

What does the ratio $\frac{P(x|\Theta)}{P(x)}$ represent in the Bayesian update rule?

It is the updating factor, which quantifies how much the observed data shifts our belief in each possible parameter value.

Why is the marginal likelihood $P(x)$ often computationally intractable in practice?

Because the integral over the parameter space has no closed form for general distributions.

What special types of prior distributions yield closed-form expressions for the posterior, making the marginal likelihood analytically tractable?

Conjugate priors.

What type of parameter is the Beta distribution typically used to model in Bayesian inference?

A continuous parameter that represents a probability defined on the interval $[0, 1]$.

What is the probability density function of the Beta distribution $$ Beta(x

\alpha,\beta) $$?

How is Euler's beta function $B(\alpha,\beta)$ mathematically defined?

$B(\alpha,\beta):=\int_{0}^{1}x^{\alpha-1}(1-x)^{\beta-1}dx$

Does the value of Euler's beta function $B(\alpha,\beta)$ depend on the variable $x$?

No, its value depends only on the shape parameters $\alpha$ and $\beta$.

What is the formula for the mean of a Beta distribution?

$\frac{\alpha}{\alpha+\beta}$

What is the formula for the variance of a Beta distribution?

$\frac{\alpha\beta}{(\alpha+\beta)^{2}(\alpha+\beta+1)}$

What is the formula for the mode of a Beta distribution?

$\frac{\alpha-1}{\alpha+\beta-2}$

What distribution is recovered from $$ Beta(x

\alpha,\beta) $when$ \alpha=1 $and$ \beta=1 $$?

What shape property does a Beta distribution possess when $\alpha=\beta$?

It is a symmetric distribution with zero skewness.

What type of skewness occurs in a Beta distribution when $\alpha<\beta$?

Positive skewness, which shifts the probability mass towards $0$.

What type of skewness occurs in a Beta distribution when $\alpha>\beta$?

Negative skewness, which shifts the probability mass towards $1$.

Where is the peak located for a symmetric, unimodal Beta distribution where $\alpha=\beta>1$?

At $\frac{1}{2}$.

Why does the Beta distribution serve as an analytically convenient prior for a Binomial likelihood?

Because of the structural similarity between $$ Beta(x

The Beta distribution is a special case of which parametric distribution family?

The Dirichlet distribution.