Generalized Linear Models Notes

Chapter 9: Generalized Linear Models

Section 1: Components of the Model

• Definition: A generalized linear model formally describes the relationship between a response variable and a collection of predictors.

• It specifies three components:

  1. Random Component:
     • Specifies the response variable $Y$ and its distribution conditional on predictors $X<em>1, X</em>2, …, X_{p-1}$.
  2. Systematic Component:
     • Specifies the predictors in the linear predictor $\eta = \beta<em>0 + \beta</em>1X<em>1 + \beta</em>2X<em>2 + \cdots + \beta</em>{p-1}X_{p-1}$.
  3. Link Function:
     • Specifies the relationship between the systematic component and the expected value of the random component.
     • $g(\mu|X<em>1, X</em>2, …, X<em>{p-1}) = \beta</em>0 + \beta<em>1X</em>1 + \beta<em>2X</em>2 + \cdots + \beta<em>{p-1}X</em>{p-1}$
     • Link functions are chosen based on the probability distribution of the random component.
     • 'Natural' link functions are called canonical link functions.

• Canonical Link Functions:

  • Normal: Identity link; $\, \eta = \mu, \, \mu = \eta$
  • Poisson: Log link; $\, \eta = ln(\mu), \, \mu = e^{\eta}$
  • Exponential/Gamma: Inverse link; $\, \eta = \mu^{-1}, \, \mu = \eta^{-1}$
  • Binomial: Logit link; $\, \eta = ln(\frac{\mu}{1-\mu}), \, \mu = \frac{1}{1 + e^{-\eta}}$
  • Probit: Normal Quantile; $\, \eta = \Phi(\mu), \, \mu = \Phi^{-1}(\eta)$

• Example: Normal Error Regression Model

  • Uses the identity link function: $g(x) = x$
  • $Y = \beta<em>0 + \beta</em>1X<em>1 + \cdots + \beta</em>{p-1}X_{p-1} + \epsilon$
  • $E[Y] = \beta<em>0 + \beta</em>1X<em>1 + \cdots + \beta</em>{p-1}X_{p-1}$

• Example: Binomial Distribution

  • Response $Y$ follows a binomial distribution with parameters n and $\, \pi = P(Y = 1)$
  • $ln(\frac{\pi(X<em>1, …, X</em>{p-1})}{1 - \pi(X<em>1, …, X</em>{p-1})}) = \beta<em>0 + \beta</em>1X<em>1 + \cdots + \beta</em>{p-1}X_{p-1}$
  • $\pi(X<em>1, …, X</em>{p-1}) = E[Y | X<em>1, …, X</em>{p-1}]$

• Example: Poisson Distribution

  • Response $Y$ follows a Poisson distribution with parameter $\, \lambda$
  • $ln(\lambda(X<em>1, …, X</em>{p-1})) = \beta<em>0 + \beta</em>1X<em>1 + \cdots + \beta</em>{p-1}X_{p-1}$
  • $\lambda(X<em>1, …, X</em>{p-1}) = E[Y | X<em>1, …, X</em>{p-1}]$

Section 2: Maximum Likelihood Estimators

• Definition: Given a random sample $Y<em>1, Y</em>2, …, Y<em>n$ with density functions $f(y</em>i|\theta, X)$, that depends on a parameter $\, \theta$ (possibly a vector) and predictor $X$, the likelihood function is:

  • $L(\theta|y<em>1, y</em>2, …, y<em>n, X) = f(y</em>1|\theta, X)f(y<em>2|\theta, X) \cdots f(y</em>n|\theta, X)$

• The log-likelihood function is:

  • $l(\theta|y<em>1, y</em>2, …, y<em>n, X) = ln(L(\theta|y</em>1, y<em>2, …, y</em>n, X)) = ln(f(y<em>1|\theta, X)) + ln(f(y</em>2|\theta, X)) + \cdots + ln(f(y<em>n|\theta, X)) = \sum</em>{i=1}^{n} ln(f(y_i|\theta, X))$
  • Likelihood function of a random sample is a product of density functions.
  • Log-likelihood function of a random sample is the sum of the logs of density functions.
  • The response and predictors are fixed (observed), and the parameters are the unknowns to be estimated.

• Definition: The maximum likelihood estimate (MLE) of a parameter $\, \theta$ is the value that maximizes the observed data's likelihood.

  • $L(mle \, \theta|y<em>1, y</em>2, …, y<em>n, X) = max</em>{all \, possible \, \theta} L(\theta|y<em>1, y</em>2, …, y_n, X)$
  • $\, \theta$ could represent multiple parameters (e.g., mean and variance).
  • MLE is usually computed using the log-likelihood function (since log is increasing, both maximize at the same place).
  • Theoretically, try all possible values for $\, \theta$ until you find the one that maximizes the (log)-likelihood function, subject to any constraints.

• Example: Bernoulli Distribution

  • A random sample of size three is taken from a Bernoulli distribution with unknown proportion of successes $\, \pi$.
  • Observed values: one success and two failures.
  • Density function: $f(y | \pi) = \pi^y(1 - \pi)^{1-y}$, where y = 0, 1 and $0 \leq \pi \leq 1$
  • Likelihood function: $L(\pi | y<em>1 = 1, y</em>2 = 0, y<em>3 = 0) = f(y</em>1 = 1 | \pi)f(y<em>2 = 0 | \pi)f(y</em>3 = 0 | \pi) = [\pi^1(1 - \pi)^{1-1}][\pi^0(1 - \pi)^{1-0}][\pi^0(1 - \pi)^{1-0}] = \pi(1 - \pi)^2$
  • Log-likelihood function: $l(\pi | y<em>1 = 1, y</em>2 = 0, y_3 = 0) = ln(\pi) + 2ln(1 - \pi)$

Section 3: Likelihood Ratio Test

• Suppose $M<em>0$ and $M</em>1$ are two models with $M<em>0$ a restricted version of $M</em>1$:

• $M<em>0: g(\mu) = \beta</em>0 + \beta<em>1X</em>1 + \beta<em>2X</em>2 + \cdots + \beta<em>{p}X</em>{p}$(restricted)

• $M<em>1: g(\mu) = \beta</em>0 + \beta<em>1X</em>1 + \beta<em>2X</em>2 + \cdots + \beta<em>{p}X</em>{p} + \beta<em>{p+1}X</em>{p+1} + \cdots + \beta<em>{p+k}X</em>{p+k}$(full)

• Let $l_m$ be the maximum log-likelihood of a model M.

• Definition: The likelihood ratio test statistic is given by $LRT = -2[l<em>{M</em>0} - l<em>{M</em>1}]$.

  • If n >>> 0, then the likelihood ratio test statistic has an approximate $\, \chi^2$ distribution with $k$ degrees of freedom.
  • $k$ is the number of extra parameters or the difference in the number of parameters.

• Definition: The likelihood ratio test of

  • $H<em>0: \beta</em>{p+1} = \cdots = \beta_{p+k} = 0$ (restricted)
  • $H_1$: at least one is not equal to 0 (full)
  • Rejects the null hypothesis if:
    1. The test statistic $LRT$ is greater than the appropriate critical value, $LRT \geq \chi^2_{\alpha, k}$
    2. p-value = P(\chi^2_k \geq LRT) < \alpha

• Example: Bernoulli Distribution (continued)

  • Maximum log-likelihood function:$l(\pi | y<em>1 = 1, y</em>2 = 0, y_3 = 0) = ln(\pi) + 2ln(1 - \pi)$
  • $\frac{dl}{d\pi} = \frac{1}{\pi} - \frac{2}{1 - \pi} = 0$
  • $\frac{\pi}{1-\pi} = 1/2$
  • $mle \, \pi = \frac{1}{3}$
  • $l(mle \, \pi | y<em>1 = 1, y</em>2 = 0, y_3 = 0) = ln(\frac{1}{3}) + 2ln(1 - \frac{1}{3})$

Section 4: Binary Data: Logistic Regression

• Using a linear regression model with a binary response leads to fitting a line with an infinite range to the binary response.

• Logistic Regression fits a curve that will estimate the probability of a response at various predictor values.

• Example: Challenger Space Shuttle Disaster

  • On January 28, 1986, the Challenger space shuttle broke apart 73 seconds into the flight, killing all seven crew members.
  • Investigation focused on O-rings, and damage to these O-rings may be related to the ambient temperature during the launch.

• Definition: Suppose $Y<em>i$ is a Bernoulli random variable with success probability $\, \pi</em>i$. The odds of success are given by $o<em>i = \frac{\pi</em>i}{1 - \pi_i}$.

• Definition: A random variable $Y$ has a Binomial distribution with parameters $n$ (number of independent trials) and success probability $\, \pi$ if its probability mass function is:

  • $p(y) = \binom{n}{y} \pi^y(1 - \pi)^{n-y}$, where $y = 0, 1, …, n$ and 0 < \pi < 1

• $ln(\frac{\pi(X<em>1, …, X</em>{p-1})}{1 - \pi(X<em>1, …, X</em>{p-1})}) = \beta<em>0 + \beta</em>1X<em>1 + \cdots + \beta</em>{p-1}X_{p-1}$

• $\pi(X<em>1, …, X</em>{p-1}) = \frac{1}{1 + e^{-(\beta<em>0 + \beta</em>1X<em>1 + \cdots + \beta</em>{p-1}X_{p-1})}}$

• $L(\beta<em>0, \beta</em>1, …, \beta<em>{p-1}|Y</em>i, X<em>{i,1}, …, X</em>{i,p-1} \, i = 1,2, …, n) = \prod<em>{i=1}^{n} (\frac{1}{1 + e^{-(\beta</em>0 + \beta<em>1X</em>{i,1} + \cdots + \beta<em>{p-1}X</em>{i,p-1})}})^{Y<em>i} (1 - \frac{1}{1 + e^{-(\beta</em>0 + \beta<em>1X</em>{i,1} + \cdots + \beta<em>{p-1}X</em>{i,p-1})}})^{1-Y_i}$

• R Code Examples:

  • See transcript for R code examples applying logistic regression to the Challenger disaster and Donner party datasets, including:
    • Fitting generalized linear models using glm() with `family = binomial(link =