Topic Modeling & Latent Dirichlet Allocation (LDA)

Introduction & Session Context

• Instructor resumes after a 2-week gap; previous unit covered Part-of-Speech tagging (HMM, Maximum Entropy, Bi-LSTM, CRF, neural models).
• New unit: Topic Modeling – automatically discovering the themes a document (or corpus) discusses.
• Central real-world driver: filtering massive news/email/social feeds so that only content matching a user’s topical interests is shown.

Motivation & Use-Cases

• Summarisation (e.g. Gmail “Summarise this thread”).
• Measuring topic coverage and intensity inside a single document.
• Finding topic–topic correlations (e.g. "war" ↔ "arms race", "batting" ↔ "bowling").
• Recommendation systems: given one liked document, find others sharing its hidden topical mix.
• Tracking how topics evolve through time (e.g. Sachin Tendulkar → Virat Kohli in cricket articles).
• Domain examples: book/movie tagging, Research papers, protein–DNA sequence analysis, even image generation (analogy to DALL·E).

Key Terminology & Symbols (will re-appear throughout)

• $V$ : size of vocabulary (distinct word types).
• $K$ : number of topics we wish/assume.
• $D$ : number of documents in corpus.
• $N_d$ : number of word tokens in document $d$.
• $w_{d,n}$ : the n-th observed word in document d.
• $z_{d,n}$ : latent topic id of that word.
• $\alpha$, $\eta$ (or $\beta$) : Dirichlet hyper-parameters – priors over topic mixtures and word mixtures respectively.
• $\theta_d$ : $K$-dimensional vector – topic proportions for document d.
• $\phi_k$ : $V$-dimensional vector – word distribution for topic k.

Historical Approaches

• Unigram Model: choose one word → infer topic. Weak due to polysemy ("bat" = cricket bat or mammal).
• Mixture of Unigrams: combine several unigram models; still brittle.
• Latent Semantic Indexing (LSI / pLSI)
  • Uses co-occurrence patterns + Singular Value Decomposition to embed documents/words in lower-dimensional space.
  • Good when each doc focuses on a single theme; less effective when documents mix themes.
• Latent Dirichlet Allocation (LDA)
  • Probabilistic/Bayesian upgrade over LSI.
  • Explicitly models documents as mixtures of topics and topics as mixtures of words.

Breaking Down the Name “Latent Dirichlet Allocation”

• Latent: topics are hidden; word “cricket” need not appear for text to be about cricket.
• Dirichlet: the prior used for both $\theta$ and $\phi$; convenient because it’s conjugate to the multinomial.
• Allocation: assigning each word token to a topic.

Mathematical Foundations

• Dirichlet PDF (for $K$-vector $\theta$):
  $p(\theta\mid\alpha)=\frac{1}{B(\alpha)}\prod<em>{k=1}^{K}\theta</em>k^{\alpha_k-1}$
• Conjugacy: Multinomial likelihood × Dirichlet prior ⇒ Dirichlet posterior (keeps inference tractable).
• Plate Diagram Essence
  • Outer plate (repeat over documents $D$) contains inner plate (repeat over word positions $N_d$).
  • Arrows: $\alpha\to\theta<em>d$, $\theta</em>d\to z<em>{d,n}$, $\beta\to\phi</em>k$, $z<em>{d,n},\phi\to w</em>{d,n}$.
• Simplex Geometry
  • Any $K$-dimensional probability vector lives on a $(K-1)$-simplex ⇒ can visualise 3-word toy case on a triangle.

LDA Generative Process (Top-down description)

1. For each topic $k\in{1..K}$
   • Draw word distribution $\phi_k \sim \text{Dirichlet}(\eta)$.
2. For each document $d\in{1..D}$
   a. Draw topic mixture $\theta<em>d \sim \text{Dirichlet}(\alpha)$. b. For each word position $n\in{1..N</em>d}$
   i. Draw topic $z<em>{d,n} \sim \text{Multinomial}(\theta</em>d)$.
   ii. Draw word $w<em>{d,n} \sim \text{Multinomial}(\phi</em>{z_{d,n}})$.

• This hierarchy allows multiple topics per document and ambiguous words across topics.
• Example narratives:
  • "bat", "ball", "over" → cricket topic vs. "bat", "nocturnal", "echolocation" → zoology topic.
  • "bank" + "loan" vs. "bank" + "river".

Posterior Inference Challenge

We need $p(\theta,\phi,z\mid w,\alpha,\eta)$ but exact marginalisation
$\int<em>\theta \int</em>\phi \sum_{z} p(\theta,\phi,z,w\mid\alpha,\eta)$
is intractable for realistic corpora.

Approximate Inference Techniques

1. Gibbs Sampling (collapsed)

• Integrate out $\theta$ and $\phi$ analytically; iterate over tokens updating $z_{d,n}$ conditioned on other assignments.
• Conditional update formula:
  $p(z<em>{d,n}=k \mid z</em>{-}, w, \alpha, \eta) \propto \frac{n<em>{k}^{w</em>{d,n},-n}+\eta}{n<em>{k}^{\cdot,-n}+V\eta}\times\frac{n</em>{d}^{k,-n}+\alpha}{n<em>{d}^{\cdot,-n}+K\alpha}$ • $n</em>{k}^{w,-n}$ : count of word w assigned to topic k (excluding current position).
  • $n_{d}^{k,-n}$ : count of tokens in doc d already assigned to k.
• Iterate until Markov chain mixes; sample draws approximate posterior.
• Parameter estimates after convergence:
  $\hat{\phi}<em>{k,v}=\frac{n</em>{k}^{v}+\eta}{\sum<em>{v'} n</em>{k}^{v'}+V\eta},\qquad \hat{\theta}<em>{d,k}=\frac{n</em>{d}^{k}+\alpha}{\sum<em>{k'} n</em>{d}^{k'}+K\alpha}$

2. Variational Inference (mentioned, not detailed)

• Replaces sampling with optimisation of a tractable surrogate distribution.

Practical Questions LDA Helps Answer

1. What mixture of topics does a given document contain?
2. Given one document, find other documents with similar topic vectors.
3. Track topic prevalence/intensity across time (diachronic corpora).
4. For recommendation: map user interest vector ↔ document topic vectors.

Connections to Earlier Course Material

• Dimensionality Reduction: LDA’s simplex view echoes PCA/SVD; both aim to compress information without losing signal.
• Smoothing: Dirichlet prior plays role similar to add-one or Kneser–Ney smoothing in n-gram models.
• Probabilistic tagging models: Markov-style reasoning (HMM → sequence, LDA → bags-of-words).

Computational & Implementation Considerations

• Must choose $K$ (can use CV, non-parametric methods, or business knowledge).
• Hyperparameter tuning for $\alpha,\eta$ influences sparsity: small $\alpha$ ⇒ each doc uses few topics; small $\eta$ ⇒ each topic uses few signature words.
• Large corpora → high-dimensional counts; Gibbs sampling still linear in tokens but needs many iterations.

Ethical & Philosophical Reflections

• Hidden topic discovery can profile users implicitly (privacy concerns).
• Topic ambiguity emphasises need for transparent AI explanations – why did system tag my article as “war”?
• Potential misuse in disinformation: generating plausible documents on sensitive topics at scale.

Formula Recap (Cheat-Sheet)

• Dirichlet prior: $p(\theta \mid \alpha)$.
• Joint model: $p(w, z, \theta, \phi \mid \alpha, \eta)=p(\phi\mid\eta)p(\theta\mid\alpha)\prod p(z\mid\theta)p(w\mid z,\phi)$.
• Collapsed Gibbs conditional (core update):
  $p(z<em>i=k\mid z</em>{-i},w) \propto \frac{n<em>{k}^{w</em>i,-i}+\eta}{n<em>{k}^{\cdot,-i}+V\eta}\frac{n</em>{d<em>i}^{k,-i}+\alpha}{n</em>{d_i}^{\cdot,-i}+K\alpha}$.
• Parameter expectations after sampling: see $\hat{\phi}$ and $\hat{\theta}$ formulas above.

Visual & Intuitive Aids Discussed

• Plate diagrams – rectangles denote replication, circles latent vars, shaded circles observed words.
• Simplex triangles – 3-word toy example: corners = pure topics, interior points = document mixtures.
• Heat-map images (25-pixel toy example) – each topic visualised as distinct 5×5 pattern; documents generated by mixing patterns.

Take-Away Summary

• LDA treats each document as a distribution over hidden topics; each topic is a distribution over words.
• Dirichlet priors make Bayesian inference neat (conjugacy) and provide built-in smoothing.
• Exact inference is infeasible; collapsed Gibbs sampling delivers a practical solution by updating one word-assignment at a time.
• Outcomes: interpretable topic–word lists, per-document topic proportions, ability to generate or classify documents, and a gateway to many modern NLP/ML pipelines.