GenAI by genAI

Generative model — learns $P(data)$ / the posterior $P(data \mid label)$ by inverting Bayes (likelihood $\to$ posterior), to create new data

Discriminative model — learns $P(label \mid data)$ directly, to classify/decide, discarding data generation

Autoencoder — bottleneck compresses data into a latent space, decoder must reconstruct (the latent is relevant)

Variational AE (VAE) — encoder predicts a Gaussian (μ(x), σ(x)) per input, KL-regularized, so the latent is structured/samplable

GAN — Generator vs Discriminator zero-sum game: G forges, D tells real/fake, improving each other

Diffusion — instead of a game, a forward Markov noise chain and a learned reverse denoiser (predict the noise), U-Net backbone

Latent space — the "backbone": with an encoder you can do math on data (king − man + woman = queen), it carries compressed meaning

Conditioning — injecting a desired value/label/prompt (conditional GAN, latent diffusion prompt) to steer generation toward a target

Autoregressive LLM (Transformer) — masked self-attention + feedforward, predict next token, KV-cached

RAG (Retrieval-Augmented Generation) — grounds the LLM on external retrieved context to cut hallucination (understand → ground)

**KL divergence** 📏 — measures the difference between two probability distributions

**VAE's Gaussian prior** 🎲 — the target shape KL **pulls the encoded latent toward**; without regularization the latent would be arbitrary

**Mode collapse** 🕳️ — GAN degenerates: G learns to output a single convincing sample to always fool D

**Training instability in GAN** 🎢 — the two-player game has no single optimum; G and D can chase each other in infinite improving-but-wrong directions

**Conditional GAN** 🏷️ — injects a **desired feature between encoder and generator** to control output

**Image-to-image translation** 🖼️ — almost-noising input x, feeding G(x) **and** x to D, transforms domains (e.g. photo ↔ sketch)

**QKV self-attention** 🔍 — $QK^\top/\sqrt{d_k}$, softmax over similarity; Q="what I seek", K="what I contain", V="what I pass"

**√dₖ scaling** ⚖️ — prevents large dot-products from saturating softmax and **killing gradients**

**KV caching** ⚡ — store past keys/values across decode steps to avoid recomputing (FlashAttention optimizes it)

**Autoregression** 🔁 — token-by-token generation; caching is what makes long autoregressive decoding affordable

**MoE (Mixture of Experts)** 🧮 — a **router/load-balancer** picks sparse experts per token

**Params ≠ FLOPs** ⚖️ — MoE scales **memory (params) but not activation (compute)** per token; fewer active params, huge total

**Explicit density** 📐 — tractable, you can directly compute $P(x)$ (e.g. autoregressive, NADE)

**Implicit density / likelihood-free** 🎯 — you can *sample* from the model but not evaluate density (GAN, VAE-style)

**LoRA / QLoRA** 🪛 — parameter-efficient fine-tuning: train low-rank adapters instead of all weights

**RLHF alignment** 🎯 — fine-tune the model so its outputs match human preferences, a key "align" step

**FID / CLIPScore** 📊 — FID measures *distributional fidelity* of generated images; CLIPScore measures *prompt-alignment*

**Hallucination rate & toxicity** 🚨 — eval also targets *truthfulness/safety*; judge step of "understand, build, ground, orchestrate, **judge**"

**Flow Matching** 🌊 — learns a **direct vector field** mapping noise→data (vs. a stepwise denoising)

**Optimal Transport** 🧭 — gives **straight** noise→data paths, the theoretical ideal flow matching aims for