CSCI 496/595 Generative AI

Min-Ho Lee, NAZARBAYEV UNIVERSITY

Theoretical Introduction to Generative Modeling

Differences Between Discriminative and Generative Models

  • Exploration of distinctions between generative and discriminative models in machine learning.
    • Generative models: Focus on modeling the joint probability distribution of observations and labels, thereby capturing the underlying data structure.
    • Discriminative models: Aim to model the conditional probability of a label given the observations.

Desirable Properties of a Good Generative Model

  • Identification of key attributes that a high-quality generative model should possess.
    • Realism: The generated samples should closely resemble real-world data.
    • Diversity: The model should produce a wide variety of outputs rather than repetitive results.
    • Quality: Outputs should exhibit high fidelity and detail.
    • Efficiency (train/infer): The model should be computationally efficient during training and inference.
    • Generalization: The ability to generate novel instances that were not part of the training data.

Core Probabilistic Concepts in Generative Models

  • Overview of fundamental probabilistic principles underpinning generative models.
    • Generative models are inherently probabilistic, implying some random or uncertain characteristics in the data.
    • Sampling and estimation are key components in the understanding of generative models.

Introduction to Dominant Generative Approaches

  • Leading approaches in generative modeling, highlighting methodologies and applications.
    • Image Generation:
    • Methods include Stable Diffusion, ControlNet, and Diffusion Transformers.
    • Modalities include Text-to-Image (T2I) and Image-to-Image (I2I) transformations, leveraging edge maps, masks, and sketches.
    • Text Generation:
    • Models such as GPT, Claude, Gemini, LLaMA, Qwen, and Mistral focus on generating coherent and contextually relevant text.
    • Video Generation:
    • Involves models that create video from input text or image sequences, exemplified by systems like Sora.
    • Audio & Speech Generation:
    • Audio generation models that synthesize speech or music based on text or prior audio prompts.
    • Multimodal Generation:
    • Integrates generation across various modalities (text, image, audio) using shared representations.
    • Involves modality transition, cross-modal mapping, feature alignment, and joint embedding techniques.

Discriminative Model vs. Generative Model

Key Differences

  • Nature of Generative Models: Probabilistic rather than deterministic.
    • A generative model must account for inherent randomness within the data, allowing for variability in output.
    • If a model always produces the same output, it cannot be classified as generative.

Generative Model Representation

  • Latent Space Representation:
    • The model represents attributes of images in a latent space that supports sampling.
    • Questions surrounding generalization arise: What does it imply for the generated outputs?

Framework of Generative Models

Understanding Generative Models

  • Probability Distributions:
    • Pdata: Represents the actual probability distribution of the real data.
    • Pmodel: Estimated probability distribution obtained from modeling.
    • Models can include graphical mixture models (GMM), hidden Markov models (HMM), variational autoencoders (VAE), GANs, and diffusion models.
    • Quote: "All models are wrong, but some are useful." - George Box

Inference & Density Functions

  • Inference on Models: Understanding how to define distributions (e.g., Gaussian).
  • Maximal Likelihood Estimation (MLE):
    • Generative modeling links to MLE where parameters (\theta) minimize the negative log-likelihood.
    • The goal is to maximize the likelihood of observing the given data by adjusting parameters accordingly.

Dimensionality Reduction Techniques

  • Dimensionality and Latent Space:
    • A representation of high-dimensional data into a lower-dimensional space captures essential features for model interpretability, efficiency, and performance.
    • Terms like "latent" refer to hidden layers revealing underlying patterns, incorporating methods like PCA and t-SNE for feature reduction.

Generative Model Architecture

Encoder-Decoder Framework

  • Components of a Generative Model:
    • The model consists of an encoder and decoder:
    • The encoder maps high-dimensional data to a latent space, while the decoder reconstructs data from this latent representation.
    • A unique feature includes variability generation for objects via transformations in the latent space.

Model Capabilities & Characteristics

  • Management of Latent Space:
    • Similar data points (e.g., digits) are located close together in latent space, supporting model reliability and explainability.
    • Continued understanding and manipulation of model outputs are feasible through vector arithmetic, allowing attribute adjustments without affecting other features.

Sample Space and Probability Density Function

Sample Space Definition

  • Complete Set of Observations:
    • The sample space comprises all values an observation can take, defining the boundaries of model inference.

Probability Density Function (PDF)

  • Mathematical Function Analysis:
    • The PDF, represented as $p(x)$, maps a point x in the sample space to a probability value between 0 and 1.
    • There exists a singular true density function, Pdata(x), while numerous density functions, Pmodel(x), can estimate Pdata(x).
  • Core Probability Theory Insights:
    • Addressing concerns around overfitting, interpretation, and generalization within model performance.

Generative Model Taxonomy

Classification Types

  • Types of Generative Models:
    • Tractable Density Models: Methods yielding explicit probability densities (e.g., MLE generative models).
    • Variational Approaches: Such as Variational Autos, Boltzmann machines, and statistical belief networks.
    • Implicit Probability Densities: Including techniques like GANs, which rely on adversarial training.

Generative Models Summary

  • Examples of Key Models:
    • GAN (Generative Adversarial Networks): Involves adversarial training for data generation.
    • VAE (Variational Autoencoder): Focuses on maximizing the variational lower bounds.
    • Flow-based Models: Use invertible transformations on distributions for generation.
    • Diffusion Models: Introduce gradual Gaussian noise and reverse processes for data generation.