LN 10 Generative Models GANs & AEs

Discriminative Model

• Learns the boundary between classes

• Goal: Estimate P(Y|X)

• Examples: Logistic Regression, SVMs, CNNs

Generative Model

• Learn the distribution of data itself

• Goal: Estimate P(X) or (X,Y)

• Examples: Naive Bayes, GANs, VAEs

Goal of Generative Modeling

Take as input training samples from some distribution and learn a model that represents that distribution

Why Generative Models?

• Generative models learn probability distributions

• Sampling (new data instances)

• Backbone of Generative AI

Latent Variable Models

1. Autoencoders & Variational Autoencoders (VAEs)

2. Generative Adversarial Networks (GANs)

Autoencoder

• Unsupervised deep learning method that learns a compressed embedding for a data input.

• Goal is to reproduce the input data at the output layer

Autoencoder uses

Commonly used for dimensionality reduction or generating an embedding for input into other classification or clustering methods.

Autoencoder components

1. Encoder

2. Compressed embedding

3. Decoder

Autoencoder process steps:

1. Encoder encodes input data into an embedding using non-linear activation functions

2. Decoder reconstructs output by using non-linear layers to decode embedding

3. MSE is calculated between reconstructed output and original input

4. Backpropagation of error to adjust autoencoder weights.

5. Repeat until MSE is minimised

Autoencoder variations

1. Denoising

2. Sparsing

Denoising

Corrupts input data by injecting Gaussian noise.

Sparse

Restricts the number of active nodes in the training process

Autoencoder variations intuition

Injecting noise or restricting active nodes forces autoencoder to learn the most important features.

Variational Autoencoders

1. Compress representation of world to something we can use to learn

2. Reconstruct for unsupervised learning

3. Reparameterization trick to train end-to-end

4. Interpret hidden latent variables using perturbation

5. Generate new examples

Generative Adversarial Networks

Machine learning framework where two neural networks, a generator and a discriminator, compete against each other to create realistic, synthetic data

How to train a GAN

1. Train the discriminator using current ability of the generator

2. Train the generator to beat the discriminator

3. Feedback loop