L21_Generative Deep Learning

Flashcards based on Deep Learning Lecture 21 about Generative Deep Learning

What is the key difference in text generation for generative deep learning?

Sampling instead of deterministically picking the top prediction. Think of it like choosing your next word based on probability rather than certainty, allowing for creativity. Analogy: Instead of always choosing your favorite dish at a restaurant, you sometimes pick something new at random.

Name three deep learning methods for converting sampled values into realistic data.

Variational autoencoders, Generative adversarial networks, and Diffusion models. These are like different artists using various techniques to turn abstract ideas into concrete images or text. Analogy: VAEs are like sketching, GANs like painting, and diffusion models like sculpting.

What is the primary goal of an autoencoder?

To reconstruct its input by encoding (compressing) data to a smaller dimensionality (latent space) and then decoding it back to the original dimensions. Imagine compressing a file to send it faster, then decompressing it to its original form; autoencoders do this with data. Analogy: Think of it as zipping and unzipping a file.

What can be done with the latent space in autoencoders to generate new outputs?

Randomly sample values, although not all sampled values are meaningful. However, this is not always reliable since random sampling might lead to noise. z \sim P_z Analogy: It's like tuning a radio - sometimes you find a clear station, sometimes just static.

In language models, what do directions in the latent space relate to?

Concepts or word meaning. In the latent space, z, models learn that you can traverse the semantic meaning via z + z_{\text{concept}}. Analogy: It's like a map where certain directions always lead to specific landmarks or cities.

What is one improvement that can be made to autoencoders for generating realistic output?

Imposing requirements on the structure of the latent space, such as learning a distribution rather than a fixed encoding. This is crucial for smooth and meaningful generations. Analogy: Like organizing your closet by item type, instead of throwing things in randomly.

What type of network predicts parameters of normal distributions?

Mixture density network. This allows each data point to belong to potentially multiple different clusters. p(x) = \sum{i=1}^K \pii N(x \mid \mui, \Sigmai), where \pi_i is the mixing component. Analogy: Imagine sorting candies into different jars, where each candy can belong to multiple jars with varying probabilities.

What is the aim of the variational autoencoder (VAE)?

To model distributions in latent space. Unlike regular autoencoders, VAEs learn the parameters (mean and variance) of the data distribution. This is represented by p_{\theta}(z). Analogy: It's like learning not just the notes in a song, but also the range of possible variations.

Describe the three main steps of a variational autoencoder (VAE).

1. Encode input data into parameters of a probability distribution. 2. Sample a point from this distribution. 3. Decode this point back to the input format. This creates a continuous latent space.

Analogy: Like sending a letter – you encode it into a summary, sample a specific wording, then decode it back to the recipient.

How does a variational autoencoder (VAE) force a normal distribution in the latent space?

By modifying the loss function to combine reconstruction error and the difference between the latent distribution and a normal distribution (measured by KL divergence). The loss function is L = L{\text{reconstruction}} + L{\text{KL}}. Analogy: Like adjusting a recipe to match a desired flavor profile.

What does making the latent space continuous allow?

Sampling along the different latent space dimensions to combine (or remove) concepts. This enables smooth transitions and manipulations of the generated data. Analogy: Like having a color palette where you can mix different shades seamlessly.

What are the two models that work in tandem in Generative Adversarial Networks (GANs)?

A generator that decodes random values into an image and a discriminator that takes in images and predicts whether it is real or made by the generator. It’s like a counterfeiter (generator) trying to fool a detective (discriminator). Analogy: It's a constant cat and mouse game between the generator and discriminator.

Are GANs commonly used in practice?

No, GANs are well studied in research but seldom used in practice because they are very sensitive to the configuration of the training procedure and difficult to work with. They have many parameters that need tuning to minimize instability.

Analogy: Similar to cooking a soufflé; it requires precise measurements to minimize instability.