AVAI flashcards

0.0(0)
Studied by 13 people
call kaiCall Kai
learnLearn
examPractice Test
spaced repetitionSpaced Repetition
heart puzzleMatch
flashcardsFlashcards
GameKnowt Play
Card Sorting

1/84

flashcard set

Earn XP

Description and Tags

A set of Advanced Visual AI flashcards made by yours truly

85 Terms

1
New cards

Explain the Inverse Problem

The inverse problem is the problem of taking a degraded image and restoring it. We represent the degraded image as a degradation function H applied to the original image with added noise. The process then involves estimating the original image from the observed data by reversing the effects of degradation and noise.

<p>The inverse problem is the problem of taking a degraded image and restoring it. We represent the degraded image as a degradation function H applied to the original image with added noise. The process then involves estimating the original image from the observed data by reversing the effects of degradation and noise.</p>
2
New cards

What is the Wiener filter?

A loss function which minimises the Mean Square Error (MSE) between an output and a ground truth

<p>A loss function which minimises the Mean Square Error (MSE) between an output and a ground truth</p>
3
New cards
<p>What are the names of the four components of the Bayes Formula?</p>

What are the names of the four components of the Bayes Formula?

knowt flashcard image
4
New cards

What does the Bayesian Estimator do?

Minimises the loss function averaged over the posterior given the evidence.

<p>Minimises the loss function averaged over the posterior given the evidence.</p>
5
New cards

What is the Maximum a Posteriori (MAP) estimator?

The Bayesian Estimator with a hit or miss loss function

<p>The Bayesian Estimator with a hit or miss loss function</p>
6
New cards

What are the steps to solve for an estimator?

Take the logs => differentiate with respect to ฮธ => set equal to 0 => rearrange for ฮธ

7
New cards

What is the variational formulation of the Bayesian?

knowt flashcard image
8
New cards
<p>The Constrained Least Squares regularisation technique consists of choosing x that minimises the Lagrangian. What are each of the terms?</p>

The Constrained Least Squares regularisation technique consists of choosing x that minimises the Lagrangian. What are each of the terms?

ฮป is the regularisation parameter, controlling the relative contribution of both main terms. ||๐‘ฆโˆ’๐ป๐‘ฅ||2 is the data attachment term. ||๐ถ๐‘ฅ||2 is the prior term.

9
New cards

What is the solved form of the Constrained Least Squares?

knowt flashcard image
10
New cards

What is the idea behind Alternate Direction of Multipliers Method (ADMM)

We split a problem which is too hard to solve into separate smaller problems through an additional term. We then iteratively minimise each new variable until we reach a minimum.

<p>We split a problem which is too hard to solve into separate smaller problems through an additional term. We then iteratively minimise each new variable until we reach a minimum.</p>
11
New cards

What is step 1 of ADMM?

Update x

<p>Update x</p>
12
New cards

What is step 2 of ADMM?

Update v

<p>Update v</p>
13
New cards

What is step 3 of ADMM?

Update u

<p>Update u</p>
14
New cards

What is the idea behind Deep Image Prior (DIP)

We optimise a neural network to reconstruct an image from noise. We train it enough for the weights to reconstruct the structure of the image but not enough to also reconstruct the noise.

<p>We optimise a neural network to reconstruct an image from noise. We train it enough for the weights to reconstruct the structure of the image but not enough to also reconstruct the noise.</p>
15
New cards

In Deep Image Prior (DIP) what are the key points regarding the input noise, ground truth and output image?

We use the same input noise each iteration. The ground truth is the degraded image we are trying to fix. The output image is a surjective mapping of noise onto the image. The NN will reconstruct the structure of the image before it reconstructs the noise, so if training is stopped at the right time, we will end up with a denoised image as output.

16
New cards

What is the maths behind Deep Image Prior (DIP)

knowt flashcard image
17
New cards

What is the key idea behind Regularisation by Denoising (RED)

knowt flashcard image
18
New cards

What is the key idea behind DeepRED

knowt flashcard image
19
New cards

What is the maths behind DeepRED?

knowt flashcard image
20
New cards

What is the idea behind Regularisation by Artifact Removal (RARE)

We replace the denoiser in RED with a CNN which we train for more general artifact removal. The artifact-removal network ๐‘…๐œƒ, where ๐œƒ denotes the weights of the CNN can learn the imaging prior despite having no fully-sampled ground truth.

21
New cards

What does the estimator look like for RARE?

knowt flashcard image
22
New cards

What is the maths behind RARE?

knowt flashcard image
23
New cards

What is the idea behind Deep Unfolding?

We take an iterative algorithm and turn it into a DNN by designing each layer to represent a single iteration of the algorithm.

24
New cards

What are the four steps involved in creating a Deep Unfolding network?

  1. Find a suitable iterative optimisation algorithm for the problem

  2. Fix the number of iterations (and hence the number of layers)

  3. Design the layers to imitate the free parameters of each iteration such that they can be trained

  4. Train the network


25
New cards

What is the idea behind ISTA?

An iterative image reconstruction algorithm which solves the sparse coding problem using two steps:

  1. A Gradient Descent Step where we adjust the image to match the raw data

  2. A Soft-Thresholding Step where we force small/noisy pixel values to zero to keep the image data sparse


26
New cards

What is LISTA?

Learned ISTA. It is the ISTA algorithm unfolded into the layers of a NN.

27
New cards

What is the maths behind the training of LISTA?

knowt flashcard image
28
New cards

What are the benefits of LISTA over ISTA?

LISTA may achieve higher efficiency than ISTA

Useful when W (Weights representing the degradation matrix H) is hard to determine

Number of trained layers L can be an order of magnitude smaller than the number of iterations required of ISTA to converge

29
New cards

What are the three types of image fusion?

Pixel level fusion

Region level fusion

Compressive fusion

30
New cards

In image fusion, what are the pixel level approaches?

Maximum selection scheme

Average scheme

Weighted average scheme

31
New cards

How do we represent image fusion as an inverse problem?

We define two degradation functions that have been applied to the same image to create two degraded ones.

<p>We define two degradation functions that have been applied to the same image to create two degraded ones.</p>
32
New cards

What is the maths behind image fusion?

knowt flashcard image
33
New cards

What is the key concept behind generative modelling?

IDEA: Can we build a NN that can capture the distribution of the data?

34
New cards

What does an Autoencoder do?

It takes images as inputs and encodes them in a lower dimension latent space from which it aims to recreate the images via the decoder.

35
New cards

What is the latent space?

The latent space is a low dimensional representation z of the data x which captures the essential structure. It is a learned, abstract representation where key features are stored. By manipulating the latent space, new data complying with the captured patterns can be generated.

36
New cards

Summarise what the Encoder and Decoder of the Autoencoder do.

The encoder takes data as input and creates the latent space from it.

The decoder learns the mapping from the latent space to a reconstructed image.

37
New cards

What loss do we use for the Autoencoder?

knowt flashcard image
38
New cards

What are the uses and drawbacks of Autoencoders?

They are useful for denoising.

The drawback is that the latent space may not be continuous or easy to interpolate.

39
New cards

What is the key difference between a Variational Autoencoder and a standard Autoencoder?

Whereas an autoencoder maps an input to a point in the latent space, a variational autoencoder maps an input to a probability distribution. This promotes encoding to be normally distributed around the centre of the latent space and penalises deviations.

<p>Whereas an autoencoder maps an input to a point in the latent space, a variational autoencoder maps an input to a probability distribution. This promotes encoding to be normally distributed around the centre of the latent space and penalises deviations.</p>
40
New cards

What is the loss used in Variational Autoencoders?

The sum of the reconstruction loss used in standard autoencoders and a regularisation term. The term used is called the KL-divergence.

<p>The sum of the reconstruction loss used in standard autoencoders and a regularisation term. The term used is called the KL-divergence.</p>
41
New cards

What does KL divergence measure?

KL divergence measures how different a distribution is from another.

<p>KL divergence measures how different a distribution is from another.</p>
42
New cards

In Variational Autoencoders, why do we need re-parameterisation and how is it done?

As is, VAEs cannot be trained using standard gradient descent. That is why we model the distribution z as the mean summed with the element-wise product of the variance and random noise. This separates the stochastic part of the network from the deterministic part (because modelling z as mean + variance is deterministic) and allows for the gradient to be properly back-propagated.

43
New cards

What is the idea behind a Generative Adversarial Network (GAN)

We train two models - a generator G and a discriminator D. The generator aims to create as realistic of an image as it can. The discriminator aims to distinguish between fake generated images and real dataset images. We alternate training between the two until an equilibrium is reached.

44
New cards

What loss function is used for a GAN?

knowt flashcard image
45
New cards

What are the challenges of training a GAN?

In early training the part of the loss function that optimises D may saturate

Mode collapse occurs: the generator can only produce one type of image and the discriminator thinks its life-like so no further exploration occurs

Convergence failure: Equilibrium cannot be reached, usually because of one model training quicker than the other

46
New cards

What are the solutions for improved training in a GAN?

Euclidean loss between generated and example image

Add regularisation term to help with overfitting to certain images

Better hyperparameters

Leaky ReLU for D

Batch normalisation

Increase convolution stride in D

Scheduling G and D training to be balanced

47
New cards

What is a Conditional Generative Adversarial Network (CGAN)?

A GAN with additional conditions/information like text, classes or images which guide the generation process.

<p>A GAN with additional conditions/information like text, classes or images which guide the generation process.</p>
48
New cards

What is a Cycle-Consistent Generative Adversarial Network?

A modification of a GAN used for image-to-image translation tasks where paired data is not available. It learns to convert an image in set X into an image from set Y and vice versa.

<p>A modification of a GAN used for image-to-image translation tasks where paired data is not available. It learns to convert an image in set X into an image from set  Y and vice versa. </p>
49
New cards

What is a diffusion model?

It is a special case of a VAE. Rather than generating an image directly from the latent space, a diffusion model starts with noise and iteratively turns it into an image.

It is a parameterised Markov Chain where we learn the transitions of the chain that reverse a diffusion process (a diffusion process is a process by which we add noise to an image until the signal is no longer distinguishable)

<p>It is a special case of a VAE. Rather than generating an image directly from the latent space, a diffusion model starts with noise and iteratively turns it into an image.</p><p>It is a parameterised Markov Chain where we learn the transitions of the chain that reverse a diffusion process (a diffusion process is a process by which we add noise to an image until the signal is no longer distinguishable)</p>
50
New cards

What are the two processes involved in denoising diffusion models?

Forward Diffusion Process (Fixed)

Backward Diffusion Process (Generative)

51
New cards

How do we do the Forward Diffusion Process in a Diffusion Model?

We take an image and gradually add noise each iteration using the formula:

<p>We take an image and gradually add noise each iteration using the formula:</p>
52
New cards

What is the final equation for the Forward Process of the Diffusion Model?

knowt flashcard image
53
New cards

What is the maths behind the Backward Process in a Diffusion Model

knowt flashcard image
54
New cards

How do we approximate the posterior using an NN in the Backword Process of a Diffusion Model

knowt flashcard image
55
New cards

What architecture do Diffusion Models use?

knowt flashcard image
56
New cards

What is there to consider regarding the scheduling of the Forward Process in a Diffusion Model?

A linear schedule is used most often and works well in a lot of cases. For small images, the image can become pure gaussian noise very early in the process with a linear schedule, so a cosine schedule would work better then.

57
New cards

What changes must be made to a Diffusion Model to allow for Conditional Generation?

knowt flashcard image
58
New cards

What are the algos for a Diffusion Model

knowt flashcard image
59
New cards

What are the techniques for conditioning in Diffusion Models?

Concatenation

Cross-Attention

Adaptive Layer Norm

60
New cards

What are the strengths of Diffusion Models over other models?

Stable and high quality - Less prone to artifacts than GANs

Probabilistic foundation - Based on modelling data distribution through noise transition

Controllable generation - Can guide generation with text, class labels or images

61
New cards

How does Flow Matching differ from Diffusion Models?

Diffusion Models learn to reverse a noise adding process whereas Flow Matching learns to map noise directly into data.

62
New cards

What is a velocity field in the context of Flow Matching?

It is a space that defines the image as a field of velocities, one per pixel. Each pixels velocity specifies how the noise is transformed into meaningful image data.

63
New cards

In Flow Matching, how are points moved from the source distribution to the target distribution?

The model uses an ODE (Ordinary Differential Equation) to gradually move the noise until it matches the target distribution.

<p>The model uses an ODE (Ordinary Differential Equation) to gradually move the noise until it matches the target distribution. </p>
64
New cards

Explain Probability Paths in Flow Matching

The path pt that the datapoint takes during the transition from the source distribution to the target distribution. A velocity field ut generates pt if it moves probability mass without creating or destroying any. In other words, if the equation in the image holds.

<p>The path p<sub>t </sub>that the datapoint takes during the transition from the source distribution to the target distribution. A velocity field u<sub>t</sub> generates p<sub>t </sub>if it moves probability mass without creating or destroying any. In other words, if the equation in the image holds. </p>
65
New cards

Does the source distribution have to be a noisy distribution?

No, it does not. Flow Matching allows you to map between any two distributions.

66
New cards

What are the three ways to explicitly store data and their pros and cons?

Voxel Grid - Pros: Simple, works with 3D CNNs - Cons: Blocky results, N3 memory requirment

Point Cloud - Pros: Comes straight from LIDAR and depth sensors - Cons: No connectivity, holes, no inside or outside

Mesh - Pros: Compact, the graphics standard - Cons: Irregular structure, networks usually deform fixed template

67
New cards

What is the benefit or representing an image as a NN?

With a NN, we can store an image in constant memory no matter the image size. We can increase or decrease the resolution at will and sample sub-pixel values. We also get exact derivatives via auto-differentiation.

68
New cards

What is the difference between explicit and implicit representation?

Explicit - We state where an object is (voxel, triangle, point)

Implicit - We state a rule f that works for any point p inside, outside or on the surface

<p>Explicit - We state where an object is (voxel, triangle, point)</p><p>Implicit - We state a rule f that works for any point p inside, outside or on the surface</p>
69
New cards

What is an Implicit Neural Representation (INR)?

A small neural network that has been trained to represent an image such that given a pixel coordinate as input, it can output the corresponding pixel colour value.

<p>A small neural network that has been trained to represent an image such that given a pixel coordinate as input, it can output the corresponding pixel colour value. </p>
70
New cards

How would you represent an INR as a function?

knowt flashcard image
71
New cards

How do we represent the input, hidden layers and output in an INR?

knowt flashcard image
72
New cards

How do we train an INR?

We use each pixel as a training example and average over the MSE between the predicted colour and the actual colour.

<p>We use each pixel as a training example and average over the MSE between the predicted colour and the actual colour.</p>
73
New cards

What is the problem with using ReLU and tanh activation functions when modelling a NN to represent an image?

ReLU and tanh will model the smooth parts first. This is called Spectral Bias and it refers to low frequencies being learned first and high frequencies being learned slowly. Additionally, ReLU is piecewise linear, so almost all second derivatives are 0, and tanh is smooth and saturating, meaning we get low frequency fits and vanishing gradients.

74
New cards

What is the first fix for Spectral Bias?

Convert the signal to frequencies using the Fourier Transform. The network then can learn to combine waves instead of making them from scratch. Here, the variance acts as a dial: too low = blurry, too high = noisy

<p>Convert the signal to frequencies using the Fourier Transform. The network then can learn to combine waves instead of making them from scratch. Here, the variance acts as a dial: too low = blurry, too high = noisy</p>
75
New cards

What is the second fix for Spectral Bias?

SIREN: Use sine as the activation function. This not only always has a smooth second derivative since the derivative of sine is cos but it also represents the special detail within the image. The only downside is that it requires a special initialisation to train properly.

<p>SIREN: Use sine as the activation function. This not only always has a smooth second derivative since the derivative of sine is cos but it also represents the special detail within the image. The only downside is that it requires a special initialisation to train properly.</p>
76
New cards

What is the difference between occupancy and signed distance?

Occupancy is whether a point resides inside, outside or on the edge of a mesh. Signed distance is the distance from the edge of the mesh, with negative being inside and positive being outside.

77
New cards

How would you compute the signed distance?

f(p) = ||p|| - 1

d<0 inside surface

d=0 on surface

d>0 outside surface

78
New cards

What is the idea behind sphere tracing?

We can map points onto the surface of a sphere using the iterative formula:

pk+1 = pk + f(pk) . d

79
New cards

What is DeepSDF?

A singular NN that represents a shape space, meaning its weights encode many different shapes. Each code z is learned jointly with the weights, and if we have a new partial scan to add, we freeze the weights and optimise z and the prior will fill in the gaps.

<p>A singular NN that represents a shape space, meaning its weights encode many different shapes. Each code z is learned jointly with the weights, and if we have a new partial scan to add, we freeze the weights and optimise z and the prior will fill in the gaps.</p>
80
New cards

How do we train a DeepSDF model?

knowt flashcard image
81
New cards

What is an occupancy network?

A neural classifier which returns whether the probability of whether a point is inside or outside of the surface. Trained using binary cross entropy on sampled points labelled inside or outside. Itโ€™s also conditioned. It reconstructs the same object from a single image or scan.

82
New cards

How do we go from a field to a mesh?

Evaluate f

Where neighbouring cells donโ€™t match we have the surface

Place vertices on each crossed edge by linear interpolation

connect the vertices

83
New cards

What are 3 critical things required for an INR to solve an inverse problem?

A learned prior

A forward model (Knowing how the image was degraded)

More observations (several frames or viewpoints of the same scene)

84
New cards

How does an INR solve an inverse problem?

knowt flashcard image
85
New cards

What are the limits and solutions to INRs?

Slow to fit โ†’ Hash grid encodings

Every query runs the MLP โ†’ cache into grids

One network per signal โ†’ Learn a prior

Hard to edit โ†’ hybrid and explicit representations