1/84
A set of Advanced Visual AI flashcards made by yours truly
Explain the Inverse Problem
The inverse problem is the problem of taking a degraded image and restoring it. We represent the degraded image as a degradation function H applied to the original image with added noise. The process then involves estimating the original image from the observed data by reversing the effects of degradation and noise.

What is the Wiener filter?
A loss function which minimises the Mean Square Error (MSE) between an output and a ground truth


What are the names of the four components of the Bayes Formula?

What does the Bayesian Estimator do?
Minimises the loss function averaged over the posterior given the evidence.

What is the Maximum a Posteriori (MAP) estimator?
The Bayesian Estimator with a hit or miss loss function

What are the steps to solve for an estimator?
Take the logs => differentiate with respect to ฮธ => set equal to 0 => rearrange for ฮธ
What is the variational formulation of the Bayesian?


The Constrained Least Squares regularisation technique consists of choosing x that minimises the Lagrangian. What are each of the terms?
ฮป is the regularisation parameter, controlling the relative contribution of both main terms. ||๐ฆโ๐ป๐ฅ||2 is the data attachment term. ||๐ถ๐ฅ||2 is the prior term.
What is the solved form of the Constrained Least Squares?

What is the idea behind Alternate Direction of Multipliers Method (ADMM)
We split a problem which is too hard to solve into separate smaller problems through an additional term. We then iteratively minimise each new variable until we reach a minimum.

What is step 1 of ADMM?
Update x

What is step 2 of ADMM?
Update v

What is step 3 of ADMM?
Update u

What is the idea behind Deep Image Prior (DIP)
We optimise a neural network to reconstruct an image from noise. We train it enough for the weights to reconstruct the structure of the image but not enough to also reconstruct the noise.

In Deep Image Prior (DIP) what are the key points regarding the input noise, ground truth and output image?
We use the same input noise each iteration. The ground truth is the degraded image we are trying to fix. The output image is a surjective mapping of noise onto the image. The NN will reconstruct the structure of the image before it reconstructs the noise, so if training is stopped at the right time, we will end up with a denoised image as output.
What is the maths behind Deep Image Prior (DIP)

What is the key idea behind Regularisation by Denoising (RED)

What is the key idea behind DeepRED

What is the maths behind DeepRED?

What is the idea behind Regularisation by Artifact Removal (RARE)
We replace the denoiser in RED with a CNN which we train for more general artifact removal. The artifact-removal network ๐ ๐, where ๐ denotes the weights of the CNN can learn the imaging prior despite having no fully-sampled ground truth.
What does the estimator look like for RARE?

What is the maths behind RARE?

What is the idea behind Deep Unfolding?
We take an iterative algorithm and turn it into a DNN by designing each layer to represent a single iteration of the algorithm.
What are the four steps involved in creating a Deep Unfolding network?
Find a suitable iterative optimisation algorithm for the problem
Fix the number of iterations (and hence the number of layers)
Design the layers to imitate the free parameters of each iteration such that they can be trained
Train the network
What is the idea behind ISTA?
An iterative image reconstruction algorithm which solves the sparse coding problem using two steps:
A Gradient Descent Step where we adjust the image to match the raw data
A Soft-Thresholding Step where we force small/noisy pixel values to zero to keep the image data sparse
What is LISTA?
Learned ISTA. It is the ISTA algorithm unfolded into the layers of a NN.
What is the maths behind the training of LISTA?

What are the benefits of LISTA over ISTA?
LISTA may achieve higher efficiency than ISTA
Useful when W (Weights representing the degradation matrix H) is hard to determine
Number of trained layers L can be an order of magnitude smaller than the number of iterations required of ISTA to converge
What are the three types of image fusion?
Pixel level fusion
Region level fusion
Compressive fusion
In image fusion, what are the pixel level approaches?
Maximum selection scheme
Average scheme
Weighted average scheme
How do we represent image fusion as an inverse problem?
We define two degradation functions that have been applied to the same image to create two degraded ones.

What is the maths behind image fusion?

What is the key concept behind generative modelling?
IDEA: Can we build a NN that can capture the distribution of the data?
What does an Autoencoder do?
It takes images as inputs and encodes them in a lower dimension latent space from which it aims to recreate the images via the decoder.
What is the latent space?
The latent space is a low dimensional representation z of the data x which captures the essential structure. It is a learned, abstract representation where key features are stored. By manipulating the latent space, new data complying with the captured patterns can be generated.
Summarise what the Encoder and Decoder of the Autoencoder do.
The encoder takes data as input and creates the latent space from it.
The decoder learns the mapping from the latent space to a reconstructed image.
What loss do we use for the Autoencoder?

What are the uses and drawbacks of Autoencoders?
They are useful for denoising.
The drawback is that the latent space may not be continuous or easy to interpolate.
What is the key difference between a Variational Autoencoder and a standard Autoencoder?
Whereas an autoencoder maps an input to a point in the latent space, a variational autoencoder maps an input to a probability distribution. This promotes encoding to be normally distributed around the centre of the latent space and penalises deviations.

What is the loss used in Variational Autoencoders?
The sum of the reconstruction loss used in standard autoencoders and a regularisation term. The term used is called the KL-divergence.

What does KL divergence measure?
KL divergence measures how different a distribution is from another.

In Variational Autoencoders, why do we need re-parameterisation and how is it done?
As is, VAEs cannot be trained using standard gradient descent. That is why we model the distribution z as the mean summed with the element-wise product of the variance and random noise. This separates the stochastic part of the network from the deterministic part (because modelling z as mean + variance is deterministic) and allows for the gradient to be properly back-propagated.
What is the idea behind a Generative Adversarial Network (GAN)
We train two models - a generator G and a discriminator D. The generator aims to create as realistic of an image as it can. The discriminator aims to distinguish between fake generated images and real dataset images. We alternate training between the two until an equilibrium is reached.
What loss function is used for a GAN?

What are the challenges of training a GAN?
In early training the part of the loss function that optimises D may saturate
Mode collapse occurs: the generator can only produce one type of image and the discriminator thinks its life-like so no further exploration occurs
Convergence failure: Equilibrium cannot be reached, usually because of one model training quicker than the other
What are the solutions for improved training in a GAN?
Euclidean loss between generated and example image
Add regularisation term to help with overfitting to certain images
Better hyperparameters
Leaky ReLU for D
Batch normalisation
Increase convolution stride in D
Scheduling G and D training to be balanced
What is a Conditional Generative Adversarial Network (CGAN)?
A GAN with additional conditions/information like text, classes or images which guide the generation process.

What is a Cycle-Consistent Generative Adversarial Network?
A modification of a GAN used for image-to-image translation tasks where paired data is not available. It learns to convert an image in set X into an image from set Y and vice versa.

What is a diffusion model?
It is a special case of a VAE. Rather than generating an image directly from the latent space, a diffusion model starts with noise and iteratively turns it into an image.
It is a parameterised Markov Chain where we learn the transitions of the chain that reverse a diffusion process (a diffusion process is a process by which we add noise to an image until the signal is no longer distinguishable)

What are the two processes involved in denoising diffusion models?
Forward Diffusion Process (Fixed)
Backward Diffusion Process (Generative)
How do we do the Forward Diffusion Process in a Diffusion Model?
We take an image and gradually add noise each iteration using the formula:

What is the final equation for the Forward Process of the Diffusion Model?

What is the maths behind the Backward Process in a Diffusion Model

How do we approximate the posterior using an NN in the Backword Process of a Diffusion Model

What architecture do Diffusion Models use?

What is there to consider regarding the scheduling of the Forward Process in a Diffusion Model?
A linear schedule is used most often and works well in a lot of cases. For small images, the image can become pure gaussian noise very early in the process with a linear schedule, so a cosine schedule would work better then.
What changes must be made to a Diffusion Model to allow for Conditional Generation?

What are the algos for a Diffusion Model

What are the techniques for conditioning in Diffusion Models?
Concatenation
Cross-Attention
Adaptive Layer Norm
What are the strengths of Diffusion Models over other models?
Stable and high quality - Less prone to artifacts than GANs
Probabilistic foundation - Based on modelling data distribution through noise transition
Controllable generation - Can guide generation with text, class labels or images
How does Flow Matching differ from Diffusion Models?
Diffusion Models learn to reverse a noise adding process whereas Flow Matching learns to map noise directly into data.
What is a velocity field in the context of Flow Matching?
It is a space that defines the image as a field of velocities, one per pixel. Each pixels velocity specifies how the noise is transformed into meaningful image data.
In Flow Matching, how are points moved from the source distribution to the target distribution?
The model uses an ODE (Ordinary Differential Equation) to gradually move the noise until it matches the target distribution.

Explain Probability Paths in Flow Matching
The path pt that the datapoint takes during the transition from the source distribution to the target distribution. A velocity field ut generates pt if it moves probability mass without creating or destroying any. In other words, if the equation in the image holds.

Does the source distribution have to be a noisy distribution?
No, it does not. Flow Matching allows you to map between any two distributions.
What are the three ways to explicitly store data and their pros and cons?
Voxel Grid - Pros: Simple, works with 3D CNNs - Cons: Blocky results, N3 memory requirment
Point Cloud - Pros: Comes straight from LIDAR and depth sensors - Cons: No connectivity, holes, no inside or outside
Mesh - Pros: Compact, the graphics standard - Cons: Irregular structure, networks usually deform fixed template
What is the benefit or representing an image as a NN?
With a NN, we can store an image in constant memory no matter the image size. We can increase or decrease the resolution at will and sample sub-pixel values. We also get exact derivatives via auto-differentiation.
What is the difference between explicit and implicit representation?
Explicit - We state where an object is (voxel, triangle, point)
Implicit - We state a rule f that works for any point p inside, outside or on the surface

What is an Implicit Neural Representation (INR)?
A small neural network that has been trained to represent an image such that given a pixel coordinate as input, it can output the corresponding pixel colour value.

How would you represent an INR as a function?

How do we represent the input, hidden layers and output in an INR?

How do we train an INR?
We use each pixel as a training example and average over the MSE between the predicted colour and the actual colour.

What is the problem with using ReLU and tanh activation functions when modelling a NN to represent an image?
ReLU and tanh will model the smooth parts first. This is called Spectral Bias and it refers to low frequencies being learned first and high frequencies being learned slowly. Additionally, ReLU is piecewise linear, so almost all second derivatives are 0, and tanh is smooth and saturating, meaning we get low frequency fits and vanishing gradients.
What is the first fix for Spectral Bias?
Convert the signal to frequencies using the Fourier Transform. The network then can learn to combine waves instead of making them from scratch. Here, the variance acts as a dial: too low = blurry, too high = noisy

What is the second fix for Spectral Bias?
SIREN: Use sine as the activation function. This not only always has a smooth second derivative since the derivative of sine is cos but it also represents the special detail within the image. The only downside is that it requires a special initialisation to train properly.

What is the difference between occupancy and signed distance?
Occupancy is whether a point resides inside, outside or on the edge of a mesh. Signed distance is the distance from the edge of the mesh, with negative being inside and positive being outside.
How would you compute the signed distance?
f(p) = ||p|| - 1
d<0 inside surface
d=0 on surface
d>0 outside surface
What is the idea behind sphere tracing?
We can map points onto the surface of a sphere using the iterative formula:
pk+1 = pk + f(pk) . d
What is DeepSDF?
A singular NN that represents a shape space, meaning its weights encode many different shapes. Each code z is learned jointly with the weights, and if we have a new partial scan to add, we freeze the weights and optimise z and the prior will fill in the gaps.

How do we train a DeepSDF model?

What is an occupancy network?
A neural classifier which returns whether the probability of whether a point is inside or outside of the surface. Trained using binary cross entropy on sampled points labelled inside or outside. Itโs also conditioned. It reconstructs the same object from a single image or scan.
How do we go from a field to a mesh?
Evaluate f
Where neighbouring cells donโt match we have the surface
Place vertices on each crossed edge by linear interpolation
connect the vertices
What are 3 critical things required for an INR to solve an inverse problem?
A learned prior
A forward model (Knowing how the image was degraded)
More observations (several frames or viewpoints of the same scene)
How does an INR solve an inverse problem?

What are the limits and solutions to INRs?
Slow to fit โ Hash grid encodings
Every query runs the MLP โ cache into grids
One network per signal โ Learn a prior
Hard to edit โ hybrid and explicit representations