Deep Learning 330

Define learning for machines?

Any process by which a system improves performance from experience

What are the two stages of a traditional machine learning pipeline, and which one is fixed vs. learned?

Hand-crafted feature extractor (fixed) → classifier (learned)

What is the main weakness of traditional ML that deep learning addresses?

The front-end features are hand-designed and fixed, not learned from data

How do high-level and low-level features differ in a deep hierarchy?

High-level: more global and more invariant. Low-level: shared among categories

Shallow vs. deep models: what is the difference?

Shallow: fixed hand-crafted feature extractor + simple trainable classifier. Deep: stacked trainable feature-transform/classifier modules, all learned

What is predictive modeling, and why is it called function approximation?

Using historical data to predict on new data without answers; it approximates a mapping function f from inputs X to outputs y

Classification vs. regression: what is the difference?

Classification predicts a discrete class label; regression predicts a continuous quantity

What is unsupervised learning, and what are its main tasks?

Learning structure from unlabeled data: clustering, quantization/compression, dimensionality reduction, density estimation (e.g., anomaly detection), and learning to sample (e.g., GANs)

What is the 'semantic gap' in image classification?

The computer sees only a grid of numbers (0–255, e.g., 800x600x3 RGB), not the concept 'cat'

What makes image classification challenging, and why can't we hard-code a classifier?

Viewpoint variation, illumination, deformation, occlusion, background clutter, intraclass variation, fine-grained categories; there's no obvious algorithm for 'cat'

How does the Nearest Neighbor classifier work, and what are its training and testing costs for N examples?

Memorize all data and labels; predict the label of the most similar training example. Training O(1), testing O(N) — the wrong way around, since we can afford slow training but need fast testing

What is K-Nearest Neighbors, and why use K > 1?

Take a majority vote among the K closest training points. More neighbors smooth the decision boundary and reduce the effect of outliers

What are hyperparameters, and what is the right way to set them?

Choices set before learning rather than learned (e.g., K, distance metric).

Split data into train / validation / test: tune on validation, evaluate once on test

What is the curse of dimensionality?

To cover space uniformly, the number of training points needed grows exponentially with dimension

32x32 binary images: 2^1024 ≈ 10^308 possibilities

What is a linear classifier function

f(x,W) = Wx + b gives class scores

How can a linear classifier be interpreted, and when does it fail?

Visually: one 'template' per class (a single template can't capture multiple modes, e.g., a two-headed horse). Geometrically: hyperplanes carving up the space. Fails on XOR-like quadrants, ring-shaped classes, and multi-modal classes

What is a loss function, and what are the two steps for choosing a good W?

It quantifies how good the current classifier is (low = good, high = bad). Step 1: use a loss to score W. Step 2: find the W that minimizes it

If a W gives SVM loss L = 0, is it unique? Why does that matter?

No. 2W also gives L = 0, so the data loss alone can't choose between W and 2W; we need an extra preference (regularization)

What is regularization, and why use it?

A term that prevents the model from doing too well on training data. It expresses preferences among models, favors simpler ones that generalize, and avoids overfitting to noise

How does optimization use the gradient to minimize the loss?

1. Find the slope: The gradient points in the direction of the steepest increase in loss (uphill).

2. Go the opposite way: The optimizer moves in the negative gradient direction (downhill) to reduce error the fastest.

3. Take controlled steps: It multiplies that direction by a learning rate to control step size, repeating until the slope hits zero (the bottom).

What are gradient descent and stochastic gradient descent (SGD)?

GD steps repeatedly in the negative gradient direction. SGD approximates the full sum over N examples with a minibatch

Why aren't linear classifiers powerful enough, and how do feature transforms help?

One template per class can't capture multiple modes. A feature transform can make data linearly separable

What are depth, width, and activation functions, and what happens with no activation function?

Depth = number of layers; width = size of each layer. The activation adds nonlinearity. With none, the network collapses to a linear classifier.

What are locally connected and convolutional layers, and what is spatial filtering?

Locally connected: each unit sees a small filter-sized patch (10x10 filter on 200x200 with 40K units ≈ 4M parameters).

Convolutional: share the same learned kernel across locations. Filtering slides a kernel over the image

How does backpropagation work on a computational graph?

Forward pass computes outputs; backward pass computes derivatives via the chain rule

What are the patterns of gradient flow for add, multiply, and max gates?

Add: gradient distributor(copies upstream gradient to all inputs)

Multiply: gradient switcher(swaps the inputs' values to scale it)

Max: gradient router(sends gradient only to the larger input)

What is spatial filtering?

Replace each pixel with a weighted sum of its neighborhood (kernel)

What are the components of a convolutional network?

Fully-connected layers and activation functions (as before), plus convolution layers, pooling layers, and normalization

What happens if we stack convolution layers, and why add ReLU between them?

Without nonlinearity, two stacked convolutions are just another convolution

Conv → ReLU → Conv → ReLU ... builds a deep, nonlinear feature hierarchy

e.g. 3→6→10→12 channels

What is padding and why is it important to the output size of a convolution

Without padding, feature maps shrink each layer

What is strided convolution, and what is the general output size formula?

Move the filter S pixels at a time to downsample.

Output = (W − K + 2P)/S + 1.

What is pooling, and what are its properties?

Non-linear downsampling of each channel. Reduces size and computation, helps control overfitting, gives invariance to small shifts, and has no learnable parameters

How did ImageNet challenge winners progress, and what was the trend?

Trend: deep networks replaced hand-crafted features, and error fell as networks got deeper

What did batch normalization add to the architecture toolbox?

A differentiable normalization layer (usually after conv/FC, before the nonlinearity) that makes deep networks much easier to train, allowing higher learning rates and less sensitivity to initialization.

Why do deep learning frameworks and GPUs/TPUs matter

GPUs have thousands of slower cores, ideal for parallel work like matrix multiplication TPUs are matrix-multiply hardware. Frameworks give rapid prototyping, automatic gradients, and efficient GPU/TPU execution.

Static vs. dynamic graphs: what are the trade-offs?

static computational graphs are defined and compiled once before execution. Hard to debug

dynamic computational graphs are built on-the-fly during each forward pass of the model. Easy to debug

Problems with Sigmoid activation function

squashes to [0,1] and has 3 problems:

1. saturated neurons kill gradients

2. outputs aren't zero-centered (gradients on w are all positive or all negative, giving zig-zag updates)

3. exp() is costly.

Problems with Tanh activation function

Is zero-centered ([-1,1]) but still kills gradients when saturated

ReLU activation function

max(0,x) not zero-centered, and can 'die' (zero gradient for x < 0, so never updates)

What is data augmentation?

Transform each training image randomly (flips, random crops and scales, translation, rotation, stretching, shearing, lens distortions) while keeping the label.

What are the problems with plain SGD, and how do momentum and Nesterov momentum help?

SGD gets stuck at local minima/saddle points (zero gradient) and minibatch gradients are noisy. SGD+Momentum builds up a 'velocity' as a running mean of gradients and steps along it. Nesterov momentum 'looks ahead' to where the velocity would take us, computes the gradient there, and mixes it with the velocity

What learning rate schedules exist, and how long should you train?

Start with a large learning rate and decay it: step, cosine, linear, inverse sqrt, or constant. Early stopping: stop when validation accuracy decreases, or train long but always keep the best-on-validation snapshot

How do you read learning curves?

Loss flat from the start: bad initialization is a prime suspect.

Loss plateaus: try LR decay.

Loss still dropping when LR was decayed: you decayed too early.

Accuracy still rising: train longer.

Huge train/val gap: overfitting

No gap: underfitting

Why do we need RNNs instead of feedforward networks?

Feedforward nets need fixed-length inputs and outputs, but tasks like speech recognition and time-series prediction need to store and use context. A fixed context window can't work since a longer sample can always appear

What kinds of sequence tasks do RNNs handle?

One-to-many: image captioning

Many-to-one: video classification

Many-to-many: per-frame video labeling

What is the key idea of an RNN?

An internal state updated as the sequence is processed, using a recurrence at every time step with the same function and same parameters at every step

What is backpropagation through time (BPTT)

treat the unrolled network as one big feedforward network that takes the entire sequence, run forward to compute the loss and backward for gradients; the shared weights get gradients summed (accumulated) over all time steps.

What are the issues with vanilla RNNs when training?

Backprop through many time steps multiplies many matrices, and a product of matrices can shrink to zero (vanishing gradients, gradient clipping) or blow up to infinity (exploding gradients, change RNN Architecture).

What is the key architectural change in an LSTM compared to a vanilla RNN?

It adds a memory cell state c_t that is not subject to matrix multiplication or squishing, in addition to the hidden state h_t.

Gates, computed from, control what is written to, erased from and read out of the cell, so gradients avoid decay

What are multilayer RNNs and GRUs, and how do they change the basic RNN?

Multilayer (stacked) RNNs: pass the hidden states of one RNN layer as the inputs to the next, adding depth in addition to time.

GRU: a simpler gated variant that gets rid of the separate cell state and merges the 'forget' and 'output' gates into an 'update' gate

How is attention generalized into a standalone layer (queries, keys, values)?

turning information routing into a soft, differentiable database lookup using three learned linear projections: queries keys and values

What is a self-attention layer

a neural network layer where every element in a single input sequence attends to every other element in that same sequence to build context-aware representations

What is masked self-attention, and where is it used?

a variant of self-attention that prevents tokens from attending to future positions in a sequence. It is primarily used in autoregressive text generation model and Transformer decoders to prevent the model from "cheating" by seeing upcoming words during training.

What is multihead self-attention?

An extension of self-attention that runs multiple independent attention mechanisms (called "heads") in parallel on the same input sequence.

Instead of relying on a single set of attention weights to summarize a sentence, it allows a model to simultaneously focus on different types of relationships such as grammar, semantic meaning, and word proximity at the same time

What does a transformer block do?

It takes a sequence of token embeddings, lets the tokens share context with one another, and processes that updated information to produce richer representations of the exact same shape.

What is the difference between supervised and unsupervised learning?

Supervised: data (x, y) with labels; goal is to learn a function mapping x → y

Unsupervised: just data x, no labels; goal is to learn some underlying hidden structure of the data(dimensionality reduction).

What is density estimation, and what does a density function p(x) mean?

Given a large dataset, learn a probability distribution that gives high probability to the data we have seen and low probability to data we have not seen. A density function p(x) assigns a positive number to each possible x

How do discriminative and generative models differ?

Discriminative: learn p(y|x); the possible labels for each input compete for probability mass, but must output a label for unreasonable input

Generative: learn p(x); all possible images compete for probability mass, which requires deep image understanding, and the model can 'reject' unreasonable inputs by assigning them small values

What is an autoencoder

An unsupervised method for learning feature vectors from raw data x without labels: an encoder maps the input to lower-dimensional features, and a decoder reconstructs the input from them

What is the difference of a Variational Autoencoder compared to a regular autoencoder

A standard autoencoder compresses data into fixed, deterministic points for reconstruction, while a VAE maps data into continuous probability distributions so it can generate brand-new data

What is the idea behind Generative Adversarial Networks (GANs)?

train two neural networks simultaneously in a competitive, zero-sum game so that one network learns to create realistic synthetic data from scratch. Pits a Generator against a Discriminator

Problems with GAN training

Stability: parameters can oscillate or diverge, the generator loss does not correlate with sample quality, and behavior is very sensitive to hyperparameters

What is a conditional GAN(CGAN)

Allows you to direct and control the data generation process by feeding additional information (such as class labels, tags, or source images) into both the generator and the discriminator

What are the main computer vision tasks, and how is object detection defined?

Classification: one label for the image, no spatial extent.

Semantic segmentation: label every pixel

Object detection: multiple objects, each with a category and a box.

Instance segmentation: multiple objects with per-pixel masks.

Detection task: input a single RGB image; output a set of detected objects, and for each predict a category label from a fixed, known set and a bounding box

What makes object detection challenging?

Multiple outputs: need to output a variable number of objects per image

Multiple types of output: must predict 'what' (category label) and 'where' (bounding box)

Large images: typically needs higher resolution for detection

What is the sliding window approach for multiple objects, and why is it impractical?

Apply a CNN to many different crops of the image, classifying each crop as one of the object categories or background. Problem: the number of possible boxes is huge so we can’t evaluate them all.

What are region proposals, and how does R-CNN use them?

Sliding window over all boxes is infeasible, so find a small set of boxes likely to cover all objects, often with heuristics; fast to run.

How do we compare boxes (IoU) and remove duplicate detections (NMS)?

IoU = area of overlap between predicted and ground-truth box divided by area of their union.

NMS: select the next highest-scoring box, eliminate lower-scoring boxes with IoU > a threshold. Problem: NMS may eliminate good boxes when objects are overlapping

How is an object detector evaluated with mean Average Precision(mAP)

evaluates an object detector by measuring both localization accuracy (how well the predicted bounding boxes match the ground truth) and classification accuracy (whether the object is assigned the correct class label)

What does Fast R-CNN change compared to 'slow' R-CNN?

several major bottlenecks of the original ("slow") R-CNN by moving from independent regional processing to a single, shared-computation architecture

How does Faster R-CNN make region proposals learnable?

replaces hand-crafted, non-learnable region proposal methods with a deep, learnable neural network called the Region Proposal Network (RPN)

How many learnable parameters does a 2x2 max pooling layer with stride 2 have?

0 learnable parameters because pooling layers don't have them. These layers apply a fixed operation (ex. taking a max value) so there are no weights to learn

Match the probabilistic distribution to the appropriate model.

Discriminative model: p(y|x)

Generative model: p(x)

Conditional Generative Model: p(x|y)

Why aren’t RNN’s parallelizable?

Because they process data sequentially, each step depends directly on the output of the previous step

What problem motivated the development of residual blocks?

developed to solve the degradation problem in very deep neural networks, and inherit identity functions so the deep models can mimic shallower ones

Match these three core Pytorch Concepts: Tensor, Autograd and Module

Tensor = like a numpy array but it can run on a GPU.

Autograd = setting that makes PyTorch build a computational graph and compute gradients automatically .

Module = neural network layer that may hold state or learnable weights

What are the five components of CNN

1. fully connected layers

2. activation layers

3. convolution layers

4. pooling layer

5. Normalization

What is the difference between deep learning and traditional machine learning?

Traditional machine learning requires hand-crafted features that are then provided to the model for training; whereas, deep learning is end-to-end. This means that the neural network automatically learns the relevant features directly from raw data and uses them to make predictions.

What are the 4 techniques that GoogleNet uses for efficiency

1. aggressive stem

2. global average pooling

3. inception module

4. auxiliary classifier

Is the result of a Laplacian filtering an enhanced image compared to the original?

No. The Laplacian filter highlight edges and other discontinuities, but to obtain the enhanced image some extra work is required. Original image + Laplacian filtering = ENHANCED IMAGE.

By using pooling layers we can control overfitting? (Y/N)

Yes, because it reduced the number of parameters and the number of computations required, therefore controlling overfitting.

Why it is not a good idea to use sigmoid or tanh as activation functions in general?

In both saturated neurons kill the gradient. In addition for the sigmoid function: this is not zero centered and it involves exp() which make it a bit compute expensive.

How do you know from the loss function or graph if a model is overfitting or underfitting?

Overfitting: The training loss decreases, while the validation loss decreases initially, bottoms out, then starts increasing. Fix: Regularization, stop training early

Underfitting: Both the training loss and the validation loss remain high and plateau (flatline) without improving much, even as training continues. Fix: Increase model complexity, train longer

Good fit: Both training and validation decrease together during early training and eventually flatten out at a low value with a small, stable gap between them

What does batch normalization do?

standardizes the outputs (activations) of a hidden layer for each mini-batch during training, this stabilizes training allowing for higher learning rates

Why do we use a validation set separate from the test set?

Tuning hyperparameters and picking checkpoints on the validation set makes its score slightly optimistic. The test set stays untouched until the end, so it gives an unbiased estimate on new data.

What does dropout do?

randomly deactivates a fraction of neurons during the training process to prevent a neural network from overfitting

What is transfer learning?

Taking a model pretrained on a big dataset like ImageNet and reusing it for a new task.

Why do Transformers train faster than RNNs?

Self-attention processes the whole sequence in parallel, while an RNN has to go one step at a time

What is Predictive Modeling?

uses historical data, statistical techniques, and algorithms to forecast future events, behaviors, or unknown outcomes

Where do we draw the line to say we both have a good generator and a good discriminator, as their being “good” is making the other look "bad"?

The ideal stopping point for a Generative Adversarial Network is Nash equilibrium where the discriminator can no longer tell real data from fake data and simply guesses with 50% accuracy

GAN is a zero-sum game, a rising loss for one network does not mean it is failing; it usually means its opponent is getting stronger.

Compare masked vs multihead self-attention layers

Masked: Restricts what tokens a position can attend to. It applies a causal mask so that token at position i can only attend to past and current tokens

Multi: Splits the query (Q), key (K), and value (V) projections into multiple lower-dimensional "heads" that run in parallel. This allows the model to jointly attend to information from different representation subspaces and capture diverse relationships simultaneously

Why are standard fully connected layers poorly suited for processing raw, high-resolution images directly?

Fully connected layers require flattening the image into a single column vector, which destroys the 2D spatial structure and local spatial correlations of the pixels.

How do LSTM and GRU architectures solve the vanishing gradient problem at a structural level compared to a vanilla RNN?

Unlike vanilla RNNs that rely solely on repeated matrix multiplications, LSTMs and GRUs use gated architectures with additive updates to their cell or hidden states.

What is the "dying ReLU" problem, and what activation functions are commonly used to fix it?

When a ReLU neuron receives negative inputs, it outputs 0 and has a gradient of 0, meaning it stops updating its weights and can permanently "die." Activation functions like Leaky ReLU, GELU, or Swish solve this by allowing a small, non-zero gradient for negative inputs.

After training an autoencoder on unlabeled data, how do you use it to transfer knowledge to a supervised downstream task?

You throw away the decoder and keep the trained encoder. You then use the encoder to initialize a supervised model by attaching a classifier on top of its learned features (z), and fine-tune the encoder jointly with the classifier on your smaller, labeled dataset

Explain the consequences of decaying learning rate too quickly and decaying learning rate too slowly during training.

Decaying learning rate too quickly can result in slow learning, which can lead to long training times and convergence to a suboptimal model.

Decaying learning rate too slowly can cause the model to overshoot the optimal model repeatedly, and the parameters will struggle to converge.

What methods can we use to down sample and up sample and why do we use them?

For down sampling we can use pooling or strided convolutions. It helps reduce the dimensionality while keeping main characteristics to reduce the amount of memory and computation needed

For upsampling we can use unpooling. It increases our dimensionality and gives more resolution needed for deeper networks.

What are the four gates in an LSTM and what is the main purpose of an LSTM

The LSTM helps to "remember" long-term information about sequential data. The four gates are:

1. Input Gate (whether to write to cell)

2. Forget Gate (whether to erase cell)

3. Output gate (how much to reveal to cell)

4. Gate Gate (how much to write to cell)

What are the differences between semantic segmentation and instance segmentation?

semantic: labels pixels by class, merging touching objects into the same region. includes background regions

instance: separately labels each object, defines boundaries between, focuses on countable foreground objects instead of background information

What are the query, key, and value in self-attention?

Each token emits a query (what it's looking for), a key (what it advertises), and a value (what it contributes)