Transformers Oral Exam Study Set

what does vocab_size represent in the embedding layer?

it represents the number of unique tokens the model can embed, typically determined during tokenization

what does embed_dim represent in the embedding layer?

it refers to the length of each embedding vector and is set as a model hyperparameter

how are embeddings initialized during training?

they are initialized randomly unless pretrained, and are updated as the model learns

why do transformers need positional embeddings?

because transformers lack inherent order awareness, and positional embeddings inject sequence information

are sinusoidal positional embeddings trainable?

no, they are fixed and not updated during training

what happens when the same token appears in the same position in two sequences?

the final embedding is the same since both word and positional embeddings are identical

why use multiple attention heads in a transformer?

to capture diverse relationships between tokens in parallel, improving representation power

how many attention heads are used in the given implementation?

8 attention heads are used

what is the dimensionality of each attention head in a 512d embedding?

64, since 512 is split equally among 8 heads

how many sets of Q, K, and V are logically created for multi-head attention?

8 sets logically, but only one set of Q, K, and V matrices is implemented and then split

how can shared Q, K, V projections still support diverse attention?

because each head processes a separate slice and learns patterns independently

what does Add & Norm do in a transformer block?

it adds residual connections and applies layer normalization to stabilize and preserve information

what is the structure of the transformer feedforward network?

two linear layers with ReLU in between, typically sized 512 → 2048 → 512

why are the same vectors used as Q, K, and V in the encoder?

because the encoder uses self-attention to relate tokens within the same input

how many encoder blocks are in the original transformer?

6 blocks, each with their own parameters and stacked sequentially

why is a look-ahead mask used in the decoder?

to prevent attention to future tokens during training and ensure autoregressive behavior

what is cross-attention in the decoder for?

it allows the decoder to focus on relevant encoder outputs for generating context-aware outputs

what does the final linear layer in the decoder do?

it projects the output to vocabulary size, allowing the model to score and choose the next token

why use softmax after the final decoder layer?

to convert scores into probabilities for each possible next token

how is the encoder output used in the decoder?

it’s passed as both key and value in cross-attention layers for every decoder block

what does model.eval() do?

it switches the model to evaluation mode, disabling training behaviors like dropout

why use torch.no_grad() during inference?

to save memory and computation by disabling gradient tracking

what happens to model parameters during inference vs. fine-tuning?

they stay fixed during inference but are updated during fine-tuning

why fine-tune instead of training from scratch on small datasets?

because small datasets can’t train a model effectively from scratch, but fine-tuning leverages existing knowledge

what is the main innovation of the transformer architecture?

it removes recurrence and convolutions, using only attention mechanisms

how is the transformer structured?

it follows an encoder-decoder architecture with stacked layers of attention and feed-forward networks

how many layers are in the transformer encoder?

6 identical layers

how many layers are in the transformer decoder?

6 identical layers

what are the two main components of each encoder layer?

multi-head self-attention and a feed-forward network

what additional component does each decoder layer have?

a cross-attention layer over the encoder output

why is masking used in the decoder?

to prevent attending to future tokens and preserve auto-regressive behavior

what is scaled dot-product attention?

it is a mechanism that computes attention scores by scaling dot products of queries and keys

why do we scale the dot product in attention?

to prevent large values that push softmax into regions with very small gradients

what is multi-head attention?

it applies attention multiple times in parallel with different learned projections and then combines the results

how many heads are used in multi-head attention?

8 heads

what are the advantages of multi-head attention?

it allows the model to attend to information from different representation subspaces simultaneously

what is the feed-forward network in each layer?

two linear layers with a ReLU activation in between, applied independently to each position

what is the input and output size of the feed-forward network?

both are 512, with an inner layer of size 2048

what is the role of the embedding layer?

to convert tokens into vectors of dimension 512

what is positional encoding used for?

to inject order information into the model since it lacks recurrence and convolution

how is positional encoding implemented?

using fixed sinusoidal functions of different frequencies

why were sinusoidal positional encodings chosen?

they help the model attend by relative position and generalize to longer sequences

how is self-attention different from convolution?

self-attention connects all positions with constant sequential operations, while convolution does not

what makes self-attention efficient?

it enables parallelization and has shorter paths for learning long-range dependencies