Transformers Oral Exam Study Set

what does vocab_size represent in the embedding layer?

it represents the number of unique tokens the model can embed, typically determined during tokenization

what does embed_dim represent in the embedding layer?

it refers to the length of each embedding vector and is set as a model hyperparameter

how are embeddings initialized during training?

they are initialized randomly unless pretrained, and are updated as the model learns

why do transformers need positional embeddings?

because transformers lack inherent order awareness, and positional embeddings inject sequence information

are sinusoidal positional embeddings trainable?

no, they are fixed and not updated during training

what happens when the same token appears in the same position in two sequences?

the final embedding is the same since both word and positional embeddings are identical

why use multiple attention heads in a transformer?

to capture diverse relationships between tokens in parallel, improving representation power

how many attention heads are used in the given implementation?

8 attention heads are used

what is the dimensionality of each attention head in a 512d embedding?

64, since 512 is split equally among 8 heads

how many sets of Q, K, and V are logically created for multi-head attention?

8 sets logically, but only one set of Q, K, and V matrices is implemented and then split

how can shared Q, K, V projections still support diverse attention?

because each head processes a separate slice and learns patterns independently

what does Add & Norm do in a transformer block?

it adds residual connections and applies layer normalization to stabilize and preserve information

what is the structure of the transformer feedforward network?

two linear layers with ReLU in between, typically sized 512 → 2048 → 512

why are the same vectors used as Q, K, and V in the encoder?

because the encoder uses self-attention to relate tokens within the same input

how many encoder blocks are in the original transformer?

6 blocks, each with their own parameters and stacked sequentially

why is a look-ahead mask used in the decoder?

to prevent attention to future tokens during training and ensure autoregressive behavior

what is cross-attention in the decoder for?

it allows the decoder to focus on relevant encoder outputs for generating context-aware outputs

what does the final linear layer in the decoder do?

it projects the output to vocabulary size, allowing the model to score and choose the next token

why use softmax after the final decoder layer?

to convert scores into probabilities for each possible next token

how is the encoder output used in the decoder?

it’s passed as both key and value in cross-attention layers for every decoder block

what does model.eval() do?

it switches the model to evaluation mode, disabling training behaviors like dropout

why use torch.no_grad() during inference?

to save memory and computation by disabling gradient tracking

what happens to model parameters during inference vs. fine-tuning?

they stay fixed during inference but are updated during fine-tuning

why fine-tune instead of training from scratch on small datasets?

because small datasets can’t train a model effectively from scratch, but fine-tuning leverages existing knowledge