LLM Midterm Quiz 3

We know query key and value concepts were borrowed from the domain of information retrieval and databases. Please match the respective concept.

query → store , value →retrieve, key→ search

while using softmax function if dim is set to -1 then it indicates that the normalization is applied along the 

last dimension

Which of the following are true w.r.t RNNs?

All of the above

Which of the following is accurate with respect to attention weights from a softmax function?

higher weights indicate greater importance 

Both weight parameters and attention weights represent the same entities in a neural network. T/F

False

During dropout, randomly selected ______ layer units are ignored 

hidden

Before applying the softmax function, mask the attention scores with 

negative infinity

Normalized attentions mean the attention weights in 

each row sum to 1

Which of the following represent the meaning of the dot product

quantifies how closely two vectors are aligned, measures similarity, higher dot product implies à greater degree of alignment, multiplying two vectors element-wise and then summing the products

Softmax takes attention scores and returns attention weights that are always positive t/f

True

Self-attention module allows each position in the input sequence to consider the relevancy of all other positions in the same sequence T/F

True

Masked self attention, also known as causal attention used in LLMs allows a model to only consider _____ and ____ tokens or input in a sequence during text generation.

previous and current

attention weights

normalized dot products between the inputs

attention score

dot products between the inputs

context vectors

weighted sum over the inputs

To avoid information leakage, the distribution of attention weights is calculated only among the unmasked positions

True

Large dot products can result in very ______ gradients during backpropagation

small