Self-Attention

What is the goal of self-attention?

To compute a new representation of each token by allowing it to incorporate information from every other token in the sequence.

Why can't a token representation be computed independently in language?

Because the meaning of a token often depends on the surrounding context (e.g.

What is the input to a self-attention layer?

A sequence of token embeddings X=[x₁

What is the dimensionality of a token embedding?

Each embedding xᵢ belongs to ℝ^(d_model).

How is the Query vector computed?

Q = XW_Q.

How is the Key vector computed?

K = XW_K.

How is the Value vector computed?

V = XW_V.

What are W_Q

W_K and W_V?

Why are three different projection matrices used instead of one?

Because the model must solve three different optimization problems: deciding what to look for

What are the dimensions of W_Q?

W_Q ∈ ℝ^(d_model × d_k).

Are W_Q

W_K and W_V initialized with meaningful values?

Does the token embedding itself contain Query

Key and Value?

Why can the same embedding produce three different vectors?

Because it is multiplied by three different matrices (W_Q

What does the attention score measure?

The compatibility between one token's Query and another token's Key.

How is the attention score computed?

score(i

Why is the dot product used in attention?

Because it measures similarity or compatibility between two vectors in the learned space.

Why is the attention score divided by √d_k?

To prevent large dot products from making the Softmax saturate and producing tiny gradients.

What is the formula for scaled dot-product attention?

Attention(Q

What does the Softmax operation produce?

A probability distribution over all tokens.

What properties do attention weights satisfy?

Each weight is non-negative and all weights sum to 1.

How is the output representation computed?

As the weighted sum of all Value vectors.

Why is the output a weighted sum of Values instead of Keys?

Because Values contain the information to propagate after attention weights have been determined.

Do Query

Key and Value vectors have intrinsic meanings?

Why do people describe Query as 'What am I looking for?' Because it behaves that way after training

not because the vector explicitly contains a question.

Why do people describe Key as 'What information do I contain?' Because it behaves like a descriptor that other Query vectors compare against.

Why do people describe Value as 'The information to transfer'?

Because it is the only vector multiplied by the attention weights and propagated to the next layer.

What determines the meaning of W_Q

W_K and W_V?

Do the names Query

Key and Value matter mathematically?

What would happen if we renamed Query

Key and Value to A

What determines the role of Query?

Its output is used only on the left side of the dot product QKᵀ.

What determines the role of Key?

Its output is used only on the right side of the dot product QKᵀ.

What determines the role of Value?

Its output is used only after the attention weights have been computed.

Why can't Query become a Value vector?

Because Query disappears after computing attention scores and is never propagated to the next layer.

Why can't Key become a Value vector?

Because Key is only used to compute attention scores and is discarded afterward.

Why can't Value determine which tokens receive attention?

Because Value is not involved in computing the attention scores.

Which matrices influence the attention weights?

Only W_Q and W_K.

Which matrix influences the information that is transmitted?

Only W_V.

If the model attends to the wrong token

which matrices receive the strongest corrective gradients?

If the model attends to the correct token but extracts poor information

which matrix receives the strongest corrective gradients?

How does gradient descent know that W_Q should become a Query matrix?

It doesn't. W_Q specializes because its output only influences attention scores.

How does gradient descent know that W_V should become a Value matrix?

It doesn't. W_V specializes because its output only influences the transmitted information.

What does gradient descent optimize?

The model's loss function by adjusting parameters in the direction that reduces prediction error.

What is backpropagation?

The algorithm that computes how every parameter contributed to the loss using the chain rule.

What does ∂L/∂W_Q represent?

How changing W_Q would affect the loss.

What does ∂L/∂W_K represent?

How changing W_K would affect the loss.

What does ∂L/∂W_V represent?

How changing W_V would affect the loss.

Why are the gradients for W_Q

W_K and W_V different?

What mathematical rule allows gradients to flow backward through the network?

The chain rule of calculus.

Why does specialization emerge naturally?

Because each matrix repeatedly receives gradients corresponding only to its architectural role.

Does the transformer explicitly encode 'look for nouns' or 'find verbs' in Query vectors?

No. Such behaviors emerge because they minimize prediction loss.