Temporal models

Why does sequential recurrence limit parallel computation over a trajectory?

A recurrent model must compute step t from an earlier hidden state before computing step t+1, creating a serial dependency chain across time.

Why can recurrent models struggle with very long-range dependencies?

Information and gradients must pass through many sequential transitions, so distant signals can be weakened, distorted, or hard to optimize.

What core mechanism lets a Transformer model dependencies without recurrence?

Attention directly relates sequence elements by computing content-dependent interactions between queries and keys and aggregating values.

What training-time parallelism advantage does a Transformer have over a recurrent model?

All positions in an unmasked encoder, and all causally masked positions during teacher-forced decoder training, can be processed in parallel within a layer.

How does the computational path between two distant tokens differ in self-attention and recurrence?

A self-attention layer can connect the tokens directly, whereas a recurrent model normally passes information through a chain whose length grows with temporal distance.

What sequence mapping was the original encoder-decoder Transformer designed to learn?

It maps an input sequence (x₁, x₂, …, x_T) to an output sequence, originally for sequence-to-sequence tasks such as machine translation.

Which two main sublayers does each encoder layer contain in the supplied description?

A multi-head self-attention sublayer and a two-layer position-wise feed-forward network.

What additional sublayer distinguishes the original Transformer decoder from the encoder?

Encoder-decoder cross-attention, which lets decoder states attend to encoder outputs.

What does self-attention mean in a Transformer encoder?

Queries, keys, and values are all derived from the same sequence representation, so each position aggregates information from positions in that sequence.

What is the difference between modeling a trajectory in observation space and in latent space?

Observation-space modeling operates on observed states or sensory tokens; latent-space modeling operates on learned compressed representations.

Why is an embedding layer used before Transformer processing?

It maps each discrete or raw input token into a continuous d_model-dimensional representation on which learned projections and neural transformations can operate.

Why must a non-recurrent Transformer receive positional information?

Content attention alone does not encode sequence order, so the model needs positional signals to distinguish where otherwise similar tokens occur.

What are the two decoder modifications emphasized in the supplied text?

Its self-attention is causally masked to hide future tokens, and it adds encoder-decoder attention over encoder outputs.

What temporal capability do the reviewed deep models provide to reinforcement-learning agents?

By predicting or encoding temporal regularities, they can support anticipation, planning, and generalization across environments.

What empirical information is absent from the supplied passage?

It gives no datasets, quantitative results, evaluation metrics, baselines, uncertainty estimates, or ablation outcomes, so it motivates mechanisms but does not empirically establish superiority.

For X ∈ ℝ^(T×d_model), how are the query, key, and value matrices formed?

Q = XW^Q, K = XW^K, and V = XW^V, using learned linear projections.

If W^Q and W^K are in ℝ^(d_model×d_k), what are the shapes of Q and K?

Both Q and K have shape T×d_k.

What is the general shape of W^V and why is the supplied notation slightly restrictive?

Generally W^V ∈ ℝ^(d_model×d_v), giving V ∈ ℝ^(T×d_v); the passage writes d_k for all three projections even though it later distinguishes d_v.

In attention, what functional role does a query vector q_t play?

It represents what position t is seeking from the available tokens and is compared with their keys to form relevance scores.

In attention, what functional role does a key vector k_j play?

It represents the features by which token j can be matched against a query.

In attention, what functional role does a value vector v_j play?

It carries the information from token j that will be aggregated into an output if the corresponding attention weight is large.

What does the unscaled dot product q_t·k_j represent?

It is a learned compatibility score measuring how relevant token j is to the information sought at position t.

Given Q,K ∈ ℝ^(T×d_k), what is the shape and meaning of QKᵀ?

It is T×T; entry (t,j) is the query-key compatibility score from target position t to source position j.

State the scaled dot-product attention equation.

Attention(Q,K,V) = softmax(QKᵀ/√d_k)V.

Why divide attention logits by √d_k?

If query and key components have roughly unit variance, their dot product variance grows like d_k; division by √d_k keeps logit scale roughly stable and reduces softmax saturation.

Along which axis is softmax normally applied to QKᵀ in self-attention?

Across the key positions for each query row, so every query obtains a distribution over the tokens it may attend to.

What property does each row of an unmasked attention-weight matrix have after softmax?

Its entries are nonnegative and sum to 1.

Write the output for a single query position t as a sum over values.

o_t = Σ_j α_tj v_j, where α_tj = exp(q_t·k_j/√d_k) divided by Σ_m exp(q_t·k_m/√d_k).

If V ∈ ℝ^(T×d_v), what is the shape of a single-head attention output?

T×d_v, because a T×T attention matrix multiplies the T×d_v value matrix.

Why is a head output described as context dependent?

The weights on the values change with the current query and all available keys, so the representation of a token depends on the surrounding sequence.

If all allowed attention logits for a query are equal, what weights result?

A uniform distribution over the allowed keys, so the output is their arithmetic mean in value space.

If one allowed attention logit is much larger than all others, what happens to the output?

Softmax places nearly all weight on that key, so the output approaches its associated value vector.

If every value vector in a head is identical, can changing the attention weights change that head’s output?

No. Any normalized weighted average of identical values equals that same value vector.

Without positional information, what ordering property does self-attention have?

It is permutation equivariant: permuting input tokens permutes the outputs in the same way, rather than revealing an intrinsic order.

How do positional encodings change content-only self-attention?

They alter each token representation as a function of position, allowing the learned queries, keys, and values to depend on both content and location.

Why can attention help with distant temporal dependencies?

A token can place weight directly on any visible earlier position instead of relying only on repeated step-to-step state transitions.

What is the leading time complexity of dense self-attention in sequence length T?

Forming and using the T×T attention matrix costs O(T²d) up to projection terms, where d is a head or model width.

What is the leading memory issue of dense self-attention?

Storing attention logits or weights requires O(T²) memory per layer and head group, which can dominate for long trajectories.

For q = (1,0), k₁ = (1,0), k₂ = (0,1), d_k = 2, and scalar values v₁ = 2, v₂ = 0, what is the attention output?

The logits are (1/√2, 0), giving weights about (0.67, 0.33); the output is about 0.67·2 + 0.33·0 = 1.34.

What happens if the same vector c is added to every key for a fixed set of queries?

For each query, q·c adds the same scalar to every logit; row-wise softmax cancels that common shift, so the attention weights are unchanged.

What happens to attention sharpness if a query is multiplied by a large positive scalar while keys are fixed?

Its logits spread farther apart, usually making softmax more peaked around the largest compatibility score.

How can the attention scale be interpreted as a softmax temperature?

Dividing logits by √d_k is equivalent to using temperature √d_k; a higher temperature produces flatter weights, while a lower temperature produces sharper weights.

How is a causal attention mask applied mathematically?

Disallowed future logits are set to −∞, or a sufficiently negative value, before softmax so their resulting weights are zero.

Why must masking occur before softmax rather than zeroing logits before softmax?

A zero logit still receives positive probability; setting a disallowed logit to −∞ makes its softmax probability exactly zero in theory.

What is the purpose of a padding mask in attention?

It prevents real tokens from assigning weight to padded sequence positions that contain no valid data.

For cross-attention with U decoder positions and T encoder positions, what is the score-matrix shape?

U×T, because U decoder queries are compared with T encoder keys.

In encoder-decoder cross-attention, where do Q, K, and V normally come from?

Queries come from the decoder stream, while keys and values come from encoder outputs.

Why is scaled dot-product attention not identical to cosine-similarity attention?

The dot product depends on vector magnitudes as well as angle, whereas cosine similarity explicitly normalizes vector norms.

What optimization problem can occur if unscaled dot products grow very large?

Softmax can saturate near one-hot probabilities, producing small gradients for most alternatives and making learning less stable.

Are attention weights fixed after training for a given position index?

No. They are dynamically recomputed from the current input-derived queries and keys, so they vary with sequence content.

Before the output projection, where does a single attention output lie relative to its allowed value vectors?

Because softmax weights are nonnegative and sum to one, it is a convex combination of the allowed value vectors.

If Q = K, is the final attention-weight matrix necessarily symmetric?

No. QKᵀ is symmetric, but row-wise softmax uses a different normalizer for each row, which can break symmetry.

What is the maximum rank of the unnormalized score matrix QKᵀ before masking?

At most d_k, and also at most T, because it is the product of a T×d_k matrix and a d_k×T matrix.

Why does “attend across arbitrary time spans” require qualification?

A model can directly connect any positions inside its processed context, but finite context windows, memory cost, positional behavior, and training distribution still bound usable time spans.

Does a short attention path guarantee that the model will use a distant dependency?

No. It makes the interaction representable, but learning it still depends on data, optimization, capacity, positional signals, and whether the target rewards using it.

Why might a single attention head be insufficient?

One head has one set of projections and one attention pattern, which can limit its ability to represent several distinct relation types simultaneously.

State a standard per-head multi-head attention formula using a common input X.

head_i = Attention(XW_i^Q, XW_i^K, XW_i^V).

Why is the supplied formula head_i = Attention(QW_i^Q, KW_i^K, VW_i^V) potentially ambiguous?

The passage previously defines Q, K, and V as projected from X, so multiplying them by head projections can imply an unintended second projection unless Q, K, and V are instead generic inputs.

For h heads with per-head value width d_v, what is the shape after concatenating their outputs over T tokens?

T×(h d_v).

What does the multi-head output projection W^O do?

It maps the concatenated h d_v-dimensional head representation back to d_model and learns how to mix information across heads.

What shape must the multi-head output projection have?

W^O ∈ ℝ^((h d_v)×d_model).

If d_k = d_v = d_model/h, what width does concatenation produce?

h d_v = d_model, so concatenating all heads restores the model width before W^O.

Under the common split-head configuration and ignoring biases, roughly how many parameters do Q, K, V, and output projections contain together?

About 4d_model²: three d_model×d_model combined projections plus one d_model×d_model output projection.

Does using multiple heads guarantee that the heads learn distinct relations?

No. Heads can become redundant or unused; diversity is an empirical outcome, not a mathematical guarantee.

What ablation tests whether multiple heads are necessary rather than merely increasing parameter count?

Compare different head counts while matching total model width, parameter count, training budget, and downstream evaluation; optionally prune heads after training.

Why is W^O useful even when concatenated heads already have width d_model?

It lets the model learn arbitrary linear combinations across head subspaces instead of leaving them as permanently separated feature blocks.

State the two-layer feed-forward network used in the supplied text.

FFN(x) = ReLU(xW₁ + b₁)W₂ + b₂, where ReLU(a) = max(0,a) elementwise.

What are the FFN weight shapes in the supplied architecture?

W₁ ∈ ℝ^(d_model×d_ff) and W₂ ∈ ℝ^(d_ff×d_model).

What does “position-wise” mean for the Transformer FFN?

The same FFN is applied independently to every sequence position; it transforms features within a token but does not directly mix tokens.

Are different FFN parameters learned for different sequence positions?

No. The parameters are shared across positions within a layer, although different Transformer layers normally have different FFN parameters.

Why is d_ff often larger than d_model?

The expansion gives each token a higher-dimensional nonlinear feature space before projection back to d_model, increasing representational capacity.

What role does ReLU play in the supplied FFN?

It adds an elementwise nonlinearity between the two linear transformations, allowing the FFN to represent more than a single affine map.

Why is it imprecise to say that only the FFN introduces nonlinearity into a Transformer?

The FFN does add an explicit activation, but softmax attention and layer normalization are also nonlinear operations.

Ignoring biases, how many parameters does one FFN contain?

2d_model d_ff, from W₁ and W₂.

What behavior would an FFN-removal ablation test?

It would test whether attention-based token mixing alone provides enough per-token feature transformation; a drop would support the FFN’s contribution, though parameter-count changes must be controlled.

State the sinusoidal positional encoding for even and odd dimensions.

PE(pos,2i) = sin(pos/10000^(2i/d_model)) and PE(pos,2i+1) = cos(pos/10000^(2i/d_model)).

Why are sine and cosine paired at each positional frequency?

The pair provides two phases of the same frequency, yielding a smooth two-dimensional representation of position at that scale.

How do positional frequencies change as dimension index i increases in the supplied encoding?

The denominator grows with i, so the angle changes more slowly with position and the corresponding wavelength becomes longer.

How is a token embedding combined with its positional encoding?

z_pos = x_pos + PE_pos.

What shape must x_pos and PE_pos have for elementwise addition?

Both must have model width d_model, producing z_pos ∈ ℝ^d_model.

What is the shape of the stacked initial encoder representation h^(0) = Z?

T×d_model, with one combined token-plus-position vector per row.

What failure would be expected if positional encodings were removed from a task where order matters?

The model could still compare token content but would have no intrinsic way to distinguish permutations containing the same token multiset, so order-sensitive performance should degrade.

Why do sinusoidal encodings not automatically guarantee reliable extrapolation to longer sequences?

Although they can be evaluated at unseen positions, the model may not have learned how to interpret those phase patterns outside the training range.

What useful relative-position property do sinusoidal encodings have?

For a fixed offset, the sine-cosine vector at pos+offset can be expressed as a linear transformation of the vector at pos, which can make relative displacement accessible.

What is one tradeoff between learned absolute positions and fixed sinusoidal positions?

Learned positions can adapt to training data but have no native parameters beyond trained indices; sinusoids require no learned table and are evaluable beyond training length, though neither guarantees length generalization.

Write the supplied post-normalization encoder attention update.

h_tilde^(l) = LayerNorm(h^(l−1) + MultiHead(h^(l−1), h^(l−1), h^(l−1))).

Write the supplied post-normalization encoder FFN update.

h^(l) = LayerNorm(h_tilde^(l) + FFN(h_tilde^(l))).

What is the purpose of a residual connection around a Transformer sublayer?

It preserves a direct identity path for representations and gradients while allowing the sublayer to learn a refinement.

State the scalar feature update performed by layer normalization.

For feature i, x̂_i = (x_i−μ)/√(σ²+ε), then y_i = γ_i x̂_i + β_i.

Across which dimensions are μ and σ² computed in the supplied LayerNorm description?

Across the feature dimensions of each token’s vector, not across sequence positions or batch examples.

What are γ and β in layer normalization?

Learned per-feature scale and shift parameters that let the network adjust the normalized representation.

Why is ε included in layer normalization?

It prevents division by zero and improves numerical stability when the feature variance is very small.

Is the supplied encoder update pre-norm or post-norm?

Post-norm, because LayerNorm is applied after adding each sublayer output to its residual input.

How does a common pre-norm block differ from the supplied post-norm block?

Pre-norm applies LayerNorm before attention or the FFN and then adds the residual; it often improves optimization in deep Transformers, but it is not the equation shown in the passage.

Which encoder operations mix information across token positions?

Self-attention mixes positions; the FFN and standard LayerNorm act independently at each position.

Why can stacking Transformer layers help even though one self-attention layer has global visibility?

Depth allows repeated cycles of relation-dependent aggregation and nonlinear feature transformation, composing higher-order representations.

What exactly does causal masking prevent in a decoder?

At position t, it prevents attention to tokens at positions greater than t, blocking direct access to future outputs.

State the supplied autoregressive output distribution.

P(y_t | y_<t, X) = softmax(h_t W_vocab + b_vocab), where the projection maps the decoder state to vocabulary logits.

What are the shapes of the decoder vocabulary projection and its output logits?

W_vocab ∈ ℝ^(d_model×|V|), so h_t W_vocab + b_vocab has |V| logits.

What notation collision occurs in the supplied text?

W^O denotes both the multi-head output projection and the decoder-to-vocabulary projection, even though they have different roles and shapes.