Recurrent Neural Networks: Architectural Motivation and Sequence Modeling Foundations

Feedforward Neural Networks and Sequence Processing

  • Standard Feedforward Classifiers:

    • Multilayer Perceptrons (MLPs) traditionally encompass architectures such as Fully-Connected (FC) Neural Networks and Convolutional Neural Networks (CNNs).
    • Both architectures process the complete input pattern simultaneously in a single forward pass during both training and inference.
  • Input Data Representations:

    • Multidimensional Matrices: Data structured spatially as 2D or 3D matrices (e.g., image channels and pixel grids).
    • One-Dimensional Vectors: Data represented as a 1D array of values (e.g., concatenated token representations of a text sentence).
    • Monolithic Feeding: Regardless of structure, standard feedforward neural networks (FNNs) mandate that the entire sequence be formatted and ingested as a single, static input vector x⃗=[x1,x2,x3,x4,x5]\vec{x} = [x_1, x_2, x_3, x_4, x_5].

Comparison of Fully-Connected Neural Network and Convolutional Neural Network architectures for vector input

Sentiment Analysis Case Study and Dataset Structure

  • Task Objective:

    • Formulation of a binary text classifier to evaluate sentiment in a movie review corpus, categorizing reviews into either POSITIVE or NEGATIVE polarity.
  • Exemplary Corpus Records:

    1. Fellini created magic in Eight-and-Half! (Label: POSITIVE)
    2. Disgusting! (Label: NEGATIVE)
    3. The longest bore on celluloid! (Label: NEGATIVE)
    4. I can watch Roma forever! (Label: POSITIVE)
    5. Worst than the previous one! (Label: NEGATIVE)
    6. Coppola's masterpiece! (Label: POSITIVE)
  • Key Structural Observations:

    • Natural language sequences display extreme length variance; in the sample dataset, reviews range from single-token expressions (Disgusting!, length 11) to five-token sequences (Fellini created magic in Eight-and-Half!, length 55).

Preprocessing Text for Feedforward Classifiers

  • Prerequisites for Vector Space Ingestion:

    • Numeric Conversion: Raw linguistic tokens (words and punctuation) cannot be directly processed by arithmetic perceptrons and must be mapped to numerical values.
    • Dimensional Uniformity: Fixed-topology networks (FC and CNN) require inputs to conform strictly to a predetermined vector length.
  • Token Embeddings:

    • Embedding refers to the mapping of discrete tokens or words into numerical representations.
    • While operational embeddings typically project tokens into dense vectors of higher dimensionality, they can be conceptually represented as scalar numbers for illustrative simplicity:
    • Coppola →[9.1]\rightarrow [9.1]
    • Fellini →[8.4]\rightarrow [8.4]
    • Mapping reviews to numerical vectors:
    • Review 1 (Fellini created magic in Eight-and-half): transformed to the 5-dimensional vector [8.4,6.1,5,9,2][8.4, 6.1, 5, 9, 2].
    • Review 2 (Disgusting): transformed to the 1-dimensional vector [4.3][4.3].
  • Zero-Padding Technique:

    • Because MLP models cannot accept inputs of differing dimensions across samples, the maximum length found in the dataset (Lmax⁡=5L_{\max} = 5) is used to define the universal input vector size.
    • Shorter vectors are padded with zero values to match Lmax⁡L_{\max}:
    • Review 2 with zero-padding: Disgusting (padding) (padding) (padding) (padding) →[4.3,0,0,0,0]\rightarrow [4.3, 0, 0, 0, 0].

Inherent Limitations of Feedforward Architectures for Text

  • Inefficiency of Variable-Length Inputs:

    • Standardizing textual length via zero-padding introduces artificial sparsity, increases redundant computation, and requires allocating fixed input layers sized to the largest potential sequence.
  • Inability to Support Variable-Length Outputs:

    • Feedforward networks map fixed inputs to fixed-dimension targets.
    • Applications such as machine translation, summarization, or dialogue generation require dynamically sized outputs where the generated sentence length often differs from the input sentence length.
  • Neglect of Temporal and Sequential Inherent Structure:

    • Linguistic meaning is fundamentally determined by the sequential order and contextual interdependencies of tokens over time.
    • In image classification, CNNs achieve superior performance by exploiting spatial locality and spatial translations.
    • Analogously, processing natural language and time-series data requires an architecture capable of explicitly manipulating and retaining temporal, order-dependent relationships.

Recurrent Sequential Modeling and Dynamic Memory

  • Token-by-Token Processing:

    • Rather than consuming a complete sentence as a single monolithic block, the sequence is segmented into sequential tokens:
    • x1=Fellinix_1 = \text{Fellini}
    • x2=createdx_2 = \text{created}
    • x3=magicx_3 = \text{magic}
    • x4=inx_4 = \text{in}
    • x5=Eight-and-Halfx_5 = \text{Eight-and-Half}
    • Tokens are fed into the network individually across sequential timesteps (t1,t2,…,tTt_1, t_2, \dots, t_T).
  • Sequential Language Model Dynamics:

    • In an autoregressive language modeling formulation, each token's target output is the subsequent token:
    • At timestep t1t_1, input x1=Fellinix_1 = \text{Fellini} is processed to predict y1=createdy_1 = \text{created}.
    • Information from t1t_1 is retained and propagated to timestep t2t_2.
    • At timestep t2t_2, the network receives two inputs: the context from t1t_1 and the current token x2=createdx_2 = \text{created}, computing y2=magicy_2 = \text{magic}.
    • At timestep t3t_3, inputs comprise historical context from preceding timesteps and x3=magicx_3 = \text{magic}, yielding y3=iny_3 = \text{in}.
    • At timestep t4t_4, the model ingests context from t1t_1 through t3t_3 alongside x4=inx_4 = \text{in} to produce y4=Eight-and-Halfy_4 = \text{Eight-and-Half}.
    • At timestep t5t_5, the final hidden state ingests x5=Eight-and-Halfx_5 = \text{Eight-and-Half} alongside cumulative history from t1t_1 to t4t_4 to produce the overall sequence sentiment label (y=Class Labely = \text{Class Label}, e.g., POSITIVE).
  • Cognitive Reading Analogy:

    • This sequential processing parallels human language comprehension: humans read token-by-token, continually updating a mental model of context and past information rather than digesting an entire document instantaneously as a single perception.
  • Internal Memory and Hidden States:

    • The intermediate recurrent layer functions as an internal hidden state (hh) that persists contextual memory across time.
    • The output at any step tkt_k incorporates the aggregated signal from timesteps t1t_1 up to tkt_k.

Unrolled recurrent neural network architecture processing text token-by-token across timesteps

Mathematical Formulation of the Recurrent Neural Network (RNN)

  • Core Structural Layers:

    • Input Layer: Supplies the vector representation xtx_t of the token at timestep tt.
    • Hidden State Layer: Maintains the internal latent memory vector hth_t.
    • Output Layer: Emits the network prediction yty_t.
  • Hidden State Evolution Equation:

    • The hidden state hth_t at timestep tt is governed by a fully connected transformation combining the current input xtx_t and the antecedent hidden state ht−1h_{t-1}:     ht=f(xt,ht−1)h_t = f(x_t, h_{t-1})
    • The function ff typically incorporates a linear combination followed by a non-linear activation function.
  • Output Computation Formulations:

    • Direct Hidden Output (Identity Formulation): In simplified architectures, the output directly mirrors the hidden state:     yt=hty_t = h_t
    • Parameterized Output Mapping: In standard and deep formulations, the output is produced by applying an additional fully connected layer with function gg to the hidden state:     yt=g(ht)y_t = g(h_t)
  • Recurrent Parameter Sharing:

    • Unlike feedforward networks that allocate separate weights to each element of a fixed-size input vector, an RNN utilizes the identical parameter sets across all timesteps.
    • The transformation matrices governing f(⋅)f(\cdot) and g(⋅)g(\cdot) remain invariant over time, allowing the model to generalize across variable sequence lengths.

Cyclic Representation vs. Unrolled Computation Graph

  • Folded (Cyclic) Topology:

    • The foundational definition of an RNN consists of a single processing unit containing a self-directed recurrent loop.
    • The activation of hidden state hh is fed back into itself alongside input xx, forming a directed cycle.
  • Unrolled (Unfolded) Graph Through Time:

    • To visualize and compute network states over a sequence of length TT, the recurrent loop is unrolled into an acyclic computational chain representing timesteps t1,t2,…,tTt_1, t_2, \dots, t_T.
    • Each node in the unrolled chain represents the network state at a discrete timestep tt, explicitly displaying how hidden representations propagate forward through time (ht−2→ht−1→ht→ht+1h_{t-2} \rightarrow h_{t-1} \rightarrow h_t \rightarrow h_{t+1}).

Recurrent Neural Network represented as unrolled through time and as a compact cyclic architecture