RNNs are designed to process sequences of data by maintaining an internal state that can carry information across time steps.
Key components include:
Input at time t (x_t): The current input vector at time step t.
Hidden state (h_t): The internal state that gets updated at each time step, holding information from past inputs.
Output (y_t): The final output generated from the hidden state.
Sequence and Frame Processing
Sequence Length (n): Represents the number of time steps in the input sequence.
Frame Length (m): Represents the number of features in each data point (vector).
The process involves feeding sequences of vectors (x1, x2, …, x_m) into the RNN, capturing activities over time.
Activities must be observed continuously across these multiple frames for accurate classification (e.g., detecting whether a person is running or sitting).
Temporal Dependency in RNNs
Data points across different sequences (x_i for i = 1 to n) are independent from each other, but sequences themselves are dependent across time steps (e.g., the vector at time t influences the vector at time t+1).
This structure supports the understanding of the entire activity pattern (e.g., a person’s motion).
For classification, maintaining the order and connections within sequences is crucial.
Patterns in Time Series and Sequences
Time Series: Variation of activities over time, showcasing patterns (e.g., increments in a series like 2 to 5).
The order and consistency of data points (x_t) are pivotal for predicting future data points.
Data must be structured as sequences to enable effective processing and output through RNNs.
Data Format in Neural Networks
In feedforward networks, each input is treated as a single vector, while in recurrent networks, inputs are treated as sequences or matrices of vectors.
Sequence representation may be in forms such as:
x=x<em>1,x</em>2,…,xn (sequence of n vectors)
Each vector can have dimensions d, making it a d-dimensional input for the recurrences.
Independence of Data Points
For successful machine learning algorithms:
Data points (xi) must be mutually independent; independence allows for shuffling methods.
Training and testing datasets must be drawn from the same distribution to ensure the model predicts accurately on new data.
If the training data has a different distribution from testing data, the model performance deteriorates significantly.
This adheres to the i.i.d. (independent and identically distributed) assumption, a fundamental principle in ML.
Learning in RNNs
The architecture allows for adjustments (learning) through weights (w) and biases (b) in neuron structures.
Updates happen across iterations:
y=w⋅x+b
Here, w changes and adapts with each training epoch to minimize prediction error.
Learning algorithms adjust gradients but remain constrained to a structured path without altering the entire shape (i.e., centroid shifts).
Memorization vs Generalization
Understanding how well a model can generalize involves:
Balancing its ability to encode learned features versus its ability to apply those features to unseen data.
The effectiveness of predictions relies on consistent data representing the identified distribution throughout training and evaluation phases.
Recap of Core Principles
RNNs are vital for sequence-based tasks where contextual information is necessary.
Datasets must be structured and sourced correctly to facilitate model success, adhering to the i.i.d. assumption.
Learning processes involve weight adaptations based on training efficacy, maintaining a balance between new learning (generalization) and previously learned information (memorization).