Sequence Learning: Foundations and Time Series Processing

Fundatmentals of Sequential Data and Time Series

• The Global Prevalence of Sequences: Sequential data is foundational to understanding the world. Sensory inputs (visual, auditory, tactile) are perceived as continuous streams of sequential data.
• Defining Sequential Data: In sequences, the order of elements is not just a feature but is essentially important and meaningful.
• Coherence: Sequential data typically exhibits coherence along its sequential dimension, which can be the temporal axis or a spatial axis.
• Pattern Distribution: Patterns are not localized in single points but are distributed along the sequential dimension.
• Challenges in Sequential Data Processing:
  • Feature Variance: The positions of important features may vary across different sequences.
  • Length Variability: Sequences can have varying lengths, making fixed-input models difficult to apply.
  • Non-linear Warping: Patterns can be warped non-linearly over time or space.
  • Extended Dependence: Patterns can extend over very large sequence lengths, requiring long-term memory mechanisms.
• Required Model Capabilities:
  • Structural flexibility to handle varying lengths.
  • Elasticity (warping tolerance) to handle temporal distortions.
  • Robustness against nonstationarity (shifting statistical distributions).

Time Series Definitions and Formalisms

• Time Series as a Subset: Time series are a specific type of sequential data where time is the primary sequential dimension (the time axis).
• Process Nature: Data usually originates from an underlying continuous-time process but is sampled in discrete time.
• Univariate Time Series:
  • Data points are pairs $(t, x)$.
  • Continuous Time: $t \in \mathbb{R}$.
  • Discrete Time: $t \in \mathbb{N}$ ($t$ is known as the time index).
  • Data Point: At every time step, the data point is a single scalar value $x \in \mathbb{R}$.
  • Equidistant Sampling: Usually, sampling is equidistant, defined as $\forall t \exists k \in \mathbb{N} : t = \Delta t \cdot k$ for a constant time delay $\Delta t$.
  • Example: A sample rate of $100$ samples per second corresponds to $\Delta t = 10\,ms$.
• Multivariate Time Series:
  • Data points consist of more than one variable: $x \in \mathbb{R}^n$ where $n > 1$.
  • Variables $(x_t^1, \dots, x_t^n)$ can be correlated, independent, or form a high-dimensional trajectory.
  • The generation and handling of multivariate series are significantly more complex than univariate ones.

Notational Conventions for Sequential Data

• Function-Style Notation: Expressed as $s(t)$. This is common for continuous-time signals (e.g., $s(0)$, $s(0.1)$, $s(\pi)$).
• Discrete Notation:
  • Array-style: $s[t]$ (less common).
  • Index-style (most common): Uses superscript $x^t$ or subscript $x_t$. A sequence is denoted as $(x_1, x_2, x_3, \dots, x_T)$, where $T$ refers to the sequence length.
• Note: In notation, $T$ for length should not be confused with the transposition operator $A^\top$.
• Matrix/Tensor Representation: Sequential data can be organized in a matrix where the first dimension typically refers to time:     $\begin{pmatrix} x_1^1 & x_1^2 & \dots & x_1^n \\ x_2^1 & x_2^2 & \dots & x_2^n \\ \vdots & \vdots & \ddots & \vdots \\ x_T^1 & x_T^2 & \dots & x_T^n \end{pmatrix}$

Sequential Data in PyTorch

• Data Layouts:
  • Sequence First [T, B, D]: (Sequence length, Batch size, Data dimension). This is the default layout for most Recurrent Neural Network (RNN) modules in PyTorch.
  • Batch First [B, T, D]: (Batch size, Sequence length, Data dimension). Often considered more intuitive for certain data pipelines.
• Implementation Details: The batch_first=True argument can be used in modules like nn.LSTM to toggle layouts.
• Variable Length Sequences: These are often handled more efficiently with the "sequence first" layout using functions like pack_padded_sequence.

Statistical Properties: Stationarity and Seasonality

• Mean ($\bar{x}$): The average of a time series $(x_t)_{1 \le t \le T}$ of length $T$:     $\bar{x} = \frac{1}{T} \sum_{t=1}^{T} x_t$
• Variance ($\sigma^2$): Measures the spread of the data:     $\sigma^2 = \frac{1}{T} \sum_{t=1}^{T} (x_t - \bar{x})^2$
• Standard Deviation: $\sigma = \sqrt{\sigma^2}$.
• Stationarity: A time series is stationary if its statistical properties (mean, variance) do not change over time.
• Nonstationarity: A time series is nonstationary if its statistics change over time. Common examples include time series with a "trend" where the local mean shifts.
• Seasonality: Refers to reoccurring periodic trends within nonstationary data.
• Processing Challenges: Nonstationarity is problematic for forecasting. Detection depends heavily on temporal resolution (the "zoom" on the data).
• Trend Removal Techniques:
  1. High pass filtering: Attenuates low-frequency trends.
  2. Differencing: Calculating the difference between successive steps: $s'[t] = s[t] - s[t-1]$.

Quantifying Distance Between Time Series

• Mean Square Error (MSE): Often used as a loss function:     $MSE(X, Z) = \frac{1}{T \cdot n} \sum_{t=1}^{T} \sum_{i=1}^{n} (x_t^i - z_t^i)^2$
  • Drawback: It does not share the same "unit" as the data because error decreases/increases quadratically.
• Root Mean Square Error (RMSE):     $RMSE(X, Z) = \sqrt{\frac{1}{T \cdot n} \sum_{t=1}^{T} \sum_{i=1}^{n} (x_t^i - z_t^i)^2}$
  • Advantage: Shares the unit with the data; better interpretability. It is similar to Euclidean distance.
• Alignment Problem: Point-wise measures (like MSE/RMSE) produce high error for unaligned sequences even if they are structurally similar. Proper distance measures should incorporate alignment.

Dynamic Time Warping (DTW)

• Core Logic: DTW finds the best "warping path" to align two sequences by minimizing the discrepancy (warping effort) between them [Sakoe and Chiba, 1978].
• Temporal Flexibility: Allows for non-linear alignment, accommodating differences in timing and speed of events.
• Capabilities:
  • Compares sequences of different lengths.
  • Is robust to noise and timing irregularities.
• Applications: Speech recognition, gesture recognition, and pattern recognition.
• Modern Implementations: Improved algorithms like fastDTW [Salvador and Chan, 2007] are used for computational efficiency.

Principles of Signal Filtering

• Noise Reduction: Removing interference from signals.
• Smoothing: Attenuating high-frequency components to aid analysis.
• Frequency Selectivity: Selectively passing or attenuating specific frequency bands.
• Domains: Filters operate in either the time domain or frequency domain.

Exponential Moving Average (EMA) Filtering

• Recursive Formula: For a series $(x_t)_{1 \le t \le T}$, the EMA-filtered signal $\tilde{x}_t$ is:     $\tilde{x}_t = \alpha \tilde{x}_{t-1} + (1 - \alpha) x_t$
• Filter Coefficient ($\alpha$): The closer $\alpha$ is to $1$, the smoother the output.
• Characteristics:
  1. Online Computation: Incremental processing with every new data point.
  2. Low Memory: Ideal for data streams.
  3. Low Pass Filter: Effectively cancels high frequencies.
  4. Causality: It is a causal filter.
  5. Delay Trade-off: Better smoothing results in higher signal delay.
• High Pass Application: EMA can be used for high-pass filtering by subtracting the low-pass EMA output from the original signal: $s_{high}[t] = s[t] - EMA(s[t])$. This helps remove trends.

Causality and Noncausal Filtering

• Causality: Past observations influence future values (unidirectional flow). Causal models prevent "data leakage" (future information biasing current predictions).
• Noncausal Models: Require the entire sequence or future values to be present, which introduces delays.
• Forward-Backward Filtering (Noncausal Operation): Solves the delay problem of causal filters by filtering twice:
  1. Forward Pass: $x' = EMA(x)$.
  2. Backward Pass: $\tilde{x} = \text{flip}(EMA(\text{flip}(x')))$.
• Result: The bidirectional application cancels out the temporal delay, but causality is lost. It requires the full sequence or accurate future predictions.

Course Outlook and Advanced Applications

• Key Sequence Learning Tasks:
  • Time Series Classification: Predicting class labels for entire sequences or steps.
  • Time Series Forecasting: Predicting future values of a signal.
  • Sequence-to-Sequence (Seq2Seq) Mappings: Translating one sequence space to another (e.g., text translation).
• Recurrent Neural Networks (RNNs): Described as the "most beautiful sequence learning structure," to be discussed in future lectures.
• Boss Level Application: Spatiotemporal process forecasting on a global scale (e.g., weather prediction).
  • Requires sophisticated system design and strong inductive biases.
  • Involves complex, nonstationary, multiscale data.
  • Computationally intensive resources required.

Referenced Literature

• Bache and Lichman (2013). UCI machine learning repository.
• Bonev et al. (2023). Spherical Fourier neural operators (ICML '23).
• Otte et al. (2013). OCT A-Scan based lung tumor tissue classification with Bi-LSTM (MLSP).
• Sakoe and Chiba (1978). Dynamic programming algorithm optimization for spoken word recognition.
• Salvador and Chan (2007). Toward accurate dynamic time warping in linear time and space.
• Thummel et al. (2024). Inductive biases in deep learning models for weather prediction.