L7_Sequences and Time Series

Flashcards covering key concepts related to sequences, time series, and recurrent neural networks.

What are sequences?

Sequences are ordered series. Think of them like a train where each car follows the one before it in a specific order.

Give examples of sequence prediction tasks for classification.

Speech recognition (understanding spoken words), Fraud detection (identifying unusual transaction patterns), Network intrusion detection (spotting malicious activity), Fault detection and predictive maintenance (predicting equipment failures), Medical diagnostics (analyzing patient history), Sentiment analysis (gauging public opinion), Topic classification (categorizing documents).

Give examples of sequence prediction tasks for forecasting.

Predicting weather (using past conditions), energy prices (based on market trends), stock prices (analyzing historical data), Text generation (creating new content).

Give examples of sequence-to-sequence learning tasks.

Language translation (converting text between languages), Image captioning (generating descriptions for images), Text summarisation (condensing large documents).

What Keras layers are analogous to Conv2D and Conv2DTranspose for sequences?

keras.layers.Conv1D and keras.layers.Conv1DTranspose. These are like applying a filter over a 1D sequence to extract features.

What Keras layer is analogous to MaxPooling2D for sequences?

keras.layers.MaxPooling1D. Similar to picking the most important signal from a set of inputs in a sequence.

What is cross-correlation?

A similar operation to convolution where one of the functions is reversed. Think of it as finding how similar two signals are as you slide one past the other.

Why are CNNs typically not used for forecasting?

CNNs are great for classification because of translation invariance (detecting patterns regardless of their position), which we typically don’t want for forecasting. In forecasting, the position in time matters!

What is the key assumption for forecasting?

Recent data is more informative than old data. The weather today is a better predictor of the weather tomorrow than the weather a year ago.

What is a key limitation of standard neural networks in sequence modeling?

They have no state and can’t remember anything. Each input is processed independently without considering past inputs.

How is state introduced to a recurrent neural network (RNN)?

Each node stores its previous output. Like a memory cell remembering what it saw before.

In a recurrent node, what are the two sets of weights and how are they applied?

\Wx applied to current input, and \Wy applied to the previous state (output). One set of weights transforms the current input, the other transforms the memory of the past.

What is an autoregressive model?

A model where the value tomorrow is predicted based on the value today (or a weighted sum of previous time steps). For example, predicting tomorrow's temperature based on today's temperature.

What are common issues in training RNNs, and what are two approaches used to mitigate them?

Vanishing/exploding gradients that make it difficult to learn long-term dependencies. Mitigated using LSTMs and GRUs. Imagine trying to pass a message through a long line of people; the message might get lost or amplified along the way. LSTMs and GRUs are like adding mechanisms to ensure the message gets through clearly.

How do LSTMs add long-term memory?

By having two states in each cell: a short-term state and a long-term state. This allows the network to remember information over longer sequences.

What are some tricks for efficiently training recurrent networks?

Use saturating activation functions (tanh, sigmoid), layer normalization, recurrent dropout, and optionally unroll for loops. These are like tuning the settings on a machine to make it run smoother and faster.

What is a CNN processing bonus trick for handling long sequences in RNNs?

Extract small-scale patterns with convolutional layers first, then apply recurrent layers. Use CNNs to find local patterns before feeding the sequence into an RNN.

What is a bidirectional RNN and when might it be useful?

Processes sequences both forwards and in reverse. Useful when chronological ordering isn't strictly important, such as in text analysis. It's like reading a sentence from both ends to understand the context better.