Recurrent Neural Networks and LSTMs

Comprehensive practice flashcards covering the fundamental concepts of RNNs, LSTMs, GRUs, and their specific applications in medical informatics as presented in the lecture.

Recurrent Neural Networks (RNNs)

A type of neural network used when temporal information is more important (or equally important) than spatial information, commonly applied to speech and video.

Vanishing Gradient Problem

A challenge in RNNs where the derivative of the sigmoid function becomes zero at both tails, causing learning to be slow and the network to forget old events.

LSTM (Long-Short Term Memory)

A specific type of RNN designed with gates to decide what information to forget and what to remember, addressing the problem of memorizing distant information.

Sigmoid Gate

A layer in an LSTM that outputs values in the range of $[0, 1]$ to act as a filter, deciding how much information should pass through.

tanh Layer

A layer in an LSTM that outputs values between $[-1, 1]$, used to produce candidate values or produce final output values from the state.

Forget Gate

A component of the LSTM that determines which information should be discarded from the cell state.

Input Gate

The gate in an LSTM that, along with a tanh layer, decides which values in the state should be updated.

Output Gate

The gate in an LSTM that decides what the next hidden state should be based on the current cell state.

Cell State

The 'conveyor belt' of the LSTM that carries information across the sequence, modulated by gates to maintain long-range interactions.

GRU (Gated Recurrent Unit)

A variant of an RNN that lacks a cell state, utilizing only a hidden state and gates to modulate information flow.

Reset Gate

A component in a GRU that modulates the amount of past information to keep.

Update Gate

A component in a GRU that defines a positive and negative mask to forget and input information simultaneously.

Backpropagation Through Time (BPTT)

The training process for recurrent networks where errors are propagated backwards through each time step of the sequence.

S2VT

A sequence-to-sequence model used for video-to-text conversion, such as generating the description 'A man is doing stunts on his bike' from video frames.

ChronoNet

A deep recurrent neural network for identifying abnormal EEG activity, formed by stacking 1D convolution layers and deep GRU layers.

Alzheimer’s Disease (AD) progression modeling

An application of LSTMs used to model disease progression using volumes from six MRI regions, including the hippocampus and entorhinal cortex.

LSTM-Mean

A network trained using standard backpropagation through time with missing values imputed based on the mean imputation method prior to training.

NEURO-DRAM

A framework for medical imaging where classification and glimpse networks are optimized via cross entropy, and location and context networks via total reward.

C-LSTM

Bidirectional Convolutional Long Short-Term Memory used in deep sequential segmentation to address memory restrictions and volume resizing issues in 3D CNNs.

Multi Head Attention

A mechanism that spreads cognitive load across multiple attention processes to prevent oversaturation of a single attention mechanism.