Tutorial on Multimodal Machine Learning Notes

Introduction to Multimodal Machine Learning

• Definition: Multimodal Machine Learning (MMML) is a vibrant multi-disciplinary research field bridging artificial intelligence goals by integrating multiple communicative modalities. These include linguistic, acoustic, and visual messages.
• Context: Based on the CMU Course 11-777 "Advanced Multimodal Machine Learning" taught by Louis-Philippe (LP) Morency and Tadas Baltrusaitis.
• Core Goal: Addressing unique challenges involving data heterogeneity and the contingency between modalities, such as audio-visual speech recognition and image/video captioning.
• Technical Scope:
  • Multimodal representation learning (Auto-encoders, Deep Learning).
  • Multimodal component analysis and fusion (Deep CCA, Multi-kernel learning).
  • Multimodal alignment and multi-stream modeling (Multi-instance learning, Multimodal RNNs).
  • Multi-sensory computational modeling (Nonparametric Bayesian networks).

Defining Modality and Medium

• Modality: The way in which something happens or is experienced. It refers to a specific type of information and its storage format.
  • Sensory Modality: Primary forms of sensation such as vision, touch, hearing, aroma, and taste.
• Medium: The "middle" or means for communicating/storing information (e.g., a system of transmission).
• Examples of Modalities:
  • Natural language (spoken or written).
  • Visual (images, videos).
  • Auditory (voice, sounds, music).
  • Haptics/Touch.
  • Smell, taste, and self-motion.
  • Physiological signals (Electrocardiogram [ECG], skin conductance).
  • Other: Infrared images, depth images, fMRI.
• Communicative Behaviors:
  • Verbal: Lexicon, syntax, pragmatics.
  • Vocal: Prosody, intonation, laughter, vocal expressions.
  • Visual: Gestures (head, eye, arm), body language, eye contact (gaze), facial expressions (FACS action units).

Historical Eras of Multimodal Research

• The Behavioral Era (1970s - late 1980s):
  • Focused on psychology and human behavior.
  • Arnold Lazarus (1973): Multimodal Behavior Therapy (7 dimensions of personality).
  • McGurk Effect (1976): Demonstrated the interaction of hearing and vision ("Hearing lips and seeing voices").
  • Language/Gestures: David McNeill proposed gestures as integral components of thought in action, not just additions to speech.
  • Trivia: Geoffrey Hinton received his B.A. in Psychology.
• The Computational Era (late 1980s - 2000):
  • Audio-Visual Speech Recognition (AVSR): First system in 1986 ("Automatic lipreading to enhance speech recognition").
  • Multimodal HCI: Designing interfaces like "Glove-talk" (Fels & Hinton, 1995) to translate hand gestures to speech.
  • Affective Computing (Rosalind Picard): Computing that relates to or influences emotion.
  • Multimedia Computing: Digital Video Libraries (Informedia Project, 1994-2010) and shot-boundary detection.
• The Interaction Era (2000 - 2010):
  • Human Interaction Modeling: Projects like AMI (100+ hours of synchronized meeting data) and CHIL (Alex Waibel).
  • Cognitive Assistants: CALO (Personalized Assistant that Learns), from which Siri was a spinoff.
  • Social Signal Processing (SSP): Coined by Sandy Pentland (2007).
  • Multimedia Information Retrieval: TRECVID (NIST-hosted competition from 2001-2016).
• The Deep Learning Era (2010 - Present):
  • Enablers: Large-scale datasets, faster GPUs, high-level visual features.
  • Key milestones: Multimodal Deep Learning (2011), Visual Attention/"Show, Attend and Tell" (2015).

Core Technical Challenges in Multimodal ML

• Challenge 1: Representation: Learning to represent and summarize data to exploit complementarity and redundancy.
  • Joint Representations: Combining modalities into a single multimodal space (e.g., Multimodal Autoencoders, Tensor Fusion).
  • Coordinated Representations: Keeping modalities separate but enforcing constraints to coordinate them (e.g., Deep CCA).
• Challenge 2: Alignment: Identifying relations between sub-elements of different modalities.
  • Explicit: Directly finding correspondences (e.g., matching a word to a video frame).
  • Implicit: Latent alignment while solving a different task (e.g., Attention mechanisms).
• Challenge 3: Fusion: Joining information for prediction.
  • Model-Agnostic: Early fusion (feature level) or Late fusion (decision level).
  • Model-Based: Custom architectures like Multiple Kernel Learning or Multi-View LSTMs.
• Challenge 4: Translation: Process of changing data from one modality to another (e.g., Image Captioning, Video Description).
• Challenge 5: Co-Learning: Transferring knowledge between modalities (e.g., zero-shot learning, concept grounding).

Basic Concepts: Linear Classification and Loss

• Score Function: Maps input observation to class scores.
  • Linear classifier: $f(x_i; W, b) = Wx_i + b$
  • Weights ($W$) and Bias ($b$).
• Cross-Entropy Loss (Softmax Loss): Minimizing the negative log-likelihood.
  • Probability for class $y_i$: $P(y_i | x_i; W) = \frac{e^{f_{y_i}}}{\sum_j e^{f_j}}$
  • Loss for example $i$: $L_i = -\ln\left(\frac{e^{f_{y_i}}}{\sum_j e^{f_j}}\right)$
• Hinge Loss (Multi-class SVM loss):
  • Formula: $L_i = \sum_{j \neq y_i} \max(0, f_j - f_{y_i} + \Delta)$
  • Standard margin $\Delta = 10$ (example provided).

Basic Concepts: Neural Networks and Optimization

• Foundation: Inspired by artificial neurons (dendrites, cell body, axon).
• Activation Functions:
  • Sigmoid: $f(x) = \frac{1}{1 + e^{-x}}$
  • ReLU (Rectified Linear Unit): $f(x) = \max(0, x)$. Advantaged by faster training and no gradient vanishing.
  • tanh: $f(x) = \tanh(x)$
• Multi-Layer Perceptron (MLP): Chained functions: $y_i = f_3(W_3(f_2(W_2(f_1(W_1 x_i)))))$.
• Gradients and Chain Rule:
  • Local Jacobian (matrix of partial derivatives).
  • Chain rule for gradients: $\nabla_x y = \left(\frac{\partial h}{\partial x}\right)^T \nabla_h y$.
• Backpropagation: Efficient gradient computation through forward and backward passes.
• Parameter Update (Gradient Descent): $\theta^{(t+1)} = \theta^{(t)} - \epsilon_k \nabla_{\theta} L$.
  • Methods: SGD, Momentum, AdaGrad, RMSProp.

Unimodal Representations: Language, Vision, and Audio

• Language Modality:
  • Bag-of-Words: Documents represented by word counts.
  • Word2Vec: Learning representations based on the distribution hypothesis (words in similar contexts lie close together).
  • Arithmetic: $\text{Vec(king)} - \text{Vec(man)} + \text{Vec(woman)} \approx \text{Vec(queen)}$.
• Visual Modality:
  • Convolutional Neural Networks (CNNs): Exploit translation invariance and redundant image structures.
  • Main Components: Convolution layers (shared weights), Pooling layers (sub-sampling, e.g., Max pooling).
  • AlexNet: Classic architecture for 1000-way object classification.
• Acoustic Modality:
  • Digitalization: Samples at 8-96 kHz, bit depth 8-24 bits, window size 20ms.
  • Spectrograms: Visual representation of the spectrum of frequencies over time.
  • Autoencoders: Feed-forward networks intended to reproduce input ($x' = f(g(x))$). Includes Denoising Autoencoders for robustness.

Advanced Multimodal Representations

• Deep Multimodal Boltzmann Machines: Individual modalities trained as DBNs, joined using Variational approaches.
• Deep Multimodal Autoencoders: Bimodal structure used for AVSR (Ngiam et al., 2011).
• Multimodal Vector Space Arithmetic:
  • Example: $\text{blue} - \text{red} + \text{white} \approx \text{nearest images}$.
• Multimodal Tensor Fusion Network (TFN):
  • Models unimodal, bimodal, and trimodal interactions explicitly.
  • Formula for two modalities: $h_m = [h_x, 1] \otimes [h_y, 1]$.
• Canonical Correlation Analysis (CCA):
  • Goal: Find linear projections $u$ and $v$ such that $\text{corr}(u^T X, v^T Y)$ is maximized.
  • Deep CCA (DCCA): Non-linear projections using neural networks.
  • Deep Canonically Correlated Autoencoders (DCCAE): Jointly optimizes DCCA and reconstruction loss.

Recurrent Neural Networks (RNN) and LSTMs

• RNN: Models temporal sequences where $h(t) = \tanh(U x(t) + W h(t-1))$.
• Long Short-Term Memory (LSTM): Specifically designed to solve the vanishing gradient problem in standard RNNs.
  • Components: Memory cell, Input gate ($i$), Forget gate ($f$), Output gate ($o$).
  • Equations:
    1. $g = \tanh(W_g [h_{t-1}, x_t] + b_g)$
    2. $i = \text{sigm}(W_i [h_{t-1}, x_t] + b_i)$
    3. $f = \text{sigm}(W_f [h_{t-1}, x_t] + b_f)$
    4. $o = \text{sigm}(W_o [h_{t-1}, x_t] + b_o)$
    5. $c_t = f \odot c_{t-1} + i \odot g$
    6. $h_t = o \odot \tanh(c_t)$

Multimodal Alignment: DTW and Attention

• Dynamic Time Warping (DTW): Finds the lowest cost path to align two unimodal temporal signals.
  • Cost: $L(p_t^x, p_t^y) = \sum_{t=1}^l \|x_{p_t^x} - y_{p_t^y}\|_2^2$.
  • Constraints: Monotonicity, continuity, boundary conditions.
• Canonical Time Warping (CTW): Combines DTW and CCA to align multimodal or multi-view data.
• Generalized Time Warping (GTW): Extension for multiple sequences of different modalities.
• Attention Models: Implicitly align modalities by weighting context features.
  • Image Captioning: Distribution over $L$ locations in an image while generating words.
  • Temporal Attention-Gated Model (TAGM): Uses recurrent attention gates for robust sequence classification.

Multimodal Fusion Techniques

• Model-Free Approaches:
  • Early Fusion: Feature concatenation before classification.
  • Late Fusion: Decision-level combination (voting, weighted sum).
  • Hybrid Fusion: Combines early and late architectures.
• Model-Based Approaches:
  • Multiple Kernel Learning (MKL): Learns weightings for different modality kernels.
  • Multi-View Hidden CRF: Models modality-private and modality-shared structures for sequential data.
  • Multi-View LSTM (MV-LSTM): Extends LSTM for multi-view structured learning with specific topologies (Coupled, Fully-connected, Hybrid).
  • Memory Fusion Network (MFN): Uses a Delta-memory Attention Network to accumulate information over time from diverse source networks.