Chapter 1-8: Deep Learning & Imitation Learning - Vocabulary Flashcards

Logistics and course setup

Enrolled in class of 2020; use organized groups for questions; Ed Discussion is used for class questions and announcements; accounts linked or create new if needed.
Office hours/logistics: questions about logistics or lectures should be asked via Ed Discussion.
Homework plan: two to three weeks per assignment; programming-focused; first homework is a supervised behavior cloning problem with a provided environment and demonstrations; students fill in implementations based on lecture content.
Purpose of today: a rapid refresher on deep learning to prepare for Homework 1; assumes some introductory ML knowledge; recommended external resources for deeper understanding.
Resources mentioned:
- Stanford CS231n: Deep Learning for Computer Vision (lectures available online).
- Other courses: CMU, Berkeley, etc.
- Ian Goodfellow DL course (free online).
- Patterns, Predictions, and Actions; PyTorch tutorials.
Recommendation: if homework basics aren’t clear, review the cited resources, especially before starting Homework 1.

Quick recap of machine learning types

Supervised learning
- Input x, output y provided; model f maps x to y; trained with statistical learning techniques.
Unsupervised learning
- No outputs y provided; only inputs x; aim to learn structure from data.
Self-supervised learning
- Only inputs x are provided; model generates its own labels y; still learning y = f(x).
- Common in large language models (mass language modeling).
- Simple illustration: teacher provides a sequence; model fills in missing parts to learn the mapping.
Reinforcement learning (RL)
- An agent interacts with an environment; discovers actions y given observations x; uses a reward signal to optimize the mapping y = f(x).
- In this class, RL is a central focus, but we start with supervised learning basics.
Examples of how these concepts appear in practice:
- Large language models use self-supervised learning to predict missing tokens.
- Behavior cloning (a form of imitation learning) is a supervised learning approach to RL.

Supervised learning with a classic example: the Iris dataset

Problem setup: classify iris flower species based on four features.
- Input x:
- Sepal length, sepal width, petal length, petal width (four measurements).
- Output y: iris species ∈ {Setosa, Versicolor, Virginica} (three classes).
Training/test split (common default): 80/20 split between training and testing data.
Goal: learn a function f that maps x to y (x → y) via training on labeled data.
Conceptual distinction: optimization vs machine learning
- Optimization (in a traditional sense) finds an optimal solution for a given dataset without regard to generalization.
- Machine learning seeks models that generalize well to unseen data, evaluated via a separate test set.
Important question (often asked): What is the difference between optimization and ML?
- In ML, we care about generalization to new samples, not just minimizing error on the training set.
Models commonly considered for classification (in ML courses): KNN, linear classifiers, multinomial logistic regression, SVMs, neural networks.
In this RL-focused course, neural networks are the primary model used for learning policies.
Data split and evaluation: training set for learning, test set for evaluating generalization.

Neural networks: structure, training, and fundamentals

Anatomy of a neural network (illustrated as a standard feedforward network):
- Input layer, one or more hidden layers, and an output layer.
- Learnable parameters: weights and biases (collectively, θ).
- Forward pass: compute activations layer by layer; end with output ŷ.
Activation functions and their purpose
- Activation functions introduce nonlinearity to model complex functions and boundaries between classes.
- Without nonlinear activations, a multi-layer network collapses to a single linear transformation.
- ReLU (Rectified Linear Unit) is a common default:
- $\text{ReLU}(z) = \max(0, z)$
- Derivative:
  $\frac{d}{dz}\text{ReLU}(z) = \mathbf{1}_{z>0}$
- Why nonlinearities are necessary: real-world decision boundaries (e.g., separating blue vs red classes) are often not linearly separable; nonlinear activations allow the network to learn complex boundaries.
- Other activations include sigmoid and tanh, but ReLU is widely used due to computational efficiency and simple gradient.
Forward computation and output
- A simple 4-feature input example yields a few hidden units; outputs pass through an activation (e.g., softmax for multi-class classification).
- Softmax for multi-class output:
 $\text{softmax}(z)i = \frac{e^{zi}}{\sumj e^{zj}}$
Training dynamics: why gradients, learning rate, and gradient steps
- Objective: minimize a loss L(θ) that measures discrepancy between predictions ŷ and true labels y.
- Gradient descent update (one step):
  $\theta \leftarrow \theta - \eta \nabla_{\theta} \mathcal{L}(\theta)$
- η is the learning rate (a hyperparameter, e.g., η = 10^{-4} in many settings).
- Why gradients matter: gradients point in the direction to adjust weights to reduce loss; magnitude scaled by learning rate.
- Noisy gradients and mini-batches: in practice, gradients are estimated on batches, which introduces stochasticity; this can help escape poor local minima but requires proper learning rate tuning.
Backpropagation and automatic differentiation
- Explicitly computing analytical gradients for deep networks is impractical.
- Backpropagation uses the chain rule to compute derivatives efficiently from end to start (backward pass).
- Automatic differentiation frameworks (e.g., PyTorch, TensorFlow) build a computation graph during the forward pass, and loss.backward() computes all gradients automatically.
- Practical note: you can treat autograd as a "black box" in early coursework; you will implement tasks that rely on loss.backward().
From simple feedforward nets to deep nets
- Deep neural networks: stacking more layers (hidden layers) to form deep architectures.
- There is no formal cutoff between shallow and deep; it’s a matter of practical depth and learning capabilities.
From image data to convolutional networks (CNNs)
- Fully connected networks flatten images into a long vector and connect every input pixel to every neuron; this is inefficient for images due to:
- Exploding number of parameters for realistic image sizes.
- Lack of spatial structure exploitation; small translations should not drastically change activations.
- Convolutional neural networks (CNNs) address this by using filters/kernels that operate on local regions and slide across the image (weight sharing, parameter efficiency).
- Key benefits of CNNs:
- Spatial locality: captures local patterns (edges, textures).
- Translation invariance: same features detected anywhere in the image.
- Parameter sharing: fewer parameters than fully connected layers for images.
- Typical components: convolutional layers, pooling/downsampling, and eventually flatten to feed into a classifier (e.g., a fully connected head or global pooling).
- Famous CNN milestones: AlexNet, LeNet (for handwriting digit recognition).
- Practical workflow: multiple convolutional layers followed by a fully connected head for classification.

Convolutional neural networks: intuition and mechanics

How a convolution operation works
- Input: a feature map (image) and a filter (kernel) that slides across the input.
- For each position, the filter computes a dot product with the local region (the receptive field) to produce one value in the output feature map.
- As the filter slides, it creates a 2D (or 3D) output called a feature map.
- Parameters: the filter weights are shared across all positions, enabling translation invariance.
- Result: a reduced spatial dimension, depending on padding and stride.
Benefits summarized
- Spatial locality, translation invariance, and parameter sharing reduce the number of parameters and capture meaningful patterns like edges and textures.
Typical CNN architecture traits
- Sequences of convolutional layers to learn hierarchical representations.
- A final flattening step or global pooling to produce a vector fed into a classifier (often a small fully connected network).
Practical examples
- Early CNNs: LeNet; later: AlexNet; many modern networks are deeper and include residual connections (ResNet) or other architectural innovations.
Takeaway for this course
- CNNs are the standard for image-like inputs and are used in practice for vision-based RL, imitation learning, and perception modules.

Behavior learning and imitation learning

Core idea
- Imitation learning: a teacher demonstrates correct behavior; the learner tries to imitate those actions given states.
- The teacher provides state-action pairs; the agent learns a policy mapping states to actions, typically via supervised learning.
Behavior cloning (the simplest form)
- Assumes access to demonstrations without an interactive environment.
- Two main steps:
  1) Collect demonstrations: a dataset of sequences of (state, action) pairs.
  2) Train a policy to mimic the demonstrated actions (supervised learning).
How demonstrations are obtained in practice
- Human demonstrations, domain experts performing tasks, or previously trained policies.
- Kinesthetic teaching (e.g., robotics): physically guide the robot arm to demonstrate a motion.
- Teleoperation: a human operator controls a device (e.g., dual grippers on a moving platform) and records actions and states.
Real-world imitation-learning exemplars
- Autonomous driving (NVIDIA, 2016): train a CNN to map camera images to steering commands via behavior cloning; used to initialize policies before RL.
- Surgical robotics: demonstrations to learn controlling needle threading and knot tying.
- Hunting robots and other teleoperation-based tasks demonstrate recovery behaviors and robustness.
Key takeaway: starting from imitation learning provides a strong initialization for further RL-based improvement.
State and action representation
- States: current observations (e.g., camera frames, joint positions, velocities).
- Actions: discrete or continuous control commands (e.g., steering angle, joint torque).
On-policy vs off-policy imitation learning
- Behavior cloning typically uses the demonstrated trajectories directly (on-policy with respect to the collected data).
- Other methods (e.g., DAG) involve interaction and correction loops to improve robustness.

Multimodality and stochastic policies in imitation learning

Multimodality: there can be multiple viable actions in a given state (e.g., two equally good ways to move around an obstacle).
Deterministic mappings vs stochastic policies
- A deterministic policy may fail to capture multiple viable behaviors and can get stuck in local optima.
- A stochastic (probabilistic) policy can model multiple modes of behavior and explore alternative actions.
Diffusion models/policies (mentioned briefly for later in the course)
- Used to represent multiple behaviors by sampling actions from a learned distribution; capable of capturing multimodality.
- Will be covered in depth later; the idea is that these models generate sequences of actions and can reflect variability in behavior.

Dataset aggregation (DAgger) and iterative imitation learning

Dataset Aggregation (DAgger): a hybrid approach between imitation learning and RL.
- Access to a teacher policy and an interactive environment.
- Start with demonstrations from the teacher; gradually mix in actions from the student policy.
- Iteratively collect trajectories using a mix of teacher and student actions.
- Replace problematic student actions with teacher actions to correct errors, aggregating data over iterations.
- The data collection strategy typically involves a decaying probability beta that determines whether to sample from the teacher or the student at each step:
- Beta ∈ [0,1], often decays over time.
- Result: a richer dataset that covers states the student is likely to encounter, improving generalization and robustness.
High-level workflow for DAG
- Collect trajectories with a teacher-student mix (pi and pi*).
- Aggregate and retrain the policy on the growing dataset.
- Repeat until performance stabilizes.
Practical notes and caveats
- Requires access to a capable teacher policy and an environment capable of interaction.
- The teacher must be able to recover from mistakes (i.e., provide corrections when the student errs).
- In practice, such supervision may be difficult to obtain in some domains; synthetic data and simulation can help.

Real-world examples and considerations

Alternative demonstrations and data collection methods
- Surgical robotics demonstrations for precise, risky tasks.
- Teleoperation setups for complex manipulation tasks.
The role of data and distribution shift in imitation learning
- The IID (independent and identically distributed) assumption often does not hold in sequential data with dynamics.
- Sampling from a teacher may not cover the states the student will encounter; distribution shift can degrade performance.
- Small amounts of synthetic or augmented data can help bridge gaps (see data augmentation and synthetic data in later sections).
IID discussion and intuition
- For probabilistic models, training often assumes data are IID; violations can undermine formal guarantees.
- In RL-like settings with temporal dependence, successive states are highly correlated, and the next state depends on the current state and action.
- When the learner diverges from the expert trajectory, it enters unseen states, where the policy may be under-specified.
Illustrative example: logistic regression and likelihood
- For a binary classifier, the probability of a positive label given x is p = P(y = 1 | x).
- The likelihood over n data points is a product of Bernoulli probabilities; maximizing likelihood corresponds to minimizing negative log-likelihood (cross-entropy).
- IID assumption underpins many theoretical guarantees for ML algorithms; when violated, guarantees may fail.
Implications for real-world systems
- In car driving and robotics, distribution shifts occur due to weather, lighting, road conditions, traffic patterns, etc.
- To mitigate risks, approaches include data augmentation, synthetic data generation, and training on diverse scenarios.
- For long-tail or rare events (e.g., a deer jumping in front of a car), generation of synthetic data or specialized curricula can help.

Practical takeaways and connections to the course trajectory

Start with supervised behavior cloning to bootstrap RL-style policy learning without environment interaction.
Data quality and diversity are crucial; demonstrations should cover a range of states and scenarios.
Use of neural networks (and CNNs for image inputs) is central to this course; backpropagation with automatic differentiation makes training feasible on modern architectures.
Expect trade-offs: model complexity, data availability, and distribution shifts affect performance; plan to experiment with data augmentation, DAG, and multimodal policies.
The next topics will extend from this refresher into reinforcement learning specifics (non-Markovian dynamics, time-series considerations, and more advanced imitation learning methods).

Key formulas and concepts to memorize

Mapping notation and goal:
- Classification: input x maps to label y via a function f:
  $f: \mathcal{X} \to \mathcal{Y},\quad \mathcal{Y} = {\text{labels}}$
Training update (gradient descent):
- $\theta \leftarrow \theta - \eta \nabla_{\theta} \mathcal{L}(\theta)$
Activation functions (example):
- ReLU: $\text{ReLU}(z) = \max(0, z)$ ; discriminative derivative: $\frac{d}{dz}\text{ReLU}(z) = \mathbf{1}_{z>0}$
Softmax for multi-class output:
- $\text{softmax}(z)i = \frac{e^{zi}}{\sumj e^{zj}}$
Loss and likelihood (binary example):
- For binary classification with probability p = P(y=1|x): cross-entropy/negative log-likelihood
 $\mathcal{L}(\theta) = -\sum{i=1}^n [yi \log pi + (1-yi) \log(1-p_i)]$
Dataset split and generalization concept:
- 80/20 training/testing split is a common default; goal is to measure generalization beyond training data.
Diffusion/policy multimodality (conceptual):
- Multimodal action distributions can be captured with stochastic policies or diffusion-based policies to model multiple viable behaviors.
DAG (Dataset Aggregation) update idea (high level):
- Interleave teacher and student actions; aggregate datasets and retrain; progressively rely more on student actions while ensuring corrections from the teacher.

Ethical, philosophical, and practical implications

Distribution shift and safety:
- Real-world deployment must consider that training data (IID assumptions) may not cover all scenarios; testing and validation must be robust to shifts.
Data quality and bias:
- Demonstrations come from humans or heuristics; biases in demonstrations can propagate into learned policies.
Explainability and trust:
- Understanding what the network learns (e.g., feature maps in CNNs showing edge detection) helps with trust and debugging; visualizing intermediate features is encouraged.
Safety in robotics and autonomous systems:
- Imitation learning reduces the need for hard-to-tune RL exploration in dangerous settings; however, one must still ensure robust recovery from errors and manage long-tail risks with augmentation or additional training regimes.
Modularity and system design:
- Depending on the scenario, single monolithic policies vs modular, specialized policies can be advantageous; trade-offs include memory, adaptability, and performance.

Quick study pointers for Homework 1 and beyond

Focus on understanding supervised learning basics first: data splits, loss functions, gradient-based optimization, backpropagation, and CNN basics for image inputs.
Practice with simple PyTorch-style forward passes and a loss.backward() workflow to solidify autograd concepts.
Grasp the intuition behind why data augmentation and DAG can help mitigate distribution shift and improve robustness.
Be comfortable distinguishing IID assumptions and how they relate to generalization, especially in sequential/temporal data contexts.
Familiarize yourself with real-world imitation-learning examples (autonomous driving, robotic manipulation) to contextualize the methods.
Expect future topics to cover reinforcement learning in more depth, including non-Markovian dynamics, time-series modeling, and advanced imitation-learning methods (e.g., GAIL, diffusion policies).