Introduction to Deep Reinforcement Learning
Course Overview and Logistics
Course Assignment and Instructor: The Deep Reinforcement Learning course (AML ZC 512) is taught by Professor Nipun.
Schedule and Structure:
Classes are conducted weekly from 10:30 to 12:30.
Sessions follow a structure with a logical stop after the first hour for a 10-15 minute break, followed by a second half.
Standard protocol disables participant microphones, but for this course, they are explicitly enabled to allow for real-time doubt clarification. Interruption during the flow of the lecture for clarification is encouraged over waiting for a final Q&A session.
Attendance and Participation:
Live attendance is highly recommended for working professionals despite weekend constraints because reinforcement learning is a continuous topic; failing to grasp one session significantly hinders understanding of the next.
The course is designed to move from basic Reinforcement Learning (RL) to Deep Reinforcement Learning (DRL).
Learning Objectives and Outcomes
Course Focus:
The primary focus is on the theoretical concepts and mathematical foundations of RL and DRL.
Mathematics is kept as light as possible, providing practical intuition for every equation rather than just theoretical derivation.
The curriculum covers implementation in various scenarios: planning, control, and decision-making.
Lab Components (Webinars):
Webinars are laboratory sessions handled by Teaching Assistants (TAs).
These sessions involve implementing DRL algorithms and analyzing outputs in Python environments.
Prerequisites:
Mathematics: A basic understanding of probability, specifically conditional probabilities, Bayes' theorem, and Markov decision processes.
Computing: Proficiency in Python programming using Jupyter Notebook or Google Colab.
Primary Textbook:
"Reinforcement Learning: An Introduction" by Richard S. Sutton and Andrew Barto.
This is the standard global text for the subject, covering concepts mathematically, practically, and intuitively.
Professor Richard S. Sutton remains active in the field and provides updates via social media platforms like Twitter.
Evaluation Schema
EC1 (Continuous Evaluation 1):
Includes two quizzes (5% weightage), where the best of two scores is recorded. No makeup exams are provided if both are missed.
Includes two programming assignments tied to the webinar laboratory sessions.
EC2 (Midterm Exam):
A theoretical, in-person, closed-book exam conducted at designated centers.
EC3 (Comprehensive Exam):
An open-book exam focusing on application, practical scenarios, and open-ended questions.
Conceptual Foundations of Reinforcement Learning
Core Definition: Reinforcement Learning is a type of machine learning where an agent learns from experience through interaction with an environment. It is goal-oriented learning under uncertainty.
Nature of Learning:
Trial and Error: The agent makes moves, encounters errors, and learns from them systematically.
Without Supervision: Unlike supervised learning, RL does not rely on a teacher or labeled dataset; it relies on self-discovery through interaction.
Reward-Based: Learning is driven by rewarding desired behaviors and punishing undesired ones.
The Walking Child Metaphor:
A child learning to walk acts as an agent. The brain processes interactions with the environment (the floor, furniture) without explicit instructions.
Initial Discovery: Standing up involves realizing that holding onto a chair prevents falling. Falling is a "negative reward" or penalty.
Fine-tuning: The child learns that widening their stance increases stability. The brain processes these "episodes" even during sleep, refining neural connections so performance improves the next day.
Complex Actions: Learning the gait requires balancing step length; too far forward leads to an off-balance fall, while internal adjustments lead to successful walking.
Comparisons in Machine Learning
Supervised Learning:
Uses labeled datasets (input/output pairs).
Goal: Learn a mapping from inputs to outputs ().
Classic Example: Classifying dog vs. cat pictures based on existing labels.
Unsupervised Learning:
Uses unlabeled data.
Goal: Find underlying patterns or clusters in data.
Classic Example: Clustering animals into groups based on feature similarities without knowing the category names.
Reinforcement Learning:
Does not have a predefined dataset or labels.
Goal: Learn a policy ()—the probability of taking action given state —to maximize cumulative rewards.
Characteristics: Sequential decision-making where time plays a critical role and feedback is often delayed.
Key Elements of Reinforcement Learning
Agent: The learner or decision-maker (e.g., the child or a robot's brain).
Environment: Everything the agent interacts with (e.g., the physical world, a game board).
Action (): The moves made by the agent at a specific time step.
State (): The current situation or configuration of the agent within the environment (e.g., coordinates of a robot or a specific board state in Tic-Tac-Toe).
Reward (): A numerical signal from the environment indicating success or failure. The agent acts greedily to maximize the cumulative reward over time.
Policy (): The strategy the agent employs to determine the next action based on the current state. Learning RL means evolving this policy for optimality.
Value Function (): A function that estimates the "goodness" of a state, representing the total reward an agent can expect to accumulate in the future starting from that state.
Model of the Environment (Optional): A representation of how the environment behaves (state transitions). Knowing the model is helpful but not always necessary for learning.
Case Study: Tic-Tac-Toe and Temporal Difference Learning
System Setup:
States: Every possible configuration of the board (e.g., is empty, has one X).
Actions: Placing an X in any of the 9 available positions.
The Learning Table: A lookup table storing every state and its corresponding value function estimate.
The Temporal Difference (TD) Rule:
The value of a state is updated after taking an action and observing the next state.
The equation for updating the value of the current state () is:
In this equation:
is the value of the current state.
is the value of the succeeding state.
is the Temporal Difference (TD) error.
(Alpha) is the step-size parameter or learning rate, a hyperparameter between and .
Practical Intuition of :
If , the agent gives 100% weight to the difference, immediately updating the current state value to match the next state value.
If , only half of the difference is added, leading to a more gradual update.
A high alpha leads to faster but shallower learning; a lower alpha leads to more stable, deep learning over many episodes.
Real-World Applications
Self-Driving Cars: Swiat Robotics, an Indian company, retrofitted a Bolero to run autonomously in cluttered environments like Mumbai and Delhi. The vehicle is trained using DRL to navigate traffic and follow road rules.
Robotics and Drones: RL is used to control drone flight and mobile robot navigation. Because hardware is expensive, training usually begins in an 80-90% accurate simulation before transitioning to real hardware.
Gaming: Google's AlphaGo uses DRL to outperform human champions. RL is also effective for Atari games like Tetris or Breakout (paddle and ball games).
Healthcare (Drug Discovery): DRL helps in designing drugs for diseases like Cancer and Malaria. The agent learns the optimal sequence of elements to form a stable compound, which is significantly faster than traditional methods.
Questions & Discussion
Environment Interaction: A participant asked if knowing the environment model means having all inputs/outputs. It was clarified that knowing the environment model means understanding the state transition probabilities, but the agent still must interact to learn the best policy; data is not provided upfront.
Delayed Rewards: A question was raised about whether a failure (e.g., falling in a pit) is caused by the immediate action or a sequence of previous actions. The answer is that feedback in RL is often delayed; steps taken ten positions earlier might contribute to a crash, and TD learning propagates this reward info back through the states.
Production Readiness: A student asked if RL becomes supervised learning once it is "production-ready." The clarification provided was that even after the policy is learned and fixed, it remains the result of an interaction-based process, not a labeled dataset process.
Temporal Difference vs. Gradient Descent: While gradient descent minimizes a global loss function based on targets, TD learning compares current and succeeding states to locally update values based on observed goodness.