Introduction to Deep Reinforcement Learning

Course Overview and Logistics

  • Course Assignment and Instructor: The Deep Reinforcement Learning course (AML ZC 512) is taught by Professor Nipun.

  • Schedule and Structure:

    • Classes are conducted weekly from 10:30 to 12:30.

    • Sessions follow a structure with a logical stop after the first hour for a 10-15 minute break, followed by a second half.

    • Standard protocol disables participant microphones, but for this course, they are explicitly enabled to allow for real-time doubt clarification. Interruption during the flow of the lecture for clarification is encouraged over waiting for a final Q&A session.

  • Attendance and Participation:

    • Live attendance is highly recommended for working professionals despite weekend constraints because reinforcement learning is a continuous topic; failing to grasp one session significantly hinders understanding of the next.

    • The course is designed to move from basic Reinforcement Learning (RL) to Deep Reinforcement Learning (DRL).

Learning Objectives and Outcomes

  • Course Focus:

    • The primary focus is on the theoretical concepts and mathematical foundations of RL and DRL.

    • Mathematics is kept as light as possible, providing practical intuition for every equation rather than just theoretical derivation.

    • The curriculum covers implementation in various scenarios: planning, control, and decision-making.

  • Lab Components (Webinars):

    • Webinars are laboratory sessions handled by Teaching Assistants (TAs).

    • These sessions involve implementing DRL algorithms and analyzing outputs in Python environments.

  • Prerequisites:

    • Mathematics: A basic understanding of probability, specifically conditional probabilities, Bayes' theorem, and Markov decision processes.

    • Computing: Proficiency in Python programming using Jupyter Notebook or Google Colab.

  • Primary Textbook:

    • "Reinforcement Learning: An Introduction" by Richard S. Sutton and Andrew Barto.

    • This is the standard global text for the subject, covering concepts mathematically, practically, and intuitively.

    • Professor Richard S. Sutton remains active in the field and provides updates via social media platforms like Twitter.

Evaluation Schema

  • EC1 (Continuous Evaluation 1):

    • Includes two quizzes (5% weightage), where the best of two scores is recorded. No makeup exams are provided if both are missed.

    • Includes two programming assignments tied to the webinar laboratory sessions.

  • EC2 (Midterm Exam):

    • A theoretical, in-person, closed-book exam conducted at designated centers.

  • EC3 (Comprehensive Exam):

    • An open-book exam focusing on application, practical scenarios, and open-ended questions.

Conceptual Foundations of Reinforcement Learning

  • Core Definition: Reinforcement Learning is a type of machine learning where an agent learns from experience through interaction with an environment. It is goal-oriented learning under uncertainty.

  • Nature of Learning:

    • Trial and Error: The agent makes moves, encounters errors, and learns from them systematically.

    • Without Supervision: Unlike supervised learning, RL does not rely on a teacher or labeled dataset; it relies on self-discovery through interaction.

    • Reward-Based: Learning is driven by rewarding desired behaviors and punishing undesired ones.

  • The Walking Child Metaphor:

    • A child learning to walk acts as an agent. The brain processes interactions with the environment (the floor, furniture) without explicit instructions.

    • Initial Discovery: Standing up involves realizing that holding onto a chair prevents falling. Falling is a "negative reward" or penalty.

    • Fine-tuning: The child learns that widening their stance increases stability. The brain processes these "episodes" even during sleep, refining neural connections so performance improves the next day.

    • Complex Actions: Learning the gait requires balancing step length; too far forward leads to an off-balance fall, while internal adjustments lead to successful walking.

Comparisons in Machine Learning

  • Supervised Learning:

    • Uses labeled datasets (input/output pairs).

    • Goal: Learn a mapping from inputs to outputs (P(y∣x)P(y|x)).

    • Classic Example: Classifying dog vs. cat pictures based on existing labels.

  • Unsupervised Learning:

    • Uses unlabeled data.

    • Goal: Find underlying patterns or clusters in data.

    • Classic Example: Clustering animals into groups based on feature similarities without knowing the category names.

  • Reinforcement Learning:

    • Does not have a predefined dataset or labels.

    • Goal: Learn a policy (π(a∣s)\pi(a|s))—the probability of taking action aa given state ss—to maximize cumulative rewards.

    • Characteristics: Sequential decision-making where time plays a critical role and feedback is often delayed.

Key Elements of Reinforcement Learning

  1. Agent: The learner or decision-maker (e.g., the child or a robot's brain).

  2. Environment: Everything the agent interacts with (e.g., the physical world, a game board).

  3. Action (AtA_t): The moves made by the agent at a specific time step.

  4. State (StS_t): The current situation or configuration of the agent within the environment (e.g., coordinates of a robot or a specific board state in Tic-Tac-Toe).

  5. Reward (RtR_t): A numerical signal from the environment indicating success or failure. The agent acts greedily to maximize the cumulative reward over time.

  6. Policy (π\pi): The strategy the agent employs to determine the next action based on the current state. Learning RL means evolving this policy for optimality.

  7. Value Function (VV): A function that estimates the "goodness" of a state, representing the total reward an agent can expect to accumulate in the future starting from that state.

  8. Model of the Environment (Optional): A representation of how the environment behaves (state transitions). Knowing the model is helpful but not always necessary for learning.

Case Study: Tic-Tac-Toe and Temporal Difference Learning

  • System Setup:

    • States: Every possible configuration of the 3×33 \times 3 board (e.g., S1S_1 is empty, S2S_2 has one X).

    • Actions: Placing an X in any of the 9 available positions.

    • The Learning Table: A lookup table storing every state and its corresponding value function estimate.

  • The Temporal Difference (TD) Rule:

    • The value of a state is updated after taking an action and observing the next state.

    • The equation for updating the value of the current state (V(St)V(S_t)) is:       V(St)=V(St)+α×(V(St+1)−V(St))V(S_t) = V(S_t) + \alpha \times (V(S_{t+1}) - V(S_t))

    • In this equation:

      • V(St)V(S_t) is the value of the current state.

      • V(St+1)V(S_{t+1}) is the value of the succeeding state.

      • (V(St+1)−V(St))(V(S_{t+1}) - V(S_t)) is the Temporal Difference (TD) error.

      • α\alpha (Alpha) is the step-size parameter or learning rate, a hyperparameter between 00 and 11.

  • Practical Intuition of α\alpha:

    • If α=1\alpha = 1, the agent gives 100% weight to the difference, immediately updating the current state value to match the next state value.

    • If α=0.5\alpha = 0.5, only half of the difference is added, leading to a more gradual update.

    • A high alpha leads to faster but shallower learning; a lower alpha leads to more stable, deep learning over many episodes.

Real-World Applications

  • Self-Driving Cars: Swiat Robotics, an Indian company, retrofitted a Bolero to run autonomously in cluttered environments like Mumbai and Delhi. The vehicle is trained using DRL to navigate traffic and follow road rules.

  • Robotics and Drones: RL is used to control drone flight and mobile robot navigation. Because hardware is expensive, training usually begins in an 80-90% accurate simulation before transitioning to real hardware.

  • Gaming: Google's AlphaGo uses DRL to outperform human champions. RL is also effective for Atari games like Tetris or Breakout (paddle and ball games).

  • Healthcare (Drug Discovery): DRL helps in designing drugs for diseases like Cancer and Malaria. The agent learns the optimal sequence of elements to form a stable compound, which is significantly faster than traditional methods.

Questions & Discussion

  • Environment Interaction: A participant asked if knowing the environment model means having all inputs/outputs. It was clarified that knowing the environment model means understanding the state transition probabilities, but the agent still must interact to learn the best policy; data is not provided upfront.

  • Delayed Rewards: A question was raised about whether a failure (e.g., falling in a pit) is caused by the immediate action or a sequence of previous actions. The answer is that feedback in RL is often delayed; steps taken ten positions earlier might contribute to a crash, and TD learning propagates this reward info back through the states.

  • Production Readiness: A student asked if RL becomes supervised learning once it is "production-ready." The clarification provided was that even after the policy is learned and fixed, it remains the result of an interaction-based process, not a labeled dataset process.

  • Temporal Difference vs. Gradient Descent: While gradient descent minimizes a global loss function based on targets, TD learning compares current and succeeding states to locally update values based on observed goodness.