List of Questions for Entrance Test - 2. Reinforcement Learning

Describe the basic idea of Reinforcement Learning in 2-3 sentences

Reinforcement learning is a class of problems in which an agent must learn how to behave through a process of trial-and-error interactions within a dynamic environment. The agent receives a scalar reinforcement signal that acts as a reward or punishment, and its goal is to choose actions that maximize the long-run sum of these values. Unlike supervised learning, the agent is never told which action is best; instead, it must actively gather experience to determine the optimal policy for achieving its task

Name the components of the Reinforcement Learning approach and describe them briefly.

• Agent: The entity that learns behavior through trial-and-error interactions. It is responsible for perceiving the state, choosing actions, and receiving reinforcement.

• Environment: The dynamic system with which the agent interacts. It transitions between states in response to the agent's actions and provides feedback in the form of reinforcement signals

• States (S): A discrete set representing the different possible configurations or situations of the environment. At each step, the agent receives an indication of the current state (s)

• Actions (A): A discrete set of possible outputs or moves the agent can make. Choosing an action (a) causes the environment to transition to a new state

• Reinforcement Signal (r): A scalar value (typically 0, 1, or a real number) communicated to the agent after a state transition. It serves as a reward or punishment that the agent seeks to maximize over the long run

• Policy (pi): The agent's internal strategy or mapping from states to actions. The goal of reinforcement learning is for the agent to find an optimal policy that maximizes its long-term reinforcement

Name 3 possibilities to measure the quality of learning performance. Describe each approach briefly.

1. Eventual convergence to optimal

2. Speed of convergence to optimality

3. Regret

Briefly describe the main differences between Reinforcement Learning and Supervised Learning.

Reinforcement learning differs from supervised learning because it lacks explicit input/output training pairs; instead, an agent must discover optimal behavior through trial-and-error interactions and scalar rewards. This creates a necessary trade-off between exploiting known rewards and exploring the environment to find better ones. Unlike supervised systems, RL performance is typically evaluated "on-line" as the agent learns, rather than after a separate training phase

Name the components of Markov decision process and describe them briefly

• States (S): A finite set of possible configurations or "situations" the environment can be in.

• Actions (A): A finite set of choices available to the agent in each state

• Transition Function (T): A model that defines the probability of moving from one state to another after taking a specific action, denoted as T(s, a, s'). It captures the dynamics and potential non-determinism of the environment

• Reward Function (R): A mapping that provides a scalar reinforcement signal (reward or punishment) based on the state transitions, denoted as R(s, a)

Explain the approach of Policy Shaping in a few sentences

Policy Shaping is an interactive reinforcement learning approach where an external trainer provides direct advice on which actions the agent should take. Unlike "reward shaping," which modifies the environment's reward signal, policy shaping allows a trainer to intercept and replace the agent's selected action with a more suitable one before it is executed.

Explain the approach of Reward Shaping in a few sentences

Reward Shaping is an interactive reinforcement learning approach where an external trainer provides additional reward signals to supplement the environment's original reward function. Unlike policy shaping, which influences the action selection directly, reward shaping modifies the reinforcement signal itself to guide the agent toward desired behaviors.

Name 4 open challenges in the field of Reinforcement Learning and explain each in 1-2 sentences

1. Feedback Modality and Combinations: While most current systems use a single form of feedback (like binary rewards), an open challenge is effectively combining multiple modalities—such as natural language, gestures, and evaluative signals—to allow more natural and flexible teaching

2. Robustness to Imperfect Human Input: Humans are often inconsistent, provide delayed feedback, or make mistakes; developing algorithms that can distinguish between intentional guidance and "noisy" or incorrect human signals is a critical hurdle

3. User Modeling and Personalization: Creating agents that can adapt to different teaching styles and the varying levels of expertise of different users remains difficult, as the system must understand the human's underlying intent rather than just their raw actions

4. Scaling to Complex Real-World Environments: Moving beyond simple grid-worlds or simulations to high-dimensional, continuous real-world tasks (like home robotics) is challenging because human attention is limited and cannot provide the massive amount of guidance typically required by deep RL models.

Describe the Use Case described in Chapter 3 of the paper by Zinn, Vogel-Heuser, and Ockier (2020)

The use case described in Chapter 3 of the paper by Zinn, Vogel-Heuser, and Ockier (2020) focuses on an automated sorting system within a simulated production environment. The system's objective is to correctly identify and sort workpieces by color using a Programmable Logic Controller (PLC) controlled by a Deep Q-learning agent.

The production system is characterized by multiple end-effectors that actuate in one or two axes, creating a challenging control task where only a few actuators affect a specific workpiece at any given time. The study specifically evaluates how well different Deep Reinforcement Learning algorithms can generalize their sorting strategy from 30 known workpiece combinations to all 81 possible variations

Describe the challenge of using Reinforcement Learning for industrial control.

According to chapter 2 of the paper by Zinn, Vogel-Heuser, and Ockier (2020), the challenge of using Reinforcement Learning for industrial control lies in the high complexity and multidimensionality of Automated Production Systems (aPS). Specifically, these systems often feature a large number of actuators and multiple end-effectors, making it difficult to track workpieces within a single, consistent coordinate system.

Additionally, industrial environments require high robustness and real-time capabilities, yet traditional RL often suffers from the "curse of dimensionality" when applied to these high-dimensional state-action spaces. The authors highlight that while RL offers the potential for self-learning and fault-tolerant control, bridging the gap between simulated training and reliable PLC-based execution remains a significant engineering hurdle.