Lecture 32 Sequential Decision Problems

Sequential Decision Making

• Building upon rational one-step decision making by maximizing expected utility. This involves making the best choice at each step, considering the potential outcomes and their associated values, to achieve a specific goal.

• Extending decision-making to sequences of actions. Rather than making isolated decisions, this approach considers the impact of a series of interconnected choices over time.

• Introduction to Markov Decision Processes (MDPs) as the mathematical foundation for reinforcement learning. MDPs provide a framework for modeling decision-making in situations where outcomes are uncertain and depend on the actions taken.

Background Information

• Rational One-Step Decision Making: This is the foundation, focusing on making the best choice in a single instance by evaluating potential outcomes and their utilities.

• Markov Decision Processes (MDPs): These are mathematical models used in reinforcement learning to formalize sequential decision-making problems. They are essential for creating systems that can learn to make optimal decisions through trial and error.

Markov Decision Processes (MDPs)

• Basics of MDPs. Includes defining the state space, action space, transition probabilities, and reward functions.

• Utility values for a Markov decision process. Represents the long-term benefit of being in a particular state or taking a specific action.

• Algorithms for computing utility values and optimal decisions. Such as value iteration and policy iteration, which are used to find the best possible strategy.

Background Information

• State Space: All possible situations or states an agent can be in.

• Action Space: The set of actions an agent can take in each state.

• Transition Probabilities: The likelihood of moving from one state to another given an action.

• Reward Functions: The immediate reward received after performing an action in a particular state.

Reinforcement Learning

• MDPs assume known transition model (probability of outcomes for each action). This means you know how likely each action is to result in different states.

• MDPs assume known reward function (utility of a sequence of actions is the sum of rewards). This means you know the immediate reward you get for each action.

• Reinforcement learning addresses scenarios where the reward function and transition model are unknown, requiring learning from experience. Instead of being given the probabilities and rewards, the agent must learn them by trying different actions and observing the outcomes.

Context

• Transition Model: A function describing the probability of moving to a new state after taking a specific action in a given state. In simpler terms, it tells you what will happen when you take an action.

• Reward Function: A function that defines the immediate reward an agent receives after taking an action in a specific state. It’s the feedback the agent gets for its actions.

Sequential Decision Problems

• Utility depends on the sequence of actions. The overall value or benefit is determined by the entire series of decisions, not just individual actions.

• Stochastic environment: non-deterministic outcomes with associated probabilities. Outcomes are not certain; instead, there’s a probability associated with each possible outcome.

Background

• Utility: A measure of the desirability of different outcomes or states.

• Stochastic Environment: An environment where the outcomes of actions are not predictable with certainty.

Transition Probability

• Conditional probability of moving to state $s'$ from state $s$ by taking action $a$: $P(s' | s, a)$.

• A fully specified transition model includes these probabilities for all state pairs and actions.

• In reinforcement learning, this transition model is learned.

Explanation

• Conditional Probability: The probability of an event occurring given that another event has already occurred. In this context, it’s the probability of ending up in state $s'$ given that you started in state $s$ and took action $a$.

• Transition Model: A comprehensive set of these probabilities for all possible state transitions. Understanding this model is critical for making informed decisions.

Simplifying Assumption (Markov Property)

• The probability of ending up in $s'$ given $s$ and $a$ is independent of previous states.

• The history of states doesn't matter; only the current state affects the outcome of an action.

• Conditional independence: $P(s' | s, a)$ is independent of earlier states given state $s$.

• Analogy to probabilistic language models where the next word depends only on the current word.

Background

• Markov Property: This property simplifies the problem by assuming that the future state depends only on the current state and action, not on the entire history of previous states and actions.

• Conditional Independence: This means that once the current state is known, the previous states do not provide any additional information about the future state.

Bounded Reward Function

• The reward received for any given action is limited (cannot be infinite).

• Reward function: specifies the numeric reward $R(s, a, s')$ for performing action $a$ to transition from state $s$ to $s'$.

Explanation

• Bounded Reward Function: This ensures that no single action can result in an infinitely large reward, which helps in making the problem solvable.

• Reward Function: This assigns a numerical value to each transition, indicating how desirable that transition is. The goal of the agent is to maximize the cumulative reward over time.

Utility Maximization

• Utility is the sum of rewards obtained during the sequence of actions.

• Discounted rewards: rewards further in the sequence matter less than immediate rewards.

Details

• Utility: A measure of the total reward an agent expects to receive over time by following a particular policy.

• Discounted Rewards: This technique reduces the impact of future rewards, making immediate rewards more significant. This is crucial for ensuring that the agent doesn’t wait indefinitely for a far-off reward.

Markov Decision Process Characteristics

• Environment is fully observable (like search algorithms).

• Does not consider partially observable Markov decision processes (though generalization is possible).

Explanation

• Fully Observable Environment: The agent knows the current state of the environment perfectly. This simplifies the problem because the agent doesn’t need to guess or estimate its current state.

• Partially Observable MDPs: These are more complex and involve cases where the agent only has partial or noisy information about the current state.

Example: Robot Navigation

• Robot in a room needs to return to a charging station (reward +1).

• Hole in the room leads to robot being stuck (reward -1).

• Negative reward (-0.04) for each action encourages shorter solutions.

• The agent cannot move to certain locations.

Scenario

• Imagine a robot tasked with navigating a room to reach its charging station. The robot gets a positive reward for reaching the charger and a negative reward for falling into a hole. To encourage efficiency, there’s a small negative reward for each step the robot takes.

Probabilistic Environment

• Actions have uncertain outcomes.

• If the robot tries to move up, it might move up with a probability of 0.8, move to the left with a probability of 0.1, and move to the right with a probability of 0.1.

• The transition model is assumed to be similar for each state-action pair.

Explanation

• Probabilistic Environment: In the real world, actions don’t always have the intended outcome. There’s a chance that the robot might slip or be pushed off course.

• Transition Model: This defines the probabilities of different outcomes for each action.

Policies

• A solution (policy) specifies the optimal action for every state, not just a single list of states.

• The policy is commonly denoted by the Greek letter $\pi$.

• An optimal policy maximizes the expected utility.

Definition

• Policy: A strategy that tells the agent what action to take in each possible state. It’s the agent’s “plan of action.”

• Optimal Policy: The policy that results in the highest possible cumulative reward over time.

Policy Evaluation

• Follow the policy to evaluate its quality.

• Execute the policy multiple times (episodes) to account for probabilistic transitions.

• Calculate the sum of rewards for each episode.

• Average the sums of rewards across all episodes to estimate the policy's expected utility.

• Monte Carlo policy evaluation: estimating the expected utility of the policy by running trials and averaging the obtained utility values.

Methods

• Policy Evaluation: Assessing how good a policy is by estimating the expected utility of following it.

• Monte Carlo Method: A technique that relies on repeated random sampling to obtain numerical results. In this context, it involves running multiple simulations and averaging the results to estimate the policy's expected utility.

Policy Optimization

• Use a genetic algorithm to find the best policy with the highest expected utility.

• Alternatively, use value iteration and policy iteration algorithms (to be discussed).

Techniques

• Genetic Algorithm: An optimization algorithm inspired by natural selection. It involves creating a population of policies, evaluating their performance, and iteratively improving them through processes like mutation and crossover.

• Value Iteration and Policy Iteration: Dynamic programming algorithms used to find the optimal policy by iteratively improving estimates of state values and policies.

Optimal Policy Example

• An optimal policy specifies an action for each state (e.g., move up, move left).

• The optimal policy depends on the reward setup.

Illustration

• For each possible state, the optimal policy tells the agent the best action to take. This ensures the agent always makes the most effective move towards its goal.

Impact of Reward Structure on Policies

• Positive reward for each move encourages the agent to stay in the environment.

• Large negative reward encourages the agent to reach the goal state quickly.

Insight

• The way rewards are structured significantly influences the agent’s behavior. For example, if there’s a reward for each step, the agent might wander aimlessly. Conversely, a large penalty for time spent encourages quick solutions.

Discounted Rewards

• Rewards encountered further in the sequence are discounted to ensure finite utility values even for infinite sequences.

• Discount factor $\gamma$ is between 0 and 1.

• Utility of an episode (sequence of states and actions): $U(s<em>0, a</em>0, s<em>1, a</em>1, …) = \sum<em>{t=0}^{\infty} \gamma^t R(s</em>t, a<em>t, s</em>{t+1})$.

Rationale

• Discounted Rewards: This technique addresses the issue of infinite rewards by reducing the value of future rewards. It ensures that the agent prefers immediate rewards over distant ones.

• Discount Factor ($\gamma$): A value between 0 and 1 that determines how much future rewards are discounted. A higher value means future rewards are valued more.

Infinite Horizon MDPs

• With discounted rewards, infinite sequences of actions can have finite utility.

• Assuming bounded individual rewards (between +Rmax and -Rmax), the sum of discounted rewards is less than or equal to $R_{max} / (1 - \gamma)$.

Explanation

• Infinite Horizon MDPs: These models deal with continuous tasks where there is no clear end. Discounting ensures that the total reward remains finite and manageable.

Expected Utility

• Approximate the expected utility by executing the policy many times, computing the discounted sum of rewards, and averaging the results.

Details

• Expected Utility: The average utility an agent expects to receive by following a particular policy over many episodes. This is a key metric for evaluating the effectiveness of a policy.

Utility of a State

• The utility of a state $s$ with respect to policy $\pi$ is the expected utility of executing the policy starting in state $s$.

• The optimal policy maximizes utility across all possible policies.

• The optimal policy for an infinite horizon MDP with discounted rewards does not depend on the starting state.

Definition

• Utility of a State: The long-term value of being in a particular state, given that the agent follows a specific policy.

• Optimal Policy: The policy that maximizes the utility of all states.

Bellman Equation

• $U(s) = \max<em>{a} \sum</em>{s'} P(s' | s, a) [R(s, a, s') + \gamma U(s')]$

• The utility of a state $s$ is the expected utility of the best available action for that state, assuming optimal actions are taken afterward.

• The equation relates the utility of the current state to the utility of the next state.

• It can be solved iteratively using value iteration and policy iteration algorithms.

Explanation

• Bellman Equation: A fundamental equation in dynamic programming that expresses the relationship between the value of a state and the values of its successor states. It is used to compute the optimal policy in an MDP.

Extracting the Policy from Utility Values

• $\pi^*(s) = \arg \max<em>{a} \sum</em>{s'} P(s' | s, a) [R(s, a, s') + \gamma U(s')]$

• Given the utility values of states, this equation indicates the optimal action at a given state. It represents the maximization over immediate reward plus discounted future reward based on optimal policy.

• Iterate through the states based on the calculated data and choose the move that earns the maximum value from the formula.

Details

• Extracting the Policy: Once the utility values for each state are known, this equation allows the agent to determine the best action to take in each state.

Q-Function (Action-Utility Function)

• $Q(s, a)$ represents the expected utility of taking action $a$ in state $s$.

• $U(s) = \max_{a} Q(s, a)$

• $Q(s, a) = \sum<em>{s'} P(s' | s, a) [R(s, a, s') + \gamma \max</em>{a'} Q(s', a')]$

• Optimal action: $\pi^*(s) = \arg \max_{a} Q(s, a)$

• The Q-function allows for directly reading off the optimal decision without additional computation.

Explanation

• Q-Function: This function represents the quality of taking a specific action in a given state. It provides a way to directly assess the value of each action, making it easier to choose the optimal one.