Reinforcement Learning

Return

Total discounted long term future reward. To evaluate the decision process of the agent.

Estimates the quality of the agent’s sequence of actions

Agent goal

Maximizing cumulative rewards over time.

Markov Property

The furure is independent of the past given the present.

Value functions

Estimate how good it is for an agent following policy \pi to be in a certain state or how good it is to perform a certain action from a state, based on expected rewards.

State Value Function

Expected return starting from state s anf following policy \pi

State-action Value Function

Expected return starting from state s, performing action a and following policy \pi

Advantage Function

Relative measure of action quality.

Estimates how much better (or worse) an action is compared to the average action in a given state

Mainly used for variance reduction

If A(s,a)>0, action a is better than the expected return from state s

Bellman Equation

The value of a state equals the immediate reward plus the discounted value of future states

Recursive relationship

Bellman optimality equation for V*

What does P(s′∣s,a) mean?

The probability of transitioning to state s′ after taking action a in state s.

What does R(s,a,s′) represent?

The immediate reward received when transitioning from s to s′ after taking action a.

What does the discount factor γ do?

It determines how much future rewards matter relative to immediate rewards

What does a small γ mean?

The agent focuses more on immediate rewards.

What does a γ close to 1 mean?

The agent places more importance on long-term rewards.

Why does the Bellman optimality equation contain max_a​?

Because an optimal agent chooses the action with the greatest expected return.

What is the Bellman expectation equation for a policy π?

What is the difference between the Bellman expectation equation and Bellman optimality equation?

The expectation equation averages actions according to a fixed policy π, while the optimality equation chooses the best action using maxa​.

What question does policy evaluation answer?

How good is this particular policy?

What question does the Bellman optimality equation answer?

What is the best possible return I can achieve?

What is iterative policy evaluation?

Repeatedly applying the Bellman expectation update to estimate V^\pi

Why can value functions be estimated iteratively?

Because the value of a state depends recursively on the values of successor states.

What does V_k^\pi(s) mean?

The estimate of the value of state s under policy \pi after k iterations of policy evaluation.

What is value iteration?

An iterative method for estimating the optimal value function using:

What is the key difference between iterative policy evaluation and value iteration?

Policy evaluation uses (sum \pi(s|a)) = Weighted average

Value iteration uses max_a

iterative policy evaluation, you already have a policy π, and you want to find out how good it is. So you follow that policy. If the policy sometimes chooses different actions, you take the weighted average

value iteration: you want to find the optimal behavior. So instead of averaging according to a policy, you choose the best action:

What is the intuitive meaning of

What I get now + what I expect to get later.

What is the difference between V^\pi(s) and V*(s)?

Vπ(s) is the expected return from state s when following a specific policy π, while V*(s) is the maximum expected return achievable from s over all possible policies.

When do we use Vπ instead of V*?

Use Vπ when evaluating a fixed policy; use V* when reasoning about the optimal policy.