Model-free-RL-DE

Artificial Intelligence and Intelligent Agents (F29AI)

• Focuses on Model-Free Reinforcement Learning (RL)

• Direct Evaluation by Arash Eshghi

• Contributions based on works from Ioannis Konstas, Verena Rieser, and Dan Klein.

Example: Expected Age

• Goal: Compute expected age of Heriot-Watt students.

• Approaches:

  • Model Based: Requires knowledge of probability P(A).

  • Model Free: Does not require prior probabilities but uses sample data.

• Why Model Free Works?:

  • Samples obtained reflect true distribution over time leading to accurate model learning.

HERIOT WATT University – Model-Free Learning

• Practical example with potential outcomes:

  • Earnings: $2/$0 or Nothing.

HERIOT WATT University – Passive Reinforcement Learning

• Simplified Task

  • Focuses on policy evaluation.

  • Input: Fixed policy p(s).

  • Unknown Elements:

    • Transitions T(s,a,s').

    • Rewards R(s,a,s').

  • Goal: Learn state values.

  • Key Concept: Learner follows the existing policy without altering actions or engaging in offline planning.

Direct Evaluation

• Objective: Compute value estimates for each state under the policy p.

• Method:

  • Average observed sample values.

  • Record the discounted rewards encountered at each state while acting according to p.

  • This technique is termed direct evaluation.

Example: Direct Evaluation Input Policy p

• Assumption: Discount factor g = 1.

• Observational Episodes during training lead to output state values:

  • A:

  • B:

  • C:

  • D:

  • E:

• Sample Outputs:

  • B: +8

  • C: +4

  • D: +10

  • E: -10

  • A: -2

Problems with Direct Evaluation

• Advantages:

  • Clear understanding of the process.

  • No requirement for knowledge of T or R.

  • Correct average values are eventually computed using sample transitions.

• Disadvantages:

  • Neglects connections between states, leading to inefficient learning.

  • Learning must occur independently for each state, prolonging the process.

  • Example Query: Why can the values from states B and E to C differ under the same policy?

Suggested Readings

• Text: Russell & Norvig, Chapters 21.1-3.

Looking Ahead

• Upcoming Lectures:

  • Lecture 14: Introduction to NLP.

  • Lecture 15: Language Modelling.

  • Lecture 16: Syntactic Parsing as a Search Problem.

  • Lecture 17: If time permits, covering Perceptrons & Deep Learning.