Reinforcement Learning

What is the goal of reinforcement learning?

The goal of reinforcement learning is to learn a policy that maximizes the expected cumulative reward over time.

What is the difference between passive and active reinforcement learning?

Passive reinforcement learning involves a fixed policy, while active reinforcement learning involves changing the policy based on learned experiences.

What is value iteration in MDPs?

Value iteration is an algorithm used to compute the optimal values for states in an MDP by iteratively updating the value estimates until they converge.

What is policy extraction in the context of MDPs?

Policy extraction is the process of deriving the optimal policy from the computed state values.

What is Q-learning?

Q-learning is a model-free reinforcement learning algorithm that seeks to learn the value of actions in states, allowing an agent to learn how to act optimally.

What is the purpose of the reward function R(s,a,s')?

The reward function R(s,a,s') provides feedback to the agent by assigning a numerical value to the outcome of taking action a in state s and transitioning to state s'.

What is temporal difference learning?

Temporal difference learning is a reinforcement learning method that updates value estimates based on the difference between predicted and actual rewards.

What is the significance of the exploration-exploitation trade-off?

The exploration-exploitation trade-off refers to the balance between exploring new actions to discover their rewards and exploiting known actions that yield high rewards.

What is a bandit problem?

A bandit problem is a simplified reinforcement learning scenario where an agent must choose between multiple actions (arms) to maximize rewards without knowing the reward distribution.

What is model-based learning in reinforcement learning?

Model-based learning involves creating an approximate model of the environment based on experiences and using that model to make decisions.

What is the difference between model-based and model-free learning?

Model-based learning uses an estimated model of the environment to derive values, while model-free learning directly learns values from experiences without a model.

What is the purpose of policy iteration?

Policy iteration is an algorithm that alternates between policy evaluation and policy improvement to find the optimal policy in an MDP.

What does the term 'reward function' refer to in reinforcement learning?

The reward function quantifies the immediate benefit received after taking an action in a given state.

What is the role of an agent in reinforcement learning?

The agent is the entity that interacts with the environment, takes actions, and learns from the feedback received in the form of rewards.

How does Q-value iteration differ from value iteration?

Q-value iteration focuses on calculating the value of taking specific actions in states, while value iteration calculates the value of states themselves.

What is the significance of convergence in value iteration?

Convergence in value iteration indicates that the value estimates for states have stabilized and no longer change significantly with further iterations.

What does 'exploration' mean in the context of reinforcement learning?

Exploration refers to the agent's strategy of trying new actions to discover their potential rewards, rather than relying solely on known actions.

What is the expected outcome of a reinforcement learning agent?

The expected outcome is to maximize the cumulative reward over time through optimal decision-making.

What is the significance of the term 'policy' in reinforcement learning?

A policy is a strategy that defines the actions an agent will take in each state to maximize rewards.

What is the role of feedback in reinforcement learning?

Feedback in the form of rewards helps the agent learn which actions are beneficial and guides its future decision-making.

What is the difference between direct utility estimation and temporal difference learning?

Direct utility estimation evaluates the utility of actions based on observed outcomes, while temporal difference learning updates estimates based on the difference between predicted and actual rewards.

What is the purpose of the transition model in an MDP?

The transition model describes how the environment responds to the agent's actions, defining the probabilities of moving between states.

What is the significance of the term 'adversarial games' in AI?

Adversarial games involve scenarios where multiple agents interact, often with competing objectives, such as in chess.

What is the goal of passive reinforcement learning?

To learn values from experiences and use them to make decisions.

What are the two main families of approaches in reinforcement learning?

Model-based and model-free approaches.

What does direct evaluation in passive reinforcement learning aim to estimate?

It aims to estimate Vπ(s), the expected total discounted reward from state s onwards.

What is the method used in direct evaluation?

Using returns, which are the actual sums of discounted rewards from state s, averaged over multiple trials.

What is the main advantage of direct estimation?

It is easy to understand and does not require prior knowledge of transition probabilities or rewards.

What are the drawbacks of direct estimation?

It requires learning each state separately, ignores state connection information, and can take a long time to learn.

What is the purpose of temporal difference (TD) learning?

To learn from every experience by updating value estimates with each transition observed.

What is the running average method in TD learning?

It updates the value estimate using a running mean of observed values.

What is the Bellman equation used for in Q-learning?

To express the relationship between the value of a state-action pair and the expected return from that action.

What does Q*(s,a) represent?

The expected return from taking action a in state s and then behaving optimally thereafter.

What is the significance of the learning rate in Q-learning?

It determines how quickly the Q-values are updated based on new information.

What is off-policy learning in Q-learning?

It refers to the ability of Q-learning to converge to an optimal policy even when samples are generated from a suboptimal policy.

What is the main challenge in reinforcement learning regarding exploration?

Finding a balance between exploring new actions and exploiting known rewarding actions.

What is the role of the policy in reinforcement learning?

The policy defines the behavior of the agent, determining which actions to take in given states.

What does the term 'state' refer to in reinforcement learning?

A representation of the current situation or environment in which the agent operates.

What is the purpose of the Q-value table in Q-learning?

To store the estimated values of state-action pairs to guide the agent's decision-making.

What is the expected outcome of using Q-learning?

To learn the optimal policy that maximizes expected rewards over time.

What does the term 'discount factor' (γ) represent in reinforcement learning?

It represents the importance of future rewards compared to immediate rewards.

What is the significance of the sample mean in TD learning?

It provides an unbiased estimate of the value of a state based on observed rewards.

What is the effect of a fixed weight in running averages?

It allows for exponential forgetting of old values, giving more weight to recent observations.

What is the main idea behind learning from each experience in TD learning?

To continuously update value estimates based on new transitions and rewards observed.

What is the purpose of the running mean formula in reinforcement learning?

To maintain an updated average of values as new data is collected.

What is the challenge of scaling reinforcement learning to complex environments?

To efficiently explore large state spaces without excessive regret or computational cost.