World Models, Exploration, and Actions

3 objectives to put agent in interactive environments (intertwined)

model world, explore world, make right action.

Agent Exploration-Exploitation? Tradeoff? (RL/Interactive Environments)

Exploration: deliberately select actions with uncertain consequences to explore environment and discover better long-term behavior.

Exploitation: leverage existing knowledge to select action yielding highest, immediate reward

The tradeoff is the choice between exploration and exploitation.

Measure Exploration

Action Entropy, State Coverage, Information Gain, Regret, Pass@k (LLMs)

Common failure modes in World Models

Policy exploits errors in world model

Agent explores things that are novel but useless (noisy TV problem)

World Models, Policy, World Action Models? Why is it good to combine world models and policy?

World Models: predict next state given action and state

Policy: predict next action given state

World Action Models: predict next action and state given action and state

Pretraining world model provides strong prior environment knowledge (physics) essential for policy.

RL Terminology: States, Actions, Observations, Rewards

what makes interactive setting different from predicting from previous data?

In the interactive/embodied setting, the probability distribution for actions at a state is defined/learend by an agent/algorithm, not predefined. Unlike generative modeling, we don’t have to study a fixed policy distribution, actions are like causal interventions (based on consequences)

Open vs. Closed loop planning and when suitable? Random shooting w/ world model? Cross-entropy?

Open-loop planning: produce an action sequence once. Deterministic

Closed-loop planning: produce actions per step using a policy. Stochastic

Random shooting: try multiple action sequences in the world model, then execute the best first action.

Cross-entropy: iteratively refine optimal action trajectories from some distribution; sample actions from P(A) distribution, pick M actions with highest value, refit P(A) on M elites