Do Large Language Models Learn World Models or Just Surface Statistics

Fundamental Problem: Surface Statistics vs. World Models in LLMs

• Mechanism of Large Language Models (LLMs):

  • LLMs utilize a simplistic training paradigm: next-token prediction ("guess-the-next-word") executed over vast datasets with substantial computational resources.

  • Training loop details: the model receives a partial sequence, predicts the subsequent word, and updates its internal parameters to reinforce correct guesses or adjust based on prediction errors.

  • Scale-driven emergence: scaling model capacity and training data yields qualitative breakthroughs, enabling models to generate software code, answer reasoning prompts, and solve logic puzzles.

• Core Debate: Memorization vs. Emergent World Models:

  • Surface Statistics Viewpoint: theoretical arguments in philosophy and mathematics suggest that training solely on ungrounded text tokens limits models to memorizing surface-level statistics, statistical correlations, and sequence patterns without acquiring underlying semantics or causal models of world processes.

  • World Model Viewpoint: an alternative hypothesis posits that optimizing next-token prediction forces neural networks to build internal representations of the causal data-generating process (a world model).

  • Practical Implications: distinguishing between these mechanisms is vital for AI alignment, purging spurious correlations, and ensuring robust performance on out-of-distribution data.

The Crow Parable: A Thought Experiment

• Premise and Setting:

  • Two friends play Othello regularly in a room with an open window, verbally announcing every move using standard board notation without discussing game rules or strategies.

  • A crow sitting outside on the windowsill observes these verbal calls over many games without ever seeing the board.

  • Eventually, the crow begins calling out moves that are consistently legal under standard Othello rules.

• Investigating Internal Representations:

  • Initial Question: Is the crow stitching together surface statistics (e.g., common opening moves or frequencies of corner calls late in the game), or is it tracking internal board states?

  • Observation of Physical Representation: a grid-like arrangement of two types of birdseed on the windowsill matches the current $8 \times 8$ Othello board state. The crow actively nudges seeds with its beak following each called move and examines the grid before announcing its next move.

• Causal Intervention Experiment (The Prank):

  • While distracting the crow, a human changes the seed grid to a completely new board configuration.

  • When the crow inspects the modified seed grid, it announces a move that is legal exclusively under the new, modified board state.

  • Conclusion: the crow relies on an interpretable and controllable representation of the game state to generate predictions, demonstrating functional use of an internal world model beyond static statistical memorization.

Othello-GPT: A Synthetic Testbed

• Experimental Setup:

  • Model Architecture: a GPT variant trained strictly on sequence streams of Othello game transcripts (termed Othello-GPT) without any explicit rule input, board geometry, or prior game instructions.

  • Othello Mechanics: played on an $8 \times 8$ grid by two players (black and white) alternating placing discs. Every legal move must outflank and flip at least one opponent disc along a straight line. The game ends when no legal moves remain, and the player with the majority of discs wins.

• Distinction from Reinforcement Learning (AlphaGo):

  • AlphaGo / Traditional RL: game rules, board structures, and victory conditions are hardcoded; game histories serve to maximize winning probability.

  • Othello-GPT: receives no predefined board structure or game rules; transcripts are treated as raw sequences. Each tile coordinate is tokenized as a distinct word (e.g., E3, D3, C4). The training objective is solely predicting the next move token.

• Empirical Legal Move Accuracy:

  • Untrained Baseline Error Rate: $93.29\%$

  • Trained Othello-GPT Error Rate: $0.01\%$

  • The dramatic drop in error demonstrates that the sequence model successfully learns to predict legal moves with high accuracy.

Probing Internal Representations in Othello-GPT

• Probing Methodology:

  • Probing is an established interpretability technique in Natural Language Processing (NLP) that uses linear or low-capacity classifiers (probes) on intermediate neural layer activations to predict specific properties of the input data.

  • Assumes that if a simple classifier can accurately predict target properties from layer activations, the underlying network has encoded those concepts within its representations.

• Othello Tile Classification Setup:

  • Probe architecture: $64$ independent two-layer Multi-Layer Perceptron (MLP) classifiers, corresponding to the $64$ tiles on the $8 \times 8$ Othello board.

  • Target state: classifying each tile into one of $3$ discrete states: black, white, or blank.

  • Input: intermediate layer activations extracted from Othello-GPT during game processing.

• Probing Accuracy Results:

  • Randomly-Initialized Othello-GPT Probe Error Rate: $26.2\%$

  • Trained Othello-GPT Probe Error Rate: $1.7\%$

  • The substantial accuracy improvement indicates that Othello-GPT constructs internal representations of the board state across its layers.

• Emergent Geometry of Conceptual Representations:

  • Prototype vectors from trained probes represent individual tile concepts in high-dimensional activation space.

  • Principal Component Analysis (PCA) reduces the dimensionality of the $64$ concept vectors to $3$ dimensions for visualization.

  • Neighboring tiles on the physical board are connected in the plot (horizontal connections mapped using an orange gradient palette across vertical positions; vertical connections mapped using a blue gradient palette).

  • Outcome: while a random network displays an unstructured cluster of points, the trained Othello-GPT reveals an emergent visual geometry resembling a "draped cloth on a ball," structurally reflecting the physical coordinates of the $8 \times 8$ Othello board.

Controlling Model Predictions via Interventions

• Causal Intervention Protocol:

  • To test if identified world representations causally drive next-step predictions (rather than acting as passive side-effects), internal layer activations are directly modified during forward-pass computations.

  • Target Modification: intervention vectors alter specific internal tile states in the model's latent space (e.g., flipping tile E6 from black to white) on the fly.

• Intervention Results:

  • Post-intervention model predictions are evaluated against ground-truth legal moves generated by an exact Othello game engine configured to the modified board state.

  • Average Prediction Error: $0.12\,\text{tiles}$

  • The high accuracy demonstrates that intermediate layer activations causally steer downstream token predictions according to the altered world state.

• Synthetic and Unreachable Board State Testing:

  • Evaluated on impossible board configurations unreachable via standard game sequences (e.g., boards with disconnected blocks of discs), similar to Fischer random chess setups.

  • The model maintains high performance under these synthetic interventions, confirming that the internal world model operates independently of specific surface sequence memorization.

Applications for Interpretability: Latent Saliency Maps

• Attribution via Intervention Method:

  • Interventions enable counterfactual analysis: evaluating how changing latent state variables alters predicted token probability distributions.

  • Procedure:

  1. Compute base probability for a candidate move (e.g., move D6) using standard forward pass.

  2. Apply latent interventions to flip individual occupied board tiles (e.g., flipping D4 from white to black) and recalculate predicted probabilities for D6.

  3. Measure the probability differential between factual and counterfactual outputs to assign an attribution score to each tile.

• Latent Saliency Map Properties:

  • Heatmaps highlight board regions that causally justify move legality: specifically pin-pointing the opponent discs being sandwiched and the matching-color disc at the opposing end of the line.

  • Unlike traditional saliency methods that highlight input surface tokens, latent saliency maps visualize causal dependencies directly within the model's internal latent space.

Structural Framework and Unified Bidirectional Mapping

• Parallel Process Mapping:

  • Sequence generation involves two parallel flows: a human-understandable World Model (game engine) and an uninterpretable deep neural network.

• Directional Mapping Mechanics:

  • Probing (Internal Activations $\rightarrow$ World Representation): probes extract readable world states (board representations) from intermediate high-dimensional neural activations.

• Intervention (World Representation $\rightarrow$ Internal Activation): targeted interventions inject modified world representations directly back into intermediate activation vectors.

• Unified Bidirectional Architecture:

  • Combining probing and intervention establishes a bidirectional link between high-dimensional neural representations and human-understandable world states.

  • Higher layers execute downstream predictions based on these latent world representations, bypassing reliance on exact input token sequences.

• Open Challenges & Future Work:

  • Reverse-Engineering Latent Spaces: extracting latent world structures automatically without assuming prior knowledge of state dimensions (e.g., $64$ tiles with $3$ states).

  • Minimally Invasive Steering: controlling complex LLMs with precise, minimal interventions that alter target state variables while preserving collateral internal representations.

Citations and References

• Primary Source Citation:

  • Kenneth Li, "Do Large Language Models learn world models or just surface statistics?", The Gradient, 2023.

• Academic Paper Citation:

  • Kenneth Li, Aspen K. Hopkins, David Bau, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg, "Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Task," arXiv preprint arXiv:2210.13382, 2022 (Presented at ICLR 2023).

• Key Theoretical and Methodological Literature:

  • E. M. Bender and A. Koller, "Climbing towards NLU: On Meaning, Form, and Understanding in the Age of Data," ACL, 2020.

  • W. Merrill, Y. Goldberg, R. Schwartz, and N. A. Smith, "Provable Limitations of Acquiring Meaning from Ungrounded Form: What Will Future Language Models Understand?" arXiv:2104.10809, 2021.

  • E. M. Bender, T. Gebru, A. McMillan-Major, and S. Shmitchell, "On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?" ACM FAccT, 2021.

  • L. Floridi and M. Chiriatti, "GPT-3: Its Nature, Scope, Limits, and Consequences," Minds & Machines, 2020.

  • Y. Belinkov, "Probing Classifiers: Promises, Shortcomings, and Advances," arXiv:2102.12452, 2021.

  • A. Conneau, G. Kruszewski, G. Lample, L. Barrault, and M. Baroni, "What you can cram into a single vector: Probing sentence embeddings for linguistic properties," ACL, 2018.

  • J. Hewitt and C. D. Manning, "A Structural Probe for Finding Syntax in Word Representations," NAACL, 2019.

  • M. Abdou, A. Kulmizev, D. Hershcovich, S. Frank, E. Pavlick, and A. Søgaard, "Can Language Models Encode Perceptual Structure Without Grounding? A Case Study in Color," arXiv:2109.06129, 2021.

  • R. Patel and E. Pavlick, "Mapping Language Models to Grounded Conceptual Spaces," 2022.

  • B. Z. Li, M. Nye, and J. Andreas, "Implicit Representations of Meaning in Neural Language Models," arXiv:2106.00737, 2021.