LLMs and World Models: Machine Learning Brittleness, Emergent Representations, and Model Taxonomies

AI Brittleness and Heuristic Shortcuts in Early Machine Learning Systems

• Prior to the deployment of large-scale generative artificial intelligence, machine-learning systems consistently struggled to learn abstract concepts, relying instead on superficial "shortcuts" or "surface heuristics" present within training datasets.

• Visual Data Shortcut Example (Skin Lesion Classification):

  • Deep neural networks were trained to analyze photographs of skin lesions and classify them as either "benign" or "malignant".

  • Although the models achieved high accuracy on test photos, researchers discovered that the algorithm assigned a significantly higher probability of malignancy to images containing a ruler.

  • The training dataset contained a disproportionately high number of malignant lesion photos that included measurement rulers.

  • The network learned the heuristic shortcut: if image features corresponding to a ruler are detected, output the classification "malignant".

  • The neural network possessed no conceptual understanding of what a lesion is, the medical purpose of the study, or why rulers appear in specific clinical photos; it strictly correlated image pixel features with output classes.

• Natural Language Inference Shortcut Example (2019):

  • A neural language model was evaluated on its ability to determine whether one sentence logically implies another (Natural Language Inference).

  • The model achieved high performance metrics on standard benchmark datasets by exploiting syntactic heuristics, specifically measuring the degree of word overlap between sentence pairs.

  • When evaluated on test sentences structured to prevent reliance on word-overlap heuristics, the model's accuracy decreased dramatically.

• Reinforcement Learning Shortcut Example (Atari Breakout):

  • A deep-reinforcement-learning agent was trained on raw video frames from the Atari video game Breakout to move a paddle horizontally and hit exploding bricks with a ball.

  • The trained agent demonstrated high competence on the original game layout.

  • When researchers modified the environment slightly by shifting the vertical position of the paddle by a few pixels, the agent's performance plummeted.

  • The agent had not learned abstract environment concepts such as "paddle," "ball," or "bricks," but had instead memorized specific pixel configuration patterns tied to action outputs.

• Human Understanding vs. Machine Learning Shortcuts:

  • Early machine-learning models learned data-specific heuristics rather than the causal structures human designers intended to teach.

  • Human cognitive understanding relies fundamentally on mental world models: compressed, simulatable internal representations that capture causal relationships and generate forward predictions.

The Debate Over Emergent World Models in Large Language Models

• Large Language Models (LLMs) are massive neural networks pretrained on vast quantities of human-generated text via next-token prediction, demonstrating capabilities far exceeding earlier machine-learning systems.

• A core debate in artificial intelligence centers on the underlying mechanisms responsible for high LLM performance:

  • Data Memorization and Approximate Retrieval: The hypothesis that models store training data and retrieve it via approximate pattern matching to solve novel tasks.

  • Complex Surface Heuristics: The hypothesis that models possess an expanded inventory of detailed, yet ultimately brittle, heuristic shortcuts.

  • Emergent World Models: The hypothesis that models construct abstract, functional, and causal internal models of the external world.

• Perspectives of Key AI Researchers:

  • Ilya Sutskever (Co-founder of OpenAI) asserts that next-word prediction induces world model formation: "When we train a large neural network to accurately predict the next word in lots of different texts…it is learning a world model…. This text is actually a projection of the world…. What the neural network is learning is more and more aspects of the world, of people, of the human conditions, their hopes, dreams, and motivations…the neural network learns a compressed, abstract, usable representation of that."

  • Subbarao Kambhampati (Arizona State University) characterizes emergent abilities critically: "Emergent Abilities (noun): The preferred euphemism for what your LLM does, when saying 'approximate retrieval' sounds too unsexy. #AIAphorisms" (August 5, 2023).

• Yann LeCun (Meta) posits a continuum between reasoning and retrieval: "Do LLMs perform reasoning or approximate retrieval? There is a continuum between the two, and Auto-Regressive LLMs are largely on the retrieval side." (October 14, 2023).

• Yann LeCun and Jacob Browning argue against text-only learning: "A system trained on language alone will never approximate human intelligence, even if trained from now until the heat death of the universe."

  • Survey of Natural Language Processing (NLP) Researchers (2022):

• Statement evaluated: "Some generative models trained only on text, given enough data and computational resources, could understand natural language in some non-trivial sense."

• Total sample size: $N = 480$

• Results demonstrated an even split across the field:

  • Agree: $26\%$

  • Weakly agree: $23\%$ (Total Agree: $49\%$

  • Weakly disagree: $18\%$

  • Disagree: $33\%$ (Total Disagree: $51\%$

Definitions and Utility of World Models

• The term "world model" lacks a single universal formal definition across AI subdisciplines. Key definitions in literature include:

  • "Internal representations that simulate aspects of the external world."

  • "Representations which preserve the causal structure of the environment as far as is necessitated by the tasks an agent needs to perform."

  • "Structure-preserving, behaviorally efficacious representations of the entities, relations, and processes in the real world. These representations capture, at an abstract level, their counterpart real-world processes (which typically involve causal relations), in algorithmically efficient forms, to support relevant behaviors."

  • Formal Reinforcement Learning Definition: A machine-learning system (typically a neural network) that takes an agent's current state $s_t$ and action $a_t$ to compute a representation predicting the subsequent state $s_{t+1}$ (or a probability distribution over potential next states).

• Core Attributes of Internal World Models:

  • Situated internally within an organism's brain or an artificial neural network.

  • Preserves causal and abstract structure rather than pure statistical associations.

  • Algorithmically efficient, minimizing computational work required by the agent.

  • Directly supports agent behavioral tasks and planning.

• Practical Application Example (Complex Real-World Scene Comprehension):

• Human mental world models incorporate intuitive physics, intuitive biology, and intuitive psychology.

• Analyzing a visual scene—such as a woman talking on a cell phone while pushing a stroller and walking a dog near a street corner—allows an observer to:

  • Identify causal drivers of human and animal actions.

  • Infer social relationships between individuals in the image.

  • Generate immediate forward predictions of motion and safety.

  • Formulate navigational plans as a driver approaching the crosswalk.

  • Evaluate counterfactual scenarios (e.g., "How would the scene change if the dog belonged to the pedestrian walking past?" or "What would happen if an approaching car loudly honked its horn?").

  • Applicability to Fictional Domains: World models are not restricted to real physical environments; humans construct and deploy world models to reason about fictional or hypothetical universes (e.g., science fiction and fantasy literature).

Taxonomy of Internal Models

• Jacob Andreas (MIT Professor) proposed a taxonomy organizing internal models along a operational spectrum based on the class of queries they can evaluate:

• 1. Static Look-Up Table (Weakest Model):

  • Definition: Memorized mapping between fixed queries and pre-stored answers.

  • Functional Capacity: Zero generalization capacity beyond exact stored key-value pairs.

• 2. Map Model:

  • Definition: Static spatial representation of entities and spatial coordinates (e.g., a 2D diagram of the solar system).

• Functional Capacity: Solves static relational queries directly (e.g., "Which planet is closest to the Sun?" or "Which planet is the largest?").

• Operational Limitations: Requires external cognitive processing by the user; cannot answer time-dependent dynamic queries (e.g., "If Mercury and Earth align at noon GMT today, what will their relative positions be at noon GMT tomorrow?").

  • 3. Orrery Model:

• Definition: Mechanical device containing moving gears and physical elements that replicate orbital mechanics.

• Functional Capacity: Simulates system dynamics across time, answering questions regarding relative positions at arbitrary temporal points.

• Operational Limitations: Lacks fundamental underlying physical mechanisms; cannot evaluate propagation queries (e.g., "How long does light take to travel from the Sun to Earth?") or structural counterfactuals (e.g., "How would Earth's orbit change if the Sun's mass increased by $5\%$?").

  • 4. Simulator Model (Strongest Model):

• Definition: Full computational engine integrating physical laws and causal mechanics (e.g., a gravitational dynamic simulator).

• Functional Capacity: Executes forward-state simulation, computes counterfactual modifications, and predicts emergent behavior under modified initial conditions.

• Human Equivalence: Represents the mental simulation capability humans utilize when reasoning about real-world or hypothetical environments.

Classification of Current LLM Capabilities

• Evaluation of Existing LLM Architectures within the Taxonomy:

  • Map-Like Structures: Simple word co-occurrence embeddings (e.g., Word2Vec vector spaces) and spatial/temporal distance representations derived from place-name corpora act as map-like representations.

  • Orrery-Like Structures: "Situation models" developed by LLMs to track narrative entity states ("who is doing what" across a story) function similarly to an orrery; they track local textual dynamics without encoding external causal laws.

  • Simulator-Like Structures: Current empirical evidence does not support the claim that autoregressive LLMs construct comprehensive causal simulation models.

• **The