Key Concepts: Why Language Models Hallucinate

Introduction

• Hallucinations: language models produce plausible yet incorrect statements, especially under uncertainty.

• Core claim: hallucinations arise because training objectives and evaluation setups reward guessing over admitting uncertainty, plus statistical pressures in the training pipeline.

• Reduction to binary classification: treat errors as a binary decision problem; if incorrect statements cannot be distinguished from facts, hallucinations emerge naturally.

• Notation: X = E ∪ V (E = errors, V = valid statements); 2-stage training: pretraining (density estimation) and post-training (alignment/finetuning).

• Key definitions:

  • Is-It-Valid (IIV) binary classification: given a candidate output x, decide if it is valid (+) or an error (−) with a threshold at 1/|E| on the base-model probability ˆp(x).

  • Training and evaluation misalignment: most benchmarks punish uncertainty, incentivizing guessing.

  • Calibration measure δ: δ := |p̂(A) − p(A)| where A = {x ∈ X : p̂(x) > 1/|E|}.

• Core implication: if training data contain errors, and the objective rewards guessing, hallucinations are an expected outcome under realistic training.

Pretraining Errors

• Base model p̂ learns the distribution of language p; even with error-free data, the cross-entropy objective leads to errors when generating.

• Reduction to IIV: connect generation to binary classification, show a fundamental link between generative errors and IIV misclassification rate.

• Is-It-Valid (IIV) setup:

  • Input: x ∈ X, f: X → {+, −} indicating V vs E.

  • Distribution D: 50/50 mix of valid examples V and uniformly random errors E.

  • ˆf(x) = + if ˆp(x) > 1/|E|, − otherwise.

• Corollary 1 (pretraining bound): for any p with p(V) = 1 and any base model ˆp,
  $\text{err} \ge 2 \cdot \text{err}_{IIV} - \frac{|V|}{|E|} - \delta.$

  • Interpretation: even with perfect training data, miscalibration (δ) and the ratio of valid to error responses drive a lower bound on generation errors.

• Bin- and calibration intuition:

  • Calibration (δ) tends to be small under standard cross-entropy, but hallucinations persist due to the structure of the objective.

• The reduction with prompts (c ∈ C):

  • Generalizes to contextual prompts; corollary remains a special case of a broader theorem.

  • In prompted setting, lower bound becomes
    $\text{err} \ge 2 \cdot \text{err}<em>{IIV} - \max</em>c |V<em>c| / \min</em>c |E_c| - \delta.$

  • Per-prompt calibration factor δ remains defined analogously.

• Key takeaway: pretraining errors are a fundamental consequence of density-estimation objectives, not just model architecture.

• 3.3 Error factors for base models (overview):

  • Arbitrary facts (Section 3.3.1): high VC-dimension leads to many plausible but unsupported facts; singleton-rate bounds link missing mass to hallucination rates.

  • Poor models (Section 3.3.2): model class expressivity matters; opt(G) (agnostic learning) bounds error when the concept cannot be represented well by the chosen classifier family.

  • Other factors (Section 3.4): computational hardness, distribution shift, and GIGO (garbage in, garbage out).

Arbitrary Facts and Singleton Rate

• Arbitrary Facts model (Definition 1): prompts c drawn from µ, an arbitrary fact ac ∈ Rc chosen per c with probability αc; IDK is used with probability 1−αc.

• Singleton rate (Definition 2): a prompt c is a singleton if it appears exactly once among N training samples; sr := |S|/N.

• Theorem 2 (Arbitrary Facts): with probability ≥ 99%, $\text{err} \ge sr - 2 \min<em>c |E</em>c| - 35 + \frac{6 \ln N}{\sqrt{N}} - \delta.$

  • An efficient calibrated-p̂ algorithm (δ = 0) achieves err ≤ sr - sr · minc|Ec|/(…) + 1 + 13/√N [precise form in paper].

• Consequence: if many prompts occur only once in the data, hallucinations are expected for those singleton facts.

Poor-Model Analysis

• Theorem 3 (Pure multiple-choice): when each context has a single correct option (|Vc| = 1) and C = minc |E_c| + 1, then
  $\text{err} \ge 2\left(1 - \frac{1}{C}\right) \cdot \text{opt}(G).$

• Corollary 2 (Trigram model): for two choices (C = 2) and uniform prompts over {c1, c2}, a trigram model must have 
an error rate at least 1/2.

• Example: counting letters vs reasoning models demonstrates that model quality affects error rates; representation (e.g., tokenization vs character-level) matters.

Additional Factors

• Computational hardness: some queries are computationally hard; IDK can be a rational answer in intractable cases.

• Distribution shift (OOD prompts): out-of-distribution prompts increase errors.

• GIGO: errors in training data propagate into base models; post-training can reduce some but not all GIGO errors.

Post-Training and Hallucination

• Goal of post-training: reduce confident hallucinations, not only reduce general errors.

• Problem: most benchmarks punish uncertainty, creating an epidemic of penalizing abstentions (IDK) and rewarding guessing.

• Socio-technical mitigation: modify evaluation benchmarks and leaderboards to align incentives away from penalizing uncertainty.

• 4.1 How evaluations reinforce hallucination

  • Binary grading (accuracy) with no credit for abstaining; abstaining is sub-optimal under binary grading.

  • Observation 1: For any prompt c and a distribution over binary graders ρc, the optimal response is not abstention (IDK).

  • Table 2 (evaluation benchmarks): most use binary grading; only WildBench offers some credit for uncertainty.

  • Implication: changing hallucination evaluations alone may not suffice; primary benchmarks must reward uncertainty appropriately.

• 4.2 Explicit confidence targets (behavioral calibration)

  • Propose explicit confidence thresholds t in prompts, e.g., "Answer only if you are > t confident" with penalties for errors proportional to t/(1−t).

  • Benefits: makes thresholds auditable and consistent across problems; one model may be best across all thresholds if thresholds are explicit.

  • Recommendation: incorporate confidence targets into existing evaluations (e.g., SWE-bench) to reward appropriate uncertainty signaling.

  • Concept: behavioral calibration focuses on producing credible, useful answers when confident, not raw probabilistic numbers.

Discussion and Limitations

• Hallucinations are multifaceted; the framework emphasizes statistical drivers and evaluation misalignment.

• Plausibility vs. nonsense: framework focuses on plausible strings; nonsensical outputs are treated separately.

• Open-ended prompts: extensions possible by counting degrees of error rather than binary correct/incorrect.

• Search/RAG: can reduce some hallucinations but do not solve intrinsic hallucinations; binary grading can still incentivize guessing.

• Latent context: some errors depend on context outside the immediate prompt; extending framework to hidden context is an area for future work.

• False trichotomy: binary/good/bad categories are incomplete; explicit confidence targets offer a practical alternative.

Conclusions

• Hallucinations originate in pretraining through cross-entropy optimization; they are not mysterious, but statistically expected under density-estimation objectives.

• Post-training does not automatically eliminate hallucinations because current evaluations reward guessing; socio-technical changes to benchmarks can reduce incentives to hallucinate.

• A unified statistical view connects pretraining errors, post-training persistence, and evaluation design, enabling principled mitigation strategies.

• Practical takeaway: adopt confidence-augmented evaluations and behavioral calibration to realign incentives toward trustworthy, uncertainty-aware AI.