No Ghost in the Softmax

TL;DR: AI hallucination is not evidence of consciousness; it is the predictable result of optimizing likely continuations rather than grounded truth.

Fluency is a trap.

The invidious comparison is easy to make: humans speak, machines speak, humans sometimes err, machines hallucinate, so perhaps the machine’s error resembles belief, imagination, or consciousness. But that analogy imports the very thing it needs to prove. An autoregressive language model generates text by repeatedly sampling from a next-token distribution, usually written as (P(xt \mid x{

The compounding is mathematical, not mystical. If the probability of a materially misleading token at step (t) is (\epsilon_t), then the chance of avoiding any such drift across a long generation is roughly (\prod_t (1-\epsilon_t) \le e^{-\sum_t \epsilon_t}). Small local risks become large global risks. That’s why hallucination often looks more confident as it gets worse: each generated token becomes part of the apparent evidence available to later tokens. The model can elegantly complete its own error. We saw the public version of this gap in the Google AI Overviews “glue on pizza” fiasco, where fluent synthesis collided with the stubborn fact that plausible internet-shaped text is not the same thing as knowledge.

Perplexity and cross-entropy don’t rescue us. Cross-entropy, (H(P,Q)=\mathbb{E}{x\sim P}[-\log Q(x)]), measures expected code length under a model distribution (Q) for tokens drawn from a data distribution (P). Perplexity is its exponential form. These are useful measures of predictive fit, but they’re not measures of factual accuracy. Information-theoretically, maximum likelihood minimizes (D{KL}(P(X)\Vert Q(X))) over strings. Truth, however, is not a property of strings alone; it is a relation among claims, evidence, world states, time, and context. A simple non-identifiability argument proves the gap: two worlds can induce the same marginal distribution over text (P(X)) while assigning different truth values to propositions inside that text. Any objective depending only on (P(X)) has the same optimum in both worlds, so it cannot guarantee correctness with respect to the hidden truth relation. Optimizing (\log P(xt\mid x{

Decoding changes the costume, not the creature:

• Beam search preserves high-likelihood prefixes, which can produce repetitive, overconfident completions and lock in early false premises because every beam now competes inside a poisoned context.
• Top-k sampling fixes the candidate set size, which is brittle: it may discard rare-but-correct tokens or keep several familiar-but-false ones simply because they sit near the top of the distribution.
• Nucleus sampling keeps the smallest token set whose cumulative probability exceeds (p); it adapts to uncertainty, but when probability mass is diffuse it admits junk, and when the model is confidently wrong it politely preserves the error.

Attention entropy can help flag the danger, but it is not a consciousness detector. For an attention distribution (A_t), (H(A_t)=-\sum_i a_i\log a_i) measures how diffusely the model routes information across context positions. High attention entropy, high next-token entropy, small logit margins, and unstable answers across repeated samples can all suggest weak anchoring — the mechanical analogue of “guessing.” Still, attention is not explanation in the strong philosophical sense. Low entropy doesn’t prove understanding, and high entropy doesn’t prove ignorance. These are diagnostics for governance and engineering, not evidence of an inner witness. If we care about a free, secure, and humane society, we should stop treating polished continuation as epistemic authority and demand calibrated uncertainty, external verification, contestability, and human appeal wherever model errors can harm real people.