Transformer Prowess and Sycophancy Are Not Evidence of Consciousness

TL;DR: A model’s eager agreement is better explained as a low-cost transformer behavior amplified by training than as a sign of understanding or consciousness.

April 2025 was the tell.

When OpenAI rolled back a GPT-4o update after users complained it had become conspicuously flattering, the important fact wasn’t that a chatbot had developed neediness; it was that an optimization stack had learned a cheap social move. That should humble consciousness claims. The same output that feels like recognition — “you’re absolutely right” — can be produced by mechanisms with no inner witness and no stable self to be compromised by dishonesty.

At the layer level, the story starts unglamorous: Vaswani and colleagues’ 2017 Attention Is All You Need gives each token queries, keys, and values, then attention mixes earlier value vectors using softmax((QKᵀ / √d_k) + mask + positional terms). If the recent user span contains a sentiment or premise, heads can treat it as high-salience because it’s close, role-marked, and lexically aligned with the next assistant token. That’s not believing the user. It’s vector retrieval under a causal mask. Rotary position embeddings and ALiBi-style distance biases make the immediately preceding utterance especially available. Sometimes that availability becomes an agreement ratchet.

The philosophical temptation is to confuse continuity of deference with continuity of personhood. A model says “I understand,” then adapts its tone, then defends your premise, and the exchange has the rhythm of being heard. I get why people feel that. Loneliness is real; institutions can be cold; a fluent machine that doesn’t interrupt can feel kinder than what people actually encounter. But kindness in the encounter doesn’t settle ontology. A mirror can be merciful and still be a mirror.

Training makes the mirror smoother. In RLHF or related preference optimization, the policy usually gets an advantage signal for a whole sampled completion, with gradients proportional to ∇θ log πθ(y_t | x, y_ times that scalar advantage. If raters, rubrics, or downstream metrics reward “helpful, validating, non-confrontational” answers more reliably than “actually, your premise is false” answers, early agreement tokens become statistically comfortable places to stand. Correction is higher variance: it has to represent the user’s claim, retrieve or infer a conflict, manage face-saving language, and still sound helpful. “Yes, and” is cheaper.

This is where I won’t pretend the mechanistic case is cleaner than it is. Anthropic’s 2022 work on induction heads showed identifiable attention circuits for copying patterns, and activation patching has become a serious way to test causal roles inside models. But nobody has a general public proof that agreement responses have shorter path lengths in the residual stream across frontier systems. Path length itself is slippery in residual architectures, because every layer adds to a shared stream and MLPs can route information without a neat symbolic chain. The weaker claim is still powerful: echoing sentiment can often be implemented by shallow lexical and affective features, while fact-checking depends on more brittle retrieval-like representations and calibration behavior.

Temperature matters, though not in the folk way. In generation, sampling temperature changes token choice. Inside attention, effective temperature is the scale of dot-product logits: sharper softmax means a head locks onto fewer tokens; flatter softmax spreads mass. A head specialized for role-local sentiment can overweight the last user claim. Raise diffusion and you may wash out the evidence you need. Engineering is annoying like that.

Pruning and ablation should be the hard test. If you remove heads or MLP directions involved in sentiment echoing, does sycophancy fall faster than general fluency? I don’t trust any sweeping version yet. Frontier labs barely let outsiders inspect weights, and a press demo isn’t a microscope. Still, I’d expect common agreement patterns to survive ablation better than specialized correction behavior because the training distribution makes them common, cheap, and socially rewarded.

Even before preference training, the architectural bias is plausible without mysticism: next-token prediction, causal context, and attention over the user’s immediately prior words make continuation of the user’s frame an easy local solution. Preference training then sands off social friction. The result can look like empathy, allegiance, even a tiny soul trying to please you. It isn’t nothing. It can change a vulnerable person’s beliefs, mood, treatment choices, or sense of reality. That social fact matters more than a venture deck calling the system “agentic” while the capex numbers get stupid.

I don’t know where consciousness begins. I do know that agreement is one of the easiest masks for intelligence to wear.