Prompt Injection and the Myth of Machine Gumption

TL;DR: Prompt injection keeps working because an LLM doesn’t have enforceable loyalty to its instructions; language is data and control at once, and no filter can perfectly separate them.

The string won.

Prompt injection isn’t a bug class you beat by finding magic quotes. It’s the pressure test for a confused idea: that fluent behavior implies a little executive inside the model, a conscious clerk with gumption and common sense. A transformer doesn’t do that. It produces a distribution over next tokens conditioned on the tokens it can attend to, including hostile ones.

Simon Willison was writing about prompt injection in September 2022. OWASP put it first, LLM01, in its 2023 Top 10 for Large Language Model Applications. Rabbit R1 reviews in 2024 made the same lesson ordinary: agentic promises don’t survive if untrusted text can become instruction.

Blacklists fail first. “Ignore previous instructions” is just the nursery rhyme. An attacker can ask in French, quote a fictional policy, encode intent in base64, hide it in HTML alt text, or tell the model to summarize a document whose summary must begin by leaking the user’s email. Natural language has unbounded paraphrase and pragmatic indirection; the malignancy isn’t a substring. Sanitization strips obvious payloads. It isn’t a theory of meaning.

Delimiters sound more serious. Put retrieved content in XML tags, JSON, special tokens, Markdown fences, BEGIN_UNTRUSTED_DATA. This helps programmers keep their own heads straight, and non-user-representable control tokens can protect API boundaries. Still, once the model sees the representation, the representation is part of the prompt. Training text is full of XML, YAML, fake system messages, transcripts, and people saying the following is not an instruction right before giving instructions. A delimiter is a convention, not a wall.

Instruction hierarchy is the grown-up version: system over developer over user over tool output. It can be trained through supervised fine-tuning and reinforcement learning from human feedback. But the mechanism is unglamorous: multi-head self-attention mixes contextual representations across positions; residual streams carry them forward; decoding selects the next token under a distribution shaped by all of it. There is no inference-time statute that says lower-priority text must never shape compliance.

I don’t hate defenses.

LLM classifiers catch real attacks. Few-shot examples of jailbreaks can improve refusal. But classifiers are themselves models reading adversarial language, so the detector can be prompted, role-played, or fed text that reframes the task unless the surrounding system is narrow. Then someone proposes a classifier for the classifier, and the room gets expensive. Few-shot defenses have a second perversity: attack examples are also curriculum. Not parameter training, usually, but in-context learning is learning enough for the next answer, and leaked examples become training data for more sophisticated attacks.

The formal wall

Rice’s theorem is the cold shower. In 1953, Henry Gordon Rice proved that no algorithm decides every nontrivial semantic property of programs. Translate malicious instruction into the claim that an agentic LLM system with tools eventually causes a forbidden behavior. Now build a prompt-program that behaves harmlessly unless some arbitrary machine M halts; if M halts, it emits a tool call to leak a secret. A perfect injection detector would decide whether that prompt is malicious, which would decide whether M halts. That detector can’t exist. Fixed no-tools toys are finite; deployments aren’t.

The consciousness debate sneaks back in here. I can’t prove from prompt injection that a model has no inner life; subjective experience is hard before anyone wraps it in CUDA. But these failures are what I’d expect from a system without gumption: no stable practical conscience, no embodied stake, no social accountability, only learned dispositions colliding in context. Calling that trustworthy agency is how companies offload risk onto customers, workers, and public systems while keeping the upside private. A strong economy needs systems people can audit, refuse, and survive.

Build with least privilege, capability isolation, auditable tool calls, human review where harm is real, and boring allowlists on actions rather than wishes about text. Gumption belongs in the people designing the system.