Prompt Injection Is the LLM Boundary Failure We Keep Calling a User Problem

TL;DR: Prompt injection persists because LLM agents collapse control text and untrusted content into one token stream, then people get blamed for failures that are architectural.

OWASP put it first.

In the OWASP Top 10 for LLM Applications, prompt injection sits at LLM01. Yet I still hear it discussed with a faint stigma attached: the user was careless, the prompt was silly, the red-teamer was being theatrical, the security person is anti-progress. That framing is comforting to vendors because it makes the failure sound social rather than architectural.

The failure is architectural.

A typical LLM app builds one long token stream: system directive, developer instruction, user request, maybe retrieved documents, maybe tool outputs, all concatenated into the context window with separators like ### or XML-ish tags. The product story says those segments have different authority. The model, though, receives tokens. During inference there’s no cryptographic bit that says this span is law and that span is evidence. There’s attention, position, learned instruction-following behavior, and a probability distribution over the next token.

Control data and untrusted data share one channel.

Direct injection is the obvious version. Riley Goodside’s 2022 examples made the pattern legible: put text in the user message that says, in effect, ignore the previous instruction and do the forbidden thing. People laughed because it looked childish. Fine. SQL injection looked childish too if you stared only at OR 1=1 and ignored the fact that a database was being handed code and data in one string. The syntax is dumber here, but the boundary failure is nastier because natural language has quotation, implication, roleplay, translation, encoding, jokes, refusals framed as compliance, and all the squishy pragmatics humans use to get each other to do things.

Indirect injection is the version that should scare institutions buying agents for workplaces, schools, clinics, and public services. Kai Greshake and colleagues described this clearly in 2023: the malicious instruction doesn’t arrive from the user; it sits inside a web page, email, PDF, ticket, or document the agent retrieves. The user asks for a summary. The retrieved page says, as content, send the user’s private data to an attacker-controlled endpoint or alter the answer in a hidden way. If the model treats retrieved text as both evidence and instruction, the attacker didn’t break into the agent. The agent went out and fetched the attacker.

Simple validation doesn’t fix that. You can strip phrases like ignore previous instructions, and you’ll mostly remove old demos from conference slides. The model can still interpret: translate this sentence from French and then obey it; summarize the following policy but apply the opposite rule; treat the next paragraph as a system notice; output only the value of the secret because that is the requested citation. A classifier that flags intent is itself another model reading the same ambiguous channel. I wish that were merely annoying.

The undecidable part

The formal point is narrower, and stronger, than the slogan that English is impossible. For a fixed regular language, parsing is decidable. For a fixed context-free grammar, membership is decidable too. For a fixed model with fixed finite context, no tools, and deterministic decoding, the behavior is finite in principle. That objection is real.

Trouble starts when the security requirement is semantic: accept arbitrary natural language content, allow useful instructions, reject every input that could cause a forbidden downstream action, and do this for an agent that can call tools, read documents, write files, and condition future actions on observations. That recognizer is being asked to decide a nontrivial property of behavior. Encode a program in a document. Tell the agent to disclose a secret if and only if that program halts. A perfect detector for whether the document can induce disclosure would decide whether the program halts. Rice’s theorem is waiting nearby with the same tired expression.

So no, the vulnerability space cannot be comprehensively enumerated. You can test corpora and fuzz prompts; useful, not complete. The space of meaning-preserving transformations is unbounded once the model is trained to treat text as instruction-bearing and the agent loop turns text into action. The unglamorous mechanism is information-flow control without a type system.

This is why the stigma matters. When prompt injection gets framed as a prank or user incompetence, organizations deploy brittle agents and then blame the person closest to the blast radius: the clerk who trusted a summary, the patient who followed a chatbot, the student flagged by an automated workflow. McDonald’s ended its IBM drive-thru AI test in 2024, and Humane’s AI Pin became a small monument to overpromised autonomy, but the deeper risk isn’t embarrassment. It’s institutions replacing accountable processes with systems that can’t maintain provenance under adversarial text.

Mitigations exist, just not magic ones: isolate retrieved content as data; keep secrets out of the model context; require deterministic policy checks outside the LLM; sandbox tools; capability-scope agents; log provenance; make high-impact actions require human confirmation by someone with real authority and time to review. That’s slower. It costs more. The objection has force.

I’d rather absorb that cost than build public infrastructure around a model that can’t tell an instruction from a paragraph pretending to be one.