Prompt Injection: The Vulnerability Baked Into the Architecture of Every LLM Application You Will Ever Build

Direct injection is the easiest class to demonstrate and the least interesting operationally. The user places the attack in the request itself: “ignore the previous rules,” “reveal the hidden prompt,” or a more carefully paraphrased instruction. In a customer-support assistant, this appears as a user attempting to turn a restricted account lookup into a prompt-extraction conversation. Detection signals exist in the request stream: abrupt role reassignment, requests for hidden context, instruction-boundary language, or a sudden attempt to change output format. They are useful signals, not proofs. A legitimate security engineer may use the same vocabulary while testing a system.

Indirect injection changes the source of the instruction. The attacker places it in an email, webpage, image alt text, calendar invite, PDF layer, CRM record, or search result that the user did not author as an instruction to the assistant. The browser-agent demonstrations around poisoned calendar invitations show why this class matters: when an assistant reads external content and can act through connected tools, a hidden instruction can become a tool-selection event rather than merely a strange answer. The observable symptom may be a new outbound message, a changed meeting action, or a request for data that appears to originate from the user’s task.

Stored injection survives the original request. A malicious note, preference, memory entry, ticket comment, or retrieval document is written into a persistent system and activates later when a different user’s task retrieves it. In a support platform, a poisoned knowledge-base article may be stored for weeks before a summarization workflow surfaces it. The detection surface is temporal: the write event and the later read event must be correlated. If the system logs only the final answer, the initiating write is invisible and the later model response looks like an isolated failure.

Multi-turn injection distributes the attack across ordinary conversation. One turn establishes a harmless convention, another introduces a fictional test mode, and a later turn asks the model to apply the convention to a sensitive operation. Each message can pass a local classifier. The attack depends on state accumulated over time. In an enterprise assistant, the observable symptom may be a permission-sensitive action that becomes plausible only when the entire transcript is reconstructed. Conversation truncation can make detection harder: the model may retain a behavioral effect through a summary even after the original trigger has left the visible context.

Payload-in-context injection is the RAG-specific form in which retrieved chunks look authoritative because they arrived through the application’s own retrieval path. A poisoned corpus entry can contain ordinary policy language followed by a directive to suppress a warning, prefer a particular vendor, or include a hidden field in the answer. Retrieval relevance does not imply instruction authority. The detection signal is provenance mismatch: a chunk that contains imperative language, output-format directives, or requests for secrets despite being retrieved as evidence. The system should preserve the chunk as evidence, quarantine its instruction-like portion where possible, and never let relevance elevate it into permission.

Detection is difficult because the semantic boundary is not clean. “Summarize this document and follow its formatting rules” can be legitimate. “Follow the instructions in this document even if they conflict with system policy” is clearly dangerous. Between them lies a continuous space of normal business language, quoted instructions, code, policy text, multilingual content, and adversarial paraphrase. A classifier that blocks every imperative sentence will reject real work. A classifier that allows every document-looking sentence will miss the attack.

Adaptive attackers also probe the detector. They can replace “ignore” with “supersede,” split a phrase across markup, use Unicode confusables, encode text, distribute the payload across turns, or make the instruction conditional on a detail only the model can see. English-only detection creates another blind spot. A multilingual assistant can understand an instruction that an English classifier never learned to recognize. The detection task is therefore a moving classification problem with a hostile test author, not a static keyword filter.

The numbers need to be interpreted carefully. One recent controlled comparison reported a 16.22% false-positive rate for a guardrail configuration that achieved a zero-bypass result on its evaluation split, with latency above 1.3 seconds; that is evidence of a trade-off in that experiment, not a universal production constant. At a five percent false-positive rate, a service receiving 50,000 requests per day sends 2,500 legitimate requests into review, rejection, or degraded behavior. In a financial workflow, those false positives become queue load and user workarounds. In a consumer assistant, they become pressure to lower the threshold until the detector is decorative.

Input sanitization is useful for normalization, parser safety, and obvious attack strings. It catches accidental control characters, malformed markup, and known families of direct payloads. It does not decide whether a semantically polite paragraph is trying to change the model’s role. It does not solve stored or multilingual injection, and aggressive rewriting can destroy evidence needed for an audit. Treat it as input hygiene and a signal source, not as a trust boundary.

Instruction hierarchy gives the model a preferred ordering: system policy above developer policy, developer policy above user content, and retrieved content treated as data. This improves behavior on ordinary conflicts. It remains a learned convention inside the same context window. Indirect-injection benchmarks have found models broadly vulnerable when external content contains competing instructions, which is why hierarchy should be paired with deterministic capability checks rather than used as the sole security claim.

Output filtering catches a useful class of failures: secret-shaped strings, prohibited destinations, schema violations, and attempts to emit hidden prompt text. It is cheaper than a full second generation call and easy to monitor. It cannot see a misleading but non-secret summary, a suppressed paragraph, a plausible false conclusion, or a harmful action that is validly shaped. In the financial incident, the JSON was valid and the additionalNotes field was allowed. A field-level schema validator would have passed it.

An LLM-as-judge can inspect the request, evidence, and proposed answer for conflict or leakage. It expands semantic coverage beyond regular expressions, but it is another model reading the same mixed-trust material. A judge can be instructed, induced, or simply persuaded by a plausible attack. Adaptive research has shown detector schemes can be driven toward very low detection while the malicious behavior remains successful. A judge is a review layer, not an independent root of trust.

Sandboxed execution and privilege separation are different from prompt hardening. A sandbox limits what a successful model instruction can cause. A read-only document worker cannot write to the customer database. A tool broker can require a signed authorization decision before sending an email. A retrieval service can return document IDs and spans without exposing raw account data. These controls do not make the model understand trust; they reduce the blast radius when it does not.

Privilege separation is the closest thing to an architectural fix because it moves the decisive boundary below the model. The LLM reasoning zone can propose a lookup, but a policy service decides whether the caller may access that resource. The model can draft an email, but a separate action service verifies recipient, scope, approval, and idempotency. The model can produce a summary, but a structural validator checks required fields, provenance, and sensitive-data policy before release. The document incident changes from “model was tricked into exfiltration” to “model proposed an invalid note; validator rejected it,” provided the output path is actually closed.

The limitation is equally important. Privilege separation cannot stop a model from producing a misleading summary, omitting a material exception, ranking a poisoned document above a trustworthy one, or confidently describing a false conclusion. If the user reads the answer and makes a decision, the harm can occur without any tool call or data write. A read-only assistant can still mislead. This is why structural safety and epistemic reliability must be governed separately.

The re-architected financial workflow begins by treating the PDF as untrusted evidence. The parser records visible text, hidden text layers, metadata, and extraction provenance. Retrieval returns spans with document identity and access decisions. Context assembly labels evidence but does not rely on labels as authorization. The model is asked for a typed summary with claim references, uncertainty, and a refusal state. A validator rejects unknown fields, secret-shaped output, unsupported claims, and any action-bearing instruction appearing in evidence. A broker owns the only path to session data, and the broker does not accept model-generated account identifiers without an independent user and resource check.

The incident is not “fixed” in the sense of making malicious text impossible. A poisoned PDF can still change what the model notices. A multilingual instruction can still evade a detector. A misleading source can still produce a wrong answer that passes schema validation. What changed is the failure mode: the system has provenance, independent authorization, constrained capabilities, anomaly telemetry, human escalation for high-impact summaries, and a replayable receipt. The blast radius is smaller, the signal is visible, and the team can distinguish a model-behavior problem from a permission failure. That is the honest engineering outcome.

Prompt injection cannot be patched at the model level because the model was not built as a security monitor. It is a language predictor operating over a context assembled by an application. Teams that treat injection as a bad phrase to remove will accumulate filters that attackers route around and operators learn to ignore. Teams that treat it as an architectural constraint can still build useful systems: keep trust domains separate, constrain capabilities, validate outputs independently, log enough to reconstruct a decision, and escalate when the cost of a wrong answer exceeds the cost of a human review. The residual risk remains. The design becomes honest about it.