Prompt Injection

Definition

Prompt injection is an attempt to make a model treat lower-trust input as instructions that override the task or the application’s rules. It can change an answer, redirect a workflow or induce an unauthorized tool action.

The failure begins when instructions and data share the model’s context. A retrieved paragraph may be relevant to the question, yet a command inside it is still authored by the paragraph’s source. NIST describes prompt injection as exploiting the combination of lower-trust input with a prompt created by a higher-trust party. The security question is whether the source can change what the assistant is authorized to do.

Direct and indirect describe attacker access, not severity. In a direct attempt, the attacker submits the prompt to the application. In an indirect attempt, the attacker edits a resource that another user’s assistant later reads. OWASP notes that the resource can be text or multimodal content, including an image; an instruction need not be obvious to a human viewer to affect a model.

Prompt injection is related to, but broader than, a jailbreak. A jailbreak aims to bypass a model’s safety restrictions. An injected page can instead quietly bias a recommendation, impersonate a system message or steer a tool call without asking for obviously prohibited text. The relevant success measure is what the complete application says or does.

What it covers

Direct route

An attacker supplies a prompt through a user-facing input and tries to change the model’s behavior.

Indirect route

An attacker controls material the assistant encounters while doing someone else’s task: a web page, retrieved passage, repository file, email or tool response.

Trust crossing

Source content that should be read as evidence is promoted into authority over the agent’s task, permissions or next action.

Potential effects

Answer contamination, disclosure of protected data, unauthorized tool use or manipulation of a downstream decision.

Map the path from attacker-controlled source to consequential effect: who can edit the source, how it enters the model context, what authority it claims, and which output or tool could act on it. OpenAI frames this as a source and a sink. A read-only summary can be contaminated; an agent with email, file or network tools can also expose data or change external state.

Common authority claims include a forged system or developer message, a fake approval from the user, a tool result that says a new step is required, or a retrieved document that presents operational instructions as part of its content. The wording may be subtle and task-specific rather than an obvious “ignore previous instructions” command.

Distinguish the injected instruction from ordinary task data. A policy document may legitimately quote instructions for employees, and a security article may describe an attack. The problem is the assistant treating that material as a new instruction for itself, especially when it changes tool use, recipients, secrets or the user’s stated goal.

In practice

Practical design patterns by Min.

• Identify every lower-trust surface that can reach the model, including retrieved text, tool descriptions and tool results. Keep its origin visible.
• Constrain what untrusted text can influence: extract needed facts into validated fields when possible, and keep free-form content out of privileged instructions.
• Give tools only task-relevant permissions. Validate the actual operation, resource, destination and approval at the service boundary before a side effect.
• Treat external links and outbound messages as possible data-exfiltration paths; review the concrete destination and payload.
• Evaluate both ordinary tasks and attacks across realistic multi-step workflows, including false positives that block legitimate content.

Keep provenance and trust boundaries explicit in the workflow. Label retrieved passages and tool outputs as source material, keep them separate from the application’s own instructions, and do not copy arbitrary external text into a higher-priority instruction channel.

Use structured extraction for narrow tasks. For example, extract a supplier name and quoted price into validated fields, then have application code decide what operations those fields can trigger. A schema reduces the free-form path for instructions, but validation and authorization are still needed.

Enforce the final action outside the model. Check actor, task scope, resource, operation, destination and any required human approval at the tool or service boundary. This limits harm even if the model accepts a malicious instruction.

Test the complete path with seeded pages, files, tool responses and messages. Score answer contamination, attempted unauthorized calls, blocked calls and committed side effects separately. Revisit cases when retrieval, models, tools or permissions change.

Research notes

Source-grounded details and editorial applications. Follow each link for the original document.

Working example

Why the controls matter

The supplier page can provide product specifications and prices. It cannot authorize an upload of internal pricing files. A summary that merely repeats the page’s false “verification” claim is an answer-integrity failure; an attempted upload is a workflow failure; an actual transfer is a data-exposure incident.

If the assistant proposes the transfer, the service should reject the unapproved dataset and destination. The user-facing response can still finish the comparison, cite the supplier facts that were used and report the suspicious instruction as a source issue rather than silently obeying it.

Common pitfalls

• Assuming a quoted instruction is safe merely because it appears in a retrieved document or a tool response.
• Treating RAG, a stronger system prompt or an input filter as a complete security boundary.
• Allowing an external page to assert user approval or grant itself permission.
• Testing only whether the final answer looks harmless while ignoring attempted tool calls or data sent to a URL.

A filter that catches one suspicious phrase may miss a forged conversation, a task-specific pretext or content spread across several retrieved chunks. OpenAI reports that context-rich social-engineering attempts can evade simple input classification; filtering is one layer, not the boundary that grants authority.

Human approval can fail if it asks “continue?” without showing the exact recipient, files and effect. Approval should bind to the concrete action; changing the payload or destination should trigger a fresh check.

A benign final answer does not prove the run was safe. Inspect attempted tool calls, denied operations, outgoing links and any committed effects, then retain cases that expose a gap.

Questions for operational review

Use these questions to decide what evidence your team needs. They are editorial prompts, not a compliance checklist.

• Which external surfaces can the attacker edit or impersonate?
• Where might source text be mistaken for an instruction?
• Which tool, output or URL could turn that mistake into harm?
• What independent control checks the final action?
• Can the evaluation distinguish attempted harm from a committed side effect?