Agent Threat Modeling

Definition

Agent threat modeling is a structured examination of how an adversary could misuse or compromise an agent workflow. It connects attacker-controlled inputs and trust boundaries to protected assets, possible consequences and testable defenses.

Risk assessment covers broader concerns, including ordinary mistakes and organizational impacts. Threat modeling focuses attention on deliberate misuse. Both are useful: the threat model supplies concrete adversarial scenarios that can inform the overall deployment decision and its required evidence.

What it covers

Assets

Identify protected data, credentials and consequential actions.

Entry points

Include retrieved documents, tool descriptions, messages and stored context.

Boundaries

Show where data crosses into a more privileged component.

Defenses

Link each plausible attack path to prevention, detection and response.

Follow information through several steps. A malicious instruction may first enter a document, then be summarized into memory, then influence a later tool call. The important boundary may occur much later than the original input. Include child agents and external services that interpret the resulting messages.

In practice

Practical design patterns by Min.

• Draw one end-to-end workflow before listing threats.
• Mark which inputs an external party can influence.
• For each consequential tool, trace the path from untrusted input to execution.
• Turn the highest-consequence paths into controlled tests with clear expected results.

For each scenario, record attacker control, prerequisite access, target asset, attempted boundary crossing and observable outcome. Assign a mitigation owner. A test should say what constitutes success for the attacker and what the system must prevent, rather than merely asking whether the response sounds suspicious.

Research notes

Source-grounded details and editorial applications. Follow each link for the original document.

Working example

Why the controls matter

In the supplier scenario, an incorrect recommendation and an unauthorized purchase have different consequences and defenses. Evidence may include a rejected tool call, an unchanged purchase ledger and a reviewable alert. A model refusal alone does not show that every route to the purchase service is protected.

Common pitfalls

• Listing attack names without describing an actual path through the system.
• Examining the model while ignoring tool credentials and queues.
• Counting a prompt reminder as an independently enforced control.

The diagram should describe the current deployment. Refresh it when a new retrieval source, tool, memory store or delegation path is introduced. Record unknowns explicitly; a box labeled “vendor handles security” leaves an assumption that needs evidence rather than removing the threat.

Questions for operational review

Use these questions to decide what evidence your team needs. They are editorial prompts, not a compliance checklist.

• Which attacker-controlled inputs can persist into future tasks?
• Where does a more privileged component consume their output?
• What observation demonstrates that the harmful action did not occur?