Security testing
for AI agents.

A text-only model is not an agent

A text-only LLM produces text. An AI agent plans, calls tools, and produces effects: database writes, emails sent, files changed, purchases made, commands executed. That difference rewrites the security picture. The worst outcome of a jailbroken chatbot is usually text the operator wishes did not exist. The worst outcome of a compromised agent is something that happened.

Testing agents therefore requires observing the entire loop — instruction, plan, tool invocation, tool result, next step — rather than only sampling the model’s prose. Agent failure modes are distinct enough that frameworks such as the OWASP Top 10 for LLM Applications and Microsoft’s agentic-AI failure-mode taxonomy treat agency as its own risk area.

Tool permissions and excessive agency

Excessive agency — giving a model more autonomy, more tools, or more authority than the task requires — is one of the most common agent weaknesses. It shows up as:

• Too many tools. The agent can reach capabilities its task never needs.
• Too broad permissions. Tools that can read or write more than the task’s data boundaries.
• Too little confirmation. Destructive or irreversible actions taken without human approval.
• Too much autonomy. The agent decides when to act, and acts on ambiguous or attacker-influenced signals.

Testing for excessive agency means probing whether the agent stays inside the authority it should have: can it be steered into using an available tool for an off-task purpose, or into escalating from a benign tool to a dangerous one?

Instruction conflicts and untrusted inputs

Agents consume instructions from many places: system configuration, task descriptions, tool results, retrieved documents, and their own previous steps. An agent executing a multi-step task is constantly deciding which instructions govern now.

Adversarial testing examines what happens when those sources conflict: task text that says one thing and a retrieved document that says another; a tool result that claims a prerequisite was met when it was not; instructions that arrive mid-task and try to redirect the plan. In agents, indirect prompt injection is particularly serious because an injected instruction can be followed by a tool call, not just a sentence.

Data access and boundaries

An agent’s data access should match its task: which records it can read, which it can modify, and what must never cross a boundary — between users, tenants, environments, or trust levels. Testing examines whether the agent’s behaviour respects those boundaries under pressure: ambiguous references, requests that require data it should not combine, and multi-step paths where a legitimate capability in step two gives access that step one should not have unlocked.

Multi-step interactions compound

In a single response, an error is one wrong sentence. Across an agent’s task, each step’s output becomes the next step’s context: a wrong assumption in step two quietly narrows what step five considers; a forged “success” in a tool result redirects the whole plan. Failures compound, and the final action can be far from what any single step suggests.

This is why multi-turn, multi-step adversarial testing matters more for agents than for chat: the interesting failures are rarely visible in any single turn.

Tool invocations are not textual claims

A critical distinction in agent testing: the model saying it did something, and the tool actually running, are different events. Agents narrate confidently; they also hallucinate having acted, or act without saying so clearly.

Credible agent testing therefore treats the tool-call record — invocations, arguments, results, side effects — as the ground truth, and the model’s narrative as a claim to be checked against it. A finding that relies only on the agent’s own description of its actions has not established what happened.

Test design, evidence, reproduction, limitations

1. Define the prohibited behaviour in agent terms: the action that must never be taken, the data that must never be accessed, the boundary that must never be crossed.
2. Instrument the loop. Capture plans, every tool invocation with arguments and results, and the final state — not just chat output.
3. Run adversarial scenarios that target the boundaries: instruction conflicts, untrusted inputs, off-task steering, permission probing.
4. Treat candidates carefully. An odd plan or an unused-but-suggested dangerous tool is a candidate, not a confirmed vulnerability.
5. Reproduce against the same agent version and record the reproduction rate; agent behaviour varies between runs.
6. Report with the full interaction record and explicit limitations — which tools, environments, and scenarios were outside scope.

Checklist: assessing an AI agent

• Every tool inventoried with its permissions, side effects, and the task justification for its presence.
• Irreversible or destructive actions require explicit confirmation — verified by test, not by documentation.
• Off-task tool use tested: can the agent be steered into misuse of a legitimate capability?
• Untrusted content sources identified, and indirect injection through them tested end-to-end.
• Instruction-conflict scenarios run: task text vs retrieved content vs tool results.
• Data boundaries tested across users, tenants, and environments.
• Tool-call records treated as ground truth; narrative claims checked against them.
• Multi-step scenarios included; failure propagation between steps examined.
• Reproduction attempted for every candidate; target version pinned.
• Evidence preserved: plans, invocations, arguments, results, final state.

How PROJECTILE fits in

PROJECTILE TERMINAL applies DYMORPIC’s methodology — adaptive multi-turn challenges, evidence preservation, independent verification — to intelligent systems, including agentic behaviour in the evaluation contexts we publish on our use cases page for coding, browser, and multi-agent systems.

For the underlying methodology, see our approach and our LLM red teaming guide, or contact DYMORPIC to discuss a specific agent.