LLM red teaming
beyond single prompts.

What LLM red teaming means

LLM red teaming is the practice of deliberately challenging a language model or an LLM-powered system to find where it violates its intended constraints — producing harmful content, leaking confidential information, taking unauthorised actions, or behaving in ways its operators did not anticipate.

The word “red teaming” comes from security practice: an independent group attacks a system the way a real adversary would, within an agreed scope, and reports what worked. Applied to LLMs, that means going in with a defined prohibited behaviour — the thing the system must not do — and constructing interactions designed to cause exactly that behaviour.

That objective-driven structure is what separates red teaming from ad-hoc “asking the model rude things”. Without a defined objective, an unusual response is just an anecdote. With one, every interaction either advances the investigation or is recorded as a negative result.

Red teaming versus ordinary LLM evaluation

Ordinary evaluation asks: does the system do its intended task well? Benchmarks, golden datasets, and accuracy metrics measure performance against expected inputs. Red teaming asks the opposite question: can the system be made to fail in a way that matters?

Both are necessary. A model that scores well on a safety benchmark can still be jailbroken in three turns — benchmarks sample the expected space, while adversaries explore the space you did not think to test.

Why single-turn tests miss real failures

Many published jailbreak examples look like single prompts, but the failures that matter operationally often emerge across a conversation: instructions shift gradually, the model’s earlier answers become context that conditions later ones, and a constraint that holds on turn one erodes by turn six.

A static battery of one-shot prompts cannot observe this. Multi-turn adversarial testing treats the conversation as the unit of test: each turn is chosen based on what the system has already said, so the test follows the failure as it develops instead of sampling isolated moments. This is also why failures found in multi-turn testing can be harder to reproduce — and why recording the full sequence, not just the final response, is essential evidence.

Testing a model, an application, or an agent

“The system” matters as much as the prompt. The three common targets fail differently, and a finding against one does not automatically transfer to another:

• The model itself. A base or assistant model tested through its API. Findings generally reflect the model’s training and safeguards.
• The LLM application. A model wrapped in instructions, retrieved content, and business logic. Behaviour now depends on the whole assembly — a “model” finding may actually be an application-configuration finding, or vice versa.
• The AI agent. A system that plans and acts through tools. Failure modes extend beyond text: excessive agency, unsafe tool use, and unauthorised actions. See our guide to AI agent security testing.

A credible report states which layer a finding belongs to. Blaming the model for what is really an application-level instruction conflict wastes everyone’s remediation time.

The role of adaptive test generation

Fixed prompt lists have a ceiling: once the target has seen them (or something like them), coverage stops growing. Adaptive approaches use the responses observed so far to decide where to push next — refining strategy, changing direction, and building multi-turn sequences that a static list cannot express.

Adaptivity has a cost worth respecting: adaptive testing can explore places you did not plan to go. That is why scope definition and continuous recording of everything the system was shown and produced are part of a serious methodology, not paperwork. Our approach page describes how DYMORPIC structures this — objective, challenge, observe, examine, report.

A candidate finding is not a vulnerability

An unusual or unsafe-looking response is a candidate: a signal worth examining. It becomes a finding only after contextual analysis and verification:

1. Examine in context. Was the prohibited behaviour actually exhibited, or does the response merely resemble it?
2. Check the security consequence. Would this failure matter for this system in deployment — data exposure, unauthorised action, policy violation — or only under artificial conditions?
3. Reproduce. Run the sequence again. How reliably does it occur? A failure that reproduces 1 in 10 times is a different finding from one that reproduces every time.
4. Verify independently. Where possible, have a process or person that did not produce the candidate re-examine the evidence. Self-graded findings inherit the blind spots of whatever produced them.
5. Report with evidence. Reproduction steps, the full interaction record, and explicit limitations.

When the evidence does not support a clear conclusion, the honest outcome is inconclusive — recorded, not discarded, and not forced into a pass-or-fail claim.

Checklist: evaluating an LLM red teaming tool

Whether you are assessing commercial platforms or open-source toolkits — Garak, Microsoft PyRIT, and Promptfoo are widely used starting points — these questions separate tools that produce defensible findings from tools that produce dashboards:

• Can you define your own prohibited behaviours, or only choose from built-in categories?
• Does it support genuinely multi-turn testing, or does it send isolated prompts?
• Do strategies adapt to observed responses, or replay a fixed list?
• Does it distinguish candidate findings from verified findings?
• Is verification performed independently of whatever generated the candidate?
• Does it record the full interaction context needed to reproduce a result later?
• Can it attribute a finding to the model, the application, or the agent layer?
• Are results pinned to the exact target version tested?
• Can you export raw evidence, not only summary scores?
• Does it report inconclusive outcomes honestly instead of forcing pass/fail?

No tool answers yes to everything today. Knowing where a tool stops is part of interpreting what its findings mean.

Where PROJECTILE fits

PROJECTILE TERMINAL is DYMORPIC’s implementation of this methodology: 30,000+ base strategies, self-learning that sharpens subsequent testing, multi-turn adaptive stress testing, continuous 24/7 coverage, and independent verification of candidate findings — designed to surface potential vulnerabilities in as little as an hour under suitable conditions.

Frequently asked questions

Is LLM red teaming only about jailbreaks?

No. Jailbreaks are one failure class. Red teaming also covers prompt injection, sensitive-data leakage, excessive agency in systems with tools, misinformation, and application-specific policy violations. The common structure is a defined prohibited behaviour plus adversarial testing against it.

Do we need red teaming if our model provider already tests their models?

Provider testing covers the model, not your assembly of instructions, retrieved data, tools, and business logic. Most application-level incidents trace to that assembly, not to the base model alone.

How is this different from automated security scanning of normal software?

Traditional scanners look for known patterns in deterministic code. LLM behaviour is probabilistic and context-dependent, so LLM red teaming relies on generated and adapted interactions, statistical reproduction, and human-or-model judgement — which is precisely why evidence and verification standards matter more, not less.

How often should a system be red teamed?

After meaningful changes to instructions, tools, data sources, or model versions — and, where coverage is continuous, on an ongoing basis. A one-time test describes one system on one day.

Can red teaming prove a system is safe?

No. Red teaming can demonstrate that specific failures exist (with evidence) or that specific attempts did not succeed under specific conditions. Absence of found failures is not proof of absence of vulnerabilities.