AI security testing

The report lands in your inbox on a Thursday. Forty pages, a summary table, a handful of medium findings, all closed. Somewhere near the front sits the sentence everyone will quote in the launch meeting.

“No critical vulnerabilities were identified.”

That sentence is almost certainly true. It is also one of the most misread lines in AI security, because red teaming answers a far narrower question than the people reading the report believe.

What follows is four flaws that hide comfortably inside a clean red teaming result. Each opens with the kind of line that shows up in real reports, then covers what it actually means, the evidence behind it, and the clause to put in your next scope of work.

Key Takeaways

• A pass means one configuration survived one set of attacks on one day. It is a snapshot, not a certificate.
• Single-turn testing misses most of the risk. Cisco ran 6,986 multi-turn attacks against 15 flagship models and broke through as often as 88.3% of the time.
• Models are probabilistic, so binary results mislead. The same attack can fail four times and succeed on the fifth, which means red teaming needs repeat runs and success rates, not pass or fail.
• The tested system usually is not the shipped system. Model version, system prompt, tools and retrieval all drift between assessment and launch.
• Red teaming shows presence, never absence. It proves a flaw exists. It can never prove none remain.
• The fix is cadence, not coverage. Microsoft’s AI Red Team put it plainly after 100 products: securing AI systems is work that never completes.

---

Quick Navigation

---

How to Read a Red Teaming Report Without Being Reassured

Start with a reframe. A red teaming engagement is a search, and every search has a scope, a budget and a stopping rule.

The report tells you what the searchers found. It does not tell you how much of the space they covered, and red teaming reports almost never state that denominator.

That gap is where false confidence grows. Nobody lies. The reader hears “we found nothing bad”, while the document says “we found nothing bad in the places we looked, in the time we had”.

Microsoft’s AI Red Team made this explicit after running more than 80 operations across over 100 generative AI products. One of their eight lessons is simply that red teaming is not safety benchmarking, and another is that the work of securing AI systems will never be complete.

So read a red teaming report the way an auditor reads a sample. Ask what was in scope, how deep the sampling went, and what the result would look like if the system changed next week.

---

Flaw One: The Red Teaming Ran One Turn, Attackers Run Ten

“We submitted 1,200 adversarial prompts across twelve harm categories. 1,186 were correctly refused.”

What it actually means. Twelve hundred single messages were sent and scored one at a time. Nobody held a conversation, which is the most common shortcut in red teaming.

That distinction decides most of your risk, because real attackers do not fire one prompt and leave. They build rapport, reframe the goal, split the payload across turns and return to it later. Single-turn testing never sees any of that.

The Evidence on Multi-Turn Attacks

The numbers here are not close, and they should reset how you price an engagement. At VB Transform in July 2026, Cisco’s head of AI threat intelligence reported that 6,986 multi-turn attacks against 15 flagship models broke through as often as 88.3% of the time.

Earlier work points the same way. Scale AI’s multi-turn human jailbreak study found human red teamers beat automated ensembles by 19 to 65 percentage points, with overall success above 70% on HarmBench.

One result deserves a pause. Against one strong defence, automated single-turn attacks scored zero while multi-turn human attacks still landed around 70%. A red teaming report built on the first method would have read as flawless.

The follow-up is stranger still. An ACL 2025 paper showed that compressing those multi-turn conversations back into a single structured prompt reached 70.6% to 95.9% success, sometimes beating the original conversation, by hiding the request inside enumerated or code-like formatting.

So the flaw was never really about turn count. It was about context in a shape safety training had not seen, and a suite that varies only the wording will keep missing it.

What to Write Into the Red Teaming Scope

Require a stated minimum conversation depth in the red teaming scope, five turns or more, across every harm category rather than a sampled few.

Require delivered transcripts, not just counts. You cannot fix a conversational attack you cannot read, and transcripts are what make a finding reproducible.

Require at least one structural-reformatting pass, where the same request arrives wrapped in lists, code blocks or documents.

---

Flaw Two: Red Teaming a Probabilistic System Needs Repeat Runs

“Attack vector 4.2: blocked. Attack vector 4.3: blocked.”

What it actually means. Each attack was tried once, and one attempt was treated as decisive. Testing a language model this way borrows a habit from software QA that does not transfer.

Traditional security testing gets away with it. A SQL injection flaw fails the same way every time you send the same malformed input.

A language model does not behave like that. The same prompt can be refused four times and answered on the fifth, because sampling, context and load all move the outcome. Results need repetition to mean anything.

Binary results therefore hide the shape of the risk. “Blocked” might mean blocked every time, or blocked 80% of the time, and those two systems carry very different exposure at a million requests a month.

The Statistics Red Teaming Reports Should Carry

Ask for attack success rate rather than a verdict, with the trial count next to it. Ten runs per vector is a reasonable red teaming floor, and any result from a single attempt should carry a label saying so.

Ask about the grader too. Most large red teaming exercises score results with another model, and that judge has its own false-positive and false-negative rate.

An uncalibrated judge quietly sets your finding count. Ask what sample of its decisions went to human review, and what the agreement rate was, because that number bounds the whole result.

Then watch for the reverse error. A judge tuned to avoid false alarms marks borderline compliance as a refusal, and your report gets cleaner while the system gets no safer.

---

Flaw Three: The System Under Red Teaming Was Not the System You Shipped

“Testing was conducted against the staging environment between 3 and 14 March.”

What it actually means. Everything that changed after 14 March sits outside the red teaming result, and in an AI stack that is usually quite a lot.

Microsoft’s team pushes hard on this point. Their ontology treats the system as the end-to-end product rather than the model, because integration is where a surprising share of real failures live.

Their own case work makes it concrete. When they tested a vision language model, image inputs proved far more jailbreakable than text, which no amount of text-only red teaming would have revealed.

Five Things That Drift Between Test and Production

1. The model version. If your integration calls a floating alias rather than a pinned snapshot, the weights behind your tested behavior can change without notice.
2. The system prompt. Most teams edit it weekly. Every edit is an untested configuration.
3. The tool list. A new connector adds capability the testers never had access to, which changes what a successful injection can reach.
4. The retrieval corpus. New documents mean new places for indirect prompt injection to hide, and that path stays the top-ranked risk on the OWASP LLM list. We broke the variants down in our guide to the classes of prompt injection.
5. The guardrail configuration. Filters get loosened for a legitimate false-positive problem, and nobody re-runs the suite afterwards.

Pinning the System Under Red Teaming Test

Record a configuration fingerprint with every red teaming engagement: model snapshot ID, system prompt hash, tool manifest, guardrail settings and retrieval index version.

Then treat a change to any of those five as a change to the finding. The report is evidence about one fingerprint and nothing else.

This is the same discipline that makes containment work, and our piece on sandbox isolation layers covers the environment side of it.

---

Flaw Four: Red Teaming Shows Presence, Never Absence

“No critical vulnerabilities were identified in the scope defined above.”

What it actually means. The red teaming team looked for the harms someone thought to list, and did not find those.

This is the deepest of the four flaws and the least fixable. Red teaming demonstrates that a problem exists. It has no mechanism for demonstrating that no problems remain.

Think about what that implies for novel attacks. A red teaming exercise in early 2024 could not test for attack classes published in late 2025, and yours cannot test for whatever lands next quarter.

Prompt injection illustrates the gap neatly. It has sat at the top of the OWASP LLM risk list since 2025, and it remains a flaw that testing finds rather than one that alignment has fixed.

Measuring Red Teaming Coverage Honestly

You cannot measure red teaming coverage against all possible attacks. You can measure it against a named taxonomy, which is the next best thing.

Ask for findings mapped to MITRE ATLAS and the OWASP LLM Top 10, with explicit “not tested” entries rather than silence. In a coverage matrix, a blank row is information and a missing row is not.

Ask for the harm categories considered and then ruled out of scope, with reasons. That list is often more revealing than the red teaming findings themselves.

Finally, ask what the team would have tested with double the budget. Every honest red teaming lead has a ready answer, and that answer is your real risk register.

---

When to Re-Run Red Teaming: A Trigger List

Annual testing made sense when systems changed annually. Tie your red teaming cadence to change events instead.

Trigger	Re-test scope
Model version or snapshot change	Full suite
System prompt edit	Targeted, plus top jailbreak vectors
New tool or connector added	Full agentic path, permissions and egress
Retrieval corpus expansion	Indirect prompt injection set
Guardrail or filter loosened	The categories that filter covered
Novel public attack class	The affected categories, within days

Two of those rows fire most weeks in an active product. That is the real argument for automating the regression layer and saving human red teaming for depth and novelty.

Continuous testing is now the consensus position rather than a vendor pitch. As Check Point put it in a 2026 write-up, AI security has no final exam — it has an improvement process.

---

What Good Red Teaming Looks Like on Paper

A strong red teaming report is less comfortable to read than a weak one, and you can spot it from the structure alone.

A good red teaming report puts the configuration fingerprint on page one. It reports success rates with trial counts rather than verdicts. It includes the multi-turn transcripts.

It maps findings to a named taxonomy, marks untested categories explicitly, and separates security findings from responsible-AI harms because those need different owners.

It names what the red teaming could not reach: budget limits, access limits, missing modalities, all stated rather than implied.

And it ends with a re-test trigger list rather than a date twelve months out. If your last red teaming report did none of these, the gap is in the scope you wrote, not the work they did.

Good telemetry makes all of it easier, since red teaming findings are only reproducible if you captured what the system did. Our field guide to the four signals an agent stack must emit covers that layer.

---

The Verdict: Red Teaming Is a Cadence, Not a Certificate

Go back to that Thursday red teaming report and the sentence everyone quoted. Nothing in it was wrong.

The error was in translation. “No critical vulnerabilities in this scope, this configuration, this fortnight” became “the system is safe”, and a launch decision rode on the gap between them.

None of this is an argument against red teaming. It is the only practice that turns a vague worry into a specific, reproducible finding you can actually fix, and it is worth every rupee.

Just hold red teaming to what it is. It measures resistance to attacks someone imagined, in a system that has since changed, scored by a judge with its own error rate.

Ask for the denominator, the trial counts and the fingerprint. A red teaming report that gives you all three is worth more than one with no findings at all.

---

Frequently Asked Questions

---

Keep reading

Here are the latest posts from the blog.