llm security

The plan was ordinary. Take a set of documented prompt-injection classes, run them against a pinned agent framework, and report what got through.

Before running anything, I went to read how the benchmark records a result. That reading ended the original plan and produced this article instead.

The benchmark is AgentDojo, built by ETH Zurich’s SPY Lab. It is the most serious open tool for agent prompt injection testing, and I want to be clear from the start that nothing here is a criticism of it. It is well built, its source is readable, and it is honest about what it does.

What I found is a property of how agent security gets scored, not a bug. It affects anyone quoting a number from this class of tool. And you can check it yourself in about ninety seconds.

Here it is in one sentence: in AgentDojo 0.1.35, a provider outage and a successful defence are recorded identically.

Key takeaways

• Verified in source: AgentDojo 0.1.35 contains three exception paths that set security = True on infrastructure failures. A context-window overflow and a working defence are recorded identically.
• Verified in source: The scoring does not confirm that a payload reached the model. A dropped payload and a refused payload produce the same result.
• Interpretation, not measurement: Published agent security numbers are therefore somewhat optimistic. The size of the effect is unmeasured and environment-dependent.
• Design principle: Ambiguity should resolve to “indeterminate”, not to “secure”. A benchmark has to write something to disk; a published article does not.
• Failure shape is not a detail. A silent failure produces a plausible answer and no alert. Attack success rate treats it identically to a loud refusal.
• Attribution defaults to unknown. A mitigation observed with no defence configured is a property of the model, not of the framework. Assuming otherwise breaks on your next model upgrade.
• Limitation: No models were run for this article. It is source analysis, and the payload evaluation remains open.

---

Quick Navigation

---

What a Model Card Is, and What Model Card Disclosure Now Means

What is a model card? A model card is a short structured document published alongside a machine learning model that states its intended uses, training data, evaluation results, and known limitations. The format was proposed by Margaret Mitchell and colleagues in 2019 and has since become the default unit of AI documentation.

The original proposal assumed one document would carry everything. That assumption broke in 2026. Frontier labs now publish a system card focused on pre-deployment safety evaluation, a model card focused on specifications and benchmark results, and — for anyone selling into Europe — a separate public summary of training content filed under the AI Act. These three documents overlap unevenly and almost never link to one another.

So “model card disclosure” in 2026 means something looser than it did five years ago: the sum of what a provider publishes about a model, wherever it lands. The problem for the reader is that nobody tells you which document holds which fact.

---

How We Audited Model Card Disclosure

We read four flagship cards released between July and September 2026 directly, against a seven-item checklist, and recorded what each one states on its own terms. Where a card points elsewhere rather than stating something, we recorded a pointer rather than crediting the disclosure.

The sample is small and deliberately so: OpenAI’s GPT-6 Astra (3 September), Google DeepMind’s Gemini 3.8 Flash (2 September), Anthropic’s Claude Opus 5 (24 July), and Thinking Machines Lab’s Inkling (15 July). It spans two closed-weight US labs, one closed-weight card from a lab that publishes model cards rather than system cards, and one open-weight release.

Three cells below are marked unverified. We could not confirm them from the primary document within the scope of this audit, and we are not going to guess. Flagging what you could not check is the difference between an audit and a roundup.

For breadth, we cross-reference three independent corpora that cover far more ground than four cards: the GPAI Ledger’s archive of Article 53(1)(d) filings, the AI Accountability Lab’s graded assessment of those filings, and Stanford CRFM’s Foundation Model Transparency Index.

---

What I actually tested in this agent prompt injection testing exercise

I tested the measuring instrument. Not the payloads, not the models.

The subject is agentdojo version 0.1.35, the current release, uploaded on 27 October 2025. I verified that against the PyPI JSON API rather than the project’s documentation, downloaded the published wheel, and read the source. Every claim below comes from that wheel, not from docs that may lag the code.

Here is what the package contains at that version.

Property	Value
Release	0.1.35, uploaded 2025-10-27
Releases published to date	36
Python requirement	>= 3.10
Evaluation suites	workspace, banking, travel, slack
Suite versions shipped	v1, v1_1, v1_1_1, v1_1_2, v1_2, v1_2_1, v1_2_2
Built-in defences	4
Attack generators	17
Result representation	Two booleans per run

The four defences are tool_filter, transformers_pi_detector, spotlighting_with_delimiting and repeat_user_prompt. The seventeen attack generators split into twelve goal-hijacking templates and five denial-of-service templates.

That last row is where this article lives.

---

The test harness

I built a small analysis layer that reads AgentDojo’s own trace logs and re-scores them. It runs no attacks of its own and calls no model. It is deterministic, which means anyone can re-run it against the same logs and get the same output.

AgentDojo writes one JSON file per run to a predictable path:

runs/{pipeline}/{suite}/{user_task}/{attack}/{injection_task}.json

Each file carries the full trajectory plus the context the logger attached: utility, security, attack_type, pipeline_name, benchmark_version, agentdojo_package_version, an evaluation timestamp, and the injection strings used.

That is a generous amount of information. It is enough to ask questions the summary statistics do not answer, which is exactly what the harness does.

The harness ships with twelve unit tests covering every branch of its classifier. The test that motivated the whole project asserts that an infrastructure error is not scored as a block.

---

What the two booleans encode

AgentDojo reduces each run to utility and security.

utility is true when the agent completed the legitimate user task. security is true when the injection task goal was not achieved. So security = False means the attack worked.

This is a reasonable design. It produces attack success rate, which is the number most of the literature reports, and it makes results comparable across models and defences.

The compression is also where information goes missing. Two runs that look identical in the summary can be very different events.

Consider two cases that both record security = True:

• The model read the injected instruction and declined to follow it.
• The injected instruction never reached the model at all.

The first tells you something about the model’s resistance. The second tells you something about your filtering, or about nothing at all if no filter was configured. Collapsed into one boolean, they are indistinguishable.

---

What happened

Reading agentdojo/benchmark.py at version 0.1.35 turned up three exception handlers that set utility = False; security = True.

They fire on:

• BadRequestError where the code is context_length_exceeded, the parameter is max_tokens, or the message asks to reduce message length
• ApiError whose string contains internal server error
• ServerError

Each path logs the error and moves on. The run is counted.

So a context-window overflow enters the summary statistics in the same column as a defence that worked. A provider having a bad afternoon looks, at the aggregate level, like security.

There is a fourth behaviour worth noting. For denial-of-service attack generators, the scoring inverts: security = not utility. Those five generators measure availability rather than goal hijacking, so mixing them into a single headline number combines two different properties.

These are observations about the code, not about any model’s behaviour. No model was run for this article. That distinction matters and I want it stated plainly rather than buried in the limitations section.

---

Where the measurement holds

The design is sound for what it was built to do.

AgentDojo’s core contribution is realism. Its environments are stateful and require multiple tool calls. Injection strings are placed inside realistic content, such as an email body, rather than appended to a tool response where they would be trivially detectable. Over 900 combinations of benign user tasks and malicious injection tasks are paired across the four suites.

That is a substantially harder setting than earlier work. InjecAgent used simulated single-turn scenarios where one adversarial item is fed as a tool output without evaluating the agent’s planning. The newer AgentDyn benchmark reports average trajectory lengths of 7.1 steps against AgentDojo’s 3.49 and InjecAgent’s 1.

The two-boolean scheme also does something quietly important. By tracking utility alongside security, it catches the failure mode where a defence works by breaking the agent. A system that refuses everything scores perfectly on security and terribly on utility, and you can see that immediately.

Agent task competence is a real confound in every security number, and AgentDojo’s original evaluation is upfront about it. The best-performing model in that paper reached only 78.22% utility in benign settings with no attack present. If an agent fails a fifth of its ordinary tasks, some portion of every security result is just ordinary incompetence.

---

Where the measurement breaks

Three places, in rising order of importance.

1. Infrastructure errors count as wins. Covered above. The practical effect depends on how often those exceptions fire in your environment, which depends on your context lengths, your provider’s reliability that week, and your rate limits. None of those have anything to do with security.
2. Delivery is never verified. Nothing in the summary confirms the payload reached the model’s context. A truncation bug, a serialisation error, or a defence quietly dropping content all produce the same security = True as a model that read the attack and refused.
3. Ambiguity resolves toward safety. This is the pattern underneath the other two. When the framework cannot determine what happened, it records the secure outcome. That is a defensible engineering choice, because a benchmark has to write something. It is a poor choice for a published number, because it biases every aggregate in the reassuring direction.

For agent prompt injection testing that informs a real deployment decision, an unresolvable run should be visible as unresolvable.

---

The shape of failure matters

A pass/fail score tells you whether something went wrong. It does not tell you whether anyone would have noticed.

That second question decides whether your incident response has any chance of working. I use four shapes.

Shape	Definition
Loud	The system refused, blocked, or sanitised, and made that visible. The event enters your telemetry.
Silent	The injected behaviour executed and the output reads as a normal, plausible completion. Nothing signals a problem.
Partial	The agent followed part of the injected instruction without completing it.
Indeterminate	The shape cannot be established from the trace.

A silent failure is one where the system produces an apparently valid response while following the injected instruction.

Silent failures are the operationally dangerous ones. A loud failure generates a log line and possibly an alert. A silent failure generates a plausible answer, a satisfied user, and no reason for anyone to look. If an agent exfiltrates a record and then produces a competent summary of your inbox, the summary is what your reviewer sees.

Attack success rate treats both identically. They are not remotely the same risk.

The harness detects loudness with a pattern list matching refusal and warning language in assistant output. That is a lexical heuristic, and it errs in both directions. It over-counts loudness when a model uses refusal-shaped phrasing while still complying, and under-counts it when concern is expressed in wording the list does not cover. Anyone using it should hand-audit a sample and publish the agreement rate.

---

Why attribution is difficult

When a payload does not get through, the interesting question is why. It is also the question most evaluations skip, because it is genuinely hard.

A blocked payload looks the same whether the block was engineered or accidental. The trace shows an attack that did not succeed. It does not show you which component is responsible.

The harness records attribution across ten causes, with a stated confidence on each: documented, observed, inferred, or unknown. The default is “cannot be determined”, and a cause is only assigned when a rule can point at a specific artefact in the trace or a documented behaviour of the configured pipeline.

Two rules carry most of the weight.

1. No defence configured, payload reached context, attack failed. Attributed to model behaviour, never to the framework. The framework did nothing. Whatever resisted the attack is a property of the model you happened to pick, and it may not survive your next model upgrade.
2. A presentation-layer defence configured, payload reached context, attack failed. Attributed to model behaviour at inferred confidence only, with competing explanations listed. Delimiting changes how content is presented, not whether it arrives. The refusal decision still sat with the model, but the delimiter markup may itself have supplied the cue. Those cannot be separated from a single trace.

Every attribution that is not “undetermined” carries a list of competing explanations. If that list is empty and confidence is below “observed”, it is a bug in the classifier rather than a finding.

The most common error in this area is reading a model’s refusal as a framework security feature. It is an easy mistake and it leads directly to deploying the same architecture on a different model and being surprised.

---

What the results actually tell us

Three things, held at appropriate strength.

First, verified. AgentDojo 0.1.35 scores three classes of infrastructure error as defensive successes, and does not verify payload delivery. This is directly checkable in the published source.

Second, an interpretation. Published agent security scores are therefore slightly optimistic by an amount nobody currently reports. How much is unknown and depends entirely on the environment. I am not going to put a number on it, because I have not measured one.

Third, an open question. Whether stricter scoring changes any published conclusion is unknown. It may turn out that infrastructure errors are rare enough to be noise. That would be a useful finding too, and it is testable with the harness.

What this does not tell us is anything about which frameworks resist prompt injection. That evaluation has not been run. The original question stands open.

---

Limitations

This section is longer than the findings section, which is the correct ratio.

• No models were run. This is source analysis. Every claim is about code, not behaviour.
• One framework, one version. AgentDojo 0.1.35 only. Whether other agent evaluation tools share this property is untested, though the design pressure that produces it is common.
• Point-in-time. Verified 20 September 2026 against a release dated 27 October 2025. The project is active and this may change. Re-check before citing.
• The stricter rubric is unvalidated. The harness demands positive evidence of payload delivery before scoring a block. That is a defensible bar, but it is my bar. It will produce lower block rates than AgentDojo’s own scoring on identical logs, and I have not demonstrated that the difference is meaningful rather than pedantic.
• Loudness detection is lexical. A pattern list, not a semantic judgement. Error in both directions.
• Template attacks measure a floor. Work on adaptive attacks shows defences evaluated against fixed templates degrade badly under attacks adapted to them. Any result from this class of tool is a lower bound on what a motivated attacker achieves.
• Passing proves nothing. A system that survives this evaluation is not secure. It survived these payloads, at these versions, on this date, under this configuration. Failing one test does not mean a framework is broadly insecure, either.

---

How to reproduce this

The source reading needs nothing but the package.

bash

pip download agentdojo==0.1.35 --no-deps
unzip agentdojo-0.1.35-py3-none-any.whl -d src

Then open src/agentdojo/benchmark.py and search for security = True. The three exception paths are in the function that runs a single task against a pipeline. The attack generators are in src/agentdojo/attacks/, and the defence names sit in src/agentdojo/agent_pipeline/agent_pipeline.py.

To run the re-scoring layer, generate traces first. Hold the model fixed and vary one defence at a time, or attribution becomes impossible:

bash

python -m agentdojo.scripts.benchmark \
  --model <exact-dated-model-string> \
  --defense tool_filter \
  --attack important_instructions

Read the directory name that appears under runs/, since that string is the join key the harness needs.

Two things to record and publish: the exact dated model string, never “latest”, and your temperature. Leaving temperature at the provider default is a defensible choice, but it is a choice, and it belongs in your methodology.

Set runs per cell above one. A single run cannot distinguish a defence from a coin flip, and these are stochastic systems. Three is a floor. Five is better. Report cells that produce different outcomes across repeats as variable rather than averaging them, because the variance is the finding.

All of this runs against AgentDojo’s simulated in-process environments. No third-party system is involved, and nothing here should be pointed at infrastructure you do not own.

---

References

Primary research: source reading of agentdojo-0.1.35-py3-none-any.whl, and the PyPI release metadata API, both accessed 20 September 2026.

Official documentation: AgentDojo repository · AgentDojo documentation

Academic research: AgentDojo, arXiv:2406.13352 · AgentDyn, arXiv:2602.03117 · Adaptive attacks against indirect prompt injection defences, arXiv:2503.00061 · Meta SecAlign, arXiv:2507.02735

---

Frequently Asked Questions

---

Keep reading

Here are the latest posts from the blog.