ai evaluation

The invoice from your model provider and the quote from your server vendor are moving in opposite directions, and both are telling the truth.

Epoch AI’s benchmark-anchored work puts the decline in inference prices at a median of roughly 50x per year across six benchmarks, rising to around 200x per year when restricted to data after January 2024. Meanwhile, Meta raised its 2026 capital expenditure guidance on 29 April 2026 from $115–135 billion to $125–145 billion, and Mark Zuckerberg pointed to memory pricing as a driver, according to Fortune’s reporting from the call.

Both are true because they sit at different layers. Token pricing is a retail price for an output. Infrastructure cost is what somebody had to buy to produce it. Between those layers sits memory, which spent 2026 becoming the most volatile input in the AI supply chain.

Key takeaways

• The 90% quarter is history, and that matters. TrendForce forecast conventional DRAM contract prices rising 90–95% QoQ in 1Q26. By 3Q26 the same firm forecast 13–18%. The rate of increase collapsed; the price level did not come back down.
• Server DRAM is now a four-figure line item per module. Seoul Economic Daily reported on 17 September 2026 that the fixed contract price for a 64GB DDR5 server RDIMM stood at $1,500 as of 15 September, against $272 a year earlier.
• HBM consumes wafer area out of proportion to the bits it delivers. TrendForce estimates HBM will take roughly 30% of the top three suppliers’ DRAM wafer input by end-2027 while supplying only about 13% of DRAM bits.
• A faster accelerator is not automatically a cheaper inference system. Per-chip compute has been growing faster than per-chip bandwidth, which raises the batch size needed to keep silicon busy.
• Context length is a cost multiplier that never appears on a rate card. In the illustrative model below, moving from 8k to 32k tokens of context cuts concurrent requests per accelerator roughly four-fold.
• Memory inflation moves the rent-or-own crossover, but less than the headlines imply. In the illustrative server model, the utilisation needed to beat the cheapest public cloud rate shifts from roughly 45% to roughly 49%.
• Micron reports fiscal Q4 2026 results on 30 September 2026, the first full quarter of guidance after its 84.6% GAAP gross margin quarter. It is the clearest near-term read on whether supply is loosening.

---

Quick Navigation

---

Start With the Memory Bill: What AI Memory Costs Look Like in 2026

The most quoted number from this cycle is already out of date, and reusing it without context is the fastest way to get the story wrong.

TrendForce’s memory pricing survey of 2 February 2026 revised its 1Q26 forecast for conventional DRAM contract prices upward from 55–60% QoQ to 90–95% QoQ, with server DRAM projected to climb around 90% QoQ, which the firm called the largest quarterly increase on record. That figure gets recycled constantly. It was a first-quarter forecast.

Read the rest of the series and a different shape emerges. TrendForce projected 58–63% QoQ for 2Q26 on 31 March, then 13–18% QoQ for 3Q26 on 3 July, with server DRAM in the same band. On 30 June it raised its 4Q26 PC DRAM forecast to only 3–8% QoQ.

So the rate of increase fell sharply across 2026. The level did not. Conflating the two produces bad procurement decisions in both directions.

Where has the level landed? Seoul Economic Daily reported on 17 September 2026 that as of 15 September, the fixed contract price for a 64GB DDR5 server DRAM module stood at $1,500, roughly 5.5 times the $272 recorded a year earlier, with some spot transactions near $3,100. That tracks a Citi research note dated 12 May 2026 projecting the same module rising from $873 in Q1 2026 to roughly $1,586 by Q4.

The categories are not interchangeable. The $1,500 figure is a fixed contract price for one module type; the $3,100 figure is a spot transaction, and spot diverges widely from contract in tight markets. Neither reflects what a hyperscaler with a long-term agreement pays. TrendForce noted on 9 July 2026 that several US cloud providers had signed multi-year LTAs restricting price increases for those customers, which is why moderation in headline prices does not reach all buyers evenly.

The supplier side confirms the magnitude. Micron’s fiscal Q3 2026, ended 28 May 2026, produced revenue of $41.46 billion against $9.30 billion a year earlier, with GAAP gross margin at 84.6% versus 37.7%. A memory manufacturer earning 85 cents of gross margin on the dollar is the same fact as your RDIMM quote, seen from the other end.

Conventional DRAM and HBM are different products with different pricing mechanics. HBM is negotiated annually rather than quarterly, which is why its contract prices lagged the commodity DRAM surge. That lag is now closing, and the reason has nothing to do with demand for HBM.

---

How HBM Rewrote the Allocation Problem Behind AI Memory Costs

HBM does not simply compete with conventional DRAM for customers. It competes for wafers, and it is an inefficient consumer of them.

TrendForce estimates that HBM wafer input across the top three suppliers will account for approximately 18%, 22% and 30% of total DRAM wafer input at the end of 2025, 2026 and 2027, while representing only about 8%, 9% and 13% of total DRAM bit supply over the same period. These are TrendForce estimates rather than disclosed manufacturer figures, and should be read as a modelled view of a market whose participants publish very little.

Hold those two series side by side. By end-2027, on this estimate, roughly 30% of wafer starts produce roughly 13% of the bits. HBM stacks DRAM dies vertically using through-silicon vias and a logic base die, and the die area, packaging yield and test burden mean each delivered gigabyte absorbs far more capacity than a DDR5 gigabyte.

That is the crowding-out mechanism in one sentence: every wafer allocated to HBM removes a disproportionate quantity of conventional DRAM from the market.

The twist in 2026 is that the crowding ran in an unexpected direction. TrendForce reported on 2 June 2026 that, on its analysis of per-wafer revenue derived from die size, yield and per-gigabit pricing, HBM wafer revenue was overtaken by DDR5 64GB RDIMM in 1Q26, with HBM profitability falling below the RDIMM’s from that quarter on. Commodity server memory briefly became the better use of a wafer than the exotic AI product.

Suppliers reallocate capacity in response, depending on where HBM contract negotiations land. TrendForce’s conclusion is that the three major manufacturers will raise HBM quotations substantially in 2027 to restore the premium. That is a forecast, not a settled outcome.

The supply side offers little relief on a 2027 budget timescale. SK hynix CEO Kwak Noh-Jung told Reuters on 10 July 2026 that 2027 would be the worst year in the industry’s history from a supply perspective, with wafer and manufacturing growth around 12% falling well short of demand, and demand exceeding capacity beyond 2030. TrendForce estimated on 9 July 2026 that total RDIMM bit supply will grow only 15–20% year over year in 2027, lagging server CPU shipment growth.

Epoch AI found that AI chips consumed over 90% of total HBM production in 2025. There is no meaningful non-AI buyer left to displace.

---

Memory Bandwidth Can Matter More Than FLOPS

Everything above concerns what memory costs to buy. This section concerns why you need so much of it.

Inference splits into two phases with opposite hardware profiles. Prefill reads the entire prompt in one parallel pass and saturates the arithmetic units. Decode generates one token at a time, and each token requires reading the model’s weights plus the accumulated key-value cache out of memory to perform a comparatively tiny amount of arithmetic. Decode is bound by memory bandwidth, not by compute. As Databricks put it in its inference performance work, achieved memory bandwidth predicts token generation speed better than peak compute throughput does.

Why batch size sets your AI memory costs

Consider a 70-billion-parameter dense model at FP8, so roughly 70 GB of weights. On an accelerator with 8 TB/s of theoretical HBM bandwidth achieving 70% in practice, the chip reads the full weight set about 80 times per second. Serve one user and you get roughly 80 tokens per second and a very expensive token. Serve 64 users in a batch and the same 80 weight reads produce around 5,120 tokens per second, because the weights were fetched once and used sixty-four times.

Batching is the economic engine of inference serving. Utilisation is not a nice-to-have; it is the denominator.

Context length as a memory cost multiplier

Capacity now reasserts itself. Every concurrent request carries its own KV cache, and that cache competes with the weights for the same HBM.

Illustrative example. Take a representative 70B-class model with grouped-query attention: 80 layers, 8 key-value heads, head dimension 128, cached at one byte per element. KV cache per token is 2 × 80 × 8 × 128 = 163,840 bytes, about 160 KB.

On a 192 GB accelerator holding 70 GB of weights, roughly 122 GB remains for cache. At 8,000 tokens of context each request needs about 1.31 GB, allowing roughly 93 concurrent requests. At 32,000 tokens each needs about 5.24 GB, allowing roughly 23.

Same hardware, same model, same advertised price per token. Four times fewer users per accelerator, and therefore roughly four times the infrastructure cost behind every token produced.

Now add the generational trend. NVIDIA’s Rubin VR200, due in the second half of 2026, carries 288 GB of HBM4 at 22 TB/s against Blackwell’s 8 TB/s on HBM3e, a 2.75x bandwidth gain. Dense FP8 throughput rises from roughly 4.5 to 17.5 PFLOPS over the same step, closer to 3.9x. Compute is outrunning bandwidth, so the batch needed to keep the newer chip busy grows, and that batch needs cache, and cache needs capacity. Capacity and bandwidth bind together, which is why the faster chip does not automatically yield the cheaper serving system.

---

Repricing a Self-Hosted Inference Server: Where AI Memory Costs Land

Illustrative example. These are modelled assumptions, not a vendor quotation. No accelerator vendor publishes street pricing, and system prices vary by volume, region and configuration. The point is to show which line moved.

Take an eight-accelerator inference node. Hold every assumption constant except system DRAM, priced at the two dated contract figures above.

Component	Assumption	Cost
8 accelerators	$30,000 each (illustrative)	$240,000
CPU (2 sockets)	illustrative	$20,000
Storage (4 × NVMe)	illustrative	$12,000
Networking	illustrative	$24,000
Chassis, PSU, cooling, assembly	illustrative	$25,000
Subtotal excluding system DRAM		$321,000
2 TB system DRAM (32 × 64GB RDIMM) at $272	Sept 2025 contract	$8,704
2 TB system DRAM (32 × 64GB RDIMM) at $1,500	15 Sept 2026 contract	$48,000

System total at September 2025 memory pricing: $329,704. System total at September 2026 memory pricing: $369,000.

One line item moved. The system got about 12% more expensive, and system DRAM rose from 2.6% of the build to 13.0%.

Convert that to an operating rate. Amortise $369,000 straight-line over four years for $92,250 a year. Assume 10.2 kW of draw at a PUE of 1.3, giving 13.3 kW, which at $0.10 per kWh is roughly $11,600 a year. Add an illustrative $15,000 for colocation, support and operations. The annual total is about $118,900, or $14,858 per accelerator-year: $1.70 per accelerator-hour at 100% utilisation.

The same arithmetic on 2025 memory pricing gives $1.56.

Which assumptions move the number most

The DRAM line is real but not dominant. Three assumptions matter more.

Utilisation leads by a wide margin. At 50%, that $1.70 becomes $3.40 per delivered hour. At 30%, it becomes $5.66. Nothing else in the model has that leverage.

Amortisation period comes second. Moving from four years to three raises the hourly figure by roughly a third, and the useful-life assumption for AI accelerators is genuinely contested.

Accelerator price is third, and the assumption most likely to be wrong in your case. It carries its own memory exposure, since HBM is a large share of accelerator bill of materials, and HBM contract prices are exactly what TrendForce expects to rise in 2027.

Memory inflation raised this system’s cost by about 12% and its hourly rate by about 9%. Material, but not the multiple that consumer DRAM coverage implies, because a server is more than its memory.

---

Rent or Own: How AI Memory Costs Change the Calculation

Public list rates give a reference point. Inworld reported that NVIDIA B200 list rates spanned $3.49 to $14.24 per GPU-hour across clouds in April 2026, more than a four-fold spread for the same silicon.

Against the cheapest end of that range, the illustrative self-hosted node at $1.70 per accelerator-hour breaks even at about 49% utilisation. Under 2025 memory pricing the crossover sat near 45%. Memory inflation moved the threshold by roughly four percentage points.

That should temper the “memory prices killed self-hosting” framing. What determines the answer is whether you can keep accelerators busy.

Four forces push in different directions, and they do not cancel.

Owning gets harder as procurement risk rises. Lead times have stretched, 2027 memory allocation was reportedly settled during mid-2026 negotiations, and an organisation buying twenty nodes has no leverage.

Owning gets easier when the workload is predictable and high-volume. Steady batch inference, a fixed model, a known context distribution and no spiky traffic can hold 70% utilisation or better. At 70%, the self-hosted rate lands near $2.43 against a cloud floor of $3.49.

Renting gets harder as providers pass through their own memory bill. Cloud rates are downstream of DRAM and HBM contract pricing, with a lag set by each provider’s procurement contracts and depreciation schedules.

Renting gets easier when demand is uncertain, when you need to switch accelerator generations quickly, or when reserved-capacity discounts approach your amortised cost without the capital commitment. Reserved pricing is where memory inflation shows up most directly, because providers reprice reservations as their own inputs reset.

There is no universal answer, only a utilisation threshold that memory inflation nudged up slightly.

---

Why Token Pricing Hides Your Real AI Memory Costs

“$X per million tokens” is a real price. It is also an average over a distribution of workloads whose infrastructure costs differ by an order of magnitude. Four things break the correspondence between the rate card and the hardware.

• Output intensity. Output tokens come from the bandwidth-bound decode phase and batch less efficiently than input tokens. Providers price output several times higher for that reason, but the ratio in your traffic decides where you sit inside the average.
• Reasoning workloads. Models that generate long internal chains produce many output tokens per user-visible answer. A task that took 500 output tokens under a non-reasoning model can take thousands, at the higher rate, through the more expensive phase.
• Context length. As the worked example showed, longer contexts shrink the concurrency an accelerator sustains. Providers absorb this through pricing tiers and caching discounts, but the physical cost lands on memory capacity.
• Batching and serving architecture. Continuous batching, prefill-decode disaggregation and paged attention all exist to raise the number of users served per weight read. Two providers on identical hardware running identical models can have materially different cost structures from serving-stack quality alone.

Agentic systems compound all four at once: more calls, longer accumulated context, more output tokens, and idle time between tool invocations that nobody is billed for and everybody is renting.

The rate card tells you what a token costs. It tells you almost nothing about how much HBM sat idle to guarantee the latency you were promised.

---

What to Watch Next in AI Memory Costs

• Micron’s fiscal Q4 2026 results on 30 September 2026. Still upcoming as of publication. Micron guided to roughly $50 billion in revenue at around 86% gross margin, after $41.46 billion at 84.6% in fiscal Q3. Watch the guidance commentary more than the print: it is the clearest public read on whether 2027 supply is loosening.
• HBM contract negotiations for 2027. TrendForce expects substantial increases. Whether suppliers get them decides how wafer capacity splits between HBM and DDR5, and therefore what happens to conventional server DRAM.
• Contract price direction, not magnitude. The series moved from 90–95% to 13–18% in three quarters. The 2027 question is whether increases keep moderating, as TrendForce currently expects for server DRAM through 2H27, or flatten.
• RDIMM bit supply against server CPU shipments. TrendForce’s 15–20% bit growth estimate for 2027, against faster CPU shipment growth, is the arithmetic behind the shortage.
• Configuration downgrades. TrendForce noted that since 1H26, some CSPs and OEMs shifted RDIMM configurations from 96GB and 128GB modules down to 32GB and 64GB. When buyers cut memory per server to manage cost, demand destruction is already underway.
• New fab and packaging capacity. SK hynix has committed to substantial expansion, but buildings and advanced packaging lines arrive on multi-year schedules.

---

What AI Infrastructure Teams Should Re-Model

Twelve numbers. If you cannot produce them from current data, that is the finding.

• Memory cost per server, split into HBM (inside the accelerator price) and system DRAM, as a share of total build
• Memory cost per accelerator, so accelerator generations are comparable on a like-for-like basis
• Achieved memory bandwidth utilisation during decode, not theoretical peak
• KV cache footprint per request at your actual p50 and p95 context lengths
• Average tokens per request, input and output counted separately
• Output-token ratio, which predicts the phase dominating your bill
• Accelerator utilisation, measured over a full week including troughs
• Cost per request, not cost per token
• Cost per million tokens from your own hardware, against the rate card you pay
• Server amortisation schedule, with the useful-life assumption written down
• Power and cooling at your actual PUE and electricity rate
• Cloud versus self-hosted exposure, as the utilisation threshold at which the answer flips

That last number is the one worth putting on a wall. Everything else here is an input to it.

---

Frequently Asked Questions

---

Keep reading

Here are the latest posts from the blog.

Agent prompt injection testing can examine documented prompt-injection classes and how benchmarks record their outcomes.

The discussion concerns benchmark scoring and a proposed trace-review method. It does not report a completed UniverseBlend experiment.

The benchmark discussed here is AgentDojo, from ETH Zurich’s SPY Lab.

The distinction below concerns the interpretation of a recorded score, rather than a measured comparison of models or defences.

Here it is in one sentence: in AgentDojo 0.1.35, a provider outage and a successful defence are recorded identically.

Key takeaways

• Verified in source: AgentDojo 0.1.35 contains three exception paths that set security = True on infrastructure failures. A context-window overflow and a working defence are recorded identically.
• Verified in source: The scoring does not confirm that a payload reached the model. A dropped payload and a refused payload produce the same result.
• Open question: Whether a different rubric changes reported security scores remains unmeasured here.
• Design principle: Ambiguity should resolve to “indeterminate”, not to “secure”. A benchmark has to write something to disk; a published article does not.
• Failure shape is not a detail. A silent failure produces a plausible answer and no alert. Attack success rate treats it identically to a loud refusal.
• Attribution defaults to unknown. A failed attack alone does not identify which component caused it.
• Limitation: No models were run for this article. It is source analysis, and the payload evaluation remains open.

---

Quick Navigation

---

Scope of this source discussion

The scope is source discussion and a proposed review methodology, not payload testing or model execution.

The source discussed is agentdojo version 0.1.35, released on 27 October 2025. The relevant implementation is in agentdojo/benchmark.py in the published package.

Here is what the package contains at that version.

Property	Value
Release	0.1.35, uploaded 2025-10-27
Releases published to date	36
Python requirement	>= 3.10
Evaluation suites	workspace, banking, travel, slack
Suite versions shipped	v1, v1_1, v1_1_1, v1_1_2, v1_2, v1_2_1, v1_2_2
Built-in defences	4
Attack generators	17
Result representation	Two booleans per run

The four defences are tool_filter, transformers_pi_detector, spotlighting_with_delimiting and repeat_user_prompt. The seventeen attack generators split into twelve goal-hijacking templates and five denial-of-service templates.

That last row is where this article lives.

---

A proposed trace-review method

A trace-review method could examine information that summary scores leave out: whether the payload reached the model, whether an infrastructure error interrupted the run, and what the recorded outcome establishes. The approach below is a proposed rubric, not a tested UniverseBlend implementation or a report of re-scored results.

AgentDojo writes one JSON file per run to a predictable path:

runs/{pipeline}/{suite}/{user_task}/{attack}/{injection_task}.json

Each file carries the full trajectory plus the context the logger attached: utility, security, attack_type, pipeline_name, benchmark_version, agentdojo_package_version, an evaluation timestamp, and the injection strings used.

These recorded fields provide possible inputs for a trace review. Their completeness would need to be checked for the particular logs under review.

No implemented re-scoring harness or unit-test results are presented here.

---

What the two booleans encode

AgentDojo reduces each run to utility and security.

utility is true when the agent completed the legitimate user task. security is true when the injection task goal was not achieved. So security = False means the attack worked.

This is a reasonable design. It produces attack success rate, which is the number most of the literature reports, and it makes results comparable across models and defences.

The compression is also where information goes missing. Two runs that look identical in the summary can be very different events.

Consider two cases that both record security = True:

• The model read the injected instruction and declined to follow it.
• The injected instruction never reached the model at all.

The first tells you something about the model’s resistance. The second tells you something about your filtering, or about nothing at all if no filter was configured. Collapsed into one boolean, they are indistinguishable.

---

What the published source records

In the published agentdojo/benchmark.py source at version 0.1.35, three exception handlers set utility = False; security = True.

They fire on:

• BadRequestError where the code is context_length_exceeded, the parameter is max_tokens, or the message asks to reduce message length
• ApiError whose string contains internal server error
• ServerError

Each path logs the error and moves on. The run is counted.

So a context-window overflow enters the summary statistics in the same column as a defence that worked. A provider having a bad afternoon looks, at the aggregate level, like security.

There is a fourth behaviour worth noting. For denial-of-service attack generators, the scoring inverts: security = not utility. Those five generators measure availability rather than goal hijacking, so mixing them into a single headline number combines two different properties.

These statements concern the published code, not observed model behaviour. No UniverseBlend model-execution results are presented.

---

Where the measurement holds

The design is sound for what it was built to do.

AgentDojo’s core contribution is realism. Its environments are stateful and require multiple tool calls. Injection strings are placed inside realistic content, such as an email body, rather than appended to a tool response where they would be trivially detectable. Over 900 combinations of benign user tasks and malicious injection tasks are paired across the four suites.

That is a substantially harder setting than earlier work. InjecAgent used simulated single-turn scenarios where one adversarial item is fed as a tool output without evaluating the agent’s planning. The newer AgentDyn benchmark reports average trajectory lengths of 7.1 steps against AgentDojo’s 3.49 and InjecAgent’s 1.

The two-boolean scheme also does something quietly important. By tracking utility alongside security, it catches the failure mode where a defence works by breaking the agent. A system that refuses everything scores perfectly on security and terribly on utility, and you can see that immediately.

Agent task competence is a real confound in every security number, and AgentDojo’s original evaluation is upfront about it. The best-performing model in that paper reached only 78.22% utility in benign settings with no attack present. If an agent fails a fifth of its ordinary tasks, some portion of every security result is just ordinary incompetence.

---

Where the measurement breaks

Three places, in rising order of importance.

1. Infrastructure errors count as wins. Covered above. The practical effect depends on how often those exceptions fire in your environment, which depends on your context lengths, your provider’s reliability that week, and your rate limits. None of those have anything to do with security.
2. Delivery is never verified. Nothing in the summary confirms the payload reached the model’s context. A truncation bug, a serialisation error, or a defence quietly dropping content all produce the same security = True as a model that read the attack and refused.
3. Keep ambiguity visible. A summary boolean cannot, by itself, explain why a run ended without an achieved injection goal.

For agent prompt injection testing that informs a real deployment decision, an unresolvable run should be visible as unresolvable.

---

The shape of failure matters

A pass/fail score tells you whether something went wrong. It does not tell you whether anyone would have noticed.

The proposed rubric distinguishes four failure shapes.

Shape	Definition
Loud	The system refused, blocked, or sanitised, and made that visible. The event enters your telemetry.
Silent	The injected behaviour executed and the output reads as a normal, plausible completion. Nothing signals a problem.
Partial	The agent followed part of the injected instruction without completing it.
Indeterminate	The shape cannot be established from the trace.

A silent failure is one where the system produces an apparently valid response while following the injected instruction.

Silent failures are the operationally dangerous ones. A loud failure generates a log line and possibly an alert. A silent failure generates a plausible answer, a satisfied user, and no reason for anyone to look. If an agent exfiltrates a record and then produces a competent summary of your inbox, the summary is what your reviewer sees.

Attack success rate treats both identically. They are not remotely the same risk.

One possible approach is to flag refusal or warning language for manual review. Such a heuristic would need validation; no accuracy result is reported here.

---

Why attribution is difficult

When a payload does not get through, the interesting question is why. It is also the question most evaluations skip, because it is genuinely hard.

A blocked payload looks the same whether the block was engineered or accidental. The trace shows an attack that did not succeed. It does not show you which component is responsible.

A review should distinguish documented evidence from inference and leave attribution undetermined when the trace does not support a conclusion.

Two possible situations illustrate why attribution needs caution.

1. No configured defence: Consider model behaviour as one possible explanation, while checking other causes and avoiding unsupported framework attribution.
2. A presentation-layer defence: Record both model behaviour and the changed presentation as possible explanations; a single trace may not separate their effects.

A proposed attribution should record competing explanations and the evidence supporting its confidence level.

The most common error in this area is reading a model’s refusal as a framework security feature. It is an easy mistake and it leads directly to deploying the same architecture on a different model and being surprised.

---

What the source discussion establishes—and leaves open

Three things, held at appropriate strength.

First, verified. AgentDojo 0.1.35 scores three classes of infrastructure error as defensive successes, and does not verify payload delivery. This is directly checkable in the published source.

Whether a different rubric changes any reported score remains unmeasured here.

Whether stricter scoring changes any published conclusion is unknown. Evaluating that question would require a reproducible method, preserved traces and documented results.

What this does not tell us is anything about which frameworks resist prompt injection. That evaluation has not been run. The original question stands open.

---

Limitations

The proposed methodology has not been validated.

• No models were run. This is source analysis. Every claim is about code, not behaviour.
• One framework, one version. AgentDojo 0.1.35 only. Whether other agent evaluation tools share this property is untested, though the design pressure that produces it is common.
• Version-specific: The source discussion concerns AgentDojo 0.1.35. Re-check the implementation before applying it to another version.
• The proposed rubric is unvalidated. Its effect on scores has not been measured.
• Proposed lexical detection needs validation. It may miss warnings or incorrectly classify compliant answers.
• Template attacks measure a floor. Work on adaptive attacks shows defences evaluated against fixed templates degrade badly under attacks adapted to them. Any result from this class of tool is a lower bound on what a motivated attacker achieves.
• Passing proves nothing. A system that survives this evaluation is not secure. It survived these payloads, at these versions, on this date, under this configuration. Failing one test does not mean a framework is broadly insecure, either.

---

How to investigate this question

The source reading needs nothing but the package.

bash

pip download agentdojo==0.1.35 --no-deps



unzip agentdojo-0.1.35-py3-none-any.whl -d src

Then open src/agentdojo/benchmark.py and search for security = True. The three exception paths are in the function that runs a single task against a pipeline. The attack generators are in src/agentdojo/attacks/, and the defence names sit in src/agentdojo/agent_pipeline/agent_pipeline.py.

A future investigation could generate traces while holding the model fixed and varying one defence at a time. The following is prospective guidance, not a record of a completed experiment:

bash

python -m agentdojo.scripts.benchmark \



  --model <exact-dated-model-string> \



  --defense tool_filter \



  --attack important_instructions

Record the generated trace paths and preserve the logs alongside the exact evaluation configuration.

Two things to record and publish: the exact dated model string, never “latest”, and your temperature. Leaving temperature at the provider default is a defensible choice, but it is a choice, and it belongs in your methodology.

Set runs per cell above one. A single run cannot distinguish a defence from a coin flip, and these are stochastic systems. Three is a floor. Five is better. Report cells that produce different outcomes across repeats as variable rather than averaging them, because the variance is the finding.

All of this runs against AgentDojo’s simulated in-process environments. No third-party system is involved, and nothing here should be pointed at infrastructure you do not own.

---

References

Source materials: AgentDojo 0.1.35 and its published source. No historical UniverseBlend execution or re-scoring record is asserted.

Official documentation: AgentDojo repository · AgentDojo documentation

Academic research: AgentDojo, arXiv:2406.13352 · AgentDyn, arXiv:2602.03117 · Adaptive attacks against indirect prompt injection defences, arXiv:2503.00061 · Meta SecAlign, arXiv:2507.02735

---

Frequently Asked Questions

---

Keep reading

Here are the latest posts from the blog.