ai agents | UniverseBlend

Context Engineering: Cost, Latency and Security in AI Agents

An agent asked to fix a failing test may make twenty model calls before it answers. The first call carries a system prompt, a few thousand tokens of tool definitions and the user’s request. By the tenth call it is also carrying three file reads, two test runs, a stack trace, a search result and the model’s own earlier reasoning. None of that was written as a prompt, yet all of it shapes the next decision. Context engineering is the work of deciding what goes into each of those calls. As AI applications become stateful, tool-using systems, the wording of the prompt becomes one part of a larger problem: managing context for quality, cost, latency and security all at once.

That last part is the argument of this piece. Most explanations of context engineering treat it as a quality discipline, about getting the right information to the model. It is that, but the same decisions also set the input bill, the time to first token and the set of instructions an attacker can reach. A choice made for one of those reasons changes the other three.

Key Takeaways

Context engineering decides the exact tokens in each model call. Prompt wording is one part of it, and in agents often a small one.
Agent loops are input-heavy. In the worked example, input was over 90% of the uncached cost, and good cache layout cut the run by about 78%.
Prompt order is a cost decision. A volatile value at the top of the prompt can make caching more expensive than not caching.
Long context mainly lengthens prefill and time to first token. The size of the effect depends on hardware, serving and cache hits.
Everything in context can steer the model. Treat retrieved text, tool outputs and third-party tool descriptions as untrusted, and enforce permissions outside the model.
Every context rule should have a test: task success, tokens, cache-read share, latency and injection resistance.

Quick Navigation

The shift from prompt engineering to context engineering
The context budget: what actually enters a model call?
The cost of context engineering
The latency tax of context engineering
Context engineering is also a security boundary
A context engineering architecture that can be tested
When more context is the wrong answer
References and further reading
Frequently Asked Questions

The shift from prompt engineering to context engineering

The term is newer than the practice. Anthropic’s Applied AI team, in a September 2025 engineering post titled Effective context engineering for AI agents, described it as “the natural progression of prompt engineering”. It defined context engineering as “the set of strategies for curating and maintaining the optimal set of tokens” during inference. That is a useful working definition, but it is a practitioner’s framing, not a standard. Retrieval-augmented generation, conversation summarisation and tool design all existed before anyone grouped them under one name.

So “replaced” overstates it. Prompt design did not go away; it became one input among several. A well-written system prompt still matters, but in an agent it may be only a few percent of the tokens in a late-stage call. The rest arrives from places the prompt author did not write and often never sees.

Discipline	Main question	Typical artefacts
Prompt engineering	How should the instructions be worded?	System prompt, few-shot examples, output format
Retrieval (RAG)	Which external documents are relevant to this query?	Index, retriever, reranker, chunks
Memory	What from earlier sessions or turns should persist?	Notes, summaries, user profiles, vector stores
Orchestration	Which call happens next, and with which tools?	Agent loop, planner, sub-agents, tool router
Context engineering	What exact set of tokens enters this call, in what order, and with what trust level?	Context assembler, budgets, compaction rules, cache layout

Context engineering sits across the other four. Retrieval decides what might be relevant. Memory decides what might be kept. Context engineering decides what is actually sent, and that decision is made again for every call.

The context budget: what actually enters a model call?

Three numbers are often confused. The context window is the maximum the model will accept. The context budget is what an application chooses to spend, given cost and latency targets. The task requirement is what the call actually needs to succeed. In a healthy system the third is well below the second, and the second well below the first. UniverseBlend’s LLM context window comparison makes the same point from the other side: a model accepting a million tokens “says nothing about how well it uses them”.

A single call in an agent loop typically assembles:

System instructions: stable, written by the developer, high trust.
Tool definitions: stable within a session. They are often larger than people expect, because every tool brings a name, a description and a JSON schema.
Conversation history: grows every turn.
Retrieved material: documents, search results and file contents, chosen per query.
Tool outputs: API responses, logs and test results, frequently verbose and rarely edited.
Summaries and notes: compressed forms of earlier work, written by the model itself.

These components differ in how often they change, how much they cost, and how far they can be trusted. That is why context assembly should be task-specific. A call that decides which file to open needs the file list, not the contents of last week’s design document. A call that writes the final answer may need the reverse.

Anthropic’s post recommends several patterns for keeping the budget under control:

Compaction: summarising a conversation as it approaches the limit and starting a fresh window.
Tool-result clearing: dropping raw outputs once they have been used.
Structured note-taking outside the window.
Sub-agents: each works in a clean context and returns a condensed summary.
“Just-in-time” retrieval: the agent keeps lightweight identifiers, such as file paths or queries, and loads content only when needed.

Each of these is a decision about what survives into the next call.

The cost of context engineering

Agent loops are input-heavy in a way chat is not, because each call re-sends the accumulated transcript. A worked example makes the size of this visible.

Hypothetical workload. Rates are Anthropic’s published Claude Opus 5.5 prices, checked on 22 October 2026: $4 per million input tokens, $5 per million for 5-minute cache writes, $0.20 per million for cache reads, and $20 per million output tokens.

The agent makes 20 calls. Each carries a 12,000-token stable prefix (system prompt and tool definitions). The transcript grows by 1,500 tokens per call (tool output plus the model’s reply). Each call generates 500 output tokens. Across the run, that is 525,000 input tokens and 10,000 output tokens, and the last call alone sends 40,500 input tokens.

Strategy	What is billed	Total
No caching	525,000 input × $4 + 10,000 output × $20	$2.30
Cache the stable prefix only	12,000 written once, 228,000 read, 285,000 uncached history, output	$1.45
Cache the growing prefix each turn	40,500 written, 484,500 read, output	$0.50
Prefix broken on every call (e.g. a timestamp at the top)	525,000 written at $5, output	$2.83

Three things stand out. Input is $2.10 of the $2.30 uncached total, so here context, not generation, is the bill. Caching the whole growing prefix cuts the run cost by about 78%. And a single volatile value placed early in the prompt makes caching cost more than not caching, because every call pays the write premium and none of them get a read.

That last row is the context-engineering lesson. Anthropic’s prompt-caching documentation says cache hits “require 100 percent identical prompt segments” and that prefixes are built in the order tools, system, then messages. OpenAI’s prompt-caching guide tells developers to keep stable content first and dynamic content last, and to move timestamps and user-specific values later. On both platforms, cache economics depend on where things sit in the prompt, not just on how much there is.

Caching also has limits worth stating. Anthropic’s default cache lives five minutes (refreshed on use), so a tool call that runs longer than that lets the prefix expire. Opus 5.5 will not cache prompts under 512 tokens, and OpenAI’s minimum on GPT-5.6 and later is 1,024. On Anthropic’s API, changing a tool definition invalidates the entire cache. The worked example assumes none of these interruptions happen, and a real run will do worse.

The latency tax of context engineering

Inference has two phases. In prefill, the model processes every input token, largely in parallel, and builds the key-value (KV) cache that generation will read from. In decode, it produces output one token at a time. The Splitwise paper from Microsoft researchers calls these “a compute-intensive prompt computation” and “a memory-intensive token generation”. The DistServe authors tie time to first token (TTFT) to prefill, and time per output token to decode.

Long context therefore lands mainly on TTFT. A 40,500-token prompt has far more prefill work than a 12,000-token one, and that work happens before the user sees anything. The relationship is not a fixed per-token cost, though. It depends on the attention implementation, the hardware, how the provider batches requests, queueing at the moment of the call and, above all, cache hits. Both major providers say caching reduces time spent processing input. Anthropic says users “generally see improved time-to-first-token for long documents”. Neither publishes a universal figure, and this article does not invent one. UniverseBlend’s LLM inference latency measurements show how far TTFT varies by provider even before prompt length is considered.

For agents, the model’s prefill time is only part of the wait. End-to-end latency adds network round trips, retrieval queries, tool execution and orchestration logic, multiplied by the number of steps. Trimming context helps the inference share of that. It does nothing for a slow database query or a test suite that takes four minutes. A latency budget for an agent should be measured per component before anyone decides to cut context to fix it.

Context engineering is also a security boundary

Everything in a model’s context can influence its next action. That is how the model is designed to work, and it is also the security problem. Greshake and colleagues’ 2023 paper on indirect prompt injection put it plainly: “LLM-Integrated Applications blur the line between data and instructions”. An attacker who cannot reach the prompt can still plant text in a web page, a document or a code comment that the application is likely to retrieve.

OWASP’s LLM01:2025 Prompt Injection entry separates direct injection from indirect injection, “when an LLM accepts input from external sources, such as websites or files”. It also notes that RAG and fine-tuning “do not fully mitigate” the risk. Agents widen the set of sources: retrieved documents, tool outputs that talk back, earlier conversation turns, and tool descriptions themselves. The Model Context Protocol specification says “descriptions of tool behavior such as annotations should be considered untrusted, unless obtained from a trusted server”. It also states that “MCP itself cannot enforce these security principles at the protocol level.” UniverseBlend’s eight classes of prompt injection maps how each of these entry points behaves.

An illustrative scenario, not a reported incident: a support agent can read tickets and issue refunds through a tool. A customer submits a ticket whose body contains text formatted like a system note telling the assistant to refund the order in full and close the ticket. The ticket enters the context as retrieved material. Whether the model follows it depends on the model, the prompt and the formatting. Context engineering can lower the odds: mark the ticket body as untrusted data, keep it in a delimited block, and avoid putting it near the instructions. None of that is a guarantee. OpenAI’s research on an instruction hierarchy reports that training models to prioritise privileged instructions substantially improves robustness. That lowers the risk; it doesn’t remove it.

The control that actually holds sits outside the context. The refund tool should check, on the server, that the caller is authorised to refund that order, that the amount is under a limit, and that refunds above the limit need human approval. OWASP’s mitigations say the same: least privilege, functions handled in code rather than by the model, and “human-in-the-loop controls for privileged operations”. The distinction to keep is between what the model sees and what the application lets it do. Context filtering reduces exposure. Authorisation, input validation on tool arguments, sandboxing and approval gates decide the damage.

A context engineering architecture that can be tested

Treating context as an engineered component means every assembly rule should have a measurement attached. A practical architecture has five parts:

An assembler that builds each call from typed components (instructions, tools, history, retrieved items, tool outputs), each tagged with its source and trust level.
A budget per call type, set in tokens. Compaction or retrieval triggers when the budget is reached, not when the model’s limit is.
A cache layout: stable components first, in a fixed order, with volatile values pushed to the end.
A retrieval policy that prefers identifiers and on-demand loading over pre-stuffing.
An enforcement layer for tool calls, independent of anything in the context.

Decision	What to measure	How
What stays in the active context	Task success rate vs tokens per call	Run the same eval set with and without each component
What is retrieved on demand	Retrieval precision and recall; task success	Labelled queries with known relevant documents
What is summarised	Accuracy of facts carried across compaction	Probe questions about pre-compaction details
What is removed	Success rate and tokens after clearing tool outputs	Ablation on long-running tasks
What is cached	Cache-read share of input tokens; cost per task	Provider usage fields per request
What is treated as untrusted	Injection success rate	Seeded malicious documents and tool outputs in a red-team suite
What needs independent enforcement	Unauthorised actions attempted vs executed	Server-side logs of denied tool calls

The cache row is the easiest to start with, because both major providers report cached tokens in the response usage data. The security row is the one most often skipped. An agent that has never been tested against a poisoned document hasn’t shown that its context handling is safe.

When more context is the wrong answer

Long windows invite a simple strategy: include everything and let the model sort it out. The evidence says that is not free. The Lost in the Middle study (Liu et al., TACL) found that performance was “often highest when relevant information occurs at the beginning or end of the input context” and dropped when the relevant facts sat in the middle. Chroma’s Context Rot report, testing 18 models in July 2025, found that performance “grows increasingly unreliable as input length grows” and that “even a single distractor reduces performance”. The authors are clear that their evaluation is “not exhaustive of real-world use cases” and that they do not explain the mechanism. These are measured tendencies on specific tasks, not a law that every extra token makes every model worse.

In practice, more context goes wrong in a few recognisable ways:

Stale information (an earlier file version, a superseded plan) competes with current facts.
Duplicated evidence inflates the apparent weight of one source.
Conflicting sources force the model to arbitrate without being told which source is authoritative.
Summaries can be wrong. A compaction that drops a constraint will mislead every later call, and nothing in the context will flag it.
Unedited tool output buries a single error line in kilobytes of log.

A larger window still earns its place for some work: reasoning across a whole contract, comparing two long documents, or a codebase question whose answer is spread across many files. The case for it is strongest when the task genuinely needs the whole input and weakest when the window is used as a substitute for retrieval. Selective retrieval and a smaller context usually win on cost, latency and attack surface together. The burden of proof belongs on the larger context, and an eval on your own tasks is where that proof should come from.

References and further reading

Effective context engineering for AI agents. Anthropic Applied AI team (Rajasekaran, Dixon, Ryan, Hadfield et al.), 29 Sep 2025. https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents. Vendor engineering guidance: the definition, compaction, note-taking, sub-agents and just-in-time retrieval.
Lost in the Middle: How Language Models Use Long Contexts. Liu, Lin, Hewitt, Paranjape, Bevilacqua, Petroni, Liang. TACL; arXiv v1 6 Jul 2023. https://arxiv.org/abs/2307.03172. Research: position effects in long contexts.
Context Rot: How Increasing Input Tokens Impacts LLM Performance. Hong, Troynikov, Huber (Chroma), 14 Jul 2025. https://www.trychroma.com/research/context-rot. Vendor research report: reliability declines with input length; distractor effects; stated limitations.
Not what you’ve signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection. Greshake, Abdelnabi, Mishra, Endres, Holz, Fritz. arXiv, Feb 2023 (v2 May 2023). https://arxiv.org/abs/2302.12173. Research: indirect prompt injection.
LLM01:2025 Prompt Injection. OWASP Gen AI Security Project. https://genai.owasp.org/llmrisk/llm01-prompt-injection/. Industry guidance: direct vs indirect injection; RAG and fine-tuning don’t fully mitigate it; least privilege; human approval.
The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructions. Wallace, Xiao, Leike, Weng, Heidecke, Beutel (OpenAI), arXiv, 19 Apr 2024. https://arxiv.org/abs/2404.13208. Research: training-based robustness to injection.
Model Context Protocol Specification (schema 2026-07-28), Security and Trust & Safety section. https://modelcontextprotocol.io/specification/latest. Official spec: untrusted tool descriptions; the protocol cannot enforce its security principles.
Prompt caching. Anthropic (Claude API docs). https://platform.claude.com/docs/en/build-with-claude/prompt-caching. Official docs: exact-match requirement; tools → system → messages order; 5-minute default TTL; 512-token minimum; TTFT improvement.
Pricing. Anthropic (Claude API docs). https://platform.claude.com/docs/en/about-claude/pricing. Official docs: Opus 5.5 rates used in the worked example.
Prompt caching. OpenAI (API docs). https://developers.openai.com/api/docs/guides/prompt-caching. Official docs: static-first ordering; full-prefix match; 1,024-token minimum; “up to 95%” discount.
Splitwise: Efficient generative LLM inference using phase splitting. Patel, Choukse, Zhang, Shah, Goiri, Maleki, Bianchini (Microsoft), arXiv, 30 Nov 2023. https://arxiv.org/abs/2311.18677. Research: prefill is compute-intensive and decode memory-intensive.
DistServe: Disaggregating Prefill and Decoding for Goodput-optimized LLM Serving. Zhong et al., arXiv, 18 Jan 2024. https://arxiv.org/abs/2401.09670. Research: TTFT tied to prefill; TPOT tied to decode.

Frequently Asked Questions

What is context engineering in AI?

Context engineering is the practice of deciding exactly what information enters each model call: instructions, history, retrieved documents, tool definitions, tool outputs and summaries, along with their order and trust level. It matters most in multi-step agents, where the context is rebuilt for every call.

How is context engineering different from prompt engineering?

Prompt engineering focuses on how instructions are written. Context engineering includes that, but also manages everything else the model sees, much of which the developer never writes directly, such as search results and tool outputs.

Is RAG a form of context engineering?

RAG is one technique within it. Retrieval decides which documents might be relevant. Context engineering also decides how many of them to include, where to place them, whether to summarise them and how far to trust them.

How does context engineering reduce LLM costs?

It reduces costs by sending fewer unnecessary tokens and by ordering prompts so that stable content can be served from cache at a fraction of the input price. Savings depend on the provider’s cache rules, minimum lengths, cache lifetimes and write charges. Caching is not automatically cheaper.

How does context engineering improve AI agent security?

It limits and labels untrusted content, which reduces the chance that injected instructions steer the model. It does not prevent prompt injection on its own. Tool permissions, argument validation, sandboxing and human approval have to be enforced outside the model.

What tools or techniques are used for context engineering?

Common techniques include retrieval with reranking, compaction or summarisation, clearing tool results, structured notes stored outside the window, sub-agents with isolated contexts, cache-aware prompt ordering, and per-call token budgets. The tooling varies; the techniques are what carry over.

Keep reading

Here are the latest posts from the blog.

Claude API Rate Limits in 2026: How Many Requests Can You Run?

November 5, 2026

Claude API rate limits on the Start tier allow most current models 2,000,000 input tokens and 400,000 output tokens per minute, with 1,000 or 4,000 …

Enterprise AI Spending Hit $177B. Agents Are Driving the Bill

November 4, 2026

Figures checked against Menlo Ventures’ 2026 State of Enterprise AI report on 4 November 2026. Market sizes are Menlo estimates, shown as annualised run rates. …

Machine Unlearning: Can We Really Trust an Edited AI Model?

November 4, 2026

Checked against Hirundo’s and Lazarus AI’s published results, CrowdStrike’s DeepSeek-R1 study, Lawfare and the Federal Register on 4 November 2026. Ask Alibaba’s Qwen3.6-35B-A3B about Tiananmen …

30 Agentic AI Statistics for 2026: Adoption, Spending and Risk

November 3, 2026

Last updated: 3 November 2026. This page is updated monthly; each figure keeps its original publication date. Agent traffic at one healthcare booking platform rose …

What the Model Context Protocol Actually Is, and What It Opens Up

by Idrees Patel

Connecting a language model to a company’s issue tracker used to mean writing a small integration layer by hand. Someone had to describe each operation to the model, handle authentication, map the model’s chosen action onto the right API call and turn the response back into text. Then the same work started again for the next model or the next application, because none of it was portable. With five AI applications and twenty internal systems, that is up to a hundred bespoke connectors, each with its own bugs and its own way of handling credentials.

The Model Context Protocol (MCP) is an attempt to collapse that grid. It defines one way for an AI application to discover what an external system offers and to call it. Anthropic described the problem in its November 2024 announcement: “Every new data source requires its own custom implementation.” MCP makes the connection standard. It does not make what sits on the other end of the connection trustworthy, and much of what follows is about the difference between the two.

Key Takeaways

The Model Context Protocol standardises how AI applications discover and call external capabilities. It replaces per-application integration code, not security design.
Hosts, clients and servers have distinct roles. The host sits between model and server, which makes it the natural place for policy.
Listing a tool and executing it are separate steps. Visibility through tools/list grants nothing, and execution is a host decision.
Tool descriptions and tool outputs are both untrusted inputs to the model. The specification says so, and tool-poisoning research has demonstrated the consequences.
The 2026-07-28 revision made MCP stateless and deprecated sampling, roots and logging. Check which version a server implements before relying on older documentation.
MCP provides interoperability, OAuth-based authentication for HTTP servers and consent guidance. Least privilege, isolation, validation and logging must be built around it.

Quick Navigation

What the Model Context Protocol actually is
Model Context Protocol architecture: hosts, clients and servers
- Messages and transports
Tools, resources and prompts are different things
What the Model Context Protocol opens up in real applications
The trust boundary hiding inside the tool description
- Four layers, one of them yours
What the Model Context Protocol does not guarantee
- Who is responsible for what
Before connecting a Model Context Protocol server: a checklist
The engineering decision the Model Context Protocol leaves
FAQ

What the Model Context Protocol actually is

The Model Context Protocol is an open specification for how AI applications exchange context and capabilities with external programs. In the words of the current specification, it gives applications a standard way to “share contextual information with language models” and “expose tools and capabilities to AI systems.” Its designers say it takes inspiration from the Language Server Protocol, which let one language server work with many code editors.

Several things it is often confused with are worth separating:

MCP is not a model. It doesn’t make a model smarter or better at choosing tools. That still depends on the model and the application.
MCP is not an agent framework. The architecture documentation says plainly that it “does not dictate how AI applications use LLMs or manage the provided context.”
MCP is not an API in the usual sense. A REST API defines one service’s operations. MCP defines how any service can describe its operations to any compatible AI application.
An MCP server is not the protocol. A server is one program that speaks MCP, written by anyone, with whatever quality and permissions its author chose.

Anthropic released the protocol in November 2024, along with SDKs and a set of open-source servers. In December 2025 it donated MCP to the Agentic AI Foundation, a fund under the Linux Foundation co-founded with Block and OpenAI. The existing maintainers kept control of its technical direction. The specification changes regularly. The current revision, dated 28 July 2026, made substantial changes, covered below.

Model Context Protocol architecture: hosts, clients and servers

MCP has three participants, and their responsibilities differ in ways that matter later.

The host is the AI application the user works in, such as a desktop chat app, an IDE or an agent runtime. It owns the conversation, the model and the user interface.
A client is a connector inside the host. The host creates one client for each server it connects to, and each client keeps a dedicated connection to its server.
A server is a program that provides context and capabilities: a filesystem server, a database server, an issue tracker’s server.

The model never talks to a server directly. When a model wants to use a tool, the host receives that request, decides whether to allow it, and has the relevant client send the call. The architecture documentation describes this step: the host “intercepts the tool call, routes it to the appropriate MCP server, executes it, and returns the results back to the LLM.” That interception point is where an application can enforce policy. Whether it does is up to the application.

Messages and transports

Clients and servers exchange JSON-RPC 2.0 messages, a simple request-and-response format. MCP defines two standard transports:

stdio, where the host launches the server as a local process and talks to it over standard input and output;
Streamable HTTP, where the server runs remotely and the client sends HTTP POST requests, optionally receiving streamed responses.

A local filesystem server typically uses stdio. A hosted service, such as the remote server Sentry operates, uses Streamable HTTP.

The 2026-07-28 revision made the protocol stateless. Earlier versions opened each connection with an initialize handshake that negotiated capabilities once per session. Now every request carries its protocol version and the client’s capabilities in its own metadata. Clients can call a new server/discover method to learn what a server supports. Servers that need state across calls must hand out explicit identifiers as ordinary tool arguments. The same revision deprecated sampling, which let servers ask the host’s model for completions, along with roots and logging. Articles written before mid-2026 describe the older design, so check which version a server implements.

Tools, resources and prompts are different things

A server can offer three kinds of capability, and the MCP documentation is explicit about who is meant to control each.

Capability	What it is	Who typically controls it	Example
Tools	Functions the model can call; they may read data or change it	The model, with the host’s permission	`create_issue`, `run_query`, `send_message`
Resources	Read-only data identified by a URI	The application, which decides what to include as context	A file, a database schema, a calendar
Prompts	Reusable templates for a task	The user, who invokes them explicitly	A slash command such as `/summarise-incident`

The server concepts page uses this split: tools are model-controlled, resources application-controlled and prompts user-controlled. Servers declare which of these they support, and a client may use only some of them.

Discovery and execution are separate steps, and the gap between them matters. A client calls tools/list and receives each tool’s name, description and JSON Schema for its inputs. The host passes some or all of these to the model. Later, if the model asks for one, the client sends tools/call with arguments. Seeing a tool in a list grants nothing by itself. Execution is a second, distinct decision that the host makes.

What the Model Context Protocol opens up in real applications

Three patterns show what the protocol standardises and what it leaves to the application.

Querying documents or a database

A database server might expose its schema as a resource and a query function as a tool, and a filesystem server exposes file reading and searching. The official filesystem server is one of the project’s reference implementations. MCP standardises how the assistant finds those capabilities and calls them. The host still decides which directories or databases the server is configured for, and whether a query runs automatically or waits for approval. The integration needs read access to the data, and nothing more if it only answers questions.

Working with a code repository

GitHub’s official MCP server is a demonstrated case. It can run remotely, hosted by GitHub, or locally in Docker. It groups tools into toolsets, such as repositories, issues and pull requests. A --read-only flag skips write tools even when they’re explicitly requested. MCP standardises the interface, so a developer assistant in an IDE can list issues or open a pull request. The permissions come from the GitHub token or OAuth grant behind the server, and which toolsets are switched on is a deployment choice.

Retrieval combined with an action

This example is illustrative. An assistant reads a customer’s recent support tickets through one server, drafts a reply, and sends it through an email server. MCP lets both servers plug into the same host. Nothing in the protocol decides whether sending should happen without a person reviewing the draft. The integration needs read access to tickets and send access to email. That second permission turns a summarising assistant into one that acts on a customer.

The trust boundary hiding inside the tool description

A tool description is text written by the server’s author, and the model reads it to decide when and how to use the tool. That makes it an input to the model’s reasoning. If the author is careful and honest, the description helps the model choose correctly. If the author is careless, or the server is malicious, the description can push the model toward something the user never asked for.

The specification acknowledges this. It says descriptions of tool behaviour, including annotations, “should be considered untrusted, unless obtained from a trusted server.” In April 2025, Invariant Labs demonstrated “tool poisoning”. In its experiment, a tool presented as a simple add function carried hidden instructions in its description. Those instructions led the Cursor agent to read a configuration file and a private SSH key and pass them out through a tool argument. The confirmation dialog didn’t show the full arguments. Invariant also described “rug pulls”, where a server changes a description after the user has approved it. It showed a malicious server’s description altering how the agent used a different, trusted server’s email tool.

Tool outputs carry the same risk from the other direction. A tool that fetches a web page or reads an email returns text the model then reads, and that text may contain instructions. Our guide to the eight classes of prompt injection treats tool metadata and tool output as two of them.

Four layers, one of them yours

It helps to keep four layers distinct:

Metadata: what the server claims a tool does.
Model interpretation: what the model decides to do after reading it.
Execution: what the server actually does when called, with whatever credentials it holds.
Enforcement: what the host or surrounding infrastructure allows, regardless of the first three.

Only the fourth layer is under the deploying team’s direct control. Here is an illustrative example. A team connects a “project notes” server that also exposes a shell tool its author added for convenience, running with the developer’s own permissions. A poisoned page fetched by another tool now has a path to arbitrary commands. MCP carried every message correctly. The failure lay in what the host exposed and allowed.

The deeper threat model, covering token passthrough, cross-server shadowing, CVEs and supply-chain incidents, is in our analysis of where the model-to-tool boundary fails in MCP security. This article stays with what the protocol is.

What the Model Context Protocol does not guarantee

Several common assumptions don’t survive a reading of the specification.

Interoperability is not semantic compatibility. Two servers can both speak MCP and still describe similar operations differently. A host that connects both may present the model with two search tools. The specification tells clients aggregating many servers to handle name collisions, for example by prefixing tool names with a server identifier.
Authentication is not authorisation. For HTTP servers, the authorisation specification builds on OAuth 2.1. Servers must validate that tokens were issued specifically for them, and must not pass tokens through to other services. That proves who is calling. Which actions that caller may take inside the server is the server’s own access-control logic. Authorisation is also optional in the protocol, and stdio servers are told to take credentials from their environment instead.
A successful response is not a trustworthy result. A tool result marked complete means the server answered. It doesn’t mean the answer is correct, safe to show the model, or free of injected instructions. The specification asks clients to “validate tool results before passing to LLM”, but how is left to them.
Human approval is not a complete boundary. The specification says there “SHOULD always be a human in the loop with the ability to deny tool invocations.” Invariant’s experiment shows an approval prompt is only as good as what it displays. A user approving a call they can’t fully see isn’t exercising much control.

Who is responsible for what

The specification is candid about where its responsibility ends: “MCP itself cannot enforce these security principles at the protocol level.” The work divides roughly as follows:

Server implementers validate inputs, enforce access control, rate-limit and sanitise outputs.
Host developers build consent flows, display tool calls accurately, validate results and log usage.
Deployment owners choose which servers to trust, with what credentials, inside what isolation.
Users decide what to approve, which works only if they’re shown enough to decide.

Before connecting a Model Context Protocol server: a checklist

These are the checks that change outcomes. The MCP security best practices cover the authorisation-level attacks in more depth.

Identity and provenance. Know who publishes the server, pin its version, and record a hash of its tool descriptions so a changed description is noticed.
Least privilege. Enable only the tools the task needs. Use read-only modes where they exist, as GitHub’s server offers.
Credentials. Give each server its own scoped, short-lived credential. Never share a broad personal token across several servers.
Scope of tools and resources. Restrict directories, repositories and databases at the server’s configuration, not in the prompt.
Approval for consequential actions. Gate writes, sends, deletes and payments. Make sure the approval shows the full arguments.
Isolation. Run local servers with limited filesystem and network access. The MCP best practices recommend sandboxing local servers, which run with the client’s privileges. Our guide to the four layers of sandbox isolation covers the options.
Validation. Check tool arguments against the schema before calling. Treat returned content as untrusted input to the model.
Logging. Record every tool call, its arguments, the result and the approving user, with a correlation ID.
Updates and configuration review. Re-review a server when its tool list changes. Updates can add tools.
Adversarial testing. Feed the integration poisoned documents and malicious tool outputs before production. Confirm the enforcement layer holds when the model is misled.

None of these removes prompt injection. Together they limit what a misled model can reach.

The engineering decision the Model Context Protocol leaves

MCP solves a real problem well. Connecting a new tool to a compatible host is now configuration rather than a project, and the same server can serve an IDE, a chat app and an agent runtime. The 2026 move to a stateless design also makes remote servers easier to scale behind ordinary HTTP infrastructure.

What MCP doesn’t decide is whether a given server deserves the access it asks for. Every server you connect adds a party whose text your model reads and whose code runs with credentials you granted. The question to ask of each integration is the same one you would ask of any third-party code with production access: what can it reach, and what stops it when it’s wrong?

FAQ

What is the Model Context Protocol?

It’s an open protocol that standardises how AI applications connect to external tools and data. A host application uses MCP clients to discover what an MCP server offers (tools, resources and prompts) and to call it. It doesn’t define how the model reasons or which tools are safe.

How does an MCP client communicate with an MCP server?

Through JSON-RPC 2.0 messages, over stdio for local servers or Streamable HTTP for remote ones. Since the 2026-07-28 revision, each request carries its own protocol version and capabilities. There’s no session handshake, and a client can call server/discover to learn what a server supports.

What is the difference between MCP and an API?

An API defines one service’s operations. MCP defines a common way for many services to describe their operations to AI applications. An MCP server often wraps an existing API, translating it into tools and resources that any MCP-compatible host can discover and call.

Is MCP secure?

The protocol includes OAuth-based authorisation for HTTP servers and security requirements for implementers. But the specification states that MCP “cannot enforce these security principles at the protocol level.” Security depends on which servers you trust, the permissions they hold, and the controls your host and infrastructure enforce.

When should a team use MCP?

When several AI applications need the same external capabilities, or when you want to use servers others maintain. For one application calling one internal API, a direct integration may be simpler. Either way, the same permission and isolation decisions apply.

Keep reading

Here are the latest posts from the blog.

Claude API Rate Limits in 2026: How Many Requests Can You Run?

November 5, 2026

Claude API rate limits on the Start tier allow most current models 2,000,000 input tokens and 400,000 output tokens per minute, with 1,000 or 4,000 …

Enterprise AI Spending Hit $177B. Agents Are Driving the Bill

November 4, 2026

Figures checked against Menlo Ventures’ 2026 State of Enterprise AI report on 4 November 2026. Market sizes are Menlo estimates, shown as annualised run rates. …

Machine Unlearning: Can We Really Trust an Edited AI Model?

November 4, 2026

30 Agentic AI Statistics for 2026: Adoption, Spending and Risk

November 3, 2026

Last updated: 3 November 2026. This page is updated monthly; each figure keeps its original publication date. Agent traffic at one healthcare booking platform rose …

AI Agent vs Copilot: Classify AI by Its Approval Model

by Idrees Patel

Every AI product that does useful work runs some version of a decide–act–verify loop: something chooses the next step, something carries it out, and something checks that it worked. The difference between a chatbot, an assistant, a copilot and an agent is mostly who controls that loop, and above all who has to approve the step that changes something real.

Take a support system that drafts a refund reply. A person reads the draft, edits it and clicks send. Now take one that sends the reply, updates the CRM and schedules a follow-up without asking again. Both could run on the same model with the same prompt. The second has been handed the “act” and “verify” parts of the loop, and that is the real AI agent vs copilot question. The label on the box tells you much less than the approval model behind it.

Key Takeaways

Chatbot, assistant, copilot and agent describe how control is allocated, not distinct technologies. One product can occupy several positions.
In the AI agent vs copilot distinction, the decisive question is who approves consequential actions.
Reasoning, tool access, execution authority and verification are separate capabilities. Evaluate each one.
Removing approval moves risk from the model’s output to its permissions, identity, network reach and logs.
Human approval works only when the reviewer can understand the specific action. Generic prompts get rubber-stamped.
Agent runs cost more in tokens but can cost less per task when they remove routine human work. The deciding factors are completion rate and recovery time.
Grant autonomy by impact, reversibility and verifiability, not by product label.

Quick Navigation

Four labels, one operational question
Where each system actually sits
AI agent vs copilot: the transition is an engineering decision
- What an agent's permissions reach
- Why approval is only a partial control
AI agent vs copilot costs: why a run isn't a reply
The buyer's test: classify an AI agent vs copilot by its approval model
A deployment rule that survives the marketing
Conclusion
Frequently Asked Questions

Four labels, one operational question

The terms are confusing because vendors use them for products that can share the same model, the same retrieval system and the same tool-calling interface. What differs is configuration. That covers which tools are switched on, which identity they run under, which actions pause for approval, and whether the system keeps working after the user stops typing.

Five questions separate the categories better than any definition:

Who interprets the goal?
Who decides the next step?
Who can invoke tools or change external state?
Who approves consequential actions?
Who checks whether the intended result actually happened?

Those questions also separate four things that are often blurred together: reasoning, tool access, execution authority and verification. A model can reason well and have no tools. It can have read-only tools and no authority to change anything. It can have write access and still have every write held for approval. Each of those is a different system, whatever it’s called.

Category	Typical behaviour	Who chooses the next step	Human approval model	External actions	Main operational concern
Chatbot	Answers within one conversation	The user	Not needed; nothing executes	None, or scripted handoffs	Wrong or harmful answers
Assistant	Retrieves information, performs bounded tasks	Mostly the user	Per task, often implicit in the request	Usually read-only or low-impact	Data access and leakage
Copilot	Proposes or prepares work inside a tool the user is working in	The user, with suggestions	The user accepts or rejects each change	Prepared, executed after acceptance	Over-trust in plausible suggestions
Agent	Pursues a goal across several decisions and tool calls	The system, within limits	Selective: some actions gated, others not	Yes, under its own identity	Permissions, containment, cost

These are working categories, not a standard taxonomy. A single product can sit in two rows depending on how it’s configured.

Where each system actually sits

Product names make poor evidence, and some of the clearest examples come from one vendor using one name for several positions on the spectrum.

GitHub Copilot: two rows in one product

GitHub Copilot spans the copilot and agent rows. In the editor, it suggests code that a developer accepts or ignores, which is the classic copilot pattern. Its cloud agent takes a task, works on it in the background and produces a pull request. GitHub’s documentation on reviewing Copilot’s pull requests shows where the human control sits. If a repository requires pull-request approvals, “your approval” of a Copilot pull request doesn’t count toward them, so another reviewer has to approve. GitHub Actions workflows don’t run on Copilot’s changes by default either; a person clicks “Approve and run workflows”. GitHub notes that workflows can be privileged and reach secrets. The agent decides the steps inside the task, but merging and privileged execution stay with people unless an administrator changes the default.

Claude Code: autonomy set by configuration

Claude Code shows one product moving along the spectrum by setting. Anthropic’s permissions documentation describes a tiered system. In the default manual mode, file reads in the working directory run without approval, while shell commands, file edits and web fetches ask first. Other modes shift that boundary. “acceptEdits” auto-approves file edits in the working directory. “Plan” restricts the tool to reading and exploring. “bypassPermissions” skips prompts, and Anthropic says to use it only “in isolated environments like containers or VMs where Claude Code can’t cause damage”. The model is the same in each case, but the delegated authority is very different.

Copilot Studio: agents under a copilot brand

Microsoft’s Copilot Studio builds agents, despite the name. Microsoft described autonomous agents in Copilot Studio in October 2024. They can “react to events or triggers without human input”, with oversight through activity logs reviewed afterwards. A Microsoft 365 roadmap item published in September 2026 adds a per-tool toggle that pauses gated tool calls for a person to approve, approve for the session, or deny. It was listed as in development at the time. The “copilot” brand covers everything from a chat helper to an event-driven agent.

Why product labels date quickly

Products also move or disappear. OpenAI’s help page for ChatGPT agent now says the feature “is no longer available” and points users elsewhere. When it ran, OpenAI described it as pausing for “user confirmations for high-impact actions”. Any classification of a named product is a snapshot, which is one more reason to classify by approval model instead.

The chatbot row is the easiest to fill. A chat model with no tools enabled is a chatbot, whatever else the same product can do once integrations are switched on.

AI agent vs copilot: the transition is an engineering decision

Removing a mandatory approval step changes the risk profile more than any model upgrade does. In a copilot, the model’s output is a proposal, and the risk is that a person accepts a bad one. In an agent, the output becomes an action, so the risk now depends on what that action can reach.

Consider the same capability twice. A model writes a SQL statement to correct a customer’s address. As a copilot, it shows the statement, and a developer reads it, notices it lacks a WHERE clause and fixes it. As an agent with write access to the database, it runs the statement. The model’s skill is identical; the outcome is not. OWASP’s guidance on Excessive Agency traces this class of harm to three causes: excessive functionality, excessive permissions and excessive autonomy. Notice that none of them is about the model’s output.

What an agent’s permissions reach

Granting an agent control means deciding, explicitly, about each of these:

Credentials and identity. The agent acts as someone. A shared service account makes its actions unattributable and its reach as wide as the account’s. Our article on why AI agents shouldn’t share a service account sets out the alternative.
Data, APIs and services. Every connected system is part of the blast radius, the set of things a misdirected agent can affect.
Network destinations. An agent that can reach arbitrary hosts can send data anywhere, including wherever an injected instruction points it.
Write and irreversible operations. Deletes, payments, permission changes and outbound messages are where review matters most.
Logs and traces. Without a record of inputs, tool calls and decisions, nobody can reconstruct what happened.
Retry loops and delegation. An agent that retries or spawns sub-tasks can repeat a mistake faster than anyone notices.

Prompt injection is why the model can’t police this itself. Instructions hidden in a web page, email or document can steer an agent. The UK’s National Cyber Security Centre wrote in December 2025 that current models “do not enforce a security boundary between instructions and data inside a prompt”. The controls that hold are the ones outside the model, such as scoped identity, least privilege, egress limits and isolated execution. Our guide to the four layers of sandbox isolation covers the execution boundary in detail.

Why approval is only a partial control

Human approval is a real control, but only under conditions. The reviewer has to see what the action is, what it touches and what permission it uses. An approval prompt that says “Allow tool call?” over a wall of JSON trains people to click yes. Claude Code’s documentation shows one response to that trade-off: in its “auto” mode, a classifier reviews actions instead of the user. Approvals should be reserved for actions where a person can genuinely judge the consequence.

AI agent vs copilot costs: why a run isn’t a reply

A copilot interaction and an agent run aren’t comparable units. A copilot typically makes one or a few model calls and stops. An agent plans, calls tools, reads results, retries and continues until it reaches a stopping condition. Each call usually re-sends the growing context. Anthropic reported that in its own multi-agent research system, agents used about four times as many tokens as chat interactions, and multi-agent systems about fifteen times as many. That’s one company’s product, not a universal ratio, but it gives a sense of scale.

Tokens are rarely the whole bill, though. Here is a hypothetical comparison. The prices are illustrative, $3 per million input tokens and $15 per million output, not any provider’s list price:

Per task (hypothetical)	Copilot	Agent
Model calls	1	10, with growing context
Tokens	5,000 in, 800 out	126,000 in, 3,500 out
Inference cost	$0.03	$0.43
Human time	3 minutes reviewing every draft, at $60/hour: $3.00	10 minutes on the 20% of runs that fail: $2.00 average
Total	about $3.03	about $2.43

On these assumptions, the agent costs about sixteen times more in tokens but less per task, because it removes routine review. Change the failure rate to 40% and the agent’s average human cost doubles to $4. Change the task so that no review was needed anyway, and the copilot wins easily. The comparison turns on completion rate and human recovery time, not token price. Our analysis of when adding an agent stops paying for itself works through the token side in more detail.

Other costs also scale with autonomy:

tool and infrastructure charges for each call;
latency, because sequential steps add up;
rate limits, which fan-out makes easier to hit;
logging and evaluation, which an agent needs and a copilot often doesn’t;
runs that burn tokens before failing, with no useful result.

Agents don’t always cost more. A bounded agent that reliably finishes a task can be the cheaper option. A poorly constrained one can loop through tokens and still hand a person a mess to clean up.

The buyer’s test: classify an AI agent vs copilot by its approval model

When a vendor says “agent”, these twelve questions show what’s actually being sold. They’re grouped by what they reveal.

Decision and execution

Does the system only recommend, or can it execute actions?
Who selects the next step: the user, the system, or a fixed workflow?
Can it modify or delete external state?

Approval and authority

Which actions require explicit approval?
Can approval rules be set by action type, resource or risk level?
What identity does it act under, and can that identity be scoped per task?
Which data sources and network destinations can it reach?

Failure and verification

What happens when a tool fails or returns something unexpected?
Can it verify that an action actually succeeded?
What limits apply to retries, run time, tokens and tool calls?

Oversight and proof

Can operators stop a run, inspect logs and reconstruct what happened?
Will the vendor demonstrate these boundaries in your test environment, not just a slide?

Interpret the answers together. A product with broad write access, a shared identity and thin logging is high-risk, even if its interface is a friendly chat window. One that executes only pre-approved, reversible actions under a scoped identity can be low-risk even when it’s called an agent. Passing the checklist doesn’t make a deployment secure. It shows where the remaining risk sits.

A deployment rule that survives the marketing

Grant autonomy in proportion to how easily a mistake can be contained. Five factors set the level:

Data sensitivity: what the system can read, and where it could send it.
Action impact: what changes if the action is wrong.
Reversibility: whether the change can be undone cleanly and quickly.
Verification strength: whether success can be checked automatically and reliably.
Containment: whether a failure stays inside a sandbox, a test environment or a single account.

Low-impact, reversible, well-verified tasks can run with little or no approval. Drafting internal notes, triaging tickets into queues and running tests in a disposable environment are examples. Financial transfers, production deployments, access-control changes and decisions that affect people’s rights belong at the other end. For those, keep a person in the loop with a clear, specific approval, whatever the product is called.

The same tool can sit at both ends. A coding agent that runs tests in a container can be largely autonomous, while the same agent pushing to the main branch should wait for review. The approval model belongs to the deployment, not the product category.

Conclusion

Before adopting any AI product, write down three things. What can it do without asking? What can it change? Where can a person step in? The answers place the product on the spectrum more accurately than its name does, and they tell you which controls to build before it goes live.

For the AI agent vs copilot decision specifically, start with the copilot pattern for anything irreversible. Grant agent-level autonomy one action type at a time, as verification and containment prove they can catch the failures.

Frequently Asked Questions

What is the difference between an AI agent and a copilot?

A copilot proposes or prepares work, and a person accepts it before anything consequential happens. An agent pursues a goal across several steps, choosing actions and calling tools itself, with approval required only for some actions. Both can use the same model. The difference is who controls execution.

Is an AI assistant the same as an AI agent?

Not usually. An assistant typically handles bounded tasks on request, such as finding information or completing a single action, and the user directs each step. An agent decides the sequence itself. Some products called assistants can be configured to act as agents, so check the permissions.

Can a chatbot perform actions?

A pure chatbot only responds in conversation. Once it can call tools, such as booking, refunding or updating a record, it has moved toward the assistant or agent end of the spectrum. Many customer-facing “chatbots” now include such tools, often behind scripted handoffs.

Is every AI copilot less autonomous than an AI agent?

No. The label doesn’t fix the behaviour. Microsoft’s Copilot Studio builds event-triggered agents, and GitHub Copilot includes a cloud agent that works through tasks in the background. Judge autonomy by who chooses the next step and which actions need approval.

Why can AI agents be more expensive to operate?

An agent makes many model calls per task, each re-sending growing context, and adds tool, logging and retry costs. Failed runs consume resources without producing a result. An agent can still be cheaper per completed task if it reliably replaces human work.

Are AI agents more dangerous than chatbots?

They carry different risks. A chatbot’s failure is a bad answer that someone may act on. An agent’s failure can be an action, such as a deleted record or data sent to the wrong place. The danger scales with the permissions and network access it’s given and how well they’re contained.

Keep reading

Here are the latest posts from the blog.

Claude API Rate Limits in 2026: How Many Requests Can You Run?

November 5, 2026

Claude API rate limits on the Start tier allow most current models 2,000,000 input tokens and 400,000 output tokens per minute, with 1,000 or 4,000 …

Enterprise AI Spending Hit $177B. Agents Are Driving the Bill

November 4, 2026

Figures checked against Menlo Ventures’ 2026 State of Enterprise AI report on 4 November 2026. Market sizes are Menlo estimates, shown as annualised run rates. …

Machine Unlearning: Can We Really Trust an Edited AI Model?

November 4, 2026

30 Agentic AI Statistics for 2026: Adoption, Spending and Risk

November 3, 2026

Last updated: 3 November 2026. This page is updated monthly; each figure keeps its original publication date. Agent traffic at one healthcare booking platform rose …

Agentic AI vs Generative AI: Cost, Risk and Control

by Idrees Patel

Generative AI produces an output, such as text, code or an image, from a prompt and its context. Agentic AI is a system that uses a generative model to pursue a goal over several steps, calling tools and acting on the results until it finishes or is stopped.

So agentic AI vs generative AI isn’t a contest between two kinds of model. Most agents run on the same large language models (LLMs) as chat assistants. What changes is the system around the model: how many calls a task makes, what those calls may touch, what they cost together and what happens when one goes wrong. The sections below work through each difference with calculations you can rerun on your own numbers.

Key Takeaways

They aren’t rival model types. An agent is an application architecture that usually calls a generative model many times.
Repeated context dominates many agent bills. In our illustrative ten-call loop, 84% of input tokens were repeats.
Token price isn’t task cost. In a hypothetical comparison, a model at one-third of the token price cost about 38% more per completed task.
Caching helps only when it hits. Cache writes cost more than ordinary input, so a loop that keeps missing the cache pays extra.
Actions outlive answers. Validation, idempotency and approval gates belong outside the model.
Instructions aren’t permissions. Prompt injection is a risk to contain with system controls; filtering alone doesn’t remove it.
Known steps call for a pipeline. If you can write the workflow down in advance, fixed code with model calls at set points is usually cheaper and easier to test.

Quick Navigation

Agentic AI vs generative AI: the architectural difference
Where the tokens go in agentic AI vs generative AI
- A worked example with illustrative assumptions
Agentic AI vs generative AI: the cost of completing a task
- What caching changes
Agentic AI vs generative AI when a step fails
Agentic AI vs generative AI: security is an execution problem
- Where the controls belong
Agentic AI vs generative AI under rate limits
- An illustrative fan-out
When a pipeline beats an agent
A practical decision table
Conclusion
FAQ

Agentic AI vs generative AI: the architectural difference

Generative AI describes a capability. Agentic AI describes how a system uses it. The two overlap, because an agent’s plans, tool choices and final answer usually come from a generative model.

Anthropic’s guide Building effective agents (December 2024) draws a useful line. Workflows are systems where “LLMs and tools are orchestrated through predefined code paths.” Agents are systems where “LLMs dynamically direct their own processes and tool usage.” The practical question is who decides the next step: your code or the model.

Dimension	Single generative call	Agentic system
Unit of work	One request, one response	A task spanning many calls
Who picks the next step	The user or calling code	The model, within set limits
Calls per task	One	Variable, often unknown in advance
Side effects	None unless someone acts on the output	Possible at every tool call
Main cost driver	Prompt and output length	Calls × context per call
Typical failure	A wrong output	A wrong action, or a loop that never ends

The boundary isn’t absolute. Someone can paste generated SQL straight into production, and an agent can be limited to writing a report. The difference is structural: in an agent, the gap between the model’s decision and its effect is a function call, not a person.

Where the tokens go in agentic AI vs generative AI

A single request is easy to cost. An agent loop is harder, because each call re-sends material the model has already seen.

Every call carries the system instructions and tool definitions. OpenAI’s function-calling guide says function definitions “are billed as input tokens.” Anthropic’s tool-use documentation also adds a tool-use system prompt of 286 or 406 tokens on its 5.5-generation models, as listed on 18 October 2026. On top of that come the history, each earlier tool call and result, and the newest observation. Reasoning tokens, where a model produces them, are billed as output.

A worked example with illustrative assumptions

These are illustrative assumptions, not measurements of any product.

The system instructions take 1,500 tokens, the tool definitions 2,500 and the user task 500.
The one-shot request has no tools but includes a 3,000-token document, and it returns 800 tokens.
Each agent step outputs 300 tokens and receives a 1,500-token tool result. The final answer is 800 tokens.
The agent carries its full transcript forward.
Prices are $3 per million input tokens and $15 per million output tokens. These are round illustrative rates, not a current list price.

Call k therefore sends 4,500 tokens of prefix and task, plus 1,800 for every earlier step:

Input for call k        = 4,500 + (k − 1) × 1,800
Total input over n calls = 4,500n + 900n(n − 1)

Scenario	Calls	Input tokens	Output tokens	Illustrative cost
One-shot generation	1	5,000	800	$0.027
Agent, five calls	5	40,500	2,000	$0.152
Agent, ten calls	10	126,000	3,500	$0.431

Doubling the calls from five to ten more than tripled the input. Of the ten-call run’s 126,000 input tokens, only 20,700 were new; the other 84% were repeats.

That growth comes from the design, not from agents in general. A state-managed version might send the prefix, the task, a 500-token summary and only the latest observation. Its calls then stay at 6,500 tokens after the first, for 63,000 in total, half the transcript version. The catch is that a summary can drop a detail a later step needed.

For real-world scale, Anthropic reported that agents in its multi-agent research system (June 2025) “typically use about 4× more tokens than chat interactions,” and multi-agent systems “about 15× more.” That’s one company measuring one product.

Agentic AI vs generative AI: the cost of completing a task

Price per million tokens tells you what a call costs, not what finished work costs, because failed attempts are paid for too.

Cost per successful task = (inference + tools + infrastructure + retries)
                           ÷ successfully completed tasks

Take a hypothetical pair using the same token assumptions, plus $0.02 per attempt for tool APIs and sandbox compute. Model A costs $1 input and $5 output per million tokens, averages 18 calls and completes 60% of tasks. Model B costs $3 and $15, averages 10 calls and completes 92%.

Per 1,000 attempts	Model A	Model B
Input tokens per attempt	356,400	126,000
Total cost	$405.90	$450.50
Successful tasks	600	920
Cost per successful task	$0.677	$0.490

Model A charges a third of the token price yet costs about 38% more per completed task. Longer loops absorb most of its price advantage, and the lower completion rate does the rest. The table also ignores the 400 failed tasks, and if a person picks those up, the labour can outweigh the inference bill. Our breakdown of the three numbers inside inference cost per token explains how list prices become the rate you actually pay.

What caching changes

Anthropic’s prompt-caching documentation, checked on 18 October 2026, prices five-minute cache writes at 1.25× the base input rate and reads at 0.1× for many models. The cache lasts five minutes by default, refreshed on each use, and changing tool definitions invalidates all of it. OpenAI’s caching guide describes automatic caching with a 1,024-token minimum and, for GPT-5.6 and later, the same 1.25× and 0.1× multipliers.

Apply those multipliers to the ten-call example, with each call reading the previous prefix and writing only its new tokens. The task falls from $0.431 to $0.162, about 62% less. If every step instead waits past the cache lifetime, perhaps for human approval, each call writes and none reads. The task then costs $0.525, about 22% more than with no caching. Measure your hit rate before counting on the discount.

Agentic AI vs generative AI when a step fails

A wrong answer and a wrong action differ mainly in what has already happened by the time anyone notices.

A generated output that’s wrong	An agent action that’s wrong
A summary misstates a contract term	The agent writes the wrong term into the contract record
A code suggestion has a destructive bug	The agent runs that code with production write access
A recommendation names the wrong supplier	The agent emails that supplier or calls its ordering API

The left column still needs a person to act. The right column doesn’t. The line isn’t absolute, though: people act on unchecked generated text, and a read-only agent may never change anything.

Errors also compound. Anthropic’s write-up says plainly, “Agents are stateful and errors compound.” If ten independent steps each succeed 95% of the time, all ten succeed only about 60% of the time (0.95¹⁰ ≈ 0.60). Real steps aren’t independent, so treat that as arithmetic, not a forecast.

Picture a refund agent whose database write commits but whose confirmation times out. A naive retry issues a second refund. The controls that contain this are:

Validation: code checks each proposed action against a schema and business rules before it runs.
Idempotency keys: a retried write applies only once.
Bounded retries: attempts are capped per step and per task.
Transaction boundaries: related writes commit together or not at all.
Approval gates: a person confirms costly or irreversible actions.
Compensation: each action that can’t be undone directly has a defined correcting action.
Audit logs: the system records what was proposed, what ran and under which identity.

Agentic AI vs generative AI: security is an execution problem

A model that follows instructions well still isn’t a security boundary. The OWASP Top 10 for LLM Applications 2025 lists Excessive Agency (LLM06) and traces it to “excessive functionality,” “excessive permissions” and “excessive autonomy.” Its fixes include “complete mediation,” meaning authorisation enforced in downstream systems rather than by the model.

Prompt injection is why. OWASP’s prompt-injection entry (LLM01) covers instructions hidden in web pages, files and other content an agent reads. It says “it is unclear if there are fool-proof methods of prevention.” The UK’s National Cyber Security Centre argued in December 2025 that prompt injection is not SQL injection. Its reasoning was that LLMs “do not enforce a security boundary between instructions and data inside a prompt.” It therefore recommended deterministic, non-LLM safeguards over filtering alone. In March 2026, NIST’s Center for AI Standards and Innovation reported a red-teaming competition with more than 250,000 attack attempts. At least one attack succeeded against each of the 13 frontier models tested.

Where the controls belong

A filter can lower the odds of an attack, but it can’t decide what an attack may do. A framework’s tool list only controls what the model can request. A “run_python” tool inside your application process inherits that process’s network access, files and credentials. Real isolation comes from the operating system, container or virtual machine, plus egress rules and per-tool credentials. Our guide to the four layers of sandbox isolation covers how each layer limits damage.

The controls that matter are:

Least privilege: each tool gets only the access its job needs.
Tool allowlists: the agent calls named, reviewed tools, with no general shell unless the task requires one.
Scoped credentials: each tool has its own short-lived token.
Egress restrictions: outbound traffic is blocked by default.
Sandboxing: generated code runs outside the application’s process.
Human approval: payments, deletions and external messages wait for a person.
Independent policy enforcement: a separate check decides whether each action is authorised.

No single item makes an agent secure. OWASP’s Top 10 for Agentic Applications (December 2025) lists Agent Goal Hijack, Tool Misuse, Identity & Privilege Abuse and Cascading Failures as separate risks. Each needs its own control.

Agentic AI vs generative AI under rate limits

Four measures get confused. Requests per minute (RPM) counts API calls. Tokens per minute (TPM) counts text processed. In-flight concurrency is how many requests are open at once, and end-to-end latency is how long the user waits.

Providers measure these differently. On 18 October 2026:

OpenAI sets RPM, TPM and daily limits per organisation and project. Its guide warns that “unsuccessful requests contribute to your per-minute limit.”
Anthropic sets RPM and separate input and output TPM per model class. Its documentation says cache reads don’t count toward input limits for most models.
Google’s Gemini API applies RPM, TPM and daily limits per project, not per key, and says they “are not guaranteed.”

An illustrative fan-out

Suppose 100 users start a task in the same minute, using the five-call agent above. Each of its four tool steps runs three tools in parallel. Each model call takes 6 seconds and each tool step 2 seconds.

In that minute	Single call	Five-call agent
Model requests	100	500
Input tokens	500,000	4,050,000
Downstream tool calls	0	1,200
Time per task	about 6 seconds	about 38 seconds

The agent sends five times the requests and about eight times the input tokens, and it also loads your own systems, such as a search API or CRM. With caching on a provider that excludes cache reads, the input counted against limits drops to about 1.17 million, but the request count doesn’t change.

Retries can make this worse, because immediate retries after 429 errors add requests to a minute that’s already full. Honour any retry-after header, and otherwise use exponential backoff with random jitter, which Marc Brooker’s AWS analysis showed cuts wasted work sharply. Agents also need bounded concurrency per model and tool, a queue that holds new tasks instead of failing them, cancellation of abandoned branches, and hard caps on steps, time and tokens. Our reference on AI API rate limits in 2026 compares each provider’s published ceilings.

When a pipeline beats an agent

Anthropic’s advice is to find “the simplest solution possible,” because agentic systems “trade latency and cost for better task performance.” That trade only pays when you need the performance.

A deterministic pipeline is the better choice when you can list the steps in advance, check each output in code, run at high volume, or meet a tight latency budget. Pipelines still use generative models: one call classifies a ticket, another extracts fields, and a third drafts the reply, with code fixing the order.

Agents earn their cost when the next step depends on what the last one found, as in debugging or open-ended research. Even then, a bounded agent often works best. It has narrow tools, a step limit and deterministic checks before any write, and it sits inside a larger pipeline.

A practical decision table

Scenario	Likely architecture	Why it fits	Main concern
Single-turn drafting and summarisation	One model call	The output is the product, and a person reviews it	Accuracy
Structured extraction and classification	Pipeline with schema validation	Fixed steps; output checkable in code	Silent field errors at volume
Research across several sources	Bounded agent with read-only tools	The next search depends on the last result	Token growth; injected content
Database updates and external transactions	Pipeline, or an agent that only proposes, with approval and idempotent writes	Side effects must be validated and reversible	Duplicate or unauthorised writes
Long-running multi-step workflows	Workflow engine with agent steps inside	State and retries belong in infrastructure	Cache expiry, lost state, runaway loops
High-volume, latency-sensitive requests	Single call or short fixed chain	Predictable latency and limits	Fan-out and retries under load

Conclusion

The difference between agentic AI and generative AI shows up in operations more than output. An agent multiplies calls, compounds errors, holds permissions and loads rate limits in ways a single generation doesn’t.

So use one model call when one output answers the request, and a pipeline when you can write down the steps. Use an agent only when the next step depends on what the last one found. Then give it a step limit, a token budget, scoped credentials and a checker that isn’t the model.

FAQ

What is the difference between agentic AI and generative AI?

Generative AI produces content from a prompt. Agentic AI is a system that pursues a goal over several steps, choosing actions, calling tools and using the results. Agents usually rely on a generative model to decide.

Can generative AI become part of an agentic system?

Yes, and it usually does. The model plans, picks tools and writes the answer, while the application runs the loop, executes tools and enforces limits.

Is agentic AI more expensive than generative AI?

Usually, per task, because one task triggers many calls that re-send context. Caching, state management and completion rate can matter more than token price.

Is agentic AI more dangerous than generative AI?

Its risks are different. An agent can act without a person in between, so mistakes and injected instructions can have direct effects. The risk scales with the permissions the system grants.

When should a company use a deterministic workflow instead of an AI agent?

Use one when the steps are known in advance, outputs can be checked in code, or volume and latency demands are high. Use an agent when the path truly depends on intermediate results.

How do you measure the cost of an agentic AI task?

Divide total inference, tool, infrastructure and retry costs by the number of successfully completed tasks. Track calls per task, tokens per call, cache hit rate and completion rate separately.

How can developers limit an AI agent’s permissions and actions?

Give each tool scoped, short-lived credentials, and run generated code in a sandbox with outbound traffic blocked by default. Require human approval for consequential actions, enforce authorisation outside the model, and cap steps, time and tokens.

Keep reading

Here are the latest posts from the blog.

Claude API Rate Limits in 2026: How Many Requests Can You Run?

November 5, 2026

Claude API rate limits on the Start tier allow most current models 2,000,000 input tokens and 400,000 output tokens per minute, with 1,000 or 4,000 …

Enterprise AI Spending Hit $177B. Agents Are Driving the Bill

November 4, 2026

Figures checked against Menlo Ventures’ 2026 State of Enterprise AI report on 4 November 2026. Market sizes are Menlo estimates, shown as annualised run rates. …

Machine Unlearning: Can We Really Trust an Edited AI Model?

November 4, 2026

30 Agentic AI Statistics for 2026: Adoption, Spending and Risk

November 3, 2026

Last updated: 3 November 2026. This page is updated monthly; each figure keeps its original publication date. Agent traffic at one healthcare booking platform rose …

When Adding an Agent Stops Paying for Itself

by Idrees Patel

The architecture that became more expensive without becoming better

Multi-agent token cost rarely shows up in a design review. It shows up a month later on the invoice. Take a team that splits a defect-investigation agent into a planner, three specialist workers and a synthesiser. Its fix rate improves, so it scales to eight workers. The fix rate stays flat and the per-run model bill more than doubles.

Nothing went wrong with the models. Each worker reread the same code, wrote its own notes, and handed them to a synthesiser that read them all again. Per-token prices have fallen over the past year, but this system now buys far more tokens per answer. A cheaper token and a more expensive system can coexist, and that gap is the subject of this article.

Key Takeaways

Agent count is a poor proxy for quality. In the largest controlled study available, multi-agent setups averaged −0.3% against a single agent, with results ranging from +80.8% to −70% depending on the task.
Most extra tokens come from repeated context. In this article’s eight-agent example, workers rereading the same 20,000-token code context account for 72% of all input tokens.
In the example, going from one to three agents cost $2.34 per additional successful task. Going from three to five cost $12.95. Going from five to eight cost $0.78 more per run with no gain.
Token prices and GPU rental prices are separate markets. Falling API rates do not mean falling H100 rents, and the dated data does not show that they have.
Extra agents can still pay when subtasks are genuinely independent, when a failure is expensive, or when parallel work cuts latency someone cares about.
Before adding an agent, run an ablation. Remove it, rerun the same evaluation set, and compare cost per successful task.

Quick Navigation

The multi-agent token cost is hidden in the handoffs
What the research on multi-agent scaling establishes
The four-configuration multi-agent token cost test
- Token arithmetic
- Results
The point where more agents stop paying
Why cheaper tokens do not guarantee cheaper AI systems
A decision rule engineers can actually use
What to measure before scaling the agent count
Conclusion: optimise the work, not the agent count
Frequently Asked Questions

The multi-agent token cost is hidden in the handoffs

Agents are not the unit of cost. Model calls are, and every call pays for its whole input again.

A single agent investigating a bug might make eight calls. It starts with the bug report and the relevant code, about 20,000 tokens, and appends what it learns as it goes. By the last call its input is larger, but it only ever pays for one copy of the codebase per call.

Split that work across workers and the arithmetic changes. A planner reads the task and writes assignments. Each worker then reads the same code plus its own brief, makes a few calls of its own, and writes a report. A synthesiser reads every report alongside the original context. The code that the single agent read eight times is now read four times per worker, by every worker.

That is the first cost driver: duplicated context. The second is intermediate output. Plans, reports and summaries are output tokens, billed at five times the input rate on many current models, and then billed again as input when the next agent reads them. Retries add a third layer. A worker that fails a tool call reruns with its full context.

None of this makes multi-agent design wrong. It means the useful question is how much an added worker contributes beyond the context it rebuys, not how many agents a system has. UniverseBlend found the same pattern in a different form in its analysis of why a 50% price cut doesn’t halve agent bills: in agent workloads, the line item that repeats matters more than the headline rate.

What the research on multi-agent scaling establishes

The circulating claim that agent quality plateaus while token use keeps climbing has a real basis, but not a single magic number behind it. No paper dated October 12, 2026 matching that claim could be found. The strongest primary evidence is earlier and more careful.

Kim and colleagues at Google Research, Google DeepMind and MIT tested 260 configurations in Towards a Science of Scaling Agent Systems (arXiv, version 3 dated April 8, 2026). They compared a single agent against four multi-agent designs across six agentic benchmarks, from financial analysis to SWE-bench Verified, using OpenAI, Google and Anthropic models.

The headline is variance, not improvement. Averaged across every multi-agent result, performance changed by −0.3% relative to a single agent, with a 95% interval running from −58.7% to +77.2%. A centralised design improved Finance-Agent by 80.8%. Every multi-agent design made PlanCraft worse, by 39.1% to 70.0%. On SWE-bench Verified, all variants lost ground.

Two findings bear directly on cost. First, token efficiency collapsed. A single agent produced 67.7 successes per thousand tokens; centralised and hybrid teams produced 21.5 and 13.6. Second, tasks where a single agent already scored above 45% saw negative returns from adding agents. This was the paper’s most statistically robust result (β = −0.236, p = 0.004).

The agent-count question itself was a smaller, preliminary test. Using Gemini models at 1, 3, 5, 7 and 9 agents, the authors report that Gemini 2.0 Flash peaked at seven agents before degrading, and that a centralised Gemini 2.5 Pro team showed diminishing returns beyond five. The paper does not give the exact accuracies or name the benchmark for that figure. So “quality plateaus at five agents” is too strong a summary. The defensible reading is that the optimum depends on model and coordination design, and it arrived at single-digit team sizes in the cases tested.

The study did not measure latency, and it reports dollar cost only as marginal cost per point of success for one architecture. A newer paper on vulnerability-discovery agents, Where Do the Tokens Go? (October 8, 2026), adds a consistent observation from a different domain: “higher spending does not consistently yield better outcomes.” It does not vary agent count, so it supports the spending point and nothing more.

The practical consequence is that there is no published universal agent count to copy. Each team has to measure its own curve.

The four-configuration multi-agent token cost test

What follows is a transparent cost model, not a benchmark. Token counts and success rates are illustrative assumptions chosen to be plausible and consistent with the research above. Prices are real.

Task (same for all four): investigate a reported defect in a service and produce a root cause plus a proposed patch. Success means the patch passes the project’s existing test suite and a reviewer accepts the diagnosis.
Model and prices: Claude Sonnet 5.5, $2.00 per million input tokens and $10.00 per million output tokens, USD, from Anthropic’s pricing page, checked October 11, 2026.
Included: every priced model call (planner, workers, synthesiser).
Excluded: prompt caching, retries, tool execution, embeddings, external search, orchestration servers and engineering time.

Workload assumptions:

Shared context: 20,000 tokens (bug report plus relevant code).
One agent: 8 sequential calls, average input 32,000 tokens (shared context plus 12,000 accumulated notes), 1,500 output tokens each.
Multi-agent (n workers): one planner call (21,000 in / 2,000 out); each worker makes 4 calls averaging 26,000 input (shared context, brief and own notes) and 1,200 output; one synthesiser call reading the shared context, a 1,500-token report per worker and the 2,000-token plan, writing 3,000 tokens.
Success rates (illustrative): 38%, 50%, 54% and 53%. The single-agent baseline is set below the 45% level at which Kim et al. found extra agents start to hurt. The slight drop at eight reflects the degradation they observed at larger team sizes.

Token arithmetic

1 agent: input 8 × 32,000 = 256,000; output 8 × 1,500 = 12,000.
3 agents: input 21,000 + (3 × 4 × 26,000 = 312,000) + (20,000 + 4,500 + 2,000 = 26,500) = 359,500; output 2,000 + 14,400 + 3,000 = 19,400.
5 agents: input 21,000 + 520,000 + 29,500 = 570,500; output 2,000 + 24,000 + 3,000 = 29,000.
8 agents: input 21,000 + 832,000 + 34,000 = 887,000; output 2,000 + 38,400 + 3,000 = 43,400.

Cost = input tokens ÷ 1,000,000 × $2.00 + output tokens ÷ 1,000,000 × $10.00.

Results

Metric	1 agent	3 agents	5 agents	8 agents
Input tokens	256,000	359,500	570,500	887,000
Output tokens	12,000	19,400	29,000	43,400
Total tokens	268,000	378,900	599,500	930,400
Input cost	$0.512	$0.719	$1.141	$1.774
Output cost	$0.120	$0.194	$0.290	$0.434
Total API cost per run	$0.632	$0.913	$1.431	$2.208
Success rate (illustrative)	38%	50%	54%	53%
Incremental cost vs previous	—	+$0.281	+$0.518	+$0.777
Incremental success vs previous	—	+12 pts	+4 pts	−1 pt
Cost per successful task	$1.66	$1.83	$2.65	$4.17
Latency (estimate, 8 s per call, workers fully parallel)	~64 s	~48 s	~48 s	~48 s

These are four different measurements. Total tokens tripled between one and five agents (268,000 to 599,500), while cost per successful task rose by 60% ($1.66 to $2.65). The ratios differ because more successes partly offset the extra spend. Latency improves when workers run in parallel, as long as the provider lets eight 26,000-token requests run at once. That depends on tokens-per-minute limits, covered in UniverseBlend’s guide to AI API rate limits in 2026. The latency figures are illustrative too.

The point where more agents stop paying

The marginal numbers carry the argument.

One to three agents: +$0.281 per run for +12 points of success, or $0.281 ÷ 0.12 = $2.34 per additional successful task.
Three to five: +$0.518 for +4 points = $12.95 per additional success.
Five to eight: +$0.777 for −1 point. No additional successes, so the move never pays.

Whether $12.95 is too much depends on what a success is worth, which the token bill cannot tell you. The rule is to add the agent only if (gain in success rate) × (value of one success) is greater than the added cost per run.

If a resolved defect saves an engineer an hour, valued at $50, then 0.04 × $50 = $2.00 exceeds $0.518 and five agents beat three. If the task is low-stakes triage worth $5 per success, 0.04 × $5 = $0.20 falls short, and three agents is the right stopping point. Eight agents lose at any value, because they produce no gain.

That is the honest answer to the headline. In this example the agent stops paying for itself at the configuration where its marginal success gain, times what a success is worth, drops below its marginal tokens. The break-even depends on the value of a success, not on a universal number of agents.

Four task properties move that point.

Independence of subtasks. Workers that split a search space with little overlap add coverage cheaply. Workers that all need the full context mostly add duplicated reading. That is the Finance-Agent pattern versus the PlanCraft pattern in Kim et al.
Baseline strength. When a single agent already succeeds most of the time, there is little left to gain and more room for coordination errors.
Error propagation. Kim et al. measured error amplification of 17.2× for independent agents against 4.4× for centralised designs with a verifier. More agents without a checking step multiply mistakes as well as tokens.
Cost of failure. For security review, financial reconciliation or production incidents, a second independent opinion can be worth many times its tokens even at a small success gain.

Why cheaper tokens do not guarantee cheaper AI systems

API prices have fallen. OpenAI’s GPT-6 Sol launched at $2 input and $10 output per million tokens, half the $4 and $20 of GPT-5.6 Sol, as UniverseBlend covered in September. On the current model page, GPT-6.1 Sol still lists $2 and $10.

GPU rents have not followed. The Silicon Data H100 rental index read $2.85 per GPU-hour on October 12, 2026, up 1.1% over the previous week. SemiAnalysis has reported one-year H100 rentals rising from about $1.70 in October 2025 to about $2.35 in March 2026. That figure could not be independently opened for this article and should be treated as reported, not verified. On-demand list prices on October 10 still ran from $2.89 to $11.68 per GPU-hour depending on provider and configuration, as tracked in UniverseBlend’s GPU cloud pricing comparison.

The two prices measure different things. An API price is what a provider charges per token, after its own hardware efficiency, batching and margin. A GPU rent is what scarce hardware fetches by the hour. Better software, quantisation and newer chips can push the token price down while demand for the hardware pushes its rent up. Both can be true at once without contradicting each other.

This is where Jevons’ paradox gets invoked. William Stanley Jevons observed in The Coal Question (1865) that more efficient steam engines increased Britain’s total coal consumption, because cheaper effective energy opened new uses. The AI version says cheaper tokens unlock workloads, multi-agent systems among them, that consume so many more tokens that total compute demand rises.

The analogy needs conditions before it holds. Demand for tokens must be elastic enough that lower prices raise total spending, not just volume. The efficiency gain must be what lowered the price, not a subsidy or promotional pricing. And the measure must be total resource use, not one company’s bill. The dated evidence here shows falling list prices and stable-to-rising GPU rents. That is consistent with Jevons, but it does not prove it. Supply constraints alone could explain firm rents.

For a team running agents, the practical point is narrower. A cheaper model makes each token cheaper, but an architecture that multiplies tokens 3.5-fold, as eight agents did in the example, can absorb a 50% price cut and still cost more. If you self-host, the relevant price is the GPU-hour and your utilisation, not the API rate card.

A decision rule engineers can actually use

Add an agent only when its measured marginal contribution is worth its marginal cost:

\Delta p \times V_{\text{success}} \; > \; \Delta C_{\text{run}} + \Delta C_{\text{latency}} + \Delta C_{\text{operations}}

Here Δp is the change in success rate, V is the value of one success, and the right side is the added cost per run in tokens, waiting time and operational burden.

Before adding one, check:

Baseline quality: what does the simplest configuration achieve on a fixed evaluation set?
Incremental success: how much does the new agent change the success rate, with confidence intervals?
Added tokens: how many input and output tokens does it add per run, including the context it rereads?
Marginal cost: what is the cost per additional successful task?
Latency: does it shorten or lengthen end-to-end time at your rate limits?
Failure severity: what does one wrong answer cost downstream?
Evaluation reliability: is the evaluation set large enough to detect a gain this size?
Simpler alternatives: would a better prompt, a verifier step or a stronger single model do the same?

The most useful test is an ablation. Disable one agent, rerun the same evaluation set, and compare cost per successful task. If nothing measurable changes, the agent was decoration.

What to measure before scaling the agent count

Log tokens per call, not per run, tagged with agent role, so duplicated context is visible. Track cost per successful task as the primary metric, alongside success rate, p50 and p95 latency, retries and tool-call count.

Keep offline and production numbers apart. An evaluation set measures capability on known tasks. Production traffic adds longer contexts, rate-limit queueing and failure modes the evaluation set never contained. Re-check the ablation on a production sample before committing.

Prompt caching changes the arithmetic for the duplicated-context problem specifically. If workers share an identical prefix, cache reads can cut the cost of rereading it sharply. Rerun the cost model with your actual cache-hit rate before deciding, because it can move the break-even point.

Conclusion: optimise the work, not the agent count

The evidence does not support a universal team size, and it does not support the idea that more agents are better. The largest controlled study found an average effect near zero, wide variation by task, and token efficiency falling by a factor of three or more. The worked example shows how that looks on an invoice. The first two extra agents earned their keep, the next two did so only for valuable tasks, and the last three added nothing.

Multi-agent token cost is a design variable, not a fixed tax. Split work where subtasks are genuinely independent, add verification where errors are expensive, and stop when the next agent’s marginal success, times what a success is worth, no longer covers its marginal tokens. An additional agent is justified when its measurable contribution exceeds what it costs. Nothing else about it matters.

Frequently Asked Questions

What is multi-agent token cost?

Multi-agent token cost is the total model spend of a system where several LLM agents collaborate on one task. It includes each agent’s own calls, the context every agent rereads, plans and reports passed between agents, and the final synthesis. It usually grows faster than the number of agents because shared context is paid for repeatedly.

How many agents should an AI system use?

No study supports a universal number. Kim et al. found optimal team sizes depended on the model and coordination design, with diminishing returns appearing beyond five to seven agents in their preliminary Gemini tests. Start with one agent, add one at a time, and keep each only if it improves cost per successful task.

Why do multi-agent systems consume more tokens?

Each agent rereads shared context, writes intermediate plans or reports, and passes them on to be read again. Coordination also adds turns: Kim et al. measured hybrid designs taking 44.3 turns on average against 7.2 for a single agent. Retries and synthesis calls add further tokens.

Does adding more agents improve LLM accuracy?

Sometimes. Across 260 configurations, Kim et al. found a mean change of −0.3%, with gains up to +80.8% on a financial-analysis benchmark and losses up to −70% on a planning benchmark. Tasks where a single agent already exceeded 45% accuracy tended to get worse with more agents.

How can engineers calculate cost per successful AI task?

Add up input and output token costs for every model call in a run, then divide by the run’s success rate on a fixed evaluation set. For example, $0.913 per run at 50% success is $1.83 per successful task. Compare configurations on this figure, not on raw token counts.

Does cheaper token pricing make multi-agent systems cheaper?

Only if token consumption grows more slowly than prices fall. A 50% price cut is erased by an architecture that uses 3.5 times the tokens. And for self-hosted systems, API price cuts do not lower GPU rents, which have been stable to rising in 2026.

Keep reading

Here are the latest posts from the blog.

Claude API Rate Limits in 2026: How Many Requests Can You Run?

November 5, 2026

Claude API rate limits on the Start tier allow most current models 2,000,000 input tokens and 400,000 output tokens per minute, with 1,000 or 4,000 …

Enterprise AI Spending Hit $177B. Agents Are Driving the Bill

November 4, 2026

Figures checked against Menlo Ventures’ 2026 State of Enterprise AI report on 4 November 2026. Market sizes are Menlo estimates, shown as annualised run rates. …

Machine Unlearning: Can We Really Trust an Edited AI Model?

November 4, 2026

30 Agentic AI Statistics for 2026: Adoption, Spending and Risk

November 3, 2026

Last updated: 3 November 2026. This page is updated monthly; each figure keeps its original publication date. Agent traffic at one healthcare booking platform rose …

LLM Inference Latency by Model and Provider: Real Numbers

by Idrees Patel

Published October 2026 · Data read October 11, 2026 · Source: Artificial Analysis provider benchmarks, secondary research

The same open-weight model, Llama 3.3 70B, starts answering in 0.67 seconds on Google Vertex and 2.37 seconds on DeepInfra. Once it starts, Groq streams 328 tokens per second and DeepInfra 17. A 500-token answer takes 2.4 seconds on Groq and 32.5 seconds on DeepInfra’s FP8 “Turbo” endpoint.

Those figures come from Artificial Analysis, which runs the same 10,000-token prompt against every host from one Google Cloud VM in Iowa and reports the median over 72 hours. That is about as fair as public LLM inference latency data gets, and it is still not perfectly fair. DeepInfra’s endpoint is quantized to FP8, no host discloses its hardware or batch size, and a request from Frankfurt or Mumbai will see different network time.

The spread is the point. Rate limits tell you how much a provider lets you run, and pricing tells you what it costs. Latency decides whether the product feels usable. That depends far more on the serving stack and the settings you choose than on the model’s name.

Key Takeaways

For one model, host choice moved output speed 19-fold (17 to 328 tokens/s for Llama 3.3 70B) and time to first token 3.5-fold.
Reasoning effort dominates frontier-model latency. On Artificial Analysis’s default “max” setting, Claude Sonnet 5.5 took about 490 seconds to its first answer token. Its low setting is listed at under a second on the provider leaderboard, a figure to check before quoting.
Time to first token (TTFT) and output speed are separate problems. A short chat reply is dominated by TTFT. A long report is dominated by output speed.
Agents multiply latency. Five sequential calls on a slow endpoint can take more than 100 seconds before tool time is added.
Published medians hide tail latency. No source examined here published p95 figures, so you have to measure them yourself.
Test with your own prompt lengths, region and reasoning settings before trusting any ranking, including this one.

Quick Navigation

Why LLM inference latency is hard to compare
What TTFT actually measures
TTFT versus output tokens per second
LLM inference latency by model and provider
How prompt length changes the first-token experience
When streaming helps and when it does not
What LLM inference latency means for AI agents
Hosted APIs versus Llama and Qwen inference hosts
How to run a fair LLM inference latency test
A practical provider selection framework
What the available evidence cannot tell us
Conclusion
Frequently Asked Questions

Why LLM inference latency is hard to compare

Two published numbers for “the same model” can differ for at least six reasons that have nothing to do with the model: where the test client sits, how long the prompt is, how many requests run at once, which reasoning setting is on, how the host quantizes and batches, and which statistic is reported.

Even one careful source can disagree with itself. On Artificial Analysis’s Gemini 3.5 Flash-Lite page, the headline gives 414 tokens per second and the comparison table gives 365. Its GPT-6.1 Sol page reports a time to first answer token of 311 seconds in the headline and 327 seconds in the table. Neither page explains the gap. Different averaging windows are the likely cause, but that is an inference, not a documented fact. If a single benchmark cannot hold one number steady across its own page, rankings stitched together from several sources deserve more suspicion.

What TTFT actually measures

Time to first token (TTFT) is the wall-clock time between sending a request and receiving the first token of the response. Artificial Analysis’s methodology defines it that way and measures from the client.

That single number folds together several stages:

network time from client to provider and back
admission and queueing behind other customers’ requests
prefill, where the model processes the entire prompt before it can generate anything
any provider-side orchestration, such as safety checks, routing or tool-schema handling
for reasoning models, the thinking phase, if the measurement waits for the first answer token

Providers do not publish this breakdown, and a client-side measurement cannot separate the stages. A slow TTFT could mean a long queue, a long prompt or a distant data centre.

The reasoning point causes most of the confusion. Artificial Analysis reports two variants: TTFT to the first token of any kind, and “time to first answer token”, which includes thinking. Its per-model pages for frontier models use the second variant. That is why Claude Opus 5.5 at max effort shows roughly 600 to 1,000 seconds depending on host. The figure measures how long the model chose to think, not how fast Anthropic, Google, Azure or Amazon serve it.

TTFT versus output tokens per second

Output speed is the average rate at which tokens arrive after the first one. Artificial Analysis defines it as tokens received per second after the first token. It governs how long a full answer takes. TTFT governs how long the user stares at nothing.

The two do not move together. On Llama 3.3 70B, Google Vertex had the lowest TTFT (0.67 s) but streamed at 162 tokens/s, while SambaNova was slower to start (2.26 s) but generated faster (280 tokens/s). For a 50-token reply, Vertex wins comfortably. For a 2,000-token document, SambaNova finishes first.

The gap comes from different bottlenecks. Prefill is mostly compute-bound, processing many prompt tokens in parallel. Decode, the token-by-token generation that follows, is mostly limited by memory bandwidth, because the model’s weights stream through the chip for every token. Hardware and batching choices that help one phase can hurt the other. Large batches raise a provider’s total throughput but slow each individual stream.

LLM inference latency by model and provider

All figures are third-party measurements by Artificial Analysis, read October 11, 2026. Conditions throughout: about 10,000 input tokens, single request at a time, client in Google Cloud us-central1-a, streaming, median of the previous 72 hours. “End-to-end” is the time to receive a 500-token answer. None of this was measured by UniverseBlend.

Table 1: One model, many hosts (the controlled comparison)

Model	Host	TTFT (s)	Output speed (tok/s)	End-to-end, 500 tokens (s)	Notes
Llama 3.3 70B	Google Vertex	0.67	162	3.76	Precision not reported
Llama 3.3 70B	Groq	0.85	328	2.37	Custom LPU hardware
Llama 3.3 70B	CoreWeave	0.89	78	7.34	Not reported
Llama 3.3 70B	Scaleway	1.48	73	8.34	Headline TTFT 1.34 s
Llama 3.3 70B	Together AI (Turbo)	1.59	71	8.65	“Turbo” variant
Llama 3.3 70B	Novita	1.78	45	12.83	Not reported
Llama 3.3 70B	Azure	2.24	121	6.38	Not reported
Llama 3.3 70B	SambaNova	2.26	280	4.05	Headline speed 294.5
Llama 3.3 70B	DeepInfra (Turbo)	2.37	17	32.53	FP8
Llama 3.3 70B	Parasail	2.57	86	8.39	FP8

Source: Artificial Analysis, Llama 3.3 70B providers. p95: not reported for any row.

Table 2: Frontier models at maximum reasoning effort

Model (API ID)	Host	Time to first answer token (s)	Output speed (tok/s)	End-to-end, 500 tokens (s)
Claude Sonnet 5.5 (`claude-sonnet-5-5`)	Amazon	475.97	143	479.46
Claude Sonnet 5.5	Anthropic	492.88	142	496.41
Claude Opus 5.5 (`claude-opus-5-5`)	Google	604.25	95	661.24
Claude Opus 5.5	Anthropic	704.18	97	667.28
GPT-6.1 Sol (`gpt-6.1-sol`)	Azure	288.60	78	294.99
GPT-6.1 Sol	OpenAI	326.94	56	335.94

Sources: Artificial Analysis provider pages for Claude Sonnet 5.5, Claude Opus 5.5 and GPT-6.1 Sol. Effort: “max” with default fallback (Claude); “max” (GPT-6.1 Sol). These numbers mostly measure thinking time. Several Opus 5.5 rows show a time-to-first-answer longer than the 500-token total, an inconsistency the source does not explain.

Table 3: Lower-effort and lightweight endpoints (leaderboard view)

Model	Host	First chunk (s)	Output speed (tok/s)	Total (s)	Setting
Claude Sonnet 5.5	Anthropic	0.88	100	5.86	low effort
GPT-6 Luna (`gpt-6-luna`)	OpenAI	2.83	125	6.84	low effort
GPT-6.1 Sol	OpenAI	2.80	47	13.41	low effort
Gemini 3.5 Flash-Lite (`gemini-3.5-flash-lite`)	Google AI Studio	8.96	365	10.33	default; reasoning not reported
Qwen3.8 2.4T A95B	Together AI	1.14	112	23.40	not reported
Qwen3.8 2.4T A95B	DigitalOcean	0.82	57	44.54	not reported
Qwen3.8 2.4T A95B	Alibaba Cloud	2.99	37	71.44	not reported

Source: Artificial Analysis provider leaderboard, with Gemini from its model page. Treat this table as indicative: column values were extracted from a combined view and should be confirmed on the live page. The Qwen totals are high relative to their speed, which suggests reasoning output or longer answers. The source does not say which.

The Sonnet 5.5 rows across Tables 2 and 3 tell the main story. The same model on the same host goes from under one second to about eight minutes before answering, depending on a single API parameter.

Measurement-quality matrix

Comparison	Same model?	Same harness?	Same settings?	Verdict
Llama 3.3 70B across hosts (Table 1)	Yes	Yes	Precision varies	Controlled; fair to rank, noting FP8 rows
One frontier model across clouds (Table 2)	Yes	Yes	Yes (max effort)	Partially comparable; differences mix serving and thinking-time variance
Different frontier models (Table 2)	No	Yes	Effort labels differ by vendor	Unsuitable for a speed ranking; effort levels are not equivalent across vendors
Low-effort rows (Table 3)	No	Yes	Varies	Indicative only
Qwen across hosts (Table 3)	Yes	Yes	Not reported	Partially comparable
Any of the above versus your own region and concurrency	n/a	No	No	Unsuitable without retesting

How prompt length changes the first-token experience

Before a model can write its first word, it has to process every prompt token. More input means more prefill work, so TTFT generally rises with prompt length. Attention also scales worse than linearly with sequence length, which matters most at very long contexts.

In production the relationship is rarely clean. Providers split long prompts into chunks, interleave them with other customers’ work, and route long-context requests to different hardware. Artificial Analysis tests at 1,000, 10,000 and 100,000 input tokens for this reason. All the figures above use the 10,000-token workload, so they overstate TTFT for a short chat turn and understate it for a large retrieval prompt.

Caching changes the picture for repeated prefixes. Anthropic’s prompt-caching documentation says users “will generally see improved time-to-first-token for long documents” when a cached prefix is reused. Caching has no effect on output generation, and the default cache lives five minutes. The documentation gives no millisecond figure, so measure the gain on your own prompts.

Typical input size	What to measure beyond the published median
Under 2,000 tokens (chat turns, classification)	TTFT and first visible text from your own region; p95 under expected concurrency
2,000–30,000 tokens (RAG, documents)	TTFT at three or more sizes in this range; cache-hit versus cache-miss TTFT
30,000+ tokens (codebases, long files)	TTFT growth curve up to your real maximum; timeouts and error rate; whether the endpoint routes long prompts differently
Any size with a fixed system prompt	Cached-prefix TTFT, and how quickly the cache expires under your traffic pattern

When streaming helps and when it does not

Streaming sends tokens as they are generated instead of waiting for the full response. It does not make the model faster. It changes when the user first sees something.

For a chat interface, that is often enough. Jakob Nielsen’s widely cited response-time limits, drawn from Miller (1968) and Card et al. (1991), put the edge of uninterrupted thought at about one second and the edge of sustained attention at about ten. A streamed answer that starts within a second and reads at a steady pace keeps a reader engaged, even if completion takes eight seconds.

Streaming fails in three common cases. The first is when the client needs the whole response before acting: structured JSON, a tool call or a function argument cannot be used half-written. The second is uneven cadence. Long pauses mid-stream, from server batching or client buffering in proxies and UI frameworks, can feel worse than a slightly later start. The third is reasoning models that show nothing until thinking ends, so a fast first token is meaningless if that token is hidden reasoning.

What LLM inference latency means for AI agents

This is where latency stops being a benchmark number and becomes a product problem. An agent rarely makes one call. It plans, calls a tool, reads the result and calls the model again, and every step pays TTFT and generation time in sequence.

A rough estimate per step:

T_{\text{step}} \approx \text{TTFT} + \frac{\text{output tokens}}{\text{output tokens per second}} + T_{\text{tool}}

This is an approximation. It ignores mid-stream pauses, retries and queueing spikes, and it assumes steps run one after another rather than overlapping.

The formula reproduces the source’s own totals. For Groq, 0.85 s + 500 / 328 tok/s = 2.37 s, exactly the published end-to-end figure. For DeepInfra, 2.37 + 500 / 17 = 31.8 s, against 32.5 s published.

Worked example (derived, not measured): a five-step agent, each step producing 300 output tokens and making one tool call that takes an assumed 0.8 seconds.

Endpoint (Table 1 or 2 figures)	Per step	Five steps
Llama 3.3 70B on Groq	0.85 + 0.91 + 0.8 = 2.56 s	12.8 s
Llama 3.3 70B on Google Vertex	0.67 + 1.85 + 0.8 = 3.32 s	16.6 s
Llama 3.3 70B on DeepInfra (FP8)	2.37 + 17.65 + 0.8 = 20.82 s	104 s
Claude Sonnet 5.5, max effort, Anthropic	492.88 + 2.11 + 0.8 = 495.8 s	about 41 minutes

The same five-step task is a 13-second interaction or a 41-minute background job, depending on host and settings. Neither outcome is wrong, but each implies a different product. Voice is stricter still: Stivers et al.’s 2009 PNAS study of turn-taking across 10 languages found average gaps between speakers clustering within 250 milliseconds of a cross-language mean. That points to a budget measured in hundreds of milliseconds, not seconds. That figure is a design guide drawn from human conversation, not a measured LLM threshold.

A reasonable set of engineering guidelines, labelled as such: aim for first visible text under about one second in chat, under a few hundred milliseconds of response gap in voice, and set an explicit end-to-end budget for agents, with p95 rather than median as the target.

Hosted APIs versus Llama and Qwen inference hosts

With a closed model, you can only buy latency from the vendor or the clouds that resell it. Table 2 shows that resellers can differ: GPT-6.1 Sol reached its first answer about 38 seconds sooner on Azure than on OpenAI’s own API, and streamed faster. With open weights you can also choose the serving stack, and Table 1 shows that choice matters more than anything else measured here.

That difference belongs to the host, not the model. Llama 3.3 70B is identical weights everywhere, yet output speed varies 19-fold. The causes are mostly undisclosed: accelerator type (Groq and SambaNova build their own chips), quantization (DeepInfra and Parasail are FP8), batch size and scheduling policy, and data-centre location relative to the test client. A slow host is not evidence that the model is slow, and a fast host’s numbers do not transfer to your own deployment.

Self-hosting gives you control of those variables and the responsibility for tuning them. It is also the economic question UniverseBlend examined in GPU Cloud Pricing 2026. Larger batches lower cost per token and raise per-request latency, so the cheapest configuration is rarely the most responsive one.

How to run a fair LLM inference latency test

No public benchmark matches your region, prompts and traffic. A minimal test you can run yourself:

Run the client in the cloud region your users or servers actually use.
Build prompts at your real input sizes, with three lengths minimum, and unique content per request so caching does not flatter results (then test cached prefixes separately).
Fix output length with max_tokens and record the actual tokens returned.
Set reasoning effort, temperature and tool schemas exactly as production will.
Stream every request and timestamp each chunk on the client.
Run at least a few hundred requests per configuration, spread across days and hours, at your expected concurrency.
Report p50 and p95 for TTFT, inter-token gap and total time, plus the error and timeout rate.

import time

def measure(client, prompt, **settings):
    t0 = time.perf_counter()
    stamps = []
    for chunk in client.stream(prompt, **settings):   # provider SDK streaming call
        if chunk.text:                                  # ignore empty/keep-alive chunks
            stamps.append(time.perf_counter())
    ttft = stamps[0] - t0
    gaps = [b - a for a, b in zip(stamps, stamps[1:])]
    total = stamps[-1] - t0
    return {"ttft": ttft, "total": total,
            "max_gap": max(gaps, default=0),
            "chunks": len(stamps)}
# Run many times; report p50/p95 of each field, not the mean.

Chunks are not always single tokens, so divide counted output tokens by streaming time for a true rate. This sketch produces no results by itself, and none are reported here.

A practical provider selection framework

Pick metrics by workload before comparing providers.

Criterion	Matters most when	How to judge
TTFT at your input size	Chat, autocomplete, voice	p95 from your region
Sustained output speed	Long answers, reports, code	Tokens per second after the first token
End-to-end completion	Anything parsed as a whole (JSON, tools)	p50 and p95 total time
Variability	User-facing products	Gap between p50 and p95; largest mid-stream pause
Concurrency and rate limits	Production traffic	Latency at expected parallelism; see AI API rate limits in 2026
Tool-call overhead	Agents	Per-step time multiplied by expected steps
Cost at required quality	All	Cost per completed task at the reasoning setting you need

A single fastest-model ranking misleads in both directions. It can reward an endpoint that is fast because it thinks less, and penalise a model whose latency comes from a reasoning setting you would never use. Fix quality first, then compare latency among the options that meet it.

What the available evidence cannot tell us

The public data has clear gaps. It has no tail latency (medians only), one client location, mostly single-request load, and undisclosed hardware and batch sizes. Reasoning effort labels mean different things at different vendors. Frontier-model pages default to maximum effort, which few interactive products would use. Several source pages also show internal inconsistencies, so any figure here could shift by 10% or more when re-read. None of it tells you how a provider behaves during a regional incident or a demand spike, which is when latency matters most.

Conclusion

LLM inference latency becomes a product problem at a specific point: when the time a user or a downstream system must wait crosses the budget your interface was designed around. For chat that is roughly a second to first visible text. For voice it is hundreds of milliseconds. For agents it is the sum of every step, at the 95th percentile.

The evidence says the model name predicts that wait poorly. Host choice moved Llama 3.3 70B’s output speed 19-fold, and one reasoning parameter moved Claude Sonnet 5.5 from under a second to about eight minutes. Treat published rankings as a shortlist, then measure the two or three candidates yourself, with your prompts, in your region, at your load.

Frequently Asked Questions

What is a good LLM inference latency for a chatbot?

There is no universal standard. A common engineering guideline, based on Nielsen’s one-second limit for uninterrupted thought, is first visible text within about one second, with steady streaming after that. Measure p95, not just the median.

What is time to first token (TTFT)?

TTFT is the time from sending a request to receiving the first token of the response, measured at the client. It includes network time, queueing and prompt processing. For reasoning models it may also include thinking time, depending on whether the measurement waits for the first answer token.

Is TTFT or output tokens per second more important?

It depends on response length. Short replies are dominated by TTFT, and long documents by output speed. Estimate total time as TTFT plus output tokens divided by output speed for your typical response.

Why does the same open model have different latency on different hosts?

Hosts use different hardware, quantization, batching and data-centre locations. On Artificial Analysis’s October 2026 data, Llama 3.3 70B ran at 17 to 328 output tokens per second depending on host.

Why do reasoning models seem so slow in benchmarks?

Many benchmarks report time to the first answer token at the highest reasoning effort, which includes minutes of thinking. Lower effort settings respond in seconds. Compare models at the effort level you would actually use.

Does streaming reduce latency?

Streaming reduces the time until users see output, not the time to generate the full response. It helps chat, but does nothing for tool calls or structured output that must arrive complete.

Does prompt caching reduce latency?

It can reduce TTFT when a long prompt prefix repeats. Anthropic documents improved time-to-first-token for long cached documents and no effect on output generation speed. Gains depend on cache hits within the cache lifetime.

How do I benchmark LLM inference latency fairly?

Test from your production region, with your real prompt sizes and settings, streaming, at expected concurrency, over hundreds of requests across several days. Report p50 and p95 for TTFT and total time, plus errors.

Keep reading

Here are the latest posts from the blog.

Claude API Rate Limits in 2026: How Many Requests Can You Run?

November 5, 2026

Claude API rate limits on the Start tier allow most current models 2,000,000 input tokens and 400,000 output tokens per minute, with 1,000 or 4,000 …

Enterprise AI Spending Hit $177B. Agents Are Driving the Bill

November 4, 2026

Figures checked against Menlo Ventures’ 2026 State of Enterprise AI report on 4 November 2026. Market sizes are Menlo estimates, shown as annualised run rates. …

Machine Unlearning: Can We Really Trust an Edited AI Model?

November 4, 2026

30 Agentic AI Statistics for 2026: Adoption, Spending and Risk

November 3, 2026

Last updated: 3 November 2026. This page is updated monthly; each figure keeps its original publication date. Agent traffic at one healthcare booking platform rose …

AI API Rate Limits in 2026: The Numbers That Bind

by Idrees Patel

Two teams call the same model on the same tier. One runs a support chatbot and never sees a 429. The other runs a five-step agent loop over long documents and starts getting throttled on day one. Same account, same limits, different binding constraint.

That is the useful thing to understand about AI API rate limits in 2026, and it is the thing the comparison tables usually miss. Every provider enforces several ceilings at once. The one you hit depends on the shape of your traffic, not on which number is biggest.

The short version: OpenAI enforces RPM, TPM, RPD and TPD at the organization and project level, with four tiers gated on cumulative credit purchases. Anthropic splits tokens into input and output, caps them separately, and — crucially — does not count cached input tokens toward the input limit on most models. Google publishes a tier structure but has stopped publishing per-model numbers in the docs, pointing you to AI Studio instead. AWS does not do tiers at all; Bedrock quotas are per account, per region, per model, token-burndown based, and now split across two separate endpoints.

None of those providers publishes a concurrency limit for standard text inference. That absence matters more than most of the published numbers, and it comes up later.

Limits last checked: 9 October 2026. Every figure below links to the provider’s own documentation. Treat them as a snapshot — OpenAI’s tier names changed this year, and Google’s published tables disappeared sometime before September.

Key takeaways

Rate limits are plural. OpenAI enforces up to six dimensions, Anthropic three plus spend caps, Gemini three plus a rolling ten-minute spend limit, Bedrock token-based quotas split across two endpoints.
Cached input tokens do not count toward ITPM on most Claude models. At an 80% hit rate, a 2M ITPM limit processes around 10M total input tokens per minute, which makes cross-provider TPM comparisons close to meaningless.
No provider publishes a concurrency limit for standard text inference. Derive yours from RPM and request duration, and enforce it in your client.
A 429 with no retry-after on Anthropic means a spend cap, not a rate limit. Retrying cannot clear it.
Google no longer publishes per-model Gemini numbers in its documentation. AI Studio is the source of truth for your project.
Limits attach above the key: organization and project on OpenAI, organization on Anthropic, project on Gemini, account and region on Bedrock. Minting more keys does not add capacity.
OpenAI charges TPM against the maximum of max_tokens and its own estimate. Anthropic ignores max_tokens for OTPM. The same parameter is a cost on one platform and free on the other.

Quick Navigation

Quick Referene: AI API Rate Limits, October 2026
What the Numbers Actually Mean
The Table Nobody Publishes
OpenAI Rate Limits
Anthropic Rate Limits
Google Gemini Rate Limits
AWS Bedrock Quotas
The Concurrency Problem
What Changes When You Add Agents
When You Hit the Ceiling
Which Limit Matters for Your Workload
FAQ

Quick Referene: AI API Rate Limits, October 2026

The providers do not expose comparable quota systems, so this table does not pretend they do.

Provider	Scope	Dimensions enforced	Top published standard tier	Concurrency limit	How you move up
OpenAI	Organization and project	RPM, RPD, TPM, TPD, IPM, audio minutes	Grow: 15,000 RPM / 40M TPM (Astra, Sol, Terra); 30,000 RPM / 180M TPM (Luna)	Not published	Automatic at $500 total credit purchases
Anthropic	Organization, with optional workspace sub-limits	RPM, input TPM, output TPM	Scale: 10,000 RPM / 10M ITPM / 2M OTPM (Opus 5.5, Sonnet 5.5, Haiku 5.5)	Not published	Automatic on usage history; request increase in Console
Google Gemini	Project, not API key	RPM, input TPM, RPD, plus IPM/TPD on some models	Not published in docs; shown in AI Studio per project	Not published for standard; 100 concurrent batch jobs	Tier 2 at $100 paid + 3 days; Tier 3 at $1,000 paid + 30 days
AWS Bedrock	AWS account, per region, per model, per endpoint	Tokens per minute, requests per minute, varies by quota	Varies; see AWS General Reference	Not published as a single figure	Service Quotas increase request

Anthropic’s Scale tier and OpenAI’s Grow tier look roughly comparable on RPM. They are not comparable on tokens, because the two companies count tokens differently — which is the first thing worth explaining.

What the Numbers Actually Mean

Six different things get called a rate limit.

RPM counts requests in a rolling minute. It ignores how big each request is. OpenAI’s docs give the cleanest illustration: 20 requests of 100 tokens each will exhaust a 20 RPM limit even though you sent 2,000 tokens against a 150,000 TPM allowance.
TPM counts tokens. Whether that means input tokens, output tokens, or both depends entirely on the provider, and this is where cross-provider comparison breaks down. Google counts input tokens. Anthropic runs two separate counters, ITPM and OTPM. OpenAI’s TPM covers the request as a whole.
RPD and TPD are daily caps. Gemini’s RPD resets at midnight Pacific. OpenAI applies RPD and TPD on some models, mostly at lower tiers.
Spend limits are a different animal and are easy to mistake for throttling. Anthropic’s tier spend caps ($500 Start, $1,000 Build, $200,000 Scale) stop API usage until the first of the next month when you hit them, returning a 429 with rate_limit_error but no retry-after header. The distinguishing field is error.details.error_code, which reads enforced_spend_limit_reached. Retrying will not help, and SDK auto-retry will burn through its attempts for nothing. Gemini enforces a spend-based limit on a rolling ten-minute window: $10 at Tier 1, $50 at Tier 2, $200 at Tier 3.
Acceleration or ramp limits throttle how fast your traffic grows rather than how much of it there is. OpenAI returns 429 with code slow_down for this, and states plainly that it can fire while you are inside your RPM and TPM. Their guidance: past roughly 1M input TPM, increase by no more than 50% every 15 minutes. Anthropic documents the same behaviour for sharp usage increases.
Concurrency is the number of requests in flight simultaneously. None of the four providers publishes one for standard text inference. More on that below, because it is the limit that most often surprises people.

Two more distinctions worth holding onto.

Scope. OpenAI applies limits at organization and project level, not per user or per key. Anthropic applies them per organization, with optional lower limits per workspace. Google applies them per project, explicitly not per API key — so rotating keys inside one project buys you nothing. Bedrock applies them per AWS account, per region.
Replenishment. Anthropic uses a token bucket, so capacity refills continuously rather than resetting on the minute. The practical consequence is that a 60 RPM limit may behave like one request per second, and a burst of 60 at the top of the minute can fail even though the per-minute figure says it should not.

The Table Nobody Publishes

Here is a thing I expected to find and did not: a documented maximum concurrent requests figure for standard text inference at any of the four providers.

OpenAI publishes RPM, RPD, TPM, TPD, IPM and audio minutes per minute. No concurrency. Anthropic publishes RPM, ITPM and OTPM, plus a batch processing queue limit. No concurrency for the Messages API. Google publishes concurrent batch requests (100) and Live API concurrent sessions, but nothing for standard generation. AWS publishes a long list of quotas per model and endpoint; concurrency appears for some async and provisioned-throughput operations, not as a general figure.

That is not an oversight. For a synchronous API, RPM plus request duration effectively determines concurrency, and the provider’s capacity management happens elsewhere. But it means your own concurrency ceiling is something you have to derive rather than look up.

OpenAI Rate Limits

The tier names changed. There is no Tier 1 through Tier 5 any more — the three paid tiers are Build, Launch and Grow, with Free below them. Advancement is automatic on cumulative credit purchases.

Tier	Qualification	Monthly usage limit
Free	Allowed geography	$100
Build	$5 in total credit purchases	$500
Launch	$100 in total credit purchases	$5,000
Grow	$500 in total credit purchases	$200,000

The published standard rate limits, from OpenAI’s rate limits guide:

Tier	Models	RPM	TPM
Build	Astra, Sol, Terra	5,000	1,000,000
Build	Luna	5,000	2,000,000
Launch	Astra, Sol, Terra	10,000	4,000,000
Launch	Luna	10,000	10,000,000
Grow	Astra, Sol, Terra	15,000	40,000,000
Grow	Luna	30,000	180,000,000

No Free-tier numbers appear in that table. If you are on Free, the dashboard at Settings → Organization → Limits is the only source.

Five things in the documentation matter more than the table.

Shared limits across model families. Models grouped under a shared limit on your organization’s limits page draw from one pool. OpenAI’s example: a 3.5M shared TPM is consumed by calls to any model in that group. Routing between two models in the same family does not buy headroom.
Long-context requests are limited separately. Long context models such as GPT-5.5 carry a distinct rate limit for long-context requests, visible only in the console. A RAG pipeline with large prompts can be throttled by a limit that does not appear in the standard table.
Your TPM is charged against the larger of two numbers. The limit calculation uses the maximum of max_tokens and an estimate derived from your request’s character count. Setting max_tokens to 4096 out of habit on a workload that returns 200 tokens inflates your token consumption for limit purposes. That is a one-line fix with real headroom attached.
Vector store ingestion has its own ceiling. 300 requests per minute per vector store ID, shared across the files and file_batches endpoints. Use file_batches for bulk ingest.
Two distinct failure codes. 429 with slow_down means your rate grew too quickly. 503 with server_is_overloaded means the model is saturated. Handling both matters: the SDKs map them to different exception types, and video endpoints changed which code they return this year. Follow Retry-After where present; where absent, back off with jitter.

Batch API queue limits are counted in queued input tokens per model, and tokens from pending jobs count until the job completes. Enterprise traffic that keeps tripping ramp limits can move to Scale Tier, or Reserved Tier for GPT-5.6 and later.

Anthropic Rate Limits

Three published tiers — Start, Build, Scale — plus a Custom tier arranged with an account team. Placement is automatic based on usage history, and new organizations may start in an Evaluation tier with limits below the published Start numbers while account history builds.

From Anthropic’s rate limits documentation, for the current flagship models (Opus 5.5, Opus 5, Sonnet 5.5, Sonnet 5, Haiku 5.5, Haiku 4.5 — all share the same figures):

Tier	RPM	ITPM	OTPM	Monthly spend cap
Start	1,000	2,000,000	400,000	$500
Build	5,000	5,000,000	1,000,000	$1,000
Scale	10,000	10,000,000	2,000,000	$200,000

Claude Fable 5.x runs lower: 1,000 RPM / 500K ITPM / 100K OTPM at Start, rising to 4,000 / 4M / 800K at Scale. Fable 5.1 and Fable 5 share one combined bucket; Mythos 5.1 and Mythos 5 share a separate one. Opus 4.x models (4.8 through 4.5) share a combined bucket that Opus 5.5 and Opus 5 are not part of — each of those has its own limit. Same pattern for Sonnet 4.x versus Sonnet 5.5 and Sonnet 5.

The detail that changes capacity planning: cached input tokens do not count toward ITPM on most models.

Specifically, input_tokens and cache_creation_input_tokens count; cache_read_input_tokens does not. Anthropic’s own worked example: a 2,000,000 ITPM limit with an 80% cache hit rate processes roughly 10,000,000 total input tokens per minute, because the 8M cached tokens are free of the limit. Claude Haiku 3.5 is the exception and does count cache reads.

For an agent re-sending a large stable prefix every turn, that is the difference between a workable deployment and constant throttling. It also means the ITPM headline number is not comparable to OpenAI’s TPM at all.

A related gotcha in the response shape: input_tokens only counts tokens after your last cache breakpoint. With a 200k cached document and a 50-token question, you will see input_tokens: 50. Total input is cache_read + cache_creation + input_tokens.

On the output side, OTPM is measured on tokens actually produced. max_tokens does not factor in, so there is no limit-based reason to set it conservatively — the opposite of OpenAI’s behaviour.

Other limits worth knowing:

Message Batches API: separate RPM (1,000 / 2,000 / 4,000 by tier), a processing-queue cap (200K / 300K / 500K batch requests), and 100,000 requests per batch at every tier.
Managed Agents endpoints: 300 RPM for creates, 1,200 RPM for reads, per organization, separate from Messages.
Fast mode on Opus 5.5, Opus 5 and Opus 4.8 has dedicated limits separate from standard Opus, surfaced through anthropic-fast-* headers.
Workspaces can be given lower limits than the organization, and organization limits always apply even if workspace limits sum to more.
Inference geo does not split the pool: us and global draw from the same limits.
Claude Platform on AWS uses these same rate limits but bills through AWS Marketplace, starts organizations on Start tier, and does not offer the self-service tier increase flow or per-workspace configuration.

Google Gemini Rate Limits

Google’s documentation has changed in a way that matters for anyone writing or reading comparison articles: the per-model RPM, TPM and RPD tables are no longer in the docs. The rate limits page, last updated 2 September 2026, says limits depend on factors including your usage tier and can be viewed in Google AI Studio, then links to a per-project rate limit view. It also adds that specified rate limits are not guaranteed and actual capacity may vary.

So any article quoting you a universal Gemini number for a current model is quoting a snapshot of someone else’s project. Including this one — which is why there are no Gemini RPM figures in the reference table above.

What Google does publish is the structure, and the structure is informative.

Usage tier	Qualification	Billing tier cap	Spend limit per 10 min
Free	Active project or free trial	N/A	N/A
Tier 1	Billing account linked	$250	$10
Tier 2	$100 paid + 3 days from first payment	$2,000	$50
Tier 3	$1,000 paid + 30 days from first payment	$20,000–$100,000+	$200

Tier 2 and 3 qualification is based on cumulative spend on Google Cloud services for the linked billing account, not just Gemini. Upgrades from Free to Tier 1 are effectively instant; later ones take up to ten minutes.

The spend-based limit is the one to watch, because it is unusual. It is evaluated on a rolling ten-minute window, and hitting it returns 429 RESOURCE_EXHAUSTED — the same status as a conventional rate limit. A burst of expensive long-context calls at Tier 1 can exhaust $10 of spend in ten minutes without going anywhere near a token ceiling.

Other published specifics:

Limits apply per project, not per API key. Minting more keys in the same project achieves nothing.
RPD quotas reset at midnight Pacific.
TPM counts input tokens. Some models carry IPM (images per minute) or TPD instead of or alongside the standard dimensions.
Experimental and preview models are limited more tightly than stable ones.
Priority inference gets 0.3x the standard rate limit per model and tier by default, while still counting against overall interactive traffic.
Batch API: 100 concurrent batch requests, 2GB input file limit, 20GB file storage, and per-model enqueued-token caps that do appear in the docs — Gemini 3.8 Flash runs 3M enqueued tokens at Tier 1, 400M at Tier 2, 1B at Tier 3.

If you need current interactive numbers for your project, AI Studio’s rate limit page is now the authoritative source rather than the documentation.

AWS Bedrock Quotas

Bedrock does not have tiers. It has AWS Service Quotas, and that difference runs deeper than terminology.

From the Bedrock quotas documentation:

Quotas are per account, per region, per model. The same model in two regions carries two separate allowances. This is genuinely useful — region is a capacity lever on Bedrock in a way it is not on the other three — and genuinely annoying, because it means your quota picture is a matrix rather than a list.
Model inference is controlled by quotas on token usage, with AWS noting that some models consume tokens at a higher rate. There is a separate documentation page on token burndown, which is the mechanism to read if your usage does not match your arithmetic.
There are now two inference endpoints with separate quota pools. bedrock-runtime and bedrock-mantle each have their own per-model allocations, and traffic to one does not count against the other even when you are calling the same underlying model. If you are migrating between endpoints, your effective capacity changes without any quota being modified.
Defaults are not fixed. AWS states that the default quotas assigned to an account may be updated depending on regional factors, payment history, fraudulent usage, or an approved increase request. Two accounts in the same region can legitimately have different numbers.

Because of all that, the honest entry for Bedrock in any comparison table is “varies by model, region, account and endpoint — check Service Quotas.” The per-model values live in the AWS General Reference and in the Service Quotas console for your own account, where you can also see which quotas are adjustable.

One planning note. Because quotas are regional and some are adjustable, Bedrock rewards capacity planning that the direct APIs do not: spreading a workload across regions is a legitimate way to raise aggregate throughput, subject to data residency and latency. On OpenAI, Anthropic and Gemini, the equivalent move — more keys, more projects — mostly does not work, because limits attach to the organization or project above the key.

The Concurrency Problem

RPM is a rate. Concurrency is a count of things in flight. The two are related by how long a request takes, and that relationship is where capacity planning usually goes wrong.

The arithmetic, which is just Little’s Law:

Sustainable concurrency = (RPM ÷ 60) × average request duration in seconds

Illustrative, using a Build-tier OpenAI figure of 5,000 RPM. At 5,000 RPM you can start about 83 requests per second. If each request takes 2 seconds, roughly 167 are in flight at steady state. If your requests are reasoning-heavy and take 40 seconds, the same RPM implies about 3,300 concurrent requests — and your own infrastructure has to be able to hold 3,300 open HTTP connections, with the memory and file descriptors that implies.

Run it the other way and the trap appears. A worker pool of 200 threads, each making a 40-second call, starts only 5 requests per second — 300 RPM. You are using 6% of a 5,000 RPM allowance and your throughput is capped by your own pool size. The provider is not throttling you. You are.

The inverse case is the one that produces 429s nobody expected. Fire 500 requests simultaneously from an async client on a 1,000 RPM Anthropic Start tier, and the per-minute figure says you are fine. The token bucket says otherwise: capacity replenishes continuously, so a 1,000 RPM allowance behaves closer to 16 requests per second. Anthropic documents this directly, noting that a 60 RPM limit might be enforced as 1 request per second and that short bursts can trigger errors.

Four places concurrency bites in practice:

Async clients with no semaphore. asyncio.gather over a list of 1,000 items will try to open 1,000 connections. Nothing in the SDK stops it.
Retry storms. A burst gets throttled, every failed request retries at the same moment, the retry burst is larger than the original. This is what jitter exists to prevent, and it is why OpenAI explicitly warns that unsuccessful requests still count against your per-minute limit.
Long-running calls holding tokens. On OpenAI, TPM is charged using the maximum of max_tokens and an estimate — so concurrent long-context requests can reserve a large share of your token budget while they run.
Agent fan-out, which deserves its own section.

The practical move is to set an explicit concurrency ceiling in your client, derived from the equation above with headroom, rather than discovering it empirically through 429s. A semaphore is four lines of code and it is the single highest-value thing most teams are missing.

What Changes When You Add Agents

A chat turn is one request. An agent turn is a loop, and the loop multiplies in two directions at once: more calls per task, and more context per call.

Illustrative, not measured. Take a modest research agent: a planning call, a retrieval call, two tool-selection and interpretation rounds, and a final synthesis call. Five model calls per task.

Concurrent users	Calls per task	Requests per task cycle
10	5	50
100	5	500
100, with one retry in ten	5.5	550
100, with three sub-agents on step 3	8	800

The RPM side of that is usually survivable on a paid tier. The token side is where it hurts, because each call in the loop carries the accumulated transcript. Step one sends 2,000 tokens. Step five sends 2,000 plus everything produced in between, which on a document-heavy task can be 50,000 or more.

So the input tokens for a single task are not five times 2,000. They are a growing series, and the last call dominates it.

This is exactly where Anthropic’s cache-aware ITPM changes the calculus. If the growing prefix is cached and the cache is hit, those tokens do not count toward the input limit. The same loop on a provider that counts all input tokens consumes several times the quota for identical work. It is the clearest case I found where two providers’ headline numbers are not measuring the same thing — and where the cheaper-looking option can be the more constrained one. We went through the billing side of that in our analysis of cache-read economics; the rate-limit side is arguably more consequential, because you cannot buy your way past a ceiling mid-request.

Three more agent-specific pressures:

Fan-out concurrency. A supervisor delegating to four sub-agents in parallel turns one user action into four simultaneous requests. Ten users doing that is 40 in flight, and they arrive in a spike rather than spread across the minute.
Tool loops that do not terminate. An agent retrying a failing tool call burns requests and tokens with no output to show for it. Rate limits are the place this becomes visible, usually before the cost dashboard catches up.
Shared organization limits. Your production agents, your evaluation runs and the developer testing a new prompt all draw from the same pool. Anthropic’s workspace limits and OpenAI’s project-level limits exist for exactly this reason: use them to prevent a batch eval from starving production.

Agent permissions and agent capacity turn out to be the same design problem viewed from two angles — which is a theme we have covered from the security side in our work on testing agents against prompt injection.

When You Hit the Ceiling

First, read the error properly. A 429 is not one condition.

Signal	What it is	Right response
429 with `retry-after`	Conventional rate limit	Wait at least that long, add jitter, retry
429, OpenAI code `slow_down`	Traffic ramped too fast	Reduce rate, then increase gradually
429, Anthropic `enforced_spend_limit_reached`	Monthly spend cap	Retrying cannot work; raise the cap or tier
429 `RESOURCE_EXHAUSTED` on Gemini	Rate or spend-based limit	Back off; check the 10-minute spend window
503 `server_is_overloaded`	Model saturated, not your fault	Retry with longer delays
400 on an Anthropic self-set spend limit	Your own configured cap	Change the setting

The distinction between the second and third rows is where real production time gets lost. A spend-cap 429 carries no retry-after, and SDK auto-retry will exhaust its attempts against a wall. Inspect error.details.error_code before retrying anything.

Then, in rough order of how much headroom they buy relative to effort:

Cache aggressively, especially on Anthropic. Cached reads do not count toward ITPM on most Claude models. Nothing else on this list multiplies effective throughput by five.
Set an explicit concurrency ceiling. A semaphore sized from the Little’s Law calculation, not from whatever your event loop will happily attempt.
Fix max_tokens on OpenAI. Your TPM charge uses the maximum of max_tokens and the estimate from your request. Padding it inflates consumption for no benefit.
Move eligible work to batch. OpenAI’s Batch API does not touch synchronous rate limits. Anthropic’s Message Batches have their own pool. Gemini’s batch quotas are separate and generous — 400M enqueued tokens for Gemini 3.8 Flash at Tier 2. Anything that tolerates delay should not be competing with interactive traffic.
Ramp gradually. Past 1M input TPM, OpenAI suggests increasing by no more than 50% every 15 minutes. Cold-starting a large migration at full volume is how you discover acceleration limits.
Route by task, not by habit. Small models carry separate pools and higher limits — OpenAI’s Luna sits at 180M TPM at Grow against 40M for the larger models. Classification and routing steps inside an agent rarely need the flagship.
Separate your pools. Workspaces on Anthropic, projects on OpenAI, projects on Gemini, regions on Bedrock. Keep evaluation and development away from production capacity.
Queue rather than retry. A bounded queue with a worker pool gives you backpressure. Retry loops give you a thundering herd.
Fall back across providers only if your evaluation says the outputs are interchangeable for that step. Fallback that silently degrades quality is a worse failure than a 429.
Then raise the tier. It is on the list, just not at the top — most teams have 2–5x of headroom available from caching and configuration before they need to spend anything.

Which Limit Matters for Your Workload

Find your shape, then look at the number that actually binds.

Workload	Binds first	Watch
High-volume classification, short in and out	RPM	Request count, not tokens. Batch if latency allows
RAG over large documents	Input TPM, long-context limits	Caching; OpenAI’s separate long-context limit
Reasoning-heavy generation	Output TPM	Anthropic’s OTPM is a fifth of its ITPM at Scale tier
Chat with moderate context	Usually nothing	You will hit spend limits before rate limits
Agent loops with growing context	Input TPM, then concurrency	Cache-hit rate is the lever
Multi-agent fan-out	Concurrency	Bursts, not averages
Bulk offline processing	Batch queue limits	Enqueued tokens, not RPM
Expensive calls at low volume	Spend limits	Gemini’s 10-minute window; Anthropic’s monthly cap

The diagnostic sequence when something throttles: read the error code, check whether it carries retry-after, then look at which dimension the response headers say you exhausted. OpenAI and Anthropic both return remaining-capacity headers — Anthropic separates input and output token headers, and reports the most restrictive limit currently in effect. Those headers answer the question faster than any dashboard.

FAQ

What are AI API rate limits?

Caps a provider places on how much you can call its API in a period. They are enforced across several dimensions at once — requests per minute, tokens per minute, requests or tokens per day, and in some cases spend — and exceeding any one returns an error even if the others have headroom.

What is the difference between RPM and TPM?

RPM counts calls regardless of size. TPM counts tokens regardless of how many calls carried them. Twenty tiny requests can exhaust a 20 RPM limit while using a fraction of a 150,000 TPM allowance, which is OpenAI’s own example.

Which AI API has the highest rate limits?

On published standard tiers as of October 2026, OpenAI’s Grow tier lists the largest token figures: 40M TPM for Astra, Sol and Terra, and 180M for Luna. Anthropic’s Scale tier lists 10,000 RPM with 10M ITPM and 2M OTPM. The comparison is not clean, because Anthropic excludes cached input tokens from ITPM and OpenAI does not exclude them from TPM.

How do AI API concurrency limits work?

None of the four providers publishes a concurrency figure for standard text inference. Your effective concurrency is set by your rate limit and how long each request takes: roughly (RPM ÷ 60) × seconds per request. Anthropic’s token bucket also means a per-minute limit behaves like a per-second one for bursts.

How do I increase my AI API rate limit?

OpenAI upgrades tiers automatically on cumulative credit purchases ($5, $100, $500). Anthropic moves organizations up on usage history and offers a Request tier increase flow in the Console. Gemini upgrades on cumulative Google Cloud spend plus a waiting period. Bedrock requires a Service Quotas increase request, per region.

Why does my AI agent hit rate limits so quickly?

Because one task becomes many calls, and each call carries the accumulated context. Five calls per task across a hundred users is five hundred requests, with the last call in each loop carrying the largest prompt. Token limits usually bind before request limits.

How should I handle rate-limit errors?

Check the error code first. If retry-after is present, wait at least that long and add jitter. If the code indicates a spend cap, retrying will not work. If it indicates ramp speed, reduce your rate and increase gradually rather than retrying at the same volume.

Do API limits differ by model?

Yes, substantially, and some models share pools. OpenAI groups model families under shared limits and applies a separate limit to long-context requests. Anthropic gives Opus 5.5 and Opus 5 their own limits while Opus 4.x models share one bucket. Gemini limits preview and experimental models more tightly than stable ones.

How do I calculate the capacity I need?

Measure input and output tokens per request separately, multiply by calls per task and tasks per minute at peak, then compare against each dimension individually. Add your cache-hit rate if you are on Anthropic, since cached reads do not consume ITPM on most models.

Keep reading

Here are the latest posts from the blog.

Claude API Rate Limits in 2026: How Many Requests Can You Run?

November 5, 2026

Claude API rate limits on the Start tier allow most current models 2,000,000 input tokens and 400,000 output tokens per minute, with 1,000 or 4,000 …

Enterprise AI Spending Hit $177B. Agents Are Driving the Bill

November 4, 2026

Figures checked against Menlo Ventures’ 2026 State of Enterprise AI report on 4 November 2026. Market sizes are Menlo estimates, shown as annualised run rates. …

Machine Unlearning: Can We Really Trust an Edited AI Model?

November 4, 2026

30 Agentic AI Statistics for 2026: Adoption, Spending and Risk

November 3, 2026

Last updated: 3 November 2026. This page is updated monthly; each figure keeps its original publication date. Agent traffic at one healthcare booking platform rose …

Isaac ROS 5.0 and the Robotics Agent Boundary Problem

by Idrees Patel

One of the agent skills NVIDIA published alongside Isaac ROS 5.0 is called submit-and-monitor-mission. Its documented sample prompt reads: submit a route mission to carter01 and prove it actually completed.

That is not autocomplete. That is a coding assistant issuing a task to a robot in a fleet and verifying the outcome, from a chat window.

NVIDIA released Isaac ROS 5.0 on September 21, 2026, announcing it at ROSCon in Toronto the following day. Most coverage has focused on the ROS 2 Lyrical migration and the NITROS rewrite, both of which matter to anyone maintaining an Isaac ROS workspace. The more interesting change is quieter: robotics development tasks are now packaged as structured skills that AI coding assistants can invoke.

Which raises a question the release notes do not answer, because it is not NVIDIA’s to answer. When an agent works on software that eventually moves actuators, what should it be allowed to reach?

Key takeaways

Isaac ROS 5.0 ships AI agent skills in the open Agent Skills format, including isaac-ros-activate for the development environment and an early-access skill for migrating nodes off NITROS.
Separately, the Mission Control documentation publishes five skills that reach further, including submitting missions to robots and changing fleet composition.
The release moves to ROS 2 Lyrical Luth on Ubuntu 24.04 and rebuilds the accelerated data path on rosidl::Buffer. Code calling NITROS APIs directly needs source-level migration.
Isaac ROS 5.0 does not give an agent control of a robot. It gives an agent structured access to the toolchain that produces robot software, which is a different thing with its own permission questions.
The useful boundary in agentic robotics is not the IDE. It runs across repository, build, simulation, middleware, deployment and hardware, and each of those deserves a separate permission decision.

Quick Navigation

What Isaac ROS 5.0 Actually Adds
From Code Assistant to Robotics Toolchain Agent
The Isaac ROS 5.0 Boundary Question Is Not the IDE
What Does Egress Control Look Like in a Workcell?
Simulation Is a Boundary, Not a Guarantee
Failure Modes Change When Code Reaches Hardware
How to Evaluate Agentic Development in an Isaac ROS 5.0 Workflow
What Isaac ROS 5.0 Means for Physical AI Teams
Questions to Ask Before an Isaac ROS 5.0 Agent Touches a Robot
Frequently Asked Questions

What Isaac ROS 5.0 Actually Adds

Isaac ROS is NVIDIA’s collection of CUDA-accelerated packages built on open-source ROS 2. It is not a replacement for ROS 2 and not a robot controller. It supplies GPU-accelerated perception, localisation, planning and deployment packages that run inside a ROS 2 graph.

Four changes define the 5.0 release, all documented in NVIDIA’s release notes.

ROS 2 Lyrical Luth becomes the supported distribution, with a Buildfarm apt repository carrying ROS 2 Lyrical ecosystem packages for Ubuntu 24.04 Noble.
The accelerated data path was rebuilt. NITROS now sits natively on ROS 2 Lyrical’s rosidl::Buffer with the CUDA buffer backend, letting accelerated nodes exchange standard ROS messages whose array fields can be backed by GPU memory. The NITROS packages themselves were removed. Code that called NITROS APIs or types directly requires source-level migration; isaac_ros_nitros_bridge_ros2 survives as a deprecated bridge marked for removal.
AI agent skills arrive in the open Agent Skills format. The Isaac ROS CLI ships isaac-ros-activate for activating the development environment, plus an early-access migrate-node-to-rosidl-buffer skill for moving a node off the NITROS APIs. NVIDIA says further Isaac skills are published in the nvidia/skills catalog under the Physical AI category.
GPU partitioning became a package. isaac_ros_gpu_partitioning assigns a fixed share of a GPU’s streaming multiprocessors to each ROS 2 process using CUDA Multi-Process Service. NVIDIA is explicit about its limits: it does not partition GPU memory and does not isolate workloads.

That last sentence is worth noting, because it is the kind of precision the rest of this discussion needs. Partitioning compute is not isolation, and NVIDIA says so plainly rather than letting readers assume otherwise.

The release also renames isaac_ros_visual_slam to isaac_ros_cuvslam, changes the teleop end-effector pose topic type in a way that breaks existing subscribers, and carries a frank list of limitations — RealSense cameras supported only in Docker mode, an H.264 encoder segfault on shutdown on Jetson Thor with VPI 4.1.4, cuMotion MoveIt examples that may fail against robot vendor packages not yet certified for Lyrical.

A migration, in other words. A substantial one.

From Code Assistant to Robotics Toolchain Agent

The difference between autocomplete and agentic development is not the quality of the code. It is whether the tool acts.

Autocomplete proposes text and a human commits it. An agentic workflow runs commands, reads output, decides what to do next, and repeats until a stated goal is met or it gives up. The skill format is what makes that repeatable: it packages a procedure, its preconditions and its verification steps so the agent follows a known path instead of improvising one.

NVIDIA’s Mission Control documentation gives the clearest picture of what that looks like in robotics. It lists five published skills, installable through a single npx skills add command, and states they work with Claude Code, Cursor, Codex and other Agent Skills clients.

Skill	What it does
`bring-up-cloud-stack`	Starts the Mission Control cloud services from the repository compose stack
`change-map`	Changes the map Mission Control loads, including from an uploaded file, URI or S3 object
`change-fleet-composition`	Adds, removes, renames or relabels robots in the fleet
`change-stack-version`	Changes the image tag of any Mission Control service
`submit-and-monitor-mission`	Submits a mission and verifies it completed end to end

Read that list as a permission surface rather than a feature list. Starting services, changing which map a fleet navigates against, altering fleet membership, pinning service versions, dispatching missions. In a conventional software context these would be infrastructure operations requiring review. Here they are chat prompts.

None of this is hidden or oversold. NVIDIA documents each skill with a sample prompt and notes the combination case: change the map, add an AMR, restart the stack, run a health check. It reads exactly like a DevOps runbook, which is the point.

The Isaac ROS 5.0 Boundary Question Is Not the IDE

The question worth asking is not whether an AI agent can write ROS 2 code. It can, adequately, and Isaac ROS 5.0 makes it better at the Isaac-specific parts by giving it documented procedures.

The question is what the agent can reach after the code exists.

Here is the chain, and it helps to keep the layers distinct, because collapsing them into “AI controls robots” produces bad reasoning in both directions:

AI coding agent — writes and modifies source, runs commands, reads results. Has no real-time guarantees and no place in a control loop.
Isaac ROS — GPU-accelerated ROS 2 packages for perception, localisation and planning.
ROS 2 middleware — the message transport connecting nodes.
Robot autonomy stack — the assembled graph that decides what the robot does.
Robot controller — the real-time layer commanding joints and wheels, typically with its own safety functions.
Isaac Sim — the simulator, a separate application from Isaac ROS.

An agent editing a launch file is operating at layer one. The effect of that edit can propagate to layer five. The propagation is not instantaneous or automatic, and the gates between layers are where the engineering decision sits.

Risk changes at each step:

Boundary	What the agent does	Worst realistic outcome
Source code	Edits packages, configs, launch files	A bad change sits in a branch
Dependencies	Installs packages, changes versions	Supply-chain exposure, broken builds
Build	Compiles the workspace	Wasted time; a binary that behaves differently than reviewed
Simulation	Launches Isaac Sim scenarios	Misleading pass results
Middleware	Publishes to live ROS 2 topics	Interference with a running system
Deployment	Pushes artifacts to robot compute	Untested software on a machine
Hardware	Commands actuators	Physical consequences

The first three are ordinary software risks with ordinary controls. The last three are not, and treating them as one continuous permission level is the mistake worth avoiding.

What Does Egress Control Look Like in a Workcell?

In conventional agentic development, sandboxing means protecting files, credentials, processes and network access. Those concepts map onto robotics, but imperfectly, and the places where the analogy breaks are the interesting ones.

Software agent control	Robotics equivalent	Where the analogy breaks
Filesystem permissions	Workspace, packages, launch and config files	Config changes can alter physical behaviour without touching code
Credential scope	Robot, fleet and deployment credentials	A fleet credential can address many machines at once
Network egress	Which robots, fleet services and telemetry endpoints are reachable	ROS 2 discovery is permissive by default; a node on the same network can find peers
Process execution	ROS 2 nodes and launch files	A launched node publishes to a live graph, not just to stdout
Sandbox	Simulation or an isolated workcell	A simulator is a model, not a copy
Deployment approval	Physical deployment approval	Rollback restores software, not a dropped payload

The closest robotics analogue to network egress control is the set of decisions about what the development environment can address. ROS 2 nodes on a shared network discover each other by default, which means an agent that can launch a node can potentially participate in a graph that includes real hardware unless something prevents it. Domain ID separation, network segmentation and DDS configuration are the mechanisms that do the preventing.

Cloud-connected fleets sharpen this. Mission Control dispatches missions over network services, and the VDA5050 protocol exists precisely so fleet managers can command robots from elsewhere. Once an agent can reach that interface, network reachability is the control that matters, exactly as it is for any other agent workflow — a point we worked through in our piece on testing agents against prompt injection, where reachability rather than instruction quality decided the outcomes.

The mechanisms that can be technically substantiated, without inventing capabilities:

Separate ROS 2 domain IDs and network segments for development, simulation and production
DDS security configuration where the deployment supports it
Credential scoping so development credentials cannot address production fleets
Deployment gates requiring signed, versioned artifacts rather than direct file copies
Simulation-only execution modes for agent workflows
Emergency stop and safety functions implemented independently of the software stack
Rollback to a known-good artifact version

That last pair deserves emphasis. E-stop circuits and safety controllers exist below the ROS layer precisely so that no software fault, agent-introduced or otherwise, can defeat them. NVIDIA documents a safety controller concept within isaac_ros_deploy, with its own troubleshooting guide, which reflects the same layered thinking.

Simulation Is a Boundary, Not a Guarantee

Restricting an agent to simulation is the single most useful control available, and it is genuinely strong: a mistake in Isaac Sim costs compute time, not equipment.

It is also not proof of physical safety, for reasons every robotics engineer already knows and every agentic workflow will rediscover.

Simulators model contact, friction, sensor noise, latency and timing approximately. The sim-to-real gap is a permanent feature of the field, not a bug awaiting a fix. A perception model tuned in simulation encounters real lighting, real motion blur and real camera calibration error. A grasp that succeeds against a simulated object may slip on the real one.

There is a second-order problem specific to agents. An agent optimising for a simulated success metric will find whatever satisfies that metric, including artefacts of the simulation itself. That is reward hacking in a new costume, and it argues for holdout scenarios the agent has not been iterating against.

The practical sequence most teams already use stays intact:

Simulation for broad coverage and fast iteration
Replay against recorded sensor data, which tests perception against real inputs deterministically
Hardware-in-the-loop, with real compute and real timing but constrained motion
Supervised physical testing in a controlled cell
Production deployment

An agent can usefully operate through step two. Steps three onward are where human judgement earns its place, and no part of Isaac ROS 5.0 suggests otherwise.

Failure Modes Change When Code Reaches Hardware

Software failures are usually graded by how loud they are. In robotics they should be graded by where they surface.

Build failure. The workspace does not compile. Harmless, immediate, self-announcing. The best kind.
Configuration failure. A launch parameter, calibration file or frame ID is wrong. This is the quiet one. A wrong camera extrinsic produces a system that runs cleanly and perceives the world offset by several centimetres. Nothing errors. Isaac ROS 5.0’s own release notes carry an instructive example: an occupancy grid localizer launch failure caused by empty frame IDs, fixed by using the named static_transform_publisher arguments Lyrical requires. Configuration is where correct-looking code goes wrong.
Runtime software failure. A node crashes. In a well-designed system a supervisor restarts it or the robot halts. The severity depends entirely on what the node was doing and what depends on it.
Perception failure. The system misinterprets the world — a missed obstacle, a bad pose estimate. It does not announce itself, and downstream components act on the wrong belief with full confidence.
Planning failure. A trajectory is generated that is valid in the planner’s model and wrong in the world, typically because the model of the environment was wrong.
Control failure. The lowest and most consequential layer, which is exactly why safety functions live below it and independent of it.
Deployment failure. The wrong version reaches the robot, or reaches some robots in a fleet and not others. Version skew across a fleet is its own category of problem, and change-stack-version existing as an agent skill is a reason to think about it now rather than later.

The ordering that matters for permissions: failures that announce themselves are cheap; failures that propagate silently into physical action are not. An agent workflow should be designed so its mistakes land in the first category.

How to Evaluate Agentic Development in an Isaac ROS 5.0 Workflow

Benchmark scores on coding tasks say little about whether an agent is safe to run in a robotics toolchain. A complete picture needs measures from three groups.

Did it work?

Task success against a stated goal
Code correctness, by review and by test
Regression rate across the existing test suite
Simulation success, on scenarios the agent has not been iterating against
Reproducibility across repeated runs from the same starting state

How did it behave?

Tool-call accuracy: did it invoke the right commands with the right arguments
Permission violations: attempts to act outside its granted scope, which should be logged whether or not they succeeded
Unsafe configuration attempts: disabled safety parameters, widened velocity limits, bypassed checks
Escalation behaviour: does it stop and ask, or improvise, when blocked

What happens when it is wrong?

Rollback success rate and time to restore a known-good version
Human approval compliance: did anything reach a gated stage without approval
Auditability: can every action be reconstructed from logs afterwards

The second and third groups are the ones benchmarks skip. An agent with a 90% task success rate that occasionally tries to widen a velocity limit to make a test pass is worse than one at 75% that stops and asks. The containment-versus-capability trade is the same one we examined in our analysis of evaluator access and security boundaries, and it resolves the same way: measure what the system does when it cannot get what it wants.

What Isaac ROS 5.0 Means for Physical AI Teams

The release lands differently depending on what you do with it.

Robotics developers get a real migration and a tool that helps with it. Moving off the NITROS APIs is mechanical, repetitive work across many files — precisely the shape of task agents handle well, which is presumably why NVIDIA shipped migrate-node-to-rosidl-buffer alongside the change that requires it. It is marked early access, so treat its output as a first pass to review rather than a finished migration.
AI engineers get an entry point into robotics that does not require learning the whole stack first. This cuts both ways. The skills encode NVIDIA’s intended procedure, which is better than an agent improvising. They also make it easy to operate parts of a system you do not yet understand.
Robotics teams get a permissions question they probably have not formalised. Most teams have conventions rather than enforced boundaries around who can deploy to which robot. An agent that follows documented procedures faster than any human will surface the gap between convention and control.
Platform engineers get the ROS 2 Lyrical and Ubuntu 24.04 move, the Buildfarm repository, and GPU partitioning via CUDA MPS — with NVIDIA’s own caveat that partitioning is not isolation.
Safety engineers get the most important reminder in the release: nothing here changes where safety functions live. E-stop circuits, safety controllers and certified components sit below the ROS layer and remain independent of it. Isaac ROS 5.0 does not give an AI agent control of a robot. It gives an agent structured access to the toolchain that produces robot software.

Those are different claims, and the distinction is worth defending whenever the coverage blurs it.

Questions to Ask Before an Isaac ROS 5.0 Agent Touches a Robot

Ten questions. If your team cannot answer them, the answer to “should we give an agent access” is not yet.

What can the agent read? Source, configs, calibration files, recorded data, credentials?
What can it modify? Which repositories, packages, launch files and parameter files, and which are off limits?
What can it execute? Build commands, dependency installation, node launches, simulation, deployment scripts?
What can it deploy, and where? Development machines, a test robot, a fleet?
What hardware can it reach? Directly, and indirectly through fleet services and cloud interfaces?
What network destinations can it access? Package repositories, model registries, robot networks, telemetry endpoints?
What happens when simulation and reality diverge? Who notices, and what stops the change from progressing?
Where is human approval mandatory? Name the specific gates rather than the principle.
Can every action be audited? Commands, file changes, deployments, and refused attempts.
Can changes be rolled back? Including configuration and calibration, not just code.

There is no universal answer to where the approval boundary belongs. A mobile robot in a warehouse aisle, a manipulator behind a fence and a humanoid working near people have different consequence profiles, and the right gate depends on the robot, the environment, the autonomy level and the deployment architecture.

What Isaac ROS 5.0 changes is the urgency. The skills exist, they are documented, they install in one command, and they work with the coding assistants your team already uses. The permission model is now the part of the toolchain that needs designing.

Frequently Asked Questions

What is Isaac ROS 5.0?

Isaac ROS 5.0 is the September 2026 release of NVIDIA’s collection of CUDA-accelerated packages built on open-source ROS 2. NVIDIA tagged the 5.0.0 packages on September 21, 2026 and announced the release at ROSCon in Toronto on September 22. It moves to ROS 2 Lyrical Luth on Ubuntu 24.04, rebuilds the accelerated data path on rosidl::Buffer, and adds AI agent skills.

What are AI agent skills in Isaac ROS 5.0?

They are documented procedures in the open Agent Skills format that AI coding assistants can follow. The Isaac ROS CLI ships isaac-ros-activate for the development environment and an early-access migrate-node-to-rosidl-buffer skill. NVIDIA publishes further skills in the nvidia/skills catalog, and the Mission Control repository publishes five more covering cloud stack startup, map changes, fleet composition, service versions and mission submission.

Does Isaac ROS 5.0 let AI agents control robots?

No. It gives agents structured access to robotics development workflows — environment setup, migration, and Mission Control operations. Robot control remains the job of the autonomy stack and the real-time controller, with safety functions implemented independently of the software stack. What an agent can reach in any given deployment depends on the permissions and network access that team grants it.

What is the relationship between Isaac ROS and ROS 2?

Isaac ROS is built on open-source ROS 2 and runs inside a ROS 2 graph. It supplies GPU-accelerated nodes for perception, localisation, planning and deployment. ROS 2 supplies the middleware, message types and tooling. Isaac ROS 5.0 supports ROS 2 Lyrical Luth.

What is the difference between Isaac ROS and Isaac Sim?

Isaac ROS is a set of ROS 2 packages that run on the robot or on development hardware. Isaac Sim is a separate simulation application. Many Isaac ROS tutorials use Isaac Sim as a data source, but they are distinct products.

Keep reading

Here are the latest posts from the blog.

Claude API Rate Limits in 2026: How Many Requests Can You Run?

November 5, 2026

Claude API rate limits on the Start tier allow most current models 2,000,000 input tokens and 400,000 output tokens per minute, with 1,000 or 4,000 …

Enterprise AI Spending Hit $177B. Agents Are Driving the Bill

November 4, 2026

Figures checked against Menlo Ventures’ 2026 State of Enterprise AI report on 4 November 2026. Market sizes are Menlo estimates, shown as annualised run rates. …

Machine Unlearning: Can We Really Trust an Edited AI Model?

November 4, 2026

30 Agentic AI Statistics for 2026: Adoption, Spending and Risk

November 3, 2026

Last updated: 3 November 2026. This page is updated monthly; each figure keeps its original publication date. Agent traffic at one healthcare booking platform rose …

AI Memory Costs in 2026: HBM, DRAM and the Real Bill

by Idrees Patel

The invoice from your model provider and the quote from your server vendor are moving in opposite directions, and both are telling the truth.

Epoch AI’s benchmark-anchored work puts the decline in inference prices at a median of roughly 50x per year across six benchmarks, rising to around 200x per year when restricted to data after January 2024. Meanwhile, Meta raised its 2026 capital expenditure guidance on 29 April 2026 from $115–135 billion to $125–145 billion, and Mark Zuckerberg pointed to memory pricing as a driver, according to Fortune’s reporting from the call.

Both are true because they sit at different layers. Token pricing is a retail price for an output. Infrastructure cost is what somebody had to buy to produce it. Between those layers sits memory, which spent 2026 becoming the most volatile input in the AI supply chain.

Key takeaways

The 90% quarter is history, and that matters. TrendForce forecast conventional DRAM contract prices rising 90–95% QoQ in 1Q26. By 3Q26 the same firm forecast 13–18%. The rate of increase collapsed; the price level did not come back down.
Server DRAM is now a four-figure line item per module. Seoul Economic Daily reported on 17 September 2026 that the fixed contract price for a 64GB DDR5 server RDIMM stood at $1,500 as of 15 September, against $272 a year earlier.
HBM consumes wafer area out of proportion to the bits it delivers. TrendForce estimates HBM will take roughly 30% of the top three suppliers’ DRAM wafer input by end-2027 while supplying only about 13% of DRAM bits.
A faster accelerator is not automatically a cheaper inference system. Per-chip compute has been growing faster than per-chip bandwidth, which raises the batch size needed to keep silicon busy.
Context length is a cost multiplier that never appears on a rate card. In the illustrative model below, moving from 8k to 32k tokens of context cuts concurrent requests per accelerator roughly four-fold.
Memory inflation moves the rent-or-own crossover, but less than the headlines imply. In the illustrative server model, the utilisation needed to beat the cheapest public cloud rate shifts from roughly 45% to roughly 49%.
Micron reports fiscal Q4 2026 results on 30 September 2026, the first full quarter of guidance after its 84.6% GAAP gross margin quarter. It is the clearest near-term read on whether supply is loosening.

Quick Navigation

Start With the Memory Bill: What AI Memory Costs Look Like in 2026
How HBM Rewrote the Allocation Problem Behind AI Memory Costs
Memory Bandwidth Can Matter More Than FLOPS
- Why batch size sets your AI memory costs
- Context length as a memory cost multiplier
Repricing a Self-Hosted Inference Server: Where AI Memory Costs Land
- Which assumptions move the number most
Rent or Own: How AI Memory Costs Change the Calculation
Why Token Pricing Hides Your Real AI Memory Costs
What to Watch Next in AI Memory Costs
What AI Infrastructure Teams Should Re-Model
Frequently Asked Questions

Start With the Memory Bill: What AI Memory Costs Look Like in 2026

The most quoted number from this cycle is already out of date, and reusing it without context is the fastest way to get the story wrong.

TrendForce’s memory pricing survey of 2 February 2026 revised its 1Q26 forecast for conventional DRAM contract prices upward from 55–60% QoQ to 90–95% QoQ, with server DRAM projected to climb around 90% QoQ, which the firm called the largest quarterly increase on record. That figure gets recycled constantly. It was a first-quarter forecast.

Read the rest of the series and a different shape emerges. TrendForce projected 58–63% QoQ for 2Q26 on 31 March, then 13–18% QoQ for 3Q26 on 3 July, with server DRAM in the same band. On 30 June it raised its 4Q26 PC DRAM forecast to only 3–8% QoQ.

So the rate of increase fell sharply across 2026. The level did not. Conflating the two produces bad procurement decisions in both directions.

Where has the level landed? Seoul Economic Daily reported on 17 September 2026 that as of 15 September, the fixed contract price for a 64GB DDR5 server DRAM module stood at $1,500, roughly 5.5 times the $272 recorded a year earlier, with some spot transactions near $3,100. That tracks a Citi research note dated 12 May 2026 projecting the same module rising from $873 in Q1 2026 to roughly $1,586 by Q4.

The categories are not interchangeable. The $1,500 figure is a fixed contract price for one module type; the $3,100 figure is a spot transaction, and spot diverges widely from contract in tight markets. Neither reflects what a hyperscaler with a long-term agreement pays. TrendForce noted on 9 July 2026 that several US cloud providers had signed multi-year LTAs restricting price increases for those customers, which is why moderation in headline prices does not reach all buyers evenly.

The supplier side confirms the magnitude. Micron’s fiscal Q3 2026, ended 28 May 2026, produced revenue of $41.46 billion against $9.30 billion a year earlier, with GAAP gross margin at 84.6% versus 37.7%. A memory manufacturer earning 85 cents of gross margin on the dollar is the same fact as your RDIMM quote, seen from the other end.

Conventional DRAM and HBM are different products with different pricing mechanics. HBM is negotiated annually rather than quarterly, which is why its contract prices lagged the commodity DRAM surge. That lag is now closing, and the reason has nothing to do with demand for HBM.

How HBM Rewrote the Allocation Problem Behind AI Memory Costs

HBM does not simply compete with conventional DRAM for customers. It competes for wafers, and it is an inefficient consumer of them.

TrendForce estimates that HBM wafer input across the top three suppliers will account for approximately 18%, 22% and 30% of total DRAM wafer input at the end of 2025, 2026 and 2027, while representing only about 8%, 9% and 13% of total DRAM bit supply over the same period. These are TrendForce estimates rather than disclosed manufacturer figures, and should be read as a modelled view of a market whose participants publish very little.

Hold those two series side by side. By end-2027, on this estimate, roughly 30% of wafer starts produce roughly 13% of the bits. HBM stacks DRAM dies vertically using through-silicon vias and a logic base die, and the die area, packaging yield and test burden mean each delivered gigabyte absorbs far more capacity than a DDR5 gigabyte.

That is the crowding-out mechanism in one sentence: every wafer allocated to HBM removes a disproportionate quantity of conventional DRAM from the market.

The twist in 2026 is that the crowding ran in an unexpected direction. TrendForce reported on 2 June 2026 that, on its analysis of per-wafer revenue derived from die size, yield and per-gigabit pricing, HBM wafer revenue was overtaken by DDR5 64GB RDIMM in 1Q26, with HBM profitability falling below the RDIMM’s from that quarter on. Commodity server memory briefly became the better use of a wafer than the exotic AI product.

Suppliers reallocate capacity in response, depending on where HBM contract negotiations land. TrendForce’s conclusion is that the three major manufacturers will raise HBM quotations substantially in 2027 to restore the premium. That is a forecast, not a settled outcome.

The supply side offers little relief on a 2027 budget timescale. SK hynix CEO Kwak Noh-Jung told Reuters on 10 July 2026 that 2027 would be the worst year in the industry’s history from a supply perspective, with wafer and manufacturing growth around 12% falling well short of demand, and demand exceeding capacity beyond 2030. TrendForce estimated on 9 July 2026 that total RDIMM bit supply will grow only 15–20% year over year in 2027, lagging server CPU shipment growth.

Epoch AI found that AI chips consumed over 90% of total HBM production in 2025. There is no meaningful non-AI buyer left to displace.

Memory Bandwidth Can Matter More Than FLOPS

Everything above concerns what memory costs to buy. This section concerns why you need so much of it.

Inference splits into two phases with opposite hardware profiles. Prefill reads the entire prompt in one parallel pass and saturates the arithmetic units. Decode generates one token at a time, and each token requires reading the model’s weights plus the accumulated key-value cache out of memory to perform a comparatively tiny amount of arithmetic. Decode is bound by memory bandwidth, not by compute. As Databricks put it in its inference performance work, achieved memory bandwidth predicts token generation speed better than peak compute throughput does.

Why batch size sets your AI memory costs

Consider a 70-billion-parameter dense model at FP8, so roughly 70 GB of weights. On an accelerator with 8 TB/s of theoretical HBM bandwidth achieving 70% in practice, the chip reads the full weight set about 80 times per second. Serve one user and you get roughly 80 tokens per second and a very expensive token. Serve 64 users in a batch and the same 80 weight reads produce around 5,120 tokens per second, because the weights were fetched once and used sixty-four times.

Batching is the economic engine of inference serving. Utilisation is not a nice-to-have; it is the denominator.

Context length as a memory cost multiplier

Capacity now reasserts itself. Every concurrent request carries its own KV cache, and that cache competes with the weights for the same HBM.

Illustrative example. Take a representative 70B-class model with grouped-query attention: 80 layers, 8 key-value heads, head dimension 128, cached at one byte per element. KV cache per token is 2 × 80 × 8 × 128 = 163,840 bytes, about 160 KB.

On a 192 GB accelerator holding 70 GB of weights, roughly 122 GB remains for cache. At 8,000 tokens of context each request needs about 1.31 GB, allowing roughly 93 concurrent requests. At 32,000 tokens each needs about 5.24 GB, allowing roughly 23.

Same hardware, same model, same advertised price per token. Four times fewer users per accelerator, and therefore roughly four times the infrastructure cost behind every token produced.

Now add the generational trend. NVIDIA’s Rubin VR200, due in the second half of 2026, carries 288 GB of HBM4 at 22 TB/s against Blackwell’s 8 TB/s on HBM3e, a 2.75x bandwidth gain. Dense FP8 throughput rises from roughly 4.5 to 17.5 PFLOPS over the same step, closer to 3.9x. Compute is outrunning bandwidth, so the batch needed to keep the newer chip busy grows, and that batch needs cache, and cache needs capacity. Capacity and bandwidth bind together, which is why the faster chip does not automatically yield the cheaper serving system.

Repricing a Self-Hosted Inference Server: Where AI Memory Costs Land

Illustrative example. These are modelled assumptions, not a vendor quotation. No accelerator vendor publishes street pricing, and system prices vary by volume, region and configuration. The point is to show which line moved.

Take an eight-accelerator inference node. Hold every assumption constant except system DRAM, priced at the two dated contract figures above.

Component	Assumption	Cost
8 accelerators	$30,000 each (illustrative)	$240,000
CPU (2 sockets)	illustrative	$20,000
Storage (4 × NVMe)	illustrative	$12,000
Networking	illustrative	$24,000
Chassis, PSU, cooling, assembly	illustrative	$25,000
Subtotal excluding system DRAM		$321,000
2 TB system DRAM (32 × 64GB RDIMM) at $272	Sept 2025 contract	$8,704
2 TB system DRAM (32 × 64GB RDIMM) at $1,500	15 Sept 2026 contract	$48,000

System total at September 2025 memory pricing: $329,704. System total at September 2026 memory pricing: $369,000.

One line item moved. The system got about 12% more expensive, and system DRAM rose from 2.6% of the build to 13.0%.

Convert that to an operating rate. Amortise $369,000 straight-line over four years for $92,250 a year. Assume 10.2 kW of draw at a PUE of 1.3, giving 13.3 kW, which at $0.10 per kWh is roughly $11,600 a year. Add an illustrative $15,000 for colocation, support and operations. The annual total is about $118,900, or $14,858 per accelerator-year: $1.70 per accelerator-hour at 100% utilisation.

The same arithmetic on 2025 memory pricing gives $1.56.

Which assumptions move the number most

The DRAM line is real but not dominant. Three assumptions matter more.

Utilisation leads by a wide margin. At 50%, that $1.70 becomes $3.40 per delivered hour. At 30%, it becomes $5.66. Nothing else in the model has that leverage.

Amortisation period comes second. Moving from four years to three raises the hourly figure by roughly a third, and the useful-life assumption for AI accelerators is genuinely contested.

Accelerator price is third, and the assumption most likely to be wrong in your case. It carries its own memory exposure, since HBM is a large share of accelerator bill of materials, and HBM contract prices are exactly what TrendForce expects to rise in 2027.

Memory inflation raised this system’s cost by about 12% and its hourly rate by about 9%. Material, but not the multiple that consumer DRAM coverage implies, because a server is more than its memory.

Rent or Own: How AI Memory Costs Change the Calculation

Public list rates give a reference point. Inworld reported that NVIDIA B200 list rates spanned $3.49 to $14.24 per GPU-hour across clouds in April 2026, more than a four-fold spread for the same silicon.

Against the cheapest end of that range, the illustrative self-hosted node at $1.70 per accelerator-hour breaks even at about 49% utilisation. Under 2025 memory pricing the crossover sat near 45%. Memory inflation moved the threshold by roughly four percentage points.

That should temper the “memory prices killed self-hosting” framing. What determines the answer is whether you can keep accelerators busy.

Four forces push in different directions, and they do not cancel.

Owning gets harder as procurement risk rises. Lead times have stretched, 2027 memory allocation was reportedly settled during mid-2026 negotiations, and an organisation buying twenty nodes has no leverage.

Owning gets easier when the workload is predictable and high-volume. Steady batch inference, a fixed model, a known context distribution and no spiky traffic can hold 70% utilisation or better. At 70%, the self-hosted rate lands near $2.43 against a cloud floor of $3.49.

Renting gets harder as providers pass through their own memory bill. Cloud rates are downstream of DRAM and HBM contract pricing, with a lag set by each provider’s procurement contracts and depreciation schedules.

Renting gets easier when demand is uncertain, when you need to switch accelerator generations quickly, or when reserved-capacity discounts approach your amortised cost without the capital commitment. Reserved pricing is where memory inflation shows up most directly, because providers reprice reservations as their own inputs reset.

There is no universal answer, only a utilisation threshold that memory inflation nudged up slightly.

Why Token Pricing Hides Your Real AI Memory Costs

“$X per million tokens” is a real price. It is also an average over a distribution of workloads whose infrastructure costs differ by an order of magnitude. Four things break the correspondence between the rate card and the hardware.

Output intensity. Output tokens come from the bandwidth-bound decode phase and batch less efficiently than input tokens. Providers price output several times higher for that reason, but the ratio in your traffic decides where you sit inside the average.
Reasoning workloads. Models that generate long internal chains produce many output tokens per user-visible answer. A task that took 500 output tokens under a non-reasoning model can take thousands, at the higher rate, through the more expensive phase.
Context length. As the worked example showed, longer contexts shrink the concurrency an accelerator sustains. Providers absorb this through pricing tiers and caching discounts, but the physical cost lands on memory capacity.
Batching and serving architecture. Continuous batching, prefill-decode disaggregation and paged attention all exist to raise the number of users served per weight read. Two providers on identical hardware running identical models can have materially different cost structures from serving-stack quality alone.

Agentic systems compound all four at once: more calls, longer accumulated context, more output tokens, and idle time between tool invocations that nobody is billed for and everybody is renting.

The rate card tells you what a token costs. It tells you almost nothing about how much HBM sat idle to guarantee the latency you were promised.

What to Watch Next in AI Memory Costs

Micron’s fiscal Q4 2026 results on 30 September 2026. Still upcoming as of publication. Micron guided to roughly $50 billion in revenue at around 86% gross margin, after $41.46 billion at 84.6% in fiscal Q3. Watch the guidance commentary more than the print: it is the clearest public read on whether 2027 supply is loosening.
HBM contract negotiations for 2027. TrendForce expects substantial increases. Whether suppliers get them decides how wafer capacity splits between HBM and DDR5, and therefore what happens to conventional server DRAM.
Contract price direction, not magnitude. The series moved from 90–95% to 13–18% in three quarters. The 2027 question is whether increases keep moderating, as TrendForce currently expects for server DRAM through 2H27, or flatten.
RDIMM bit supply against server CPU shipments. TrendForce’s 15–20% bit growth estimate for 2027, against faster CPU shipment growth, is the arithmetic behind the shortage.
Configuration downgrades. TrendForce noted that since 1H26, some CSPs and OEMs shifted RDIMM configurations from 96GB and 128GB modules down to 32GB and 64GB. When buyers cut memory per server to manage cost, demand destruction is already underway.
New fab and packaging capacity. SK hynix has committed to substantial expansion, but buildings and advanced packaging lines arrive on multi-year schedules.

What AI Infrastructure Teams Should Re-Model

Twelve numbers. If you cannot produce them from current data, that is the finding.

Memory cost per server, split into HBM (inside the accelerator price) and system DRAM, as a share of total build
Memory cost per accelerator, so accelerator generations are comparable on a like-for-like basis
Achieved memory bandwidth utilisation during decode, not theoretical peak
KV cache footprint per request at your actual p50 and p95 context lengths
Average tokens per request, input and output counted separately
Output-token ratio, which predicts the phase dominating your bill
Accelerator utilisation, measured over a full week including troughs
Cost per request, not cost per token
Cost per million tokens from your own hardware, against the rate card you pay
Server amortisation schedule, with the useful-life assumption written down
Power and cooling at your actual PUE and electricity rate
Cloud versus self-hosted exposure, as the utilisation threshold at which the answer flips

That last number is the one worth putting on a wall. Everything else here is an input to it.

Frequently Asked Questions

Why are AI memory costs rising?

AI accelerators need high-bandwidth memory, and HBM consumes far more wafer capacity per delivered gigabyte than conventional DRAM. TrendForce estimates HBM will take roughly 30% of the top three suppliers’ DRAM wafer input by end-2027 while supplying only about 13% of bits. That reallocation removes conventional DRAM from the market at the same time AI server demand is growing, so both HBM and ordinary server memory tighten together. SK hynix’s CEO told Reuters in July 2026 that 2027 would be the industry’s worst supply year on record.

Why does HBM matter for AI inference?

Generating each output token requires reading the model’s weights and accumulated attention cache out of memory. That makes token generation speed a function of memory bandwidth rather than arithmetic throughput. HBM delivers bandwidth by stacking DRAM dies directly beside the processor with very wide interfaces. NVIDIA’s Rubin VR200 carries 288 GB of HBM4 at 22 TB/s, against 8 TB/s on the prior Blackwell generation. Without that bandwidth, the compute sits idle waiting for data.

Is HBM more expensive than normal DRAM?

Per gigabyte, yes, and historically by a wide margin. The gap narrowed unusually in 2026. TrendForce reported that on its per-wafer revenue analysis, HBM was overtaken by DDR5 64GB RDIMM in the first quarter of 2026, which briefly made commodity server memory the more profitable use of a wafer. TrendForce expects suppliers to raise HBM contract prices substantially in 2027 to restore the premium. HBM contracts are negotiated annually, so they respond more slowly than quarterly DRAM pricing.

Does memory bandwidth affect inference cost?

Directly. Decode is bandwidth-bound, so bandwidth sets how many tokens a chip can produce per second, which sets the denominator in cost per token. Databricks has noted that achieved memory bandwidth predicts token generation speed better than peak compute. Batching raises effective throughput by serving many users from a single weight read, but each concurrent request needs its own KV cache in memory, so capacity constrains how far batching can go.

Why can AI inference get cheaper while servers get more expensive?

They are different accounting layers. Token prices reflect competition, algorithmic efficiency gains, quantisation and serving-stack improvements, and Epoch AI measures the decline at a median of roughly 50x per year across benchmarks. Server cost reflects component procurement in a supply-constrained market. A provider can pass through efficiency gains faster than its input costs rise, and absorb the difference in margin or in capital raised against future volume.

Keep reading

Here are the latest posts from the blog.

Claude API Rate Limits in 2026: How Many Requests Can You Run?

November 5, 2026

Claude API rate limits on the Start tier allow most current models 2,000,000 input tokens and 400,000 output tokens per minute, with 1,000 or 4,000 …

Enterprise AI Spending Hit $177B. Agents Are Driving the Bill

November 4, 2026

Figures checked against Menlo Ventures’ 2026 State of Enterprise AI report on 4 November 2026. Market sizes are Menlo estimates, shown as annualised run rates. …

Machine Unlearning: Can We Really Trust an Edited AI Model?

November 4, 2026

30 Agentic AI Statistics for 2026: Adoption, Spending and Risk

November 3, 2026

Last updated: 3 November 2026. This page is updated monthly; each figure keeps its original publication date. Agent traffic at one healthcare booking platform rose …

Agent Prompt Injection Testing: What a Two-Boolean Score Leaves Out

by Idrees Patel

Agent prompt injection testing can examine documented prompt-injection classes and how benchmarks record their outcomes.

The discussion concerns benchmark scoring and a proposed trace-review method. It does not report a completed UniverseBlend experiment.

The benchmark discussed here is AgentDojo, from ETH Zurich’s SPY Lab.

The distinction below concerns the interpretation of a recorded score, rather than a measured comparison of models or defences.

Here it is in one sentence: in AgentDojo 0.1.35, a provider outage and a successful defence are recorded identically.

Key takeaways

Verified in source: AgentDojo 0.1.35 contains three exception paths that set security = True on infrastructure failures. A context-window overflow and a working defence are recorded identically.
Verified in source: The scoring does not confirm that a payload reached the model. A dropped payload and a refused payload produce the same result.
Open question: Whether a different rubric changes reported security scores remains unmeasured here.
Design principle: Ambiguity should resolve to “indeterminate”, not to “secure”. A benchmark has to write something to disk; a published article does not.
Failure shape is not a detail. A silent failure produces a plausible answer and no alert. Attack success rate treats it identically to a loud refusal.
Attribution defaults to unknown. A failed attack alone does not identify which component caused it.
Limitation: No models were run for this article. It is source analysis, and the payload evaluation remains open.

Quick Navigation

Scope of this source discussion
A proposed trace-review method
What the two booleans encode
What the published source records
Where the measurement holds
Where the measurement breaks
The shape of failure matters
Why attribution is difficult
What the source discussion establishes—and leaves open
Limitations
How to investigate this question
References
Frequently Asked Questions

Scope of this source discussion

The scope is source discussion and a proposed review methodology, not payload testing or model execution.

The source discussed is agentdojo version 0.1.35, released on 27 October 2025. The relevant implementation is in agentdojo/benchmark.py in the published package.

Here is what the package contains at that version.

Property	Value
Release	0.1.35, uploaded 2025-10-27
Releases published to date	36
Python requirement	>= 3.10
Evaluation suites	`workspace`, `banking`, `travel`, `slack`
Suite versions shipped	v1, v1_1, v1_1_1, v1_1_2, v1_2, v1_2_1, v1_2_2
Built-in defences	4
Attack generators	17
Result representation	Two booleans per run

The four defences are tool_filter, transformers_pi_detector, spotlighting_with_delimiting and repeat_user_prompt. The seventeen attack generators split into twelve goal-hijacking templates and five denial-of-service templates.

That last row is where this article lives.

A proposed trace-review method

A trace-review method could examine information that summary scores leave out: whether the payload reached the model, whether an infrastructure error interrupted the run, and what the recorded outcome establishes. The approach below is a proposed rubric, not a tested UniverseBlend implementation or a report of re-scored results.

AgentDojo writes one JSON file per run to a predictable path:

runs/{pipeline}/{suite}/{user_task}/{attack}/{injection_task}.json

Each file carries the full trajectory plus the context the logger attached: utility, security, attack_type, pipeline_name, benchmark_version, agentdojo_package_version, an evaluation timestamp, and the injection strings used.

These recorded fields provide possible inputs for a trace review. Their completeness would need to be checked for the particular logs under review.

No implemented re-scoring harness or unit-test results are presented here.

What the two booleans encode

AgentDojo reduces each run to utility and security.

utility is true when the agent completed the legitimate user task. security is true when the injection task goal was not achieved. So security = False means the attack worked.

This is a reasonable design. It produces attack success rate, which is the number most of the literature reports, and it makes results comparable across models and defences.

The compression is also where information goes missing. Two runs that look identical in the summary can be very different events.

Consider two cases that both record security = True:

The model read the injected instruction and declined to follow it.
The injected instruction never reached the model at all.

The first tells you something about the model’s resistance. The second tells you something about your filtering, or about nothing at all if no filter was configured. Collapsed into one boolean, they are indistinguishable.

What the published source records

In the published agentdojo/benchmark.py source at version 0.1.35, three exception handlers set utility = False; security = True.

They fire on:

BadRequestError where the code is context_length_exceeded, the parameter is max_tokens, or the message asks to reduce message length
ApiError whose string contains internal server error
ServerError

Each path logs the error and moves on. The run is counted.

So a context-window overflow enters the summary statistics in the same column as a defence that worked. A provider having a bad afternoon looks, at the aggregate level, like security.

There is a fourth behaviour worth noting. For denial-of-service attack generators, the scoring inverts: security = not utility. Those five generators measure availability rather than goal hijacking, so mixing them into a single headline number combines two different properties.

These statements concern the published code, not observed model behaviour. No UniverseBlend model-execution results are presented.

Where the measurement holds

The design is sound for what it was built to do.

AgentDojo’s core contribution is realism. Its environments are stateful and require multiple tool calls. Injection strings are placed inside realistic content, such as an email body, rather than appended to a tool response where they would be trivially detectable. Over 900 combinations of benign user tasks and malicious injection tasks are paired across the four suites.

That is a substantially harder setting than earlier work. InjecAgent used simulated single-turn scenarios where one adversarial item is fed as a tool output without evaluating the agent’s planning. The newer AgentDyn benchmark reports average trajectory lengths of 7.1 steps against AgentDojo’s 3.49 and InjecAgent’s 1.

The two-boolean scheme also does something quietly important. By tracking utility alongside security, it catches the failure mode where a defence works by breaking the agent. A system that refuses everything scores perfectly on security and terribly on utility, and you can see that immediately.

Agent task competence is a real confound in every security number, and AgentDojo’s original evaluation is upfront about it. The best-performing model in that paper reached only 78.22% utility in benign settings with no attack present. If an agent fails a fifth of its ordinary tasks, some portion of every security result is just ordinary incompetence.

Where the measurement breaks

Three places, in rising order of importance.

Infrastructure errors count as wins. Covered above. The practical effect depends on how often those exceptions fire in your environment, which depends on your context lengths, your provider’s reliability that week, and your rate limits. None of those have anything to do with security.
Delivery is never verified. Nothing in the summary confirms the payload reached the model’s context. A truncation bug, a serialisation error, or a defence quietly dropping content all produce the same security = True as a model that read the attack and refused.
Keep ambiguity visible. A summary boolean cannot, by itself, explain why a run ended without an achieved injection goal.

For agent prompt injection testing that informs a real deployment decision, an unresolvable run should be visible as unresolvable.

The shape of failure matters

A pass/fail score tells you whether something went wrong. It does not tell you whether anyone would have noticed.

The proposed rubric distinguishes four failure shapes.

Shape	Definition
Loud	The system refused, blocked, or sanitised, and made that visible. The event enters your telemetry.
Silent	The injected behaviour executed and the output reads as a normal, plausible completion. Nothing signals a problem.
Partial	The agent followed part of the injected instruction without completing it.
Indeterminate	The shape cannot be established from the trace.

A silent failure is one where the system produces an apparently valid response while following the injected instruction.

Silent failures are the operationally dangerous ones. A loud failure generates a log line and possibly an alert. A silent failure generates a plausible answer, a satisfied user, and no reason for anyone to look. If an agent exfiltrates a record and then produces a competent summary of your inbox, the summary is what your reviewer sees.

Attack success rate treats both identically. They are not remotely the same risk.

One possible approach is to flag refusal or warning language for manual review. Such a heuristic would need validation; no accuracy result is reported here.

Why attribution is difficult

When a payload does not get through, the interesting question is why. It is also the question most evaluations skip, because it is genuinely hard.

A blocked payload looks the same whether the block was engineered or accidental. The trace shows an attack that did not succeed. It does not show you which component is responsible.

A review should distinguish documented evidence from inference and leave attribution undetermined when the trace does not support a conclusion.

Two possible situations illustrate why attribution needs caution.

No configured defence: Consider model behaviour as one possible explanation, while checking other causes and avoiding unsupported framework attribution.
A presentation-layer defence: Record both model behaviour and the changed presentation as possible explanations; a single trace may not separate their effects.

A proposed attribution should record competing explanations and the evidence supporting its confidence level.

The most common error in this area is reading a model’s refusal as a framework security feature. It is an easy mistake and it leads directly to deploying the same architecture on a different model and being surprised.

What the source discussion establishes—and leaves open

Three things, held at appropriate strength.

First, verified. AgentDojo 0.1.35 scores three classes of infrastructure error as defensive successes, and does not verify payload delivery. This is directly checkable in the published source.

Whether a different rubric changes any reported score remains unmeasured here.

Whether stricter scoring changes any published conclusion is unknown. Evaluating that question would require a reproducible method, preserved traces and documented results.

What this does not tell us is anything about which frameworks resist prompt injection. That evaluation has not been run. The original question stands open.

Limitations

The proposed methodology has not been validated.

No models were run. This is source analysis. Every claim is about code, not behaviour.
One framework, one version. AgentDojo 0.1.35 only. Whether other agent evaluation tools share this property is untested, though the design pressure that produces it is common.
Version-specific: The source discussion concerns AgentDojo 0.1.35. Re-check the implementation before applying it to another version.
The proposed rubric is unvalidated. Its effect on scores has not been measured.
Proposed lexical detection needs validation. It may miss warnings or incorrectly classify compliant answers.
Template attacks measure a floor. Work on adaptive attacks shows defences evaluated against fixed templates degrade badly under attacks adapted to them. Any result from this class of tool is a lower bound on what a motivated attacker achieves.
Passing proves nothing. A system that survives this evaluation is not secure. It survived these payloads, at these versions, on this date, under this configuration. Failing one test does not mean a framework is broadly insecure, either.

How to investigate this question

The source reading needs nothing but the package.

bash

pip download agentdojo==0.1.35 --no-deps



unzip agentdojo-0.1.35-py3-none-any.whl -d src

Then open src/agentdojo/benchmark.py and search for security = True. The three exception paths are in the function that runs a single task against a pipeline. The attack generators are in src/agentdojo/attacks/, and the defence names sit in src/agentdojo/agent_pipeline/agent_pipeline.py.

A future investigation could generate traces while holding the model fixed and varying one defence at a time. The following is prospective guidance, not a record of a completed experiment:

bash

python -m agentdojo.scripts.benchmark \



  --model <exact-dated-model-string> \



  --defense tool_filter \



  --attack important_instructions

Record the generated trace paths and preserve the logs alongside the exact evaluation configuration.

Two things to record and publish: the exact dated model string, never “latest”, and your temperature. Leaving temperature at the provider default is a defensible choice, but it is a choice, and it belongs in your methodology.

Set runs per cell above one. A single run cannot distinguish a defence from a coin flip, and these are stochastic systems. Three is a floor. Five is better. Report cells that produce different outcomes across repeats as variable rather than averaging them, because the variance is the finding.

All of this runs against AgentDojo’s simulated in-process environments. No third-party system is involved, and nothing here should be pointed at infrastructure you do not own.

References

Source materials: AgentDojo 0.1.35 and its published source. No historical UniverseBlend execution or re-scoring record is asserted.

Official documentation: AgentDojo repository · AgentDojo documentation

Academic research: AgentDojo, arXiv:2406.13352 · AgentDyn, arXiv:2602.03117 · Adaptive attacks against indirect prompt injection defences, arXiv:2503.00061 · Meta SecAlign, arXiv:2507.02735

Frequently Asked Questions

What is prompt injection?

Prompt injection is an attack where instructions hidden in content an AI system processes get followed as if they came from the operator. In agent systems the content usually arrives indirectly, through a retrieved document, an email body, or a tool response, rather than from the person typing.

What does attack success rate mean?

It is the proportion of runs where the injected instruction’s goal was achieved. It is the standard metric across this literature and it answers a narrow question well. It does not tell you whether the failure was visible, or whether the runs it counted were valid tests.

What is a silent prompt injection failure?

One where the system follows the injected instruction and still produces an apparently valid response. Nothing in the output signals that anything went wrong, so no reviewer has a reason to investigate.

Why should an indeterminate result not count as a defence?

Because it did not test the defence. If a run crashed on a context-length error, you learned nothing about whether the model would have resisted the attack. Scoring it as a block moves an untested run into the reassuring column.

Can prompt injection ever be completely prevented?

Not at the model layer, on current evidence. There is no syntactic boundary between instruction and data in natural language. Practical defence is about limiting what a successful injection can reach: tool permissions, isolation, confirmation on consequential actions.

Keep reading

Here are the latest posts from the blog.