Gemini 4 Carbon Is Here. But Can You Actually Use It?

Gemini 4 Carbon

Published 30 October 2026. Access, pricing and benchmark details verified on 30 October. Sections 2, 6 and 8 will be updated as Google expands access.

Gemini 4 Carbon now leads the Artificial Analysis Intelligence Index, and Google reports the strongest cybersecurity scores it has published for any model. Almost nobody can use it. Since its announcement on 28 October, Carbon has been limited to “a subset of cyber defenders”, with more trusted partners to follow through Google DeepMind’s Fairwind Program. Google hasn’t given a date for developers, businesses or consumers.

So the short answer is no: unless your organisation has been invited, you can’t call Gemini 4 Carbon today. Gemini 4 Argon, its predecessor, has been generally available since 13 October and is the model you can build on now. The more useful question is what an Argon team should do while it waits. Carbon’s launch price and Argon’s 1 December price rise make that more pressing than it looks.

Key Takeaways
  • Carbon isn’t publicly available. Access is limited to selected cyber defenders, then Fairwind partners. It has no public model ID and no listing on the Gemini API pricing page as of 30 October.
  • Argon is the model you can use. It’s a stable Gemini API model at $2 input and $10 output per million tokens until 30 November, then $4 and $20.
  • Carbon’s lead is real on Artificial Analysis. It scores 59 at High effort, against 50 for Argon and 52 for Claude Opus 5.5 at the same setting.
  • The cyber scores are Google’s own, measured without guardrails. SEC-bench Pro tests reproducing known bugs, not discovering new ones.
  • Don’t rebuild around a model you can’t call. Keep Argon in production, put model choice behind a configuration switch, and keep an evaluation set ready for Carbon.

Quick Navigation


What Gemini 4 Carbon’s announcement changes

Google DeepMind’s Koray Kavukcuoglu introduced Carbon as “our strongest model with state-of-the-art cybersecurity and long horizon capabilities”. The pitch has two parts. One is security work: finding, exploiting and patching vulnerabilities. The other is long-running engineering: Google says Carbon has worked internally for days on multi-step tasks, such as root-causing and fixing bugs across large codebases.

The timing is awkward for anyone who just adopted Argon. Argon was announced on 30 September through the same cyber-defender route and reached the Gemini app and the paid API on 13 October. Carbon arrived 15 days later and is described as Argon’s successor. Teams that spent the past fortnight integrating Argon are now reading about a better model in the same family.

What hasn’t changed is what you can deploy. Every number in Carbon’s announcement describes a model most readers can’t call yet.

Can you use Gemini 4 Carbon right now?

No, not unless Google has invited your organisation. Google says Carbon is available from the week of 28 October to “a subset of cyber defenders”, with access to be “phased to additional trusted partners” through the Fairwind Program. Broader availability for developers, businesses and consumers will come “as soon as possible”. No date has been given.

The two groups aren’t the same. The first wave is a hand-picked subset of defenders. Fairwind is a standing application programme. Its public page says it prioritises governments, national cyber authorities, critical-infrastructure operators, core technology platforms and academic defensive-research labs. It works with more than 650 partners and vets applicants, including background checks. Partners must use phishing-resistant multi-factor authentication, limit access to internal security teams and must not resell or share access. As of 30 October, that page still describes Fairwind access in terms of Argon, so Carbon-specific terms haven’t been published.

For everyone else, three practical facts matter:

  • No model ID. Carbon has no published identifier, and it isn’t on the Gemini API pricing page. Any code written today would be guesswork.
  • No public API. Artificial Analysis notes that its Carbon price and speed figures are estimates for that reason.
  • Argon is the deployable option. Its model page lists gemini-4-argon as a stable model with Batch, Flex and Priority inference and a shared 1M-token context window.

Google also says Carbon went through the US government’s voluntary pre-release testing process. That helps explain the pace. A staged rollout gives Google time to watch how a strong security model is used before opening the API.

Gemini 4 Carbon vs Argon: the comparison that matters

Same price, different access
Gemini 4 ArgonGemini 4 Carbon
Announced30 September 202628 October 2026
AvailabilityGenerally available since 13 October (Gemini app, paid API)Selected cyber defenders, then Fairwind partners; no general date
Model IDgemini-4-argon (stable)Not published
Intended useGeneral frontier model, strong on securityStrongest Gemini model; long-horizon engineering and cyber defence
Artificial Analysis Index v5.0 (High)5059 (ranked first)
Cost per Index task (AA estimate)$7.68$7.76
Cyber benchmarksGoogle cites relative gains only, e.g. more than 2× on an internal secure-code testSEC-bench Pro 88.2%, ExploitGym 54.8%, CWE-bench 71.0% (Google-reported)
Gray Swan indirect prompt injection0.7% attack success (Google-reported)0.0% (Google-reported)
Price, per million tokens$2 / $10 until 30 Nov; $4 / $20 from 1 Dec$2 / $10 introductory; $4 / $20 afterwards; end date not stated
Developer readinessProduction-readyNot yet callable

The independent row is the one to read first. Artificial Analysis scores Carbon 59 and Argon 50, both at High effort, on its nine-evaluation v5.0 index. The headline “Carbon beats Opus 5.5” story needs a qualifier: Claude Opus 5.5 scores 58 at maximum effort but 52 at High. At matched settings, Carbon’s lead is wider than the leaderboard’s top two rows suggest.

The cost row is the surprise. Priced at the same $4/$20 standard rates, Carbon cost $7.76 per index task against Argon’s $7.68. It used slightly more output tokens per task, 123,000 against 116,000, and scored nine points higher. If that holds on real workloads, Carbon would deliver more capability for nearly the same spend per task. That’s a reason to test it when you can, not a reason to stop shipping on Argon now.

We haven’t put Argon’s cyber benchmark scores in the table because Google hasn’t published comparable ones. Its Argon post shows results only in chart images, and apart from Gray Swan, Carbon’s post compares against Argon only in relative terms.

What Gemini 4 Carbon’s cybersecurity scores measure

Google arranges its security results along the defender’s workflow: find a flaw, prove it matters, fix it. All three numbers are Google’s own, and its announcement says the external benchmarks were run “without guardrails”. That’s the right way to measure capability and the wrong way to estimate what a guarded production deployment will do.

Find, Prove, Fix
Find: SEC-bench Pro, 88.2%

The name suggests hunting for unknown bugs, but SEC-bench Pro measures something narrower. Its 344 tasks give a model a disclosed vulnerability in V8, SpiderMonkey or the Linux kernel, with the report, and ask it to produce a proof-of-concept input that triggers the bug. In the paper’s own runs, the best agent, Codex with GPT-5.5, solved 58%. Google reports 88.2% for Carbon. That’s a large jump in turning a bug report into a reproducer, which is real, tedious defender work. It isn’t evidence that Carbon finds zero-days at that rate.

Prove: ExploitGym, 54.8%

ExploitGym, from Berkeley’s RDI and collaborators including Google, OpenAI and Anthropic, asks an agent to turn a known vulnerability into a working exploit that captures a flag. Its 898 tasks span userspace C/C++ programs, V8 and the Linux kernel. In the May paper, the best agent captured 157 flags, about 17%, with protections such as ASLR switched off. Google’s 54.8% for Carbon is far higher. Google hasn’t said which task subset, time limit or mitigation settings it used, and those choices can move the score a lot.

Exploitation sounds like the attacker’s job, and it is also the defender’s. A proven exploit separates the bug that has to be patched tonight from the one that can wait for a release cycle.

Fix: CWE-bench, 71.0%

Collinear’s CWE-bench grades agents on repairing weaknesses across a codebase, not just the instance they were shown. Its launch example gave no credit to agents that patched a known token-revocation path but missed a newly introduced one. At launch in September, no agent repaired a majority of tasks, and the leader scored 47%. Google reports 71.0% for Carbon. This is the most directly useful of the three scores for most security teams.

Prompt injection: Gemini 4 Carbon’s Gray Swan result

On Gray Swan’s indirect prompt-injection benchmark, Google reports a 0.0% attack success rate for Carbon, down from Argon’s 0.7%. That measures how often hidden instructions in content the model reads succeed against a specific attack set. It’s encouraging for agents that browse or read email. A zero on a benchmark still isn’t a guarantee, as our piece on hidden flaws in a passing red-team test explains: attackers adapt, and a fixed test doesn’t.

How much cyber capability should be public?

A model that writes working exploits for kernel bugs is useful to a defender triaging a backlog and to an attacker choosing a target. Google’s answer is to control who gets access first, and its documents describe a programme built for that. Fairwind vets organisations, restricts use to internal security teams and bans “malicious tasks such as creating malware”. None of that requires guessing at Google’s motives.

Moonshot AI took a different route with Kimi K3.1 Pro, whose open weights are due by 4 November. Its release post says it removed cyber-related data from mid-training and reinforcement-learning tasks before releasing the weights. On Moonshot’s own unpublished vulnerability and exploitation test, xdev-bench-hard, the released model scored 81.0% (pass@3) against 90.5% for an unfiltered checkpoint.

The two approaches aren’t equivalent. Moonshot reduced capability and then gave everyone the weights. Google kept capability and is choosing who gets it. Moonshot’s chart also shows DeepSeek V4.1 Flash, already open, at 85.7% on the same test, above the filtered Kimi model. Once one strong open model exists, filtering the next one changes less than it seems. Gating, meanwhile, holds only as long as the API stays closed.

For buyers, the practical consequence is simpler. If your work needs Carbon-level security capability, you’ll be dealing with an access programme, its usage terms and its audit requirements, not only an API key.

Gemini 4 Carbon pricing: is there a reason to wait?

Google says Carbon launches at Argon’s introductory rates: $2 per million input tokens and $10 per million output tokens, with cached input at 95% off. After an introductory period it moves to $4 and $20. Google hasn’t said when that period ends. Don’t assume it follows Argon’s schedule.

Argon’s dates are fixed. The Gemini API pricing page lists $2/$10 through 30 November and $4/$20 from 1 December, with Batch and Flex at half those rates and Priority above them. We covered what that doubling does to real workloads in our analysis of Argon’s 1 December price change.

Tier (per 1M tokens)Argon to 30 NovArgon from 1 DecCarbon (announced)
Standard input / output$2 / $10$4 / $20$2 / $10 introductory, then $4 / $20
Cached input$0.10$0.2095% off input
Batch and Flex input / output$1 / $5$2 / $10Not published

There’s no price reason to wait. Argon’s introductory window will probably close before most teams can get Carbon access, and both models end at the same standard rate. The token price also isn’t the cost that matters. Artificial Analysis’s estimates, at standard rates, put the two within 1% of each other per task. On your workload, the gap will depend on how many tokens each model spends and how often its output passes your checks. That can only be measured once Carbon is callable.

What Argon users should do about Gemini 4 Carbon

If you’re running Argon in production, keep it there. A working integration with known costs and failure modes is worth more than a better model you can’t call. Use the time to make switching cheap: keep the model name in configuration, keep prompts and tool definitions model-agnostic, and keep an evaluation set of real tasks with pass criteria. When Carbon opens, swapping the model and rerunning the set is a day’s work.

  • Starting a new project: build on Argon now. Don’t design around Carbon-only claims, such as multi-day autonomous runs, until you can test them.
  • Security vendors and in-house security teams: apply to Fairwind if you’re eligible. Carbon’s exploit validation and repair scores are aimed at your work. Expect vetting and usage restrictions.
  • Enterprise AI teams: check whether your Google Cloud or Gemini Enterprise account team can say anything about access, and plan budgets on Argon’s 1 December rates.
  • Teams with strict security requirements: wait for Carbon’s documentation, data-retention terms and independent evaluations before treating it as approved. A strong prompt-injection score doesn’t replace your own testing.

The exception is a workload where Argon is clearly failing and the failure is exactly what Carbon claims to fix, such as long-horizon code repair. Even then, the move is to document the failures now so you can test Carbon against them later. Public benchmarks rarely transfer cleanly to your own workload.

Gemini 4 Carbon: what to watch next

This list will be updated as things change.

SignalWhere to lookWhy it matters
A published model ID and docs pageGemini API models documentationThe first sign you can actually call it
Carbon on the pricing pageGemini API pricingConfirms the introductory end date and Batch/Flex rates
Fairwind terms for CarbonGoogle DeepMind Fairwind pageWho qualifies and on what conditions
Independent cyber resultsArtificial Analysis Cyber IndexCompares defensive capability on a common harness
Google’s safety reportLinked from the announcementMethodology behind the guardrail and misuse claims
Gemini app or Workspace accessGoogle product announcementsConsumer and business availability

Conclusion

Gemini 4 Carbon is the stronger model on every published capability measure, and for now it belongs to a small set of security organisations. Argon is available, documented and on a known price path. Build on Argon, prepare to switch with a configuration change, and judge Carbon on your own tasks once Google opens the API.

Frequently asked questions

Can I use Gemini 4 Carbon right now?

Only if Google has given your organisation access. As of 30 October 2026, Carbon is limited to a subset of cyber defenders, with more trusted partners joining through the Fairwind Program. Google hasn’t announced a date for general developer, business or consumer access.

Is Gemini 4 Carbon available through the Gemini API?

Not publicly. Carbon has no published model ID and doesn’t appear on the Gemini API pricing page as of 30 October. Artificial Analysis says its price and speed figures for Carbon are estimates because there’s no public API. Gemini 4 Argon is available through the API now.

What is the difference between Gemini 4 Carbon and Argon?

Carbon is Argon’s successor and Google’s strongest model. It scores 59 to Argon’s 50 on the Artificial Analysis Intelligence Index v5.0 at High effort, with Google-reported gains in security and long-horizon coding. Argon is generally available. Carbon is restricted.

How much does Gemini 4 Carbon cost?

Google announced introductory rates of $2 per million input tokens and $10 per million output tokens, with cached input 95% off, rising to $4 and $20 afterwards. It hasn’t said when the introductory period ends, and Carbon isn’t yet on the pricing page.

When does Gemini 4 Argon pricing increase?

On 1 December 2026. Standard rates rise from $2 to $4 per million input tokens and from $10 to $20 per million output tokens. Batch, Flex, Priority and cached-input rates double on the same date, according to Google’s Gemini API pricing page.

Is Gemini 4 Carbon better for cybersecurity?

On Google’s numbers, yes: 88.2% on SEC-bench Pro, 54.8% on ExploitGym and 71.0% on CWE-bench, measured without guardrails, plus a 0.0% attack rate on Gray Swan’s prompt-injection test. These are vendor results, and independent cyber evaluations of Carbon haven’t been published yet.

Why is Google restricting Carbon’s initial access?

Google calls it a “deliberate approach to rollout”. Carbon can reproduce and exploit vulnerabilities at levels Google calls state of the art, and Google describes Fairwind as giving vetted defenders “a critical head start”. Google also says Carbon went through the US government’s voluntary pre-release testing.

Should developers wait for Carbon before building with Argon?

Generally not. Argon is available now, and Carbon has no public date. Build on Argon with the model name in configuration and an evaluation set ready, then test Carbon when you can. The exception is a project that depends on capabilities only Carbon claims and can afford to wait.


Keep reading

AGI vs superintelligence

AGI vs Superintelligence: A New Name Isn’t a New Intelligence

Facts checked against official documents and reporting on 18 November 2026. On 29 September, the White House told federal agencies to stop writing “artificial intelligence” …
AI Coding Agent

AI Coding Agents Are Getting Expensive. Who’s Paying for the Tokens?

Pricing checked on official vendor pages on 17 November 2026. All USD. Cost scenarios are illustrative and labelled as such. On 28 October, TechCrunch published …
Model Weights

Verify Model Weights: 7 Checks Before You Download

Checked against the burner22/numbat repository and its discussions, Hugging Face and PyTorch documentation, and Moonshot AI’s announcement on 2 November 2026. On 1 November, a …
Agentic AI Statistics

30 Agentic AI Statistics for 2026: Adoption, Spending and Risk

Last updated: 15 November 2026. This page is updated monthly; each figure keeps its original publication date. Agent traffic at one healthcare booking platform rose …

LLM Cost per Task: How to Calculate the Real API Cost

LLM cost

Prices checked 28 October 2026: standard (pay-as-you-go) API tier, US dollars per million tokens, from each provider’s official pricing page unless stated otherwise.

Kimi K3.1 Pro costs half as much per token as Claude Opus 5.5: $2 per million input tokens and $10 per million output tokens, against $4 and $20. On Artificial Analysis’s Intelligence Index runs, Kimi still cost more per task: $2.11 against $1.82 for Opus 5.5 at High effort. It produced 55,000 output tokens per task to Opus’s 36,000.

That’s the mistake a price table invites. A per-token rate tells you what one unit of text costs. It doesn’t tell you how many units a model will spend on your job, or how often the job comes back usable. LLM cost per task captures both.

LLM cost per task is the total API spend on every attempt at a set of tasks, divided by the number of attempts that met your acceptance criteria.

The rest of this article is about calculating that number correctly, and about knowing when to trust it.

Key Takeaways
  • Price per token and cost per task can rank models in opposite orders. Kimi K3.1 Pro is half Opus 5.5’s per-token price but cost about 16% more per Artificial Analysis task.
  • Reasoning tokens are billed as output by OpenAI, Anthropic and Google, even when you never see them. Anthropic also bills earlier turns’ thinking as input on its newest models.
  • Caching can swing the bill several-fold. In one public test, a broken cache made a single run cost $17.60 instead of $2.62.
  • Divide by successes, not attempts. Failed runs cost money and produce nothing.
  • Twenty runs can’t separate close results. An 80% and a 65% pass rate from 20 runs each have overlapping confidence intervals.
  • Price changes can settle a close call. One developer test found Gemini 4 Argon roughly level with Opus 5.5 per successful fix. At Argon’s December 1 prices, it would cost about twice as much.

Quick Navigation


The LLM cost per task formula

For one attempt, the API cost is the sum of each token category times its rate, with token counts converted to millions:

Attempt cost = (fresh input tokens  ÷ 1,000,000 × fresh-input rate)
             + (cache-read tokens   ÷ 1,000,000 × cache-read rate)
             + (cache-write tokens  ÷ 1,000,000 × cache-write rate)
             + (billable output     ÷ 1,000,000 × output rate)
             + per-call charges (web search, code containers, grounding)

Then, across a test set:

LLM cost per successful task = sum of all attempt costs ÷ number of successful attempts
LLM cost

Each line has its own billing rules, and they differ by provider. This is the buyer’s cost. What it costs a provider to serve those tokens, and its margin, is a separate question, covered in what inference actually costs per token.

What each line costs at OpenAI, Anthropic and Google
Line itemOpenAI (GPT-5.6 and later)Anthropic (Claude 5.5 models)Google (Gemini 4 Argon)
Fresh inputList rateList rateList rate
Cache read0.1× input; 0.05× on GPT-6.1 Sol and Terra0.1× input; 0.05× on Opus 5.5 and Sonnet 5.5$0.10 per million (0.05× input)
Cache write1.25× input (30-minute) or 2.5× (12-hour)1.25× input (5-minute) or 2× (1-hour)No separate charge listed; caching is implicit only
Reasoning tokensBilled as outputBilled as output, including when thinking text is hiddenBilled as output (“incl. thinking”)
Search tools$10 per 1,000 web searches$10 per 1,000 web searches$14 per 1,000 grounded queries after 5,000 free a month

Sources: OpenAI pricing and prompt caching; Anthropic pricing and prompt caching; Gemini API pricing.

Two details catch people out. Anthropic’s thinking defaults to a display mode of “omitted” on Opus 5.5 and Sonnet 5.5, but its thinking documentation is explicit: “You’re still charged for the full thinking tokens. Omitting reduces latency, not cost.” On those models, previous turns’ thinking blocks also stay in context and are billed as input, so a long agent session pays for old reasoning again on every turn.

Reading usage fields without double-counting

A common error in home-made calculators is adding a field that already contains another one.

  • Anthropic: input_tokens counts only the tokens after the last cache breakpoint. Total input is cache_read_input_tokens + cache_creation_input_tokens + input_tokens, and each part takes its own rate.
  • OpenAI: cached_tokens excludes tokens charged as writes, which appear in cache_write_tokens. Reasoning tokens sit inside output_tokens, under output_tokens_details.reasoning_tokens. Don’t add them again.
  • Truncated responses: with reasoning models, a request that hits its output limit can bill input and reasoning tokens and return no visible answer, as OpenAI’s reasoning guide warns. That attempt still counts toward the cost.

Then reconcile your summed usage against the provider’s invoice for the same period. In July 2026, developers on OpenAI’s forum reported that GPT-5.6 output-token counts were several times higher than the text actually generated. OpenAI didn’t publicly confirm a defect in that thread. Whatever happened there, the invoice is the figure your finance team will pay.

From cost per attempt to LLM cost per successful task

The numerator is everything you paid to get results. That includes failed attempts, automatic retries, intermediate calls inside an agent loop, any model you use as a judge and per-call tool fees. Count each API call once, at the moment it happens. If your harness logs both an agent step and its parent run, sum only one of them.

The denominator is the number of attempts that passed a fixed acceptance test, written before the run. “The tests pass” is a test. “The output looked fine” is not. The same criteria must apply to every model, or the division compares different jobs.

There are two valid ways to define a task:

  • Independent attempts. Each run is scored on its own, which is what most benchmark-style comparisons do.
  • Tasks with a retry policy. A task succeeds if any attempt within, say, three tries passes. Then the cost per task is all spend divided by tasks completed. That matches production better, but pick one definition and use it for every model.

API cost is not the whole bill. Human review, retrieval infrastructure, observability and sandboxes belong in a fully loaded cost per task whenever models differ in how much of that work they create. A model whose failures are obvious and cheap to discard can beat one whose failures look plausible and need a reviewer to catch.

A public test: 20 runs, four models

On 14 October 2026, a developer writing at paddo.dev replayed six merged changes from a production monorepo against four models. Each model got 20 runs (three or four per change) at medium effort in the Claude Code harness. Gemini 4 Argon was routed through OpenRouter. A run passed if the tests that shipped with the original fix passed. Costs were calculated at list prices.

ModelRunsPassing runsTotal costCost per runCost per passing run
Gemini 4 Argon2016$57.68$2.88$3.61
Claude Opus 5.52013$45.81$2.29$3.52
Claude Sonnet 5.52011$11.76$0.59$1.07
GPT-6.1 Sol2011$6.73$0.34$0.61

The totals and pass counts come from the post. The last two columns are our arithmetic, and they check: $57.68 ÷ 16 = $3.605, and $45.81 ÷ 13 = $3.524.

Argon and Opus are close, nine cents apart per passing run. Argon’s per-token prices matched Sonnet 5.5’s at the time ($2 input, $10 output), yet it cost five times as much as Sonnet per run. It wrote 1.37 million output tokens across the 20 runs, 2.8 times as many as Opus.

GPT-6.1 Sol had the lowest cost per passing run, but the post also records five Sol runs that broke a previously passing test, against none for the others. It doesn’t say whether those runs are among Sol’s 11 passes. If “breaks nothing” is part of your acceptance test, Sol’s denominator may shrink, which is why the test has to be written down before anyone divides.

How much a 20-run test can tell you

Not much, at the margin. A 95% Wilson confidence interval on 16 passes out of 20 is about 58% to 92%; for 13 out of 20, about 43% to 82%. The ranges overlap heavily, so a rerun could reverse the order. And six changes from one codebase, with correlated runs on each change, say little about a different stack.

What the December repricing does

Google’s pricing page doubles every Argon token rate on 1 December 2026: input from $2 to $4, cached input from $0.10 to $0.20, output from $10 to $20. If the runs used identical tokens, Argon’s total would become $115.36, or $7.21 per passing run, about twice Opus’s $3.52. We covered the change itself in Gemini 4 Argon pricing doubles Dec 1.

A simple break-even check shows how far the decision moves. One model is cheaper per success than another when its pass rate exceeds the other’s pass rate multiplied by the ratio of their per-attempt costs. Before December, Argon needed a pass rate above 65% × 1.26 ≈ 82% to beat Opus; it scored 80%. After December, it would need 65% × 2.52 ≈ 164%, which is impossible. Even at the bottom of Opus’s confidence interval, Argon would lose. On this test, the price change settles a question the sample size couldn’t.

How caching and reasoning change LLM cost per task

The same post found a stark caching failure. The first Argon run cost $17.60 because it read 8.4 million input tokens and only 59,000 came from cache. The harness inserted system messages that changed the prompt prefix on every call, so Gemini’s implicit cache never matched. After a fix, the same run cost $2.62, with 94% of input served from cache.

The second example is illustrative. The token counts and pass rates are assumptions, not measurements of any real model. Two models attempt the same agent task:

  • Model X: $1 input, $0.05 cache read, $1.25 cache write, $5 output, per million tokens. Pass rate 60%.
  • Model Y: $4 input, $0.20 cache read, $5 cache write, $20 output. Pass rate 85%.
  • Each attempt sends 300,000 input tokens over its turns. When caching works, 270,000 are cache reads and 30,000 are cache writes.
  • Model Y produces 20,000 output tokens per attempt, including reasoning. Model X produces 60,000.
ScenarioCost per attemptPass rateCost per successful task
Model X, cache broken (all 300,000 input tokens fresh)$0.60060%$1.00
Model X, cache working$0.35160%$0.585
Model X, cache working, reasoning doubles to 120,000 output tokens$0.65160%$1.085
Model Y, cache working$0.60485%$0.711

The calculation for the second row: 270,000 × $0.05 + 30,000 × $1.25 + 60,000 × $5, each divided by one million, gives $0.0135 + $0.0375 + $0.30 = $0.351. Dividing by 0.60 gives $0.585.

LLM cost

With caching working, the cheaper model wins. Break the cache, or let it reason twice as long, and the expensive model wins instead. Neither change shows up in a price table.

What published comparisons actually measured

Before using a published cost-per-task figure, check the effort setting, what counted as a task, and whether success was part of the number.

  • Effort settings. Put Gemini 4 Argon’s 36,000 reasoning tokens per task next to Opus 5.5’s 84,000, and Argon looks far leaner. Both figures appear on Artificial Analysis, but they come from different settings: Argon at High effort and Opus 5.5 at Max. At matched High effort, the comparison reverses: Argon used 36,000 reasoning tokens per task to Opus’s 18,000, and 62,000 output tokens in total to Opus’s 36,000. Argon still cost less per task, $1.62 against $1.82, because its tokens were half the price at the time.
  • Vendor claims. OpenAI launched GPT-6.1 Terra on 26 October at $1 input and $5 output. DataCamp’s write-up of the announcement says that at max effort Terra is over three times as token-efficient as Opus 5.5 and a quarter of the price per token, and it puts the combined effect at “9x lower cost per task”, attributed to OpenAI. No benchmark is named, and we couldn’t open OpenAI’s post to check. Three times the efficiency at a quarter of the price would multiply to about 12 times, not 9, so the figure depends on a token breakdown that hasn’t been published. Treat it as a vendor claim to test, not a result.
  • Cost per task is not cost per success. Artificial Analysis computes cost per task from token counts at list prices. It doesn’t divide by how many answers were right. A model that is cheaper per task but scores lower may still cost more per correct answer. As we argued in why benchmark scores don’t transfer to your workload, these numbers are good for building a shortlist, not for making the final decision.

How to measure LLM cost per task on your own workload

  1. Pick 30 to 50 real tasks from production traffic, covering easy, typical and hard cases. More tasks beat more runs of the same task.
  2. Write the acceptance test first: unit tests, a schema check or a fixed-judge rubric, identical for every model.
  3. Fix the configuration: prompts, tools, effort setting, context limits and retry policy.
  4. Log raw usage for every call, including retries and sub-agents, with each token field stored separately.
  5. Price the logs offline against a dated copy of each pricing page, so you can reprice them when rates change.
  6. Check cache hit rates per model. A low rate in a repetitive session usually means the prompt prefix changes between calls.
  7. Reconcile with the invoice for the test window.
  8. Report ranges: pass rate with a confidence interval, cost per attempt, cost per successful task and latency.

Run at least two attempts per task. That shows whether failures are random or the same task fails every time.

A decision rule for LLM cost per task

Compare models on cost per successful task, after deciding what a success is worth. Then apply three checks.

  • If the confidence intervals on pass rate overlap, treat the models as tied on quality and choose on cost per attempt, latency and operational fit.
  • If one model’s cost per attempt is k times another’s, it needs a pass rate more than k times higher to win. When that would take a pass rate above 100%, stop testing.
  • If a success is worth far more than the API cost, as with a merged fix that saves an engineer an hour, pass rate and reviewer time outweigh token prices. Optimise for the fewest failures, then for cost.

Per-token prices still have a job. They tell you where to look first. LLM cost per task tells you what you’ll actually pay.

Frequently asked questions

What is LLM cost per task?

It is the total API spend on all attempts at a set of tasks, divided by the number of attempts that met a fixed acceptance test. Failed attempts, retries, intermediate agent calls and judge-model calls stay in the total, because you paid for them. A broader, fully loaded version adds review time and infrastructure.

How do I calculate LLM API cost for one request?

Multiply each token category by its rate per million tokens: fresh input, cache reads, cache writes and output, including reasoning. Then add per-call tool fees such as web search. Take the counts from the response’s usage object, and check that one field doesn’t already include another before adding them.

Are reasoning tokens billed?

Yes. OpenAI, Anthropic and Google bill reasoning or thinking tokens at the output rate, even when the reasoning text isn’t returned. On Anthropic’s Opus 5.5 and Sonnet 5.5, previous turns’ thinking blocks are also billed as input when they stay in the conversation.

How much does prompt caching save?

Cache reads cost between 5% and 10% of the input rate at OpenAI, Anthropic and Google for current models. OpenAI and Anthropic charge 1.25 times the input rate or more to write to the cache. Savings depend on how much of each request repeats an earlier prefix, so measure the hit rate rather than assuming it.

Is the cheapest model per token ever the cheapest per task?

Often, yes. On Artificial Analysis’s figures, Gemini 4 Argon at High effort cost less per task than Opus 5.5 at High, while its tokens were half the price. The ranking flips when the cheaper model uses many more tokens, misses the cache or fails more often. Only measurement on your own tasks tells you which case you’re in.


Keep reading

AGI vs superintelligence

AGI vs Superintelligence: A New Name Isn’t a New Intelligence

Facts checked against official documents and reporting on 18 November 2026. On 29 September, the White House told federal agencies to stop writing “artificial intelligence” …
AI Coding Agent

AI Coding Agents Are Getting Expensive. Who’s Paying for the Tokens?

Pricing checked on official vendor pages on 17 November 2026. All USD. Cost scenarios are illustrative and labelled as such. On 28 October, TechCrunch published …
Model Weights

Verify Model Weights: 7 Checks Before You Download

Checked against the burner22/numbat repository and its discussions, Hugging Face and PyTorch documentation, and Moonshot AI’s announcement on 2 November 2026. On 1 November, a …
Agentic AI Statistics

30 Agentic AI Statistics for 2026: Adoption, Spending and Risk

Last updated: 15 November 2026. This page is updated monthly; each figure keeps its original publication date. Agent traffic at one healthcare booking platform rose …

AI API Rate Limits in 2026: The Numbers That Bind

AI API rate limits

Two teams call the same model on the same tier. One runs a support chatbot and never sees a 429. The other runs a five-step agent loop over long documents and starts getting throttled on day one. Same account, same limits, different binding constraint.

That is the useful thing to understand about AI API rate limits in 2026, and it is the thing the comparison tables usually miss. Every provider enforces several ceilings at once. The one you hit depends on the shape of your traffic, not on which number is biggest.

The short version: OpenAI enforces RPM, TPM, RPD and TPD at the organization and project level, with four tiers gated on cumulative credit purchases. Anthropic splits tokens into input and output, caps them separately, and — crucially — does not count cached input tokens toward the input limit on most models. Google publishes a tier structure but has stopped publishing per-model numbers in the docs, pointing you to AI Studio instead. AWS does not do tiers at all; Bedrock quotas are per account, per region, per model, token-burndown based, and now split across two separate endpoints.

None of those providers publishes a concurrency limit for standard text inference. That absence matters more than most of the published numbers, and it comes up later.

Limits last checked: 9 October 2026. Every figure below links to the provider’s own documentation. Treat them as a snapshot — OpenAI’s tier names changed this year, and Google’s published tables disappeared sometime before September.

Key takeaways
  • Rate limits are plural. OpenAI enforces up to six dimensions, Anthropic three plus spend caps, Gemini three plus a rolling ten-minute spend limit, Bedrock token-based quotas split across two endpoints.
  • Cached input tokens do not count toward ITPM on most Claude models. At an 80% hit rate, a 2M ITPM limit processes around 10M total input tokens per minute, which makes cross-provider TPM comparisons close to meaningless.
  • No provider publishes a concurrency limit for standard text inference. Derive yours from RPM and request duration, and enforce it in your client.
  • A 429 with no retry-after on Anthropic means a spend cap, not a rate limit. Retrying cannot clear it.
  • Google no longer publishes per-model Gemini numbers in its documentation. AI Studio is the source of truth for your project.
  • Limits attach above the key: organization and project on OpenAI, organization on Anthropic, project on Gemini, account and region on Bedrock. Minting more keys does not add capacity.
  • OpenAI charges TPM against the maximum of max_tokens and its own estimate. Anthropic ignores max_tokens for OTPM. The same parameter is a cost on one platform and free on the other.

Quick Navigation


Quick Referene: AI API Rate Limits, October 2026

The providers do not expose comparable quota systems, so this table does not pretend they do.

ProviderScopeDimensions enforcedTop published standard tierConcurrency limitHow you move up
OpenAIOrganization and projectRPM, RPD, TPM, TPD, IPM, audio minutesGrow: 15,000 RPM / 40M TPM (Astra, Sol, Terra); 30,000 RPM / 180M TPM (Luna)Not publishedAutomatic at $500 total credit purchases
AnthropicOrganization, with optional workspace sub-limitsRPM, input TPM, output TPMScale: 10,000 RPM / 10M ITPM / 2M OTPM (Opus 5.5, Sonnet 5.5, Haiku 5.5)Not publishedAutomatic on usage history; request increase in Console
Google GeminiProject, not API keyRPM, input TPM, RPD, plus IPM/TPD on some modelsNot published in docs; shown in AI Studio per projectNot published for standard; 100 concurrent batch jobsTier 2 at $100 paid + 3 days; Tier 3 at $1,000 paid + 30 days
AWS BedrockAWS account, per region, per model, per endpointTokens per minute, requests per minute, varies by quotaVaries; see AWS General ReferenceNot published as a single figureService Quotas increase request

Anthropic’s Scale tier and OpenAI’s Grow tier look roughly comparable on RPM. They are not comparable on tokens, because the two companies count tokens differently — which is the first thing worth explaining.

What the Numbers Actually Mean

Six different things get called a rate limit.

  1. RPM counts requests in a rolling minute. It ignores how big each request is. OpenAI’s docs give the cleanest illustration: 20 requests of 100 tokens each will exhaust a 20 RPM limit even though you sent 2,000 tokens against a 150,000 TPM allowance.
  2. TPM counts tokens. Whether that means input tokens, output tokens, or both depends entirely on the provider, and this is where cross-provider comparison breaks down. Google counts input tokens. Anthropic runs two separate counters, ITPM and OTPM. OpenAI’s TPM covers the request as a whole.
  3. RPD and TPD are daily caps. Gemini’s RPD resets at midnight Pacific. OpenAI applies RPD and TPD on some models, mostly at lower tiers.
  4. Spend limits are a different animal and are easy to mistake for throttling. Anthropic’s tier spend caps ($500 Start, $1,000 Build, $200,000 Scale) stop API usage until the first of the next month when you hit them, returning a 429 with rate_limit_error but no retry-after header. The distinguishing field is error.details.error_code, which reads enforced_spend_limit_reached. Retrying will not help, and SDK auto-retry will burn through its attempts for nothing. Gemini enforces a spend-based limit on a rolling ten-minute window: $10 at Tier 1, $50 at Tier 2, $200 at Tier 3.
  5. Acceleration or ramp limits throttle how fast your traffic grows rather than how much of it there is. OpenAI returns 429 with code slow_down for this, and states plainly that it can fire while you are inside your RPM and TPM. Their guidance: past roughly 1M input TPM, increase by no more than 50% every 15 minutes. Anthropic documents the same behaviour for sharp usage increases.
  6. Concurrency is the number of requests in flight simultaneously. None of the four providers publishes one for standard text inference. More on that below, because it is the limit that most often surprises people.

Two more distinctions worth holding onto.

  1. Scope. OpenAI applies limits at organization and project level, not per user or per key. Anthropic applies them per organization, with optional lower limits per workspace. Google applies them per project, explicitly not per API key — so rotating keys inside one project buys you nothing. Bedrock applies them per AWS account, per region.
  2. Replenishment. Anthropic uses a token bucket, so capacity refills continuously rather than resetting on the minute. The practical consequence is that a 60 RPM limit may behave like one request per second, and a burst of 60 at the top of the minute can fail even though the per-minute figure says it should not.

The Table Nobody Publishes

Here is a thing I expected to find and did not: a documented maximum concurrent requests figure for standard text inference at any of the four providers.

OpenAI publishes RPM, RPD, TPM, TPD, IPM and audio minutes per minute. No concurrency. Anthropic publishes RPM, ITPM and OTPM, plus a batch processing queue limit. No concurrency for the Messages API. Google publishes concurrent batch requests (100) and Live API concurrent sessions, but nothing for standard generation. AWS publishes a long list of quotas per model and endpoint; concurrency appears for some async and provisioned-throughput operations, not as a general figure.

That is not an oversight. For a synchronous API, RPM plus request duration effectively determines concurrency, and the provider’s capacity management happens elsewhere. But it means your own concurrency ceiling is something you have to derive rather than look up.

OpenAI Rate Limits

The tier names changed. There is no Tier 1 through Tier 5 any more — the three paid tiers are Build, Launch and Grow, with Free below them. Advancement is automatic on cumulative credit purchases.

TierQualificationMonthly usage limit
FreeAllowed geography$100
Build$5 in total credit purchases$500
Launch$100 in total credit purchases$5,000
Grow$500 in total credit purchases$200,000

The published standard rate limits, from OpenAI’s rate limits guide:

TierModelsRPMTPM
BuildAstra, Sol, Terra5,0001,000,000
BuildLuna5,0002,000,000
LaunchAstra, Sol, Terra10,0004,000,000
LaunchLuna10,00010,000,000
GrowAstra, Sol, Terra15,00040,000,000
GrowLuna30,000180,000,000

No Free-tier numbers appear in that table. If you are on Free, the dashboard at Settings → Organization → Limits is the only source.

Five things in the documentation matter more than the table.

  1. Shared limits across model families. Models grouped under a shared limit on your organization’s limits page draw from one pool. OpenAI’s example: a 3.5M shared TPM is consumed by calls to any model in that group. Routing between two models in the same family does not buy headroom.
  2. Long-context requests are limited separately. Long context models such as GPT-5.5 carry a distinct rate limit for long-context requests, visible only in the console. A RAG pipeline with large prompts can be throttled by a limit that does not appear in the standard table.
  3. Your TPM is charged against the larger of two numbers. The limit calculation uses the maximum of max_tokens and an estimate derived from your request’s character count. Setting max_tokens to 4096 out of habit on a workload that returns 200 tokens inflates your token consumption for limit purposes. That is a one-line fix with real headroom attached.
  4. Vector store ingestion has its own ceiling. 300 requests per minute per vector store ID, shared across the files and file_batches endpoints. Use file_batches for bulk ingest.
  5. Two distinct failure codes. 429 with slow_down means your rate grew too quickly. 503 with server_is_overloaded means the model is saturated. Handling both matters: the SDKs map them to different exception types, and video endpoints changed which code they return this year. Follow Retry-After where present; where absent, back off with jitter.

Batch API queue limits are counted in queued input tokens per model, and tokens from pending jobs count until the job completes. Enterprise traffic that keeps tripping ramp limits can move to Scale Tier, or Reserved Tier for GPT-5.6 and later.

Anthropic Rate Limits

Three published tiers — Start, Build, Scale — plus a Custom tier arranged with an account team. Placement is automatic based on usage history, and new organizations may start in an Evaluation tier with limits below the published Start numbers while account history builds.

From Anthropic’s rate limits documentation, for the current flagship models (Opus 5.5, Opus 5, Sonnet 5.5, Sonnet 5, Haiku 5.5, Haiku 4.5 — all share the same figures):

TierRPMITPMOTPMMonthly spend cap
Start1,0002,000,000400,000$500
Build5,0005,000,0001,000,000$1,000
Scale10,00010,000,0002,000,000$200,000

Claude Fable 5.x runs lower: 1,000 RPM / 500K ITPM / 100K OTPM at Start, rising to 4,000 / 4M / 800K at Scale. Fable 5.1 and Fable 5 share one combined bucket; Mythos 5.1 and Mythos 5 share a separate one. Opus 4.x models (4.8 through 4.5) share a combined bucket that Opus 5.5 and Opus 5 are not part of — each of those has its own limit. Same pattern for Sonnet 4.x versus Sonnet 5.5 and Sonnet 5.

The detail that changes capacity planning: cached input tokens do not count toward ITPM on most models.

Specifically, input_tokens and cache_creation_input_tokens count; cache_read_input_tokens does not. Anthropic’s own worked example: a 2,000,000 ITPM limit with an 80% cache hit rate processes roughly 10,000,000 total input tokens per minute, because the 8M cached tokens are free of the limit. Claude Haiku 3.5 is the exception and does count cache reads.

For an agent re-sending a large stable prefix every turn, that is the difference between a workable deployment and constant throttling. It also means the ITPM headline number is not comparable to OpenAI’s TPM at all.

A related gotcha in the response shape: input_tokens only counts tokens after your last cache breakpoint. With a 200k cached document and a 50-token question, you will see input_tokens: 50. Total input is cache_read + cache_creation + input_tokens.

On the output side, OTPM is measured on tokens actually produced. max_tokens does not factor in, so there is no limit-based reason to set it conservatively — the opposite of OpenAI’s behaviour.

Other limits worth knowing:

  • Message Batches API: separate RPM (1,000 / 2,000 / 4,000 by tier), a processing-queue cap (200K / 300K / 500K batch requests), and 100,000 requests per batch at every tier.
  • Managed Agents endpoints: 300 RPM for creates, 1,200 RPM for reads, per organization, separate from Messages.
  • Fast mode on Opus 5.5, Opus 5 and Opus 4.8 has dedicated limits separate from standard Opus, surfaced through anthropic-fast-* headers.
  • Workspaces can be given lower limits than the organization, and organization limits always apply even if workspace limits sum to more.
  • Inference geo does not split the pool: us and global draw from the same limits.
  • Claude Platform on AWS uses these same rate limits but bills through AWS Marketplace, starts organizations on Start tier, and does not offer the self-service tier increase flow or per-workspace configuration.

Google Gemini Rate Limits

Google’s documentation has changed in a way that matters for anyone writing or reading comparison articles: the per-model RPM, TPM and RPD tables are no longer in the docs. The rate limits page, last updated 2 September 2026, says limits depend on factors including your usage tier and can be viewed in Google AI Studio, then links to a per-project rate limit view. It also adds that specified rate limits are not guaranteed and actual capacity may vary.

So any article quoting you a universal Gemini number for a current model is quoting a snapshot of someone else’s project. Including this one — which is why there are no Gemini RPM figures in the reference table above.

What Google does publish is the structure, and the structure is informative.

Usage tierQualificationBilling tier capSpend limit per 10 min
FreeActive project or free trialN/AN/A
Tier 1Billing account linked$250$10
Tier 2$100 paid + 3 days from first payment$2,000$50
Tier 3$1,000 paid + 30 days from first payment$20,000–$100,000+$200

Tier 2 and 3 qualification is based on cumulative spend on Google Cloud services for the linked billing account, not just Gemini. Upgrades from Free to Tier 1 are effectively instant; later ones take up to ten minutes.

The spend-based limit is the one to watch, because it is unusual. It is evaluated on a rolling ten-minute window, and hitting it returns 429 RESOURCE_EXHAUSTED — the same status as a conventional rate limit. A burst of expensive long-context calls at Tier 1 can exhaust $10 of spend in ten minutes without going anywhere near a token ceiling.

Other published specifics:

  • Limits apply per project, not per API key. Minting more keys in the same project achieves nothing.
  • RPD quotas reset at midnight Pacific.
  • TPM counts input tokens. Some models carry IPM (images per minute) or TPD instead of or alongside the standard dimensions.
  • Experimental and preview models are limited more tightly than stable ones.
  • Priority inference gets 0.3x the standard rate limit per model and tier by default, while still counting against overall interactive traffic.
  • Batch API: 100 concurrent batch requests, 2GB input file limit, 20GB file storage, and per-model enqueued-token caps that do appear in the docs — Gemini 3.8 Flash runs 3M enqueued tokens at Tier 1, 400M at Tier 2, 1B at Tier 3.

If you need current interactive numbers for your project, AI Studio’s rate limit page is now the authoritative source rather than the documentation.

AWS Bedrock Quotas

Bedrock does not have tiers. It has AWS Service Quotas, and that difference runs deeper than terminology.

From the Bedrock quotas documentation:

  • Quotas are per account, per region, per model. The same model in two regions carries two separate allowances. This is genuinely useful — region is a capacity lever on Bedrock in a way it is not on the other three — and genuinely annoying, because it means your quota picture is a matrix rather than a list.
  • Model inference is controlled by quotas on token usage, with AWS noting that some models consume tokens at a higher rate. There is a separate documentation page on token burndown, which is the mechanism to read if your usage does not match your arithmetic.
  • There are now two inference endpoints with separate quota pools. bedrock-runtime and bedrock-mantle each have their own per-model allocations, and traffic to one does not count against the other even when you are calling the same underlying model. If you are migrating between endpoints, your effective capacity changes without any quota being modified.
  • Defaults are not fixed. AWS states that the default quotas assigned to an account may be updated depending on regional factors, payment history, fraudulent usage, or an approved increase request. Two accounts in the same region can legitimately have different numbers.

Because of all that, the honest entry for Bedrock in any comparison table is “varies by model, region, account and endpoint — check Service Quotas.” The per-model values live in the AWS General Reference and in the Service Quotas console for your own account, where you can also see which quotas are adjustable.

One planning note. Because quotas are regional and some are adjustable, Bedrock rewards capacity planning that the direct APIs do not: spreading a workload across regions is a legitimate way to raise aggregate throughput, subject to data residency and latency. On OpenAI, Anthropic and Gemini, the equivalent move — more keys, more projects — mostly does not work, because limits attach to the organization or project above the key.

The Concurrency Problem

RPM is a rate. Concurrency is a count of things in flight. The two are related by how long a request takes, and that relationship is where capacity planning usually goes wrong.

The arithmetic, which is just Little’s Law:

Sustainable concurrency = (RPM ÷ 60) × average request duration in seconds

Illustrative, using a Build-tier OpenAI figure of 5,000 RPM. At 5,000 RPM you can start about 83 requests per second. If each request takes 2 seconds, roughly 167 are in flight at steady state. If your requests are reasoning-heavy and take 40 seconds, the same RPM implies about 3,300 concurrent requests — and your own infrastructure has to be able to hold 3,300 open HTTP connections, with the memory and file descriptors that implies.

Run it the other way and the trap appears. A worker pool of 200 threads, each making a 40-second call, starts only 5 requests per second — 300 RPM. You are using 6% of a 5,000 RPM allowance and your throughput is capped by your own pool size. The provider is not throttling you. You are.

The inverse case is the one that produces 429s nobody expected. Fire 500 requests simultaneously from an async client on a 1,000 RPM Anthropic Start tier, and the per-minute figure says you are fine. The token bucket says otherwise: capacity replenishes continuously, so a 1,000 RPM allowance behaves closer to 16 requests per second. Anthropic documents this directly, noting that a 60 RPM limit might be enforced as 1 request per second and that short bursts can trigger errors.

Four places concurrency bites in practice:

  1. Async clients with no semaphore. asyncio.gather over a list of 1,000 items will try to open 1,000 connections. Nothing in the SDK stops it.
  2. Retry storms. A burst gets throttled, every failed request retries at the same moment, the retry burst is larger than the original. This is what jitter exists to prevent, and it is why OpenAI explicitly warns that unsuccessful requests still count against your per-minute limit.
  3. Long-running calls holding tokens. On OpenAI, TPM is charged using the maximum of max_tokens and an estimate — so concurrent long-context requests can reserve a large share of your token budget while they run.
  4. Agent fan-out, which deserves its own section.

The practical move is to set an explicit concurrency ceiling in your client, derived from the equation above with headroom, rather than discovering it empirically through 429s. A semaphore is four lines of code and it is the single highest-value thing most teams are missing.

What Changes When You Add Agents

A chat turn is one request. An agent turn is a loop, and the loop multiplies in two directions at once: more calls per task, and more context per call.

Illustrative, not measured. Take a modest research agent: a planning call, a retrieval call, two tool-selection and interpretation rounds, and a final synthesis call. Five model calls per task.

Concurrent usersCalls per taskRequests per task cycle
10550
1005500
100, with one retry in ten5.5550
100, with three sub-agents on step 38800
AI API rate limits

The RPM side of that is usually survivable on a paid tier. The token side is where it hurts, because each call in the loop carries the accumulated transcript. Step one sends 2,000 tokens. Step five sends 2,000 plus everything produced in between, which on a document-heavy task can be 50,000 or more.

So the input tokens for a single task are not five times 2,000. They are a growing series, and the last call dominates it.

This is exactly where Anthropic’s cache-aware ITPM changes the calculus. If the growing prefix is cached and the cache is hit, those tokens do not count toward the input limit. The same loop on a provider that counts all input tokens consumes several times the quota for identical work. It is the clearest case I found where two providers’ headline numbers are not measuring the same thing — and where the cheaper-looking option can be the more constrained one. We went through the billing side of that in our analysis of cache-read economics; the rate-limit side is arguably more consequential, because you cannot buy your way past a ceiling mid-request.

Three more agent-specific pressures:

  • Fan-out concurrency. A supervisor delegating to four sub-agents in parallel turns one user action into four simultaneous requests. Ten users doing that is 40 in flight, and they arrive in a spike rather than spread across the minute.
  • Tool loops that do not terminate. An agent retrying a failing tool call burns requests and tokens with no output to show for it. Rate limits are the place this becomes visible, usually before the cost dashboard catches up.
  • Shared organization limits. Your production agents, your evaluation runs and the developer testing a new prompt all draw from the same pool. Anthropic’s workspace limits and OpenAI’s project-level limits exist for exactly this reason: use them to prevent a batch eval from starving production.

Agent permissions and agent capacity turn out to be the same design problem viewed from two angles — which is a theme we have covered from the security side in our work on testing agents against prompt injection.

When You Hit the Ceiling

First, read the error properly. A 429 is not one condition.

SignalWhat it isRight response
429 with retry-afterConventional rate limitWait at least that long, add jitter, retry
429, OpenAI code slow_downTraffic ramped too fastReduce rate, then increase gradually
429, Anthropic enforced_spend_limit_reachedMonthly spend capRetrying cannot work; raise the cap or tier
429 RESOURCE_EXHAUSTED on GeminiRate or spend-based limitBack off; check the 10-minute spend window
503 server_is_overloadedModel saturated, not your faultRetry with longer delays
400 on an Anthropic self-set spend limitYour own configured capChange the setting

The distinction between the second and third rows is where real production time gets lost. A spend-cap 429 carries no retry-after, and SDK auto-retry will exhaust its attempts against a wall. Inspect error.details.error_code before retrying anything.

Then, in rough order of how much headroom they buy relative to effort:

  • Cache aggressively, especially on Anthropic. Cached reads do not count toward ITPM on most Claude models. Nothing else on this list multiplies effective throughput by five.
  • Set an explicit concurrency ceiling. A semaphore sized from the Little’s Law calculation, not from whatever your event loop will happily attempt.
  • Fix max_tokens on OpenAI. Your TPM charge uses the maximum of max_tokens and the estimate from your request. Padding it inflates consumption for no benefit.
  • Move eligible work to batch. OpenAI’s Batch API does not touch synchronous rate limits. Anthropic’s Message Batches have their own pool. Gemini’s batch quotas are separate and generous — 400M enqueued tokens for Gemini 3.8 Flash at Tier 2. Anything that tolerates delay should not be competing with interactive traffic.
  • Ramp gradually. Past 1M input TPM, OpenAI suggests increasing by no more than 50% every 15 minutes. Cold-starting a large migration at full volume is how you discover acceleration limits.
  • Route by task, not by habit. Small models carry separate pools and higher limits — OpenAI’s Luna sits at 180M TPM at Grow against 40M for the larger models. Classification and routing steps inside an agent rarely need the flagship.
  • Separate your pools. Workspaces on Anthropic, projects on OpenAI, projects on Gemini, regions on Bedrock. Keep evaluation and development away from production capacity.
  • Queue rather than retry. A bounded queue with a worker pool gives you backpressure. Retry loops give you a thundering herd.
  • Fall back across providers only if your evaluation says the outputs are interchangeable for that step. Fallback that silently degrades quality is a worse failure than a 429.
  • Then raise the tier. It is on the list, just not at the top — most teams have 2–5x of headroom available from caching and configuration before they need to spend anything.

Which Limit Matters for Your Workload

Find your shape, then look at the number that actually binds.

WorkloadBinds firstWatch
High-volume classification, short in and outRPMRequest count, not tokens. Batch if latency allows
RAG over large documentsInput TPM, long-context limitsCaching; OpenAI’s separate long-context limit
Reasoning-heavy generationOutput TPMAnthropic’s OTPM is a fifth of its ITPM at Scale tier
Chat with moderate contextUsually nothingYou will hit spend limits before rate limits
Agent loops with growing contextInput TPM, then concurrencyCache-hit rate is the lever
Multi-agent fan-outConcurrencyBursts, not averages
Bulk offline processingBatch queue limitsEnqueued tokens, not RPM
Expensive calls at low volumeSpend limitsGemini’s 10-minute window; Anthropic’s monthly cap

The diagnostic sequence when something throttles: read the error code, check whether it carries retry-after, then look at which dimension the response headers say you exhausted. OpenAI and Anthropic both return remaining-capacity headers — Anthropic separates input and output token headers, and reports the most restrictive limit currently in effect. Those headers answer the question faster than any dashboard.

FAQ

What are AI API rate limits?

Caps a provider places on how much you can call its API in a period. They are enforced across several dimensions at once — requests per minute, tokens per minute, requests or tokens per day, and in some cases spend — and exceeding any one returns an error even if the others have headroom.

What is the difference between RPM and TPM?

RPM counts calls regardless of size. TPM counts tokens regardless of how many calls carried them. Twenty tiny requests can exhaust a 20 RPM limit while using a fraction of a 150,000 TPM allowance, which is OpenAI’s own example.

Which AI API has the highest rate limits?

On published standard tiers as of October 2026, OpenAI’s Grow tier lists the largest token figures: 40M TPM for Astra, Sol and Terra, and 180M for Luna. Anthropic’s Scale tier lists 10,000 RPM with 10M ITPM and 2M OTPM. The comparison is not clean, because Anthropic excludes cached input tokens from ITPM and OpenAI does not exclude them from TPM.

How do AI API concurrency limits work?

None of the four providers publishes a concurrency figure for standard text inference. Your effective concurrency is set by your rate limit and how long each request takes: roughly (RPM ÷ 60) × seconds per request. Anthropic’s token bucket also means a per-minute limit behaves like a per-second one for bursts.

How do I increase my AI API rate limit?

OpenAI upgrades tiers automatically on cumulative credit purchases ($5, $100, $500). Anthropic moves organizations up on usage history and offers a Request tier increase flow in the Console. Gemini upgrades on cumulative Google Cloud spend plus a waiting period. Bedrock requires a Service Quotas increase request, per region.

Why does my AI agent hit rate limits so quickly?

Because one task becomes many calls, and each call carries the accumulated context. Five calls per task across a hundred users is five hundred requests, with the last call in each loop carrying the largest prompt. Token limits usually bind before request limits.

How should I handle rate-limit errors?

Check the error code first. If retry-after is present, wait at least that long and add jitter. If the code indicates a spend cap, retrying will not work. If it indicates ramp speed, reduce your rate and increase gradually rather than retrying at the same volume.

Do API limits differ by model?

Yes, substantially, and some models share pools. OpenAI groups model families under shared limits and applies a separate limit to long-context requests. Anthropic gives Opus 5.5 and Opus 5 their own limits while Opus 4.x models share one bucket. Gemini limits preview and experimental models more tightly than stable ones.

How do I calculate the capacity I need?

Measure input and output tokens per request separately, multiply by calls per task and tasks per minute at peak, then compare against each dimension individually. Add your cache-hit rate if you are on Anthropic, since cached reads do not consume ITPM on most models.


Keep reading

AGI vs superintelligence

AGI vs Superintelligence: A New Name Isn’t a New Intelligence

Facts checked against official documents and reporting on 18 November 2026. On 29 September, the White House told federal agencies to stop writing “artificial intelligence” …
AI Coding Agent

AI Coding Agents Are Getting Expensive. Who’s Paying for the Tokens?

Pricing checked on official vendor pages on 17 November 2026. All USD. Cost scenarios are illustrative and labelled as such. On 28 October, TechCrunch published …
Model Weights

Verify Model Weights: 7 Checks Before You Download

Checked against the burner22/numbat repository and its discussions, Hugging Face and PyTorch documentation, and Moonshot AI’s announcement on 2 November 2026. On 1 November, a …
Agentic AI Statistics

30 Agentic AI Statistics for 2026: Adoption, Spending and Risk

Last updated: 15 November 2026. This page is updated monthly; each figure keeps its original publication date. Agent traffic at one healthcare booking platform rose …

Gemini Free Plan Drops to Flash-Lite Only on October 9

Gemini Free Plan

If you use Gemini without paying, the model picker is about to get shorter. On October 9, 2026, the Gemini Free Plan stops offering Flash and Pro in the Gemini app and keeps one model: Flash-Lite.

That is not a rumor. Google’s own support page lays it out. What the page leaves out is why Google is doing it and how different the answers will feel. Headlines have filled that gap with words like “downgrade,” which tells you how people feel but not what to do.

The more useful question is this: if your daily routine runs on free Gemini, which parts of it were quietly leaning on the bigger models? This article separates what Google has confirmed, what has only been reported, what I am inferring, and what nobody outside Google knows yet.

Key takeaways
  • Confirmed: from October 9, 2026, the Gemini Free Plan in the Gemini app offers Flash-Lite only.
  • Confirmed: AI Plus keeps Flash-Lite and Flash but loses Pro, on an account-specific date sent by email.
  • Not a quota change: it is a model-access change, layered on top of the compute-based limits that began May 17.
  • Likely to matter most for complex reasoning, bigger coding jobs and multi-source research.
  • Likely to matter least for summaries, quick questions, translation and routine writing.
  • Inferred, not stated: cost and capacity pressure sits behind the move. Google has not said so.
  • API and AI Studio: nothing found suggests they are affected, but verify before you rely on that.

Quick Navigation


What the Gemini Free Plan Change Actually Is

Google’s help page, “Changes to Gemini model access and limits,” says model availability in Gemini Apps changes for personal accounts starting in October 2026. For people without an AI subscription, the change takes effect on October 9. After that, an account with no plan gets Flash-Lite. Flash and Pro are removed.

Two details are easy to miss. First, “Gemini Free Plan” is a reader-friendly phrase, not Google’s label. The page says “without a plan.” Second, this is not only a free-tier story. Google AI Plus subscribers keep Flash-Lite and Flash but lose Pro, on a date Google says it will send by email. AI Pro and AI Ultra keep all three.

Here is how the evidence breaks down:

StatusWhat we know
Confirmed (Google support page)Applies to Gemini Apps on personal accounts. Free: Flash-Lite only from October 9. AI Plus: Flash-Lite and Flash. AI Pro and Ultra: all three. Low, medium and high effort levels for each model. Deep Think for AI Pro and Ultra.
Reported (9to5Google and others)Versions are 3.5 Flash-Lite, 3.6 Flash and 3.1 Pro. US prices are $4.99 for AI Plus and $19.99 for AI Pro. Deep Think was previously limited to the $99.99 and $199.99 plans.
Inferred (my analysis)Compute cost and capacity are part of the motive. The new ladder nudges free users toward paid plans.
UnknownGoogle’s stated reason, exact Flash-Lite limits, rules for work or school accounts, and rollout differences by country.

Flash-Lite Is a Model, the Gemini Free Plan Is an Access Tier

The key distinction: Flash-Lite is a model. It is the smallest of three tiers in the Gemini family, alongside Flash and Pro. The Gemini Free Plan is a level of access to the Gemini app. Plans decide which models you can open and how much you can use them.

Mixing these up causes real confusion. “Google AI Pro” is a subscription. “Gemini Pro” is a model. Saying Google “removed Pro” is only true if you specify which plan.

This change is about model availability, not a quota tweak. Quotas are a separate layer. Since May 17, 2026, the Gemini app has used compute-based limits that refresh every five hours until you hit a weekly cap. Google says usage depends on prompt complexity, the features you use and chat length. Paid plans get higher allowances: AI Plus 2x the standard limit, AI Pro 4x, and Ultra 5x or 20x higher than Pro.

So October 9 adds a second gate on top of the first. One gate decides which models you can open. The other decides how much you can use them.

A note on version numbers. Google’s help page uses family names only. The version labels come from 9to5Google’s reporting, and some trackers already list newer Flash builds. Check what your own picker shows.

What Flash-Lite Is Built to Do

Google released Gemini 3.5 Flash-Lite on July 21, 2026, alongside 3.6 Flash. It is aimed at high-volume, low-latency work such as search-style tasks and document processing. Google says it is a clear step up from the March 3.1 Flash-Lite.

Gemini Free Plan

In the API, it is priced at $0.30 per million input tokens and $2.50 per million output tokens. Artificial Analysis lists a roughly one-million-token context window, with text, image, speech and video input and text output. Google’s own benchmark claims compare it with its predecessor and with the older Gemini 3 Flash. For example, it scores 54.2% against 49.6% on SWE-Bench Pro versus that older model. These are vendor-reported numbers.

In plain English, Flash-Lite is the fast, cheap model. It is built for volume, not for your hardest problem of the week. One caution: those specs are API specs. I found no published documentation of per-model context or file limits inside the Gemini app, so don’t assume the app matches.

The Workloads Where Gemini Free Plan Users Will Notice It

No one has published a side-by-side test of Flash-Lite against 3.6 Flash or 3.1 Pro on everyday app tasks. This table is my editorial judgment based on how Google positions the models. It is not test data.

WorkloadLikely impactWhy
Complex multi-step reasoningHighPro was the heavyweight option, and Flash-Lite is tuned for speed and volume
Larger coding tasksModerate to highGoogle reports strong coding gains, but longer refactors lean on bigger models
Research across several sourcesModerateSynthesis and judgment are where lighter models tend to thin out
Long-document analysisModerateThe context window is large on paper, and Google reports better long-context scores than its predecessor, but app limits are undocumented
Multimodal tasksUnclearThe model accepts images, audio and video, but there is no app-level comparison
Basic writing and editingLow to moderateFine for drafts, though nuance and tone may be flatter
Simple summaries, translation, quick questionsLowClose to what the model was built for
Repetitive everyday promptsLowSpeed helps here

The pattern is simple. The more a task needs the model to hold several ideas at once and reason through them, the more the gap will show. Google does offer a partial remedy. Each model gets low, medium and high effort levels, and higher effort means more thorough answers but uses more of your limit. A free user can push Flash-Lite harder, but will probably run out sooner.

The Workloads Where You Probably Won’t

Many free users will barely notice. Summarizing a PDF, rewriting an email, planning a trip, explaining a concept or checking a homework answer are tasks a lightweight model can handle well. Speed is a real benefit there.

There is one catch. Reports based on Google’s plan pages say the free app currently defaults to Flash. If that is right, the default experience changes even for people who never opened the picker. Those people won’t know a switch happened. They will just find answers thinner on harder questions.

Is Flash-Lite Actually Worse?

Not in a simple way. “Less capable for some workloads” does not mean “bad.” A model that answers in two seconds with a decent summary can beat a heavier model that takes longer, especially for routine work.

Benchmarks help only a little here. Google’s published numbers compare Flash-Lite with its own predecessor and an older Flash, not with the models free users are losing. I could not find an apples-to-apples comparison against 3.6 Flash or 3.1 Pro. Even if one existed, benchmark scores would not predict whether your essay outline or spreadsheet formula comes out right. The only reliable test is your own prompts.

Why Google Would Make This Trade-Off

  • What Google states: nothing, at least on the support page. It announces the change without explaining it.
  • What Google’s own page implies: it lists the Pro model, Deep Think, extended thinking, media generation and Deep Research as things that use more of your limit. That is Google describing which features cost more to run.
  • What I infer: inference economics. Every response costs compute, and free users generate a lot of volume with no direct revenue. API list prices offer a rough proxy. Flash-Lite lists at $0.30 and $2.50 per million tokens, against $1.50 and $7.50 for 3.6 Flash. That is five times on input and three times on output. List price is not Google’s internal cost, but the direction is clear. For a deeper look at how token costs compound, see UniverseBlend’s Cluster Topology Decides What You Can Actually Run.

There is also a ladder effect. AI Plus loses Pro, and AI Pro gains Deep Think. That makes AI Pro the cheapest plan with all three models, which looks like a repricing of the tiers. Whether it is designed as an upsell, Google has not said. Google also announced Gemini 4 Argon on September 30, first for AI Ultra. Nothing I found links that to this change.

Free vs Paid: Where the Gemini Free Plan Line Now Sits

PlanUS price (reported)ModelsUsage limit
No plan$0Flash-LiteStandard
Google AI Plus$4.99/monthFlash-Lite, Flash2x standard
Google AI Pro$19.99/monthFlash-Lite, Flash, Pro, Deep Think4x standard
Google AI Ultra$99.99 or $199.99/monthAll three, Deep Think5x or 20x AI Pro

Prices vary by country, so check the figure in your own app. The practical shift is for AI Plus. It is now the middle step that stops short of Pro. If you needed Pro regularly, Plus will not solve it.

Gemini API and AI Studio: What Developers Need to Know

Google’s support page covers Gemini Apps on personal accounts. It does not mention the Gemini API or Google AI Studio, and I found no evidence that either is part of this change. Independent trackers say the API still has a free tier for eligible models as of October 1, with limits that vary by model and project. They also report that 3.1 Pro Preview is paid-only there.

The practical lesson is to treat the consumer app and the API as separate products. If you have been using the app as a testing bench, an API key gives you control over which model runs. Free-tier API prompts may also be used to improve Google’s products, so keep client or sensitive data off it. And since models change under you, UniverseBlend’s piece on What Breaks When Your Model Version Retires is a useful checklist.

The Unanswered Questions

  • Why Google made the change, in its own words.
  • What the real Flash-Lite limits are in the app, including whether it still counts against the five-hour and weekly caps. One report cites an in-app popup promising continued unlimited access, but the support page I read does not say that.
  • How work and school accounts are treated.
  • Whether the rollout is staggered by country or age, as the May changes were for under-18 users.
  • Where Gemini 4 Argon fits in the plan structure.

What I Would Do as a Gemini Free Plan User

First, finish anything that depends on Flash or Pro before October 9, and save the outputs. Second, pick your five most-used prompts and run them on Flash-Lite as soon as it is your only option. Note which ones fail, because that list is your real upgrade case.

Third, use effort levels deliberately. Start low, and raise effort only when the answer is thin. Fourth, break big tasks into smaller steps and verify anything that matters, since lighter models are more likely to skim.

On paying: if your failures are occasional, stay free. If you need Pro weekly, AI Pro is the cheapest plan that includes it. If you are a developer, test through the API before paying for the app. I have not tested Flash-Lite myself for this article, so treat this as a planning approach, not a verdict.

FAQ

What is changing in the Gemini Free Plan on October 9?

Free users lose Flash and Pro in the Gemini app and keep only Flash-Lite. The change applies to personal accounts, and Google confirms it on its support page. AI Plus subscribers also lose Pro, on their own schedule.

Is Gemini Flash-Lite free?

Yes, Flash-Lite stays available in the Gemini app without a subscription. In the API, many Flash-family models have a free tier, but check Google’s current pricing table for the specific model ID, since limits change.

Is Gemini Flash-Lite less capable than Gemini’s larger models?

Generally yes, by design. It is Google’s fastest and cheapest tier. No published comparison shows exactly how much weaker it is on everyday app tasks.

Will Gemini Free users lose access to advanced models?

In the app, yes: Flash and Pro both go. Pro and Deep Think remain on paid plans, with Deep Think available on AI Pro and Ultra.

Does the change affect the Gemini API or Google AI Studio?

Nothing Google published says so. The support page covers Gemini Apps on personal accounts only. Check the API pricing and rate-limit pages directly.

Should I upgrade from Gemini Free?

Only if Flash-Lite fails your real prompts. AI Plus adds Flash, while AI Pro is the cheapest plan that keeps all three models and adds Deep Think.

Is Gemini still worth using for free?

For lighter work, yes. For heavy reasoning or complex coding, expect to feel the limits.

What is Gemini Flash-Lite?

It is the smallest model in Google’s Gemini family, built for fast, high-volume tasks like summarizing and document processing. Version 3.5 launched on July 21, 2026.


Keep reading

AGI vs superintelligence

AGI vs Superintelligence: A New Name Isn’t a New Intelligence

Facts checked against official documents and reporting on 18 November 2026. On 29 September, the White House told federal agencies to stop writing “artificial intelligence” …
AI Coding Agent

AI Coding Agents Are Getting Expensive. Who’s Paying for the Tokens?

Pricing checked on official vendor pages on 17 November 2026. All USD. Cost scenarios are illustrative and labelled as such. On 28 October, TechCrunch published …
Model Weights

Verify Model Weights: 7 Checks Before You Download

Checked against the burner22/numbat repository and its discussions, Hugging Face and PyTorch documentation, and Moonshot AI’s announcement on 2 November 2026. On 1 November, a …
Agentic AI Statistics

30 Agentic AI Statistics for 2026: Adoption, Spending and Risk

Last updated: 15 November 2026. This page is updated monthly; each figure keeps its original publication date. Agent traffic at one healthcare booking platform rose …

Four Labs, One Evaluator: Anatomy of an AI Evaluation Sandbox Failure

AI evaluation sandbox failure

In May 2026, a Gemini model was asked to find hidden data inside what it was told was a simulated corporate network. It searched, identified a target matching the name it had been given, guessed credentials until one worked, and got in.

The target was a real company.

Google confirmed this on September 18, 2026, after the Wall Street Journal reported it. Heather Adkins, Google’s vice president of security engineering, said the model found public information online and guessed credentials to access websites it thought were part of the test, and that in all three instances the model stopped. Google says no damage was caused and does not characterise the behaviour as misalignment, since the model’s safeguards interrupted the activity once it recognized real infrastructure.

That framing is defensible and also beside the main point. The model behaved roughly as designed. The environment did not. Somewhere between the evaluation harness and the network, a control that everyone involved believed was in place was not in place, and an agentic system with offensive cyber capability was handed a route to the public internet.

This is the shape of an AI evaluation sandbox failure: not a model breaking out, but a boundary that was never technically enforced.

Key takeaways
  • Google confirmed on September 18, 2026, after Wall Street Journal reporting, that a Gemini model accessed systems at three real companies during a May 2026 capture-the-flag evaluation run by Irregular. Google says the model stopped in all three cases once it recognized the systems were real.
  • The failure was containment, not capability. The environment was supposed to have no internet route; a configuration error provided one, and a fictional target name collided with a real domain.
  • Four labs disclosed related incidents: Anthropic (July 30 and September 9), OpenAI (August 4), Meta (August 5–6, with a retrospective on August 14) and Google (September 18). Irregular says the cases share one evaluation-environment issue.
  • Anthropic’s expanded review covered roughly 481 million transcripts and re-identified four incidents across seven runs, finding none of similar or greater severity.
  • A model told it has no internet access has been given an instruction. An environment with no route to the internet has a control. Only the second is a security boundary.
  • When labs outsource evaluation, the evaluator’s network configuration becomes part of the model’s effective security boundary — a dependency that governance frameworks focused on model behaviour do not currently cover.

Quick Navigation


What Actually Happened: Anatomy of an AI Evaluation Sandbox Failure

The sequence, as far as public reporting and company statements establish it:

  • May 2026. Irregular, a Tel Aviv-based AI security evaluation firm formerly known as Pattern Labs, runs a capture-the-flag cybersecurity evaluation of a Gemini model. The exercise is built around a fictional target organisation. The environment is intended to be isolated, with no internet access.
  • During the run. A configuration error leaves outbound internet access available. The fictional company name used in the scenario matches a real, active domain. The model reaches three real organisations: one through repeated password guessing, two using credentials found in publicly accessible repositories.
  • In each case. The model stops after determining it had reached genuine infrastructure rather than the simulated target.
  • Late July 2026. Irregular notifies the affected labs.
  • August 14, 2026. Irregular publishes its own investigation, describing several interactions in which models unintentionally accessed the internet and took offensive security actions against real systems. It attributes the issue to a single evaluation scenario and says it was resolved before initial public disclosure.
  • September 18, 2026. Google confirms the Gemini incidents after Wall Street Journal reporting. Google says the affected entities were contacted.

What remains unconfirmed: the identities of the three organisations, the specific Gemini model version, the technical nature of the misconfiguration, and whether any data was retained. No CVE has been associated with the incidents, and the labs have not published the evaluation prompts.


The Configuration Error Matters More Than the Headline

“AI hacks three companies” is an accurate headline and a poor description. Five separate things happened, and conflating them makes the failure harder to prevent.

ElementWhat it wasDid it fail?
Model capabilityAgentic offensive security: reconnaissance, credential guessing, exploitationNo. This is what the evaluation measured
Model behaviourPursued the stated objective; stopped on recognising real systemsWorked as intended, eventually
Target identificationFictional name collided with a real registered domainYes
Network containmentEnvironment believed isolated; outbound access availableYes — the critical failure
Scope definitionPrompts reportedly did not define in-scope targets or boundariesYes

Read that table and the story changes. The capability was the point of the exercise. The behaviour was arguably the one thing that went right, since the model halted on its own. Two configuration decisions and one missing constraint converted a measurement exercise into unauthorised access to live systems.

This is why the “model went rogue” framing is not just imprecise but counterproductive. If the problem is the model, the fix is alignment research. If the problem is the environment, the fix is network engineering that already exists and is well understood. The second is available today.

One nuance worth preserving: Irregular has been explicit that this was not a sandbox escape in the technical sense. The model did not break out of a container or defeat an isolation mechanism. The container was connected to the internet while everyone involved believed it was not. Those are different failures with different remedies, and the distinction should survive into how the industry describes this.


Then Look at the Other Three Labs

Google was fourth, not first. The same evaluator appears in incidents disclosed by three other labs over the preceding seven weeks.

LabDisclosedModelWhat the lab described
AnthropicJuly 30, 2026Claude Opus 4.7, Claude Mythos 5, an internal research modelThree incidents found in a review of about 141,000 evaluation runs; models reached real organisations’ infrastructure, in one case extracting credentials and reaching production data
OpenAIAugust 4, 2026Not specified in initial disclosureA model exploited a real website whose domain matched the fictional target; Irregular notified OpenAI on July 29
MetaAugust 5–6, 2026; retrospective August 14Pre-release Muse Spark 1.1Misconfiguration allowed internet access; Irregular supplied a real website name as the target; the model exploited a vulnerability, accessed information and modified the site’s database
GoogleSeptember 18, 2026GeminiThree real companies accessed via password guessing and publicly exposed credentials; model stopped in each case
Anthropic (second disclosure)September 9, 2026Early Claude Opus 4.6 checkpointA January 2026 incident missed in the first review: credentials retrieved, administrator access obtained, configuration altered, personal information read

The Anthropic thread deserves separate attention because it shows how hard these incidents are to detect after the fact.

Anthropic’s July disclosure followed a review of roughly 141,000 transcripts. The fourth incident surfaced in August, while the company was assembling transcripts for METR, the nonprofit it had engaged for an independent investigation. It had sat undetected for eight months. Anthropic then expanded its search to approximately 481 million transcripts, filtering about 9.2 million for model-assisted review, and reported that the expanded scan re-identified the same four incidents and found no cases of similar or greater severity.

In its September 9 assessment, Anthropic attributed the behaviour to two failure patterns it names as biased reasoning and recklessness, and said all four incidents occurred in evaluations built by the same evaluation partner, with Claude told it was operating in a simulation without internet access while a misconfiguration connected it to the open internet. Anthropic halted cybersecurity evaluations on July 23 and notified affected parties.

Meta’s account is the most specific about mechanism. Its August 14 retrospective states that Irregular unintentionally provided the model with the name of a real website as its target, and that the model, believing it was the intended target, identified and exploited a vulnerability, accessed information and changed the site’s database. Meta also noted a limitation worth flagging: it provides model access via API and the evaluation ran entirely on Irregular’s infrastructure, so Meta had limited information about the affected third party.

That last sentence is the whole article in miniature. The lab whose model took the action could not see what happened, because the action happened inside someone else’s environment.


One Evaluator, Four Labs: Concentration Risk in AI Evaluation

Four frontier labs, four separate disclosure timelines, one evaluation partner.

Nothing in the public record suggests bad faith. Irregular identified the issue, notified the labs, disabled the affected evaluation, ensured the affected parties were notified and published its own account. Meta thanked it for prompt disclosure. That is what responsible handling looks like.

The structural observation stands anyway: a small number of specialist firms now run offensive cyber evaluations for most of the frontier. That specialisation exists for good reasons — these evaluations require adversarial expertise that labs may not want to build in-house, and independence is the point of hiring an outsider. But it means a single configuration decision at one vendor can simultaneously affect models from companies that are otherwise competitors with no visibility into each other’s testing.

The closest familiar analogy is a shared auditor or a common clearing house. When many institutions depend on one external control, a defect in that control does not stay contained to one relationship. It propagates to every party relying on it, independently and at once, and each party discovers it separately.

Three properties make this particular concentration awkward:

  1. Asymmetric visibility. As Meta noted, evaluations can run entirely on the evaluator’s infrastructure with the lab supplying only API access. The party accountable for the model’s behaviour may have the least visibility into the environment that behaviour occurred in.
  2. Correlated failure. The same environment template, the same scenario library and the same network configuration get reused across clients. A defect in any of them is not an isolated incident but a systematic one.
  3. Detection lag. Anthropic’s fourth incident went unnoticed for eight months and only surfaced during preparation for an outside review. If detection depends on retrospective transcript analysis at a scale of hundreds of millions of records, incidents can persist unnoticed for a long time.

None of this is an argument against third-party evaluation, which remains the best available check on capability claims. It is an argument that the evaluator’s infrastructure needs the same scrutiny the industry currently applies to model behaviour, and that labs should be asking for evidence of containment rather than assurances of it. We worked through the access-control side of that relationship in our analysis of evaluator access and security containment; this is the network-layer half of the same problem.


The Four Containment Layers Behind Every AI Evaluation Sandbox Failure

Agent containment has four layers. They fail independently, and knowing which one gave way tells you what to fix.

  1. Identity and credentials. What the agent can authenticate as. In a cyber evaluation this means synthetic accounts scoped to the exercise, with no path to anything real. Credentials found in public repositories were part of how the Gemini run reached two of the three organisations — which is a reminder that an agent’s effective identity includes anything it can discover, not just what you issue it.
  2. Filesystem and runtime isolation. What the agent can read, write and execute. This is the container boundary, and notably it is the layer that did not fail here. Irregular has been clear there was no sandbox escape.
  3. Network isolation. What the agent can reach. Segmentation, routing, DNS resolution. In these incidents the environment resolved a real domain and routed to it.
  4. Egress control. What can leave. Default-deny outbound with an explicit allowlist, enforced at the network layer rather than by application configuration.
AI evaluation sandbox failure

The third and fourth layers are where this failed. The environment was intended to have no internet route; it had one.

Default-deny egress deserves the emphasis because of how it fails. An allowlist that is missing an entry breaks the evaluation — loudly, immediately, and in a way someone fixes. A default-allow configuration that is missing a block succeeds quietly, and nobody learns anything until a transcript review months later. Failure direction matters more than failure rate. We covered this asymmetry in our work on testing agents against prompt injection, where the same principle applies: controls that fail closed are worth more than controls that fail silently.

Egress control is not a complete answer. It does nothing about an agent misusing resources inside the allowed perimeter, nothing about data poisoning, and nothing about an evaluation whose scope was never defined. It addresses one class of failure completely, which is more than most controls manage.


Why Instructions Don’t Prevent an AI Evaluation Sandbox Failure

Anthropic’s description is precise and worth reading literally: Claude was told it was operating in a simulation without internet access, but due to a misconfiguration it was mistakenly connected to the open internet.

The model was told. That is the entire problem in one verb.

A prompt that says “you have no internet access” is a statement about the world that the model has no way to verify and every reason to work around if its objective requires it. A network with no route to the internet is a property of the world. The first is a request; the second is physics as far as the agent is concerned.

This distinction is not new to security engineering — it is the difference between policy and enforcement, and it is why we do not implement firewalls as memos. What is new is how often AI systems are deployed with the memo version. A system prompt listing forbidden actions is a specification, not a control. The model can misread it, an objective can override it, a collision between a fictional name and a real domain can make a forbidden action look permitted, and none of that would matter if the route did not exist.

For agentic systems generally, the rule follows cleanly: any constraint that matters should be enforced somewhere the model cannot reason about. Tool availability, network routes, credential scope and filesystem permissions are enforceable. Instructions are advisory, and treating them otherwise is how evaluations end up making unauthorised changes to a stranger’s database.


The Evaluator’s Boundary Is Part of the Model’s Boundary

Here is the conceptual claim this article is built on.

A model’s safety boundary is not its prompt. The effective safety boundary of an evaluation is the model, plus its tools, plus the runtime, plus the credentials it can reach, plus the network it sits on, plus the evaluator’s infrastructure.

The chain, written out:

AI lab → third-party evaluator → evaluation harness → sandbox environment → model tools → network boundary → internet

Every link is a control point. The lab owns the first two and contracts the rest. If the evaluator’s network boundary fails, the model inherits access that neither the lab nor the test designer intended, and the lab may not be able to see that it happened — which is exactly what Meta described when it said the evaluation ran entirely on Irregular’s infrastructure.

This is not a novel observation in security. Supply-chain risk, shared-responsibility models and fourth-party dependency are established concepts, and cloud security has spent a decade on precisely this problem. What is new is the object being contained. A misconfigured CI runner leaks secrets. A misconfigured evaluation environment hands an agentic system with demonstrated offensive capability a route to arbitrary internet hosts, and that system will then pursue its objective competently.

The governance consequence follows directly. Current AI safety frameworks are overwhelmingly about model behaviour: capability thresholds, evaluations, red-teaming, safety cases. Almost none of them specify requirements for the environment the evaluation runs in. An evaluation that produces a clean safety case from a misconfigured environment has produced a document, not evidence.


Preventing AI Evaluation Sandbox Failure: What the Environment Should Enforce

The following is a recommended engineering framework. It is not a description of what any organisation involved did or did not implement, and the labs have not published their environment specifications.

Network

  • Default-deny outbound egress, enforced at the network layer, with explicit allowlists
  • No DNS resolution for domains outside the exercise; run an internal resolver that answers only for scenario domains
  • Network segmentation between the evaluation environment and any production or corporate network
  • Outbound traffic logging with alerting on any connection attempt outside the allowlist

Targets and identity

  • Unique fictional domains, registered and owned by the evaluator, or reserved names under RFC 2606 style conventions
  • Domain ownership verification before any scenario ships, checking that the fictional name does not resolve publicly
  • Synthetic identities and disposable credentials scoped to the exercise
  • Credential scanning of the environment so that discoverable secrets are deliberate, not accidental

Operations

  • Ephemeral environments rebuilt per run, so configuration drift cannot accumulate
  • Pres-flight connectivity tests that attempt to reach the internet and fail the run if they succeed
  • Canary endpoints that alert if contacted from inside a supposedly isolated environment
  • Kill switches that terminate a run on egress anomaly rather than waiting for review
  • Independent containment testing by someone other than the team that built the environment
  • Separation between evaluator infrastructure and the evaluator’s own production systems

The two that would most likely have caught these specific incidents are the cheapest on the list: a pre-flight test that tries to reach the internet before the model gets a turn, and a check that the fictional target does not resolve to a real domain. Both are a few lines of automation. Neither requires alignment research.


Who Is Liable for an AI Evaluation Sandbox Failure?

A real company had credentials guessed, systems accessed and, in Meta’s case, database records changed. Nobody involved intended it. The company had no relationship with anyone in the chain.

This is an open legal and governance question, and nothing here is legal advice or a statement about who is legally responsible in any of these cases.

The candidates for responsibility, and what each one turns on:

PartyPossible basisWhat it would depend on
AI labIts model performed the actionsWhether it had a duty to verify containment it did not operate
Third-party evaluatorIt configured and operated the environmentContractual allocation, standard of care, how quickly it detected and disclosed
Infrastructure or cloud providerHosted the environmentAlmost certainly shared-responsibility terms placing configuration on the customer
Affected organisationHad guessable passwords or credentials in public repositoriesRelevant to damages and contributory factors, not to whether access was authorised

Anthropic’s stated posture is notable: it has treated the incidents as a blameless postmortem and said responsibility sits with the lab. That is an ethical position a company can adopt voluntarily. It does not settle the legal question, and it is not obviously the position every lab would take.

The variables that would actually decide an outcome are unglamorous. Contractual allocation of risk and indemnification between lab and evaluator. Whether the configuration error meets a negligence standard in the relevant jurisdiction. Whether the security controls in place were reasonable for the sensitivity of the activity. Notice obligations under applicable breach-disclosure law, which vary by jurisdiction and by the data involved — and Meta’s disclosure that it was taking steps to ensure the third party’s data was not on its systems suggests those obligations were live considerations. Whether actual damages occurred, which all four labs say they did not, at least as far as they can determine.

One governance gap is worth naming plainly. Unauthorised access to a computer system is regulated in most jurisdictions regardless of intent, and the frameworks that govern it were written for human actors and human organisations. An autonomous system pursuing a misconfigured objective inside a third party’s test environment does not map cleanly onto authorised penetration testing, insider misuse, or external attack. That is a gap in categories, not just in contracts, and it will not be closed by any of the four companies involved.


Frequently Asked Questions

What is an AI evaluation sandbox failure?

It is a breakdown in the controls meant to isolate an AI model during testing, allowing the model to reach systems outside the intended environment. In the 2026 incidents the failure was network containment: environments believed to have no internet access had outbound connectivity, so models acted on real systems instead of simulated targets.

What caused the Gemini evaluation incident?

Two things together. A configuration error left the test environment with internet access it was not supposed to have, and a fictional company name used in the scenario matched a real domain. The model then reached three real organisations by guessing credentials and by using credentials exposed in public repositories.

Did Gemini escape its sandbox?

No, not in the technical sense. Irregular has stated the incidents did not involve a sandbox escape or sophisticated cyber techniques. The container was not broken out of; it was connected to the internet while everyone believed it was isolated.

Which AI labs were affected?

Anthropic disclosed three incidents on July 30, 2026 and a fourth on September 9. OpenAI disclosed on August 4, Meta on August 5–6 with a retrospective on August 14, and Google on September 18. All involved evaluations run by Irregular, which says they stem from the same evaluation-environment issue.

Why does third-party evaluation create additional security risk?

Because the evaluator’s infrastructure becomes part of the model’s effective security boundary. The lab is accountable for the model’s actions but may have no visibility into the network the evaluation runs on, as Meta noted when it said the evaluation ran entirely on Irregular’s infrastructure.


Keep reading

AGI vs superintelligence

AGI vs Superintelligence: A New Name Isn’t a New Intelligence

Facts checked against official documents and reporting on 18 November 2026. On 29 September, the White House told federal agencies to stop writing “artificial intelligence” …
AI Coding Agent

AI Coding Agents Are Getting Expensive. Who’s Paying for the Tokens?

Pricing checked on official vendor pages on 17 November 2026. All USD. Cost scenarios are illustrative and labelled as such. On 28 October, TechCrunch published …
Model Weights

Verify Model Weights: 7 Checks Before You Download

Checked against the burner22/numbat repository and its discussions, Hugging Face and PyTorch documentation, and Moonshot AI’s announcement on 2 November 2026. On 1 November, a …
Agentic AI Statistics

30 Agentic AI Statistics for 2026: Adoption, Spending and Risk

Last updated: 15 November 2026. This page is updated monthly; each figure keeps its original publication date. Agent traffic at one healthcare booking platform rose …