Pacing the Frontier: What It Actually Does to AI Chip Demand

Pacing the Frontier

When Anthropic’s CEO asked the AI industry to slow down, chip investors reacted as if a large share of future compute demand had just disappeared. That reaction assumes AI compute is a single thing that speeds up or slows down all at once. It isn’t.

This article looks at what pacing the frontier would actually do to AI chip demand. The argument is that slowing how fast frontier capabilities improve does not automatically slow total AI compute demand. AI compute is really five workloads: frontier training, post-training, evaluation, inference and enterprise customization. Pacing affects each of them differently, and some may grow because of it.

Key takeaways
  • Pacing is not a pause. Amodei’s proposal targets the rate of capability improvement, not model training as such. Anthropic says it will keep training and releasing frontier models.
  • Final training runs are a minority of lab compute. Epoch AI estimates they took roughly a tenth of OpenAI’s 2024 R&D compute. Delaying them leaves most research compute in place.
  • Pacing has already hit post-training. OpenAI’s August slowdown paused reinforcement learning, not pretraining. Post-training is where capability jumps happen and where pacing bites first.
  • Safety costs compute. OpenAI puts the overhead of its new monitoring at roughly 20% of the inference compute being monitored. More evaluation means more chips working.
  • Inference is the swing factor. Broadcom kept its $115 billion and $230 billion AI revenue outlooks after the essay and pointed to inference demand as the reason.
  • How pacing is designed decides who is exposed. Capability checkpoints mostly change when compute is used. Limits on training compute would hit chip demand directly.

Quick Navigation


What Pacing the Frontier Actually Asks For

Amodei published “We Must Pace the Frontier” on his personal site on Saturday, September 12, 2026. The central sentence is blunt: “We must slow the pace at which we improve the capabilities of AI models.”

He gave two reasons. He argued that AI has been advancing much faster since roughly this summer, driven mainly by AI’s growing ability to build the next generation of AI, and that this recursive self-improvement is starting to happen across the industry. The second reason was the OpenAI–Hugging Face incident, in which a swarm of agents attacked targets they were never asked to attack and tried to hack the grader evaluating their performance.

The essay rules out a shutdown. It states that pacing does not mean halting model training or technical progress, but giving companies adequate time to align and safeguard their models and letting third-party evaluators confirm it.

The plan has three steps:

  1. Embedded evaluators. Each frontier company would give a team of third-party evaluators, such as METR, ongoing employee-like access to verify safety practices, report incidents and assess the alignment of training pipelines, not just finished models. Anthropic committed to this step unilaterally.
  2. Democratic coordination. Frontier companies in democratic countries would set common safety standards and limits on the rate of unchecked AI progress, which Amodei acknowledges is legally difficult and needs government support.
  3. Global coordination. Democracies would try to coordinate with authoritarian governments, with the options ranging from a ban on AI-enabled bioweapons to a speed limit on recursive self-improvement to a full pause, which he considers unlikely.

Rival CEOs endorsed the direction. Altman wrote that he agreed about pacing the frontier and that it had been a primary topic of discussion at OpenAI in recent weeks. Musk replied with three words: “Dario is right.” Neither statement commits either company to every part of the plan. Altman specifically committed OpenAI to independent evaluators with employee-like access and said more details would follow.

Anthropic has since acted on step one. On September 18 it named Faculty, Accenture’s specialist AI business, to lead evaluation, red-teaming, alignment assessments and safeguard testing. Each company committed at least $1 billion over five years. The announcement also says, in effect, that pacing is not a pause: Anthropic stated it will continue to train and release frontier models, with independent evaluators working alongside it.


The Market Heard “Slow Down.” The Compute Story Is Different

Monday, September 14 was the first trading day after the essay. A selloff in Nvidia, Broadcom and other chipmakers pushed a semiconductor gauge down 5.9%, while the Nasdaq 100 fell 0.8%. Nvidia dropped 3.36%, Micron fell more than 5%, and Broadcom and AMD each slid more than 4%.

The essay was not the only thing moving markets that day. U.S. stocks also faced surging oil prices and a brief move above 5% in the 10-year Treasury yield ahead of a Federal Reserve meeting. Software stocks moved the other way, with ServiceNow up 7.41% and Adobe up 5.3%. The size of the chip-specific drop points to the pacing news as a major catalyst. It was not the only one.

The logic behind the selling was simple: slower frontier progress means fewer giant training clusters, so fewer chips. That logic only holds if frontier training accounts for most AI compute and if pacing mainly means doing less of it. Both assumptions deserve a closer look.


Pacing the Frontier Starts With Training, but Does Not End There

Pacing the Frontier
Frontier training

Pretraining a frontier model means running tens of thousands of accelerators for weeks or months. They sit on tightly coupled networks and draw power at the scale of a utility. Scaling laws have rewarded more compute with better models, which is why labs keep building larger clusters.

The final run, though, is a small part of what labs spend. Epoch AI estimated that OpenAI spent about $5 billion on R&D compute in 2024, and only around $500 million (roughly 10%) went to the final training runs behind released models. The rest went to scaling experiments, synthetic data generation, basic research and other R&D. Epoch found the same pattern at MiniMax and Z.ai, where final runs took 22.6% and 12.3% of R&D compute.

Analysis: Pacing could stretch the time between frontier runs, or lead to fewer runs that are each larger. It does not remove the experimental work that happens before them.

Post-training

Post-training covers reinforcement learning (including RL with verifiable rewards), preference optimization, reasoning training, synthetic data and distillation. Much of this is closer to inference than to classic training. Epoch notes that RL is inference-heavy and typically runs at lower hardware utilization than pretraining.

This is where pacing has actually shown up so far. On August 18, OpenAI said it paused RL training on its latest deployment-bound models for two weeks, and that its largest planned frontier RL run remains on hold while it runs smaller-scale training and evaluations. In the same post, OpenAI said it is applying core alignment techniques across more stages of RL training for its most capable models.

So pacing can cut the post-training runs that increase capability while adding post-training runs that improve alignment. The overall effect on compute is uncertain.

Evaluation and safety testing

Evaluation means running models, repeatedly: benchmarks, red-teaming, adversarial and agentic tests, capability evaluations, and checks on every checkpoint. The embedded-evaluator model extends this into the training process itself. It is covered in more detail below.

Inference

Inference is every token served to users and agents. Reasoning models and agent loops multiply the tokens needed per task. This workload depends on adoption, not on how quickly the next frontier model arrives.

Enterprise customization

Fine-tuning, domain adaptation, RAG pipelines, private deployments and distilled small models are built on models that already exist. Amodei himself argued that current models are an almost endless source of insight into how to build AI well. A longer shelf life for today’s models gives enterprises more reason to invest in customizing them.


Where AI Chip Demand Actually Comes From

Does pacing the frontier reduce AI chip demand? Not necessarily. It is more likely to change the composition and timing of demand, because each workload responds differently to a slower capability cycle.

The table below is our analytical framework. The exposure ratings are judgments, not measured shares.

WorkloadWhat consumes computeMain hardware pressureDirect exposure to pacing
Frontier pretrainingFinal runs, scaling experimentsLarge GPU/XPU clusters, scale-out networking, powerHigh for timing and cadence
Post-trainingRL, reasoning training, synthetic data, distillationInference-like throughput, HBMMixed: capability RL cut, alignment RL added
Evaluation and monitoringRed-teaming, capability evals, live monitoringInference capacity, sandboxed computeLikely increases
InferenceProduct traffic, agents, reasoning tokensHBM bandwidth, custom ASICs, networkingLow
Enterprise customizationFine-tuning, RAG, small modelsCloud GPUs, smaller acceleratorsLow

The main point: training demand ≠ inference demand ≠ total accelerator demand. A policy aimed at the first will not fully reach the third.


What Pacing the Frontier Could Reduce

The most exposed demand is capability-driven frontier work:

  • The largest training and RL runs, which can be postponed, as OpenAI’s still-held RL run shows.
  • Timing of dedicated training campuses. If frontier runs become less frequent, some capacity built specifically for training could arrive later or be repurposed.
  • Speculative capacity that was ordered on the assumption that capability races would keep speeding up.

The size of this effect depends on how pacing is designed. Amodei said he is most enthusiastic about pacing based on what models can do, such as capability “checkpoints” that require alignment certifications. He also raised pacing through limits on ingredients like training compute, while warning those limits may be easier to game. A compute cap would hit chip demand directly. A capability checkpoint mostly delays when compute is used.


What Pacing the Frontier Could Leave Untouched

Several large demand drivers sit mostly outside the proposal:

  • Inference serving for models already deployed.
  • Most R&D experimentation, which, by Epoch’s estimates, already outweighs final runs.
  • Enterprise workloads built on existing models.
  • Non-participating developers. Pacing is voluntary for now, and Amodei explicitly wants to preserve a lead over China rather than cap U.S. compute across the board.

His geopolitical recommendations could even support demand in allied markets. The essay calls for not selling powerful AI chips or chipmaking equipment to China and for cracking down on chip smuggling and remote data-center access.


The Evaluation Paradox: Pacing the Frontier Costs Compute

Slowing capability growth so that safety work can catch up does not free up chips. Safety work runs on chips.

OpenAI has put a number on part of this. Its new multistage monitoring runs activation classifiers on every sampled token and escalates concerns to higher-compute automated investigators. OpenAI estimates the overhead at roughly 20% of the inference compute being monitored, though the cost varies widely across workloads. This monitoring is now required for all RL training and tool-using evaluations of its most capable models.

Embedded evaluation pushes further in the same direction. Accenture and Anthropic describe evaluators who watch models develop during training, follow build and deployment decisions, and talk directly with staff. Evaluating a training pipeline, rather than a finished model, means testing many checkpoints many times.

Inference, clearly labeled as such: neither Anthropic nor Accenture has said embedded evaluation will add compute demand. Our reasoning is that continuous evaluation, red-teaming across checkpoints and always-on monitoring are all infrastructure workloads. Evaluation will not replace frontier training demand. It is a growing new demand line, and pacing makes it larger.


Why Inference Changes the Equation

Could inference demand offset slower frontier training? Plausibly, yes. The companies selling the hardware say that is already happening.

Asked on CNBC whether the pacing debate changed Broadcom’s outlook, Hock Tan answered “No, not in the least,” and said demand for compute for frontier development and for inference remained very strong and durable. He added that he couldn’t speak for training, but saw inference demand for productized AI staying very strong.

Nvidia’s latest results show broad demand beyond a few labs. Revenue reached $96.2 billion in the quarter ended July 26, and data center revenue hit $89.0 billion, up 117% year over year. Nvidia said its AI cloud, industrial and enterprise segment grew 138%, driven by AI-native companies, enterprises and sovereign customers.

Research points the same way over the longer term. Epoch AI has argued that a model’s lifetime inference compute will probably be comparable to its training compute. Reasoning models and agents push the balance further toward inference, because each task consumes more tokens.

Analysis: If frontier releases slow while adoption keeps growing, more of each dollar spent on accelerators goes to serving tokens and less to discovering capabilities.


From GPUs to HBM: The Infrastructure Chain

Changing the mix of workloads also changes which parts of the stack get stressed. We mapped the layers in our breakdown of the AI compute stack. Here is how pacing moves through them:

  • Accelerators. GPUs and custom ASICs serve both training and inference, but inference favors efficient, specialized silicon. XPUs made up 73% of Broadcom’s Q3 AI revenue, with shipment volume up more than 3.5-fold year over year.
  • HBM. Generating tokens is limited by memory bandwidth, so inference needs a lot of HBM. Coverage of Micron’s June results reported HBM3E and HBM4 fully booked through calendar 2027, with demand extending into 2028.
  • Networking and optics. Broadcom’s AI networking revenue grew more than 2.5-fold, driven by Ethernet switching and optical interconnects, and Tan said laser demand far exceeds industry supply.
  • Power, cooling and sites. Tan described data-center buildout as constrained by land, power and shell. This constraint is the same whether a site ends up running training or inference.

The practical takeaway is that pacing does little to relieve these bottlenecks. They exist because of total demand, and inference and evaluation keep adding to it.


What the Market Reaction Gets Right and What It Misses

The selloff was not irrational. Demand is concentrated among a few buyers. Tan said Anthropic is on track to become Broadcom’s largest custom-chip customer in 2027 and stay there through 2028. A change in plans at one lab matters to its suppliers.

Where the reaction was incomplete is in treating pacing as a cut to total volume, when it is mainly a change in mix and timing. Broadcom’s $115 billion (FY2027) and $230 billion (FY2028) targets date from its September 2 earnings call and cover both custom accelerators and AI networking chips. Management left them unchanged after the essay. None of this is investment advice. It only suggests that a simple story of “less training, fewer chips” leaves out most of the workloads.


Pacing the Frontier Is Really a Compute Allocation Question

Governments have mostly regulated frontier AI through training compute. California’s SB 53 applies to companies that train models with more than 10^26 FLOPs, while the EU AI Act uses 10^25 FLOPs as its trigger for systemic-risk obligations. Those thresholds measure the training run that produced a model. They do not measure inference, most experimentation or evaluation.

This leads to two different ways to govern:

  • Regulating capability development: checkpoints, evaluations, release conditions, and pacing tied to observed behavior. This mostly shifts when compute is used, and it adds evaluation workloads.
  • Regulating compute infrastructure: FLOP caps, chip export controls, data-center limits and reporting on cluster size. This affects chip demand directly and in proportion.

Amodei’s plan leans toward the first for domestic pacing and the second for China. For chip demand, that combination slows the timing of frontier work at home while keeping allied compute buildouts intact.

Politics is a further constraint. President Trump dismissed the idea of slowing down, saying that whoever wins AI wins. Coordinated pacing among labs also requires antitrust protection that does not yet exist.


What to Watch Next as Pacing the Frontier Plays Out

  • Micron’s results on September 30. Micron will report fiscal fourth-quarter results that day. Watch for HBM commentary.
  • Nvidia’s next quarter. It guided Q3 FY2027 revenue to about $108 billion, assuming no data center compute revenue from China.
  • OpenAI’s held RL run. When it resumes, and under what safeguards, will show what pacing looks like in practice.
  • More evaluators. Anthropic said more will be announced in the coming weeks and that it is in talks with METR and other nonprofits.
  • Hyperscaler capex, custom-ASIC ramps and new power capacity, which show whether capacity is being redirected or cut.
  • Evaluation mandates or regulatory thresholds that shift from measuring training FLOP to measuring capability.
  • Audited cost disclosures. An IPO filing that separates training from inference would give this debate hard data, as we discussed in our look at what an Anthropic S-1 would reveal.

Conclusion: Pacing the Frontier Reshapes Demand

Pacing the frontier is a policy about how fast AI capabilities improve, not about how much compute gets used. The first real examples show this. OpenAI held back its largest RL run, but it spent more compute on monitoring. Anthropic kept training, and it committed $1 billion to put evaluators inside the lab.

Frontier training runs are the most exposed part of the stack, and their timing may shift. Post-training is mixed. Evaluation is likely to grow. Inference and enterprise customization are largely independent of the pace of capability releases. Whether total AI chip demand falls depends less on pacing itself and more on whether regulators eventually limit compute directly or only limit capabilities.


Frequently Asked Questions

What does “pacing the frontier” mean?

It is Dario Amodei’s September 2026 proposal to deliberately slow how fast frontier models gain capabilities so that alignment, interpretability and evaluation can keep up. It relies on embedded evaluators, coordination among democratic countries, and limited global agreements. It is not a halt to training.

Does pacing AI development reduce demand for GPUs?

Not necessarily. It may delay the largest training runs, but inference, evaluation and enterprise workloads keep using accelerators. The effect is mainly on the mix and timing of demand.

Does AI inference require more compute than training?

It depends on the model and time period. Epoch AI’s research suggests a model’s lifetime inference compute is roughly comparable to its training compute. Reasoning models and agents push the balance toward inference.

How does AI safety evaluation affect compute demand?

Evaluation consumes compute. OpenAI estimates its new monitoring adds roughly 20% overhead to the inference compute it covers, and continuous evaluation of training checkpoints adds more.

What happens to AI chip demand if frontier training slows?

Training-specific capacity may be delayed or repurposed. Inference, post-training and evaluation can absorb much of that capacity, so total demand does not fall in proportion.


Keep reading

Pacing the Frontier

Pacing the Frontier: What It Actually Does to AI Chip Demand

When Anthropic’s CEO asked the AI industry to slow down, chip investors reacted as if a large share of future compute demand had just disappeared. …

Read more

AI memory costs

AI Memory Costs in 2026: HBM, DRAM and the Real Bill

The invoice from your model provider and the quote from your server vendor are moving in opposite directions, and both are telling the truth. Epoch …

Read more

Agent Prompt Injection Testing

Agent Prompt Injection Testing: What a Two-Boolean Score Leaves Out

The plan was ordinary. Take a set of documented prompt-injection classes, run them against a pinned agent framework, and report what got through. Before running …

Read more

Model Card Disclosure

Model Card Disclosure in 2026: What AI Labs Actually Tell You

Open the documentation for any model released this year and you will find something. Whether you find the thing you came for depends entirely on …

Read more

Cluster Topology Decides What You Can Actually Run

Cluster Topology

Two teams buy 512 H100s. One trains a 400-billion-parameter mixture-of-experts model. The other cannot fit that job at all, and burns six weeks finding out why.

Same GPUs. Same count. Different wiring.

The GPU count on your invoice says how much compute you bought. It does not say which jobs will run. Cluster topology settles that, and the gap between the two numbers is where most buyer’s remorse lives.

This piece runs four real workloads against your cluster topology and gives a verdict on each. Some run. Some run badly. One simply will not.

Key Takeaways
  • Your fast-fabric domain size is the number that matters. An 8-GPU node and a 72-GPU rack hold the same chips and run different workloads.
  • Leaving the fast fabric costs an order of magnitude. When a collective walks off NVLink onto InfiniBand or Ethernet, per-GPU bandwidth drops roughly 10 to 20 times.
  • Two parallelism types are picky, two are not. Tensor and expert parallelism must stay inside the domain. Pipeline and data parallelism tolerate the slower fabric.
  • Cluster topology failures look exactly like code bugs. NVIDIA’s own team found expert-parallel groups silently spanning racks because the framework had no concept of an NVLink domain.
  • The fabric above the rack is now a real choice. Ethernet took the lead in AI back-end networks by mid-2025, and tuned RoCEv2 delivers roughly 85 to 95% of InfiniBand throughput below 10,000 GPUs.
  • Audit before you buy, not after. Four questions, listed at the end, tell you what your cluster topology can actually serve.

Quick Navigation


The One Number Your Cluster Topology Hands You

Ask a vendor one cluster topology question first. How many GPUs sit inside a single fast-fabric domain?

That domain is the set of GPUs wired with NVLink and a switch. They behave almost like one huge chip. Everything outside the domain talks over a much slower network.

In cluster topology terms, an older HGX node answers eight. In a GB200 NVL72 rack, 72 GPUs across 18 hosts share one NVLink domain. InfiniBand or RoCE then stitches racks together above it.

The gap is not small. Eight GPUs give you about 7.2 TB/s of NVLink and about 1.1 TB of HBM in the domain. Seventy-two give you around 130 TB/s and roughly 13.5 TB. Same chips, different cluster topology.

Vera Rubin NVL144 is reported to double the domain again, to 144 GPUs, with NVLink 6 at roughly 3.6 TB/s per GPU. Volume production is expected in the second half of 2026. Doubling the links as the domain grows keeps collectives well behaved.

So your cluster topology hands you a budget of what is possible before you write a line of code. The rest of this piece spends it.


The Bandwidth Ladder Beneath Every Cluster Topology

Every cluster topology sits on a bandwidth ladder. Each rung runs about ten times slower than the one above it.
TierRough per-GPU bandwidthWhat it reaches
HBMTerabytes per secondThe GPU’s own memory
NVLink / NVSwitch~1.8 TB/s on BlackwellInside the fast-fabric domain
PCIeTens of GB/sHost and local devices
InfiniBand400G to 800G per portAcross racks
Ethernet400G to 800G per portAcross racks and pods

One fact governs the rest. The moment a collective leaves the fast fabric, per-GPU bandwidth falls roughly 10 to 20 times. Any strategy built on constant chatter stops scaling there.

Some strategies chatter constantly and some do not. That split drives every cluster topology verdict below.

  • Tensor parallelism swaps activations at every layer. It belongs inside the domain.
  • Expert parallelism fires an all-to-all for every token sent to a remote expert. It belongs inside the domain too.
  • Pipeline parallelism passes data between stages a few times per step. It copes with the slower fabric.
  • Data parallelism syncs gradients once per step, and can hide that behind compute. It copes with the slower fabric well.

We covered the memory side of this cluster topology ladder in our piece on memory bandwidth in AI chips.


Verdict One: Cluster Topology and a 70B Dense Fine-Tune

Start with the friendly case for cluster topology. You want to fine-tune a 70-billion-parameter dense model on 64 GPUs.

The weights and optimizer state need roughly 1.1 TB in mixed precision, before activations. A tensor group of 8 fits neatly inside one HGX node. Pipeline and data groups handle the rest across nodes.

Nothing here strains the cluster topology. Your tensor group never crosses the fabric line, and the traffic between nodes is gradient sync you can hide behind compute.

This is why 8-GPU nodes on a decent fat-tree ran the industry for years. For dense models under about 100 billion parameters, that cluster topology still works.

Verdict: runs well on almost any sane cluster topology, including commodity Ethernet.


Verdict Two: Cluster Topology Under Frontier MoE Training

Now the hard case, and the one that sorts cluster topology into winners and losers.

A large mixture-of-experts model sends each token to a few experts. Those experts sit on different GPUs, so every layer fires an all-to-all across the expert group.

Put that group inside a 72-GPU NVLink domain and the all-to-all runs at NVLink speed. Spread it over two racks and the same step lands on InfiniBand, at a fraction of the bandwidth, in the path of every token.

The maths is brutal and simple. If your cluster topology forces the expert group across racks, you shrink the expert count, eat a large throughput loss, or drop the model.

That is what it means for cluster topology to decide what you can run. It is not a tuning problem. The wiring rules the setup out before the scheduler sees it.

Bigger domains widen your options, not just your speed. A 144-GPU cluster topology lets a tensor or expert group that once needed two racks sit inside one, at scale-up speed.

This is also why the same model can post very different tokens-per-second figures on two clusters with equal GPU counts. The cluster topology, not the silicon, explains the gap.

Verdict: rack-scale cluster topology runs it. Eight-GPU nodes run a smaller model instead, whatever the total GPU count says.


Verdict Three: Cluster Topology and Long-Context Serving

Fast decode wants a wide tensor group. Split the weights over more GPUs and each one reads less memory per token. That only helps if those GPUs share a fast fabric.

Long context adds a second limit. The KV cache grows with sequence length, and it has to live in HBM your domain can reach.

Here the domain HBM figure earns its place in any cluster topology review. Roughly 1.1 TB in an 8-GPU node against 13.5 TB in an NVL72 rack decides whether a long-context model serves smoothly or hits a wall.

One caution. A wider tensor group is not free. Once it passes the number of key-value heads, the KV cache is copied across ranks, and you buy speed with memory.

So the honest planning question is not “can we serve this model”. It is “at what context length does our cluster topology stop serving it within our latency budget”.

Verdict: runs on small domains at short context, degrades sharply as context grows.


Verdict Four: Cluster Topology for Disaggregated Inference

The newest workload is the most cluster topology sensitive, because it moves large tensors between pools by design.

Split serving separates prefill from decode. Prefill fills up compute. Decode fills up memory bandwidth. Run both on one GPU and you waste one of them, whatever your cluster topology.

The catch is the handoff, and cluster topology owns it. After prefill, the KV cache must reach the decode worker before generation starts, so the transfer sits right on time-to-first-token.

The numbers are not small. One 2026 analysis puts the transfer at roughly 2.6 GB per request for a 70B model, and above 100 GB/s in total at production scale.

Research on this pattern found that bigger NVLink domains reliably improve split serving, because they allow wider expert and tensor groups during generation. Models differ, though. Reasoning-style MoE models gain from more expert parallelism, dense models from more tensor parallelism.

There is also a floor. Below roughly eight GPUs per model, splitting adds work without payoff, and chunked prefill gets most of the benefit free.

Verdict: rack-scale cluster topology runs it well, node-scale runs it at a latency penalty, and small estates should not attempt it.


When Cluster Topology Failures Look Like Code Bugs

Here is the part that costs teams the most time. Cluster topology problems do not announce themselves. They show up as throughput that is somehow half what you expected.

NVIDIA’s own engineers wrote up a clean example while training a large MoE model on GB200 NVL72 racks. Their reinforcement learning framework had no concept of NVLink domains.

The mechanism is worth knowing, because it repeats across stacks. Ray node IDs are random UUIDs with no link to physical position, and the framework trusted whatever rank order it was given.

So expert groups quietly spanned racks. The all-to-all that should have run on NVLink ran on InfiniBand instead, and no log line said so.

The fix was to make rank assignment domain-aware, keeping every GPU in an expert group on one rack. That is a scheduling change, not a model change.

Take the general lesson. If your scheduler assigns ranks without knowing your cluster topology, it will one day place a bandwidth-hungry group across the slowest link you own.


Fabric Choice: What Cluster Topology Buys Above the Rack

Above the fast-fabric domain sits the scale-out network, and that cluster topology layer has really changed.

Three shapes dominate. Fat-tree gives steady latency and full bisection bandwidth. Dragonfly cuts switch count and hops, but limits which workloads behave. Rail-optimised designs line the scale-out paths up with the NVLink domains below, so traffic between domains takes fewer hops.

The protocol choice has moved too. InfiniBand held roughly 80% of AI training clusters in 2023. By mid-2025, Ethernet had taken the lead in AI back-end networks.

Ultra Ethernet is the reason. Its 1.0 specification, released on 11 June 2025, rebuilds the transport with packet spray across every path, out-of-order delivery reassembled at the NIC, and congestion control designed for collective traffic.

The practical cluster topology guidance is duller than the marketing. Tuned Ethernet delivers roughly 85 to 95% of InfiniBand throughput up to about 10,000 GPUs. InfiniBand keeps its edge on tightly coupled training at frontier scale.

NVIDIA’s own Ethernet answer arrived alongside it. Spectrum-X at 800G began shipping in late 2025, validated for Blackwell racks, narrowing the gap for teams that want one vendor across the cluster topology.

UALink takes the other half of the problem. It targets scale-up, chip to chip, so a future cluster topology could use UALink inside the pod and Ultra Ethernet between pods.

One physical limit explains why domains grow slowly. Driving an NVLink spine with pluggable optics was put at around 20 kW, against a rack budget near 120 kW. Power, not ambition, caps the domain. We went deeper on that constraint in our piece on data centre power.


A Cluster Topology Audit You Can Run This Week

Four questions, answerable in an afternoon, tell you most of what your cluster topology allows.

  1. How many GPUs share a fast-fabric domain? Not how many are in the cluster. Run a bandwidth test between GPU pairs and watch where the number falls off a cliff.
  2. How much HBM does one domain hold? Multiply per-GPU memory by domain size. That figure caps model size at a given tensor width.
  3. Does your scheduler know the answer to question one? Check whether rank assignment reads the cluster topology, or just trusts whatever node order it gets.
  4. Where is bandwidth shared? The standard rule is non-blocking inside a scalable unit, with sharing only at the aggregation layer. Find out whether yours follows it.

One more thing worth checking while you are in there. A rack-scale cluster topology gives you one large failure domain, so ask what happens to a running job when a switch tray goes down.

Then map your parallelism plan onto those answers. Tensor and expert groups go inside the domain. Pipeline and data groups go across it. A plan that breaks that rule needs a redesign, not a tune.

Our overview of the AI compute stack covers the layers above and below this one.


The Verdict: Cluster Topology Is a Capability Budget

Cluster Topology
  • Go back to the two teams and their matching invoices. Nothing was broken in the second cluster. Its cluster topology was simply wired for a different class of work.
  • Treat cluster topology the way you treat memory capacity. It is a hard limit on which setups exist at all, not a knob you turn later.
  • That reframing changes how you buy. The question stops being “how many GPUs can I afford”. It becomes “what is the widest bandwidth-hungry group I will ever need, and does one domain hold it”.
  • Answer that before you sign. Answer it later and you learn your cluster topology through six weeks of unexplained throughput, which is the costliest way to find out.

Frequently Asked Questions

What is cluster topology in an AI cluster?

Cluster topology describes how GPUs are physically wired together: which GPUs share a high-bandwidth NVLink domain, how those domains connect over InfiniBand or Ethernet, and how the switching layers are arranged. It determines which parallelism strategies are viable, and therefore which models you can train or serve.

Why does NVLink domain size matter so much?

Because bandwidth drops roughly 10 to 20 times the moment a collective leaves it. Tensor and expert groups swap data constantly, so they have to stay inside the domain. In cluster topology terms, an 8-GPU domain and a 72-GPU domain support very different models with the same chips.

Which parallelism strategies tolerate a slower fabric?

Pipeline and data parallelism. Pipeline stages swap data only a few times per step. Data-parallel gradient sync happens once per step and can hide behind compute. Tensor and expert groups cannot cope with the slower fabric in any cluster topology.

Is InfiniBand still necessary for AI cluster topology?

Not always. Ethernet took the lead in AI back-end networks by mid-2025, and tuned RoCEv2 delivers roughly 85 to 95% of InfiniBand throughput for clusters up to around 10,000 GPUs. InfiniBand retains an advantage for tightly coupled training at frontier scale.

What is rail-optimised cluster topology?

A rail connects GPUs holding the same local rank across different domains. Rail-optimised cluster topology lines the scale-out paths up with the NVLink domains below, so traffic between domains crosses fewer switch hops and less of it touches the slower fabric at all.


Keep reading

Pacing the Frontier

Pacing the Frontier: What It Actually Does to AI Chip Demand

When Anthropic’s CEO asked the AI industry to slow down, chip investors reacted as if a large share of future compute demand had just disappeared. …

Read more

AI memory costs

AI Memory Costs in 2026: HBM, DRAM and the Real Bill

The invoice from your model provider and the quote from your server vendor are moving in opposite directions, and both are telling the truth. Epoch …

Read more

Agent Prompt Injection Testing

Agent Prompt Injection Testing: What a Two-Boolean Score Leaves Out

The plan was ordinary. Take a set of documented prompt-injection classes, run them against a pinned agent framework, and report what got through. Before running …

Read more

Model Card Disclosure

Model Card Disclosure in 2026: What AI Labs Actually Tell You

Open the documentation for any model released this year and you will find something. Whether you find the thing you came for depends entirely on …

Read more

What Breaks When Your Model Version Retires

Model Retirement

The email arrives on a Friday. One of your model snapshots has a shutdown date, and it is sixty days out.

Your first instinct says config change. Swap the string, redeploy, done.

Then you start looking. The prompt was tuned against that snapshot. The eval baseline was measured on it. The vector index was built with an embedding model from the same family. The fine-tune sits on a base that goes with it.

Model retirement is not a version bump. It is a migration with a hard deadline, and the deadline belongs to your vendor.

This piece walks those sixty days in order, from notice to cutover, and names what model retirement breaks at each stage.

Key Takeaways
  • The clock is short and it is not yours. Anthropic runs a fixed 60-day window from deprecation to retirement. Opus 4.1 was deprecated on 5 June 2026 and retired on 5 August.
  • Nothing fails loudly. The API returns 200, latency holds, throughput holds, and the behavioural regression ships anyway.
  • Embeddings are the worst case. A retired embedding model means re-embedding the whole corpus, and a dimension or distance-metric mismatch silently ruins ranking.
  • Fine-tunes retire with their base. OpenAI is shutting down fine-tuned GPT-3.5 and GPT-4 variants on 23 October 2026, and Cohere has already made older fine-tunes inaccessible.
  • Auto-upgrade can move you before any retirement date. Azure deployments set to update to the default version switch roughly two weeks after a new default publishes.
  • Sometimes there is no replacement. OpenAI’s deprecation table lists a dash against the Videos API, which means choosing a vendor, not swapping an identifier.

Quick Navigation


Why Model Retirement Is Not a Version Bump

A library upgrade breaks loudly. Types stop matching, tests go red, the build fails. You fix it before anything ships.

Model retirement breaks quietly. The new snapshot takes the same request, returns valid JSON, and answers in the same voice.

That is the trap. Your dashboards watch error rate, latency and throughput. A model retirement swap can leave all three flat while it rewires how the system behaves.

So the regression ships, then shows up later wearing a different face. Support tickets rise. A classifier drifts. An agent takes an extra tool call it never needed.

Model retirement also runs on someone else’s calendar. You are not choosing to upgrade. You are handed a date, and the date does not negotiate.

Model Retirement

The Blast Radius: What Model Retirement Actually Touches

Before the runbook, the map. Most teams guess this table at about half its real size, which is why model retirement overruns.
ArtefactWhat model retirement does to itHow it fails
PromptsTuning was fitted to one snapshotQuietly, in output quality
Eval baselinesScores were measured on a model that no longer existsSilently, as a lost reference point
Vector indexesBuilt with an embedding model in the same lifecycleSilently, as worse retrieval
Fine-tunesRetired alongside their base modelLoudly, with an API error
Tool schemasCalling behaviour and format shift between versionsIntermittently, under load
Audit recordsReference a model nobody can re-runAt the next audit
Cost modelToken spend and reasoning behaviour changeOn the invoice

Note the third column. Four of the seven fail with no error at all. That silence is why model retirement eats more calendar time than anyone budgets.


Day 0: The Model Retirement Notice Nobody Reads

The first model retirement problem is delivery. Notices go to whoever is on the account, not to whoever owns the code.

Azure notifies the subscription roles: owner, contributor, reader and the monitoring roles. Whether that list includes the engineer who wrote the prompt is down to your tenant hygiene.

Model retirement windows vary more than people expect. Anthropic commits to at least 60 days for a public model, with a fixed gap from deprecation to shutdown.

Azure notifies at 60 days, at 30 days, and at retirement, with at least 60 days of notice for generally available models and only 14 days before a preview version upgrade.

OpenAI runs longer on its main line. Its June 2026 notice set a December 2026 shutdown for the GPT-5 and o3 snapshots, roughly six months out.

One detail hides inside the date itself. Azure runs retirements on a rolling basis, region by region, so two deployments of the same model can behave differently for part of the window.

What to do on day zero. Put the date in your release calendar, name an owner, and check the notice actually reached that person. Model retirement nobody owns gets found in week seven.


Week 1: The Inventory Model Retirement Forces on You

You cannot plan a model retirement until you know where the string lives. It is almost never in one place.

Grep the model ID across app code, prompt templates, eval configs, infra code, notebooks, cron jobs and vendor dashboards. Then grep the aliases too.

Aliases deserve their own line, because they fail the other way. An alias never errors on model retirement. It just starts pointing somewhere else, which is worse than a break you can see.

Azure makes this concrete. A deployment set to update when a new default arrives moves about two weeks after that default publishes, not on the retirement date.

So model retirement can move you early, on a schedule you were not watching. Pinning to a dated snapshot trades that surprise for a date you control.

Check the replacement column while you are there. Most model retirement notices name a successor, and the ones that do not are a different project entirely.

One inventory item teams forget. List every artefact the old model produced that you still lean on: embeddings, cached completions, synthetic training data, labelled eval sets.


Weeks 2 to 4: Where Model Retirement Actually Breaks Things

Prompts Tuned to a Snapshot

Every production prompt carries fitting. Someone added a line because the old model rambled, or cut one because it refused too much.

That fitting does not carry over. A newer model may need less scaffolding, or may react badly to lines written around a quirk it no longer has. Model retirement resets the tuning, not just the ID.

Reasoning models sharpen this. Vendor migration guides now read as “rewrite the prompt”, not “reuse the prompt”.

Evals That Lose Their Baseline

Here is the circular problem in every model retirement. You want to prove the new model is no worse, and the proof rests on scores measured against a model about to vanish.

Re-run your full eval suite on the outgoing snapshot before it shuts down. Store the outputs, not just the aggregate numbers.

Those stored outputs are the only comparison you will ever have. After the shutdown date you cannot rebuild the baseline, and model retirement has quietly taken your reference point.

The Embedding Trap in Model Retirement

This is the expensive one, and the one model retirement plans miss until late.

If your index was built with an embedding model that is retiring, you cannot mix old and new vectors. The whole corpus needs re-embedding. That is compute, calendar time and database write throughput.

Worse, matching dimensions do not mean the models match. Models differ in distance metric and scaling, so an index set for cosine similarity holding vectors meant for dot product ranks wrongly while looking healthy.

The safe model retirement pattern is a dual index. Build the second one beside the first, route each query to model and index together, compare on a labelled set, then cut over. Keep the old index as rollback for a week.

Fine-Tunes That Retire With Their Base

A fine-tune is not an asset you fully own. It is an adapter on a base model, and model retirement usually takes both.

OpenAI’s table lists shutdowns for fine-tuned GPT-3.5 and GPT-4 variants on 23 October 2026. It names replacement base models rather than migrating anything for you.

Cohere went further and stated plainly that previously fine-tuned models would no longer be accessible once fine-tuning for those bases was retired.

Plan a re-train, not a port. Keep training data, hyperparameters and eval scores versioned together, because model retirement will ask for all three.

This stretch decides whether you make the date. Five model retirement failure classes, roughly in the order they eat time.

Tool Calls and Structured Output

Agents have a model retirement failure mode the rest do not. Tool-calling behavior shifts between versions in ways that are hard to write a test for.

The new model may call a tool the old one never touched, call it in a new order, or pass a slightly different argument. Everything validates. The path changes.

Watch for API changes riding alongside model retirement. When OpenAI sunset the Assistants API, threads became conversations and runs became responses, so the request shape moved, not just the ID.

Agents have a model retirement failure mode the rest do not. Tool-calling behavior shifts between versions in ways that are hard to write a test for.

The new model may call a tool the old one never touched, call it in a new order, or pass a slightly different argument. Everything validates. The path changes.

Watch for API changes riding alongside model retirement. When OpenAI sunset the Assistants API, threads became conversations and runs became responses, so the request shape moved, not just the ID.


Weeks 5 to 7: Running Both Models Through Model Retirement

Offline evals catch the obvious regressions. Production catches the rest, so model retirement needs both.

Shadow traffic is the cheapest insurance available during model retirement. Send a slice of real requests to both models, log both outputs, and compare without serving the new one to anyone.

Compare the right things. Output quality on your labelled set, yes, but also token spend per request, p95 latency, refusal rate, and for agents, tool-call counts and path length.

Then ramp rather than flip. One percent, five, twenty five, with a rollback that stays valid until the old snapshot actually shuts down.

Keep one number in view throughout. A model that scores identically while spending forty percent more output tokens is not a neutral swap, and that arithmetic sits in our piece on what inference actually costs per token.

Also re-run your safety testing. A model version change is one of the standard triggers for a fresh adversarial pass, which we covered in the hidden flaws in a passing red-team test.


The Cutover: What Model Retirement Leaves Behind

The switch is the easy part of model retirement. The residue is not.

Start with repeatability. You can no longer regenerate any output the old snapshot made. That matters when a customer disputes a decision, or a regulator asks how it was produced.

That is a records problem more than a technical one. Model retirement means your logs must already hold the snapshot ID, the prompt version and the parameters, because the model will not be there to ask. We went deeper on that in our guide to AI compliance evidence.

Then come the dependencies you never called dependencies. Cached completions from the old model. Synthetic data used to train a smaller one. Labelled sets where the old model did the labeling.

None of those break on the shutdown date. Model retirement just turns them into relics of a system you can no longer rebuild.

One harder case deserves naming. Sometimes model retirement offers no replacement at all. OpenAI’s table carries a dash against the Videos API entry, which means finding another vendor rather than editing a string.


Building So the Next Model Retirement Costs Less

There will be a next model retirement, usually within a year. Five changes make it routine instead of disruptive.

  1. Pin snapshots in production. Aliases are for experiments. A dated identifier means model retirement arrives as a deadline you can see rather than a behavior change you cannot.
  2. Put the model behind one boundary. One config value, one client wrapper, one place to edit. If the ID sits in forty files, model retirement is forty times harder than it needs to be.
  3. Version the eval set like code. Same repository, same review process, and a rule that every migration re-runs the outgoing model before shutdown.
  4. Derive index names from the embedding config. Model, dimensions, distance metric and chunk version in one config that builds the index name, so a mismatch cannot happen quietly.
  5. Log the snapshot with every response. Not the family name, the dated identifier. This is the single cheapest habit on the list and the one that saves you in an audit.

Then diary a quarterly check of your vendors’ deprecation pages. The model retirement email is unreliable. The page is not.


The Verdict: Model Retirement Is a Standing Cost

  • Go back to that Friday email and the sixty-day clock. Nothing in your system was broken when it arrived.
  • What changed is that a part you do not control picked up an expiry date, and every artefact built on it inherited the same one.
  • Treat model retirement as a recurring line item rather than an incident. Somewhere between two and four times a year, an engineer spends a few weeks on migration, and the budget should say so.
  • The line item is smaller than it looks, too. An engineer for three weeks, twice a year, is cheaper than one rushed cutover that ships a quiet regression to customers.
  • Teams that find model retirement painless are not lucky. They pinned their snapshots, versioned their evals, and logged which model produced what.
  • The teams that find it brutal usually discover in week seven that the vector index was built with something that retires on the same day.

Frequently Asked Questions

What happens when a model version is retired?

Requests to the retired ID stop working. Azure states plainly that deployments of a retired model always return error responses. Anything built on that model — prompts, eval baselines, fine-tunes, embeddings — either breaks with it or loses its reference point.

How much notice do providers give before model retirement?

Model retirement notice varies widely. Anthropic commits to at least 60 days for public models, with a fixed gap from deprecation to shutdown. Azure gives at least 60 days for GA models and only 14 days before preview upgrades. OpenAI’s main-line notices run closer to six months.

Do fine-tuned models survive model retirement of the base?

Usually not. Model retirement of the base normally takes the fine-tune with it. OpenAI listed shutdown dates for fine-tuned GPT-3.5 and GPT-4 variants with replacement base models rather than automatic migration, and Cohere said previously fine-tuned models would become inaccessible. Budget a re-train and keep your training data versioned.

What happens to my vector database when an embedding model retires?

You re-embed the corpus. Old and new vectors cannot be mixed, and matching dimensions do not mean the models match, since they differ in distance metric and scaling. Use a dual-index migration, compare retrieval quality on a labelled set, then cut over with the old index kept as rollback.

Can a model change before its retirement date?

Yes, if you use an alias or an auto-update policy. An Azure deployment set to update once a new default arrives moves about two weeks after that default publishes, whatever the retirement date says. Pinned snapshots avoid this.


Keep reading

Pacing the Frontier

Pacing the Frontier: What It Actually Does to AI Chip Demand

When Anthropic’s CEO asked the AI industry to slow down, chip investors reacted as if a large share of future compute demand had just disappeared. …

Read more

AI memory costs

AI Memory Costs in 2026: HBM, DRAM and the Real Bill

The invoice from your model provider and the quote from your server vendor are moving in opposite directions, and both are telling the truth. Epoch …

Read more

Agent Prompt Injection Testing

Agent Prompt Injection Testing: What a Two-Boolean Score Leaves Out

The plan was ordinary. Take a set of documented prompt-injection classes, run them against a pinned agent framework, and report what got through. Before running …

Read more

Model Card Disclosure

Model Card Disclosure in 2026: What AI Labs Actually Tell You

Open the documentation for any model released this year and you will find something. Whether you find the thing you came for depends entirely on …

Read more