AI Infrastructure & Chips

Cluster Topology Decides What You Can Actually Run

Two teams buy 512 H100s. One trains a 400-billion-parameter mixture-of-experts model. The other cannot fit that job at all, and burns six weeks finding out why.

Same GPUs. Same count. Different wiring.

The GPU count on your invoice says how much compute you bought. It does not say which jobs will run. Cluster topology settles that, and the gap between the two numbers is where most buyer’s remorse lives.

This piece runs four real workloads against your cluster topology and gives a verdict on each. Some run. Some run badly. One simply will not.

Key Takeaways

Your fast-fabric domain size is the number that matters. An 8-GPU node and a 72-GPU rack hold the same chips and run different workloads.
Leaving the fast fabric costs an order of magnitude. When a collective walks off NVLink onto InfiniBand or Ethernet, per-GPU bandwidth drops roughly 10 to 20 times.
Two parallelism types are picky, two are not. Tensor and expert parallelism must stay inside the domain. Pipeline and data parallelism tolerate the slower fabric.
Cluster topology failures look exactly like code bugs. NVIDIA’s own team found expert-parallel groups silently spanning racks because the framework had no concept of an NVLink domain.
The fabric above the rack is now a real choice. Ethernet took the lead in AI back-end networks by mid-2025, and tuned RoCEv2 delivers roughly 85 to 95% of InfiniBand throughput below 10,000 GPUs.
Audit before you buy, not after. Four questions, listed at the end, tell you what your cluster topology can actually serve.

Quick Navigation

The One Number Your Cluster Topology Hands You
The Bandwidth Ladder Beneath Every Cluster Topology
- Every cluster topology sits on a bandwidth ladder. Each rung runs about ten times slower than the one above it.
Verdict One: Cluster Topology and a 70B Dense Fine-Tune
- Start with the friendly case for cluster topology. You want to fine-tune a 70-billion-parameter dense model on 64 GPUs.
Verdict Two: Cluster Topology Under Frontier MoE Training
- Now the hard case, and the one that sorts cluster topology into winners and losers.
Verdict Three: Cluster Topology and Long-Context Serving
- Serving changes the cluster topology question. Now you care about latency per token, not throughput per step.
Verdict Four: Cluster Topology for Disaggregated Inference
- The newest workload is the most cluster topology sensitive, because it moves large tensors between pools by design.
When Cluster Topology Failures Look Like Code Bugs
Fabric Choice: What Cluster Topology Buys Above the Rack
A Cluster Topology Audit You Can Run This Week
The Verdict: Cluster Topology Is a Capability Budget
Frequently Asked Questions

The One Number Your Cluster Topology Hands You

Ask a vendor one cluster topology question first. How many GPUs sit inside a single fast-fabric domain?

That domain is the set of GPUs wired with NVLink and a switch. They behave almost like one huge chip. Everything outside the domain talks over a much slower network.

In cluster topology terms, an older HGX node answers eight. In a GB200 NVL72 rack, 72 GPUs across 18 hosts share one NVLink domain. InfiniBand or RoCE then stitches racks together above it.

The gap is not small. Eight GPUs give you about 7.2 TB/s of NVLink and about 1.1 TB of HBM in the domain. Seventy-two give you around 130 TB/s and roughly 13.5 TB. Same chips, different cluster topology.

Vera Rubin NVL144 is reported to double the domain again, to 144 GPUs, with NVLink 6 at roughly 3.6 TB/s per GPU. Volume production is expected in the second half of 2026. Doubling the links as the domain grows keeps collectives well behaved.

So your cluster topology hands you a budget of what is possible before you write a line of code. The rest of this piece spends it.

The Bandwidth Ladder Beneath Every Cluster Topology

Every cluster topology sits on a bandwidth ladder. Each rung runs about ten times slower than the one above it.

Tier	Rough per-GPU bandwidth	What it reaches
HBM	Terabytes per second	The GPU’s own memory
NVLink / NVSwitch	~1.8 TB/s on Blackwell	Inside the fast-fabric domain
PCIe	Tens of GB/s	Host and local devices
InfiniBand	400G to 800G per port	Across racks
Ethernet	400G to 800G per port	Across racks and pods

One fact governs the rest. The moment a collective leaves the fast fabric, per-GPU bandwidth falls roughly 10 to 20 times. Any strategy built on constant chatter stops scaling there.

Some strategies chatter constantly and some do not. That split drives every cluster topology verdict below.

Tensor parallelism swaps activations at every layer. It belongs inside the domain.
Expert parallelism fires an all-to-all for every token sent to a remote expert. It belongs inside the domain too.
Pipeline parallelism passes data between stages a few times per step. It copes with the slower fabric.
Data parallelism syncs gradients once per step, and can hide that behind compute. It copes with the slower fabric well.

We covered the memory side of this cluster topology ladder in our piece on memory bandwidth in AI chips.

Verdict One: Cluster Topology and a 70B Dense Fine-Tune

Start with the friendly case for cluster topology. You want to fine-tune a 70-billion-parameter dense model on 64 GPUs.

The weights and optimizer state need roughly 1.1 TB in mixed precision, before activations. A tensor group of 8 fits neatly inside one HGX node. Pipeline and data groups handle the rest across nodes.

Nothing here strains the cluster topology. Your tensor group never crosses the fabric line, and the traffic between nodes is gradient sync you can hide behind compute.

This is why 8-GPU nodes on a decent fat-tree ran the industry for years. For dense models under about 100 billion parameters, that cluster topology still works.

Verdict: runs well on almost any sane cluster topology, including commodity Ethernet.

Verdict Two: Cluster Topology Under Frontier MoE Training

Now the hard case, and the one that sorts cluster topology into winners and losers.

A large mixture-of-experts model sends each token to a few experts. Those experts sit on different GPUs, so every layer fires an all-to-all across the expert group.

Put that group inside a 72-GPU NVLink domain and the all-to-all runs at NVLink speed. Spread it over two racks and the same step lands on InfiniBand, at a fraction of the bandwidth, in the path of every token.

The maths is brutal and simple. If your cluster topology forces the expert group across racks, you shrink the expert count, eat a large throughput loss, or drop the model.

That is what it means for cluster topology to decide what you can run. It is not a tuning problem. The wiring rules the setup out before the scheduler sees it.

Bigger domains widen your options, not just your speed. A 144-GPU cluster topology lets a tensor or expert group that once needed two racks sit inside one, at scale-up speed.

This is also why the same model can post very different tokens-per-second figures on two clusters with equal GPU counts. The cluster topology, not the silicon, explains the gap.

Verdict: rack-scale cluster topology runs it. Eight-GPU nodes run a smaller model instead, whatever the total GPU count says.

Verdict Three: Cluster Topology and Long-Context Serving

Serving changes the cluster topology question. Now you care about latency per token, not throughput per step.

Fast decode wants a wide tensor group. Split the weights over more GPUs and each one reads less memory per token. That only helps if those GPUs share a fast fabric.

Long context adds a second limit. The KV cache grows with sequence length, and it has to live in HBM your domain can reach.

Here the domain HBM figure earns its place in any cluster topology review. Roughly 1.1 TB in an 8-GPU node against 13.5 TB in an NVL72 rack decides whether a long-context model serves smoothly or hits a wall.

One caution. A wider tensor group is not free. Once it passes the number of key-value heads, the KV cache is copied across ranks, and you buy speed with memory.

So the honest planning question is not “can we serve this model”. It is “at what context length does our cluster topology stop serving it within our latency budget”.

Verdict: runs on small domains at short context, degrades sharply as context grows.

Verdict Four: Cluster Topology for Disaggregated Inference

The newest workload is the most cluster topology sensitive, because it moves large tensors between pools by design.

Split serving separates prefill from decode. Prefill fills up compute. Decode fills up memory bandwidth. Run both on one GPU and you waste one of them, whatever your cluster topology.

The catch is the handoff, and cluster topology owns it. After prefill, the KV cache must reach the decode worker before generation starts, so the transfer sits right on time-to-first-token.

The numbers are not small. One 2026 analysis puts the transfer at roughly 2.6 GB per request for a 70B model, and above 100 GB/s in total at production scale.

Research on this pattern found that bigger NVLink domains reliably improve split serving, because they allow wider expert and tensor groups during generation. Models differ, though. Reasoning-style MoE models gain from more expert parallelism, dense models from more tensor parallelism.

There is also a floor. Below roughly eight GPUs per model, splitting adds work without payoff, and chunked prefill gets most of the benefit free.

Verdict: rack-scale cluster topology runs it well, node-scale runs it at a latency penalty, and small estates should not attempt it.

When Cluster Topology Failures Look Like Code Bugs

Here is the part that costs teams the most time. Cluster topology problems do not announce themselves. They show up as throughput that is somehow half what you expected.

NVIDIA’s own engineers wrote up a clean example while training a large MoE model on GB200 NVL72 racks. Their reinforcement learning framework had no concept of NVLink domains.

The mechanism is worth knowing, because it repeats across stacks. Ray node IDs are random UUIDs with no link to physical position, and the framework trusted whatever rank order it was given.

So expert groups quietly spanned racks. The all-to-all that should have run on NVLink ran on InfiniBand instead, and no log line said so.

The fix was to make rank assignment domain-aware, keeping every GPU in an expert group on one rack. That is a scheduling change, not a model change.

Take the general lesson. If your scheduler assigns ranks without knowing your cluster topology, it will one day place a bandwidth-hungry group across the slowest link you own.

Fabric Choice: What Cluster Topology Buys Above the Rack

Above the fast-fabric domain sits the scale-out network, and that cluster topology layer has really changed.

Three shapes dominate. Fat-tree gives steady latency and full bisection bandwidth. Dragonfly cuts switch count and hops, but limits which workloads behave. Rail-optimised designs line the scale-out paths up with the NVLink domains below, so traffic between domains takes fewer hops.

The protocol choice has moved too. InfiniBand held roughly 80% of AI training clusters in 2023. By mid-2025, Ethernet had taken the lead in AI back-end networks.

Ultra Ethernet is the reason. Its 1.0 specification, released on 11 June 2025, rebuilds the transport with packet spray across every path, out-of-order delivery reassembled at the NIC, and congestion control designed for collective traffic.

The practical cluster topology guidance is duller than the marketing. Tuned Ethernet delivers roughly 85 to 95% of InfiniBand throughput up to about 10,000 GPUs. InfiniBand keeps its edge on tightly coupled training at frontier scale.

NVIDIA’s own Ethernet answer arrived alongside it. Spectrum-X at 800G began shipping in late 2025, validated for Blackwell racks, narrowing the gap for teams that want one vendor across the cluster topology.

UALink takes the other half of the problem. It targets scale-up, chip to chip, so a future cluster topology could use UALink inside the pod and Ultra Ethernet between pods.

One physical limit explains why domains grow slowly. Driving an NVLink spine with pluggable optics was put at around 20 kW, against a rack budget near 120 kW. Power, not ambition, caps the domain. We went deeper on that constraint in our piece on data centre power.

A Cluster Topology Audit You Can Run This Week

Four questions, answerable in an afternoon, tell you most of what your cluster topology allows.

How many GPUs share a fast-fabric domain? Not how many are in the cluster. Run a bandwidth test between GPU pairs and watch where the number falls off a cliff.
How much HBM does one domain hold? Multiply per-GPU memory by domain size. That figure caps model size at a given tensor width.
Does your scheduler know the answer to question one? Check whether rank assignment reads the cluster topology, or just trusts whatever node order it gets.
Where is bandwidth shared? The standard rule is non-blocking inside a scalable unit, with sharing only at the aggregation layer. Find out whether yours follows it.

One more thing worth checking while you are in there. A rack-scale cluster topology gives you one large failure domain, so ask what happens to a running job when a switch tray goes down.

Then map your parallelism plan onto those answers. Tensor and expert groups go inside the domain. Pipeline and data groups go across it. A plan that breaks that rule needs a redesign, not a tune.

Our overview of the AI compute stack covers the layers above and below this one.

The Verdict: Cluster Topology Is a Capability Budget

Go back to the two teams and their matching invoices. Nothing was broken in the second cluster. Its cluster topology was simply wired for a different class of work.
Treat cluster topology the way you treat memory capacity. It is a hard limit on which setups exist at all, not a knob you turn later.
That reframing changes how you buy. The question stops being “how many GPUs can I afford”. It becomes “what is the widest bandwidth-hungry group I will ever need, and does one domain hold it”.
Answer that before you sign. Answer it later and you learn your cluster topology through six weeks of unexplained throughput, which is the costliest way to find out.

Frequently Asked Questions

What is cluster topology in an AI cluster?

Cluster topology describes how GPUs are physically wired together: which GPUs share a high-bandwidth NVLink domain, how those domains connect over InfiniBand or Ethernet, and how the switching layers are arranged. It determines which parallelism strategies are viable, and therefore which models you can train or serve.

Why does NVLink domain size matter so much?

Because bandwidth drops roughly 10 to 20 times the moment a collective leaves it. Tensor and expert groups swap data constantly, so they have to stay inside the domain. In cluster topology terms, an 8-GPU domain and a 72-GPU domain support very different models with the same chips.

Which parallelism strategies tolerate a slower fabric?

Pipeline and data parallelism. Pipeline stages swap data only a few times per step. Data-parallel gradient sync happens once per step and can hide behind compute. Tensor and expert groups cannot cope with the slower fabric in any cluster topology.

Is InfiniBand still necessary for AI cluster topology?

Not always. Ethernet took the lead in AI back-end networks by mid-2025, and tuned RoCEv2 delivers roughly 85 to 95% of InfiniBand throughput for clusters up to around 10,000 GPUs. InfiniBand retains an advantage for tightly coupled training at frontier scale.

What is rail-optimised cluster topology?

A rail connects GPUs holding the same local rank across different domains. Rail-optimised cluster topology lines the scale-out paths up with the NVLink domains below, so traffic between domains crosses fewer switch hops and less of it touches the slower fabric at all.

Keep reading

Here are the latest posts from the blog.

Cluster Topology Decides What You Can Actually Run

September 16, 2026

Two teams buy 512 H100s. One trains a 400-billion-parameter mixture-of-experts model. The other cannot fit that job at all, and burns six weeks finding out …

Multi-Agent Delegation: 4 Costs Nobody Models First

September 14, 2026

A planner agent receives a refund request. It hands the task to a billing agent. The billing agent queries an account agent, which calls a …

What Breaks When Your Model Version Retires

September 13, 2026

The email arrives on a Friday. One of your model snapshots has a shutdown date, and it is sixty days out. Your first instinct says …

Public Data Is Not a Licence: 5 Dangerous Assumptions

September 12, 2026

Somewhere in your training pipeline sits a dataset whose source note reads, in effect, “we found it on the internet”. That note is doing a …

What Breaks When Your Model Version Retires

by Idrees Patel

The email arrives on a Friday. One of your model snapshots has a shutdown date, and it is sixty days out.

Your first instinct says config change. Swap the string, redeploy, done.

Then you start looking. The prompt was tuned against that snapshot. The eval baseline was measured on it. The vector index was built with an embedding model from the same family. The fine-tune sits on a base that goes with it.

Model retirement is not a version bump. It is a migration with a hard deadline, and the deadline belongs to your vendor.

This piece walks those sixty days in order, from notice to cutover, and names what model retirement breaks at each stage.

Key Takeaways

The clock is short and it is not yours. Anthropic runs a fixed 60-day window from deprecation to retirement. Opus 4.1 was deprecated on 5 June 2026 and retired on 5 August.
Nothing fails loudly. The API returns 200, latency holds, throughput holds, and the behavioural regression ships anyway.
Embeddings are the worst case. A retired embedding model means re-embedding the whole corpus, and a dimension or distance-metric mismatch silently ruins ranking.
Fine-tunes retire with their base. OpenAI is shutting down fine-tuned GPT-3.5 and GPT-4 variants on 23 October 2026, and Cohere has already made older fine-tunes inaccessible.
Auto-upgrade can move you before any retirement date. Azure deployments set to update to the default version switch roughly two weeks after a new default publishes.
Sometimes there is no replacement. OpenAI’s deprecation table lists a dash against the Videos API, which means choosing a vendor, not swapping an identifier.

Quick Navigation

Why Model Retirement Is Not a Version Bump
The Blast Radius: What Model Retirement Actually Touches
- Before the runbook, the map. Most teams guess this table at about half its real size, which is why model retirement overruns.
Day 0: The Model Retirement Notice Nobody Reads
- The first model retirement problem is delivery. Notices go to whoever is on the account, not to whoever owns the code.
Week 1: The Inventory Model Retirement Forces on You
- You cannot plan a model retirement until you know where the string lives. It is almost never in one place.
Weeks 2 to 4: Where Model Retirement Actually Breaks Things
Weeks 5 to 7: Running Both Models Through Model Retirement
- Offline evals catch the obvious regressions. Production catches the rest, so model retirement needs both.
The Cutover: What Model Retirement Leaves Behind
Building So the Next Model Retirement Costs Less
The Verdict: Model Retirement Is a Standing Cost
Frequently Asked Questions

Why Model Retirement Is Not a Version Bump

A library upgrade breaks loudly. Types stop matching, tests go red, the build fails. You fix it before anything ships.

Model retirement breaks quietly. The new snapshot takes the same request, returns valid JSON, and answers in the same voice.

That is the trap. Your dashboards watch error rate, latency and throughput. A model retirement swap can leave all three flat while it rewires how the system behaves.

So the regression ships, then shows up later wearing a different face. Support tickets rise. A classifier drifts. An agent takes an extra tool call it never needed.

Model retirement also runs on someone else’s calendar. You are not choosing to upgrade. You are handed a date, and the date does not negotiate.

The Blast Radius: What Model Retirement Actually Touches

Before the runbook, the map. Most teams guess this table at about half its real size, which is why model retirement overruns.

Artefact	What model retirement does to it	How it fails
Prompts	Tuning was fitted to one snapshot	Quietly, in output quality
Eval baselines	Scores were measured on a model that no longer exists	Silently, as a lost reference point
Vector indexes	Built with an embedding model in the same lifecycle	Silently, as worse retrieval
Fine-tunes	Retired alongside their base model	Loudly, with an API error
Tool schemas	Calling behaviour and format shift between versions	Intermittently, under load
Audit records	Reference a model nobody can re-run	At the next audit
Cost model	Token spend and reasoning behaviour change	On the invoice

Note the third column. Four of the seven fail with no error at all. That silence is why model retirement eats more calendar time than anyone budgets.

Day 0: The Model Retirement Notice Nobody Reads

The first model retirement problem is delivery. Notices go to whoever is on the account, not to whoever owns the code.

Azure notifies the subscription roles: owner, contributor, reader and the monitoring roles. Whether that list includes the engineer who wrote the prompt is down to your tenant hygiene.

Model retirement windows vary more than people expect. Anthropic commits to at least 60 days for a public model, with a fixed gap from deprecation to shutdown.

Azure notifies at 60 days, at 30 days, and at retirement, with at least 60 days of notice for generally available models and only 14 days before a preview version upgrade.

OpenAI runs longer on its main line. Its June 2026 notice set a December 2026 shutdown for the GPT-5 and o3 snapshots, roughly six months out.

One detail hides inside the date itself. Azure runs retirements on a rolling basis, region by region, so two deployments of the same model can behave differently for part of the window.

What to do on day zero. Put the date in your release calendar, name an owner, and check the notice actually reached that person. Model retirement nobody owns gets found in week seven.

Week 1: The Inventory Model Retirement Forces on You

You cannot plan a model retirement until you know where the string lives. It is almost never in one place.

Grep the model ID across app code, prompt templates, eval configs, infra code, notebooks, cron jobs and vendor dashboards. Then grep the aliases too.

Aliases deserve their own line, because they fail the other way. An alias never errors on model retirement. It just starts pointing somewhere else, which is worse than a break you can see.

Azure makes this concrete. A deployment set to update when a new default arrives moves about two weeks after that default publishes, not on the retirement date.

So model retirement can move you early, on a schedule you were not watching. Pinning to a dated snapshot trades that surprise for a date you control.

Check the replacement column while you are there. Most model retirement notices name a successor, and the ones that do not are a different project entirely.

One inventory item teams forget. List every artefact the old model produced that you still lean on: embeddings, cached completions, synthetic training data, labelled eval sets.

Weeks 2 to 4: Where Model Retirement Actually Breaks Things

This stretch decides whether you make the date. Five model retirement failure classes, roughly in the order they eat time.

Prompts Tuned to a Snapshot

Every production prompt carries fitting. Someone added a line because the old model rambled, or cut one because it refused too much.

That fitting does not carry over. A newer model may need less scaffolding, or may react badly to lines written around a quirk it no longer has. Model retirement resets the tuning, not just the ID.

Reasoning models sharpen this. Vendor migration guides now read as “rewrite the prompt”, not “reuse the prompt”.

Evals That Lose Their Baseline

Here is the circular problem in every model retirement. You want to prove the new model is no worse, and the proof rests on scores measured against a model about to vanish.

Re-run your full eval suite on the outgoing snapshot before it shuts down. Store the outputs, not just the aggregate numbers.

Those stored outputs are the only comparison you will ever have. After the shutdown date you cannot rebuild the baseline, and model retirement has quietly taken your reference point.

The Embedding Trap in Model Retirement

This is the expensive one, and the one model retirement plans miss until late.

If your index was built with an embedding model that is retiring, you cannot mix old and new vectors. The whole corpus needs re-embedding. That is compute, calendar time and database write throughput.

Worse, matching dimensions do not mean the models match. Models differ in distance metric and scaling, so an index set for cosine similarity holding vectors meant for dot product ranks wrongly while looking healthy.

The safe model retirement pattern is a dual index. Build the second one beside the first, route each query to model and index together, compare on a labelled set, then cut over. Keep the old index as rollback for a week.

Fine-Tunes That Retire With Their Base

A fine-tune is not an asset you fully own. It is an adapter on a base model, and model retirement usually takes both.

OpenAI’s table lists shutdowns for fine-tuned GPT-3.5 and GPT-4 variants on 23 October 2026. It names replacement base models rather than migrating anything for you.

Cohere went further and stated plainly that previously fine-tuned models would no longer be accessible once fine-tuning for those bases was retired.

Plan a re-train, not a port. Keep training data, hyperparameters and eval scores versioned together, because model retirement will ask for all three.

This stretch decides whether you make the date. Five model retirement failure classes, roughly in the order they eat time.

Tool Calls and Structured Output

Agents have a model retirement failure mode the rest do not. Tool-calling behavior shifts between versions in ways that are hard to write a test for.

The new model may call a tool the old one never touched, call it in a new order, or pass a slightly different argument. Everything validates. The path changes.

Watch for API changes riding alongside model retirement. When OpenAI sunset the Assistants API, threads became conversations and runs became responses, so the request shape moved, not just the ID.

Agents have a model retirement failure mode the rest do not. Tool-calling behavior shifts between versions in ways that are hard to write a test for.

The new model may call a tool the old one never touched, call it in a new order, or pass a slightly different argument. Everything validates. The path changes.

Watch for API changes riding alongside model retirement. When OpenAI sunset the Assistants API, threads became conversations and runs became responses, so the request shape moved, not just the ID.

Weeks 5 to 7: Running Both Models Through Model Retirement

Offline evals catch the obvious regressions. Production catches the rest, so model retirement needs both.

Shadow traffic is the cheapest insurance available during model retirement. Send a slice of real requests to both models, log both outputs, and compare without serving the new one to anyone.

Compare the right things. Output quality on your labelled set, yes, but also token spend per request, p95 latency, refusal rate, and for agents, tool-call counts and path length.

Then ramp rather than flip. One percent, five, twenty five, with a rollback that stays valid until the old snapshot actually shuts down.

Keep one number in view throughout. A model that scores identically while spending forty percent more output tokens is not a neutral swap, and that arithmetic sits in our piece on what inference actually costs per token.

Also re-run your safety testing. A model version change is one of the standard triggers for a fresh adversarial pass, which we covered in the hidden flaws in a passing red-team test.

The Cutover: What Model Retirement Leaves Behind

The switch is the easy part of model retirement. The residue is not.

Start with repeatability. You can no longer regenerate any output the old snapshot made. That matters when a customer disputes a decision, or a regulator asks how it was produced.

That is a records problem more than a technical one. Model retirement means your logs must already hold the snapshot ID, the prompt version and the parameters, because the model will not be there to ask. We went deeper on that in our guide to AI compliance evidence.

Then come the dependencies you never called dependencies. Cached completions from the old model. Synthetic data used to train a smaller one. Labelled sets where the old model did the labeling.

None of those break on the shutdown date. Model retirement just turns them into relics of a system you can no longer rebuild.

One harder case deserves naming. Sometimes model retirement offers no replacement at all. OpenAI’s table carries a dash against the Videos API entry, which means finding another vendor rather than editing a string.

Building So the Next Model Retirement Costs Less

There will be a next model retirement, usually within a year. Five changes make it routine instead of disruptive.

Pin snapshots in production. Aliases are for experiments. A dated identifier means model retirement arrives as a deadline you can see rather than a behavior change you cannot.
Put the model behind one boundary. One config value, one client wrapper, one place to edit. If the ID sits in forty files, model retirement is forty times harder than it needs to be.
Version the eval set like code. Same repository, same review process, and a rule that every migration re-runs the outgoing model before shutdown.
Derive index names from the embedding config. Model, dimensions, distance metric and chunk version in one config that builds the index name, so a mismatch cannot happen quietly.
Log the snapshot with every response. Not the family name, the dated identifier. This is the single cheapest habit on the list and the one that saves you in an audit.

Then diary a quarterly check of your vendors’ deprecation pages. The model retirement email is unreliable. The page is not.

The Verdict: Model Retirement Is a Standing Cost

Go back to that Friday email and the sixty-day clock. Nothing in your system was broken when it arrived.
What changed is that a part you do not control picked up an expiry date, and every artefact built on it inherited the same one.
Treat model retirement as a recurring line item rather than an incident. Somewhere between two and four times a year, an engineer spends a few weeks on migration, and the budget should say so.
The line item is smaller than it looks, too. An engineer for three weeks, twice a year, is cheaper than one rushed cutover that ships a quiet regression to customers.
Teams that find model retirement painless are not lucky. They pinned their snapshots, versioned their evals, and logged which model produced what.
The teams that find it brutal usually discover in week seven that the vector index was built with something that retires on the same day.

Frequently Asked Questions

What happens when a model version is retired?

Requests to the retired ID stop working. Azure states plainly that deployments of a retired model always return error responses. Anything built on that model — prompts, eval baselines, fine-tunes, embeddings — either breaks with it or loses its reference point.

How much notice do providers give before model retirement?

Model retirement notice varies widely. Anthropic commits to at least 60 days for public models, with a fixed gap from deprecation to shutdown. Azure gives at least 60 days for GA models and only 14 days before preview upgrades. OpenAI’s main-line notices run closer to six months.

Do fine-tuned models survive model retirement of the base?

Usually not. Model retirement of the base normally takes the fine-tune with it. OpenAI listed shutdown dates for fine-tuned GPT-3.5 and GPT-4 variants with replacement base models rather than automatic migration, and Cohere said previously fine-tuned models would become inaccessible. Budget a re-train and keep your training data versioned.

What happens to my vector database when an embedding model retires?

You re-embed the corpus. Old and new vectors cannot be mixed, and matching dimensions do not mean the models match, since they differ in distance metric and scaling. Use a dual-index migration, compare retrieval quality on a labelled set, then cut over with the old index kept as rollback.

Can a model change before its retirement date?

Yes, if you use an alias or an auto-update policy. An Azure deployment set to update once a new default arrives moves about two weeks after that default publishes, whatever the retirement date says. Pinned snapshots avoid this.

Keep reading

Here are the latest posts from the blog.