disaggregated inference

Two teams buy 512 H100s. One trains a 400-billion-parameter mixture-of-experts model. The other cannot fit that job at all, and burns six weeks finding out why.

Same GPUs. Same count. Different wiring.

The GPU count on your invoice says how much compute you bought. It does not say which jobs will run. Cluster topology settles that, and the gap between the two numbers is where most buyer’s remorse lives.

This piece runs four real workloads against your cluster topology and gives a verdict on each. Some run. Some run badly. One simply will not.

Key Takeaways

• Your fast-fabric domain size is the number that matters. An 8-GPU node and a 72-GPU rack hold the same chips and run different workloads.
• Leaving the fast fabric costs an order of magnitude. When a collective walks off NVLink onto InfiniBand or Ethernet, per-GPU bandwidth drops roughly 10 to 20 times.
• Two parallelism types are picky, two are not. Tensor and expert parallelism must stay inside the domain. Pipeline and data parallelism tolerate the slower fabric.
• Cluster topology failures look exactly like code bugs. NVIDIA’s own team found expert-parallel groups silently spanning racks because the framework had no concept of an NVLink domain.
• The fabric above the rack is now a real choice. Ethernet took the lead in AI back-end networks by mid-2025, and tuned RoCEv2 delivers roughly 85 to 95% of InfiniBand throughput below 10,000 GPUs.
• Audit before you buy, not after. Four questions, listed at the end, tell you what your cluster topology can actually serve.

---

Quick Navigation

---

The One Number Your Cluster Topology Hands You

Ask a vendor one cluster topology question first. How many GPUs sit inside a single fast-fabric domain?

That domain is the set of GPUs wired with NVLink and a switch. They behave almost like one huge chip. Everything outside the domain talks over a much slower network.

In cluster topology terms, an older HGX node answers eight. In a GB200 NVL72 rack, 72 GPUs across 18 hosts share one NVLink domain. InfiniBand or RoCE then stitches racks together above it.

The gap is not small. Eight GPUs give you about 7.2 TB/s of NVLink and about 1.1 TB of HBM in the domain. Seventy-two give you around 130 TB/s and roughly 13.5 TB. Same chips, different cluster topology.

Vera Rubin NVL144 is reported to double the domain again, to 144 GPUs, with NVLink 6 at roughly 3.6 TB/s per GPU. Volume production is expected in the second half of 2026. Doubling the links as the domain grows keeps collectives well behaved.

So your cluster topology hands you a budget of what is possible before you write a line of code. The rest of this piece spends it.

---

The Bandwidth Ladder Beneath Every Cluster Topology

Every cluster topology sits on a bandwidth ladder. Each rung runs about ten times slower than the one above it.

Tier	Rough per-GPU bandwidth	What it reaches
HBM	Terabytes per second	The GPU’s own memory
NVLink / NVSwitch	~1.8 TB/s on Blackwell	Inside the fast-fabric domain
PCIe	Tens of GB/s	Host and local devices
InfiniBand	400G to 800G per port	Across racks
Ethernet	400G to 800G per port	Across racks and pods

One fact governs the rest. The moment a collective leaves the fast fabric, per-GPU bandwidth falls roughly 10 to 20 times. Any strategy built on constant chatter stops scaling there.

Some strategies chatter constantly and some do not. That split drives every cluster topology verdict below.

• Tensor parallelism swaps activations at every layer. It belongs inside the domain.
• Expert parallelism fires an all-to-all for every token sent to a remote expert. It belongs inside the domain too.
• Pipeline parallelism passes data between stages a few times per step. It copes with the slower fabric.
• Data parallelism syncs gradients once per step, and can hide that behind compute. It copes with the slower fabric well.

We covered the memory side of this cluster topology ladder in our piece on memory bandwidth in AI chips.

---

Verdict One: Cluster Topology and a 70B Dense Fine-Tune

Start with the friendly case for cluster topology. You want to fine-tune a 70-billion-parameter dense model on 64 GPUs.

The weights and optimizer state need roughly 1.1 TB in mixed precision, before activations. A tensor group of 8 fits neatly inside one HGX node. Pipeline and data groups handle the rest across nodes.

Nothing here strains the cluster topology. Your tensor group never crosses the fabric line, and the traffic between nodes is gradient sync you can hide behind compute.

This is why 8-GPU nodes on a decent fat-tree ran the industry for years. For dense models under about 100 billion parameters, that cluster topology still works.

Verdict: runs well on almost any sane cluster topology, including commodity Ethernet.

---

Verdict Two: Cluster Topology Under Frontier MoE Training

Now the hard case, and the one that sorts cluster topology into winners and losers.

A large mixture-of-experts model sends each token to a few experts. Those experts sit on different GPUs, so every layer fires an all-to-all across the expert group.

Put that group inside a 72-GPU NVLink domain and the all-to-all runs at NVLink speed. Spread it over two racks and the same step lands on InfiniBand, at a fraction of the bandwidth, in the path of every token.

The maths is brutal and simple. If your cluster topology forces the expert group across racks, you shrink the expert count, eat a large throughput loss, or drop the model.

That is what it means for cluster topology to decide what you can run. It is not a tuning problem. The wiring rules the setup out before the scheduler sees it.

Bigger domains widen your options, not just your speed. A 144-GPU cluster topology lets a tensor or expert group that once needed two racks sit inside one, at scale-up speed.

This is also why the same model can post very different tokens-per-second figures on two clusters with equal GPU counts. The cluster topology, not the silicon, explains the gap.

Verdict: rack-scale cluster topology runs it. Eight-GPU nodes run a smaller model instead, whatever the total GPU count says.

---

Verdict Three: Cluster Topology and Long-Context Serving

Serving changes the cluster topology question. Now you care about latency per token, not throughput per step.

Fast decode wants a wide tensor group. Split the weights over more GPUs and each one reads less memory per token. That only helps if those GPUs share a fast fabric.

Long context adds a second limit. The KV cache grows with sequence length, and it has to live in HBM your domain can reach.

Here the domain HBM figure earns its place in any cluster topology review. Roughly 1.1 TB in an 8-GPU node against 13.5 TB in an NVL72 rack decides whether a long-context model serves smoothly or hits a wall.

One caution. A wider tensor group is not free. Once it passes the number of key-value heads, the KV cache is copied across ranks, and you buy speed with memory.

So the honest planning question is not “can we serve this model”. It is “at what context length does our cluster topology stop serving it within our latency budget”.

Verdict: runs on small domains at short context, degrades sharply as context grows.

---

Verdict Four: Cluster Topology for Disaggregated Inference

The newest workload is the most cluster topology sensitive, because it moves large tensors between pools by design.

Split serving separates prefill from decode. Prefill fills up compute. Decode fills up memory bandwidth. Run both on one GPU and you waste one of them, whatever your cluster topology.

The catch is the handoff, and cluster topology owns it. After prefill, the KV cache must reach the decode worker before generation starts, so the transfer sits right on time-to-first-token.

The numbers are not small. One 2026 analysis puts the transfer at roughly 2.6 GB per request for a 70B model, and above 100 GB/s in total at production scale.

Research on this pattern found that bigger NVLink domains reliably improve split serving, because they allow wider expert and tensor groups during generation. Models differ, though. Reasoning-style MoE models gain from more expert parallelism, dense models from more tensor parallelism.

There is also a floor. Below roughly eight GPUs per model, splitting adds work without payoff, and chunked prefill gets most of the benefit free.

Verdict: rack-scale cluster topology runs it well, node-scale runs it at a latency penalty, and small estates should not attempt it.

---

When Cluster Topology Failures Look Like Code Bugs

Here is the part that costs teams the most time. Cluster topology problems do not announce themselves. They show up as throughput that is somehow half what you expected.

NVIDIA’s own engineers wrote up a clean example while training a large MoE model on GB200 NVL72 racks. Their reinforcement learning framework had no concept of NVLink domains.

The mechanism is worth knowing, because it repeats across stacks. Ray node IDs are random UUIDs with no link to physical position, and the framework trusted whatever rank order it was given.

So expert groups quietly spanned racks. The all-to-all that should have run on NVLink ran on InfiniBand instead, and no log line said so.

The fix was to make rank assignment domain-aware, keeping every GPU in an expert group on one rack. That is a scheduling change, not a model change.

Take the general lesson. If your scheduler assigns ranks without knowing your cluster topology, it will one day place a bandwidth-hungry group across the slowest link you own.

---

Fabric Choice: What Cluster Topology Buys Above the Rack

Above the fast-fabric domain sits the scale-out network, and that cluster topology layer has really changed.

Three shapes dominate. Fat-tree gives steady latency and full bisection bandwidth. Dragonfly cuts switch count and hops, but limits which workloads behave. Rail-optimised designs line the scale-out paths up with the NVLink domains below, so traffic between domains takes fewer hops.

The protocol choice has moved too. InfiniBand held roughly 80% of AI training clusters in 2023. By mid-2025, Ethernet had taken the lead in AI back-end networks.

Ultra Ethernet is the reason. Its 1.0 specification, released on 11 June 2025, rebuilds the transport with packet spray across every path, out-of-order delivery reassembled at the NIC, and congestion control designed for collective traffic.

The practical cluster topology guidance is duller than the marketing. Tuned Ethernet delivers roughly 85 to 95% of InfiniBand throughput up to about 10,000 GPUs. InfiniBand keeps its edge on tightly coupled training at frontier scale.

NVIDIA’s own Ethernet answer arrived alongside it. Spectrum-X at 800G began shipping in late 2025, validated for Blackwell racks, narrowing the gap for teams that want one vendor across the cluster topology.

UALink takes the other half of the problem. It targets scale-up, chip to chip, so a future cluster topology could use UALink inside the pod and Ultra Ethernet between pods.

One physical limit explains why domains grow slowly. Driving an NVLink spine with pluggable optics was put at around 20 kW, against a rack budget near 120 kW. Power, not ambition, caps the domain. We went deeper on that constraint in our piece on data centre power.

---

A Cluster Topology Audit You Can Run This Week

Four questions, answerable in an afternoon, tell you most of what your cluster topology allows.

1. How many GPUs share a fast-fabric domain? Not how many are in the cluster. Run a bandwidth test between GPU pairs and watch where the number falls off a cliff.
2. How much HBM does one domain hold? Multiply per-GPU memory by domain size. That figure caps model size at a given tensor width.
3. Does your scheduler know the answer to question one? Check whether rank assignment reads the cluster topology, or just trusts whatever node order it gets.
4. Where is bandwidth shared? The standard rule is non-blocking inside a scalable unit, with sharing only at the aggregation layer. Find out whether yours follows it.

One more thing worth checking while you are in there. A rack-scale cluster topology gives you one large failure domain, so ask what happens to a running job when a switch tray goes down.

Then map your parallelism plan onto those answers. Tensor and expert groups go inside the domain. Pipeline and data groups go across it. A plan that breaks that rule needs a redesign, not a tune.

Our overview of the AI compute stack covers the layers above and below this one.

---

The Verdict: Cluster Topology Is a Capability Budget

• Go back to the two teams and their matching invoices. Nothing was broken in the second cluster. Its cluster topology was simply wired for a different class of work.
• Treat cluster topology the way you treat memory capacity. It is a hard limit on which setups exist at all, not a knob you turn later.
• That reframing changes how you buy. The question stops being “how many GPUs can I afford”. It becomes “what is the widest bandwidth-hungry group I will ever need, and does one domain hold it”.
• Answer that before you sign. Answer it later and you learn your cluster topology through six weeks of unexplained throughput, which is the costliest way to find out.

---

Frequently Asked Questions

---

Keep reading

Here are the latest posts from the blog.