The AI Compute Stack: 5 Layers That Now Break First

Every AI story eventually becomes an AI compute stack story. A model launch is really a memory story. Behind a funding round sits a power story. And a pricing change is usually a utilization story.

This page is the map. It walks the AI compute stack from silicon to electricity bill, names what constrains each layer, and links to deeper coverage on each piece. Read it top to bottom once, then use it as a directory. Each layer section ends with the posts worth reading next.

Key Takeaways on the AI Compute Stack

• The AI compute stack has five layers: chips, memory, interconnect, power, and cost. Each one caps the layer above it.
• Memory bandwidth, not raw compute, is the binding constraint on most serving workloads today.
• AI racks now draw 30–110 kW against 5–15 kW for traditional racks, which broke conventional cooling.
• Grid access has replaced real estate as the main limit on new capacity.
• The IEA puts global data centre electricity at 415 TWh in 2024, heading toward roughly 945 TWh by 2030.

Quick Navigation

How the AI Compute Stack Fits Together

Think of the AI compute stack as a ladder where every rung sets the height of the next.

A chip computes only as fast as memory feeds it. Memory helps only if the interconnect moves data between chips. Nothing runs without power. Every layer converts into a cost per token at the top. So when someone says a model is expensive, the useful question is which layer of the stack is actually binding. The answer changes the fix entirely.

Why the stack beats vendor framing

Most coverage of the AI compute stack organizes around companies. Nvidia news, Google news, OpenAI news.

That framing hides the pattern. A memory shortage, a grid delay, and a networking fault all produce one symptom — a slipped deployment — yet need completely different fixes. Layer thinking also travels better, because vendors change and physics does not.

Layer 0 of the AI Compute Stack: Fabrication

Below silicon sits the ability to make silicon. It is the one layer of the stack nobody routes around.

One foundry manufactures nearly every leading-edge AI accelerator. Advanced packaging is scarcer still. That step bonds memory stacks to a processor die, and it gates output more tightly than wafer supply does.

Why packaging is the real queue in the stack

A design finished today waits on packaging slots booked a year ago. That lead time propagates upward through the whole AI compute stack, which is why accelerator roadmaps slip in quarters rather than weeks.

Memory makers face the same wall. Every vendor roadmap now runs to HBM4E, yet capacity is booked years ahead. So when a chip is sold out, the constraint is rarely the chip. It is a step in the stack you never see named.

Layer 1 of the AI Compute Stack: Chips

Silicon is where the arithmetic happens. This layer of the AI compute stack splits along one line that matters more than any other.

Training hardware chases throughput across long batch jobs. Serving hardware chases latency on single requests repeated billions of times. Those goals pull a design in opposite directions. Nvidia holds roughly 80% of accelerator revenue, the most concentrated position in the stack. The rest divides between AMD, hyperscaler silicon such as Trainium and TPUs, and a thin band of specialists.

Read next in this layer:

• Why Inference Chips Now Win a Much Narrower Race — the Groq and Cerebras story, and why specialization paid
• 5 Ways Trainium and TPU Threaten Nvidia — custom silicon and the margin question
• Google’s TPU Deal With Anthropic — what a multi-gigawatt commitment looks like in practice

Layer 2 of the AI Compute Stack: Memory

Here is the AI compute stack layer most coverage underrates, and currently the tightest.

Generating each token means reading the model’s weights again. That step is memory-bound, not compute-bound, so the processor waits on data instead of the reverse.

Why HBM decides so much

High Bandwidth Memory stacks DRAM vertically beside the processor. HBM4 entered mass production in February 2026, doubling the interface from 1,024 bits to 2,048 and raising channels from 16 to 32. Supply is concentrated. Samsung and SK Hynix together make roughly 90% of it. So memory is a single point of failure for the whole stack.

The capacity ceiling

Frontier models now exceed what one device holds. Inkling needs roughly 2TB of aggregated VRAM at BF16, and Kimi K3’s checkpoint runs 1.56TB. So capacity, not capability, decides who deploys. That gap in the stack is the quiet story behind every open-weights release.

Read next in this layer:

• The AI Glossary — HBM, KV cache, LPDDR and quantization defined
• Inkling 975B: What the Open Weights Now Really Change — why downloadable is not runnable

Layer 3 of the AI Compute Stack: Interconnect

Once a model exceeds one chip, the wires between chips join the AI compute stack as real hardware.

Every hop between accelerators costs latency. A model split across 64 devices pays that tax at every layer boundary, which is why wafer-scale designs exist. Two fabrics matter here: scale-up links join chips inside a rack, and scale-out networking joins racks into clusters.

The overlooked failure mode

Interconnect faults rarely announce themselves. They surface as low utilization, and teams blame the model instead. That mismatch is why the stack needs measuring end to end. An idle accelerator is often a networking problem wearing a compute costume.

Layer 4 of the AI Compute Stack: Power and Cooling

Now the AI compute stack layer that turned from background detail into the main constraint.

Traditional server racks draw 5–15 kW. AI racks now demand 30 kW to over 110 kW, and Blackwell-class configurations reach roughly 140 kW. That is a tenfold jump. It made conventional air cooling obsolete rather than merely inefficient.

The grid became the bottleneck

Before 2024 a large site needed 10–20 MW. New AI sites are designed for 100–300 MW, and hyperscale campuses are planned at a gigawatt or more. Interconnection queues now run three to seven years in many US regions. So grid availability, not land or capital, sets the pace of the entire stack.

Operators answered by building their own supply: on-site gas turbines, power purchase agreements, nuclear deals. That shift in the AI compute stack looks permanent.

Why PUE stopped being the right metric

Power Usage Effectiveness measures overhead, and hyperscale leaders report 1.08–1.09. Excellent numbers. Yet PUE says nothing about what the compute produced. A site with perfect overhead running idle accelerators still wastes power.

Tokens per watt is the better frame. It ties the bottom of the stack to the top, which is what an honest efficiency claim must do.

Read next in this layer:

• The Ultimate Truth About the $39B Humanoid Robot Race — where compute demand goes once it leaves the data centre

Layer 5 of the AI Compute Stack: Cost

Every AI compute stack layer below converts here, and the conversion is less obvious than it looks.

Serving now takes most accelerator spending. Training a frontier model is a one-time cost, while running it scales with every query, forever.

Three AI compute stack costs people conflate

Cost per token is what a vendor charges. It is the easiest number to compare and the least useful alone.

Cost per task includes reasoning tokens. A model with cheap tokens that thinks for three thousand of them can cost more than an expensive model that answers in four hundred.

Total cost of ownership adds hardware, utilization, engineering time, and idle capacity. A rack that serves one workload sits unused whenever traffic dips, and that gap never appears on a pricing page.

Why utilization dominates

A cluster at 30% utilization costs roughly three times per token what the same cluster costs at 90%. No chip upgrade in the stack produces that swing.

So the cheapest move is usually scheduling, not procurement. Teams reach for new hardware when better batching would have done it.

Read next in this layer:

• Anthropic S-1: What an AI Lab Must Now Disclose — compute commitments as contractual obligations
• DeepSeek IPO: What the $70B Number Now Hides — cost structures under export controls
• Why Composite Benchmarks Now Fail — why benchmark scores mislead on real cost

How the AI Compute Stack Layers Trade Against Each Other

AI compute stack layers are not independent. The trades between them are where real decisions live.

Quantization is the clearest example. Cutting from 16-bit to 4-bit roughly quarters memory use, which eases Layer 2. It cuts power draw too, easing Layer 4. And it trims accuracy slightly, a cost at Layer 5.

A worked example

Suppose serving costs run too high. Four fixes sit at four different stack layers.

Buy faster chips, and you fix Layer 1 at the highest capital cost. Quantize the model, and you fix Layer 2 for a small accuracy loss. Improve batching, and you fix Layer 5 for engineering time alone. Move the work to cheaper power, and you fix Layer 4 at the cost of latency. Three of those four cost less than the one most teams try first.

Why the cheapest fix is usually highest in the stack

Capital moves slowly and software moves fast. So changes near the top land in weeks, while changes near the bottom land in quarters.

That asymmetry should shape the order in which you investigate a cost problem, and it usually does not.

What Changed in the AI Compute Stack This Year

Four shifts reshaped the stack in 2026, and each moved a different layer.

Memory generation. HBM4 reached mass production in February, doubling interface width. That eased a constraint that had held since 2023.

Power became structural. Grid interconnection queues stretched past the point where new capacity could be planned around them, so operators began building their own supply.

Serving overtook training. Estimates now put serving at 60–70% of accelerator spending. That changed what buyers optimize for.

Open weights arrived at frontier scale. Downloadable trillion-parameter models exist. Yet the memory needed to serve them keeps access narrow.

Read together, those four say one thing. Capability stopped being scarce, and the stack around it became the constraint.

Where the AI Compute Stack Bottleneck Sits Now

Bottlenecks migrate. Knowing the current one beats knowing all five layers in the abstract.

In 2020 it was chips, because supply could not meet demand at any price. By 2023 it was memory, since HBM allocation decided who shipped, and that still constrains capacity today.

By 2026 it moved again. Power is now the pacing item for new capacity, while memory bandwidth remains the pacing item for existing capacity.

What that means practically

If you are building capacity, your stack problem is a utility queue. If you are serving on capacity you own, it is bandwidth and batching.

Those are different problems, with different vendors, timelines, and budgets. Conflating them wastes a year.

Who Controls Each Layer of the AI Compute Stack

Concentration varies sharply across the stack, and that shapes negotiating power.

Chips are concentrated but contested, since one vendor dominates while credible alternatives ship.

Memory is the most concentrated layer of the stack. Two suppliers, roughly 90% of output, and a fabrication process that cannot expand quickly.

Interconnect sits in the middle, with proprietary fabrics competing against open standards. Power is fragmented by geography and regulated locally, which is why capacity plans differ so much between regions.

Cost is where the rest resolve, and the only stack layer a buyer directly controls.

What This Hub Does Not Cover

Three things sit outside this map, and mixing them in causes confusion.

Model architecture is not part of the stack. A better attention mechanism changes what the hardware has to do, but it does not change what the hardware is.

Software frameworks matter enormously, yet they move too fast for a hub page. Serving stacks shift release to release, so those belong in dated posts rather than here.

Policy sits adjacent to the stack, not inside it. Export controls and energy regulation shape every layer, though they follow political timelines rather than technical ones.

How to Use This AI Compute Stack Hub

Three ways to use this stack hub, depending on why you are here.

Following a story. Find the stack layer it touches, then read the linked pieces there. A chip announcement almost always has memory and power implications the announcement omits.

Making a decision. Start at Layer 5 and work down. Name the cost you are optimizing, then find which layer actually binds it.

Learning the field. Read the layers in order, and keep the glossary open alongside. Most confusion in the AI compute stack is vocabulary, not concept. The physics is simpler than the jargon.

This stack hub updates as coverage grows. Every new infrastructure post links back here, and the hub links out to the ones worth reading first.

Conclusion: The AI Compute Stack Is One System

AI compute stack layers get covered separately because different reporters cover them. That is a newsroom artifact, not a fact about the world.

In practice a memory shortage raises power costs, since idle accelerators still draw current. A grid delay raises chip costs, since capacity sits unsold. And a networking fault looks exactly like a slow model.

Reading the stack as one system is the difference between following AI news and understanding it.

Start anywhere in the stack. The layers will pull you to the rest.

FAQ About the AI Compute Stack

Keep reading

Here are the latest posts from the blog.