agentic workloads

Modal ran 50 coding tasks one at a time on NVIDIA’s new Vera CPU and on Intel, AMD and AWS chips. Vera was often faster, “but not always, and not by a particularly large margin,” the company wrote. Then Modal packed the same machines with concurrent containers. On that test, it says, Vera finished up to twice as many tasks as an Intel Granite Rapids server.

That gap between the serial and concurrent results is the most useful fact in this month’s argument about the CPU for AI agents. NVIDIA says agents are bringing single-threaded performance back. AMD has announced an agent-focused rack that can hold 32,768 CPU cores. Yet the clearest public data point so far points less at faster cores than at how a processor behaves when hundreds of small jobs compete for memory.

Key Takeaways

• Confirmed: NVIDIA has shipped its first Vera systems and markets them for agent work. AMD has announced Helion, a rack blueprint planned for November 2026. Both vendors now pitch CPUs specifically for agentic workloads.
• Reported: Modal measured up to 2x the task throughput on Vera against an Intel Granite Rapids machine. The test used one node and a synthetic benchmark with network calls removed, and nobody has independently checked it.
• Analysis: Modal attributes its result to memory bandwidth under concurrency, not single-threaded speed. That supports a narrower claim than “single-core performance is back.”
• Analysis: A faster core only speeds up the CPU-bound share of an agent’s task. When most time goes to GPU inference, external APIs or rate limits, the end-to-end gain is small.
• Practical: Before buying hardware, trace where one task’s time goes, then load-test at your real concurrency and track p99 latency and cost per successful task.

---

Quick Navigation

---

What the Vera, Modal and Helion announcements establish

NVIDIA’s position is the most developed. Vera has 88 custom Olympus cores and 176 threads through what NVIDIA calls spatial multithreading. It supports up to 1.5 TB of LPDDR5X memory at up to 1.2 TB/s, and all of this sits on a single compute die. NVIDIA markets it explicitly for “the CPU work behind agentic AI”: code execution, tool use, sandboxing and orchestration. It says it has hand-delivered the first systems to Anthropic, OpenAI, Oracle Cloud Infrastructure and SpaceXAI. It hasn’t given a date for general availability.

The design argument came in a Wccftech interview with NVIDIA’s Vera team, published on 18 October; the interviewees aren’t named. Their claim is that chiplet-based server CPUs gave up per-core performance to reach high core counts. “What suffered over that time in terms of CPU design was really single-threaded perf and IPC,” the team said. They put Vera at up to 2x faster than leading x86 chips on agentic workloads, and the product page claims “up to 1.8x” on sandbox tasks. Both are vendor figures, with no configuration disclosed.

The first customer data

Modal, which runs sandboxed code for AI agents, published the first customer-side data a day later. It has received its first Vera server and plans to move its Sandboxes product onto the platform. That first requires porting its infrastructure and image builds to Arm, and Modal hasn’t given an availability date.

AMD’s rack-scale answer

AMD’s Helion, announced on 12 October, is an open rack blueprint based on an Open Compute Project design. The CPU-only version holds up to 32,768 Zen 6 cores. Its product page lists three blades: dense 256-core servers, balanced 256-core servers and a 96-core high-frequency “performance” option. A separate configuration adds Instinct MI350P GPUs. AMD says configurations are planned for November and has published no benchmarks.

Together, these establish that NVIDIA and AMD now both sell server CPUs configured specifically for agents. They don’t yet establish that agent workloads, in general, reward per-core speed over core count. AMD hedges by offering both. Its 96-core high-frequency blade is a quiet concession that some agent work wants fast cores, while its 256-core blades bet that most of it wants density.

Why agent execution reaches the CPU

A conventional inference request is short and GPU-heavy. Text arrives, the model generates a response, and the CPU’s job is mostly moving data and scheduling batches. An agent loop is different, because it alternates between model calls and work the model asked for:

Agent request → orchestration → model inference → tool execution → tool response → next decision → … → final response

Each tool step may compile code, run tests, search files, query a database or call an external API. Several of those run on the CPU. Each step also depends on the previous one: the model can’t choose the next action until the tool returns. That serial dependency is what makes per-step latency matter.

Real systems aren’t this tidy. Many agents issue several tool calls in parallel, and some frameworks overlap model inference with tool execution. Many tool calls leave the machine entirely. The diagram is a skeleton, not a description of every agent.

Independent research supports the CPU’s role, with caveats. A study from Georgia Tech and Intel profiled several agent frameworks on an Intel Emerald Rapids host with an NVIDIA B200 GPU. It found that “tool processing on CPUs can take up to 90.6% of the total latency.” The top figure came from retrieval in a RAG pipeline. In the SWE-agent coding workloads, bash and Python execution took between 43.8% and 78.7% of latency, depending on the benchmark. The authors also found throughput saturating on CPU factors, such as core over-subscription, cache coherence and synchronisation, at larger batch sizes.

Where a faster CPU for AI agents helps, and where it doesn’t

The useful question is which resource each step is waiting on. A faster core only shortens steps that are bound by that core.

Step is bound by	Typical examples	What helps	What a faster CPU core does
CPU	Compiling, running tests, parsing, local search, orchestration logic	Faster cores, more memory bandwidth, less contention	Shortens the step directly
GPU	Model inference, time to first token, long-context prefill	GPU capacity, batching, caching, smaller models	Little, beyond feeding the GPU
Network or external service	Web search, third-party APIs, remote databases	Caching, parallel calls, closer endpoints	Almost nothing
Rate limit	Provider request or token caps	Backoff, queueing, higher tiers, fewer calls	Nothing; it may hit the limit sooner

An illustrative coding agent

Amdahl’s law sets the ceiling. Take an illustrative coding agent, with assumed numbers rather than measured ones. Each of 12 steps spends 3 seconds waiting for remote model inference and 1.5 seconds compiling and testing on the local CPU, for a 54-second task. A core that is 20% faster cuts each CPU step to 1.25 seconds and the task to 51 seconds. That’s a 6% end-to-end gain from a 20% faster chip.

Now change one assumption. Under load, contention for memory bandwidth stretches each 1.5-second CPU step to 3 seconds. The task now takes 72 seconds, a third longer than on an idle machine. In this case, the hardware’s behaviour under concurrency matters far more than its single-thread speed. That is roughly the situation Modal describes.

Tail latency adds a second reason to care. In “The Tail at Scale”, Jeffrey Dean and Luiz André Barroso showed how rare slow responses multiply when a request depends on many components. Their example was a server that answers in 10 ms but takes a second at the 99th percentile. If a request has to wait on 100 such servers, 63% of requests take more than a second. An agent that runs a dozen dependent steps has the same exposure in sequence: a slow p99 on any step becomes the user’s wait.

Reading Modal’s “up to 2x” result

Modal deserves credit for publishing the method and the code. Its Agentless Agent Bench replays oracle solutions from the Terminal-Bench dataset in prewarmed Docker containers, with network calls stubbed out. Throughput is defined as verified task completions per 60 seconds, and the test raises the number of concurrent containers until the machine saturates. That isolates host CPU behaviour cleanly. It also means the benchmark contains no model inference and no external I/O, which a real agent always has.

What the published material doesn’t settle matters for anyone thinking of buying:

• Concurrency: Modal doesn’t say at which container count the 2x occurs. “Up to” means the gap is smaller elsewhere.
• Equivalence: Modal calls the Intel Granite Rapids system “equivalent”, but it had 128 physical cores to Vera’s 88. Memory configuration and SKUs aren’t given.
• Cost and power: nothing suggests the systems were matched on price, power or rack space. Throughput per dollar or per watt could differ in either direction.
• Latency: Modal says task latency under load was “equal to, or sometimes even lower than” on x86, but publishes no p95 or p99 figures.
• Sample size: Modal ran a single node, early in testing. No third party has reproduced the result.

None of that makes the result wrong. It makes it a strong hint about one kind of workload: many concurrent, CPU-heavy coding sandboxes on a single server. A purchasing decision would need the same benchmark run at your concurrency, on cost-matched systems, with tail latency reported and with your real tools, including their network waits.

Memory bandwidth, scheduling and the cost of concurrent agents

Modal’s explanation is that Vera’s memory bandwidth “allows for significantly more container workloads to pack per core, without CPU throttling.” That fits the physics. Compiling and test runs are often memory-intensive. When hundreds run at once, cores spend time stalled on memory rather than executing instructions, and adding cores past that point adds waiting, not work. NVIDIA claims Vera has twice the memory bandwidth of x86 CPUs with DDR5, and three times the bandwidth per core.

Scheduling compounds the problem. Hundreds of mostly idle agent sessions, each waking briefly to run a tool, generate context switches and run-queue pressure. The operating system must decide which waiting process gets a core. The Georgia Tech and Intel study found throughput saturating from core over-subscription in two frameworks at a batch size of 128. Its proposed batching and scheduling changes, which need no new hardware, cut P50 latency by up to 1.7x in some configurations. Software fixes come before silicon.

For the business, the unit that matters is cost per successfully completed task at a given latency target. A platform hosting agents sells sessions, not cores. If a server can carry more concurrent sandboxes before p99 latency breaks the service-level objective, each task costs less, whatever the per-core score. That’s the economic case for agent-tuned CPUs, and it depends on concurrency behaviour as much as on clock speed. Our analysis of when adding another agent stops paying for itself applies the same per-task arithmetic to model tokens.

Deciding whether a faster CPU for AI agents is worth paying for

The decision gets easier if you work through four questions in order.

1. What share of task time is CPU-bound? If tool execution and orchestration on your hosts take less than a fifth of end-to-end time, even a much faster CPU will move task latency by a few per cent. Spend on inference capacity, caching or parallel tool calls first.
2. Does CPU time grow with concurrency? Compare a step’s duration on an idle host and at production load. If it stretches sharply, the constraint is contention, which points to memory bandwidth, cache size, core count or scheduling, not raw single-thread speed.
3. Is something else capping throughput? Provider rate limits and slow external APIs put a ceiling on completed tasks that no host hardware can lift. Our reference on AI API rate limits in 2026 sets out where those ceilings sit.
4. What does a successful task cost at your SLO? Run candidate platforms at target concurrency and divide total cost by tasks completed within the latency target. A chip that wins on throughput but loses on tail latency may not win here.

For a hosted platform running thousands of untrusted coding sandboxes, the answers may well point to a high-bandwidth, agent-tuned CPU. For an enterprise agent that mostly calls SaaS APIs and waits on a remote model, they usually won’t.

What to measure before choosing a CPU for AI agents

Start small. Three measurements answer most of the question:

• A per-task time breakdown: orchestration, model inference (including time to first token), each tool call, and waits on external services.
• End-to-end latency at p50, p95 and p99, both idle and at target concurrency.
• Completed successful tasks per second, along with failure, retry and rate-limit counts.

If the breakdown shows meaningful CPU time, add:

• CPU utilisation, run-queue length and time spent waiting, to separate busy cores from contended ones;
• memory bandwidth and cache misses, where the platform exposes them;
• concurrent sessions and active work per core, to find the packing limit;
• power and rack constraints, if density is the reason for changing platform.

Then test candidates with your own workload, not a vendor benchmark, using the same metric: cost per successful task at the latency you’ve promised users.

Conclusion

The evidence supports a narrower claim than the headlines. Agent platforms do put real, serial, latency-sensitive work on CPUs, and vendors are now designing and marketing processors for it. But the one public customer benchmark points mainly at memory bandwidth and behaviour under concurrency. Single-thread speed played a smaller part. And it comes from a synthetic test on a single node, with network calls removed.

The defensible takeaway is that the right CPU for AI agents depends on where your agents spend their time. Measure that first. If the answer is CPU-bound tool work under heavy concurrency, agent-tuned chips like Vera and the high-frequency Helion blades are worth testing. If the answer is model inference or someone else’s API, they aren’t where your next dollar should go.

FAQ

Why does CPU performance matter for AI agents?

Agents alternate between model calls and tool work, such as compiling code, running tests or searching files. Much of that tool work runs on the CPU, and each step waits for the previous one. CPU-bound steps add directly to task latency.

Are AI agents more CPU-intensive than traditional inference?

Often, yes. A single chat request is mostly GPU work. An agent task adds tool execution, orchestration and sandbox overhead between model calls. How much depends on the tools: local code execution is CPU-heavy, while calls to external APIs mostly wait on the network.

Does a faster CPU reduce LLM inference latency?

Not much. Model inference runs on GPUs or other accelerators. A faster CPU can help with tokenisation, batching and data movement around inference, but the main gains for agents come from shortening the tool and orchestration steps between model calls.

What does Modal’s reported 2x throughput figure establish?

That on one Vera server, Modal’s synthetic coding benchmark completed up to twice as many tasks as on an Intel Granite Rapids server under heavy concurrency. It doesn’t establish cost or power efficiency, tail latency, or results for agents that spend time on network calls. It hasn’t been independently reproduced.

When should a team prioritise CPU upgrades over GPUs?

When tracing shows that CPU-side tool execution or orchestration takes a large share of task time, and that share grows with concurrency. If most time is spent on model inference, external APIs or rate-limit waits, GPU capacity or software changes will do more.

How should engineers benchmark CPUs for agent workloads?

Replay real agent tasks, including their tool calls, at production concurrency on cost-matched systems. Record p50, p95 and p99 latency, completed tasks per second, failures, and CPU and memory contention. Compare platforms on cost per successful task at a fixed latency target.

---

Keep reading

Here are the latest posts from the blog.