Count the Work Done. Not Just the Cores.

By: Karin Eibschitz Segal

The debate about CPUs for Agentic AI has become too narrow. Much of the industry conversation centers on single-thread performance, core density and rack throughput. All three matter. But none alone answers the question customers ultimately have: How much useful agent work can my infrastructure reliably complete within the latency, cost and power limits my business requires?

Core count tells customers how much compute capacity goes into a system. Agentic AI requires a measure of how effectively the whole system converts that capacity into completed work.

Concurrency changes the answer

A single agent loop makes performance look straightforward: finish each step faster and the workflow moves faster. Production systems run many agents at once, competing for CPU, memory, I/O and accelerator resources. Under that load, the bottleneck changes.

In a recent public analysis using 739 anonymized Claude Code conversations—coding-agent workflows involving multi-turn reasoning, code generation and tool use—CPU-side processing and wait time rose from less than 1% of total latency for a single request to as much as 15% at 32 concurrent requests. Most of that increase came from scheduling and queueing—not computation.

A faster core can execute work faster. It cannot make a queue disappear.

Waiting also reaches the most expensive part of the system. For code-execution agents, the same research estimates that GPUs perform useful work during only 50% to 60% of wall-clock time. When an agent waits for a database, retrieval system, or sandbox, the GPU waits too. Amdahl's Law still applies:

“The most expensive accelerator is the one waiting for the rest of the system.”

Measure delivered agents—not theoretical capacity

Intel proposes a more useful measure: delivered agents per rack—the agent workflows a rack can reliably complete at the required throughput, quality and latency, within a defined power and cost envelope.

One way to understand delivered capacity is:

Delivered Agents per Rack = Theoretical Agents per Rack × Routing Efficiency × Resource Balance × Offload Efficiency

Theoretical capacity reflects the rack’s available compute, memory and bandwidth. Routing efficiency reflects how well work is placed. Resource balance captures losses from bottlenecks and queueing. Offload efficiency reflects the capacity recovered by moving infrastructure tasks away from general-purpose cores. The accompanying white paper defines these factors and the methodology in greater detail.

The real limit is where adding agents pushes latency beyond the customer’s target—not the top of a theoretical throughput chart. An illustrative example is shown in Figure 1.

Figure 1. Capacity is the point where the latency curve crosses the target. Headroom-aware routing, offload and state tiering moves the knee right (illustrative).

Intel is built for the whole workflow

Intel is not choosing between fast cores and more cores. We are building the system that converts both into completed work.

Keep state available. Bandwidth keeps an agent moving; capacity keeps it from starting over. Intel Xeon 6 Flat Memory Mode combines CXL-attached memory and native DDR5 in one address space. In a joint Intel and SAP evaluation, it delivered 96% of all-DDR5 performance at roughly 25% lower memory total cost of ownership—a practical path to keeping more sessions warm at lower cost.

Return capacity to the system. Intel QuickAssist Technology, Data Streaming Accelerator and In-Memory Analytics Accelerator (Intel® IAA) offload infrastructure work that would otherwise consume general-purpose cores. Published work using Intel IAA for snapshot compression delivered approximately 1.2x faster snapshot completions. Offload is not a feature checklist; it is capacity returned to the customer.

Match the resource to the job. Orchestration, tool execution, data services and state movement have different needs. The Xeon portfolio spans high per-thread P-core performance and dense E-core throughput on a common architecture, allowing the system to place each phase on the right execution profile.

Do not strand the accelerator. Of course, enterprises will need agentic-optimized GPUs every bit as much as CPUs and Intel is preparing its Crescent Island GPU to fulfill exactly that role. But if the GPU kept waiting, it is still just an expensive piece of silicon burning rack power. That is why this discussion about CPU still matters. Xeon is the traffic controller for the agentic system, keeping sessions moving, tools executing, data flowing and accelerators fed."

A more useful scoreboard

Every agentic AI performance claim—including Intel’s—should state the concurrency, the latency target and the losses between theoretical and delivered capacity. Without those, a result describes potential, not production capacity.

Intel intends to report agents per rack at a stated p99 latency target on real agentic traces, along with routing efficiency, gains from fixed-function offload, host efficiency in GPU-attached systems and agents per watt under a defined rack power constraint.

Single-thread performance matters. Core density matters. But the system that wins will be the one that keeps every thread fed, every session warm and every accelerator productive.

The industry has spent enough time counting what goes into the rack. It is time to measure what comes out.

For more information, the Intel whitepaper, How Many Agents Will Your Rack Actually Run explores this idea in greater depth, laying out the technical framework, supporting data and practical considerations behind “delivered agents per rack”—and how organizations can apply it to evaluate real-world system performance.