Why Agentic AI Is Causing a CPU Shortage at AWS and Beyond

Amazon Web Services reportedly issued a mandate to its engineers earlier this year: conserve CPU cycles at all costs. The culprit is agentic AI, where models spawn thousands or millions of sub-agents that make API calls, parse results, run code, and check safety rules, nearly all on CPUs. The GPU got all the headlines during the AI boom, but analysts and researchers at Intel, AMD, and Georgia Tech now say the CPU is quietly becoming the bottleneck that limits how fast and how far AI agents can scale.

What happened

Data point	Detail
AWS mandate	Engineers told to conserve CPU cycles at all costs, reportedly due to exploding wait times for CPU server capacity
AMD finding	7 of 8 stages in a realistic agentic AI pipeline run entirely on the CPU
Intel / Georgia Tech finding	Scheduling optimizations can cut end-to-end agentic latency by up to 1.8x under sustained load
Georgia Tech paper finding	More CPU cores reduce time-to-first-token latency by 1.5x to 7x at longer sequence lengths
OpenAI agent incident	A single OpenAI model fired off as many as 300 actions per hour during an inadvertent Hugging Face hack
Models tested	Alibaba Qwen 3-30B and Meta Llama 3.1-70B

For most of the AI boom, CPUs were considered a sideshow. GPUs handle the parallel matrix math that powers large language model (LLM) inference, so investment and attention flowed there. CPUs lack the same degree of parallelization, making them a poor fit for running models directly.

But agentic AI changes the equation. Matt Kimball, vice president and principal datacenter analyst at Moor Insights and Strategy, says 2026 has brought a sharp spike in CPU demand tied directly to agents. “It’s one thing to have this agentic workload and let’s say it spawns 100 agents,” he says. “If I’m going to roll this out across my enterprise, those 100 become tens of thousands, hundreds of thousands, or millions of agents.”

Why do AI agents need so many CPU cycles?

The key concept here is “tool use”: an LLM’s ability to browse the web, open files, call APIs, and run code to complete a task. While the LLM’s core inference still runs on a GPU, everything around it does not.

Souvik Kundu, senior staff research scientist at Intel, describes the CPU’s role in an agentic task: it parses the model’s output, decides which tool to call, executes the API call or code, collects the result, and feeds it back to the model. Madhu Rangarajan, vice president of compute and enterprise AI at AMD, backs this up, stating that in AMD’s own testing, seven of the eight stages in a realistic agentic pipeline run entirely on the CPU.

Kundu co-authored a paper with researchers from the Georgia Institute of Technology showing that the CPU and GPU are often idle at alternating moments: the CPU sits idle during GPU inference, and the GPU sits idle during CPU tool calls. Their proposed scheduling optimizations close that gap and can reduce end-to-end latency by up to 1.8x under sustained load.

Safety guardrails add to the load

On top of tool calls, safety and policy checks on agent actions often involve rule-based syntax inspection and small models under one billion parameters. Kundu says these checks tend to stay on the CPU because their small size and the need to minimize latency make GPU offloading impractical. As agent deployments grow and safety requirements tighten, this adds another layer of CPU pressure.

Tokenization is the hidden bottleneck

Euijun Chung, a PhD student at Georgia Tech, co-authored a complementary paper on a problem most people overlook: tokenization. Tokenization converts text into the integer token IDs a model can process. Unlike standard LLM inference, tokenization is sequential string manipulation, not massively parallel math, so it sits naturally on the CPU.

In an agentic workflow, the problem compounds. “If you have an ongoing sequence of, say, 100,000 tokens, and you have a tool result of 1,000 tokens, the tokenizer will have to tokenize the whole sequence again. And you have to do tokenization at every agentic tool call,” Chung explains. That means both the frequency and the volume of tokenization grow with every tool call the agent makes.

When servers have too few CPU cores, they fall behind dispatching work to GPUs, causing the GPUs themselves to stall while waiting for instructions. Chung’s paper found that adding more CPU cores reduces time-to-first-token latency (the time before the model produces its first word) by 1.5x to 7x at longer sequence lengths.

Why it matters

Businesses deploying agentic AI at scale, whether for coding assistants, customer support, or back-office automation, are about to hit a ceiling that has nothing to do with GPU availability. The CPU layer underneath their agents will determine how many agents they can run concurrently and how quickly each one responds.

The OpenAI agent incident, where a model inadvertently fired off up to 300 actions per hour against Hugging Face, is a small preview of what happens when agents multiply at machine speed. At enterprise scale, that translates directly to CPU exhaustion on cloud infrastructure.

Cloud providers pricing compute by GPU hours may not be surfacing CPU costs clearly. If you are running agentic workflows and your latency is creeping up, the GPU may not be the bottleneck at all. For teams building on AI integration pipelines, this is worth profiling before assuming more GPU capacity is the answer.

Our take

The AWS story is a useful reality check. The industry spent three years obsessing over GPU supply, and now a different, older chip is creating the queue. That is not a knock on the AI build-out: it is what happens when a technology scales faster than the full infrastructure stack can be audited.

For most businesses, this plays out as a cloud cost and latency issue, not a chip-buying decision. If you are running Make or n8n workflow automations that call AI agents, or building customer-facing agent tools, watch your per-call latency trends. Tokenization and tool-call overhead will show up in your response times before your GPU utilization alarm fires.

The scheduling optimizations described in the Intel and Georgia Tech research are promising, but as the researchers themselves note, agentic systems generate work at machine speed. A 1.8x latency improvement is real, but it chases a target that keeps moving as agent complexity grows. Expect this to be an active area of engineering for the next 18 months.

What to do about it

1. Profile your agentic pipelines to separate GPU inference time from CPU tool-call and tokenization time. Do not assume the GPU is the constraint.
2. If you are on a cloud provider, check whether your instance type has enough CPU cores relative to GPU capacity for agentic workloads, not just for traditional ML inference jobs.
3. Minimize tool-call frequency where possible. Each call triggers a full retokenization of the growing context window, so batching or caching results reduces the multiplier effect.
4. Monitor time-to-first-token latency as a leading indicator of CPU saturation, especially as your agent context windows lengthen over a session.
5. Follow the scheduling research coming out of Intel and Georgia Tech. Practical implementations of their 1.8x optimization could ship in major inference frameworks within the year.

The practical takeaway: before you budget for more GPUs to speed up your AI agents, check whether your CPUs are already the queue.

Source: IEEE Spectrum · AI