AI Inference Is the New Workload: Memory and Storage Are the Bottleneck
AI inference has shifted the bottleneck from raw compute to memory, storage, and data movement. Here's what that means for businesses building AI infrastructure.
AI systems have moved from occasional training runs to continuous, real-time inference at scale. That shift breaks the old infrastructure playbook. According to Jim McGregor, founder and principal analyst at Tirias Research, AI is not one workload but billions of different ones, each with different demands. The bottleneck has moved away from raw processor power and onto how quickly data can be moved, cached, and delivered. For business operators, that means memory, storage, and networking now deserve as much budget attention as compute.
What happened
| Topic | Key point |
|---|---|
| Primary bottleneck | Data movement, not processor speed |
| Workload nature | Billions of distinct inference workloads, not a single job |
| Critical layers | Compute, memory, storage, and networking must be architected together |
| Key technique affected | Retrieval-augmented generation (RAG) requires constant scanning of large databases in real time |
| Business consequence | Latency affects safety, trust, and reputation in healthcare, finance, and customer AI |
MIT Technology Review published a detailed look at how AI inference, meaning the live, moment-to-moment use of a trained model to answer questions or take actions, has fundamentally changed what enterprise infrastructure must do. The core argument, built around analysis from Jim McGregor at Tirias Research, is that the old separation of compute, memory, storage, and networking into independent procurement decisions no longer works.
McGregor puts it plainly: “We tend to think of AI as a single workload, and it’s not. It’s thousands, it’s millions, it’s billions of different workloads.” Each of those workloads can have different latency tolerances, data access patterns, and throughput requirements, which means a one-size-fits-all hardware strategy is almost guaranteed to create bottlenecks somewhere.
Why data movement is now the real constraint
Traditional enterprise IT was built around relatively predictable, bursty workloads. Inference changes that. A model serving customer queries or analyzing sensor data is pulling information continuously, not in occasional batches. Techniques like RAG, where a model searches a live database to improve the accuracy of its answers, amplify this problem. The system must scan massive datasets with every query, which demands immediate data access rather than just fast processing once data arrives.
According to McGregor, this shifts the strategic question from “how fast is our GPU cluster?” to “how efficiently can we move, cache, and deliver data across the whole architecture?” Memory bandwidth (how fast data moves between storage and processor), caching strategies, and storage proximity to compute all become first-class concerns. Buying faster chips without addressing those layers just moves the bottleneck rather than removing it.
Why it matters
For businesses running or planning AI-powered services, this analysis has direct cost and performance implications. Overbuilding compute while under-investing in storage throughput or network bandwidth is a common and expensive mistake. The piece argues that the most effective AI infrastructure looks less like a collection of best-in-class individual parts and more like a balanced, co-designed system.
The stakes are not just technical. In robotics, financial services, healthcare, and customer-facing AI, a slow response is not just an inconvenience. It can undermine safety, erode trust, or trigger regulatory scrutiny. Latency becomes a reputation metric. Companies that treat infrastructure decisions as purely engineering choices, rather than business ones, are exposed to that risk.
There is also an efficiency angle. Inference workloads run continuously, so any wasted watt or unnecessary data hop compounds over time. Organizations that optimize performance per watt and reduce unnecessary data movement will carry structurally lower operating costs than those that do not.
Our take
This analysis is solid and mostly free of vendor spin, which is worth noting given the source is sponsored content. The central point, that you cannot optimize memory, storage, compute, and networking in isolation, is correct and often ignored in practice. We see it regularly: clients invest in a capable AI model but plug it into infrastructure that cannot feed it data fast enough, then wonder why response times are poor.
The workload-awareness argument is the most useful part. “You have to really have a detailed understanding of what those workloads are going to be,” McGregor says. That is harder than it sounds. Most businesses deploying AI right now are still figuring out which use cases will actually stick, which makes precise infrastructure sizing genuinely difficult. The practical answer is to start with a smaller, well-balanced system and scale horizontally rather than betting on a large, compute-heavy deployment up front.
If you are building or auditing AI tooling for your business and want to understand where integration bottlenecks actually sit, our AI integration work often starts exactly there. Separately, the workflow automation layer is frequently where data movement problems surface first. See how we approach that in our workflow automation service.
For more coverage of how AI infrastructure decisions are playing out across the industry, browse our AI news section.
What to do about it
- Map your actual inference workloads before buying hardware. List the queries, their frequency, and their latency requirements.
- Audit your current data pipeline: where does data sit, how far does it travel to reach your model, and what is the caching strategy?
- Benchmark memory bandwidth and storage throughput, not just GPU performance, when evaluating infrastructure options.
- Size for balanced throughput across all four layers: compute, memory, storage, and networking. A bottleneck in any one will cap the others.
- Plan for continuous load, not peak bursts. Inference is always on; infrastructure that is optimized for occasional spikes will underperform and overspend.
The organizations that will get the most from AI are not necessarily those with the largest clusters, but those with the clearest picture of their workloads and a system designed to serve them efficiently.
Frequently asked questions
What is the main bottleneck in AI inference systems?
According to analyst Jim McGregor of Tirias Research, the primary constraint is data movement rather than raw compute power. Systems must rapidly move, cache, and deliver data across memory, storage, and networking layers to serve continuous inference workloads.
Why can't you just buy faster GPUs to improve AI performance?
Faster processors alone shift the bottleneck rather than removing it. If memory bandwidth, storage throughput, or network capacity cannot keep pace, the GPU sits idle waiting for data. All four infrastructure layers need to be co-designed and balanced.
How does retrieval-augmented generation (RAG) affect infrastructure requirements?
RAG requires an AI system to scan large databases in real time with every query to improve accuracy. This places constant, high-volume demand on storage access and memory bandwidth, far beyond what traditional batch applications required.
How should businesses plan AI infrastructure procurement?
Start by mapping your specific inference workloads, including query frequency and latency requirements. Then size compute, memory, storage, and networking as a balanced system rather than optimizing each layer independently.
