Storage becomes the real constraint as enterprises max out GPU capacity

Enterprise spending on artificial intelligence has centered on a single priority for the past two years: acquiring compute resources. Companies competed aggressively for GPU capacity, struck major infrastructure agreements and deployed machine learning systems rapidly. Yet many organizations found that the constraint they believed they were addressing was far from the only one limiting their operations. Data storage has emerged as the actual limiting factor, starving the accelerators that enterprises worked so hard to obtain.

In a recent AppDevANGLE podcast episode, Ronen Schwartz, vice president of products for Google Cloud Storage at Google LLC, discussed how storage has become the critical limitation in enterprise AI infrastructure and what a properly designed architecture for AI operations should include. The discussion addresses a concrete, quantifiable issue that remains largely overlooked in most enterprise AI planning.

Storage just got promoted. Instead of being at the backend as an enabler, it is moving into the front as a critical part of accelerator execution.

Ronen Schwartz

The true cost of underinvesting in data infrastructure

The financial impact is straightforward. Accelerators represent an organization's most costly infrastructure investment. Every moment they sit idle waiting for data constitutes real financial loss—not a theoretical efficiency concern, but concrete, measurable expenditure. However, when teams finally secure access to their TPUs or GPUs, the infrastructure supporting data delivery frequently has not received comparable focus as the processing layer.

I speak to many customers, and what I'm hearing is the team spent a long time fighting to get their TPUs, their GPUs, to get their hands on accelerators. And once they got them, they're actually not always able to get the output they needed. Very often it actually goes back to not enough thought put into the data delivery path.

Ronen Schwartz

This situation is not uncommon. Schwartz outlined a recurring pattern in which enterprises recognize the storage limitation only after encountering it directly—when GPU utilization falls below expectations and training cycles extend beyond timelines. At that point, the expense of inadequate storage planning is already baked into project costs.

Three distinct AI workloads, three distinct storage requirements

A significant takeaway from the discussion is that artificial intelligence encompasses multiple distinct workload types, and treating them uniformly leads to gaps in storage design. Schwartz identified training, inference and agentic operations as having fundamentally different operational patterns with unique data movement requirements that demand specialized approaches.

Training workloads require massive sequential data reads, but checkpoint preservation represents an equally critical challenge. Approximately every 15 minutes, a training operation must halt and store a multi-terabyte checkpoint to safeguard progress against potential system failure. According to Schwartz, slow storage at this stage paralyzes the entire GPU cluster. Google's Rapid object storage technology achieves checkpoint writes more than three times faster and restoration more than five times faster compared to conventional object storage—a distinction that becomes significant at enterprise scale. Torin, an autonomous vehicle manufacturer, exemplifies this requirement: the company processes millions of sensor data points into storage to develop vehicle models, where checkpoint speed is not a peripheral concern but a fundamental operational necessity.

Inference presents a distinct challenge centered on real-time, delay-sensitive operations where response latency directly influences user satisfaction and token economics. Agentic systems introduce yet another layer of requirements: collaborative data spaces, state management across multiple agents, rapid data retrieval and the capacity to manage both temporary and persistent agent processes that may enter dormancy and resume operation as needed.

AI is not one workload. Really, the movement of data looks completely different. It's like different heartbeats depending on where you are in the lifecycle.

Ronen Schwartz

Hyperscale infrastructure as a competitive advantage

Google's competitive positioning rests on infrastructure credibility developed through years of operating large-scale AI systems internally before making those same capabilities available to customers. Schwartz cited specific performance metrics: 15 terabytes per second throughput on Rapid object storage and 10 terabytes per second on Google Cloud managed clusters, which he characterized as independently verified industry-leading performance.

The underlying architecture emphasizes zonal colocation, positioning data physically adjacent to GPU and TPU clusters to reduce latency between storage and accelerators. Schwartz characterized this as an early-market capability requiring fundamental changes to conventional storage assumptions, where multi-zone or multi-region redundancy was standard practice. For AI operations, proximity to the accelerator takes precedence. Google's Rapid Bucket technology streamlines the storage processing pipeline, allowing a single bucket to achieve more than 15 terabytes per second throughput while supporting 20 million queries per second with sub-millisecond response times.

It is because it is built on the Google infrastructure, which has been used internally longer than anyone else to support accelerators and their performance. But it is also because Google is working with nine out of the ten world-leading AI frontier model companies.

Ronen Schwartz

A storage architecture that demonstrably enhances GPU utilization enables faster development cycles through reduced idle compute time and decreases the effective cost per training operation.

Preparing unstructured data for agent systems

The agentic component of the discussion highlighted an often-overlooked aspect of enterprise AI infrastructure readiness. Agent systems require more than web access and database connectivity; they need large-scale access to unstructured information including photographs, video, documents and PDFs—the types of content that constitute the majority of most enterprises' actual data holdings. Without mechanisms to make this content semantically queryable, agents operate with a constrained and frequently unrepresentative understanding of organizational information.

Google's approach centers on what Schwartz termed the Smart Storage Initiative, built around a newly introduced Object Enrichment feature. When documents or images arrive in a storage bucket, Google AI models automatically analyze the content and append structured metadata to the object—accomplishing this in seconds and at scale. The outcome is that previously unstructured information becomes accessible to agents through queries without requiring manual processing or separate data pipelines.

An agent can now interact with what used to be called unstructured data because there is actually a very good understanding of the content.

Ronen Schwartz

The storage layer integrates with Google's broader agentic data infrastructure and borderless lakehouse framework, enabling agents to reach information housed outside Google's systems as well. For large-scale agent deployments, Schwartz described dedicated Filestore agent volumes connected directly into Google Kubernetes Engine and the agent sandbox setting, engineered specifically to manage the expense, latency and data-sharing demands of operating thousands or millions of agents concurrently.

The architectural shift ahead

The broader industry transformation is structural rather than incremental. Enterprise AI infrastructure has been assembled sequentially (compute first, then software frameworks, then data) with storage frequently treated as a purchasing decision rather than a performance factor. This sequencing approach is proving expensive. As GPU utilization becomes a measurable performance target and AI operations budgets face increased pressure, storage is receiving reassessment that has not occurred in years.

Google's approach demonstrates that operational experience at hyperscale in the storage domain translates into tangible enterprise value—not merely as marketing messaging, but as concrete improvements in checkpoint performance, throughput, latency and agent responsiveness. Organizations that have operated frontier model training and inference at scale have been compelled to address these challenges; enterprise teams are encountering them for the first time now.

For infrastructure engineers, machine learning operations teams and enterprise technology leaders, the takeaway is direct: storage architecture choices made during initial AI system deployment will either constrain or unlock performance for all subsequent operations. Addressing these decisions after performance limitations surface proves substantially more costly than incorporating them into the initial design.

The industry is moving toward a future where storage occupies a primary position in AI infrastructure discussions, not a secondary one. Enterprises that recognize this transition early will realize measurably greater returns from the accelerator investments they have already committed.

https://www.youtube.com/embed/w14syk7L17c?feature=oembed