AI Business

Power and cooling emerge as the real bottleneck for enterprise AI infrastructure

As companies scale AI deployments to data center levels, procuring GPUs has become trivial compared to the challenge of powering and cooling them. Infrastructure providers say the shift to megawatt-scale systems is forcing enterprises to rethink how they build and operate AI infrastructure.

·3 min read
Enterprise AI hits a wall of power and cooling, not GPU supply
Enterprise AI hits a wall of power and cooling, not GPU supply

Obtaining graphics processing units has become the straightforward part of building AI infrastructure at scale. The genuine obstacles enterprises now face involve supplying adequate power, managing thermal loads and ensuring sufficient network capacity—factors that determine whether expensive systems actually generate value or remain underutilized.

These infrastructure constraints dominated discussions at SC26, where commercial AI infrastructure increasingly competes for attention alongside traditional scientific computing. Rack power consumption has shifted dramatically from historical maximums around 10 kilowatts toward megawatt-level densities, a transformation that requires enterprises to examine infrastructure layers both above and below conventional compute, storage and networking stacks, according to Jack Hogan, vice president of advanced growth technologies at SHI International Corp.

When we look at the full stack, we recognize that customers are getting tripped up. They can go get the GPUs, but they can't power them. They can't cool them. These are not cheap systems either. Procuring the infrastructure is just the start.

Jack Hogan, SHI International Corp

Edge inference and the megawatt as AI's new yardstick

The financial model underlying AI infrastructure deployment is undergoing fundamental change. Enterprises previously evaluated deployments using individual rack units or complete racks as their measurement basis, but contemporary data centers increasingly function as unified systems with shared memory resources, where every GPU must remain continuously utilized to justify its expense. This evolution has reshaped how SHI evaluates deployment success.

Now we're talking about row and now data center scale. … We're measuring that now in megawatts. So the megawatt now starts to become a measurement that people start to grasp and understand. How many flops can you get through that megawatt? How fast can you process your tokens?

Jack Hogan, SHI International Corp

SHI manages the entire lifecycle of these systems, leveraging more than 37 years of industry experience. The organization fabricates customized racks within integration facilities distributed globally and conducts pre-deployment assessments to enable immediate operation upon arrival. In one recent engagement in New Jersey, SHI is both hosting and operating the customer's infrastructure within its own data center facilities. Geographic positioning carries particular significance for edge deployments, where applications like video processing create data streams that require local processing rather than remote transmission.

You can't backhaul that. The networks are not necessarily the right way to move that. We're creating the scenario of both the mega hyperscaler as well as the edge inference and be able to tie those two things together.

Jack Hogan, SHI International Corp

High GPU utilization does not automatically translate to productive output. SHI initiates every engagement by establishing what customers aim to accomplish, then determines the appropriate model architecture—whether that involves cutting-edge large language models or compact models suitable for edge devices. Hardware selection follows the same principle, with GPUs, advanced CPUs, inference-optimized processors and neural processing units all evaluated within the company's AI and Cyber Lab.

https://www.youtube.com/embed/xB6BBID6qX4?feature=oembed

It's not about the cost of the token or reducing the number of tokens or even finding cheaper tokens. It's about optimizing and ensuring that those tokens are turning into business value.

Jack Hogan, SHI International Corp