Inference chip makers pivot to retrofit-friendly designs as data center construction lags
Chipmakers are racing to fill the gap between surging inference demand and delayed data center buildouts by developing air-cooled systems that can operate in existing telecom and colocation facilities without major upgrades.

The buildout of new AI data centers cannot keep pace with rising inference workloads, as power constraints, capital availability and regulatory obstacles extend timelines for fresh capacity. This shortfall is pushing semiconductor vendors to seek deployment space within facilities already in operation.
Existing telecom and colocation infrastructure offers both floor space and electrical capacity, yet most cannot accommodate the power-dense, liquid-cooled GPU racks prevalent in modern deployments without substantial infrastructure investment. This reality is driving interest in air-cooled inference solutions, according to Anton McGonnell, vice president of core product at SambaNova Systems Inc.
We as an industry are just encumbered by the velocity of data center buildouts, and that's been slowed down by just the physics of how long this takes, by the access to capital, and more and more by legislative hurdles — particularly in the U.S.
Anton McGonnell, SambaNova Systems
McGonnell shared these insights during an exclusive broadcast at SC26, speaking with theCUBE Research's John Furrier and Gemma Allen on theCUBE, SiliconANGLE Media's livestreaming platform.
Reviving brownfield capacity for AI data centers
SambaNova's latest processor, the fifth-generation SN50, specializes in the decode phase of inference, where the system produces output tokens and data movement becomes the limiting factor. The company designed its racks to operate with air cooling and maintain a footprint compatible with existing facilities. Conventional GPU racks, by contrast, often require structural reinforcement and consume 120 to 250 kilowatts per rack, McGonnell noted.
On all of this stuff — the power and the cooling gear — there's 18- to 24-month lead times to get this equipment into the data centers. Never mind all of the construction costs.
Anton McGonnell, SambaNova Systems
SambaNova, valued at $11 billion by investors in July, collaborates with partners to handle prefill, the initial compute-intensive stage of inference. At SC26, the company is showcasing Intel's D-Series GPUs for prefill alongside its SambaRack SN50 for decode in an air-cooled heterogeneous configuration.
https://www.youtube.com/embed/fP4zfctTS7c?start=26&feature=oembed
They have developed a chip that has much, much better cost performance for the prefill phase of inference. They've sacrificed HBM, which isn't needed for prefill because it's a compute-bound problem.
Anton McGonnell, SambaNova Systems


