Microsoft Pursues Composable Memory to Ease AI Data Center Bottlenecks

Memory has emerged as the most constrained asset in AI data centers, prompting hyperscalers to fundamentally rethink resource allocation. Composable memory—a pooling approach that distributes capacity on demand across multiple servers—is gaining traction as a practical solution to this shortage.

The challenge manifests in AI facilities where graphics processing units frequently remain idle while waiting for memory and storage access. Rather than confining memory within individual server boundaries, Microsoft Corp.'s research team is investigating system-level approaches to transform memory into a shared, dynamically distributed resource across Azure infrastructure, according to Samir Rajadnya, principal architect at Microsoft.

One of the main things that is happening or will happen in the next several years is composable architecture. The original idea was everything is locked behind a CPU or a GPU. Memory was a servant to the CPU or GPU. Now memory is looked at as another very important resource.

Samir Rajadnya, principal architect at Microsoft

Rajadnya discussed these architectural shifts with theCUBE Research's John Furrier during the AI Data Pipeline Forum, examining how composable memory, enhanced interconnects, and workload-optimized design enable hyperscalers and enterprises to maximize utilization of limited memory resources.

Composable memory meets the memory wall

Demand for AI capabilities has outpaced memory supply, fundamentally reshaping inference economics. For cloud operators running at Azure's scale, unutilized capacity represents significant expense. Azure serves enterprises including Delta Air Lines Inc. and Walmart Inc., making reliability and security non-negotiable requirements for any new architecture, Rajadnya emphasized.

As you see in the media right now, there is a shortage of memory. So we have to use that memory more efficiently. And as a part of that, we have to make sure memory goes where the workload demands it. If we strand memory behind a workload who is not using it, we are simply wasting a valuable resource at this time.

Samir Rajadnya

Inference workloads now represent a dominant share of industry compute, yet they encounter dual constraints simultaneously. When asked about an ideal scenario without technical limitations, Rajadnya envisioned memory capable of spanning long distances with minimal power consumption and latency penalties.

AI inference is limited by power and then it is limited by memory, correct? So if I had a wish list, I would go for a memory that I can connect with the lowest possible power over a large distance and an infinite amount of memory.

Samir Rajadnya

Enterprises encounter a more constrained version of this challenge. Cloud providers must accommodate diverse workload types, whereas most organizations operate a limited set of well-understood applications, simplifying their architectural decisions. Rajadnya recommends identifying the two or three most critical workloads and engineering systems specifically for those use cases.

It's all the game of what we call vertical integration right now. Workload, the system, software, silicon, all in one. And you have to do that even at the enterprise level.

Samir Rajadnya

Scaling composable memory depends critically on the interconnections linking memory to processors and specialized accelerators—collectively termed XPU by Rajadnya. The physical distance memory travels from its source determines whether copper-based connections suffice or optical links become necessary. Microsoft has consistently contributed designs to the Open Compute Project and is presenting its AI infrastructure innovations at this year's Open Compute Project Global Summit.

There is a lot of great innovation happening on the memory component side of things. There is a lot of great innovation happening on the XPU CPU side of things, but how do you connect those two things together with the memory, right? So that is where the interconnects are very important. Some of the things that are happening in the optical space, they are going to be important. And these things are going to help us to build that composable memory.

Samir Rajadnya

https://www.youtube.com/embed/JpQ5m4csfB8?feature=oembed

Component-level breakthroughs yield limited value unless the broader ecosystem advances in parallel, Rajadnya cautioned. Progress in memory technology and processor design requires corresponding innovation in the systems that connect these elements.