AI infrastructure shifts from raw compute to systems thinking: 11 takeaways from CASPA's CEO Summit

Building AI infrastructure no longer hinges on individual chip performance. Hundreds of thousands of processors function as a single integrated machine, forcing the industry to rethink hardware as a complete ecosystem. This system spans everything from optical connections between racks through memory, storage and power delivery that keep GPUs operating at capacity.

Coming challenges will center on what surrounds the processor, not the processor itself. Optical technology is replacing copper for interconnection, while the memory bottleneck persists rather than disappearing. Power infrastructure designed for six- to ten-year lifecycles must now keep pace with chips refreshed annually. Capacity constraints are real: memory and optical laser supplies are committed years in advance.

A lot of the focus in AI has been very much on compute and on memory, but our view is actually AI is a data system. Ultimately, you need the entire ecosystem to work together and to be able to move in lockstep.

Irving Tan, CEO of Western Digital Corp.

Tan and other industry leaders shared their perspectives with theCUBE Research's John Furrier at The CEO Summit held in Santa Clara, California. The inaugural gathering coincided with the 35th anniversary of the Chinese American Semiconductor Professional Association. Discussions ranged across optical interconnects, inference cost structures, memory pooling, maskless lithography, physics-based computing, power resilience and AI-driven chip design.

11 key insights from the summit

1. Hard drives gain throughput as storage becomes the AI data layer

Intelligence emerges from data, and storage ensures that data persists, so organizations must distribute data across hot, warm and cold tiers spanning cloud, edge and on-device locations, according to Tan. Drawing on IDC projections, he noted that global data generation is forecast to reach more than 700 zettabytes within three years, tripling current levels. Western Digital anticipates storage demand will expand at a minimum compound annual rate of 25% through the next five years.

2. Connectivity becomes the choke point for AI data centers

Semtech Corp. manufactures the semiconductor components embedded in optical transceivers and copper cables that bind thousands of AI processors into one system. A single connection failure can force an entire job to restart, making connectivity stability essential to GPU uptime and revenue generation, noted Hong Hou, president and CEO of Semtech. Hyperscalers prioritize increased bandwidth paired with reduced power consumption, freeing up more of each data center's power allocation for actual computation.

3. Inference economics push chipmakers toward workload-aware racks

Agents now operate continuously, shifting the focus to performance per dollar per watt rather than raw teraflops as the defining metric of the inference phase, explained Sunghyun Park, co-founder and CEO of Rebellions Inc. The evolution beyond current rack-scale designs points toward racks optimized for specific workload profiles—whether latency-sensitive, throughput-demanding or power-constrained. Since model architectures shift every few months, Rebellions builds adaptable hardware and identifies the network as the emerging bottleneck as context windows expand.

4. Maskless e-beam lithography targets the shift from Moore's Law to advanced packaging

Moore's Law is approaching saturation, with advanced packaging—combining logic, memory and input/output components—emerging as the sector's new growth engine, according to David Lam, founder of Lam Research Corp. and CEO of Multibeam Corp. Multibeam has miniaturized an electron-beam column from 80 centimeters to pen-sized dimensions, enabling parallel column arrays to process wafers simultaneously. This maskless lithography approach circumvents the reticle constraint that currently requires large quantum and AI chips to be assembled from multiple pieces.

5. CXL memory pooling gains new life as the memory wall persists

The memory wall will likely remain unsolved for at least five years, stressed Greg Ernst, chief business officer of ScaleFlux Inc., who recently transitioned from Intel Corp. Even an 8 billion-parameter model serving multiple concurrent users demands more memory than high-bandwidth memory alone can supply. Compute Express Link, or CXL, enables multiple servers to access shared memory pools. Hyperscalers are reviving interest in the technology by repurposing older DDR4 modules within these pooled architectures.

6. In-memory computing takes aim at the memory supply crunch

New memory manufacturing facilities require three to four years to become operational, leaving inventory from major suppliers essentially exhausted through 2028, according to Faris Sbahi, co-founder and CEO of Normal Computing Corp. The company is shifting away from high-bandwidth memory toward mainstream LPDDR5 components. It also executes certain inference tasks directly within memory to reduce both expenses and power consumption.

7. Physics-based computing aims for human-level intelligence in a single rack

Unconventional AI Inc. is constructing a fully integrated system that harnesses the time-dependent physics of the physical world for computation, much as a rolling ball inherently calculates its trajectory, shared Naveen Rao, co-founder and CEO of Unconventional AI. The company's initial offering targets human-level intelligence consuming roughly 10,000 watts in a single rack, compared with the brain's 20-watt consumption. Rao is committed to progressively reducing that power requirement until it surpasses biological efficiency, making intelligence affordable enough to enable novel form factors.

8. Optics replaces copper as AI clusters outgrow electrical links

Power constraints are driving operators to connect multiple smaller data centers via fiber optics, and transitioning electrical connections within data centers to optical technology represents the next major growth opportunity, highlighted Michael Hurlston, president and CEO of Lumentum Holdings Inc. The company has expanded fourfold in just over a year, yet operates at roughly 70% below demand capacity and is essentially committed through 2029. Indium phosphide laser chips represent a critical supply constraint, prompting Lumentum to invest in manufacturing facilities and assembly capabilities as though the infrastructure expansion will continue for another four to five years.

9. Power systems race to match AI's one-year hardware cycle

Historically, power infrastructure operates on six- to ten-year replacement cycles while AI hardware refreshes annually or faster. Melton Chang, president of the Power Systems Division at Schneider Electric SE, characterized this mismatch as the industry's fundamental problem. AI workloads can fluctuate from 30% to 150% within milliseconds. These swings jeopardize voltage and frequency stability and trigger resonance phenomena that degrade on-site generator lifespan. Schneider mitigates these issues using digital twins, simulation tools and live data analysis to forecast such fluctuations.

10. Self-verifying AI agents speed chip design without sacrificing trust

Custom silicon development cannot sustain 18- to 24-month timelines when model releases occur every three to six months, yet design teams require deterministic, verifiable output from electronic design automation systems, according to Ankur Gupta, head of EDA IC and systems at Siemens Digital Industries Software. Siemens operates its design engines on Nvidia Corp. GPUs, achieving 10- to 12-fold performance gains. The company additionally deploys AI agents to streamline the setup and troubleshooting phases of the design workflow.

11. d-Matrix scales memory-centric XPUs from chip to rack for inference

d-Matrix Corp. has rapidly progressed from individual accelerator cards to complete rack-scale systems centered on its Corsair XPU, according to Sid Sheth, founder and CEO of d-Matrix. By stacking memory directly atop the processor—rather than positioning it beside compute on an interposer—the design unlocks both capacity and bandwidth, two primary constraints in AI inference workloads. The company expedited this transition by embedding into the Nvidia AI factory ecosystem, avoiding the need to engineer power, cooling, networking and software infrastructure independently.