Google donates AI agent tools to open-source community to slash idle compute costs on Kubernetes

Kubernetes has emerged as the standard infrastructure layer for distributed training and inference work, yet a significant economic problem emerges when the platform hosts AI agents that remain inactive for extended periods. The financial impact compounds because customers continue to pay for provisioned containers and virtual machines even when they sit idle. To tackle this challenge and introduce greater predictability to agent workloads, Google LLC has transferred three open-source projects to the Cloud Native Computing Foundation, according to Allan Naim, director of product management for AI infrastructure at Google Cloud.

The shift represents a fundamental change in how Kubernetes must operate. "Act 2 is a few years back; Kubernetes became the de facto for inference and training," Naim said. "What's really interesting right now is we're embarking on this Act 3 … where the typical patterns are very, very different from how Kubernetes was designed." Naim and Gari Singh, product manager at Google Cloud, discussed these developments during an exclusive broadcast at KubeCon + CloudNativeCon NA.

Addressing idle time and security in agentic systems

The economics of agentic AI present a substantial obstacle, with agents spending 90% to 95% of their time idle while waiting for model responses, testing cycles or human input. Google developed Agent Substrate, an open-source runtime environment that isolates AI agents on Kubernetes infrastructure. When an agent enters a suspended state, the runtime captures a snapshot of its memory and local files, then releases the underlying compute resources for other workloads. Upon resumption, the agent restores from that snapshot within an available sandbox environment.

The efficiency gains prove substantial. "It helps from an economic standpoint," Naim said. "We've seen 10x reduction in cost in terms of running Agent Substrate on Kubernetes versus running these agents directly on dedicated virtual machines and containers." Beyond cost considerations, security emerges as an equally critical concern. Google open-sourced gVisor in 2018, a security layer that operates above the host kernel to intercept and monitor every system call an agent executes, according to Singh.

Singh emphasized the unpredictability of agent behavior: "Agents and agent harnesses are designed to solve a problem — and to try to solve a problem — until they can't solve the problem. They will find all kinds of clever workarounds. They know everything that's around. You want to make sure that it didn't go make some low-level change to [your] low-level security, didn't make an outbound network call, didn't try to open a file that it shouldn't."

Expanding the open-source toolkit

Google is collaborating with Anyscale Inc. on KubeRay, which enables Anyscale's Ray platform to operate natively within Kubernetes, allowing platform teams to distribute training and inference capabilities across multiple research groups. OpenRL extends this model to reinforcement learning, a technique for teaching post-trained models behavioral guidelines and safety constraints.

https://www.youtube.com/embed/wv2C9Gf8aAs?feature=oembed

Democratizing reinforcement learning represents a significant shift from historical practice. "In the past, to do reinforcement learning, you needed to hire a dozen research scientists with PhDs," Naim said. "Our goal was, 'How can we bring reinforcement learning into the world of platform engineering so that you can run reinforcement learning on your Kubernetes environment and take advantage of it to actually do continuous improvement for your agentic workload?'"