Kubernetes finds new purpose managing AI inference workloads at scale

Agentic AI systems are creating unexpected demand for Kubernetes, the container orchestration platform long prized for managing reliable, predictable workloads. With AI investment increasingly flowing toward model serving and agent execution rather than model training, the open-source community is building new tools to manage surging inference traffic. Jonathan Bryce, executive director of the Cloud Native Computing Foundation, notes that Kubernetes was architected from inception to handle dynamic, distributed workloads—a capability that proved invaluable as AI companies scaled rapidly.

The AI boom has sparked a massive investment and a huge build-out of infrastructure overall. Having Kubernetes built from the beginning to handle really dynamic distributed workloads as those companies were growing like crazy — they had a really reliable, solid foundation to build on at the same time.

Jonathan Bryce, executive director of the Cloud Native Computing Foundation

Inference at scale for agentic AI systems

While new models attract attention through benchmark achievements in mathematics and coding, their practical value depends on serving them efficiently at production scale. Early chatbots simplified this problem because each user message generated a single inference call. Agents operate differently, triggering multiple inference calls to complete individual tasks—a shift that prompted the CNCF to add a dedicated AI inference and agentic track to its KubeCon conference agenda.

We're using coding agents to say, 'Build me a system that does this,' and it'll spawn 10 sub-agents. Each of those are doing thousands of inference calls over the course of hours and just churning away.

Jonathan Bryce

The surge in inference demand is driving data center expansion and raising questions about resource consumption. Two projects aim to extract greater efficiency from existing hardware: the llm-d inference engine and Agent Substrate, which provides lightweight hosting for long-running agents. Beyond performance, security and control present obstacles, since agents can produce different outcomes when assigned identical tasks. OpenShell, a security initiative now governed by the CNCF following Nvidia Corp.'s contribution, attracted over 300 participating companies within months—a growth trajectory that previously might have required two years.

What I am constantly thinking about is, 'How do we lower friction to collaboration?' How do we enable people to be able to move at the pace that the world and the industry is moving at, and how do we, while doing that, continue to drive that into collaboration around open solutions that can become those base and infrastructure components that we can all build on?

Jonathan Bryce