IBM's llm-d Framework Reaches 1.0 With Enterprise-Grade Inference Controls

IBM and a coalition of technology partners have unveiled version 1.0 of llm-d, an open-source framework for distributed artificial intelligence inference. The update introduces operational capabilities meant to enable organizations to deploy and manage large models on their own computing infrastructure with improved reliability and control over costs.

The announcement came during the PyTorch Conference North America in San Jose. The framework, built to work with Kubernetes, orchestrates model serving across distributed computing resources. IBM's Red Hat division and Google serve as lead maintainers alongside IBM itself. The project was established last year through collaboration with CoreWeave, Nvidia, Advanced Micro Devices, Cisco Systems, Hugging Face, Intel and Mistral AI.

By offering a vendor-neutral approach to inference, llm-d addresses a key concern for enterprises: the ability to reduce reliance on any single model provider, cloud platform or hardware vendor while maintaining control over infrastructure decisions and associated expenses.

Ruchir Puri, chief scientist and vice president of IBM's Technology & Technical Community and IBM Fellow, noted the framework's impact on computationally intensive work. "At IBM Research, we are seeing that open large language models and llm-d can greatly lower the costs of inference in computationally demanding problems such as designing hardware for quantum error correction, helping achieve a 11.8X efficiency advantage," Puri wrote in comments. "Llm-d 1.0 extends these savings by delivering the operational readiness needed for enterprises to extract more out of their computational capacity."

Production-Ready Operations

The framework addresses challenges that emerge after models go live. Running inference at scale requires handling variable traffic patterns, hardware failures, software maintenance and resource contention among competing users—all while maintaining acceptable response times.

Version 1.0 introduces several capabilities to address these operational demands:

• Quality of service controls for multiple tenants sharing infrastructure
• Zero-downtime operations during software updates
• Serving patterns optimized for AI agents
• Flow controls and request prioritization to prevent agent activity from degrading performance for interactive services

Carlos Costa, distinguished engineer at IBM Research and a project co-founder, described the maturity level the release achieves. "The release adds the maturity to run it like any other critical service: a stable [application programming interface], multiple tenants with competing priorities sharing the same infrastructure and upgrades that don't interrupt operations," Costa said.

Agent-Optimized Routing and Caching

AI agents behave differently from typical inference workloads. They may issue multiple requests in sequence, pause to call external tools, then return for additional model interactions. The framework handles this pattern by maintaining request context across multiple turns.

A routing system directs requests to infrastructure replicas that already have relevant cached context loaded, maximizing reuse of previously processed information. This matters significantly because agents repeatedly reference the same code, documents, instructions and conversation history as they work through assignments.

The framework also manages cached data across different memory tiers—graphics processing unit memory, CPU memory and storage—to preserve reusable information when GPU capacity runs short.

While these techniques existed previously, version 1.0 integrates them with operational controls and adds methods for separating prompt processing from response generation, optimizing batch operations and distributing large mixture-of-experts models across accelerators. In a recent test of agentic coding, the framework's router achieved 97% cache reuse.

Cost Advantages in Practice

IBM and Red Hat demonstrated the economic benefits through a benchmark running 3,000 software-engineering agents concurrently on self-hosted infrastructure. The cost was five to 10 times lower than serving the same open model through a hosted service.

The savings become more pronounced as agents perform multiple rounds of reasoning, tool use and revision for a single task. Reprocessing identical material at each step consumes additional computing resources, so the framework's caching and routing optimizations deliver compounding benefits.

Integration and Deployment

Llm-d functions as a distributed routing and operational layer around model servers, working with inference engines including vLLM and SGLang. The Cloud Native Computing Foundation currently hosts it as a sandbox project.

Stephen Watt, vice president in Red Hat's chief technology officer's office, emphasized the operational familiarity the framework brings. "It delivers high-performance, distributed inference with the reliability and scale enterprises need within an operational model they already know," Watt said.