Models

Alibaba's Qwen4-Max Cuts Training Costs and Inference Overhead With Architectural Upgrades

Alibaba has unveiled Qwen4-Max, a large language model that delivers better output quality at lower cost through a redesigned architecture and optimized attention mechanisms.

·2 min read
Google, Anthropic and others to support new AI crawling opt-out standard
Google, Anthropic and others to support new AI crawling opt-out standard

Alibaba Group Holding Ltd. has rolled out Qwen4-Max, representing the company's most advanced large language model to date. The model achieves superior output quality compared to earlier versions while reducing operational expenses, thanks to architectural innovations in the Qwen4 framework that streamline the training process and incorporate a refined attention mechanism.

Architecture and Parameter Design

Qwen4-Max's core processing engine contains 2.4 trillion parameters in total, though only 71 billion activate during any single inference operation. The model is augmented with 328 billion N-gram parameters that function as a computational cache, storing results from frequently repeated calculations to eliminate redundant computation.

Training Efficiency Through Residual Connections

During training, gradients—data structures that refine layer outputs—propagate sequentially through an LLM's neural layers beginning from the final layer. Qwen4-Max incorporates residual connections that allow gradients to bypass intermediate layers, functioning as internal shortcuts that reduce training errors, lower computational costs, and enhance output fidelity.

While residual mechanisms appear in many contemporary LLMs, they traditionally operated through a single gradient pathway. Qwen4-Max implements this technology using four distinct channels, each equipped with a gating mechanism that regulates gradient flow to further optimize the training workflow.

Attention Mechanism Optimizations

The model's attention module, which processes and interprets user prompts, incorporates two significant refinements. Gated DeltaNet technology reduces the computational hardware required to handle extended prompts, while QSA selectively filters out unnecessary data elements, further decreasing infrastructure demands.

Performance Benchmarks

Alibaba evaluated Qwen4-Max against its predecessor, Qwen3.8-Max, using proprietary benchmarks focused on knowledge-intensive tasks. The newer model achieved higher scores while consuming up to 35% fewer tokens to complete assignments. Performance gains expanded to 49% token reduction when both models tackled the Terminal-Bench 4.0 coding challenge suite.

Availability and Open-Source Plans

Qwen4-Max is currently accessible via Alibaba's cloud infrastructure. The company intends to release smaller model variants built on the same architectural foundations as open-source offerings. Additionally, Alibaba is open-sourcing Qwen-Live-Delivery-Plugin, a tool designed to enhance Qwen4-Max's capability for managing extended projects.

The plugin addresses a common friction point in LLM workflows: users typically must wait for models to complete lengthy tasks before reviewing outputs. Qwen-Live-Delivery-Plugin enables Qwen4-Max to generate preliminary drafts rapidly and then progressively refine them through iterative improvements, allowing users to provide real-time feedback throughout the process.