Kimi Linear: Hybrid Linear Attention Surpasses Full Attention
Original: Kimi Linear: An Expressive, Efficient Attention Architecture
Why This Matters
Linear attention achieving superior accuracy and efficiency over full attention could reshape large-scale LLM inference economics.
Moonshot AI's Kimi Team released Kimi Linear, a hybrid linear attention architecture with 3B activated / 48B total parameters that outperforms full attention across short-context, long-context, and RL scaling tasks while cutting KV cache usage by up to 75% and achieving 6x decoding throughput at 1M context length.
Moonshot AI's Kimi Team has published a technical report introducing Kimi Linear, a hybrid linear attention architecture claimed to be the first to outperform full attention under fair comparisons across multiple regimes, including short-context, long-context, and reinforcement learning scaling scenarios.
At the core of Kimi Linear is Kimi Delta Attention (KDA), a linear attention module that extends the Gated DeltaNet framework with a finer-grained gating mechanism. This design allows more effective utilization of finite-state RNN memory. A custom chunkwise algorithm leverages a specialized variant of Diagonal-Plus-Low-Rank (DPLR) transition matrices to reduce computation relative to the general DPLR formulation while staying consistent with the classical delta rule.
The pre-trained model uses 3B activated parameters and 48B total parameters, built on a layerwise hybrid of KDA and Multi-Head Latent Attention (MLA). Under identical training recipes, Kimi Linear outperforms full MLA across all evaluated tasks, reduces KV cache usage by up to 75%, and achieves up to 6x decoding throughput for a 1M-token context.
The team has open-sourced the KDA kernel and vLLM implementations, and released both pre-trained and instruction-tuned model checkpoints to support further research.