Kimi K3 Architecture Overview: 2.8T Open-Weight Model Breakdown
Original: Kimi K3 Architecture Overview and Notes
Why This Matters
At 2.8T parameters, Kimi K3 sets a new scale benchmark for open-weight models and advances inference-efficient MoE architecture design.
Sebastian Raschka analyzes Kimi K3, the largest open-weight model released to date at 2.8 trillion parameters, scaled up from the 48B Kimi Linear model. Key innovations include LatentMoE, multi-head latent attention, Kimi Delta Attention, attention residuals, and full NoPE adoption with native multimodal support.
Machine learning researcher Sebastian Raschka published an architectural overview of Kimi K3, which he describes as 'by far the biggest open-weight model right now' at 2.8 trillion parameters — scaled up from the 48B Kimi Linear model released previously. Raschka identifies the primary architectural theme as inference efficiency, with multiple component upgrades: standard MoE replaced by LatentMoE (also seen in Nemotron 3 Ultra), regular attention replaced by multi-head latent attention and Kimi Delta Attention. LatentMoE compresses large linear layers via down-projection, mirroring the approach used in multi-head latent attention. The attention residuals mechanism — inherited from Kimi Linear — connects residuals across layers using attention scores as contribution weights. According to the technical report, this improves validation loss and downstream performance consistently, at a cost of approximately 4% additional training compute and 2% additional inference compute. Notably, Kimi K3 eliminates all RoPE (Rotary Position Embedding) layers entirely, using NoPE (No Positional Embeddings) throughout — a first for a frontier-level model, according to Raschka. Recent architectures had trended toward RoPE in local attention layers and NoPE in global layers, but K3 breaks from that pattern. The model also introduces native multimodal support.