Kimi Delta Attention Explained from First Principles
Original: You Could Have Come Up with Kimi Delta Attention
Why This Matters
Linear attention architectures like KDA are central to scaling LLM inference efficiency for long-context applications.
Doubleword founder Jamie Dborin published a technical deep-dive showing how Kimi Delta Attention (KDA) — used in the latest Kimi and Qwen model families — can be derived step-by-step from standard softmax attention through a logical chain of design choices about hidden state properties.
Doubleword's Jamie Dborin published a blog post on July 27, 2026, walking through the full derivation of Kimi Delta Attention (KDA), a linear attention variant used in recent Kimi and Qwen model families. The post argues that KDA's equations, while appearing complex at first glance, follow naturally from a sequence of principled simplifications starting from standard softmax attention.
The derivation follows the path: softmax attention → linear attention → DeltaNet → Gated DeltaNet → KDA. At each step, the author motivates the change by asserting specific properties about the hidden state — such as how keys and values should be stored and updated — rather than presenting the final equations without context.
KDA's core state update involves a diagonal scaling matrix (Diag(αt)), a prediction of the current value from the existing state, an error term scaled by βt, and a rank-one correction added back to the state. The post uses bra-ket notation (borrowed from quantum mechanics) to make matrix shapes explicit, though a toggle allows readers to switch to conventional bold-vector notation.
The article also notes that KDA avoids the quadratic memory cost of softmax attention by maintaining a fixed-size recurrent state, making it more efficient for long-context autoregressive inference. After deriving the math, the post turns to recurrent and chunkwise Triton kernel implementations.