Deriving Transformers from Associative Memory
Original: You could have invented transformers from memory
Why This Matters
Shows attention has deep theoretical roots, which matters for principled architecture research.
Researcher Alex Wang argues that the transformer architecture can be derived from first principles using classical associative memory theory, suggesting the architecture was less arbitrary than commonly assumed.
In a technical blog post, Alex Wang walks through how the transformer's attention mechanism can be derived almost mechanically from the mathematics of associative memory — a concept studied in neuroscience and early neural network research long before 'Attention Is All You Need' appeared in 2017.
The central argument: if you start with a Hopfield network-style energy function for storing and retrieving key-value pairs, then apply standard optimization steps, you naturally arrive at softmax attention. The queries, keys, and values of standard attention are not design choices pulled from thin air — they fall out of the math when you ask how to optimally retrieve a stored pattern.
Wang's post connects dots between Modern Hopfield Networks (a 2020 paper by Ramsauer et al. that formalized this connection) and the original transformer paper, suggesting that someone fluent in associative memory literature in 2017 could, in principle, have derived the attention mechanism independently. The piece is aimed at practitioners who know transformers but not necessarily the older memory literature, offering a fresh lens on why attention works, not just how to use it.