Goodfire: Neural Networks Are Already Clean

Goodfire's Nick Cammarata has published a working hypothesis that challenges one of mechanistic interpretability's foundational assumptions: that neural networks are 'knotted' and must be untangled before study. The essay argues that canonical evidence for polysemanticity — neurons that appear to have multiple unrelated functions — was misread. The most famous example, a neuron in InceptionV1 responding to both cat faces and car fronts, was taken as proof of superposition, a proposed learning principle where networks fold unrelated concepts together when they run out of neurons. Cammarata calls this a misreading. What looked like arbitrary packing was mostly a pattern he terms 'recursive generalization,' which he considers far less problematic. The bigger culprit: architectural flaws in early models that caused later layers to overflow, compounded by weak interpretability tools that made those layers seem incoherent. After applying standard architectural improvements — no changes to model size or training data — Cammarata reports that 99-100% of neurons in InceptionV1's previously opaque last layer are now easily interpretable. The implication is significant: researchers may not need sparse autoencoders or other decomposition models to study networks. The original network itself can be the object of study. Cammarata covers vision and language models, including models up to trillions of parameters, and argues the field can return to direct circuit analysis rather than approximation.