Small transformer scores 44% on ARC-AGI-1 for $0.67
Original: I trained a small transformer in 1.5hrs and it beats many LLMs
Why This Matters
Demonstrates that extreme sample efficiency and low training cost can rival large LLMs on a key AGI benchmark.
Mithil Vakde, an IIT Bombay graduate, trained a small transformer from scratch in 1.5 hours on an NVIDIA 5090 GPU for just $0.67, achieving 44% on the ARC-AGI-1 public evaluation benchmark — matching TRM/HRM and outperforming many LLMs.
Mithil Vakde published results showing a small transformer model trained entirely from scratch at test time can achieve 44% on the ARC-AGI-1 public eval benchmark at a cost of only $0.67. Training took 1.5 hours on an RTX 5090 GPU. The model uses test-time training on both training and evaluation puzzle sets (with test labels hidden), per-puzzle additive embeddings for cross-task learning, 3D RoPE positional embeddings, and augmentation with color and dihedral permutations. Key architectural upgrades over Vakde's previous version include SwiGLU activations, RMSNorm, 8 layers instead of 4, the Muon optimizer replacing AdamW, and flash attention with varlen training. A significant change was removing the loss on input tokens, making the objective purely supervised over output tokens — which improved scores from 40% to 44%, though Vakde states the reason is not fully understood. Additional training data from non-overlapping ARC-2 tasks was carefully filtered to prevent data leakage. The model also scores 7% on ARC-AGI-2. Code is open source on GitHub.