How to measure AI-generated code sloppiness
Original: Measuring the sloppiness of code
Why This Matters
Code quality evaluation is a critical missing layer as AI coding agents move into production workflows.
As LLMs approach near-perfect functional code generation, Earendil researcher Sebastian argues that correctness alone is not enough. AI-written code often introduces unnecessary abstractions, duplication, and bloat. He reviews existing metrics — AI judges, human review, LOC counts, and verbosity scores — finding each approach has significant limits.
Sebastian from Earendil opens with a problem many vibe-coders have hit: each new feature can trigger an explosion of lines of code (LOC), eroding human oversight in codebases growing by millions of LOC per month. His key claim: agents can't fix slop either.
His survey of sloppiness metrics covers four approaches. AI-as-judge is the most common industry method, but asking a model to rate its own output on a 1–10 scale is 'basically equivalent to a random number generator.' Pairwise comparisons (A vs. B) are better but still vulnerable to label-order bias, especially in smaller models.
Human evaluation remains the gold standard for readability but doesn't scale for training data or multi-model benchmarks. The simplest proxy — raw LOC delta — turns out to be 'surprisingly effective,' with the classic Goodhart's Law caveat: optimize for it and it stops being meaningful.
The most promising formal metrics come from the paper SlopCodeBench, which introduces verbosity measures that reportedly distinguish legacy codebases from LLM-generated slop. Sebastian frames the whole problem as fundamentally harder than correctness checking: correctness has a clean reward signal via hidden tests; sloppiness requires human taste, which is hard to formalize or scale.