Better Models, Worse Tool Calls: Claude Opus 4.8 Regression
Original: Better Models: Worse Tools
Why This Matters
Highlights that frontier model upgrades can introduce tool-use regressions, raising reliability concerns for production agentic systems.
Armin Ronacher reports that Anthropic's newest models — Claude Opus 4.8 and Sonnet 5 — are producing invalid tool call arguments with invented schema fields, a regression not seen in older Claude models. The issue causes tool call failures in his Pi editor project.
Developer Armin Ronacher (creator of Flask/Jinja2) discovered a regression while debugging his Pi project: Claude Opus 4.8 and Sonnet 5 intermittently emit extra, invented fields inside the edits[] array of a file edit tool call — fields not present in the defined JSON schema. This causes Pi to reject the tool call and forces a retry loop. Notably, older Claude models do not exhibit this behavior, meaning Anthropic's state-of-the-art models have regressed on schema compliance compared to earlier releases.
Ronacher explains that LLM tool calls are not natively structured — they rely on special marker tokens (Anthropic uses ANTML-style XML-like markers) serialized into the model's context. The model was trained to emit these markers correctly, but there is no hard enforcement. He contrasts two approaches: post-hoc JSON validation (current apparent behavior) versus constrained/grammar-aware decoding, where the token sampler masks out tokens violating the schema at generation time. With constrained decoding, invented keys like "in_file" or "type" would be structurally impossible to emit. The post highlights that capability improvements in newer models don't automatically translate to better tool-use reliability, and that schema compliance is a distinct, trainable (or enforceable) property.