AI Labs Tested for 'Pelicanmaxxing' on Famous SVG Benchmark
Original: Are AI Labs Pelicanmaxxing?
Why This Matters
Rigorous informal benchmarks help the industry detect model gaming and assess true generalization capability.
Dylan Castillo of Iwana Labs generated 1,008 SVGs across 7 frontier AI models using a 48-prompt grid to test whether labs are over-optimizing for Simon Willison's famous 'pelican on a bicycle' benchmark. Analysis found no evidence of targeted benchmark gaming.
Simon Willison's informal benchmark — prompting LLMs to generate an SVG of a pelican riding a bicycle — has become one of the most recognized tests in AI, routinely drawing top comments on Hacker News. As the benchmark's fame grew, so did speculation about whether labs might be 'pelicanmaxxing': intentionally over-training on this specific prompt to boost perceived performance.
To investigate, Dylan Castillo (Iwana Labs) built a 8-animal × 6-vehicle = 48-prompt grid, making the pelican-bicycle prompt just one of 48 cells. He tested GPT-5.6 Terra, Claude Sonnet 5, Gemini 3.5 Flash, Grok 4.5, Qwen3.7-Max, GLM-5.2, and DeepSeek V4 Pro via OpenRouter, generating 3 samples per prompt at temperature 1.0 — 1,008 SVGs total. Each image was scored by GPT-5.6 Luna on animal quality, vehicle quality, and action coherence (1–5 scale), with Gemini 3.1 Flash-Lite extracting scene features.
The study examined five evidence types: visual inspection, pelican row scores vs. other animals, bicycle column scores vs. other vehicles, the pelican-bicycle cell vs. difficulty-adjusted peers, and scene memorization signals. Across all five, no evidence emerged that the pelican-bicycle combination received systematically higher scores than comparable prompts. Only 11 retries were needed across all 1,008 generations. Code and results are publicly available on GitHub.