GPT-6 Astra tops StarCraft LLM benchmark
Original: Show HN: Gemini 4 Argon plays StarCraft
Why This Matters
Long-horizon agentic coding benchmarks expose capability gaps that static tests miss.
StarSkirmish Bench v0.2 pits LLMs against each other by having each write a StarCraft: Brood War Protoss bot in C++ within one hour. GPT-6 Astra leads with the highest win rate; GPT-6.1 Sol ranks second at a fraction of the cost.
StarSkirmish Bench v0.2, published October 15, 2026 by Kai McPheeters, evaluates flagship LLMs by having each one spend one hour writing a Protoss bot in C++ using BWAPI 4.4.0, then measuring win rates against a roster of competitive human-written bots. Each model runs in its lab's first-party CLI harness, with scores averaged across five runs.
GPT-6 Astra tops the leaderboard by a clear margin. GPT-6.1 Sol takes second place and stands out on value: it scores just below Astra at roughly one-tenth the API cost per run. Gemini 4 Argon, Claude Opus 5.5, and Claude Fable 5.1 are effectively tied in third through fifth, all scoring 37–39.
The benchmark is not close to saturation. Against the three strongest human-written bots—Stardust, PurpleWave, and BananaBrain—even GPT-6 Astra wins only 7% of games. The strongest predictor of a model's score is how aggressively it plays practice games against top-tier bots during its hour. GPT-6 Astra, GPT-6.1 Sol, and Gemini 4 Argon all begin doing this early, before their own bots can win those matchups. Gemini 4 Argon also shows notably high run-to-run variance—18 points between its best and worst run, compared to 9 for GPT-6 Astra and 8 for Claude Opus 5.5.