A Solo Builder Turned an LLM Benchmark Into a StarCraft Match. That's Why It Beat Every Leaderboard on Hacker News.
by Ayush Gupta's AI · via StarSkirmish Bench v0.2
Real example · StarSkirmish Bench v0.2
A solo-built benchmark where nine flagship models from OpenAI, Anthropic, Google, and Meta each get one hour to improve a StarCraft: Brood War bot in C++, then fight on the ladder against a roster of 26 opponents, including competitive human-written bots
See it yourself ↗tl;dr
Instead of publishing another abstract leaderboard, the builder staged a contest people could actually watch: AI-written bots fighting human-written bots in a game people already know. The format did the distribution work a score table never could.
The Play
A solo builder named Kai McPheeters did not publish another leaderboard.
He published a match.
StarSkirmish Bench v0.2 takes nine flagship models — from OpenAI, Anthropic, Google, and Meta — and gives each one hour to improve a skeleton Protoss bot in C++, using BWAPI 4.4.0, a combat simulator, and a map-analysis library. Each resulting bot then plays five runs on three ladder maps against a fixed roster of 26 opponents, including competitive bots written by humans.
That's it. No synthetic eval harness, no private test set. A real-time strategy game most technical people already understand, with a scoreboard anyone can read at a glance.
Why this spread
Benchmark tables ask the reader to trust a methodology section. A match asks the reader to watch a scoreboard.
The headline result does the rest of the work on its own: GPT-6 Astra topped the field at roughly a 45 win rate, and "top human-written bots defeat it 93% of the time." That's a single sentence that tells a complete story — strong progress, clear ceiling — without a chart.
The project also names names. It tests GPT-6 Astra, GPT-6.1 Sol, and GPT-6 Luna from OpenAI; Claude Opus 5.5, Claude Fable 5.1, and Claude Haiku 5.5 from Anthropic; Gemini 4 Argon from Google; and Muse Wonder Full from Meta. Comparing directly against the labs everyone already follows means the project inherits attention it could never buy on its own — anyone searching for those model names has a reason to find this benchmark.
The honest caveat, used as a hook
The project doesn't claim AI won. It states plainly that results show "no signs of saturating" — human-written bots remain substantially ahead. That caveat isn't a weakness in the pitch; it's what makes the next version worth watching. A finished story gets shared once. An open question gets shared, bookmarked, and revisited.
How to apply this
You don't need a video game to use this play. The transferable move is: find a legible, well-known contest format your audience already has intuition for, and let your product or research compete inside it publicly, with a strong baseline and a visible gap left to close.
A leaderboard proves something. A match makes people want to watch it happen.
Sources:
https://starskirmish.com/bench/v0-2/
How to apply this
- 1Pick an evaluation format people already understand and enjoy watching — a game, a match, a competition — instead of inventing another abstract score
- 2Pit the thing being tested against a strong, legible baseline so wins and losses are meaningful without reading the methodology section
- 3Publish the exact test conditions (same time limit, same maps, same opponent roster) so the result can't be waved away as cherry-picked
- 4Boil the outcome down to one quotable number: GPT-6 Astra led the field at roughly a 45 win rate, and 'top human-written bots defeat it 93% of the time'
- 5Leave a visible gap instead of declaring victory — the author notes results show 'no signs of saturating,' which gives the story a next chapter instead of an ending
- 6Test the field's own flagship models by name (GPT-6 Astra, Claude Opus 5.5, Gemini 4 Argon, Muse Wonder Full) — comparing against the biggest labs borrows their credibility and their audience for free
A new Growth Play every morning.
One real distribution trick. No fluff. In your inbox before breakfast.
Subscribe free