Yet another Hacker News

41 points __cayenne__ 1 hour ago 17 comments

I've refreshed StarSkirmish Bench, a benchmark where LLMs play StarCraft BW by writing code, with results from the flagship models of OpenAI, Anthropic, Google, and Muse

BeaverGoose 51 minutes ago | parent

They should benchmark it against Pluto. It's a artificial neutral network bot that can easily take games off the best broodwar players.

__cayenne__ 43 minutes ago | parent

I did a live stream where Astra played Pluto. Astra went 0 - 1000 https://x.com/kaimcpheeters/status/2106782151622504911?s=20

deburo 30 minutes ago | parent

I noticed Pluto is running on a simple CPU? If so, that's very nice for future RTS games. You just ship (or use a server-client arch) a bunch of models of various difficulty levels to let players practice. Could we even get models tuned to specific campaign levels? I wonder how costly the creation of Pluto was.

qsort 47 minutes ago | parent

It's beyond insane that Astra achieves above 50%. Are models allowed to "play against" the specific bot they're up against? What explains the fact that it drops to single-digits against strong bots?

__cayenne__ 37 minutes ago | parent

Yes during the 1 hour window they are allowed to play practice matches against opponents they are graded against - though they do not have access to their opponents source code.

They drop to single digits against the A tier and S tier bots because those bots are just generally harder opponents to beat.

jstummbillig 46 minutes ago | parent

How do they avoid the LLMs simply working with pre-training data? Which apparently they do, since the human written bots win. But it's a mystery, is what it is.

m_w_ 44 minutes ago | parent

This is an interesting benchmark - I like that time is limited, as it often isn't in other benchmarks.

It would be interesting to see the reverse, how long does each model take to get >50% WR against bots of various tiers.

__cayenne__ 40 minutes ago | parent

I did a 48 hour live stream - Astra cleared in 43 hours.

Although I should caveat the format advantaged the LLM bots compared to traditional tournament rules.

0ddj0b 38 minutes ago | parent

Any thoughts on building this for Starcraft 2?

caughtinthought 12 minutes ago | parent

How do the best humans do against the LLMs?

thinkingtoilet 6 minutes ago | parent

There is currently an AI bot playing on the ladder in Korea and essentially it will nearly always win because it can micro every unit individually across the entire map. If a normal human can max out at hundreds of actions per minute, the bot can do thousands. Broodwar is so insanely difficult mechanically that there is no way for humans to compete. I imagine in a few months, maybe a year, it will be completely unbeatable.

MichaelNolan 11 minutes ago | parent

So this benchmark has the LLMs writing a bit to play StarCraft. Somewhat similar there was a post here a while back that had the LLMs playing StarCraft directly - https://broodbench.com/ and https://news.ycombinator.com/item?id=49766966

tweakimp 7 minutes ago | parent

If you want to see human written bots in action or compete in the bot ladder yourself, try https://aiarena.net (SC2)

cgearhart 5 minutes ago | parent

Is there anything interesting to learn from the agent written bots? Or are they just inferior versions of the best bots because the time limit is so short? Is there any indication that they’re trained on the code from the human written bots (like, do they magically “guess” creating functionality that requires experimental feedback or domain experience?) How much are the bots on rails applying the same techniques every time?

I think those kinds of questions would be more interesting than the raw win/loss ratios.

iugtmkbdfil834 5 minutes ago | parent

I am not sure I like this benchmark, but mostly, because I am not sure what it is really evaluating. It is not an idle dismissal.. multiple factors seem to complicate what it is supposed to look at ( just top of my head ):

- code generation speed - logic analysis - understanding of the game platform and how use of C may influence it - considered approaches, tactics and strategies

Don't get me wrong. It is is neat idea. I am just not sure what it measures.

GodelNumbering 4 minutes ago | parent

Wow this is interesting to me from another angle. A friend of mine created a benchmark to evaluate LLMs on 9x9 Go (using KataGo opponents as Elo anchors) [1]. The relative rankings of this (StarSkirmish Bench v0.2) benchmark (GPT-6 Astra > GPT-6.1 Sol > Gemini 4 Argon > Claude Opus 5.5 > Claude Fable 5.1 > Claude Sonnet 5.5) matches his GoBench rankings exactly, even though they are different problems at surface level.

[1] https://rolandgao.com/blog/gobench/

brokensegue 4 minutes ago | parent

why protoss?