Yet another Hacker News

Real-SWE: Benchmarking AI models on private, real-world, enterprise codebases

33 points theanonymousone 1 hour ago 22 comments

dgellow 43 minutes ago | parent

A bit of a meta question: what are the most relevant benchmarks by now?

andriy_koval 27 minutes ago | parent

Nvidia and OpenAI claimed AGI, but you still have a job.

tetec1 20 minutes ago | parent

Epoch.ai has a global score and tracks many benchmarks: https://epoch.ai/benchmarks

jcmontx 41 minutes ago | parent

I’ve been able to offload most tasks (coding or eles) to Codex since 5.3-codex with extra high thinking

riddlemethat 15 minutes ago | parent

Astra lets me offload entire projects without worrying about individual tasks…

jeffybefffy519 8 minutes ago | parent

Do you review the outputs?

traceroute66 30 minutes ago | parent

So TL;DR benchmarking in a completely non-reproducible manner ?

"Model X performed great, but we can't possibly tell you anything about the code it was looking at apart from it was a large code base from an unknown company".

So basically pinky-promise benchmarking ?

I'm not sure I follow the value here ?

kadoban 21 minutes ago | parent

If it builds up history and perceived reliability, this type of thing can be valuable. You're giving up transparency for it being harder to game.

traceroute66 16 minutes ago | parent

> You're giving up transparency for it being harder to game

But then if we take that argument to its natural extreme, surely it means people should take the marketing bullshit published in the 100-page system cards published by Anthropic & co as "valuable" too ?

demibabs 17 minutes ago | parent

Doesn’t it ultimately have to be this way, to prevent saturation?

bix6 28 minutes ago | parent

Wake me up when September ends or when I can do this locally.

demibabs 16 minutes ago | parent

> Each task comes from a private production codebase that we licensed from a real-world company

How does that work?

traceroute66 13 minutes ago | parent

> How does that work?

My gut feeling is that any serious real-world company with a proprietary codebase worth looking at would not be handing out the crown jewels to a third party. License or not.

I don't doubt somebody licensed their codebase to them, I just have my doubts about who the "who" could be.

InsideOutSanta 38 seconds ago | parent

Code isn't worth all that much if you don't own the associated IP, mainly copyright. And even if you disagree with that premise, if you trust that they can keep the code secret, it's basically free money.

At any rate, I'm not sure it matters whose codebase it is. I'd even say that a shitty codebase might make for a better test.

IshKebab 16 minutes ago | parent

I think these benchmarks are not that useful, e.g. this suggests Fable is better than Astra, but in practice Astra is waaaaaay faster (like 5x; it's not even close), and also waaaay less annoying to talk to.

There's only two or three sane options here - you can easily try them all and pick yourself.

rovr138 1 minute ago | parent

They're not measuring speed nor annoyance. It's there on the page

bdlowery 8 minutes ago | parent

The fact that gemini 3.8 flash is so high up there just tells you this is an awful benchmark.

Try and use gemini 3.8 yourself for any real world work and you'll see it's terrible. It'll just go in circles reading the same file 20 times for no reason making hundreds of tool calls for a simple change.

thereitgoes456 4 minutes ago | parent

Why so brazenly confident? Isn’t it possible that the benchmark is correct, and your experience is correct too, but you haven’t tried all the thousand different modalities of work that programming encompasses and so maybe you don’t actually have standing to judge?

siddbudd 1 minute ago | parent

havent tried that model, but it sounds like a potential harness issue. Have you tried it in different harnesses?

tucnak 38 seconds ago | parent

Hard disagree. I use 3.8 flash in Antigravity a lot, and thoroughly prefer it to most Pro models. It's really fast, and I've had it make crazy progress on compiler-like problems that previous models simply failed at. On ultra plan you can have it going for hours, and make incremental progress with good prompting for review interrupts. It literally solved a problem I couldn't solve for weeks. The harness and test suite is key.

lmeyerov 7 minutes ago | parent

My intuition is that many of the better & bigger 'private' code bases, at least in terms of claude code and codex... are not in fact private at this point.

One lesson of running botsbench.com, in a slightly different domain, is to measure for model contamination every time.

visiondude 3 minutes ago | parent

this is the closest benchmark to my experience using the model harness combo. Astra for as great as it is falls slightly behind Fable 5.1 for me for large feature work (although it comments code much better). in particular, Fable is able to assess priority better than Astra (meaning Astra sometimes does things that aren’t worthwhile while missing things that are clearly important, particularly on possible ballooning scenarios- fable catches “this works for x amount of data but if we run this on y way greater than x amount of data we’ll run into issues). Gemini 3.8 is under appreciated, use Google Stitch to see it in action if you haven’t used Agy yet.