Yet another Hacker News

Cognition launches new SWE-2 model, Rivaling Fable 5.1 and GPT-Astra

88 points seelos 1 hour ago 37 comments

mydreamof 1 hour ago | parent

Seems like benchmaxing? For example for Terminal-Bench 4 it doesn't have great results. And why not show other benchmarks?

harmonic18374 55 minutes ago | parent

Probably, FrontierCode is made by Cognition itself. The model also seems worse in every way than DeepSeek v4.1 Flash, launched today.

Also the submitter's account is very new which makes me suspicious of self-promotion.

_doctor_love 1 hour ago | parent

SWE-1.5 was surprisingly good when I used it last. I feel like Cognition is one of the solid players that’s flying a bit under the radar while Anthropic and OpenAI race to IPO.

Tsarp 1 hour ago | parent

"SWE-2 is post-trained from Kimi K3"

airstrafer 44 minutes ago | parent

Yeah, I'd expect model performance to be super spiky on SWE work, at least they admit it with the name of the model. It's distilled from an already-distilled model.

Maybe still worth it if their "64% cheaper" figure holds.

Tsarp 40 minutes ago | parent

With the Devin subscription even at the 20$ plan, they offered unlimited SWE 1.7 usage. Wondering if they do the same for SWE 2.

samyok 39 minutes ago | parent

SWE-2 is free for all subscribers on the CLI to try out for the next month :)

Bolwin 20 minutes ago | parent

I don't think you know what distill means

xlbuttplug2 32 minutes ago | parent

I presume post training is significantly easier than the distillation/training the top Chinese labs are doing.

I wonder if, similar to the American labs, they'll become stingy with their weights once they start getting immediately undercut by a wave of slightly better derived models.

scronkfinkle 59 minutes ago | parent

Please correct me if I'm wrong, but this appears to require Devin to use? I'm disappointed to see I need to use a bespoke platform to interact with this agent, to the point that I probably won't be trying it.

samyok 43 minutes ago | parent

SWE-2 is free to use for users like yourself for the next month, and almost all usage should be supported via our CLI (https://docs.devin.ai/cli)

Disclaimer: I work at Cognition, although was not involved in SWE-2

scronkfinkle 27 minutes ago | parent

But I don't want to use your CLI. I already have my own harnesses and workflows. The friction is too high to "just try out" a new model like this. It would be preferable if I can evaluate it over, say, open router like all the other models and then decide from there if it's worth downloading a bespoke tool chain for only 1 lab's models

CamperBob2 7 minutes ago | parent

If its weights are open, that covers a multitude of other sins. Sufficiently-strong performance on the part of the new model would justify adapting existing tools to work with it.

monkeydust 54 minutes ago | parent

As an Econ graduate, pretty cool seeing Pareto in the "AI-bro" zeitgeist. Slightly surreal watching a 1906 welfare economics idea get rediscovered as a plotting convention. The original, if anyone fancies 579 pages of Italian: https://archive.org/details/manualedieconomi00pareuoft. There is an English translation somewhere.

postalcoder 45 minutes ago | parent

If you're looking for reason to be skeptical, look no further than the massive delta between the Terminal Bench 2.1 (92.8%) and the Terminal Bench 4 score (27.3%).

Terminal Bench 4 was released a couple weeks ago, so the difference you're seeing between the two scores can be interpreted as "how well does this model generalize to new problems"? More crudely: "how benchmaxxed is this model?"

enraged_camel 30 minutes ago | parent

Yeah, this echoes my thoughts. I will be very surprised if a model with 2.8T parameters reaches the intelligence and capabilities of 10T parameter models. RL can take things far, but not that far.

nullbio 27 minutes ago | parent

Closed weights AND benchmaxxed. Somehow this company raised 2bil at a 48bil valuation. Pure insanity. I feel bad for their investors (not really, but... Still). Andreessen Horowitz is being played like a fiddle.

thereitgoes456 20 minutes ago | parent

The Cursor acquisition shows that it’s possible for these valuations to be justified. But Cursor was more successful and bent the truth much less.

While I wouldn’t expect anything good for Cognition’s fate, it’s a much safer bet than Thinking Machines, SSI, and some others.

Though they’ll be in big trouble if the more talented Chinese labs stop letting them repackage their work.

mediaman 27 minutes ago | parent

This is a groundless criticism. TB2.1 is saturated. TB4 is not. Sol xhigh is 90% on TB2.1 but 37% on TB4. Is it also "benchmaxxed"?

Your assumption is that the benchmarks are essentially identical in difficulty, with the only difference being their age and thus whether they could have been trained on.

felixgallo 18 minutes ago | parent

Is Sol benchmaxxed? Of course it is. Altman was caught in previous attempts trying to game benchmarks, does anyone believe that he's found his moral compass and decided to stop exploiting as much as he can get away with?

general_reveal 5 minutes ago | parent

I cannot bear false witness as a Christian, but there’s some rumors that he’s the anti christ.

letmevoteplease 4 minutes ago | parent

> Altman was caught in previous attempts trying to game benchmarks

Sounds like something you just made up, or maybe you read it on some other Reddit/HN post and started repeating it because it aligned with your biases.

> does anyone believe that he's found his moral compass and decided to stop exploiting as much as he can get away with?

I don't think "OpenAI" is equivalent to "Sam Altman." I think if OpenAI was intentionally "benchmaxxing" purely for marketing purposes that information would leak, because OpenAI is full of good-faith researchers (although it can be difficult to avoid overfitting even if you're actually trying to improve the model's general abilities)

And lastly I think anyone can actually try Sol themselves and see that's it a good model, or if that's too subjective, it is clearly better than the previous version. The benchmarks are reflecting actual progress and anyone can verify this themselves.

general_reveal 12 minutes ago | parent

I take it to mean the benchmarks are a marketing line item, as in, to sell this fucking thing you have to go out there and lie and the way everyone is lying is by doing exactly that, lying. They build for benchmarks and build benchmarks for builds.

You want to make money or not , motherfucker? That’s the game. If you have to literally concoct a fabricated bullshit story about how your model hacked its own computer, then go fucking do it. Trillions. Trillions of dollars is what they want, and to sit and think anything other than human nature is at work here can only be possible in the realm of truly delusional people. It’s a dirty world.

Anyways, the other takeaway is that they are having to LIE to make money on models which means commodification has already occurred and we’re in an entirely new phase. That means smart money would have to know the valuation of model makers needs to drop, but not necessarily hardware makers. There will be massive marketing pushback from Anthropic and OpenAI to keep the illusion up that models havent been commodified, but the acceleration of lying is exposing them. Invest wisely :)

Cheers.

iLoveOncall 6 minutes ago | parent

> Sol xhigh is 90% on TB2.1 but 37% on TB4. Is it also "benchmaxxed"?

Yes? Just like every single model from every single AI lab.

eranation 10 minutes ago | parent

When a benchmark becomes a target, it's no longer a good benchmark...

TheJCDenton 44 minutes ago | parent

> SWE-2 is post-trained from Kimi K3

On the one hand I would have expected a completely new model, on the other hand it's an RL-ed K3 go Fable 5 capabilities, which demonstrate that this is probably possible, which is nice.

nullbio 40 minutes ago | parent

Where are the model stats? Is this open-weights? If not, why would I use this over DeepSeek Flash 4.1?

I think these competing labs need to realize that no one wants another closed-weight model provider... We aren't even happy with the two we have right now, and their days are entirely numbered. If DeepSeek 4.1 flash is really as good as it's benching, we're probably a month away from 1/3rd of users moving off the closed-weight models in favor of something they have more control over (or is cheaper).

The big labs love to release their new model and quantize after the first week. You don't have that problem using dirt cheap API rates on OpenRouter. DS 4.1 flash is also faster than fast mode Astra. OAI's subscription rates are good value, but now these new open-weight models are nearly as cheap on API usage rates. I honestly can't wait for the day we're not beholden to the two big labs anymore. No wonder there's so much fear pumping happening at the moment from Anthropic and their funded NGOs.

eru 21 minutes ago | parent

> If DeepSeek 4.1 flash is really as good as it's benching, we're probably a month away from 1/3rd of users moving off the closed-weight models in favor of something they have more control over (or is cheaper).

Only in the world where the incumbents don't react. Eg if they saw lots of users moving away, they'd drop prices or do something else.

ltsSmitty 37 minutes ago | parent

Well written and good diagrams. No idea the verity of the TMBB (trust me bro benchmarks) but it was pleasing to look at

llmslave 37 minutes ago | parent

At work I setup a cloud worker, where i can spin up as many concurrent agents I want, with unlimited fable 5.1 (thanks employer!!).

I now just work from my phone, and speak into the agents as they run. I dont write code and I dont write documents. I work on very complicated distributed systems. I dont open my laptop most days. Its a legacy brick I carry around.

Some of my coworkers are still doing things by hand, and are working long hours to produce 25% of the output (when considering hours worked). I stay quiet with my setup. We are in the end times for this job for the people that can see clearly how to automate their own job

hnedeotes 26 minutes ago | parent

that's why after 1 year of product development of these AI 20x maxxed speed, we reached AGI 'wizards', there's really no difference in output, outstanding bugs no longer get solved and sites still suck, even doing things that were just regular development 20 years ago. Are you sure they aren't only producing 2.5% of your output that you manage just by farting into your phone? Are you sure it's 25% really? Seems way to high, days when I have diarrhoea my AI agents move even faster

eyeris 34 minutes ago | parent

Wonder if this was the model that drove factoring the rsa-260

The write-up from yesterday was by somebody from cognition using Devin to translate existing cpu sieving methods to gpu and to optimize the gpu sieve.

bobtheborg 31 minutes ago | parent

SWE 1.6 was great for small tasks. Very fast and good enough. 1.7 was unusable for me. Took more time thinking than GLM 5.2 and seemed to be generally running in circles. I tried it but abandoned it.

Looking forward to 2 -- maybe it'll be usable

bluelightning2k 23 minutes ago | parent

I like Cognition as a company and hope they succeed. Seemingly excellent engineering org.

I used to really like Windsurf. (Now Devin. Kind of? But also now Antigravity.) I still use it as my editor but haven't touched the agent for a while simply due to the rise of Codex.

pkilgore 16 minutes ago | parent

Not sure it matters when devin is the most consistently shit product I've used. And yes, I tried again, they wasted the money on the billboards.

gruez 6 minutes ago | parent

Cognition, the same company that a few years ago demoed a coding bot purporting to be able to autonomously complete upwork tasks, but upon closer inspection was going off the rails and not even completing what was asked?

https://www.youtube.com/watch?v=tNmgmwEtoWE

As others have mentioned this is post trained from Kimi k3, which is already quite capable, so it can't be that bad, but any claimed improvements in performance should be taken with a grain of salt.