91 points de_sty 3 hours ago 22 comments

robertkarl 2 hours ago | parent

I look forward to testing this against my private eval set to determine if it's benchmaxxing or actual improvement based on just 1200 tasks. downloading it now

xlayn 2 hours ago | parent

Snap, no gguf to download, will wait for one to appear to test, any improvement over an already good model is good news.

"18 days on 144 H200 GPUs" insane.. didn't thought so much hardware would be needed to train Qwen.

drdaeman 1 hour ago | parent

Rather than waiting, ask your assistant to quantize for you. It could be not the best quantization possible but it’s relatively quick, and I had an overall positive experience.

smallerize 53 minutes ago | parent

It is already quantized. You need to convert to GGUF format though. Did your assistant explain this wrong?

drdaeman 48 minutes ago | parent

My bad, I haven’t really looked yet (and haven’t asked any assistants), thought it’s BF16 safetensors as usual, and for those I used to quantize to ggufs with just a couple commands.

XCSme 2 hours ago | parent

Is it post-training or bench-maxxing?

octoberfranklin 2 hours ago | parent

FWIW that is about $331,000 on low-onboarding-grief clouds (e.g. vast.ai) right now using the SXM variants they used.

You might be able to shave that down using a different flavor of H200 (like PCIe) but it would take more hours, so benefits are unclear.

lumost 1 hour ago | parent

Savings would likely be larger with alternative accelerators such as trainium, no?

octoberfranklin 1 hour ago | parent

Savings would likely be larger with magic beans such as floobarbaz, no?

But seriously. If there were a cheaper and reliable way, I'm sure the authors would have mentioned it instead of H200 SXM modules. They're trying to brag about cheap training.

Maybe you wanted to use an opportunity to shill tranium.

phs318u 45 minutes ago | parent

> Maybe you wanted to use an opportunity to shill tranium.

This was unnecessary and against HN guidelines to engage in good faith.

woah 1 hour ago | parent

The heavy reliance on eval datasets showcased here and also evidenced by the frontier lab's high payments for such data cast doubt on the claims that AI will become self-improving soon

charcircuit 21 minutes ago | parent

>doubt on the claims that AI will become self-improving soon

It already happened. https://github.com/karpathy/autoresearch

jacquesm 1 hour ago | parent

I always wonder what you lose when you start maxing out some specific benchmark. It is really hard to do this in a balanced way and I would love if everybody that did these kind of efforts would take the time to also list the before/afters of a number of other benchmarks that they did not use in their training.

gardnr 1 hour ago | parent

Check the article. It’s the first graphic that has other benchmarks.

jacquesm 55 minutes ago | parent

Yes, they all go up. I am wondering what was lost.

nblintao 1 hour ago | parent

Good point! First of all, none of the benchmark tasks were used in training, though the training data was designed to improve capabilities on tasks like TB4.1. If you consider TB4.1 the main target, the blog actually shows before/after results on several other benchmarks, with no regressions and even some improvements.

jacquesm 56 minutes ago | parent

Yes, I've seen that, but I wonder what is missing because I find it hard to believe that you can just do some RL and then end up with a model that is better across the board. I would find it far more believable if you outperformed on the benchmark that was your goal (assuming you selected training data to give you an edge there) and underperformed on things that you were not optimizing for. But since it all moves in one direction (better) I feel there must be some kind of unknown at play here. Even so, task specific AI is definitely the future, you don't hire a plumber to do your bookkeeping either...

reissbaker 48 minutes ago | parent

Why do you find it hard to believe that you can do some RL and improve across the board (albeit less strongly on other semi-related tasks than your objective)? That's the idea behind generalization: it's not perfect but you get a little bit of cross-task generalization, so if you just keep spamming more RL environments you can hill-climb to AGI. That's what the labs are doing!

jacquesm 41 minutes ago | parent

You can hill climb but you will (or at least, I think you will!) lose generality.

And no, that won't get you to AGI, that is a completely different goal.

banghua13 21 minutes ago | parent

If you carefully curate the data, plus the original model isn't saturated for RL post-training, it's possible to improve targeted benchmarks without regression on other metrics. This experiment is done with terminal bench 4.1 as a target, and the regression on other benchmarks has been carefully controlled through data + infra design.

jacquesm 13 minutes ago | parent

Ok, so the obvious question then is: this seems (I'm not saying it is, it just looks like a small thing compared to training up a model from scratch) a relatively trivial undertaking for the authors of the original model, if there was this much left in the can why didn't they do something similar? Or is there some step here that only this team can perform? Or is it simply a matter of economics?

pkoiralap 7 minutes ago | parent

Sabi-noema also looks like a close competitor in the same space. Built from qwen3.8 and made better.

https://huggingface.co/sabicap/sabi-noema

Also HF model here: https://huggingface.co/RadixArk/DaisyRose-27B