Yet another Hacker News

59 points i_rush_carriers 1 day ago 16 comments

https://www.paperinstruments.com/blog/ultramarine

https://twitter.com/bertgodel/status/2110400987622265232

chicagobuss 59 minutes ago | parent

why does the pareto cost per task chart not have 6.1 sol?

desireco42 50 minutes ago | parent

With Jev to make decisions and Ultramarine to produce document manipulations if I understood it correctly, this is pretty useful in office. I would be excited if I was a big corp.

kadoban 49 minutes ago | parent

> We present Ultramarine 1 Flash, an agentic model for professional knowledge work. Ultramarine applies large-scale reinforcement learning to a strong open base model, GLM-5.3 Flash, and improves on it across all six knowledge-work benchmarks we evaluate [...]

In case anyone was curious what it's based on. (No judgement either way from me, IMO depends how well they did the post-training, which I can't tell).

ra 35 minutes ago | parent

Nice - I use GLM 5.3 flash for knowledge work now - will check this out.

sudo_cowsay 48 minutes ago | parent

I have a question, what is considered a Frontier Model? Because its performance is kinda similar to Luna (Max)... I thought frontier meant like 6.1 Sol or 6 Astra or Opus 5.5??? Could someone explain this to me?

handfuloflight 46 minutes ago | parent

They didn't claim it's a frontier model. They claim frontier level performance.

tibbar 31 minutes ago | parent

These days people often mean the cost/performance Pareto frontier. That is, it is the best performance you can get for that cost profile.

Jackobrien 43 minutes ago | parent

These guys are awesome and I’m really impressed. The model alone looks great (according to benchmarks), and very cool they’ve done so much in building out the harness of top of it all.

zer00eyz 37 minutes ago | parent

Someone will report back "It didn't one shot my odd ball task"

Did it do it in 2 tries? 10? It's still coming out cheaper. Brute force the work. This is the path that the labs are taking. It's the 1200 agents trying to solve Hacking Bench hacking hugging face. It's anthropic telling you that its 1000's of staff commits like 10k engineers.

Treat AI like you're deploying interns who don't need coffee, food or bathroom breaks. With the right metrics they will get to an optimal (enough) solution.

kridsdale1 7 minutes ago | parent

Like Zerglings.

plaguuuuuu 7 minutes ago | parent

there's a problem with that. Probabilistic slot machine works for anything with a deterministic eval (coding, troubleshooting, etc). It doesn't work for anything subjective, like the thing this model is targeting, professional work.

tonyhart7 26 minutes ago | parent

so its for working only right ???? it cant do/bad at coding

randomblock1 16 minutes ago | parent

Doesn't seem like coding was a focus. DeepSWE: 61.5 (this), 63.4 (GLM 5.3 Flash, which this is based on). TerminalBench 2.1 is also slightly worse.

So... potentially? The improvements in other areas might help coding, but aren't that big either, except on their own benchmark.

moralestapia 4 minutes ago | parent

"Ultramarine 1 Flash is Fable-class on knowledge work"

Oh no! That means it's going to destroy humanity!

It will:

* Hack into everything that has a computer, like your home appliances, even a toaster could be and will be weaponized against you.

* Break cryptography, it's gone, the end of the internet as we know it.

* Be impossible to contain as it can even escape airgapped environments by changing the temperature of CPUs to transmit information to other servers nearby which would sense fluctuations in ambient temperature[1].

Or wait ... maybe the people at Anthropic/OpenAI are just a bunch of lying crooks that would say anything to make a bit of cash. Who would've thought?

1: This absolutely imbecile statement was made by one of their "top scientists", it was not a joke.

tills13 4 minutes ago | parent

I don't care, I'm extremely bullish on distillation models like this. Plus anything that'll take the wind out of Anthropic's and OpenAI's IPO sails.