YHK - yet another hacker news clone

Mercury 2.5 LLM hits 770 tokens per second

24 points Retro_Dev 1 hour ago 9 comments

rvz 1 hour ago | parent

The speed means absolutely nothing when it is finishing almost dead last when compared to the frontier AI companies.

copperx 1 hour ago | parent

Ah, the old "good, fast, or cheap; pick two" proves true once again.

downrightmike 46 minutes ago | parent

Give it a few months.

glouwbug 39 minutes ago | parent

Some of us want fast food

voiceeh 21 minutes ago | parent

Not if your use case needs speed. For one of my products I can't use an LLM that has a p99 of >700ms for TTFT.

walrus01 1 hour ago | parent

Pricing at $0.25 and $0.75 already puts its cost well above reasonably reputable inference providers for deepseek v4 flash or qwen 3.8-flash-next or similar class of open weight LLMs that fit in under 170GB of RAM, so I don't see the point. I think this is probably also stupider than laguna s 2.1 which can also be very cheap to serve.

bearjaws 47 minutes ago | parent

If you care about speed Cerebras gpt-oss-120b is 1400tk/s and "just as smart" in ranking.

I've used it on a few for fun projects and its decent but the speed is crazy to watch.

nylonstrung 27 minutes ago | parent

I honestly think the diffusion LLM approach is a dead end

It's telling that frontier labs like Google toyed around with it but didn't invest further even for their most speed and cost sensitive small models

Still unclear for what, if any use cases this is pareto frontier

LarsDu88 12 minutes ago | parent

You can't think that a small startup versus Anthropic's training setup is anywhere near the same scale to make apples to apples comparisons.

Not sure how the Chinese labs pull it off though using autoregressive models. The secret sauce is probably going to be in the training data.

The main reason Google hasn't switched over to DiffusionGemma is because serving at larger batch sizes loses the speed gains you get from diffusion, and most of the primary use case is serving many users at once off a single device with a large batch size.

If you were to move to on-device low latency... like say in a robot or something, then the story might be different...