145 points ilreb 1 hour ago 28 comments
dan_thinks 1 hour ago | parent
Looking forward to the "Flash" and 27B versions.
seizethecheese 59 minutes ago | parent
And it’s a shame because the only reason it scores so low is that it’s slooooow. (Like 30 tokens per second vs opus near 80.) If you exclude voting, Qwen 3.8 does well.
(We allow early voting contrary to Arena.ai because I think it makes sense for agents, where it doesn’t for chat. This means our data takes speed into account.)
Also, it would be ideal if they served this with third parties. Part of the slowness with 3.8 is just not a lot of providers. Their open source version is much faster, but we couldn’t include it because it doesn’t handle images (another own goal by Alibaba).
forgingahead 44 minutes ago | parent
seizethecheese 40 minutes ago | parent
anana_ 43 minutes ago | parent
anon373839 39 minutes ago | parent
gr_norm 16 minutes ago | parent
stratts 15 minutes ago | parent
Yet every time I compare the two in something like Pi, Sol ends up being 5-10x the price for a simple query, something like 7-10c vs. less than 0.5c for DeepSeek. What's the deal?
blackoil 10 minutes ago | parent
the_duke 3 minutes ago | parent
Models often get great scores despite being a lot worse than the frontier in practice.
AAisthetiobeofA 21 minutes ago | parent
Ah yes, the reverse-benchmaxers of OpenAI models.
m00dy 32 minutes ago | parent
maplet 20 minutes ago | parent
esperent 14 minutes ago | parent
By comparison opus 5.5 (max) is $10.01.
If they are not testing Chinese models with less than max thinking, then we should only compare them to other models also at max thinking.
nixosbestos 13 minutes ago | parent
EDIT: oh, I guess this isn't open weights, so presumably I'll have to get it through Qwen or whoever they might partner with (Bedrock?).