40 points hn-aarvin 1 hour ago 20 comments

quantisan 29 minutes ago | parent

When and why should I trust this index over others?

ummonk 25 minutes ago | parent

I wouldn't trust any particular one, but obscure / indie indexes are going to be less affected by benchmaxxing than the well known ones, so it's a positive sign for Neon that it is performing so well on a lot of them including this one.

js4ever 24 minutes ago | parent

Is it just me or all Google models are terribly bad? They all seems benchmaxed to hell and producing total crap compared to Claude or openai models from my own experience (mostly coding)

copperx 22 minutes ago | parent

Are you saying Neon is just as bad?

mjburgess 21 minutes ago | parent

And tokens/task is very high. They don't seem to have figured out making the models usable in a typical interactive workflow, ie., they're in a Opus-4.7 era.

Very high hallucination rates too.

SwellJoe 20 minutes ago | parent

I think it's just you. Carbon is excellent but extremely slow. If Neon is better than or equal to Carbon while also being a lot faster, that'd be a great model. I haven't tried Neon, yet, but they seem to have understood the assignment with the latest round of models.

azan_ 19 minutes ago | parent

Yes. Every Google model so far has been disappointment. And their biggest moat, the integration into other google products has been laughable. Today I've asked gemini in gmail to download all attachments from one e-mail address and it turned out it's unable to do it. It's unbelievable how incompetent they are, Google has such a massive edge over OAI and Anthropic and it's still losing.

andunieee 10 minutes ago | parent

You mean they are not providing you with their best models for free?

hadlock 17 minutes ago | parent

After hearing some good things from my tokenmaxxing buddies (or whatever you want to call them, AI enthusiasts), I got a $20/mo sub to try out Neon and it's just breezed through some hardware graphics bring-up problems I've been struggling with on Sol/Astra and Fable. I'm gonna drop my $200 OpenAI sub this coming month in favor of Google's $200 plan just for more Neon tokens.

Yahivin 15 minutes ago | parent

With Neon Gemini is good now.

asadm 8 minutes ago | parent

this week I ran out of codex astra quota and switched to antigravity, boy is Gemini Carbon amazing model! It actually helped achieve a breakthrough I was stuck on for weeks with Astra.

danpalmer 4 minutes ago | parent

It's just you. Since Argon, the Gemini models have been producing good results for me side-by-side with Opus/Fable models on the same prompts. Not always winning, but increasingly so with Carbon/Neon.

YuechenLi 17 minutes ago | parent

Ok, so I can see that this is a benchmark the submitter made and submitted themselves, which is kinda misleading and should at least be disclosed.

But my question is, what does this benchmark actually measure about the LLM's engineering abilities? It just looks like a page of random bar graphs to me.

andunieee 11 minutes ago | parent

How do I use it? Do I have to create a Google Console account and navigate through 15 menus in order to generate an API key? Or are there another 7 different ways to achieve similar outcomes?

Yahivin 6 minutes ago | parent

Openrouter if you just want to pay as you go at token rates.

https://one.google.com/about/google-ai-plans/ had a 75% off on the Pro plan ($5/mo for 3 months) and limits are decent.

dudus 4 minutes ago | parent

[delayed]