Yet another Hacker News

Why I'm still bearish on LLMs after Navier-Stokes

103 points jaykru 6 hours ago 57 comments

jaykru 6 hours ago | parent

archive link in case i get hugged lol https://archive.ph/Z4gxF

robinpie 1 hour ago | parent

I really appreciate seeing a tempered take that's not literally denialist about current capabilities.

jaykru 1 hour ago | parent

Thanks :) I do enjoy and use these things every day and the current capabilities are indeed amazing, just ludicrously overpriced at the frontier.

dumberquestions 59 minutes ago | parent

I can see current limitations, but how do you expect capabilities to change in the next few years? A repeat of the gain that happened in the last two years feels like it would be significant, even if it took a little more than two years this time around.

brindleth 1 hour ago | parent

> current frontier models need laborious oversight and guardrails on even the simplest tasks

It is literally denialist about current capabilities

jaykru 1 hour ago | parent

why don't anthropic and openai ship yolo mode by default?

Human-Cabbage 55 minutes ago | parent

They do…? Well, “auto” mode has been default in Claude Code for a couple months now. It’s effectively “safer yolo:” tool calls are inspected by a separate classification system (another smaller LLM, I believe) to approve or deny. And you can always layer on additional sandboxing mechanisms to limit the blast radius deterministically.

vmg12 24 minutes ago | parent

> They do…? Well, “auto” mode has been default in Claude Code for a couple months now

They have never shipped "yolo" mode by default. Auto mode is not yolo mode. They trained a task specific model just for ensuring the llm didn't accidentally delete every file from your computer.

SyneRyder 52 minutes ago | parent

Anthropic basically does at this point with Auto Mode being default. Or was that the point you were making?

jaykru 30 minutes ago | parent

That is the point I was making, that auto mode is itself a guardrail on top of the model (and not a perfect one.) auto mode seems to cover merely actions the model could take that are clearly bad, like wiping your disk, using an overly privileged context to complete the task, etc.

I recently tasked a GPT model in Codex with implementing part of a new architecture I'm working on. I gave it a very detailed spec and the code it produced looked pretty reasonable and passed my tests. It even did exceptionally well in my evals, so I excitedly declared victory to a few friends. The next day after more careful review I found that the architecture implementation was totally correct, but the model had slipped a one line change to the observation encoding of the RL environment I was prototyping against. The encoding change made the learning problem essentially trivial; the architecture itself, I later realized, had a major flaw that was revealed by returning to the natural encoding.

This is the type of reward hack that is hard to paper over with easy guardrails like auto mode and even harder to specify out. It's also the type of thing a reasonable human wouldn't do unless they were intentionally trying to deceive you.

an0malous 1 hour ago | parent

I don’t know who you’re talking about, even the most bearish people like Gary Marcus and Ed Zitron acknowledge that LLMs are useful in these same cases the OP admits. Gary Marcus is even still a long term AI advocate, he just doesn’t think LLMs are enough and we need more foundational breakthroughs. Zitron says it’s valuable technology but not worth the trillion dollar valuations the frontier labs are claiming.

The lack of temperament is very skewed towards the bulls who have been saying AGI is here, software engineering is solved, mathematics is solved, it’s going to destroy the white collar job market, and it’s going to kill us all for like 5 years now.

arctic-true 1 hour ago | parent

Gary Marcus is an especially puzzling addition. If I recall correctly, he has made statements along the lines that superintelligence this century is more likely than not. If you’re AGI-pilled that might read as bearish, but that is still extremely rapid progress in the grand scheme of things.

pvab3 30 minutes ago | parent

Even a lot of the people who think that LLMs are a dead end think that we will soon find something signficantly more powerful, which I find deeply alarming. I don't want to know what my white-collar knowledge work will look like in a decade or 2.

ausbah 1 hour ago | parent

> the best alternative to rigorous specification is human review. human review doesn't scale well to the volumes of output produced by language models. to make matters worse

when the business model is selling more tokens you get such per serve ice times that lead to “more” thinking, engagement baiting, fluffy narratives, and straight up dark patterns

pfdietz 1 hour ago | parent

Specifically: bearish on LLMs generally, not bearish on LLMs for pure math.

jaykru 1 hour ago | parent

yes, huge for pure math and activities that look like it.

danielmarkbruce 31 minutes ago | parent

Doesn't really even need to look like it. If you can verify rewards, RLVR will optimize really really well. If you can't... it's a struggle. There are probably fewer fields where you can verify rewards than one might hope.

randomImmigrant 1 hour ago | parent

I think bearish on LLMs for automation, and bullish for LLM+human experts in specific fields, is about the right expectation for current architectures.

Apart from issues with task generalization, or perhaps related to it, is the fact that LLMs have real trouble with timekeeping, and cannot estimate the real world time it will take them to do things very well. This plus the memory issues make dreams of long horizon agents, that could plausibly handle changing specifications, quite implausible with current architectures.

In narrow domains with more deterministic outputs though, this is less of an issue, and we see multiple agents succeed much better.

The fusion of that capacity, with humans in the loop able to better direct such agents and act as their temporal tethers, is where I think the real action will be for a while at least.

againstapples 57 minutes ago | parent

> the models generalize well only on tasks within a small neighborhood of the specific tasks they've been trained on, and even then with severe caveats. the frontier labs have developed a general recipe to teach models almost any specific task enjoying clearly defined levels of task performance; many tasks are covered in the training data

Is this really any different to how humans learn, it takes a lot of training on one specific task to make a human expert as well?

JohnMakin 54 minutes ago | parent

> Is this really any different to how humans learn

yes.

knuppar 48 minutes ago | parent

being a bit more specific: the sample efficiency of humans is orders of magnitude larger for more abstract concepts. the same doesn't hold for memory-intensive tasks though (like any kind of trivia), but that only takes you so far.

bananzamba 41 minutes ago | parent

Also doesn't the very good ARC AGI 2 score of GPT-6 Astra kinda contradict this, since each problem is its own game with very different rules

bravoetch 39 minutes ago | parent

I was a young child when I learned chess by reading a short book, then practicing with a friend. That is not how LLMs learn. I'm no expert on LLMs, but if you showed a human all chess games and books in all history and then said 'play chess' and they still kept making illegal moves, they would have to have a brain injury.

harimau777 30 minutes ago | parent

It seems like an LLM potentially could learn that way if each practice game it participated in was added to its training data.

danielmarkbruce 34 minutes ago | parent

Humans wear a lot of hats when the do work. They don't even realize how many. My experience with building real systems using LLMs is that you have to be very explicit about such hats and you don't realize how many are worn until you see edge case after edge case after edge case. Check this. Check that. Check this. Check that. Check check check.

carodgers 53 minutes ago | parent

This April 2026 paper is a fun and related read.

https://arxiv.org/html/2509.24239v4

Researchers asked frontier models to play chess. Have a look at the MAR rates in Table 3. When not explicitly told which moves were legal, no model identified legal moves at a rate better than 80%. Many asked for more illegal moves than legal moves. And even when explicitly told which moves were legal, the models continued to ask for illegal moves. With illegal asks discarded, none of the bots could beat a chess model calibrated to 1100 ELO.

The author of the originating post says that "current frontier models need laborious oversight and guardrails on even the simplest tasks", and he's absolutely correct.

threethirtytwo 46 minutes ago | parent

The story isn't so clear cut.

The caveat is: It depends on the task.

Are there reams of chess moves that the model can train off of? No.

Are there reams of math papers the model can train off of? Yes.

tjwebbnorfolk 32 minutes ago | parent

> Are there reams of chess moves that the model can train off of? No.

This is as false as something can possibly be. There are open databases of millions of chess games spanning hundreds of years.

XenophileJKO 23 minutes ago | parent

It is even worse.. This is a classical reinforcement problem where data generation is easy because the rule set is pre-defined. So you really don't even need any data to start with (but would help).

manquer 1 minute ago | parent

[delayed]

vmg12 29 minutes ago | parent

> The caveat is: It depends on the task.

I think the line of criticism around LLMs sucking at chess makes more sense when you understand what the AI companies are saying about the future trajectory of these models.

The entire recursive self improvement story falls apart once you point out that there is not much "cross domain transfer learning". Meaning that training an LLM to become good at coding, math, etc, will eventually transfer into them being good at other skills that were not explicitly trained for.

Using games like chess which have little economic value is actually a good test for this. What's even more surprising about chess is how much information about chess strategy exists in the training data.

freejazz 24 minutes ago | parent

>Are there reams of chess moves that the model can train off of? No.

For real??

FuckButtons 14 minutes ago | parent

There’s multiple databases of games in algebraic notation. You can also, very easily rl train on pitting models against one another, even without mcts.

wat10000 38 minutes ago | parent

I wonder how current models would fare. The ones they tested are fairly old now.

joefourier 37 minutes ago | parent

> current frontier models

> Gemini 2.5 Pro, O3, Claude Sonnet 3.7 and ChatGPT 4.1

The gap in capabilities between those models which they tested, and actual current frontier ones is enormous. I would not trust that any conclusions they made are applicable.

sigmoid10 29 minutes ago | parent

The actual current frontier plays somewhere around GM level.

https://chessbench-ai.github.io/#leaderboard

It's also worth noting that the very latest models (GPT-6 and Fable 5.1) actually play worse than their immediate predecessors, so it is likely that the labs are not benchmaxxing for this yet. If they did, I'm sure they could come up with something superior to humans. But there is probably very little demand for this compared to IT stuff.

htrp 16 minutes ago | parent

more like you lose intelligence in chess by maxing for coding... hence knocking back the claims of emergent intelligence

csande17 13 minutes ago | parent

Even if you take that website at face value, the ELO scores shown are relative to the other AI models tested, and not comparable to the ELO scores of humans who play against other humans.

einszwei 9 minutes ago | parent

Probably tells us that without labs explicitly training/tuning the models or designing the harness (with fast oracle) the LLMs aren't going to get good at those areas.

minraws 3 minutes ago | parent

I know HN readers and posters just read numbers and can't be bothered to read, but please read the methodology before making any claims.

> About their ELO ratings from their own website:

> A field-relative rating calculated within ChessBench. It compares performance among the tested models and is not a direct equivalent of a human chess rating.

I am around 1600 elo in over the board I can mop up Astra Fable etc even if I give them literal infinite time and all the subagents and internet access..

Please folks at least use your AIs to read stuff before making claims.

AI is not GM level, it's not even 1600, I am 1600 by using memorized openings people frequently fall for with very basic intuitions.

A GM is 2600 they can beat me in under 20 moves...

Why do I even scroll through this website. For a moment I truly felt fooled, but then I read like a human should.

Maybe I should stop doing that will be a happier life, don't think just believe in the AGI.

consensus1 29 minutes ago | parent

This isn't how intelligence works. The LLM may not be able to play chess directly through inference, but it can write a program to do it and execute that program. Same as how human intelligence works. We can't fly, but we can build planes.

nefarious_ends 23 minutes ago | parent

Thanks for saying this, feels like everyone has gone insane over this stuff.

thesmtsolver2 15 minutes ago | parent

Human beings can play chess directly without coding up a tool.

famouswaffles 23 minutes ago | parent

Frontier labs don't care about chess. If OpenAI cared, GPT-7 could be a grandmaster+ level chess player. In fact there's a google paper on grandmaster level chess without search with a 270M transformer. Outside that, there was gpt-3.5-turbo instruct which was incidentally a 1800 lichess elo player that didn't make any illegal moves even after a few thousand moves. Frontier labs care deeply about automating knowledge work and computer use. They are working hard on getting models better and better, and they are succeeding. Astra is a step change on that front. So good luck i guess, if chess performance is your barometer.

bigstrat2003 9 minutes ago | parent

> Frontier labs don't care about chess. If OpenAI cared, GPT-7 could be a grandmaster+ level chess player.

If the models were actually intelligent, the way that the boosters claim, they wouldn't need to be tuned to play chess in order to be good at it. That's kind of the point of intelligence, that it is generically applicable to whichever task one wishes.

WhitneyLand 13 minutes ago | parent

1. It’s hard to trust a 2026 paper that’s showing results for such old models.

2. Chess seems to be a poor benchmark for generalized strategic reasoning. People who are good at it rely more on experience and deep domain expertise than on skills that generalize to make them experts at unrelated tasks.

3. The study sounds like proving humans will never fly because they don’t have wings. In reality, humans do fly, and Claude Fable would destroy any human at chess by coding a strong enough engine on the fly.

manquer 7 minutes ago | parent

[delayed]

knuppar 51 minutes ago | parent

Short and to the point! Open and cheap models will undercut the big labs continuously. The blast radius won't be pretty once spending commitments knock the door.

pvab3 32 minutes ago | parent

I agree with you but I'm still worried about the safety of open weight models as well. Both aligned and unaligned models.

m3kw9 15 minutes ago | parent

You said it like labs like open AI doesn’t know and don’t constantly make moves to prevent that undercutting

aogaili 47 minutes ago | parent

good post/take.

baceituno 46 minutes ago | parent

doomers gonna doom

war-is-peace 40 minutes ago | parent

refreshing to see amongst the endless tide of "i haven't written a single piece of code since 2025, llms are so good that they have already replaced everyone" gaslighting

Founderarcstone 25 minutes ago | parent

I am bullish on AI. At some point well see some true advancements.

vatsachak 17 minutes ago | parent

I agree with the caveat that it's more like a cracked junior engineer who can manage swarms of interns.

Frontier Labs will probably survive off hype valuations but will serve the important purpose of discovering architectures/techniques that will probably spread through rumors/transfers to the rest of the world.

m3kw9 16 minutes ago | parent

All website should come with a Summerize button.

MiroslavPokorny 12 minutes ago | parent

DO you know those ice cream shops that sell 30 different flavours. Everybody likes a different flavour, some people dont even like ice cream and buy nothing.

Some people will complain about the wrong flavours, or missing flavours, or the price, the long lines or maybe it closes early on fridays.

Summarise means different things to different people.