Yet another Hacker News

Vote on which of Hacker News' challenges for AI have been met

29 points stabbles 1 hour ago 28 comments

ben_w 1 hour ago | parent

Very pleased one of my predictions was totally wrong: https://news.ycombinator.com/item?id=23252711

Sure, sure, what LLMs make still isn't "efficient bug-free code": my prediction is falsified because while LLMs can write and train new models with machine learning, ML is fundamentally not advanced enough to throw arbitraty new tasks at like this.

FabCH 59 minutes ago | parent

Somewhat appropriate the site the OP links to is called „goalposts“ because as far as I can see, people keep shifting theirs.

In your case, the comment you link to says „business tasks“ and you expanded it now to „arbitrary new tasks“. Those are not the same. An LLM today sure can do many many many business-speak conversion tasks.

tripleee 44 minutes ago | parent

> An LLM today sure can do many many many business-speak conversion tasks

Not reliably, and not without supervision. That's the main point. I'm trying really hard to figure out a workflow that doesn't require me to review the code and I just don't see how it's possible (yet)

You either need a comprehensive test suite (which requires understanding the code in order to create) or you need to review the actual implementation code to make sure it does the right thing

Dylan16807 40 minutes ago | parent

You can't ignore the rest of the sentence. "every other task their business does" "everyone will be out of a job"

This means it has to handle basically all business tasks, so "arbitrary". I'm not sure what percent you have in mind by "many many many" but I would say it can't code half the things you need in an efficient and minimally buggy way.

ben_w 31 minutes ago | parent

I'm not always precise with my language, but business tasks can be pretty broad, I think "arbitrary new tasks" is not an unreasonable rephrasing on my part?

Consider I was replying to this:

> So are we all going to be out of a job?

While your boss now has the capacity to ask Claude to train a new AI model to auto-balance a tower defence game's mob, cost, and tower parameters (I know because I've done it), this only matters if you and your boss are working in a video games company.

If you and your boss are actually florists, you care if your boss can get Claude to automate a rose pruning, dead-heading, and fertilising robot.

People are trying, but I don't think they'd be happy with 91.5% success rate: https://www.emerald.com/ir/article-abstract/doi/10.1108/IR-0...

vlyan 48 minutes ago | parent

so the conditions for your prediction simply haven't been met yet.

if/when you can tell a model to do a thing and be confident that it did the thing, it's joever for 90% of knowledge workers.

ben_w 41 minutes ago | parent

The relevant condition was met; my misjudgement was that meeting it would require ML to be advanced enough to be able to train on arbitraty tasks from realistic (ie small) numbers of examples.

tripleee 47 minutes ago | parent

> reliably convert business-speak into efficient bug-free code

I actually think this would take AGI to solve, which makes me optimistic about the future of software development.

All the benchmarks are currently testing against automated tests the AI can use as an oracle

simianwords 50 minutes ago | parent

I made a bet with a guy on HN that the market value of OpenAI + Anthropic would get to at least 2.5T by 2027. I think I'm on track to winning.

https://news.ycombinator.com/item?id=48517353

I also made a bet that API inference margins are greater than 10% for OpenAI and Anthropic

https://news.ycombinator.com/item?id=48500827

I can make another prediction about Agentic Commerce and I think it will get big. Muse + Grok Bot + Dots.

mrweasel 28 minutes ago | parent

You're very lucky that market value and actual value isn't the same thing.

ngruhn 22 minutes ago | parent

I love it! Kinda wholesome how that heated discussion ended with that bet.

mcphage 12 minutes ago | parent

> API inference margins are greater than 10% for OpenAI and Anthropic

How do you measure that?

delichon 46 minutes ago | parent

If for each mistaken prediction there was some mild accountability, like someone shows up and slaps you with a trout, it would improve the site. But it should be added to the terms of service first.

Retr0id 45 minutes ago | parent

Alternatively, you can bet on your predictions. If you're wrong, you lose money.

bmenrigh 46 minutes ago | parent

At least 1/3rd of these predictions aren't clear enough to determine exactly what is being claimed/predicted. Even after reading the full comment multiple times, on a lot of them I couldn't tell where the author had set the goalposts well enough to say whether we've crossed it or not.

happytoexplain 40 minutes ago | parent

Right - people on HN are generally reasonable about objective things. The vast majority of comments (outside those chosen for this website) are not "AI will never ..." but rather, "AI does not currently ...". Of course the further you go back (I'm seeing a lot of comments from ten years ago!) the more skeptical they get, obviously. That's a funny thing to go back and see with modern context, but it doesn't really call for snideness/mockery (something I think is sadly increasing on HN).

jerf 35 minutes ago | parent

Well, I can answer this one: https://stoppels.ch/goalposts/?c=40662140

jerf, 2024: "If it could be solved with a Math Overflow-post level of effort, even from Terence Tao, it isn't what I was talking about as "high level math".

"I also am not surprised by "Consider a generation function" coming out of an LLM. I am talking about a system that could solve that problem, entirely, as doing high level math. A system that can emit "have you considered using wood?" is not a system that can build a house autonomously.

"It especially won't seem all that useful next to the generation of AIs I anticipate to be coming which use LLMs as a component to understand the world but are not just big LLMs."

The voting gloss: "An AI fully solves a research-level math problem on its own, not just suggesting an approach."

Yes, I'm satisfied. I don't even feel bad in hindsight. Coding assistants had a nice, gradual rise up the utility curve. Math went from "lol, can't add two six-digit numbers" to research-math level almost overnight in comparison.

pitched 31 minutes ago | parent

> cannot do precise things like coding software since humans will never be able to use natural language to specify their requirements.

To your point, this example. The issue expressed here is with humans, not AI. We are still pretty terrible at writing specs. TBF, the AIs are too but that wasn’t being voted on.

travisgriggs 41 minutes ago | parent

How was this assembled? From a meta point of view, how much AI was used to curate and highlite the goals; how much was used to assemble the site itself? Or deploy it?

ErrantX 32 minutes ago | parent

What is interesting to me is in 2016 people were like; pass Turing test, write code, order me a coffee.

And even in 2024 the themes are similar, generally more complex or specific about the coding/turing/action test.

But in 2026 a huge shift, we have things like; can open a physical door, emulates human pettiness convincingly, makes novel scientific breakthroughs.

That alone tells you a lot IMO

simianwords 31 minutes ago | parent

Here's a compilation of HN commenters that never believed that AI could solve Millennium problems (or same in spirit)

> I have a couple friends who did the Math tripos at Cambridge (so a pretty high level!) who work in tech and have unanimously said they have 0% expectations of an LLM doing a millennium problem anytime soon https://news.ycombinator.com/item?id=38433655

> Let's talk when we've got LLMs proving the Riemann Hypothesis (or any mathematical hypothesis) without any proofs in the training data. I'm confident in my belief that an LLM can't do that, and will never be able to. LLMs can barely solve elementary school math problems reliably.

https://news.ycombinator.com/item?id=42331654

> An LLM is like a well read college student with a nearly photographic memory that sometimes mixes things up. It's great for bouncing ideas off of and getting feedback on them. And yeah, it might product "novel ideas" by mixing and matching existing ideas, but LLMs will never create truly novel ideas. Not in their current form.

The paper didn't really answer the question sadly: their conclusion was just that humans rate LLM answers as more novel than human ones, but less feasible.

https://news.ycombinator.com/item?id=41522605

> Solving Millennium problems is a whole different ballgame. It's not known if these problems are solvable within ZFC axioms. (In one case, the Yang-Mills prize, stating the problem mathematically is part of the challenge.) All of the obvious applications of known tricks have been tried and failed. To solve such problems, one probably has to invent new and surprising mathematical definitions, building a framework in which the problem becomes solvable. This is something that LLMs will be crap at; the process of invention is not represented in any training data we have access to.

https://news.ycombinator.com/item?id=38435909

> LLMs cannot reason or use mathematics - in a way, they don't know what they are talking about. Why would such technology lead to superhuman smarts?

https://news.ycombinator.com/item?id=35752293

> But still, the questions in that test are "solved" in the sense of "I can take a dictionary and answers these questions with full certainty". Beyond established knowledge LLMs are monkeys with typewriters, at best.

> I agree but I have tried many times to intersect two ideas with a LLM that would be novel and the LLM can not do this at all. We shouldn't expect the stochastic parrot to be able to do this though and it is unfair to the stochastic parrot.

> It is like expecting a real parrot to say words it has never heard before.

> No one asks that of a real parrot because we don't anthropomorphize a real parrot like we do the LLM

https://news.ycombinator.com/item?id=41525962

mcphage 10 minutes ago | parent

> that never believed that AI could solve Millennium problems (or same in spirit)

How did that situation end up? Did it solve it on its own, or did it rip off another mathematician's work?

mrweasel 25 minutes ago | parent

The Turing test is interesting, because I believe that the current LLMs are perfectly capable of parsing the it in many situations. On the other hand we also have people are sound like they aren't real.

Looking back, was the Turing test flawed perhaps? It failed to take into account that humans can be rather bad at telling actual people from a "parrot". Turing was perhaps a little to optimistic about people.

Retr0id 23 minutes ago | parent

Heh, there's one of mine: https://stoppels.ch/goalposts/?c=39727943

"GPT-4 looks at original ASCII art of a foot, not copied from the web, and says it is a foot."

The vote is currently 64% yes, 18% no.

Just now I asked Opus 5.5 to generate an ASCII art foot, and it did a passable job. It's not great, but it's a foot. Then I pasted it into ChatGPT (whatever they're serving to the free tier by default, which seems to be 5.6 Luna), and it said it was a "train/locomotive": https://chatgpt.com/share/6abeaa39-cc80-83ed-851f-29370db089...

Maybe it's Opus's fault for drawing a bad foot but I think it's fair to say LLMs are still pretty bad at ASCII art (without additional tool calling etc).

tedsanders 11 minutes ago | parent

For me, 6.1 Sol nailed it immediately:

> A bare foot and ankle, pointing right, with three little toes.

I wonder how much of the wide variation in perceptions of LLM capabilities is driven by the gulf between free models and frontier models. Luna getting something wrong is not always great evidence for LLMs be unable to do that thing.

ben_w 7 minutes ago | parent

Case in point, this nonsense came out of 5.6 Luna: https://chatgpt.com/share/6abeae1e-42b4-83ed-8966-7e82ae0bef...

Like, is this an ice-cream? A tooth?

ben_w 11 minutes ago | parent

Mm, I kina agree with the AI on this one:

   (_)(_)(_) represents the wheels

They do look rather wheel-like; I have to assume you see them as toes though?

It's like the duck-bunny picture to me. If I focus on the "wheels", I see a steam train locomotive (but perhaps I'm only seeing that because I read your comment?); if I look at the ankle I see a foot.

howunfortunate 1 minute ago | parent

Readers: before you vote or comment, look at that foot.

I think I would have failed this test!