YHK - yet another hacker news clone

Fable 5 – Median thinking declined in August

43 points espeed 50 minutes ago 21 comments

alexjplant 27 minutes ago | parent

I seem to recall Anthropic going on record saying that they don't do anything to model performance to stretch their compute capacity. I've anecdotally noticed massive peaks and troughs in performance week to week (albeit with Opus, not Fable).

I wonder what their official explanation for this behavior is.

QwenGlazer9000 16 minutes ago | parent

Last time they were called out, it was a regression in Claude code itself.

At least that's their explanation. Either way, it wasn't a good look for "vibecoding" but it got brushed over.

Wowfunhappy 8 minutes ago | parent

When something is new, its capabilities feel incredible. Over time, those same capabilities become mundane, and you start to notice the flaws.

(Now, if TFA is actually measuring reasoning tokens, that's quite different! It's not entirely obvious to me how he is measuring.)

CamperBob2 24 minutes ago | parent

How do you measure thinking tokens? They don't send those back to the client.

ivanbakel 18 minutes ago | parent

They tell you how many tokens are used, however, right? Otherwise you couldn't see your own token consumption.

CamperBob2 16 minutes ago | parent

Good point. I suppose watching the number go up is useful information in itself.

I have been using CC with DeepSeek 4.1 Flash lately, and it's nice to see how the sausage is being made (even if it's partly illusory, as CoT always is.)

r2-129 19 minutes ago | parent

Obviously. The standard pattern is that model X is basically AGI and wins all benchmarks, followed the next day by Y and Z, which both win all benchmarks, too.

Then weeks later people find out that they have been duped and complain that the models have been quantized or employ worse inference.

Buy decent coffee instead of your $200 subscription and sidestep all the scams.

CamperBob2 11 minutes ago | parent

You forgot a stage or two:

1: "Our model will bring about the end of all things. Flee, flee for your lives"

2: "Our model is basically AGI"

3: "Our model will be available in limited release next week"

4: "Everybody who subscribes at the $200 level gets access now"

5: "Everybody who subscribes at the $20 level gets access now"

6, at least at Google: "Our model will be shoved down your throat every time you do a search, whether you want it or not"

artemonster 6 minutes ago | parent

0. "our model is too dangerous to release to pubic"

ltbarcly3 11 minutes ago | parent

Well I happen to enjoy coffee and $200 AI plans. What if Blue Bottle started watering down it's coffee? Is your answer to stop drinking coffee and make myself tea instead?

Evidence that vendors are being misleading in what they are delivering is important to share, whether or not you personally approve of that product.

mlmonkey 14 minutes ago | parent

Anecdotally, I have found the same. I spend a lot of time with these frontier models, brainstorming, etc. and the drop in performance from, say, week 1 to week 8 is often massive. Whereas in the beginning, it seemed like a capable research assistant, by the end of week 8 or so it starts acting like a puppy dog eager to make its 'master' happy for a few treats.

cloudking 12 minutes ago | parent

How do you create repeatable tests in a non-deterministic system? Every time you send the same prompt you get a different answer.

ssivark 10 minutes ago | parent

[delayed]

CharlesW 8 minutes ago | parent

This is a good overview of how this is done: https://www.anthropic.com/engineering/demystifying-evals-for...

llmslave 7 minutes ago | parent

I strongly believe that the real Fable is the one we had for a few days in June. Then they nerfed the model a bit after the government pulled it off the market. What we have now is something less, but still good

theplumber 7 minutes ago | parent

It is clear by now to me that Anthropic is constantly trying to find a kind of “auto” degradation perhaps to save money on work it thinks does not require high reasoning. I always use max reasoning and I can clearly see differences between the models when they release and after 3-4 weeks. I think they give a kind of intelligence boost also for new accounts.

jotato 6 minutes ago | parent

Just yesterday I was thinking about gpt-5.6-luna. I made it my default model in Hermes during its fist week of launch. It was just as good as 5.5 which was my previous default. But over the last 2 or 3 weeks I've seen how dumb it is now. I have to be very explicit with it.

For example, I used to be able to prompt "Check the system logs on <server> for...." and it would just figure it out. Yesterday I asked "Did <service> on <server> complete the overnight job" and all it said was "that service is not installed on my host"

I had to tell it to ssh into the server and run journlctl to check it

Anecdotal, I know, but they all seem to be less capable with time.

_edit_ I use the same reasoning level of `medium`