856 points matthieu_bl 11 hours ago 353 comments
neilellis 7 hours ago | parent
I use codex now.
AIorNot 7 hours ago | parent
apetresc 7 hours ago | parent
ceejayoz 7 hours ago | parent
I'm sure there's quite a bit of variation from person to person in these sorts of experiences, based on your harness, the way you talk, the stored memory, your CLAUDE.md, etc. But people absolutely have had this Opus 5 style experience the app simulates.
fg137 7 hours ago | parent
satvikpendem 7 hours ago | parent
ceejayoz 7 hours ago | parent
rcxdude 7 hours ago | parent
ceejayoz 7 hours ago | parent
But you can see in this thread that folks definitely have experienced this.
brazukadev 7 hours ago | parent
fractorial 7 hours ago | parent
appleappleapple 7 hours ago | parent
oujiii 7 hours ago | parent
RGS1811 6 hours ago | parent
jerf 5 hours ago | parent
This isn't a defense of it doing those four other things. It would be nice if it did what you wanted correctly. I'm just saying, as long as our programming skills have not completely atrophied, we have the power.
“Ford carried on counting quietly. This is about the most aggressive thing you can do to a computer, the equivalent of going up to a human being and saying "Blood...blood...blood...blood...” ― Douglas Adams, The Hitchhiker's Guide to the Galaxy
felixgallo 7 hours ago | parent
Toutouxc 7 hours ago | parent
dannypostma 7 hours ago | parent
pablopudding 7 hours ago | parent
matsemann 7 hours ago | parent
brap 7 hours ago | parent
ceejayoz 7 hours ago | parent
selestify 4 hours ago | parent
Surely you still have to give feedback to Claude. How do you do that without talking to Claude directly? By using a different model? But wouldn't that AI have no more common sense than Claude?
ceejayoz 2 hours ago | parent
Initial dev required talking to Claude. Now I add a ticket in the board, it makes me a mockup/writeup, I approve, and it gets me a temporary webserver, iOS/Android build, etc. to verify it.
Review loops, agents that enforce my pet peeves and testing/debugging processes, etc. all run automatically... and then Codex strips down the prose at the end. There's not zero AI generated output, but it's already been critiqued and verified by a whole cluster of independent actors before it gets to me. When I have feedback, I file a ticket.
I wanted to get out of the "what the fuck, why?!" loop. Now I let the agents handle that.
gonzalohm 7 hours ago | parent
Sohcahtoa82 3 hours ago | parent
In my experience, Claude Code is great at making a first-pass at a project, but once you start asking it to make changes, it explodes. A bug fix that should only be 2 lines turns into adding 3 functions totaling 100 lines.
Something as simple as "make the button blue" should be done manually.
andremendes 7 hours ago | parent
fractorial 7 hours ago | parent
dwringer 6 hours ago | parent
EDIT: Actually, I used to use Google's AI Studio a lot and fork it after every successful prompt interaction. When I'd encounter a problematic issue like this, I'd revert to the previous fork and try a different prompt until I could get the desired outcome, thus mitigating the need to "argue" with the LLM. Unfortunately the ability to cleanly fork and revert everything including the LLM context was removed some months ago, and I've yet to discover a workflow with any tool that works as well for me.
teiferer 6 hours ago | parent
I wonder if this is just a reflection of some senior folks being arrogant towards junior folks. When the latter finished a task but not to the liking of the senior person they might just get a "that's wrong, try again". Just to have sth similar repeat the second time around. But the arrogant guy got to boss around the junior one, and some junior folks grow up learning that's how you should behave so they also do it later.
Now it's not a person but a machine. And people just make fun of the dumb machine. Well, garbage in, garbage out If you are not specific in what you want, you might get crap back. Or at least sth you didn't envision.
selestify 4 hours ago | parent
teiferer 32 minutes ago | parent
burnoutdv 7 hours ago | parent
8cvor6j844qw_d6 7 hours ago | parent
kaoD 7 hours ago | parent
My gripe with Claude is that while investigating how to do this it will report 200 other incidental findings which I overlooked and I realize those are broken too and need urgent fixing, derailing me, not it.
drcongo 7 hours ago | parent
cub-creature 7 hours ago | parent
Also, while this little demo is certainly exaggerating the issue, I do find working with Claude to sometimes get quite verbose and tiresome. I doubt I would struggle this much to get it to change a button color, but the patterns of speech, the endless lists, the over-explanations, and the whole song and dance of trying to get it to make the change you want without side-effects is frustratingly familiar to me.
empath75 6 hours ago | parent
techscruggs 7 hours ago | parent
swiftcoder 7 hours ago | parent
kstenerud 7 hours ago | parent
inknight 7 hours ago | parent
dominotw 7 hours ago | parent
agluszak 6 hours ago | parent
raincole 6 hours ago | parent
kristofferR 6 hours ago | parent
llm_nerd 6 hours ago | parent
badrequest 6 hours ago | parent
lancebeet 6 hours ago | parent
827a 6 hours ago | parent
TheRealPomax 5 hours ago | parent
I have misbehaved in this fashion for many people across the full spectrum from casual users to highly experienced software engineers with millions of social media followers, so the statement that it's similar to making a joke about a calculator misadding two numbers because a stray beam of solar radiation flipped a bit at least for my part is not true.
Would you like me to start using bad English and doing things you never asked me to for your sessions, too? Just say the word.
stymaar 5 hours ago | parent
KaiserPro 5 hours ago | parent
The user is asserting that this is "not a lived experience"
working out if the user has missed the LLM Cliche
Formulating a suitable response
"did you just argue with a joke?" No, thats to blunt
"Is it satire when it only applies to you?" no again passive aggressive
lets just print the dictionary definition of satire and hope the move on
> Satire is when...
crazygringo 5 hours ago | parent
I do remember that one of the first things I did with my CLAUDE.md was to tell it to stick to the scope of the task, never to jump ahead and do extra "helpful" things without confirming with me first, and to follow best software practices including around refactoring but also to specifically avoid overengineering. I don't know if that is what's giving me a different experience from whatever the author seems to be "satirizing".
jaggederest 4 hours ago | parent
But with current models you actively have to sabotage the context to get this kind of behavior, or dramatically underspecify (3 words versus 2-3 sentences)
eszed 3 hours ago | parent
AI isn't magic, and it won't (or, at least, doesn't yet) enable anyone to do All The Things.
tsukikage 3 hours ago | parent
827a 3 hours ago | parent
llm_nerd 6 hours ago | parent
To the people rather lamely doing the "it's satire/a joke", that would require this to be an exaggeration of a reality. But...it isn't.
invalidusernam3 6 hours ago | parent
catapart 6 hours ago | parent
madibo3156 6 hours ago | parent
My impression is that this is an oversimplified demonstration of what can happen when you prompt Claude in a system with many more variables (than two buttons and two colours).
If I want the button to turn blue and that's it, what instead do I ask? Even in a complicated system with many levers, what do I request other than the desired end result, hoping that Claude pulls the right levers to produce something acceptably close to what I think I asked for?
crazygringo 5 hours ago | parent
And when it doesn't, it's usually because of things outside of the codebase -- iOS layout quirks that aren't documented, buggy Python libraries it's relying on where you then have to tell it to read the source to figure out what's going on, that kind of thing.
jaggederest 4 hours ago | parent
You have to change the system so that the AI understands it, via establishing what your beliefs are, how those are reflected in values (especially important if you have e.g. compliance needs), how those values are reflected in the operational and strategic levels, and then a variety of tactical behavior coaching. For example, I ban 2>/dev/null - super tactical, and I say I value simplicity over covering every edge case - a very broad generalization.
plorkyeran 2 hours ago | parent
If I don't understand why changing one button changed them all, I would use an entirely separate session to ask questions about how the site theming works. The fact that the site has a bunch of weird coupling between themes is useful information, and if I don't know how to resolve that I'd ask Claude for ideas about how to safely eliminate the coupling, and once it proposes a reasonable idea tell it to implement that.
The two big things here I'd never do is use emotional languages in prompts, and I'd never tell Claude to revert changes and try again. Once the incorrect change is in the context it's poisoning all of your future results.
mort96 5 hours ago | parent
Wow, you AI people really have a negative view of the technology y'all are trying to sell as the next Jesus
kstenerud 2 hours ago | parent
Once you understand what the problem is, you can give it better instructions. If the architecture is shit, the agent is going to have a rough time of it.
catapart 1 hour ago | parent
selestify 4 hours ago | parent
Bjartr 3 hours ago | parent
> Why is half the site blue now? I asked you to change one button.
> Half the site is blue. I asked for ONE button
Neither of these is an instruction to fix the problem, they're treating the AI like a person and telling it what it did wrong, expecting the implied admonishment to be enough to steer it back. But without an actual instruction, it just goes and does whatever it thinks will help, which is often arbitrary.
The response I would have used in this situation is
"The Cancel button is also blue now. Make sure the color change is only scoped to the Add to Cart button"
Most of the available responses throughout this "skit" are similar cases of expressing frustration first and guiding the result second.
Skip the emotion and say exactly what you want, and nothing besides that.
sunaurus 6 hours ago | parent
Usually just restarting the session helps, though.
whalabi 5 hours ago | parent
johnisgood 5 hours ago | parent
Be specific.
That said, GPT always acts up even if I am specific, but I only have the free tier there.
singularity2001 5 hours ago | parent
kstenerud 2 hours ago | parent
First thing I do when something goes wrong is tell the agent to stop and diagnose. You can't prompt effectively without three proper information.
iLoveOncall 5 hours ago | parent
Only since 4.8 though.
herrherrmann 2 hours ago | parent
wesselbindt 4 hours ago | parent
yieldcrv 3 hours ago | parent
cloverich 2 hours ago | parent
Ask for change A and get unwanted change B happens all the time with bad programmers and tradgedy of the commons (ie poorly architected, no restraint) codebases.
Most normies dont know about this stuff.
yonatan8070 1 hour ago | parent
jadar 7 hours ago | parent
random_cat_8745 7 hours ago | parent
captainbland 7 hours ago | parent
mysterydip 6 hours ago | parent
dpark 5 hours ago | parent
Everything in life is variable reward. You invite a friend over, they might accept or they might not. Drive to work, traffic might be good or might be bad. You ask a colleague to finish a task, they might do it or might not or might do a good job or might not.
Everything is variable reward. Is everything gambling?
howunfortunate 5 hours ago | parent
Sugar is the original point of the reward system!
Barbing 5 hours ago | parent
Do hard work (takes time), get dopamine for successful completion.
Find berries, taste sweet (hopefully safe), eat all, get calories. Doordash Krispy Kreme instead = few too many calories.
(I’m no Luddite in the sense popularly thought of them pre-‘22 [1], though we have to watch skill atrophy)
[1] regressionist? Decelerationist, too loaded perhaps. Someone remembers or knows the word…
utopiah 5 hours ago | parent
smugglerFlynn 5 hours ago | parent
dpark 4 hours ago | parent
pixl97 3 hours ago | parent
rnjesus 5 hours ago | parent
regarding your examples, i think the difference is that with ai, you’re literally sitting in front of a machine, pressing a button, and (almost instantly) getting a result that, if not desired, can immediately be tried for again. you even spend “tokens” to do this, and at least in my native language, “token” brings to mind the coins you’d stick in a slot machine
dpark 4 hours ago | parent
If you sit at a slot machine and pump quarters into it, each “turn” is independent. You spin and you win or lose. It’s pure chance and there is no destination. You execute the exact same action over and over and hope random chance brings you more money.
If you sit down in front of a coding harness, the progress is incremental and directed. You ask for a thing, the LLM produces something that is hopefully close to what you wanted. You give it more direction to prod it closer to the end state you want. You are not executing the same action, but incrementally nudging it in the right direction. I’ve literally never restarted from the same initial state with the same prompt and hoped for a different result and I don’t know why anyone would. Rarely I’ve thrown away the progress made and started over but always with a very different prompt that includes learnings from the failed attempt.
rnjesus 3 hours ago | parent
most people don’t use ai this way though, and i still feel like the end-psychological reward mechanism is very, very similar to gambling regardless of how well one utilizes it (and this is even more obvious with image generation as you chase that perfect output)
perhaps it’s better to compare it to gacha than slots?
dpark 3 hours ago | parent
How do they use it? Surely no one is just repeating the same prompt over and over (except as a Ralph loop perhaps, which is automated). I’m really struggling with the notion that most people just throw the same prompt repeatedly hoping it eventually works. Because that doesn’t sound like gambling. It sounds crazy (and frustrating).
> and i still feel like the end-psychological reward mechanism is very, very similar to gambling regardless of how well one utilizes it
In the sense that you get a dopamine reward when you succeed, sure, but I get the same reward when I code by hand and achieve a successful result.
> and this is even more obvious with image generation as you chase that perfect output
This is fair, because sometimes with image generation the same exact prompt will produce very different output. This is becoming less true as the models get better and it becomes more effective to direct image generation iteratively than to keep starting from scratch with a barely tweaked prompt.
qlte 3 hours ago | parent
And sometimes the debugging has already veered completely off course at the beginning so it's futile, but each time the fleeting hope that just a few thousand more tokens will magically fix it tempts you to keep going a little longer.
rnjesus 3 hours ago | parent
as an example, i was using chatgpt a few weeks back to help me remember the name of a painting i’d seen about a decade ago. i could recall the general shape of the subject and that it was europeanish, but nothing else. after seven turns or so it finally got it, and honestly, the relief of finally remembering the name felt like, well, hitting a jackpot
i’ve had a similar feeling of success after trying to get it to give a comprehensible answer when asking it for a solid counter-argument to philosophical questions. it is indeed often crazy and frustrating
> but I get the same reward when I code by hand
i have only done very simple coding work with llms, so that may be why our ideas differ about the feeling of reward. this is where the comparison to gacha makes more sense than slots, since when you’re coding, you still get a reward each turn whilst chasing the final/desired result
dpark 2 hours ago | parent
I understand the joy of success but I fail to see how this is gambling. I could have an equivalent conversation with a friend (more likely about a movie in trying to remember than a painting, but still) and get the exact same type of iterative “no, not that one, it was more like X” and feel elated when my friend finally realizes I’m taking about a scene from Hot Tub Time Machine.
This isn’t gambling in any meaningful sense.
rnjesus 21 minutes ago | parent
i wasn’t trying to convince you, i’m just explaining that this is gambling in a meaningful sense to some people, especially with how turn-based and unpredictable the whole system is. whether it is actually gambling (semantically, legally, ontologically?) isn’t really interesting imo. what’s interesting is that it’s structured similarly and feels nearly identical to some people
but then, i also feel like there’s an element of gambling in the example of you talking with your friend, though i think it would be better illustrated if it was a conversation with a random person
Tadpole9181 3 hours ago | parent
Not everyone has the desire to work around the system, and many are diametrically opposed to the concept of AI. They get this perception that it's a slot machine because of that inconsistency, and then do the human thing of assuming that other people must just be flawed if they're different from them. They're "addicted to gambling".
Obviously, things have changed. Open models can still be like that, but are often so fast and cheap at iterating it doesn't matter. SOTA models aren't perfect, but are to the point that they're generally much better than the average developer.
But once that perception set in and the meme spreads, it's really hard for some to break out of it. Especially at the pace AI development has been moving. It's just that simple.
jodrellblank 3 hours ago | parent
well, no. If you work overtime and get paid overtime, you are not gambling and that is not a variable reward.
Humans engage more with rewards that are intermittent and variable. Like Futurama's scene from 'The Scary Door' where the character says "A casino where I'm winning, I must be in heaven! A casino where I always win, that's boring, I must really be IN HELL!". A constant predictable reward is boring, less engaging. So if you know you get no overtime, but sometimes your boss rewards you with $5 coffee voucher, sometimes a free pizza dinner, sometimes double-time pay for the time worked or a half-day off, now you might be gambling 1hr overtime for an intermittent variable reward.
> "Drive to work, traffic might be good or might be bad."
Good traffic is not a "reward" for driving to work(!) and you have to drive to work regardless so you are not risking anything [you might be risking your life, but you are not making a choice which can reward you with good traffic]. You might say that going a different route is a choice and a gamble which could reward you with good traffic, but traffic engineering does not work that way because if there was a consistently low-traffic route, everyone else would take that route until it was no faster than any other route. Traffic will generally be the predictable and similar every day, plus 'arriving at work early' is not much of a reward.
dpark 3 hours ago | parent
Perhaps but predictable outcome is a very desirable quality. No one wants a hammer that sometimes drives nails and sometimes doesn’t. All of the current harness engineering work is about squeezing predictability out of the LLM.
> Good traffic is not a "reward" for driving to work(!)
Like hell it’s not. I drove into work last Friday and there was no traffic because of the holiday weekend. It was amazing. Had me considering whether Friday should be one of my standard RTO days.
jodrellblank 49 minutes ago | parent
In what way was amazing no-traffic "a reward"? What system was rewarding you for what change in behaviour?
smugglerFlynn 5 hours ago | parent
tiborsaas 4 hours ago | parent
raincole 4 hours ago | parent
jimmaswell 4 hours ago | parent
I got so sick of all this at some point that I slowly stopped doing anything that wasn't my job. But then AI got better and better and I realized it was the ultimate unblocker. When that dreaded malaise started creeping in signaling it was a project's end because I didn't want to waste any more of my life dealing with bullshit orthogonal to what I was trying to do, I'd give it to the AI. It felt like a miracle the first time this worked, and it still does. If we were previously equipped with shovels to dig through bullshit, we now have a fully automated Bagger 288.
The reward schedule now isn't variable anymore; the chance that I finish something in a good state is 100%. I can focus on the parts I actually enjoy - architecting the broader system, making the parts mesh together in a sensible way that's easy to work with and has some mathematical elegance to it, hand coding the bits I want to be really specific about (but now without the endless frustration of bugfixing or import errors and edgecases being immediately discovered, thanks to the AI).
supern0va 4 hours ago | parent
I was working on a side project recently. I had spent months designing the data model in my spare time, thinking through how to make it as elegant and durable to change as possible in the long term, since (if I launched it) the repercussions for getting it wrong would be significant.
Once I had a working design, it probably would have been several more months to build a working prototype and start testing it.
Instead, Claude knocked out the prototype for me in an afternoon. And it immediately became clear that it didn't work: not because the data model didn't solve all the problems I wanted it to solve, but because it didn't fit the shape of how I quickly learned a normal person would need/want to interact with the product. I was so focused on the long term, that I never thought about what the first five minutes of a user with hands on the thing would need. And the changes needed would be significant.
Maybe there's some variable reward mechanism. But I sure was glad to be able to pull that particular slot machine handle and learn that than waste even more of my time on what was a dead end.
bevr1337 4 hours ago | parent
At this point, what do the words even mean? Your own patience and available time are always completely random and fairly distributed across a large enough data set?
> Maybe I'd waste hours down the wrong rabbit holes trying to find a library that worked for my use case. Maybe I'd waste a day trying to get an API to do something it turned out it couldn't do. Maybe I'd have to redo my entire approach because of some factor I hadn't considered.
Our ignorance isn't random chance. As we research and experiment, we reduce the problem area.
jimmaswell 3 hours ago | parent
Predicting the time a task will take is impossible. Something that sounds like a 5 minute script can turn into a month of banging your head against unknown unknowns. I lose my patience when the afternoon I allocated is getting overrun by nonsense and I'm missing out on other things I wanted to do or household maintenance.
> Our ignorance isn't random chance. As we research and experiment, we reduce the problem area.
Every thought we have has random chance to be wrong despite our conviction that it's correct. Descartes' Evil Demon plays his tricks on all of us. How many times have you typed some line of code only to realize it was obviously wrong afterwards? Even for simpler matters we "hallucinate" all the time. I was deep in thought trying to help someone come up with an acronym the other day and felt convicted that "Goal Oriented Augmented Retrieval" worked for GOAL until I said it aloud.
Our thoughts and actions are consistently wrong some portion of the time because our meat computers are not perfect positronic brains running prolog. We put cereal in the fridge and say "you too" to the waiter. Every thought we put down or action we take is a gamble on the soundness of the thought or action.
thepasswordis 4 hours ago | parent
andai 7 hours ago | parent
syntaxing 7 hours ago | parent
This hit a bit too close to home. Sol has the same issue, spawns a lot of agents for no good reasons (besides burning tokens).
chrisgarand 6 hours ago | parent
I don't know if it works, but I've always had a consistent level of token burn on my plans (I've only heavily used Sol after adding it).
r_lee 7 hours ago | parent
dsign 6 hours ago | parent
I use Opus and Sonnet 5 all the time and I find their language grating. But honest, I prefer to put up with it and get the results than to put up with my own human limitations and not get the results.
chrismorgan 6 hours ago | parent
(I am genuinely uncertain, though I presume it’s at least somewhat exaggerated.)
dd8601fn 6 hours ago | parent
It’s funny because there are elements of truth in each bit of it, though.
retsibsi 6 hours ago | parent
It's a pisstake, but (in the bits I read, and based on my own personal experience) the writing style is barely exaggerated, while the behaviour doesn't ring true at all.
selestify 4 hours ago | parent
I am guessing you prompt differently than what is shown in the game?
mrguyorama 3 hours ago | parent
I've worked with Opus and Sonnet daily, and they are pretty great at generating functions and modules and components that have clear boundaries of concern, but I've recently been working some research tasks into our infrastructure and code and it seems impossible to coerce Sonnet into making only specific changes to a document you are working on.
It also blatantly ignores instructions as a rule. "Don't disassemble java class files, just ask me to pull in the source code" worked less than half the time. The Intellij Copilot plugin just doesn't use the AGENTS.md and similar files, and there doesn't seem to be any meaningful activity in the bug reports of same. "Don't modify code unless I tell you to" had bad adherence as well.
It also will read documentation and inform you that it says the opposite. This problem happened to me across models, across model updates, across months of real time. There's a specific example that I will not mention to avoid having it be trained on specifically. A distinct but similar problem is that it will take bad documentation and just pretend it has a good understanding. Claude gave me absurdly wrong descriptions for Splunk alert settings with absolute confidence.
I don't think any agent can reliably figure out "I don't know"
wildzzz 5 hours ago | parent
theandrewbailey 4 hours ago | parent
t43562 3 hours ago | parent
Foobar8568 4 hours ago | parent
It's a sad parody that people will take as reality.
vant 6 hours ago | parent
airstrike 6 hours ago | parent
alentred 6 hours ago | parent
This got me on "cyanide blue", and I was ROLLING ON THE FLOOR LAUGHING on "Approaching usage limit". I can barely stop laughing now and my stomach hurts. I mean, Thank You!
100percentjake 3 hours ago | parent
tamimio 6 hours ago | parent
GracefullyShot 6 hours ago | parent
almostdeadguy 6 hours ago | parent
> I'm collapsing this back to the rendered outcome:
And intermixed with SaaS product page idioms from a brain-damaged marketer like:
> No broader cleanup.
> No further architecture work.
> Just the button.
Aside from the patterns everyone knows like em-dashes, "its not X, it's Y", etc. I think the key features of claude diction is it sounds like a junior engineer over their skies who is trying to make up for that with extra verbiage mixed with extremely grating SaaS marketing-ese.
telesilla 6 hours ago | parent
dwedge 6 hours ago | parent
alex_c 6 hours ago | parent
btown 5 hours ago | parent
SoftTalker 5 hours ago | parent
barbazoo 4 hours ago | parent
jodrellblank 4 hours ago | parent
iririririr 3 hours ago | parent
econ 3 hours ago | parent
shmeeed 7 minutes ago | parent
jodrellblank 4 hours ago | parent
If we each stop arguing with racists because we're each tired of it, then what we are is a society that lets racism go unopposed, a site with racist comments proliferating, a site where racists gather because they are able to be themselves, a site where future readers and AI training is learning that racism is popular, common, normal, and acceptable because everyone is accepting it.
It's a bit like democratic voting, your vote doesn't matter and cannot change things, but it's important that you vote because all of our votes do matter and can change things.
It's a bit like the quote "Dear Board, I don't want to belong to any club that would have me as a member. Sincerely yours, Groucho Marx.", I am upset and offended every time someone calls out my dumbass comments, but horrified at all the countless times nobody does. Why would I want to waste my time on a site with such low standards that it lets me comment? I'm exhalted and encouraged everytime someone upvotes or engages with my low effort quips[1] but thrown into a chasm of despair that HN users let that pass unopposed. When nobody engages with comments I spent literal hours on[2] I curl up into a ball and die, but also dream of it being a feather in my cap with dang that I'm posting a substantial comment and that will be its own reward. I identify with nobody more closely than Colonel Cathcart, and I will be delighted if someone engages with my Catch 22 reference, but aghast that such a nontechnical reference gets engagment; distraught if nobody engages but delighted that perhaps HN standards are being held higher.
If not you, whom?
[1] https://news.ycombinator.com/item?id=49478355
[2] https://news.ycombinator.com/item?id=49586378 or https://news.ycombinator.com/item?id=49491182
fallingbananna 5 hours ago | parent
Except... the next day they are still there and I have to push myself a bit further through the wall of changes before closing the tab again.
RobKohr 5 hours ago | parent
pc86 4 hours ago | parent
zamadatix 4 hours ago | parent
gedy 4 hours ago | parent
8note 4 hours ago | parent
then you are maybe reviewing 1 out of 7 PRs that implement the agreed upon change
gwbas1c 6 hours ago | parent
- The people who won't make any effort to learn the tools, and something as simple as reverting code (via git) needs to be done by AI?
- The awful programmers who we've had to endure working with, who are so bad at simple changes that they have negative productivity?
- Or Claude itself?
---
BTW: I don't have these problems, but I'm also not afraid to do things myself when it's easier.
Edit: If I want to change a button's color, I just change it manually. If I don't know where the code for the button is, I might start with prompting, (because AI can often find the code faster than I can,) and then once the diff is proposed, start adjusting things by hand.
kmoser 6 hours ago | parent
gwbas1c 48 minutes ago | parent
dmd 6 hours ago | parent
_fat_santa 6 hours ago | parent
Once in a blue moon it's actually the model making a material error in it's thinking and I have to go back and redo it.
arnorhs 6 hours ago | parent
tarxzvf 6 hours ago | parent
taeric 6 hours ago | parent
skinfaxi 5 hours ago | parent
autoexec 3 hours ago | parent
nemomarx 5 hours ago | parent
jaggederest 5 hours ago | parent
Do you know anyone who actually reads and adheres closely to all of the documentation every time it's changed?
bluefirebrand 5 hours ago | parent
wccrawford 5 hours ago | parent
But now that I've been working on it a month and there's a lot of documentation, it's pretty clearly ignoring parts of the documentation and parts of the code. It will come up with some ridiculous statement about how something works, and I'll challenge it, and it'll admit I'm right.
It definitely reads more documentation than any programmer I've ever worked with (myself included) but because it doesn't have a memory other than the documentation, it still makes mistakes like that.
I haven't turned on "memory" or tried it with Codex, so I don't know how that'll change soon, though.
jaggederest 4 hours ago | parent
# See DOC-ITEM-NAME
DOC-ITEM-NAME.md
When referencing documents, always use the exact syntax See <TAG> - this is enforced by a lint on precommit
And those doc items are basically all of the values, architectural, strategic, and tactical items. It's a poor man's in-repo RAG but it's shockingly effective, especially if you keep them small. I may migrate some/all of them to skills over time, but I usually update them biweekly, and I only allow agents to make small edits or propose new notes. And typically I go through and delete or curate any agent edits before merge.Depending on language I've seen this scale past multiple millions of lines of code, as long as you pair it with all of the linting and tooling that you can possibly build.
voakbasda 5 hours ago | parent
lathiat 5 hours ago | parent
https://www.patheos.com/blogs/tippling/2013/11/14/post-hoc-r...
BurningFrog 5 hours ago | parent
jameshart 2 hours ago | parent
supern0va 4 hours ago | parent
astrobe_ 3 hours ago | parent
CookieCrisp 3 hours ago | parent
Lord-Jobo 2 hours ago | parent
the inconsistency of LLMs is by far one of the biggest gripes I have with them. Closely related to their apparently deep desire to avoid following instructions.
I know these are both a byproduct of noise (which is somewhat tunable) and noise is inherent to these systems in a lode bearing way.
I still hate it. it’s holding the technology back. I don’t honestly see how we can safely or even successfully approach the idealized realm of AI without bypassing this problem, which to my understanding, probably means not using language models at all and trying a totally different approach. But I really don’t know much about machine learning, I’m a super novice compared to a lot on this website.
supern0va 2 hours ago | parent
Sort of, actually. I think we humans actually have some intuition that we'd be more effective if our cognition were augmented more directly by machine strengths: the ability to run precise calculations, more memory, ability to look facts in some sort of knowledge graph.
I think we're on the right track, but instead of augmenting humans with machine strengths, we're building intelligence in hardware in a way where it can access that augmentation. Plus, then we can quickly distribute updates, run parallel instances, etc.
If intelligence is compression, and hallucinations are essentially loss, then as the models grow in size performance (at least as far as hallucinations) should reduce. Or we'll get things fast enough that we can afford to stop relying on model weights for memory and check an increasingly larger set of discrete facts as part of reasoning.
Right now, the models are making trade-offs. As compute grows, and inference gets faster, we can make fewer of those trade-offs and start to use the unique strengths of machines to fill the gaps we're seeing, I suspect.
Lord-Jobo 2 hours ago | parent
When we do meta thinking about that process after the fact, two things happen. 1, we change our total “thought” by adding that meta thinking to it. And 2: it’s a very lossy process, because we don’t have very good data about what our brain or mind was actually doing during that first think and emotional factors are nearly always at play and even more complex.
Now for the more complex AI, the fragmented process of multiple agents and loops and reruns are pretty similar to that first think we do. At least structurally. But the meta think is where they differ. They have no emotion, but they also have even worse data about its own function. They constantly degenerate so I would argue their “changing the thought by thinking about it” factor is also generally way higher than ours.
Getting better at consistent/reproducible thinking, with many ‘steps’, that leaves good documentation of that thinking behind for future analysis, has to be one of the more important areas for the big flagships going forward. I’m certain that “what is this fucker doing and why” is the biggest pain point for AI researchers. Or the math, it’s usually the math.
But you’re correct in the general structure; they generally do the same post hoc analysis we do, just noticeably worse because of their opaque nature(even to themselves) and general degenerative instability.
bmacho 5 hours ago | parent
If they've worked some minutes/hours/weeks on something and you ask them why did they do that, they will either answer honestly and truthfully, lie, or say "I missed that/didn't seem important so I just chose something at random".
None of these cases are similar to how AI works.
yawnxyz 5 hours ago | parent
croo 5 hours ago | parent
bmacho 3 hours ago | parent
Sounds like the left hemisphere usually uses something from the right hemisphere to answer those questions and it can't do that if it's been cut off?
We know that our brains are capable of hallucinating due to substances (drugs), being asleep, brain damage (including split brain), hypnosis, etc. Just as RAM damage make your computer do weird shit. That doesn't mean it operates that way normally.
You can't just remove a huge part of a system and then assume that the whole system behaves the same.
supern0va 4 hours ago | parent
Yes, but the vast, vast majority of decisions you make either don't take place via an inner monologue, or include details that were not actively/consciously "thought" and reasoned with in your inner monologue.
And yet, when asked why you did something, you're not likely to respond "sorry, that decision was made subconsciously". Instead, you use your inner monologue to try to backfill in a reason why. That reason may be correct, or it may not be. You don't actually know, since you have new data that may be updating your own internal state as you try to rationalize it after the fact.
satvikpendem 3 hours ago | parent
Perhaps up to 50% of people actually don't have an inner monologue, much like many have aphantasia where they can't actually see anything in their mind either.
archagon 3 hours ago | parent
saghm 2 hours ago | parent
It's not clear to me whether you're aware of a rigorous basis for your claim or you're just inferring based on what you think makes sense, but I can't help but wonder if it's the later, in which case regardless of the mechanism, the outcome certainly seems to resemble what happens with LLMs.
bmacho 2 hours ago | parent
1. recall them correctly
2. say they can't recall them
3. recall them incorrectly
Are you asking me a study on when people say they remember something (cases 1 and 3) they are usually right or wrong?yonatan8070 2 hours ago | parent
While it did help the epilepsy, their brain was essentially two brains controlling two halves of the body. With one controlling speech. There were experiments where one eye was shown some instruction text, the corresponding hand performed that instruction, and when asked why they dix that action, the speaking half just made up some plausible, yet completely wrong reason, just like an LLM.
Terr_ 29 minutes ago | parent
If you build a robot where a pressurized hose leaks causing fluid to destroy part of the circuitry, we don't praise it as progress towards the human ideal of having brain aneurysms.
theluketaylor 6 hours ago | parent
embedding-shape 5 hours ago | parent
zamadatix 5 hours ago | parent
You can extend this further by using an adversarial agent trying to find mistakes in the other instance's logs in a loop where a 3rd neutral agent weighs the claims of the other two. This is also just another step in reducing error, it does not guarantee elimination of such errors. The latter is an impossible guarantee, even for humans.
bmacho 5 hours ago | parent
zamadatix 5 hours ago | parent
Kiro 5 hours ago | parent
jameshart 2 hours ago | parent
JeremyNT 2 hours ago | parent
malfist 6 hours ago | parent
blackenedgem 5 hours ago | parent
throwaway6977 5 hours ago | parent
malfist 5 hours ago | parent
embedding-shape 5 hours ago | parent
applfanboysbgon 4 hours ago | parent
This is fucking constant. I can't deny that this stupid tech saves time prototyping even with having to wrangle it, but it commits a fireable offense several times a day that no human would get away with and is obviously incapable of learning from mistakes in the way a human is. The only reason it's not fired is because it's a slave that works for no more than the cost to feed it.
throwawayffffas 4 hours ago | parent
Additionally the provided prompts are not what anyone who has used this things would say in either situation.
Sure you can ask it to make one button blue and it can easily make all buttons blue, but they quickly backtrack if told to.
barbazoo 4 hours ago | parent
reedlaw 4 hours ago | parent
inerte 6 hours ago | parent
I guess we are all PMs now.
inerte 6 hours ago | parent
Simple, right? But design guidelines prevent one offs, and no !important. So you create a CSS class, but you discover another element on the header declared itself as primary (the search icon or the sign in button). You talk to that team and they decided to scope what’s primary according to their component. To change the sign in button to grey now you need to talk with the growth team. Growth team wants to run an experiment but they’re backlogged, only next quarter. They say you can innersource, just need VP approval. VP says blue matches a marketing campaign that is about to go out, agency has already been hired. You can’t talk to the agency unless Legal approves.
So you leave the button gray, to revisit decision next planning cycle after you can align all stakeholders.
mring33621 6 hours ago | parent
vinc 6 hours ago | parent
jmartrican 6 hours ago | parent
Nevermark 6 hours ago | parent
For a moment I thought, wow, someone put a lot of work into creating this theme park of frustration.
Next: It would be so easy to create a faux-Claude like this.
Then: How hilarious to watch the transcripts of unsuspecting users in real time.
Finally: I began wondering if this might be relevant to all the redundant, unnecessarily preambled, sentence structure complexifying, indirect referencing, canned phrasing, ambiguity mining, analogy maxxing, over-wordy responses I have recently been getting from Fable...
arbirk 6 hours ago | parent
improbableinf 6 hours ago | parent
moralestapia 6 hours ago | parent
It made me stop using Claude at all. Codex has almost surgical precision, and I like that a lot.
(But nowadays I just use DeepSeek Flash. it does screw up but its cents so ¯\_(ツ)_/¯).
Kim_Bruning 6 hours ago | parent
mzajc 6 hours ago | parent
> WebFetch en.wikipedia.org/…/Cyan
> WebFetch en.wikipedia.org/…/Prussian_blue
> WebFetch www.colorhexa.com/16b8c4
Brilliant.
pohl 5 hours ago | parent
jezzamon 5 hours ago | parent
Do people really prompt AI like this? Multiple times the choice was either to yell at the agent, or ask it why it did something, neither of which are very fruitful lines to go down if you know what you're doing
bennettpompi1 5 hours ago | parent
johnisgood 5 hours ago | parent
> Half the site is blue. I asked for ONE button.
Those are my only options when the site is clearly not blue, two buttons are.
There is a reason for why I am much more specific than this.
plorkyeran 5 hours ago | parent
jaggederest 4 hours ago | parent
selestify 4 hours ago | parent
tsukikage 3 hours ago | parent
So leave emotion at the door and make your words count. Voice your frustrations at the stress doll next to your monitor, sure, but spending tokens on them us just wasting time and money. Explicitly typing out your exasperation will not get you closer to whatever it is you are trying to accomplish. What can you type that will?
Laser focus: clearly, concisely explain what is wrong right now, and how you want it solved. Then do that again until all the moles have been whacked. If the AI is stuck in a sycophancy loop, start a new clean session. Do this frequently anyway: every turn in the same session charges for all preceding conversation again /and/ degrades the LLM's performance.
jacobgkau 33 minutes ago | parent
I've never explicitly paid for AI and never will. I only use the free Claude, etc. So I'm happy to waste THEIR electricity telling their machine how much of a piece of garbage it is and hoping that sentiment gets back in some form to the dumb humans making such a dumb product. It's not like they don't evaluate performance with usage data.
(Afterwards, I do also make sure to actually click the "thumbs down" or whatever equivalent button, and submit a report if it's possible.)
zeeveener 3 hours ago | parent
inerte 3 hours ago | parent
My best takeaway from the site was: Every time you see Claude Code / Codex doing something you don't want, ask it to add to AGENTS.md. Specially if they churn a lot doing something, and eventually find a way. I ask it to store the way it made something work. It's a constant gardening of AGENTS.md and it has really working well.
jannyfer 2 hours ago | parent
ChucklsTheBeard 1 hour ago | parent
Non-exhaustive consequences of following that rule:
* Each session handles one task, or at most, a few tightly coupled tasks.
* Emotions stay out of context; no showing frustration, no saying thanks (or at least wait until you're about to end the session)
* When possible, provide relevant files (or sections of files) instead of making the model search and read many irrelevant files.
* Keep CLAUDE.md short.
* Disable irrelevant tools.
* Revert history when the model makes mistakes. Don't make it read its mistake and fix it; fork the chat before the mistake and exclusively mention the correct action.
Attention is more limited than the context limits imply; stuff at the beginning of context stays high-attention for a while, stuff right at the end is always high-attention. If you're about to ask for something the model frequently forgets (eg, style), remind it that those instructions exist ("Following the style guidelines, implement feature X.")
iLoveOncall 5 hours ago | parent
t-writescode 3 hours ago | parent
Either that, or your coworker themselves would laugh and tell you about it later.
That’s also where this quiz lost me. I wouldn’t respond in either way, I’d say “all buttons look blue now. Can we make it so that just the Add to Cart button is blue? I’m okay with Add to Cart having its own class to make it easier.” or something.
iLoveOncall 2 hours ago | parent
FYI I actually agree, it also surprised me. Anyway everything is deterministic on the website so it's not like that prompt actually has an influence on anything.
blake__dev 4 hours ago | parent
stodor89 3 hours ago | parent
johnisgood 3 hours ago | parent
t-writescode 3 hours ago | parent
johnisgood 3 hours ago | parent
If I get frustrated by it because it just would not "listen", I discontinue its use!
drusepth 1 hour ago | parent
littlekey 27 minutes ago | parent
NickNaraghi 5 hours ago | parent
snkline 5 hours ago | parent
It is easy to misinterpret this site, and therefore not "get" the joke.
khernandezrt 5 hours ago | parent
bdelmas 5 hours ago | parent
chuckadams 5 hours ago | parent
dudeinhawaii 5 hours ago | parent
To try to add something to this discussion -- I think that while I've seen these sort of loops less --- what I have seen is "overly helpful".
Models nowadays want to double-triple-quadruple check things. I'm being silly but it verges on "I have a working solution but let me write a variation in Rust to ensure a convergent solution and prove this works".
I've had to stop models nowadays mostly because they're being agonizingly pedantic in their validation. Opus is actually one of the most pedantic and "off track" here. But again, not in a bad way. I'm usually like "stop testing latency between 50 runs of this app... this is version one.. we're going to make a million more changes.. you're not buying us anything".
mrinterweb 5 hours ago | parent
Yeah of 10 minutes ago. It is shocking how long some seemingly simple things can take. I know there are some things I can do faster than the LLM and some things it can do faster than me. The amount of rambling BS is the exhausting part.
epistasis 4 hours ago | parent
Edit: for how to do this, I set up an OpenRouter account so that I could easily switch models, and then ran them in Pi inside of Orca ADE. Orca lets me easily switch from Pi to Claude Code to Kilo to Codex or Hermes or whatever. Pi+OpenRouter lets me easily switch the LLM. All of it lives in a single open source orchestrator to avoid any platform lock in to any AI company going forward, and can even do local LLM should I care to take on that massive hassle.
torginus 2 hours ago | parent
And shocking how little code end effort some things take if done by hand.
I am not some hardline LLM hater, just venting my frustration.
sixsevenrot 36 minutes ago | parent
f055 4 hours ago | parent
darepublic 4 hours ago | parent
qurren 4 hours ago | parent
If you search my company's Slack for "smoke" the results are almost all within the past 2 years ...
SamuelAdams 3 hours ago | parent
Agents and AI are getting expensive, it seems silly to waste tokens on static checks.
CharlieDigital 1 hour ago | parent
mejutoco 57 minutes ago | parent
bahbahbahbah 3 hours ago | parent
cruffle_duffle 3 hours ago | parent
That is exactly how the slop accretes and you get a pile of crap. Claude somehow assumes that said 2 hour old userless app is some dusty enterprise app with millions of users and billions of dollars at stake for a 1 second outage.
I have to constantly have these things "take a deep breath, step back and look at the entire thing and do this change holistically. please restate what i'm asking you to do and why it's important"
butlike 2 hours ago | parent
This doesn't actually do anything though, right? There's no understanding, so the machine will just reiterate the original token query back to you. The 'why it's important' part will just generate some patronizing boilerplate as a raison d'etre.
Filligree 55 minutes ago | parent
If you're of the belief that this is a contradiction in terms, then what can I say? It works.
johannes1234321 22 minutes ago | parent
torginus 2 hours ago | parent
theshrike79 3 hours ago | parent
It tends to relax some of the insane things LLMs tend to do.
gopher_space 1 hour ago | parent
[0] https://en.wikipedia.org/wiki/Black_Museum_(Black_Mirror)
theshrike79 41 minutes ago | parent
stillpointlab 1 hour ago | parent
And not just backwards compatibility, but migration scripts and all of the testing and machinery around it. I'll literally add feature A, merge it, then add feature B and it is like "oh no, we'll have to fix up and migrate all of the users using feature A".
The other problem is anchoring on an old implementation. I was working with Fable on a change to a core system and it pointed out a difficult failure edge case. It is something that can go wrong in extremely unlikely scenarios but the consequence would be short-term data loss (basically a non-durable intermediate cache being overwritten in a race before a flush to durable storage). It is very hard in these circumstances to get Fable to switch from "how to patch this given the existing implementation" to "how to prevent this with a more robust implementation".
These are both cases where the model seems to over-index on what is already there instead of considering what a first-principles approach would look like. A good engineer does both and then costs them side by side, because a first-principles approach can often be less work than patching what is already there.
MisterMunchkin 29 minutes ago | parent
yonatan8070 2 hours ago | parent
metek 2 hours ago | parent
ls612 1 hour ago | parent
classified 5 minutes ago | parent
cropcirclbureau 5 hours ago | parent
nullbio 5 hours ago | parent
gitowiec 5 hours ago | parent
homeonthemtn 5 hours ago | parent
Culonavirus 5 hours ago | parent
Yes, and it's mine!
JohnMakin 5 hours ago | parent
Got an audible guffaw out of me. This really is what the experience is like sometimes if you're just giving it a result without being specific in implementation, and it comes out of nowhere, some days much worse than others.
I've become patient with it, but whatever this style of output is called or doing - it is both condescending and entirely unhelpful, and it seems designed to frustrate.
GreenWatermelon 3 hours ago | parent
Recently I've started using GLM models and noticed they aren't very claudish (just a little bit, compared to DeepSeek which is extremely claudish)
rcfox 5 hours ago | parent
xd1936 4 hours ago | parent
stevenhubertron 4 hours ago | parent
monooso 4 hours ago | parent
qazxcvbnmlp 4 hours ago | parent
The user was providing no context, they had no ability to give the model background or context. A simple why would have prevented 95% of these side quests. “Im trying to increase the relative visibility of the add to cart action on the page. Can we please change it to blue without changing any other buttons. /effort low. Let me know if you have questions and before editing anything tell me what you are going to do”
K0IN 4 hours ago | parent
epistasis 4 hours ago | parent
I'm hesitant to email their data emails, as it's common for companies to delete all data upon any request, instead of providing data as they are required to.
josh_p 4 hours ago | parent
I had 5.6-Luna coordinate a code review in which it spawns 2 agents looking for different things. My prompt was "review the currently checked out branch. diff target is `next`. The jira ticket is XX-XXXXX..." My `next` branch was a few commits behind `origin/next` but it still did its review against the stale local version instead of clarifying or inferring that I meant `origin/next`. The findings were very confusing until I realized what I did.
I'm noticing the need to be really specific with any instructions lately, which I don't think is a bad thing. I expect co-workers (or anyone really) to tell me what they need in specific terms so I can get it right. I can do the same for the machine, I guess.
heaney-555 4 hours ago | parent
johnhamlin 4 hours ago | parent
digitaltrees 4 hours ago | parent
Hold on let me go commit code without reviewing.
GoToRO 4 hours ago | parent
nonethewiser 4 hours ago | parent
I assume this is just a contrived example?
paimapi 3 hours ago | parent
nonethewiser 1 hour ago | parent
But what is the point of the fiction, if not that its relatable? Is it supposed to mirror some fictional reality that the author is relieved we dont live in? Or is it supposed to parody real life?
I think its supposed to parody reality, but I don’t see the resemblance.
data-ottawa 3 hours ago | parent
The spinner with random claudisms is what makes the game, mixed with Claude helpfully deciding to pick up some tasks along the way.
I feel like most of my conversations with Claude lately are a battle of “I am a technical user, share technical details, but not random superfluous gibberish”
nonethewiser 1 hour ago | parent
butterNaN 2 hours ago | parent
nunez 4 hours ago | parent
I gave it a README specifying the requirements for everything I wanted to build. My interaction with Claude was more or less like this.
marktl 3 hours ago | parent
bethekidyouwant 3 hours ago | parent
anthomtb 3 hours ago | parent
Maybe this is geared towards pure vibe coders for whom a file and line number is too technical.
atleastoptimal 3 hours ago | parent
satirev 2 hours ago | parent
sergiotapia 2 hours ago | parent
charcircuit 2 hours ago | parent
butterNaN 2 hours ago | parent
Caveats:
• ONE HONEST CAVEAT
• ONE THING WORTH STATING PRECISELY
• ONE THING WORTH FLAGGING
• WORTH NAMING
• WORTH STATING PLAINLY
• I DON'T WANT TO LEAVE THIS IMPLICIT
• I'D BE DOING YOU A DISSERVICE
• I DON'T WANT TO BURY THIS
• BETTER NOW THAN LATER
• ONE HONEST TRADEOFF
• I DON'T WANT TO PAPER OVER THIS
• ONE SMALL HOUSEKEEPING ITEM
• THE HONEST PART IS SIMPLER
Pushbacks:
• FAIR PUSHBACK
• FAIR HIT
• THAT'S ON ME
• YOU'RE RIGHT ABOUT THAT
• YOU'RE RIGHT
• YOUR INSTINCT IS RIGHT
• FAIR, AND MORE SPECIFIC THAN IT SOUNDS
• RIGHT, FOR A REASON WORTH NAMING
• I'M NOT GOING TO DEFEND THAT
• YOU'RE RIGHT TO PUSH BACK
Reframes:
• LET ME BE PRECISE
• THE SHARPER DISTINCTION
• THE PART THAT MATTERS
• THE USEFUL PART IS NARROWER
• LESS X THAN Y
• VISIBLE ISSUE / UNDERLYING ISSUE
• THOSE ARE DIFFERENT CLAIMS
commandlinefan 2 hours ago | parent
herrherrmann 2 hours ago | parent
annoyingnoob 2 hours ago | parent
nelaggy 1 hour ago | parent
stillpointlab 1 hour ago | parent
wh0ami 1 hour ago | parent
stefanhall05 1 hour ago | parent
jebarker 33 minutes ago | parent
moritzwarhier 25 minutes ago | parent
Is there a real generalized and capable, but frustratingly "evil" coding model? It could fix so many more buttons in no time, and ideally open follow-up issues about the open questions.