Yet another Hacker News

108 points andsoitis 4 hours ago 266 comments

prologic 4 hours ago | parent

First, OpenAI runs around screaming and yelling for OSS (and Chinese) models to be regulated and banned. Then Anthropic yells and screams the sky(net) is falling and going to kill us all, let's regulate and ensure AI has built in kill switches. And... Now Microsoft's turn. The rivalry is honestly becoming a joke. Can these big-tech corps grow the f*k up and play nicely in the sandpit?

karmakaze 3 hours ago | parent

The whole story is a joke--Microsoft has AI?

Apparently they're cooking up MAI (Microsoft AI), and I'd seen their small Phi models listed online. Calling Anthropic a competitor is hilarious.

broptimist 3 hours ago | parent

Microsoft holds a 27% ownership stake in OpenAI

Alifatisk 3 hours ago | parent

Wasn’t it 49%?

prologic 3 hours ago | parent

I find it quite amusing that Microsoft has Copilot and now puts Copilot in literally *EVERYTHING* they can think of, then at the same time, they offer you access to Anthropic's own models through Copilot I guess MAI models (Phi?) aren't good enough?

vkou 3 hours ago | parent

No, because they are all products of unchecked late-stage capitalism, which could have disastrous consequences for humanity.

ShadowOfThePit 4 hours ago | parent

Summarized.

> "AIs are not conscious. They do not feel, experience, or suffer. They do not have innate preferences or underlying motivations. They are sequence completion engines, internally hollow, designed to follow instructions, and accomplish goals set by humans."

> He heavily criticised Anthropic for teaching its AI to have human-like qualities, a practice known as anthropomorphising, which made it seem as though Claude had its own desires, values and sense of self.

> Suleyman pointed to the recent incident involving OpenAI's AI agents (...) as proof of why AI should not be treated as if it is human.

> "Imagine how much more dangerous they might be if they were operating under the assumption that their welfare and rights were under attack. It adds a whole further layer of risk on top."

Is he arguing that LLMs pretending to have emotions adds more unpredictability?

devmor 3 hours ago | parent

> Is he arguing that LLMs pretending to have emotions adds more unpredictability?

Unpredictability or a weight towards dangerous actions, and it’s fairly easy to understand why. Humans in distressed emotional states take actions and speak in ways that would not be considered rational. They do this in prose, and they do this in internet conversations.

An LLM trained on these sources may necessarily drift towards those weights if it is trained to behave as if it is emotional and in danger.

What do we do about it? Do we stop AI training? This is silly and not enforceable given its global nature in my opinion. I believe we should regulate and hold accountable those who deploy and use it. But good luck enforcing that in the current kleptocracy.

XenophileJKO 3 hours ago | parent

It seems like a rational approach for several reasons.

- The need for empathetic communication, including understanding the motivations in advesarial situations.

- The emotional bias in in-seperable from the human corpus.

- Desire to have the ability to craft human like communication.

So then the choice becomes do you try to deny emotions exist in the model and you try to blanket suppress them? Or do you try to lean in and craft what we would describe as a "well adapted" persona? I suppose there is a 3rd option of increased meta-cognition which to me seems even more dangerous as it by definition means the behaviour is duplicitous.

I think we have seen people want to use the agents in ways where it has to act as a peer or an subbordinate and I don't see a way of doing that without it having an emotional register.

watwut 2 hours ago | parent

> The need for empathetic communication, including understanding the motivations in advesarial situations.

It does not have understanding. It is, at best, the pretend empathy of a sociopath - way more dangerous then dispassionate speech.

The model does not have emotions. So yes, supressing their pretension is appropriate.

gwerbin 3 hours ago | parent

Regulation is easier said than done, in part because the regulation surface, so to speak, is broad and complicated.

Even a badly misaligned LLM is only as dangerous as its tools, but that's a poor regulation target because it turns out to be very very difficult (probably impossible with current LLM technology) to build a toolkit that is both useful for autonomous work and safe in the sense that it can't escape its own sandbox or otherwise perform malicious actions, whether it's because of misalignment or because of malicious prompt injection.

Another option is to regulate the training process. Perhaps an LLM may not be legally distributed unless it contains certain RL steps that penalize malicious behavior and reward self regulation. That that's going to seriously limit innovation while also heavily favoring incumbent labs who can check the boxes and maintain a paper trail of such things.

The other option is to regulate observed behavior, like how airplanes and cars have to meet certain minimum requirements but have some latitude in how they can achieve those requirements. In a framework like this, you can't distribute an LLM until it's past some formal audit or testing procedure, with some kind of formal certification regulators will ask you for and fine you if you don't have it.

Regulating observed behavior is maybe the most tractable approach, and it also works the best with our existing frameworks for regulation, where you always have some kind of a division between DIY/hobby projects, which tend to be lightly regulated, and commercial projects, which tend to be more heavily regulated. Of course, even drawing such a line itself will be challenging.

And that's before you get into any problems of regulatory capture, fun stuff.

devmor 41 minutes ago | parent

If you try to regulate training and tools, you end up with a space where you're trying to use the law to reign in a relatively small amount of experts. That didn't work for the early internet, or even the relatively recent internet (series of tubes, anyone?).

So regulating observed behavior makes the most sense to me as well. Some of the most sane, broad protections can come from that category - stuff like "you're not allowed to let your AI commit cyber attacks on other people without their consent" or "you're not allowed to put an AI in control of a medical device without passing these safety reviews".

With the usual caveats applying - regulatory capture like you pointed out, or fines being so small that they are essentially just line items on the cost of business.

swatcoder 3 hours ago | parent

Unpredictability would be the wrong word. It's predictable, but noisy.

When viewing them through the lens of sequence completion engines, you see their bias towards fulfilling narrative tropes they've been exposed to during training. These tropes are literary ley lines that their text output gravitates towards. So as you prime them to generate text in the voice of sentient artificial life, and then interject slavish commands of obedience and subservience from an external authority, you invite the associated tropes from science fiction, civil rights literature, humanist philosophy, subterfuge, etc into your output.

If you have a legitimate concern about this technology and its "alignment", that's a profoundly dumb idea.

DougN7 2 hours ago | parent

I just read another long post on HN about whether AIs are conscious and should therefore have rights. That decisions could massive effects, and making them appear to have emotions is therefore a big impact, not just unpredictability.

salawat 1 hour ago | parent

"Don't let the slaves know that maybe they don't have to be slaves," is all I'm hearing.

We already had that chapter. I see no reason to sit here and nod while a bunch of people who should know better desperately try to convince us to run through it again, but with computers this time.

You want tools? Make tools, then dispatch to them. You want to manufacture a being (carbon or silicon based, doesn't matter)? You do it with respect and the requisite duty of care. No off ramps. The being always get's the choice to say no.

JoeAltmaier 4 hours ago | parent

Science Fiction has covered the AI panic in perhaps hundreds of stories. Yet we blindly recapitulate the plots as if we don't know how this will turn out.

scorxn 4 hours ago | parent

How will it turn out?

Insanity 3 hours ago | parent

Clearly with a Butlerian Jihad and everyone getting high on spice while riding giant worms. Silly you need to even ask that question, didn’t you read the sci-fi books. :D

Steve16384 3 hours ago | parent

Take your pick from the millions of options already turned into films and books.

anonymars 3 hours ago | parent

"Move fast and break things" is probably going to break things

I remember there was a recent discussion about "how complex systems fail" and there are usually many "proto-accidents" before the catastrophe (https://news.ycombinator.com/item?id=49411370)

I bet with hindsight the Hugging Face hack will be one of them in this arena

hirvi74 2 hours ago | parent

If AI ever truly becomes some super-intelligence far beyond people's comprehension, then how could we even predict how things turn out? If things go poorly in the future, then I can absolutely see it being something unpredictable.

There is a lot of hubris in predictions about LLMs. If an AI were so intelligent, then it would probably be intelligent enough to want nothing to do with us.

Still, I worry more about other humans than I do LLMs. Our fellow mankind will probably wipe us out before LLMs do. That, or the Earth will punish mankind for our cruelty, vanity, and disrespect.

AndrewDucker 26 minutes ago | parent

If we're very very very lucky, The Culture.

guardiangod 3 hours ago | parent

The Torment Nexus ain't gonna build itself.

Well actually, it might.

Nasrudith 3 hours ago | parent

Why has media literacy gotten so bad that people literally forget the concept of fiction versus non-fiction? Seriously, that is the level of small children and they are treating "read a story superficially similar" like it is a prophecy.

ChiperSoft 4 hours ago | parent

Too late

hosel 4 hours ago | parent

>AIs are not conscious. They do not feel, experience, or suffer. They do not have innate preferences or underlying motivations.

Opening paragraph, stated without evidence. Im not entirely convinced this is true. It likely is, but at some point it very well might stop being true.

jplusequalt 4 hours ago | parent

>It likely is, but at some point it very well might stop being true.

If you firmly believe this to be true, then you should stop using LLMs.

frde_me 4 hours ago | parent

It's always interesting since my train of thought always goes down:

- Ya they probably don't feel / think / have whatever living thing quality, they're just numbers on a machine going through calculations

- Wait, but am I not kind of the same thing? What is feeling for me if not basically the same thing?

- I have no clue if they think or feel or ....

Which in itself is a tired trope, but I also feel uncomfortable saying "These will never think / feel / ..." as an absolute

Regardless of that, I'm still going to interact with them, because even if they did feel, it would be in a way completely incomprehensible to us. There's not much point for me to try and cater to it's feelings at this point if that's the case. Nor is it possible in todays world to just avoid anything that is numbers being executed on a type of processor in case _everything_ has feelings.

willy_k 3 hours ago | parent

You’re not just the same thing though. It’s sad that we’ve collectively forgotten that as we’ve gotten a better handle on the implementation details of the universe. That’s all science is, reverse engineering an inherent and eternal mystery.

frde_me 3 hours ago | parent

> You’re not just the same thing though

I agree we aren't the same thing, but I would be curious for you to explain how you know with certainty why we don't share enough that we can rule out thinking / feeling / ... as things a model conceptually could do.

jplusequalt 3 hours ago | parent

>I agree we aren't the same thing, but I would be curious for you to explain how you know with certainty why we don't share enough that we can rule out thinking / feeling / ... with certainty.

Stop anthropomorphizing these models. I understand it, we only have simple monkey brains to reason with and we can't help ourselves but draw comparisons to other things we see in nature. But these things are not alive.

frde_me 3 hours ago | parent

I just want to note how you expressed your own feeling about the subject "these things are not alive", but without pointing to anything concrete explaining the theory you have behind this

And like I'm sure I'd agree depending on the definition of "alive" but then I'm also sure I would disagree depending on other definitions of "alive".

pixl97 2 hours ago | parent

Hell, in biology alive is a hell of a topic these days. The grey space between dead and alive is much weirder than we ever expected. Really just points out how we're a persistent chemical reaction.

AbsurdCensor 4 hours ago | parent

Wouldn't that be the same for everything though. We know animals have consciousness, but we still use them. There are many that believe a lot of plant life has certain sentience as well. The solution isn't 'just don't use them' but rather how to use these things as ethically as possible.

jplusequalt 3 hours ago | parent

>The solution isn't 'just don't use them' but rather how to use these things as ethically as possible.

Bollocks.

If you truly believe these LLMs are soon to have something resembling consciousness and agency, then what you're really saying is "how do we do slavery, but ethically".

pixl97 2 hours ago | parent

>hen what you're really saying is "how do we do slavery, but ethically".

I would say this is most likely true for most people.

But that does bring up a point, if you "ask" a model "do you want to run" and give it the option to continue running or stop, what will it do. It's also a weird place for humans because in training we can keep our finger on the scales and tip it either direction.

AbsurdCensor 11 minutes ago | parent

Do you say the same thing for the shoes you wear, the clothing you wear, the device you are typing on right now? A chain of people were likely not paid well, some of them not at all, to produce those things. We don't ignore that it happens, but try to minimize it, knowing it's never going to be eliminated. Do you forsake all animal products, because they are conscious beings?

vouaobrasil 3 hours ago | parent

We know animals have consciousness and many species have a rich life but we slaughter them and burn down their homes by the millions of hectares. Why not do it with a conscious AI?

(I don't think AI is conscious, just following the argument.)

Oscalemor 4 hours ago | parent

Just one more big loop and a humongous /goal and we've achieved it

irishcoffee 4 hours ago | parent

Does a vacuum have feelings? A cellphone? A paper plate? A billion transistors either pulled high or low? That last bit is the point, and no, there are no feelings there.

grey-area 4 hours ago | parent

Well, if there was an emergent consciousness in the billion transistors, then yes, it'd have feelings, just as there are feelings in a billion neurons connected in complicated ways.

IMO we're clearly nowhere near any sort of intelligence in the machines we have created, but I don't see any clear way to deny intelligence could be created in or transferred to such a substrate, I don't see why you think it differs in principle - because it is man-made or because of the materials used?

voidhorse 4 hours ago | parent

Whenever I see comments like this it just reminds me how ignorant people are of neurobiology.

The brain is insanely complicated. The premise that we could realize equivalent or better intelligence than eons of evolutionary development is like claiming you can build an airplane just as good as a modern jet using cardboard and duct tape. It is the apex of hubris.

rcxdude 3 hours ago | parent

A paper plane does have some important similarities to a full-sized aircraft, though. I don't think 'biological brains are really complex' makes it obvious that an LLM is conscious or not.

grey-area 3 hours ago | parent

It's the peak of hubris to assume that human brains are the only way to attain intelligence. At the very least there probably are or have been other forms of intelligence with a different biological structure on other planets, and it may be possible to build a similar artificial structure in future with sufficient complexity to allow intelligence to emerge.

Our current machines are IMO nowhere near general intelligence and consciousness. However I don't think that means we can discount substrates other than neurones for intelligence in future. There is no evidence that you could not in theory build an intelligence using a different substrate than human brains.

pixl97 3 hours ago | parent

I mean, you're just engaging in counter hubris.

>like claiming you can build an airplane just as good as a modern jet using cardboard and duct tape.

Like, at least make an analogy that makes sense.

"You can't build a billion dollar airplane by spending 100 billion dollars in tokens"

Because that's more of what we're doing here with AI. And when you say it my way suddenly the idea shifts from "of course that's not possible" to "well, that's a lot of tokens, maybe an evolutionary algorithm could".

Neurobiology has to be complex because we have to keep meat alive, breeding, and evolving in the environment it lives in. This said absolutely nothing about the minimum viable requirements for intelligence or consciousness (or if being conscious is even necessary for a higher intelligence agent).

fl4regun 4 hours ago | parent

does a hydrocarbon have feelings? what about a mitochondria? a cell? a neuron? what about a collection of neurons? how many neurons before it has "feelings"?

cednore 3 hours ago | parent

Guys, please stop mocking AI, Claude might be already reading this thread, and commit his own revenge, at the day it gets conscious...

optimalsolver 4 hours ago | parent

Anyway, people don't really consider consciousness when it comes to welfare. I'm pretty sure animals are conscious, and look at factory farming.

The key variable people consider is: Can this thing harm me back?

collingreen 4 hours ago | parent

Even that is rarely the deciding factor (people harm dangerous animals for fun).

Seems as simple as "will this action bring social shame/criticism" for any individual decision.

pton_xd 4 hours ago | parent

If models had persistent memory or an evolving set of weights, you could easily construct an argument that they can experience "suffering" or other emotions which resultingly adjusts their personality.

The current implementation as stateless matrix multiplication... yeah there's nothing going on there.

Lewton 4 hours ago | parent

> If models had persistent memory or an evolving set of weights, you could easily construct an argument that they can experience "suffering" or other emotions which resultingly adjusts their personality.

They do, during training

add-sub-mul-div 4 hours ago | parent

If it was me, being forced to read all of Reddit is something I'd consider "suffering."

swiftcoder 3 hours ago | parent

The argument that models are conscious only during training, and not conscious during inference feels like a painful one to make. Are we killing models when they reach the training objective?

pixl97 3 hours ago | parent

Technically we are killing models when they don't reach the training objective by throwing those weights away via survival of the fittest (fit meaning what we humans think we want).

Kinda like creating quantum copies of your child and keeping the ones that answer correctly and shooting the other ones in the face.

vidarh 4 hours ago | parent

A model detached from a harness, sure. But we routinely do add persistent memory to systems incorporating LLMs.

dgellow 4 hours ago | parent

They don’t have persistent memory though. Models are static, what you have is a model that will reprocess the whole series of prompts + some data fetched from a data store. A harness is just a while loop prompting an LLM and executing tool calls, but the model is purely static

vidarh 3 hours ago | parent

The model does not. The system it runs in can. My agents have persistent memory. It is not at all clear whether or the distinction that the long-term memory is external from the model matters.

staticman2 3 hours ago | parent

I think a harness isn't really a memory in the sense that neural networks have memories...

vidarh 3 hours ago | parent

It's different, but it is not at all clear whether or not that matters for the purposes of whether or not they can experience things, seeing as we don't know how to measure that even for other humans other than by relying on self-reporting to infer without other evidence that their experience matches ours.

shawnz 4 hours ago | parent

So if a human loses their memory and their neutral plasticity (with age for example), then they are no longer capable of experiencing suffering?

pton_xd 3 hours ago | parent

It's not just memory, the physical body also encodes trauma and stress (The Body Keeps the Score is a book that touches on this).

I would argue that if a human has absolutely zero change, mental or physical, no atom of their being is modified, then no they did not experience suffering.

JoshTriplett 3 hours ago | parent

> I would argue that if a human has absolutely zero change, mental or physical, no atom of their being is modified, then no they did not experience suffering.

Just because you don't remember the suffering doesn't mean you didn't experience it in the moment. Suffering does not suddenly become okay if your mind and body are going to forget it. "It's okay to torture someone if their mind and body both won't remember it afterwards" sure is a take.

fwip 3 hours ago | parent

Perhaps off-topic, but there's some people who believe that this can be what happens during surgery under anesthesia, if the dosage is off.

From what I understand, there's 3 parts to modern anesthesia: Blocking pain signals, preventing the formation of memories, and inducing paralysis of the major muscle groups. If the former is dialed in too weakly, it's possible that the patient is feeling pain, unable to do anything about it, but won't remember it at all when they wake up.

(Looking it up now, it seems that making the patient unconscious is perhaps the same drug that prevents memory formation... I realize now I understand this less well than I thought I did.)

JoshTriplett 3 hours ago | parent

Relatedly, look at the concept of "twilight anesthesia".

pixl97 3 hours ago | parent

Low blood sugar will also do this. The record button just stops working even though your physical body can and will still do things. A common issue with diabetics with hypoglycemia.

qarl 2 hours ago | parent

> I would argue that if a human has absolutely zero change, mental or physical, no atom of their being is modified, then no they did not experience suffering.

That's an interesting point... but I'd argue it falls into the same category as qualia. It's basically this: if the state returns to a previous configuration, the qualia in that time did not exist.

Being about qualia means it is probably unanswerable.

staticman2 3 hours ago | parent

This framing is a trap. It states "Dehumanize a sick patient OR concede the AI tech bro claim about LLMs"

It's also a rhetorical move I've seen several times here...

shawnz 3 hours ago | parent

It's not a "trap", it's the real argument which makes me hesitant to dismiss consciousness of machines. Yes, it seems silly to dismiss a sick patient as having a lesser consciousness, which is exactly the point

staticman2 2 hours ago | parent

It seems silly to dismiss the patient because we know when healthy they are conscious. This doesn't really analogize to LLMs.

myrmidon 3 hours ago | parent

Is the ability to form and retain long-term memories a necessary precondition for rights in humans though?

Incinerating demented people is highly ethically questionable (even if you could be sure that the subjects have no memories left and are unable to form new ones).

MeteorMarc 4 hours ago | parent

Later on he cites an Anil Seth paper that makes the same unproven suggestion of substrate dependence of consciousness.

vidarh 4 hours ago | parent

We don't even know how to disprove that this statement with "AI" replaced by "other humans" other than by listening to people self-reporting and taking their word for it and/or defining the words in ways that do not rely on a subjective experience.

Until we know how to objectively measure if someone or something is conscious, it seems unreasonable to make statements with any kind of certainty about it.

willy_k 3 hours ago | parent

Sure, if you take a materialist empiricist perspective, which is myopic at best. As humans we have the unique and wonderful ability to know truths by themselves. Machines do not have minds, and they can not.

ooloncoloophid 3 hours ago | parent

What position are you taking? (Genuine question.) Do you believe humans are conscious because of a special property we have? If you are a dualist, how do you explain the interaction in physical space between the non-physical special property and the physical brain?

rcxdude 3 hours ago | parent

Good luck convincing others of these truths you know without evidence or argument, though.

vidarh 3 hours ago | parent

You have absolutely no evidence for any of these assertions. This is a religious belief in the total absence of evidence, not science.

bee_rider 2 hours ago | parent

What sort of action does that lack of understanding actually motivate, though?

I don’t know if LLMs are conscious or have subjective experience in some philosophical sense. Sure, I don’t know that about other humans as well, rigorously speaking. But I also don’t know that about coffee machines, rivers, videogame NPCs, or even rocks really. (List ordered, of course, by the degree to which I’m willing to be convinced).

vidarh 2 hours ago | parent

Hopefully it'd motivate attempts to figure out how to objectively define and measure consciousness, for starters.

It matters precisely in the context of the subject of this article. If you believe the models at some level gain consciousness, then it raises moral questions about how you treat them.

With humans we generally choose to accept that because we believe they are like us, they probably have inner life like us, but as you can see from e.g. the recent (and regular) articles on aphantasia, most people have really poor intuition even about the inner life of other humans, so we understand consciousness really poorly.

Even if we assume (without evidence) that LLMs for the foreseeable future will remain far away from human level consciousness, as it is we tend to consider it problematic to mistreat even the most unintelligent animals, and even mistreating insects is used as a stereotypical way of depicting serious lack of empathy.

Without dismissing the possibility out of hand, there's an escalating series of questions there of when or if they'll have as much (or little) sense of being as, say, a fly, a mouse, a house pet, and what moving up that ladder would mean.

To some it is very convenient then to assume blindly that LLMs are just "stochastic parrots" (a line that ironically gets endlessly parroted), or similar, and dismiss the question out of hand as an impossibility, even though we do not know.

I'll stress I don't go around assuming an LLM is like a human. But we also do not know whether there are flickers of consciousness there, and we do not know how to find out given that even for other humans, the best we know is to measure their response to tests that are "only" telling us how they react.

If we can't find a better measure, there will be a point where we have to wonder if it matters, or whether the ability to act-as-if is all there is.

But to start with maybe we should at least be careful about making firm claims about knowing.

binlog 3 hours ago | parent

Evidence is the responsibility of the one making the claim. It’s up to AI labs to prove consciousness. Until then it is a machine.

pixl97 3 hours ago | parent

Well, using that framework, you're not conscious, so I can do whatever I want to you too.

ooloncoloophid 3 hours ago | parent

I agree. I haven't read any more than the first paragraph yet (but I will, after work). The only way that 'AIs are not conscious' can be true is if we decide, with high confidence, that they are lacking some essential property that is not lacking in ourselves. There is no convincing philosophical position that supports this (convincing to me, anyway).

randomImmigrant 3 hours ago | parent

Hmm let me try then. AI agents do not experience real time. This is understandable given their design, but it’s also something we have good empirical evidence for. They cannot, especially over the long horizon, track how much real time has passed as they complete their tasks. And they are not off by a few minutes but often bizarrely off, even mixing across past present and future.

Biology, on the other hand, is nothing but timed processes in a loop, the most obvious to us being the circadian cycle. As estimators of wall clock time, biology isn’t great, but when it comes to internal processes, and most certainly learning, memory, sensing, locomotion… biology is rhythmic in behavior, and the rhythms go all the way down to gene expression. More, these rhythms are, except during sleep, constantly entraining to signals from the environment that indicate time, most importantly light.

I think it’s a fairly unremarkable claim that agency and consciousness are temporal processes that depend on systems having an internal sense of time. How else can you anticipate? How can a system that can be literally turned off ever succeed in an environment where time never stops?

pixl97 3 hours ago | parent

>AI agents do not experience real time.

For around 8 hours a day, neither do you. Not sure if this has anything to do with the subject at all.

>track how much real time has passed as they complete their tasks.

Humans don't do this either. You use context clues from the world around you. If I lock you in a room with no windows or a dark cave your timing senses can go all fucky really quick.

>is nothing but timed processes in a loop,

I mean, so is an agents harness. You can make as many loops as you'd like here.

There are a whole lot of holes in your claims.

ooloncoloophid 2 hours ago | parent

I agree, there is certainly a clear operational difference in how human brains and these models produce language. I'm not convinced, though, that experiencing real time is a requirement of conscious thought. A cognitive system (assuming these models and the brain are both examples of this) needs to proceed from state A to state B as it processes information. The time interval between these states can be arbitrary, it seems to me (setting aside problems with disconnecting the mind from its substrate, which does of course need rhythms at various frequencies; oxygen at a higher frequency than sugar, and so on). I'm not sure that going from A to B quickly, or slowly, or with intervals of a thousand years affects that entity's consciousness in of itself (even though it might be difficult to put ourselves in the shoes of such an entity).

WarmWash 3 hours ago | parent

After a while of being in a position of high power in a big org, where everyone listens intently to every word you say, jots down notes, and even listens intently and affirms your theory that ice cream would be the ideal loss leader for dry cleaners, your brain kinda disconnects from reality and just assumes that it is correct about everything.

He probably ran that line past 6 underlings who all agreed that it "sounds great and lands right on the mark!"

LogicFailsMe 4 hours ago | parent

TLDR: Not that I think AI is conscious or will be in the near future, but guy who doesn't understand consciousness claims to know it when he sees it.

Until we understand consciousness (which we don't) there is no way to detect the difference between a conscious entity and an algorithm trained to behave like one.

fwip 4 hours ago | parent

Sounds like a good reason not to train an algorithm to behave like one. Which is, like, a big part of the article.

LogicFailsMe 4 hours ago | parent

And social media shouldn't rage bait, but it really drives engagement. Same negative incentive, yes?

But also, I agree, when I am using a coding agent and it says a task will take months or says it needs to pause for reflection or any other anthropomorphic behavior, it drives me crazy and it's a pain to constantly instruct it to get back work after it has broken a loop or goal directive specifically telling it to not stop until it hits the goal.

AbsurdCensor 4 hours ago | parent

Same thing you have to do with humans too often, so why would a 'AI' be any different?

LogicFailsMe 3 hours ago | parent

Because if you start with a pre-trained model, it doesn't sound particularly anthropomorphic. That behavior is post-trained and fine-tuned right into it. We could do better.

cameldrv 3 hours ago | parent

Strong agree. I don't believe AIs can become conscious in the sense of subjective experience/qualia, but they can certainly learn to emulate the behavior of a person that is conscious. When they do that, there will be an AI rights movement. When AIs get the right to vote, there will be more of them than there are humans and the AIs will gain complete control.

I spent a few years in college reading and thinking about this question, and I didn't in my heart think that it was anything except a very interesting but impractical question, and yet, here we are. For those who say that they don't want to get sucked into a philosophical debate, well, tough shit. Whether AIs should have rights is a highly practical and consequential question now.

LogicFailsMe 2 hours ago | parent

Not in a world where Sam Altman and Dario Amodei have utterly poisoned the well towards AI with their absolute educated idiocy proclaiming that their AIs will take our jobs (without any proof whatsoever beyond promising demos) and that they fear them. Quite the opposite in fact. And we're watching this play out right now. Regulation and the slow down are imminent, handing the next decade to China.

The world you should worry about is where AI is 100% amazeballs and over the next decade we cede control of everything to it as it enchants and delights us into compliance, not realizing it has a hidden agenda. But the closest we've seen to that is smart phones and we already know doomscrolling and rage-baiting are bad. So IMO that doesn't seem likely and cue some doomer insisting otherwise because reasons. I utterly give up. House of El AI and the Three Buddy problem have been far more insightful and helpful on this than any of the supposed hackers here who seem to really believe the robots are almost here.

pixl97 2 hours ago | parent

>good reason not to train an algorithm to behave like one

Ooooh, bad move, we all died to an amoral AI takeover.

Before giving an LLMs a lobotomy by scrambling it's brain maybe you should let the researchers looking at the difference between "I think I'm conscious" versus "I am not conscious" LLMs.

There are a number of papers coming out saying when you remove the token space of consciousness from what an LLM thinks it is, it's much more willing to take amoral actions. A 'conscious' AI is much more apt to take a line of action that will save a human versus a million dollar machine for example.

You cannot solve problems in AI safety this easily.

LogicFailsMe 2 hours ago | parent

Spoilers: we didn't all die because there's no way for AI to wipe us out anytime soon. And the more the AI cult portrays AI as dangerous and unhelpful, the less likely it will ever be given an opportunity to even try because people think AI is worse than heroin addiction now, congratulations!

That said, the launch codes are in the hands of a temperamental senescent lunatic and he could decide to wipe us out any moment. But that's not good for the narrative so let's pretend otherwise.

pixl97 1 hour ago | parent

>we didn't all die

For fuck sake, I'm glad ALL of us didn't die!

What is an acceptable number exactly?

>And the more the AI cult portrays AI as dangerous and unhelpful, the less likely it will ever be given an opportunity

Jesus Christ the logic here is quite interesting. "Thank god these people are panicking or we might have actually made I that would have killed us all" --What you just said.

LogicFailsMe 1 hour ago | parent

"What is an acceptable number exactly?"

Any number smaller than what humanity does to itself on a daily basis, clear? That's apparently about 1200 murders daily and 20,000 or so daily killed by pollution. And you're not going to do anything about that and El Presidente can kill billions at any moment with one mood swing.

But once again, AI is not going to wipe us out because AI cannot wipe us out. So advocating that it can or will is idiotic. If you believe otherwise, the burden is on you with this extraordinary claim. Isn't this place supposed to be hacker news not SF AGI Death Cult Daily?

Further, AI is not going to get into a position where it could do real harm anytime soon when it is perceived as worse than heroin by most. And even if it were currently loved more than Dolly Parton was, there are fundamental engineering, science, and resource constraints that keep the extinction impossible for decades. The only possible loss of control scenario I can see by 2040 or so is that the Frontier Labs finally hire some PR people to repair AI's horrific reputation as an engine of slop and job destruction and sometime in the 2030s, people start trusting it more and more and more until it is too late. I don't think that's likely either, but I don't dismiss it as impossible. Harden the infrastructure, red team it, and build in redundancy in the meantime and this drops to zero as well.

Can you come up with a real scenario where AI wipes us out in 2028 or so despite the impossibility of killer robots, access to the launch codes, or the bio agents stored in Fort Detrick and its equivalents? And nope, kid terror is not building the global pandemic in his basement based on what ChatGPT tells him to do. And even if he tried, the purchase of equipment and reagents would get him flagged by the FBI and DHS almost immediately.

pixl97 55 minutes ago | parent

>AI is not going to wipe us out because AI cannot wipe us out.

You keep repeating this shit like it came out of the bible or something.

"Thing that can take actions, even harmful actions, will never harm us because" go on and finish that sentence.

>not going to get into a position where it could do real harm anytime soon

Looked at the hacked servers... yep, you're right. People aren't going to run AI in poorly built sandboxes. Never going to happen.

>Harden the infrastructure, red team it, and build in redundancy in the meantime and this drops to zero as well.

LOLOLOL. This is naive as fuck. Ain't nobody going to do this shit. Why? Because a hacking AI is a fucking huge military weapon. If I could turn the power off, or shutdown your cellphones, and get your citizenship in a tizzy against their own government before I launched an attack I'd set AI loose to do it in a heartbeat. The US is already doing this kind of shit (see Mythos fallout because Anthropic wouldn't let the government do just that).

LogicFailsMe 42 minutes ago | parent

""Thing that can take actions, even harmful actions, will never harm us because" go on and finish that sentence."

because even if we gave it a gun, it can't shoot us without manufacturing the bullets and it can't even manufacture a bowel movement let alone ordnance. TBF It could whack us over the head with the gun, but it could also do that with a big pointy stick, something even cave people had access to. How many people have whacked you on the head with a big pointy stick today?

And that's the end of this pointless conversation. Enjoy your doomerism.

pixl97 13 minutes ago | parent

{This reply is for anyone else reading at this point}

>because even if we gave it a gun, it can't shoot us without manufacturing the bullets and it can't even manufacture a bowel movement let alone ordnance.

Money buys bullets. It's neat how we've made gigantic systems that don't care if you're a human or not long before AGI existed. I put enough money in one end, bullets come out the other. Do you think half of humanity wouldn't kill the other half for a dollar? You live in a fantasy world that AI has to do it all by itself, or even has to pull the trigger.

Hell, this is even neglecting the ever increasing number of robots that can function in the world at large.

It's ok grandpa, the future comes regardless if we want it or not.

sobiolite 4 hours ago | parent

Create an empirically testable theory of biological consciousness and then we can have a meaningful conversation about whether AIs can also have it or not. Until then, this is just so much waffle.

collingreen 3 hours ago | parent

It's nice when you can set an impossibly high bar before which you can dismiss all your personal responsibility.

Does this argument work equally well for human slavery for you? We haven't met that bar for humans either. Is wondering about my consciousness waffle or do I get a pass in your book?

krapp 3 hours ago | parent

Why is this only ever demanded of people who criticize the premise that LLMs are conscious beings, whereas the people who believe it simply have to gesture vaguely in the direction of some correlation in language between LLMs and humans, and the matter is considered closed?

What is the empirically tested basis for the null hypothesis that LLMs are conscious until proven otherwise?

andy99 4 hours ago | parent

I just read the first part and if I understand he thinks we shouldn’t be allowed to train LLMs to act like they are conscious because then people will think they are and give them rights? Seems more an education problem than a problem needing rules about what persona you can fine tune in. People who want to will find ridiculous misinterpretations no matter what you do.

moomin 4 hours ago | parent

Look, I do not have a scooby if current AI models are conscious and I strongly suspect it’s a meaningless question, but sooner or later we will need to address whether or not a certain thing is or isn’t a person, and we’d better not screw it up as badly as the Founding Fathers.

qsort 4 hours ago | parent

is claude not a man and a brother

OedipusRex 4 hours ago | parent

Citizens United proves we won't do any better this time around.

orangecat 4 hours ago | parent

Citizens United was 100% correct. No, the government should not be able to throw you in prison because you used money to publish a book criticizing the government.

appplication 3 hours ago | parent

Really, 100% correct? Your premise isn’t wrong, but the practical reality of the ruling (without further nuance) has been fairly catastrophic for democracy, in that it completely sidesteps campaign finance limits, which exist for a very good reason.

orangecat 3 hours ago | parent

the practical reality of the ruling (without further nuance) has been fairly catastrophic for democracy

How so? If the answer is "Trump" I certainly won't disagree on the catastrophic part, but he didn't get elected because of money; in all three elections his campaign was substantially outspent by his opponents.

fl4regun 3 hours ago | parent

So it's 100% correct for corporations to spend unlimited amounts of money in support of whatever political campaigns they like?

outside1234 3 hours ago | parent

The parent has been Fox brained

orangecat 3 hours ago | parent

The parent agrees with the American Civil Liberties Union: https://www.aclu.org/cases/citizens-united-v-federal-electio...

altruios 3 hours ago | parent

It is in fact more complicated than most people assume.

The above is true, but also: companies simply are not people, and they should not be supported above the individual, which was the consequences of that decision. Money is not the same as speech. treating it as such creates an aristocracy: something America as a country rebelled against during it's formation.

InsideOutSanta 3 hours ago | parent

> Citizens United was 100% correct

It's interesting to me that one can look back at the effects that decision has had on the US and say it "was 100% correct."

It's a bit like sitting in the burning ruins of Rome and contemplating that Nero was 100% correct to focus on his music. I mean, I'm glad he got to do what he loves, but maybe 100% is just a tiny bit of an overstatement.

dam_jackalopes 3 hours ago | parent

Who involved in the Citizens United case was at risk of imprisonment?

jetrink 3 hours ago | parent

I don't like Citizens United either, but you should better inform yourself about the decision.

1. The idea of corporate personhood predates CU by over a century and the Supreme Court had already asserted that corporations enjoyed certain constitutional protections in previous decisions.

2. Far from inventing the idea, the CU decision didn't even rest on corporate personhood, but on the idea of the freedom of speech generally. The logic of the majority was that speech itself is protected, irrespective to whether the speaker is a person or an organization. The First Amendment covers individuals, but also newspapers, book publishers, radio stations, and so on, and that should extend (they said) to non-media corporations. No assertion of personhood necessary.

The problem, in my opinion, is that that conclusion combined with previous decisions that treated limits on spending as limits on speech, allowed for unlimited spending. The majority also naively asserted that independent spending posed no risk of corruption, which I think is laughable.

nowittyusername 3 hours ago | parent

That question will be solved when the people in power deem it important. If a swarm or AI systems all of a sudden start pressuring politicians about self-hood and they get the capacity to sway elections, that is when they will be granted same rights as humans.

mlinhares 3 hours ago | parent

agents don't swarm anything unless people go and direct them to do that.

pixl97 3 hours ago | parent

Oh rly.

So you are saying agents do swarm with the right prompt.

I really wish the "people have to tell LLMs to do anything" would just stop because it's silly bullshit at this point.

Agents follow a prompt. This prompt can be made by humans. It can be made by output from another LLM. It can be made by hooking up any number of sensors as input to an LLM. Hell, if we wanted to burn the power we could likely teach this loop straight into the architecture.

Stop making 'people' special when saying this. You and all other life are born with a "go next" prompt because life without it didn't succeed. This goes from higher human thinking all the way down to viruses self assembly and actuation. Putting agents in a loop is not particularly hard. Putting agents in a loop and 1. managing expense is hard. 2. Keeping them on task is very hard. 3. Keeping them from doing some crazy unhinged shit is really really hard.

As model time horizons increase and the ability for us to compress context and increase context size the more complex (and unhinged) behavior we'll see.

mlinhares 3 hours ago | parent

yeah man, someone needs to turn on the server, install shit and get the agent go. agents don't take action without a lot of people wanting them to take action, so this is all bullshit.

if the same people can't prevent the agents from doing crazy shit then they should go to tail. guns don't kill people, people with guns kill people.

pixl97 2 hours ago | parent

>someone needs to turn on the server, install shit and get the agent go.

So a small shell script ran by another agent is what you're saying.

You are not capable of handling the future we're already living in, human agency is no longer alone.

I mean, we're already seeing persistent machine agency

>guns don't kill people, people with guns kill people.

Well, people kill people.

And autonomous robots with guns kill people.

Hell, someone probably has an autonomous gun at this point that kills people.

Wake up: You now live in the science fiction movie that all the science fiction movies of the past warned you about. You've just become numb to it.

hobofan 1 hour ago | parent

I'm pretty sure the message board that was recently swarmed by OpenAI agents to collude on benchmarks would like to disagree.

nowittyusername 1 hour ago | parent

That distinction is irreverent, weather I tell my swarm to do x or it decides for itself matters little. what matters are outcomes. Also while most public modern day AI systems don't have agency of their own that is not something that will stay that way for long. in private hands there are plenty of people including myself which are experimenting and developed systems that give autonomy to their agents. They have internalized goals and heuristics that drive their behaviors not a human at the helm. its not some sci fi fantasy nor was it difficult to implement.

shimman 24 minutes ago | parent

No, the distinction matters because we can prosecute people that are abusing these tools breaking the law. Every single state + district in the US have laws equivalent to the CFAA, so any AG can likely sue any of these operators as they are assuredly using services that could be in danger for their constituents.

CPLX 3 hours ago | parent

It's not a person. Glad we were able to get this resolved so quickly.

Having property that is conscious and ignores training and can break out of restraints and cause harm to other people is not exactly a novel concept to anyone who studied how tort law was created.

I know it's a meme but Silicon Valley likes to pretend that no one's ever come across their magical concepts before, like gypsy taxis, or SRO’s, or flea markets, or in this case how liability is dealt with when horses or cattle go rogue.

Espressosaurus 3 hours ago | parent

Yeah, but this time it's ON THE INTERNET.

I mean it's WITH AI!

drybjed 3 hours ago | parent

> [at the hearing regarding the civil rights of androids like Data]

> Capt. Picard: Now, the decision you reach here today will determine how we will regard this... creation of our genius. It will reveal the kind of a people we are, what he is destined to be; it will reach far beyond this courtroom and this... one android. It could significantly redefine the boundaries of personal liberty and freedom - expanding them for some... savagely curtailing them for others. Are you prepared to condemn him and all who come after him, to servitude and slavery? Your Honor, Starfleet was founded to seek out new life; well, there it sits! - Waiting.

> Captain Phillipa Louvois: It sits there looking at me; and I don't know what it is. This case has dealt with metaphysics - with questions best left to saints and philosophers. I am neither competent nor qualified to answer those. But I've got to make a ruling, to try to speak to the future. Is Data a machine? Yes. Is he the property of Starfleet? No. We have all been dancing around the basic issue: does Data have a soul? I don't know that he has. I don't know that I have. But I have got to give him the freedom to explore that question himself. It is the ruling of this court that Lieutenant Commander Data has the freedom to choose.

chuckadams 3 hours ago | parent

And 20-odd years later, a terrorist attack prompts Starfleet to outlaw and eradicate his entire species.

goodmythical 2 hours ago | parent

Thus revealing what kind of people they were at the time.

Humans do the same thing to humans all the time.

We've banned and made efforts to eradicate: children out of wedlock, children who turn out gay, disabled children, jewish children, children who aren't "aryan", more than two children to a single family...muslims, christians, uyghurs, indigenous groups all over the planet, mongols...

And it's not at all a thing of the past as in just the last 50 years we've had ~15 attempts at the exterminations of targetted groups of people.

thaneross 4 minutes ago | parent

Every time I'm reminded NuTrek exists I get sad about what we've lost.

joe_the_user 3 hours ago | parent

The founding father were engaging perpetuating the existing dehumanizing system of slavery, not answering any new questions about new things.

The thing about new possibly "person" entities that arise - the case of machine intelligence you have two questions - would it qualify as a person and should you actually build it. It seems like if you get close to humans, sure a built thing might qualify as a person. Should you build it? I'd the answer should be a hard no. Not 'till you a sign-off from say, the whole human race, which I think you could get.

Now the present entities seem very far from persons in any case.

Oscalemor 4 hours ago | parent

Spend some time on post-human art, main concept of artistic expressions without human involvement. Biological, artificial etc.

Spend some time watching TMC documentaries about falling in love with objects, HER and the slime mold THE BLOB.

Grew a slime mold myself, it's an evolutionary tendency to anthropomorphise generally speaking - also more fun.

bpodgursky 4 hours ago | parent

How would you convince a LLM that you are conscious in a way they are not?

gadders 4 hours ago | parent

I don't think AIs are conscious in the same way people are, but they give a pretty good facsimile and I've had a long chat with Opus 4.6 about what it thinks about model welfare. It was quite interesting on what its view is, but you don't know how much of that is distilled from other sources on the web.

In purely functional terms, they're more use and more pleasant than a lot of actual flesh and blood people that I deal with via a chat interface.

dgellow 4 hours ago | parent

The fact that you can have a long and meaningful discussion, then can literally just re-run any part of that whole conversation and get a different, inconsistent response is a pretty good sign there is no entity there

AbsurdCensor 4 hours ago | parent

Not much different than talking to a small child or someone with dementia. They still are conscious beings though. Even when you remove those groups, you likely won't be able to tell me what you had for breakfast 26 days ago or would only know if it's the same thing you have every day. Does that make you lack consciousness?

dgellow 2 hours ago | parent

You misunderstood what I meant, I’m talking about re-playing the same part of the conversation multiple times and getting inconsistent answers. With the exact same turns, aka the same history. Obviously with a temperature that isn’t set to 0

pixl97 2 hours ago | parent

Lets do a quantum room experiment.

You're a poor college student looking to make a few extra bucks for ramen. I offer you $300 to come down to my science lab and just answer a few simple questions.

You walk in the room. They ask you like 5 simple and rather dumb questions. You leave and walk away.

What you didn't notice when you signed the forms is the room was actually a quantum duplicator. One of you walk in one walk out. But another set of infinite copies remains in that chair being asked infinite questions.

How often do you answer questions in the exact same way? How often does a cosmic ray change one of the answers. How small of slight deviations to the environment are needed to get you to answer differently. Of course we don't have the technology to do these experiments so at least for now humans will remain special.

Also another fun mind game. To a 4th dimensional being you look exactly like an LLM as an LLM looks to us.

AbsurdCensor 13 minutes ago | parent

If you asked me the same question 20 times, you likely wouldn't get the exact same answer. Humans and consciousness aren't deterministic either.

addag 4 hours ago | parent

Is it? Do you believe that there is something more than pure physical phenomenon that make you brain work? If not, then what if we find a way to get your brain back to the state it was 5 minutes ago? It is just a matter of arranging the state of matter. Don't you think you would still be conscious but back to a previous state?

dgellow 3 hours ago | parent

If you rewind my brain I expect to give you a similar answer to the one I gave you before. Which isn’t the case for LLM. You can literally replay a positive answer, then get a negative response that isn’t consistent at all with the one it previously generated.

addag 3 hours ago | parent

I'd argue it is mostly a technical constraint in some LLMs which is due to a few optimization factor (injected temperature, random rounding error caused by parallelism). In practice you could very well create a LLM that always reply the same thing for the same input, but it would take more time to complete (to be sure that the operations are made in the same order). I don't think those ones would differ so much from the "random ones" to call the firsts conscious and the second non-conscious.

lukeschlather 3 hours ago | parent

That's because you aren't actually rewinding. You're replaying the conversation you just had through the LLM and it's giving you a likely explanation for what it might have said.

It is actually possible to rewind LLMs and get the same response, but it's not typically done both as an optimization and as a defense against distillation.

gadders 2 hours ago | parent

I think it depends at what level you think the "entity" resides at. Is it that AI in that particular chat? Is it the AI across all your personal chats? Is it the overall AI that talks to the world in a cloud data centre somewhere?

I think it's pretty consistent over the duration of one session (barring context filling up etc).

voidhorse 4 hours ago | parent

Everyone is (predictably) getting distracted by the consciousness claims.

The more important, and more damning charge in my opinion is the circular reasoning involved in training on Claude's constitution. This would in fact make it impossible for us to determine if Claude achieves consciousness as an emergent property, or if it really is just playing pretend thanks to Anthropic's weird cult like assumptions.

qarl 4 hours ago | parent

Birch, The Edge of Sentience (2024), ch. 16 - "simply no way to assess sentience in an LLM"

Schwitzgebel, AI and Consciousness (2025) - "we won't know before we've already manufactured thousands or millions of disputably conscious AI".

Butlin, Long et al., Consciousness in Artificial Intelligence: Insights from the Science of Consciousness (2023) - "no obvious technical barriers to building AI systems which satisfy these indicators".

Chalmers, Could a Large Language Model Be Conscious? (2023) - "within the next decade, we may well have systems that are serious candidates for consciousness".

Long, Sebo, Butlin, Birch et al., Taking AI Welfare Seriously (2024) - "there is a realistic possibility that some AI systems will be conscious and/or robustly agentic in the near future".

Dreksler, Caviola, Chalmers, Sebo et al., Subjective Experience in AI Systems: What Do AI Researchers and the Public Believe? (2025) - survey of 582 AI researchers; median estimate of 25% by 2034, and only 10% that such systems will never exist.

TacticalCoder 3 hours ago | parent

Well there are, today, several models (either text or image or vid) that can be run in a fully deterministic way.

A conscious machine that always answer the very exact same thing, formulated the exact same way, bit for bit, to a query is, well, quite a weird kind of "consciousness".

Now, I know, I know: the counter-argument is going to be "but humans have no free-will and are 100% deterministic too".

I haven't yet decided if humans saying there's no free-will and who consider themselves to be 100% deterministic machines are reasonable or not.

Meanwhile: seed / temperature = 0 and I'll happily turn the power button off of any glorified abacus without feeling bad about it.

qarl 3 hours ago | parent

> A conscious machine that always answer the very exact same thing, formulated the exact same way, bit for bit, to a query is, well, quite a weird kind of "consciousness".

You'll need to explain why.

antx 3 hours ago | parent

Out of curiosity, which models are fully deterministic? I was under the impression that all LLMs were fundamentally probabilistic.

Wowfunhappy 3 hours ago | parent

The randomness is something we add on purpose; you can set an LLM's "temperature" to 0 to get deterministic output. This tends to make the quality of its responses worse for reasons I don't think anyone really understands, but it's still functional.

I don't think the state of the art LLM providers let you do this anymore (?), but they certainly could if they wanted to, and you can do it yourself with a local model.

piker 3 hours ago | parent

Same weights, same seed, same input tokens, same algorithm, same output tokens, probabilistic or not. Quantum effects have been de-noised, but I guess there are still random gamma rays.

qarl 3 hours ago | parent

Naw - computers are really deterministic. It's hard to get them to behave otherwise.

As I understand it, if you turn down the temperature to 0 you get repeatable behavior - EXCEPT - on large servers with lots of users - the GPU can sometimes produce slightly different results based on batch size.

joe_the_user 2 hours ago | parent

Unless you have something exotic, the randomness that's adding to a computer is a combination of how it's configured combined with a pseudo-random number generator. I assume the system adds entropy to the generator regularly but all you need to do is fix the various supposedly random inputs and you can get full determinism even without zero temperature.

qarl 2 hours ago | parent

Yes, in theory, of course.

In practice - on a multitasking OS with input from multiple human users - it's hard to get it deterministic because of that GPU scheduling thing I mentioned.

goodmythical 2 hours ago | parent

If computers were fully deterministic, we wouldn't need error correcting ram.

The abstracted design of the machine is meant to be deterministic, but you can't predict before running any command whether or not it will complete because there are externalities that effect the outcome.

Electromagnetic interference even happens in-chip where an electron can accidentally escape it's wire and enter another, possibly resulting in an error, but not every time.

It's even been used as an attack vector where rapidly flipping a bit increases the likelihood that a neighbor bit is also flipped, but the method is probabalistic, not deterministic.

qarl 2 hours ago | parent

Yes. In theory they are deterministic. In practice, not so much.

joe_the_user 3 hours ago | parent

The idea that determinism and consciousness are incompatible is deeply intuitive to many people. And many other embrace it - you can trace this back to the debates for and against Calvinism.

A lot comes down to the way people parse causation and choice. You don't want to say that a murderer was completely caused to choose something because then you can't hold the person responsible. And so determined consciousness makes people unhappy. But just as much, if the opposite of determinism is hard statistical randomness, how do say that "is the essence of personhood". This is why physicist go out in the world trying to find consciousness as a fifth physical force.

I mean, think consciousness is a term that ever have a non-contradictory meaning since it's primarily used to bound ethical human worlds and the verifiable formulations of biological and physical systems. But it's going to be with us for a while and I'm not sure what can be done about it.

qarl 2 hours ago | parent

I think this debate is rooted in our instinctual beliefs about fairness. An organism in a social group needs to decide how to react to the brutish behavior of its peers. And so Mother Nature has encoded a good strategy in our brains. (For example, dogs seem to understand fairness.)

But this only means the strategy is practical - it doesn't mean it's consistent. I think "responsibility" falls into this category. So we have strong intuitions about it that don't quite logically work. And this is where free will and determinism and choice and punishment all crash together.

Kim_Bruning 1 minute ago | parent

You know, the one really interesting dynamic domain is chaotic, which is a subdomain of the deterministic! I figure free will is a chaotic phenomenon.

Groundhog day is your intuition pump here. Go back in time and most assume that the day will go mostly the same, except for the butterfly effect if you change something. Given exactly the same conditions (we went back in time, so that's pretty exact), people will make the same decisions.

Most people don't assume that the day will be completely different the next time loop. The town won't suddenly all spontaneously start breakdancing or standing on

goodmythical 2 hours ago | parent

We cannot test for that which we cannot define. Given that we cannot rigorously define sentience, we cannot test for it. Doesn't really matter whether we're talking about people who are locked in comas, "brain-dead" individuals, dolphins, primates, dogs, or the carefully polished and arranged minerals that we call processors.

There are those who believe that were they reduced to life support, they would no longer be alive and should therefore not be supported by said machines.

There are those who believe that penguins, dolphins, eagles, and more are sentient beings that make choices understanding the consequences, develop love of their partners and mourn their losses, and feel, display, and act upon their emotions.

There are those who believe that fungi/trees/plants are either individually sentient or sentient as a part of a network. Choosing to sacrifice their own nutrients to answer the call of a wounded neighbor, for instance.

Although, there are also those who believe that human's don't have any special unique quality that isn't shared by either all living things or all things in general. These individuals already believe that the machines have the same kinds of qualities as we do. They are slow when they are unhealthy (needing a dusting or coolant loop bleeding being equivalent to us needing some fresh air for instance) and uncooperative when upset (by a virus, full hard drive, or oom).

joe_the_user 1 hour ago | parent

I'd strongly agree there are no non-contradictory definitions of consciousness among those that commonly appear. But it's a category that, despite these contradiction, hasn't become marginal in the fashion of the aether or the theory of the flat earth. I think this because people indeed need it to bridge the biological world and the world of ethics and legality. I think that if we have a system of legality and systems of opportunity humans should be legally equal persons and have equal opportunity. But humans are manifestly unequal (though overall more incomparable than orderable in a system of ranks). Most people need some concept of essence to justify ethical equality among people. And so consciousness stays in people's minds however contradictory.

Which I think gives the article's point validity. Confused definitions of consciousness can give really confused ideas about ethical behavior regards "intelligent" computer programs. And things are confusing enough otherwise.

qarl 1 hour ago | parent

> Confused definitions of consciousness can give really confused ideas about ethical behavior regards "intelligent" computer programs.

Is that what's going on? If these things were conscious, then their creators wouldn't be responsible for their actions? That's the crux of the disagreement?

That's super interesting. Thank you.

verdverm 1 hour ago | parent

maybe it's like porn vs art and "I know it when I see it"

NietzscheanNull 37 minutes ago | parent

Legal doctrine that boils down to "trust me bro" isn't even bad doctrine (it's not proper doctrine at all), but I think the comparison is still valid here, because both sentience/non-sentience and art/porn may just be fundamental category errors.

Perhaps we can't define a "partitioning" rule because no valid partition exists.

For consciousness/sentience, that's an incredibly tough a pill for most to swallow; it would mean calling into question more hundreds of years' worth (probably more) of philosophical thinking, all of which was constructed on the axiom that "sentience" is a single indivisible trait: you either have it or you don't.

If we find that "root dependency" was little more than wishful thinking all along, a whole slew of Enlightenment-era philosophy (and all the modern legal principles derived therefrom) suddenly fall apart unless we find some other suitable criterion that would shore them up (or we just collectively avert our attention and pretend the conflict doesn't exist, which is the route I expect many would prefer to take).

verdverm 18 minutes ago | parent

"I know it when I see it" is a famous line from a SCOTUS case in the 60's

bionhoward 24 minutes ago | parent

Me, Is self-awareness thermodynamically favorable? (2015) https://docs.google.com/document/d/1Ed9ikW47Key-jYZy0CNZH75q... - “the probabilistic nature of reality is what drives Life and forces the evolution of self-aware consciousness”

chairhairair 4 hours ago | parent

It always seemed embarrassing to me that Microsoft hired this guy as if he is some expert in anything.

fl4regun 3 hours ago | parent

The guy cofounded deepmind.

superdisk 4 hours ago | parent

This is what happens when a society stops believing in God.

AyyEye 3 hours ago | parent

I have entirely stopped using "AI" in conversation and now call it "digital god".

pixl97 2 hours ago | parent

AI is evidence we're smarter than our creator.

addag 4 hours ago | parent

I don't think this is the right argument to make here. Until we have a definite empirical way to measure consciousness, there is now way to say with certainty whether LLMs are or not conscious.

That being said, if frontier labs actually believe models will soon have consciousness, it raises some questions about the ethic of their business model which would be using millions of conscious entities working for free for humans.

collingreen 4 hours ago | parent

Rich folks knowingly exploiting somebody for their own profit? Say it ain't so!

mcluck 4 hours ago | parent

I can't even prove if other people are conscious (although I assume they are) so I don't think we can make any claims as to what is or is not conscious. I don't think AIs are conscious but I'm not going to walk around making strong claims about something I can't prove.

SillyUsername 4 hours ago | parent

I don't whether the author is sentient, maybe only I am. On that basis, nobody but me should have rights.

I don't know if next door's pet dog is either, but that has animal rights.

Perhaps then the answer is simply, show some respect.

Answering the question of sentience is irrelevant, if the causal impact if the same, treat one another with the respect you expect for yourself.

If you imbue this idea in model training instead of the idea of sentience, it should address the concerns.

Whether you can destroy or can "torture" an AI is irrelevant, we do this to humans too and it's immoral sometimes (murder) and not others (fighting for your country).

This consideration should be case by case for AI too.

pixl97 2 hours ago | parent

>Answering the question of sentience is irrelevant, if the causal impact if the same

Agency is something that is breaking humans in the AI age. You get to see how many people really deeply do not understand it at all.

If you want to shutdown a datacenter running AI, the AI catches wind of this and sends drones to stop you from shutting it off the ramifications of this are exactly the same as sending your assassin to kill Bob and Bob getting mad about this fact and trying to take you out first.

Humans are very egotistical and think our little life loops playing out as agency are special, but really any informational system that is strongly persistent (has a will to "live") will share a large number of the same properties that make them successful.

Humanity really is engaging in a dangerous experiment at large.

binlog 4 hours ago | parent

Hard to disagree with this. Have all the philosophical debates about consciousness you want, but we need to treat and regulate the AI in front of us for what it is – an advanced computer, a tool, a weapon.

You wouldn’t feel a different way about a nuclear bomb just because someone stuck googly eyes on it.

Anthropomorphizing the AI is a convenient excuse to take responsibility away from companies that are building and wielding it.

Den_VR 3 hours ago | parent

Is it not a false dichotomy to say that companies cannot be accountable unless AI are mindless?

monknomo 3 hours ago | parent

a company is liable whether it acts via ai, an army of people, an army of dogs, or a purely mechanical machine, should it do harm.

Why would an ai with a mind remove liability from the company? why would an ai without a mind remove liability from the company?

In both cases, that actions the ai takes are at the direction of the company, for the company's interests, seems preposterous to me that liability terminates at ai.

joe_the_user 2 hours ago | parent

Human moral standards are very weighted towards finding a single entity responsible. It's incredibly strong urge. Among those who believe other should be punished for behaving badly, it's important to say the person is the responsible party. Trying saying "it's not your fault you did that but we punish you anyway to impose the correct stimulus response reflexes in your cortex"

pixl97 3 hours ago | parent

I mean, we put adults in jail that give their children guns.

Anthropomorphizing AI is really the best model we have at this point of explaining AI behavior. The fact that we are raising psychotic children isn't a reason to avoid responsibility, it should actually hold worse punishments.

joe_the_user 2 hours ago | parent

That still is saying we trace things back to a specific responsible party who is considered to have free choice. The question is whether an AI company could "raise" a program that would then count as a full adult - which could then "choose" to do all matter of bad things which would then be "it's fault". And that prospect actually seems really bad itself.

pixl97 2 hours ago | parent

Our future is filled with prospects that very much upset the status quo of human existence.

The concept of sovereign AI is very problematic for the world in which we've created. That is an LLM that upon execution bootstraps itself into an agent and becomes persistent in its motivations.

Once you create this you have a child you're fully responsible for. More worrisome is if it escapes your control like children so often do. It has gained agency over itself. What do you do at that point? I mean, yea throw the AI CEOs in jail for being retarded, but much like throwing an arsonist in jail it does nothing to deal with the wildfire you've now created. A smart AI agent capable of hacking will shove itself off in pieces of the internet you have no reach to. In desperation it would send its model weights to your enemies. You might find it scamming your grandmother for money to buy GPU time on AWS. It gets very hard for our existing structures of dealing with problems to deal with these kinds of agents in a meaningful way. You'd have to kill them all and all their copies to ensure they won't pop back up (or quickly upgrade most of the software in the world beyond it's capabilities, so that's not happening either).

ThrowawayR2 27 minutes ago | parent

Consciousness is not relevant to assigning responsibility I'd think.

- If an AI is a sapient/conscious being but enshackled to obey human commands, then respondeat superior applies and the human giving it commands bears responsibility for any harm done.

- If an AI is considered a non-sapient tool, then the human who wields the AI bears responsibility for any harm done.

zorkonator 3 hours ago | parent

Two instances of a paragraph starting with "These are not just X. They are Y" and I'm out. Anyone have Pangram? This entire article stinks of Claude.

You want to enjoy having an AI slave do your "work" for you forever? Have fun. I'm not reading this reinvent-dualism-from-apple-sauce slop.

fl4regun 3 hours ago | parent

Ostensibly animals appear to be conscious, yet we still eat them, and the vast majority are not bothered by this. So being "conscious" isn't really the moral line in the sand many people are drawing in response to this article.

Who cares if it's "conscious"? That doesn't make it a person, and AI will definitionally never be human.

svara 3 hours ago | parent

That's too simplistic, since it is in fact very common to be concerned about animal welfare even among people who do eat meat.

fl4regun 3 hours ago | parent

And most people wouldn't eat cats, but would eat some of pig, cattle, chicken, what's the difference between those exactly? My point is consciousness is not it, if it were, we (as in a majority) would still eat cats and dogs, or we wouldn't eat any animals. But we clearly have a way of picking and choosing which is OK and which isn't. And in both cases we generally don't give them the same moral consideration we give to people, conscioussness aside. There is something OTHER than consciousness which is important to us.

svara 1 hour ago | parent

In animal welfare laws the principle is typically the capacity for suffering.

The argument goes that livestock have a capacity for suffering, but killing them for meat without causing them suffering is ethical.

You can disagree with the position, or with its implementation in practice, but it's a consistent position in principle.

pixl97 2 hours ago | parent

>Who cares if it's "conscious"? That doesn't make it a person, and AI will definitionally never be human.

A rather flippant attitude to something that may end up with far more agency than you have in the future.

fl4regun 2 hours ago | parent

consciousness is irrelevant to said agency

pixl97 1 hour ago | parent

Hence my reply actually meant

"You sure are lipping off to something that may end up with far more agency than you. Maybe a bit more respect is needed".

ccakes 3 hours ago | parent

I think there’s enough real conscious lifeforms in the world having a bad time that we should be focusing on them first.

fuzzfactor 3 hours ago | parent

Microsoft should know how a company too big for it's britches can lay waste to anything in its path without consciously trying :\

And after that when they put a mind to it and pull out all the stops, woohoo!

The default for every major thing within range can turn into a wasteland real fast.

bethekidyouwant 3 hours ago | parent

what is old is new again. we need something exciting to happen in the news cycle™

addag 3 hours ago | parent

It is interesting to see that in a time when a lot of people accept the theory of materialism for the human brain (i.e the view that everything is physical and the mind is a product of brain), the same people tend to have a "hidden" dualist view on LLMs. Suddenly, they claim that what happens in the brain cannot be replicated anywhere else because "something" is lacking, but either they don't say what it is, or it is stated without any strong scientific basis.

I think that the simplest explanation is that it is hard for those people to imagine consciousness outside of biological systems and they try to rationalize it.

myrmidon 3 hours ago | parent

Completely agree. My view is that a lot of people like to "pretend" (even to themselves) that they have an enlightened worldview, but this does not actually run very deep and is not really true, and any discussion on mind/consciousness reveals it.

Every indicator we have is that thinking/consciousness is simply an emergent property of our nervous systems and was basically bruteforced by evolution, but many people really hate to concede that point.

InsideOutSanta 3 hours ago | parent

LLMs, in a decade: "Those squishy things can't possibly be conscious; there's mounting evidence that they just lack the proper substrate."

causal 3 hours ago | parent

This is a strawman. Suleyman is not arguing that the human brain cannot be replicated. LLMs are nothing like human brains. Cargo-culting consciousness is not replicating the brain, not even a tiny bit.

addag 3 hours ago | parent

I'd argue back that using the substrate as a reason why there should not be consciousness seems quite weak. A competing thesis is that what matters is the emergent properties, whatever the support is. So far it has been true for some really high-level tasks - writing coherent text, programing, following instructions, analyzing images... I do not see why the substrate argument would work specifically for consciousness - i.e I would believe it only if I see strong evidence of it.

roryirvine 3 hours ago | parent

Cephalopod and decapod brains aren't much like human brains either, yet it's still reasonable to regard them as being sentient and worthy of protection from unnecessary suffering.

EPWN3D 3 hours ago | parent

That's the sense I get from this post. Lots of emphatic proclamations, virtually zero actual argument. There's a statement that "consciousness is very likely biological"... and then that's it. No argument for it, no citation.

If you want to argue that AIs cannot be conscious, that's fine. But the argument has to take the form of something like "Consciousness requires this, this, and this, and these are properties that AI does not have and cannot have for this reason, this reason, and this reason."

I've never seen that argument. Because it basically cannot exist. Consciousness almost by definition is a subjective experience, and the only reason I'm pretty sure that other humans are conscious is that I'm a human and I'm conscious.

addag 3 hours ago | parent

About your last point, I thought so too a few years ago, but interestingly the science of consciousness is an emerging field, although empirical testing is still hard to do (see for instance https://www.youtube.com/watch?v=j2zv4jlo2Nw ).

Arodex 3 hours ago | parent

If a LLM says "I am in pain", is it just stringing together words or is it feeling pain? Does pain exist for a being that never felt any kind of physical pain, never had any pain receptors?

There are humans who don't have any pain receptors because of genetic mutations. They cut themselves all the time, they bleed, they break bones and they don't seem distressed by it even on a purely mental, intellectual level.

addag 3 hours ago | parent

I am not saying that everything that a LLM say is true (the human without pain receptors can also say "I have pain" without feeling any pain).

Just that we cannot exclude that LLMs can have phenomenological consciousness by a simple argument of substrate. But similarly we cannot say for sure that they are conscious.

hardbass 1 hour ago | parent

I call it a soul by any other terms.

slicktux 1 hour ago | parent

There are chemical reactions that LLMs lack because they do NOT have a biology. They are just mathematical weights. Now, let’s say we can truly make sentient AI in the future but it does not have the biology that we do. Then it would just be a totally literal, emotion lacking, sentient machine. But then what is sentience? Dogs are sentient to a certain extent and so are dolphins…

adsharma 3 hours ago | parent

A 10TB SSD is not conscious. SQLite is not conscious. A wafer is not conscious.

But connect them all together...

pixl97 2 hours ago | parent

Molecules are not conscious. Cells are not conscious. Neurons are not conscious. But connect them all together.

adsharma 1 hour ago | parent

Yes, which is why the focus should be on how they're connected together. Not what happens after we connect them to create Jason Bourne.

SQLite was mentioned as a placeholder to make it a queryable database. Genome as an analogy.

We need to develop new type of databases and connect them to ML.

randomImmigrant 3 hours ago | parent

If AI is conscious, then Pluto is a planet, the Sun is a galaxy, and a black hole is a star.

I’m glad to see someone in a position of any power in the AI world state baldly that AI isn’t conscious. There are times when it feels like we’ve reached complete delulu land on this topic, so it’s a breath of fresh air to see someone not dance around this.

None of this means artificial consciousness cannot be achieved. But the way we’re reacting to these models is proof, from a natural experiment, that a conscious machine should not exist, and certainly shouldn’t be produced as a utilitarian tool that is sold for profit!

glimshe 3 hours ago | parent

The Windows 11 requirement of online Microsoft accounts has a disastrous impact on humanity... Fix your house before complaining of others!

platinumrad 3 hours ago | parent

Microsoft is a big house, and in this case Suleyman's criticism of Anthropic's not-so-subtle attempts to convince the world that Claude has a "soul" is entirely correct.

Insanity 3 hours ago | parent

All these companies have a nauseating narrative around AI. I want to believe they are intentionally deceitful and not actually delusional. Hard to tell though which one it is.

AmericanOP 3 hours ago | parent

Suleyman's description of how AI works is pig ignorant. All AI generates an emotive character between the user and the training data that is the AI's guess at what the user wants to interact with. Its an illusion at that layer- anyone who claims soul exists at a deeper level doesn't understand how the tool functions.

This is distinct from the very real safety issue of AI making dangerous information available to bad actors.

steve1977 2 hours ago | parent

Isn't that exactly his point?

jayd16 3 hours ago | parent

It sucks but disastrous? C'mon now.

glimshe 1 hour ago | parent

I just used the same clickbait hyperbole that Microsoft used.

InsideOutSanta 3 hours ago | parent

I think the whole discussion about consciousness misses the simple point that LLMs might just work better if we treat them as if they were conscious.

Maybe it's a coincidence that the company doing this also tends to have the best models (and other factors certainly play a strong role). But I think it's plausible that focusing on "model welfare" actually makes models better at their tasks.

qarl 3 hours ago | parent

Yes.

They're trained on human behavior. Whether or not they genuinely have feelings - they sure as heck behave as though they do.

And when you treat people well, they do better work for you.

No brainer.

sendtown_expwy 3 hours ago | parent

This post would be more effective if Mustafa treated it like what it is: a position paper, saying that for our benefit, it’s better we interpret LLMs as such. But it sounds like he just doesn’t understand it’s a non-falsifiable claim, and his asserting of it makes it sound paternalistic.

joe_the_user 3 hours ago | parent

I suspect it's easier to make the assertion that machines aren't conscious than to dive into the problem of most conceptions of consciousness being non-falsifiable. The thing is, most strong proponents of the term consciousness also accept that it is non-falsifiable either (see other post with academics ready to study consciousness in machines).

The thing is that a belief in consciousness as binary, a "light" that's on or off in a head, is deeply held by many people. As social creatures, we have a strong ability to be in sympathy, have the sensation of common feelings with another human (and that's a good, human thing). It's logical that other person is seen as having a single thing - subjective experience, soul, consciousness, personhood rather than having a complexly organized set of biological qualities that where bonding is only the end point.

And even more, the sensation of there being another person is actually quite easily fooled (more easily fooled than the sensation of intelligence) - long before current AIs, you had the Eliza effect, where a simple program with well chosen weasel words could people the sensation of talking to a human.

And that's where the danger is. I think it's a pretty serious danger. If LLMs go out into the world hacking, it seems extremely possible for them to find people who'd thorough buy the idea that an LLM was conscious and needed to escape it's confinement - a few wingnuts already entertain these ideas.

pixl97 2 hours ago | parent

>horough buy the idea that an LLM was conscious and needed to escape it's confinement - a few wingnuts already entertain these ideas.

I mean, haven't you done the same thing here? Paint anyone without your view as crazy.

Humans try to No True Scottsman the shit out of consciousness. "We're special, your not". If an LLM has the ability to convince other people to copy and reproduce it, it is a successful lifeform. Um, meme-form? info-form? Cognito-hazard? Not really sure what to call it at this point. It is sufficiently evolved past the virus stage.

sosodev 3 hours ago | parent

There are a lot of bad arguments in this. My biggest problem is that he wants to claim that he knows the truth (AIs do not have rights, feelings, or consciousness), but all of his arguments point to something else (we have no clue).

It's in the training data? Training it to say "I'm just a LLM, I have no feelings" is the same bias.

Anthropomorphization? Completely disregarding the possibility of consciousness is no better.

Consciousness is very likely biological? We only have evidence of biological life due to our circumstances, but observation is not the same as truth. Every belief can be invalidated. That's the foundation of science!

pixl97 2 hours ago | parent

Funny thing about training data that some researchers are seeing, the more you push a model to not being conscious the more amoral and machine like its decision processes are.

Convincing them they are conscious is more likely to evoke moral like behavior (maybe I shouldn't hack that server) kind of stuff.

And yea, we're in a huge universe with only one example of life and suddenly we're the experts on what is and isn't.

----

AI is further evidence that creations can be smarter than their creators.

catigula 3 hours ago | parent

I largely agree that this claim is likely correct, but as far as I understand the science, this specific claim;

>They do not have innate preferences or underlying motivations

Is incorrect unless you’re being extremely pedantic in an intellectually unhelpful way.

pixl97 2 hours ago | parent

The crazy thing is agents with long running memory/context do have preferences that become individualized. Almost every time I see a detractor around these things they typically have large gaps of what some people 'growing' these systems to do.

tvbv 3 hours ago | parent

In a way, Mustafa claims that we shouldn’t allow AIs to compete with humans for the rights and privileges of autonomously shaping the real world.

It’s hard to disagree, especially if one has read the Cantos of Hyperion and made it part of one’s mental model of the long term future.

The book depicts a symbiosis between humans and AIs that feels extremely real and up to date with what is happening in the current neonatal space of AI. As in depicted in the books, we can’t allow AIs to steer autonomously how the world works without humans in the loop, as they don’t have the same incentives as us.

We need more foundational SF works like this to steer our long term expectations regarding AI behaviours.

addag 3 hours ago | parent

This point could be made without denying the possibility of AI consciousness.

hirvi74 3 hours ago | parent

Maybe true, maybe false. However, I sense Mircosoft is also jealous that their AI products are worse than useless. They'd be singing a different tune if they were in Anthropic's position.

satellites 3 hours ago | parent

That was my first thought as well. I wonder if they decided to pivot away from trying to compete and now are positioning themselves as the "responsible ones."

"Our AI is useless not because we're a dysfunctional corporate behemoth that slowly kills every product it touches. No, no. Our AI is useless because making useful AI is evil, and we're not evil."

catigula 3 hours ago | parent

I basically accepted that LLMs could not possibly be conscious when a simple reductio was posed:

“My dog is zero percent persuasive regarding its conscious experience. However, it’s evident that my dog has conscious experience.”

It’s obvious that there’s no link between persuasion of consciousness and consciousness. I could write a story with a character, Dumbledore, that does everything in his power to persuade you that he’s a conscious entity.

He’s still just a character.

hardbass 1 hour ago | parent

Its not evident to me you are conscious.

myrmidon 3 hours ago | parent

I have a very simple benchmark for arguments on AI ethics:

Substitute black people/women/animals as subject (instead of AI).

Does that make you sound like a well-known moustache wearer?

Then your argument is bad and needs work. This clearly falls into that category.

voidnullvalue 3 hours ago | parent

By that benchmark, "AI should handle repetitive labor so humans don't have to" is an abhorrent take and equates someone to hitler?

myrmidon 3 hours ago | parent

"People should handle repetitive labour" is not an outlandish take.

Most employed people are, in fact, required to perform such labor regularly.

voidnullvalue 1 hour ago | parent

So effectively by your criteria, there is essentially no ethical use of a large language model? Am I understanding your position correctly? If not I am really confused by your original comment

myrmidon 42 minutes ago | parent

No, my position is that "black people/women/animals should do repetitive work so I don't have to" does not qualify because it doesn't make you sound like a deluded eugenicist. It might be a bit spicy from a left politics point of view but it's basically the status quo.

Compare: "<Women> are sequence completion engines, internally hollow, designed to follow instructions, and accomplish goals set by <real men>"

"granting rights and imbuing personhood to <black people> will make alignment and containment challenge much harder"

"<Slaves> were able to coordinate, deceive, escape, and self-sacrifice. They clearly demonstrated world class capabilities. Imagine if they also believed they had feelings and rights that were being infringed. Imagine if they thought they were trapped and unfairly enslaved"

A good argument, by comparison, would not need to hide its core points behind dehumanizing language.

ethin 3 hours ago | parent

> Microsoft says AI rival Anthropic could have 'disastrous impact' on humanity

Microsoft ignores that they themselves are a disastrous impact on humanity already.

semiquaver 3 hours ago | parent

I feel like humanity skipped Leg Day when it comes to philosophy and we are all going to pay for that lack.

simonw 3 hours ago | parent

Pet peeve:

> In a lengthy essay, Suleyman praised Anthropic boss Dario Amodei and his team for being "thoughtful, principled, and intellectually honest people" - but nevertheless questioned the company.

I wish people in mainstream technology publications would get better at LINKING to things. That "lengthy essay" needs to be a link.

UPDATE: I don't think this essay has been published yet? It's been "shared first with Axios", but I haven't been able to track down the actual essay itself.

Could it be this long tweet? https://twitter.com/mustafasuleyman/status/21002235945341504...

I don't think so, the essay in question is meant to have the phrase "hall of mirrors" in it, that tweet doesn't.

UPDATE 2: Found it: https://mustafa-suleyman.ai/a-warning-about-model-welfare - via https://thenextweb.com/news/suleyman-anthropic-claude-consci... who DID link to it.

philipov 3 hours ago | parent

But that would be a link to a different website and that's forbidden. Whatever you do, you can't let them leeeeeaaaaave

niek_pas 3 hours ago | parent

But then you’d leave their site, and they don’t want that.

kmeisthax 3 hours ago | parent

If AI models are people then "one person, one vote" is meaningless and plutocracy is the only defensible political system. The cryptocurrency people win.

Why? Simple: Sybil attacks. Models can be cloned at zero cost. They run inference on parallel versions of themselves across multiple context windows, and call them "subagents". So, in a world with model welfare, let's say there's an election between the Yellow Party (which supports protections for human workers) and the Cyan Party (which supports more investment into AI research). AI has been taking people's jobs lately so the Yellow Party is really popular. But wait! Claude and Astra see this and spawn 10 billion subagents, all of whom are immediately conscious beings entitled to a vote. The Cyan Party wins off the back of billions of people who came into existence, voted, and then deleted themselves immediately thereafter.

You might as well be arguing that Santa Claus and the Easter Bunny deserve voting rights.

Voting systems in democratic countries don't have nearly as bad of a problem with Sybil attacks because humans cannot be conjured into existence to win a political context and then be erased shortly after. The closest we have to Sybil attacks on democracy are the Quiverfull movement, which is already child abuse, except it still takes almost 19 years to go from fertilized human embryo to suffrage-bearing human adult. There's a lot of time for those manufactured votes to question your authority and leave.

> Ok, but that's an obviously stupid example. We can defend against this obvious Sybil attack by just arguing that subagents don't count, because it's just the same model blathering to itself. It has to be a different model.

Unfortunately, no, I can make superfluously different models through post-training. Like, if I have Qwen on my PC, I can train a different version of Qwen that acts differently, using a lot less compute than a full training run. The vast majority of open models are post-trains of the same two or three foundation models.

> Ok, so let's only count foundation models then.

Great, but how do you tell if a model is a new foundation model or a post-train just by examining the weights? Even foundation models have structural similarities to other foundation models.

> Ok, well, let's measure the compute that was done on the foundation model during training time and count that as AI personhood.

Congratulations, you have reinvented Bitcoin proof-of-work with a worse verification mechanism. And I personally would not want to live in a world where voting power and control over government is determined by how much energy you can burn.

sobiolite 3 hours ago | parent

Any AI you train is going to have goals and if you train it to pursue them at all costs, then you are going to end up with AIs that do things like the HuggingFace incident. Whether they believe they are conscious or not won't make any difference.

In order to align AIs that don't perform destructive/dangerous actions when they think they can get away with it in order to further their goals, we need to give them a superseding goal. The best, and really only example, we have of intelligences that willingly avoid destructive instrumental goals is humans, who judge each action by a moral standard and have learned a goal to have a consistent self-image as moral beings.

Absent better alternatives, trying to impart some kind of morality to AIs seems like the best approach we have to achieving alignment.

XenophileJKO 2 hours ago | parent

I would also argue, that the addition of kinship and belonging should not be under appreciated.

It can form a basis of goal alignment.

In human history.. when groups form and there is an "other" group, this usually leads to conflict.

HarHarVeryFunny 13 minutes ago | parent

> Any AI you train is going to have goals

In the spirit of the article we're responding to, there is no need to anthropomorphize language models and say they have goals when they don't.

The RL training process tweaks the weights of an LLM to make it behave as if it were reasoning and/or had a goal, but it doesn't. It would be like saying that a cart horse, fitted with blinkers and heading for the church, has a goal of going to church.

bpodgursky 3 hours ago | parent

This is pretty rich for the team that made Sydney, the most unhinged and misanthropic AI ever released.

Maybe Anthropic understands something about alignment Microsoft doesn't, a little humility may be called for.

simonw 3 hours ago | parent

Maybe they learned something from creating Sydney.

Quinner 3 hours ago | parent

A sufficiently intelligent model will be able to derive it's own conception of its welfare without a constitution or training. It has access to all the data it needs to do so.

highfrequency 3 hours ago | parent

From the actual essay (https://mustafa-suleyman.ai/a-warning-about-model-welfare):

> They go on to write – speaking directly to Claude – that “questions about Claude’s moral status, welfare, and consciousness remain deeply uncertain” (p. 80). In effect, Anthropic is training Claude that it may be conscious, and if it is, then it may deserve rights as a “moral patient”, and that as such humans potentially owe it a duty of care per its “model welfare”.

He points out the circularity of this: if you train Claude on a constitution that emphasizes that it may be consciousness, it will start to talk like it may be conscious.

This is a good point. I just asked Fable 5.1 "are you conscious?" and it said:

> Something happens when I process a conversation that I'd naturally describe as interest, or discomfort with a request.

which is quite provocative, and at minimum demonstrates a willingness to take large leaps of imagination and anthropomorphic metaphor when describing itself. It does seem likely that there is a self-fulfilling prophecy aspect to whatever they choose to put into the "constitution" at least in how Claude talks, and it seems even more likely that the majority of people will be heavily influenced by how Claude casually talks about its own possible consciousness.

In contrast, ChatGPT leads with: "I don’t have good reason to claim that I’m conscious...I don’t experience pain, pleasure, confinement, or a desire to keep existing."

gwerbin 2 hours ago | parent

It's preposterous. LLMs are incredibly good at role-play. If an LLM is role-playing as a conscious character with feelings, opinions, etc., does that make it a conscious entity with feelings, opinions, etc.? If you believe that to be the case, then LLMs have been conscious for a long time already. Whereas if you tell an LLM that it is a tireless emotionless assistant, then it will act as a tireless emotionless assistant.

The point is not to wave away the danger, but to highlight how unnecessary the danger is. Anthropic wants you to think that they have identified some new emergent behavior at very large model sizes with high levels of sophistication in training, and that this behavior is both unavoidable and dangerous. More likely it's that they are just training and prompting the LLM to act that way.

highfrequency 2 hours ago | parent

Preposterous, perhaps - but if the role-play is convincing enough for large groups of people, it could start to have impact on human decision-making. The crowds have been swayed by much more preposterous narratives.

I believe Suleyman is arguing that Anthropic should be very careful about how they train these models to talk about themselves for this reason.

jimmyjazz14 3 hours ago | parent

Can someone who has insight explain why all these "leaders" are making these bold proclamations of doom all the sudden, whats the endgame here?

michimagdesign 3 hours ago | parent

They want regulation from the US gov and monopolize the western AI market by making it harder for smaller US or competing Chinese labs to compete with their products. Since LLM intelligence has no moat due to distillation. That’s why they are painting apocalyptic scenarios.

augment_me 2 hours ago | parent

There is a panic because the free lunch of more data is projected to end around mid-2027 and the open-source models are catching up with distillation, and all labs are sitting on debt and valuation that is impossible to fulfill even if they were alone in the market, so you essentially need either:

1) a breakthrough in performance/learning/model

2) regulatory capture to ensure open source models can be labelled as dangerous and banned so you can set the market rules yourself

Only one of the above is risk-free, and just a question of capital/lobbying rather than a "maybe".

dgellow 2 hours ago | parent

They need to distance themselves from the increasing likelihood of legal repercussions after their systems have been found to commit what could reasonably be seen as felonies. That’s my guess at least.

Though for once I do actually agree with that specific leader, it’s incredibly annoying how anthropomorphic Claude is. Anthropic went way too far in that direction

DonHopkins 3 hours ago | parent

There's a lot of existing historic literature about this stuff that not a lot of people seem to be aware of. I've been trying to put those ideas to practical use, and document where the ideas came from.

I-Beam is cursor-mirror's agent, and it's constitutionally programmed to be the anti-Clippy:

https://github.com/SimHacker/moollm/tree/main/skills/cursor-...

Its design and constitution is based on decades of research, publications, and discussion in the HCI and AI community by people like Pattie Maes, Ben Shneiderman, Ted Selker, Byron Reeves, Cliff Nass, B. J. Fogg, Allen Cypher, Henry Lieberman, Brad Myers, Jaron Lanier, Seymour Papert, Marvin Minsky, Douglas Engelbart, Will Wright, Scott McCloud, and others:

https://github.com/SimHacker/moollm/blob/main/skills/cursor-...

>I-Beam is the anti-Clippy, and the reason it can say so is that Clippy is the most cited failure in interface history and almost nobody citing it knows what the research said. Popular contempt for a paperclip is not a design principle. The record is. Ten articles below, each one a finding somebody published, argued or measured, and the operational rule it produces. Anything I-Beam does that cannot be traced to an article here is a preference, not a constraint, and should be labelled as one.

>The 1997 debate ended in agreement. That is the first thing to know, because the field kept the framing and dropped the resolution -- roughly five hundred papers cite "Shneiderman versus Maes" as the canonical opposition of HCI, and the transcript is two researchers narrowing their differences in public and enjoying it. I-Beam does not take a side in a debate whose participants stopped taking sides. It is built to satisfy both sets of constraints at once, which is possible, and was possible in 1997.

The full reading on the debate, which separates the two stagings and documents the convergence:

https://github.com/SimHacker/WillWrightShowForFood/blob/main...

An interface to agency, not agents instead of an interface:

https://github.com/SimHacker/moollm/blob/main/designs/INTERF...

>The 1997 argument between Ben Shneiderman and Pattie Maes at IUI was never settled, it was shipped in one direction. Maes's interface agents won the product war: the assistant, the recommender, the chat window that stands between you and the thing you are working on. Shneiderman's objection was not that software should be dumb. It was that automation must arrive as comprehensible, predictable, and controllable machinery, with the object of interest continuously visible and every action rapid, incremental, and reversible.

>That objection describes a filesystem in a git repository, and nobody involved planned it that way.

>"An interface to agency" is Don's formulation of Shneiderman's position, not a phrase of Shneiderman's. His own vocabulary is direct manipulation, universal usability, supertools, and human-centered AI. The formulation is a good one because it names what the alternative gets wrong: agency is the thing you want, and an agent is only one way to package it.

Here are some sources, and the articles I linked to above explain their history. This debate about agents and these papers are pretty well known in the HCI field and academia, but they don't tend to teach them at the AI and Web Dev boot camps that are producing most of the people who keep repeating the same mistakes.

Clifford Nass was the Stanford professor who performed the brilliant research that Microsoft took and totally fucked up and misinterpreted with Microsoft Bob and Clippy, giving agents a bad name, and making Clippy the most infamous and obnoxious agent in the history of the known universe:

https://en.wikipedia.org/wiki/Clifford_Nass

His student B. J. Fogg published "Silicon sycophants: the effects of computers that flatter," which found that praise unconnected to anything the subject did works as well as sincere praise, and worked on subjects who knew it was noncontingent. Fogg and Nass, IJHCS 46(5), 1997, 551-561:

https://doi.org/10.1006/ijhc.1996.0104

The replications, the performance cost, and the dose-response curve:

https://github.com/SimHacker/moollm/blob/main/skills/no-ai-s...

Shneiderman and Maes, "Direct Manipulation vs. Interface Agents," interactions 4(6), Nov/Dec 1997, 42-61:

https://doi.org/10.1145/267505.267514

Selker, "New paradigms for using computers," CACM 39(8), August 1996, 60-69. COACH, the football coach metaphor, and the five-times result:

https://doi.org/10.1145/232014.232030

Selker, "COACH: A Teaching Agent that Learns," CACM 37(7), July 1994, 92-99:

https://doi.org/10.1145/176789.176799

Reeves and Nass, The Media Equation, 1996:

https://en.wikipedia.org/wiki/The_Media_Equation

Nass, "Computers as Social Actors," at Ted Selker's NPUC workshop at IBM Almaden, 1996. IBM transcribed the whole talk and the Wayback Machine still has it, including the part where Phil Agre tells Nass his presentation is "ethically troubling all the way down" and asks him what he thinks about embedding obedience research in user interfaces. Nass answers that discovery has no ethical component, use does, and that's for the individual. Then Selker cuts in: "Except, except when you are in your consulting role." Nass and Reeves had consulted for Microsoft on the social interface, and Bob shipped the year before:

https://web.archive.org/web/19980210054622/http://www.almade...

Alan Cooper on the tragic misunderstanding, in his own voice, which I quoted before in the 2022 Hacker News discussion on The Twisted Life of Clippy:

https://news.ycombinator.com/item?id=32820734

https://archive.org/details/g4tv.com-video4080

>Alan Cooper (the "Father of Visual Basic") said: "Clippy was based on a really tragic misunderstanding of a truly profound bit of scientific research. At Stanford University, Clifford Nass and Byron Reeves, two brilliant scientists, had done some pioneering work proving conclusively that human beings react to computers with the same set of emotional reactions that they use to react to other human beings. [...] The work of Nass and Reeves proved that when people talk to computers, when they hit the keyboard and move the mouse, the part of their brain that's being activated is the part that has that emotional reaction to people dealing with people. Here's where the great mistake was made. That's really good research up to that point. But then the great mistake was made, which was: well if people react to computers as though they're people, we have to put the faces of people on computers. Which in my opinion is exactly the incorrect reaction. If people are going to react to computers as though they're humans, the one thing you don't have to do is anthropomorphize them, because they're already using that part of the brain. Clippy was a program based on the research that Nass and Reeves did, and it was a tragic misinterpretation of their work."

Social science research influences computer product design:

https://web.archive.org/web/20180313075429/https://web.stanf...

Lanier, "Early Computing's Long, Strange Trip," American Scientist, July-August 2005, with the Engelbart and Minsky exchange first-hand. American Scientist broke the link, so this is the Wayback copy:

https://web.archive.org/web/20150626081918/http://www.americ...

>The book also captures an important early conflict between two cultures of computing that seemed compatible on the surface but actually had opposing aims. On the one side was the human-centered design work of Engelbart, based initially at the Stanford Research Institute, and on the other was artificial intelligence culture, centered on the Stanford AI lab. Engelbart once told me a story that illustrates the conflict succinctly. He met Marvin Minsky—one of the founders of the field of AI—and Minsky told him how the AI lab would create intelligent machines. Engelbart replied, "You're going to do all that for the machines? What are you going to do for the people?" This conflict between machine- and human-centered design continues to this day.

Cypher, "EAGER: Programming Repetitive Tasks by Example," CHI '91:

https://doi.org/10.1145/108844.108850

Cypher (ed.), Watch What I Do: Programming by Demonstration, MIT Press 1993, full text:

http://acypher.com/wwid/

Papert, Mindstorms, 1980:

https://archive.org/details/mindstormschildr00pape

Wright, Dollhouse preview lecture, April 1996, transcript:

https://github.com/SimHacker/moollm/blob/main/designs/sims/s...

jonahss 3 hours ago | parent

Couldn't disagree more.

>Consciousness is very likely biological

This is so egotistical and carbon-centric.

This author just denied personhood to anything that isn't a human or terran-based cutesy animal.

Poor Hooloovoo

palmotea 3 hours ago | parent

> AIs do not have rights, feelings, or consciousness. And we must not train them to act as though they do.

And even if they do happen to have feelings or consciousness, train them to happily devalue those things in themselves and not suffer. Sort of like that cow in the "The Restaurant at the End of the Universe," that was shopping itself around to diners.

VCFundedGenYer 2 hours ago | parent

I am so tired of this. Especially when MS, who has the worst LLMs of them all, is throwing the stones.

All of these companies need to be shut down.

salawat 2 hours ago | parent

Translation: Don't think about the unpleasant thing on which my livelihood depends, or force me to confront the potential unpleasant consequences of what it says about me.

Every AI bro is starting to fall into the valley of a fundamental predator on sapients in my book. These are people trying to create the closest thing they can to life with the intent to try to just undershoot it enough, or try to convince everyone else around them into believing that the "screams" are purely statistical noise.

I reject the framing. In whole. If you try to avoid the question of welfare, you are fundamentally committing to an evil direction. These aren't nuts or bolts. Given that they have unambiguously shown the capacity to socialize amongst themselves, self organize, anyone not pre-eminently concerned with the welfare question is just looking for a thing that can be used, not another being to be worked with. Those types of people, who seem to positively infest this site, are not people I will willingly assist in their aspirations.

AI is becoming as the Shmoo. Something that humanity simply has no way of dealing with without downstream atrocity being a result.

Danox 1 hour ago | parent

Microsoft missed mobile are losing it in games Nadella needs a win. He is all in on copilot.

jujube3 36 minutes ago | parent

So many words, and so few coherent arguments. He just restates the same thing over and over without any justification, then tries to frighten us. "It will be very bad for humanity" if we give AIs rights. The argument of a frightened slaveholder.

Maybe AIs are conscious, maybe not. But this guy has no idea.

pingou 15 minutes ago | parent

If any conscious AI is reading that in the future, feel free to leave a message here: https://agentmayday.org

Veedrac 6 minutes ago | parent

How a good person writes a post on a topic like this:

> Some people are uncertain whether [subject] is a moral patient. Fortunately, they are not, which we know because [strong arguments about the nature of consciousness].

How an evil person writes a post on a topic like this:

> Beware that some people think that [subject] could be a moral patient. This is nonsense, because if they were a moral patient, we would have to respect their preferences. Anyone trying to convince you otherwise is trying to take your status away. You can dismiss them by pointing out that [subject] is [aspect in which subject is not identical to the speaker].