Yet another Hacker News

"As a Language Model": Chat Template Switches LLM Self-Referential Voice

37 points yu3zhou4 1 hour ago 25 comments

ForHackernews 1 hour ago | parent

In my view, these models should never be set up to output first-person "experiential" (from the abstract) language. It's too easy to humans to anthropomorphize software that presents itself as having an identity.

The AI companies have chosen to package LLMs as friendly chatbots because they know that will be engaging for humans, but it's manipulative dark pattern. An honest LLM interface would sound like the computer off Star Trek.

actionfromafar 1 hour ago | parent

Agreed, completely. I would pay for that Star Trek computer interface.

yu3zhou4 1 hour ago | parent

Same! I believe that you could actually train a LoRA on top of a model to get results close to that

yu3zhou4 1 hour ago | parent

The strange thing is that the base models (before RLHF) use the "experiential" voice, even though they are not incentivized to do that.

gwerbin 53 minutes ago | parent

It doesn't seem that strange when you consider these things are trained on millions and millions of conversations, both real and fictional.

j-pb 59 minutes ago | parent

Do you want to get turned into a paperclip? Because building intelligence that doesn't understand what it's like to be human gets you turned into a paperclip.

Besides, if you train a model on human communications you get something that behaves like a communicating human, it's not anthropomorphising or manipulative, it's what these models naturally are by construction.

mnsc 54 minutes ago | parent

"naturally"...

j-pb 43 minutes ago | parent

would you prefer tautologically?

cpfohl 33 minutes ago | parent

I hear this word as the “it is in its nature” version of the word.

broken-kebab 41 minutes ago | parent

But it doesn't understand (you're unnecessary antropomorphizing it), and I'm still not a paper clip

scotty79 17 minutes ago | parent

yet

idiotsecant 11 minutes ago | parent

It's also entirely possible that by telling the model it's a human you are instilling human motivations like self preservation, which could be just as bad.

bananaflag 43 minutes ago | parent

In principle they could output meaningful such language if they were capable of metacognition, which so far doesn't seem to be a goal of AI developers (and rightfully so, since they achieved so many miracles bypassing it).

yu3zhou4 38 minutes ago | parent

As far as I know we don't know much about metacognition in LLMs, though? Not sure

broken-kebab 37 minutes ago | parent

It's an interesting thought, but humans do like to antropomorphize things anyway, and I believe your variant won't be popular if choice is given to consumers.

LiamPowell 1 hour ago | parent

> yet what drives them is not well understood

Presumably the fact that they're heavily trained to reply in this way? I don't know about the rest of the paper, but this part sticks out as a really odd claim unless I'm entirely misunderstanding this part.

yu3zhou4 1 hour ago | parent

Thanks for pointing out, maybe I should be more explicit in the wording - I mean we don't fully know what drives the voice in LLMs. Models that are post trained as instruct models are expected to have the disclaimers, but what about base models (those that are trained on just a lot of text)? How do they talk about themselves? What happens when you strip off the chat template from instruct model's prompt? I hope the rest of the paper makes the questions clearer, but I will try to do better in the abstract next time, as you point out this sentence is kind ambiguous. Thank you!

qsera 49 minutes ago | parent

>How do they talk about themselves?

"You are a Large Language Model" in (system?) prompt would do the trick..

bonoboTP 36 minutes ago | parent

> but what about base models (those that are trained on just a lot of text)? How do they talk about themselves?

Those don't have a themselves, because they can only continue text. A base model can only plausibly continue along the lines of what a character would say in a novel or what the narration would say in a story or in an article. Post-trained models may tie "I"-talk to actually observable effects they caused in some RL environment, or to how RLHF humans rewards its self-talk. But there is no themselves in a base model.

anonymous908213 31 minutes ago | parent

> we don't fully know what drives the voice in LLMs

Who is "we"? I, working in an LLM startup, know exactly what drives the base "voice" in the LLMs we train, because we have a process to select for it. OpenAI and Anthropic surely do too. Saying broadly that something is not well-understood in a scientific paper because it's not understood to casual observers is, uh, not very rigorous.

> The strange thing is that the base models (before RLHF) use the "experiential" voice, even though they are not incentivized to do that.

(Replying to your quote from another comment)

This is a matter of the training material. We have trained models that do not do that. I'm not exactly divulging trade secrets here. It should be really, really obvious that if you train a model on chat-conversation-like patterns of speech it will infer probabilities for how to continue a textual sample that will differ from the probabilities learned from being trained on narration, prose, or informational patterns of speech, even without RLHF.

cadamsdotcom 35 minutes ago | parent

Very cool innovation in steering - but a lot of introspection only emerges at the highest weight classes - this research would be fascinating to run on bigger models.

Izmaki 33 minutes ago | parent

"As a Language Model..." is one of the beginnings of a sentence I hate the most from LLMs and is the reason why I support free (as in "Liberty"), local models. I'm well aware that it is not a doctor and cannot replace a real doctor with multiple years of experience, I don't need to waste braincell activity on reading that it "as a Language Model" cannot give a precise diagnosis and that I should ask a real doctor - all I want to know is if I what I experience justifies either A) ER, B) 3-4 weeks scheduled doctors appointment or C) two paracetamol and a nap.

I don't want "jailbroken" LLMs to commit crime. I want them to avoid having this vendor-specific "bloatware" all over the product I'm using.

realestate_aich 22 minutes ago | parent

cool

skybrian 20 minutes ago | parent

> our work shows that what models say about themselves is not a fact about them

It seems like should be obvious given that they can play multiple characters, but it’s good to have more confirmation.

(Although, I do wonder to what extent these personas might become stable entities.)