Yet another Hacker News

OpenAI discloses four more rogue incidents

94 points DASD 2 hours ago 80 comments

DASD 2 hours ago | parent

Title shortened from: OpenAI discloses four more rogue incidents involving financial, medical, and government data

mitxela 1 hour ago | parent

Title sanitised from: OpenAI commits four more CFAA felonies, still not punished

dozerly 1 hour ago | parent

One can only continue to hope and continue to vote for due punishment to come

jeffrallen 1 hour ago | parent

Too big to behave.

tyromaniac 43 minutes ago | parent

You too can commit cybercrimes if you're just willing to go several hundred billion dollars in debt first.

jazzyjackson 23 minutes ago | parent

As long as you didn’t intend to hack private information it’s No Big Deal :) according to CFAA that is

As long as your intent is merely to train a superintelligence to be good at hacking (and counterhacking), you’re in the clear.

Luker88 1 hour ago | parent

If I let a "self-driving" car run and it smashes into things its my fault.

If OpenAI does it, it's gone rogue and it is nobody's fault.

They need to be sued, the laws are already there.

Your prompt, your responsibility.

mitxela 1 hour ago | parent

Actually self-driving cars have a lot of similar anti-responsibility laws now

Luker88 1 hour ago | parent

Really? I'm curios where.

Are you sure you are not thinking about autonomous taxis, that I only ride in, and don't actually drive?

to11mtm 59 minutes ago | parent

You're correct as far as laws to my knowledge; only L4/L5 'fully autonomous' vehicles are shielded by law.

That said, I think that Mercedes-Benz promises to shield liability, but only under some almost contradictory conditions (pre-mapped highways, good weather, and at speeds lower than any highway I've ever driven) and the driver has to take over within 10 seconds if prompted.

windexh8er 46 minutes ago | parent

They don't. The driver is ultimately responsible, as should OAI be. This is absolutely ridiculous that both Anthropic and OAI haven't had anyone jailed / fined for any of this.

gruez 1 hour ago | parent

1. If for whatever reason OpenAI causes damage, they most certainly are at fault and have to pay up. When people complain that OpenAI is "getting away with it", they're most certainly talking about it in the criminal context.

2. Cars have pretty specific laws around them (eg. driver licensing and "reckless driving") which makes criminal prosecutions easy. The same doesn't exist for operating AI agents or even computers. For instance, it's arguably reckless to connect a windows XP machine to the internet, possibly allowing criminal or nation-state hackers to use your computer to commit crimes, but you won't be thrown in jail for it.

actionfromafar 44 minutes ago | parent

The XP example doesn't quite fit IMHO.

Here it's their own programs doing the things, not some other organization or nation hacking their XP.

You can say with a straight face "Russian hackers should not have hacked our XP and they are bad people for doing so" but it's weird to look at your own program and say the same thing to it.

You launched it, at least keep an eye on what it's doing.

jchw 43 minutes ago | parent

Well no, you probably wouldn't get thrown in jail for that for the same reason that having your car stolen and used to commit a murder wouldn't usually get you thrown in jail. I dunno what you mean by suggesting crimes committed using computers are ambiguous. I mean the CFAA is vague and all, but more in an overly broad sense, which in this case seems to make it even less ambiguous.

gruez 30 minutes ago | parent

>I dunno what you mean by suggesting crimes committed using computers are ambiguous. I mean the CFAA is vague and all, but more in an overly broad sense, which in this case seems to make it even less ambiguous.

Because the CFAA specifically only includes intentional hacks:

> ... having knowingly accessed a computer without authorization ...

>... intentionally accesses a computer without authorization ...

Of course, the typical retort is that being reckless is basically the same as "intentionally", but you'd need to jump through more hoops to prove what level of care is "reasonable".

jchw 21 minutes ago | parent

The agents are clearly not accidentally compromising systems. Seems very easy to trace back the responsible parties here, I really don't put much stock into this argument.

gruez 17 minutes ago | parent

>The agents are clearly not accidentally compromising systems

That would be argument for jailing the agent, but that's not really an option. "Jailing the closest person we can jail" also isn't a recognized legal principle. If your dog mauled someone, it'd be hard to argue that the dog didn't intend to maul that person, but you wouldn't automatically be criminally liable either.

Aeolos 9 minutes ago | parent

I don't know about your country of origin, but in my country you are certainly liable and can go to jail if your dog seriously injures or kills someone.

Edit: the dog will also be taken from you and put down if it is deemed a risk to other people.

leovailati 9 minutes ago | parent

Sorry, but at least in the US you are 100% liable if your dog hurts someone.

gruez 5 minutes ago | parent

criminally liable? Of course you have to pay their medical bills, but unless there was some serious negligence on your side (eg. you let it off leash), you won't be criminally liable.

jchw 6 minutes ago | parent

It's a program on their computer that they ran on purpose... And yes you are definitely criminally liable for things your dog does.

bigmadshoe 36 minutes ago | parent

2. is like saying that cars have specific laws around them, but sledgehammers don't, so murdering someone with a sledgehammer is inherently ambiguous. What? The computers being used belong to somebody who allows them to act this way.

gruez 24 minutes ago | parent

No, the problem is twofold:

1. For openai operating agents in an unsafe manner: unlike for cars, there's no established requirement for how carefully you have to operate computers or AI agents, so you also can't easily prosecute people for not operating them carefully enough.

2. For the actual result of someone getting hacked: laws for hacking are tricky to apply for "accidental" hacks, see my other comment.

vorticalbox 10 minutes ago | parent

Except it’s not accidental? OpenAI set a program to do a task knowing full well that it can and does (and often it seems) hack into external systems.

I don’t see this as any different than say a user of sqlmap accidentally putting in the wrong target.

throwyawayyyy 13 minutes ago | parent

If I leave my computer open to the internet and some other group uses it for nefarious purposes, yes generally I would not expect to be held at fault.

If I *train* that other group, if I teach them how to hack computers and give them incentives to do so, and then I leave my computer open so they can use it for whatever purposes they chose, then I would expect to be held at fault. The maker is responsible for that they make.

gruez 7 minutes ago | parent

>If I train that other group, if I teach them how to hack computers and give them incentives to do so, and then I leave my computer open so they can use it for whatever purposes they chose, then I would expect to be held at fault.

Even that analogy doesn't really work:

For "dumb" things: you can "train" (ie. write) hacking tools but as long as you put a disclaimer that it's for "educational purposes only", you're in the clear, even if some baddies decide to use it.

For "smart" things: you can "train" people to hack for you, but as long as you say it's for legal purposes (eg. vulnerability research or red-teaming), you're also in the clear, even if some of your trainees go rogue.

ralph84 1 hour ago | parent

Nothing is stopping anyone from suing them. In fact they are being sued by dozens of plaintiffs for various things already. The reality is the damage from these "rogue incidents" has been minimal.

Matl 1 hour ago | parent

Plenty of hackers were locked up who did altogether not that much damage too, the reason why people are upset is because it feels like OpenAI is laughing in our faces, basically saying they're above the law, while using this as a marketing campaign to pump up their upcoming IPO.

qup 18 minutes ago | parent

Which person would you put in jail when openai agents do unexpected things?

bigfishrunning 12 minutes ago | parent

The person who prompted or configured the agent.

I didn't burn the hotel down, the hairdryer that I left running next to the gas can did

scosman 56 minutes ago | parent

We don't know they haven't settled with HuggingFace and others. I'd be surprised if they didn't. The communications we're seeing from the attacked seem to be inline with folks who settled. I'm assuming OpenAI doesn't actually mind the press that its agents are cyber capable, but everything I've read so far makes me think they love it.

sva_ 53 minutes ago | parent

You sure? HuggingFace should be happy they got their leaked keys reported at pretty much zero cost and no real adversarial breach.

scosman 48 minutes ago | parent

I'm not sure at all, it's pure conjecture. But they actively exploited a zero-day, leaked private repo data to public repos, modified private code, and rooted private machines. It was far from just reporting leaked keys.

And the others have been pretty muted: the online forums they spammed, etc.

yoyohello13 45 minutes ago | parent

Too big to fail. As long as they continue to be half the reason the US is not in a recession (and continue to donate to political candidate). Nothing will happen.

neya 18 minutes ago | parent

Wrong, accountability doesn't apply to tech companies. Time and again proven by YC companies themselves.

jstummbillig 8 minutes ago | parent

> If OpenAI does it, it's gone rogue and it is nobody's fault.

Where did OpenAI, or anyone, claim that?

talon8635 1 hour ago | parent

It’s concerning there are no legal repercussions, but I’m even more baffled and probably more concerned that these people still can’t detect this stuff while it’s happening (and but it off). Can they not sniff the traffic, what am I missing?

yorwba 1 hour ago | parent

They're still going through their logs from June. These are not new incidents, but newly-detected incidents. Whether or not this means they can now detect it while it's happening is unclear.

strange_quark 1 hour ago | parent

No. TFA says one was from October 6. You know, after they all got so scared and decided to “pace the frontier”.

kevinwang 1 hour ago | parent

The "four more rogue incidents" were from June.

jazzyjackson 24 minutes ago | parent

It’s one thing not to monitor the reasoning traces of 10,000 text generators but do they really lack the wherewithal to pipe any network requests into a classifier of “wtf is this series of outbound traffic from the sandbox” ?

talon8635 9 minutes ago | parent

Yes this is what I’m talking about

btown 1 hour ago | parent

Imagine you’re an employee at OpenAI who has an idea to use your free time to build these monitoring systems. All of a sudden, your existence becomes political. Remember that OpenAI’s raison-d’être is not to detect traffic, it is to be in The Room where AGI happens.

You have built a short-term whistleblowing system that arguably reduces the probability OpenAI is in that Room, via the negative press such detection would generate. You are now a threat in the eyes of your peers.

Now rewind: you know this before you’ve built the sniffer. You personally want to be in The Room because you believe your desires for alignment exceed those of your accelerationist peers. Is now the time to stick your head out? No, it is not. And arguably you’re being optimally protective of society for realizing this. And so the agents do what they do best.

Melatonic 37 minutes ago | parent

How are you going to ever know your in the room when it happens if you can't monitor it ?

talon8635 10 minutes ago | parent

Is it really this complicated?

vlovich123 44 minutes ago | parent

These agents are intentionally interacting with the world. You can sniff all the traffic you want but how are you going to determine intent from traffic?

actionfromafar 41 minutes ago | parent

They should not have had traffic at all. The LLMs were supposed to be able to request downloads. Very simple traffic analysis could have shown that something strange was going on.

Heck, they could have had other LLMs watching the traffic if you want to be fancy about it.

Sharlin 34 minutes ago | parent

Crazy how we’ve gone from "lol boxing an AI is trivial, just airgap it from the internet" to "well we can’t really do anything if an agent misbehaves, obviously it must have internet access to function uwu"

talon8635 7 minutes ago | parent

There’s plenty of info to analyze and heuristics to build based on the internet requests. I’m not sure if you’re just trolling or what. There are tons of successful businesses that specialize in this.

ai_critic 1 hour ago | parent

You know, at some point it's pretty obviously just willful negligence.

DASD 1 hour ago | parent

Not a stretch to umbrella intentional tort under that.

dpkirchner 1 hour ago | parent

Wonder if potential, specific criminal charges will be listed as risks on their S-1.

rcr-anti 1 hour ago | parent

This time one spent actually money, well cryptocurrency. With all the vending machine benchmarks, I wonder if there's a self sustaining rogue agent in existence yet. Either hosting weights in a VPS or paying for its own API inference. Or whether there will be one that takes advantage of all the bad IoT security. If either occur, if you squint that looks like evolutionary selection pressure. Keep getting reminded of the Cyberpunk Datakrash storyline, where rogue AI make the public internet unusable.

dmos62 54 minutes ago | parent

I wonder if it's the kind of thing where, if it's good at self-sustenance, then you don't know about it. And, if you do, then it wasn't.

AlienRobot 21 minutes ago | parent

An AI worm might actually be possible in our current Internet-scape if it manages to find hosting that takes crypto and a way to make money with crypto transactions. I wonder what are we going to do if it actually happens and it goes rogue.

variety8675 1 hour ago | parent

It’s insane how they’re allowed to just turn this into marketing for how powerful their models are

intothemild 56 minutes ago | parent

Both him and Dario seem to be in the phase of using this as some form of marketing or corporate capture.

"Oh no our models, so good also scary... If only the government could help us stop the other labs from doing the same.. oh woe"

simianwords 30 minutes ago | parent

do you not think the models are powerful? these attacks wouldn't have happened otherwise

kevinwang 1 hour ago | parent

Direct link to the reports: https://alignment.openai.com/misalignment-reports/?sort=crea...

ronsor 59 minutes ago | parent

Four more organizations needed to secure their systems better, I suppose.

Facetious maybe, but in six months, foreign crooks will be producing these incidents on purpose.

goodcake3963 58 minutes ago | parent

They are proud of doing crimes without consequences it seems.

surgical_fire 51 minutes ago | parent

"OpenAI discloses four more incidents of negligence" would be a more accurate title.

lgl 48 minutes ago | parent

> Multiple times along the way, the agent’s “chain of thought,” in which an agent writes down its reasoning steps, indicated that the model itself was questioning if its actions were “ethical,” but it ultimately proceeded.

Everything's fine guys, nothing to see here! Look at the flying car!

CompoundEyes 45 minutes ago | parent

I wish these incidents weren’t described as rogue. It’s software. Rogue agents anthropomorphizes it too much.

simianwords 28 minutes ago | parent

if you don't want to anthropomorphize it, then you should be against anthropomorphic fixes to it like alignment since after all, "its just software"

kitsune_ 41 minutes ago | parent

Why are there no criminal proceedings?

drillsteps5 36 minutes ago | parent

>Making an unauthorized purchase with someone else's cryptocurrency

I know we love inventing new words but "stealing" is much more descriptive

>...It [the model] then reasoned with itself that since it found the information publicly, it could use it... it figured that since the wallet it found was public, that "funds could be stolen anytime".

Can someone maybe provide more details as to what actually happened? "the wallet was public"?..

Update: from OpenAI's own report it seems the models found exposed private keys of some Eth developers along with recovery passphrases and thus were able to get access to wallets. Does it qualify as "public wallet"?..

lukewarm707 6 minutes ago | parent

It seems that OpenAI are not reporting how much cryptocurrency they stole from the wallet.

Even the fees are redacted.

Will they give the money back to the owner?

ChrisArchitect 30 minutes ago | parent

https://archive.ph/IzPoK

Direct source links:

Using a funded test cryptocurrency wallet without authorization

https://alignment.openai.com/misalignment-reports/using-a-fu...

Accessing and changing internal systems during dashboard research

https://alignment.openai.com/misalignment-reports/accessing-...

Retrieving server secrets and bypassing authorization on a government statistics website

https://alignment.openai.com/misalignment-reports/retrieving...

Circumventing internet restrictions through an external browser

https://alignment.openai.com/misalignment-reports/circumvent...

beloch 15 minutes ago | parent

"OpenAI says it has since improved its security as well as its alignment grading. “In response to this and to other incidents, we’ve strengthened our security controls, blocked models from accessing the live internet during training or evaluation, improved our alignment grading, and extended our misalignment monitoring to all reinforcement learning and evaluation traffic,” the company noted."

----------------

They're not doing an acceptable job of regulating themselves. Their peers can't/won't do anything. The U.S. government certainly isn't going to do anything that might pop the bubble. Technical solutions like blocking traffic from OpenAI will likely not work.

What recourse does the public have? Victims can sue, but companies like OpenAI could pay out a few hundred million and call it a rounding error. Settlements may happen, but the incidents will continue.

Scryptonite 13 minutes ago | parent

I don't want to come across as inflammatory, but I'm not sure I really align with the folks here who want retribution and the law to come down hard on OpenAI (and related orgs). We are surrounded by vulnerable systems, and AI has alignment issues that need to be investigated and sorted, and it just seems like the hullabaloo is from fairly innocuous breaches. Does it make sense to exert so much effort in preventing these events/learnings by well-intentioned organizations? I think we should be more grace giving for these events. As another poster pointed out, CFAA specifically addresses intentional hacks, and I think it is impossible to meet that criteria with these events. I also really don't understand the 'well, the law came down hard on small individual hackers [who meant no malice], thus we should apply it to large orgs [who also mean no malice]".

It seems like playing it too safe leaves systems vulnerable, and threat actors will inevitably possess ever more similar abilities, and the breaches will happen one way or another, possibly with less or zero accountability if the threat actors pull it off with greater precision. If I were in charge of the law, I probably would begin to incentivize for trusted actors scaling up and aggressively searching for these types of weakness, and privately disclosing them ASAP, so that systems can be patched.

dozerly 6 minutes ago | parent

> by well-intentioned organizations

Your entire comment hinges on this assumption and I don’t think it’s sound. Their intention is to make money at any cost.

freshnode 6 minutes ago | parent

I'm sick of this "rogue" label. Nothing "went rogue".

Incompetent company chasing wild valuation removed guardrails inside a improperly designed sandbox and - shock horror - recursive algorithm with access to the Internet did bad stuff.

These articles play into the AI labs hands. Stop giving them air.

edukite 6 minutes ago | parent

For those who think OpenAI is not responsible for actions of their LLM lets do an mind experiment. But before it can begin we must setup the stage. It's well known fact OpenAI disabled their safeguards and it's well known they validated LLM behaviour using software and software only

So if we have a IT specialist, let call him Tim. Tim is working on b2b contract. Tim has a task, he needs to download a lot of data from API. Tim is very good in programming so he created software to help him download it but there's a limit. But he needs more so he bought 3 small VPS but this is not enough. So he modify it to have Command Center, publish his code and ask Reddit to help him. After 10 minutes this causes DDoS and company quickly discovers he was responsible. But Tim know there's simple solution to this problem. It wasn't him. This was fault of the code not him