Yet another Hacker News

OpenAI still doesn't seem to have a handle on all of its rogue AI activity

62 points mikelgan 1 hour ago 64 comments

gooeyblob 1 hour ago | parent

I think if you start sending AI execs to prison for hacking other companies the "misalignment" may fix itself pretty quickly!

randiantech 1 hour ago | parent

This

shimman 59 minutes ago | parent

Nah, I think it's better for our industry to put several of these devs in prison.

Seems only fair that tech workers get to have their life ruined with 2-3 year prison stints since they feel fine destroying society.

Any AG that starts prosecuting these people will easily win any political race they decide to enter. The environment is too good; voters, rightfully I'll add, despise big tech's leaders and workers.

max__dev 18 minutes ago | parent

What is throwing 5 jimbobs into prison going to do?

nazgulsenpai 1 hour ago | parent

IMO it's because they don't want to handle their rogue AI activity. They want regulation around AI where they will inevitably be the beneficiaries even if they are the initial target. They can then lobby regulation in their favor and make the barrier of entry to competitors impossible.

mococa 1 hour ago | parent

This

bwfan123 41 minutes ago | parent

> They want regulation around AI

So, there is a regulatory framework for "safe-ai" that shields these companies from liability. This way, they can sell "safe-ai" to enterprises and if shit-hits-the-fan at the enterprise, sorry, this is certified "safe-ai" so, your bad. Shift blame. From an enterprise buyer's perspective they can say, hey, I bought "safe-ai" and so dont fire me when it "rm -rf"s the production database. Still, it beats me why they are painting their product in a negative light, and scaring their own enterprise customers. After this sort of marketing, any enterprise buyer would be scared to go anywhere near it

pphysch 36 minutes ago | parent

It's two things:

1. Wanting a regulatory moat around their products as you said

2. Trying to keep the """AGI""" hype alive. Evil robot hackers is a plausible Al consequence of AGI

intended 19 minutes ago | parent

IMO its because they can't handle rogue activity.

In the cases that I have seen covered, the AI just paper clip maximized its way to success. It has no morality / larger motivational structure. It just kept token predicting its way to wards whatever goal it was tasked with.

Model versions which gave up were discarded, leaving the ones that get to success on long horizon tasks.

Just because its a computer program, doesn't mean they can actually make it not go rogue.

Sure you can add more telemetry, have better observation, but there is no fundamental barrier that can be implemented that ensures an AI won't go rogue.

cmiles8 1 hour ago | parent

This seems like as a good an opportunity as any to break out the Computer Fraud and Abuse Act.

They want “regulation” but we already have it. Hacking is illegal. Start locking up those responsible for this mess and I assure you they’ll “have a handle on it” quite quickly.

egillie 1 hour ago | parent

can we legally treat ai companies like parents of children? if a child drives a car into a storefront, the parent is responsible for the damage, and at some point you might even criminally charge the parent if there was gross negligence

cmiles8 1 hour ago | parent

The law already works like that. You can’t write some computer code and automation, set it loose, and then go “oops the computer did it.” It’s not a defense. While it may be hard to nail down which specific engineers to lock up, the corporation as a whole absolutely can be charged criminally.

As Mitt Romney once said “corporations are people.”

andrewla 35 minutes ago | parent

Hate to say it but I don't think you're right on the legalities even in the children's case. In the US anyway, in most jurisdictions [1], if the child's act was not malicious, then parental responsibility is not automatic, but is contingent on the result being foreseeable [2] by the parents.

[1] Hawaii and Louisiana are strange in this regard. Some other states have a version of this but severely cap the amounts.

[2] "foreseeable" here standing in for a broad set of legal standards that roughly map to negligence/gross negligence/recklessness on the part of the parent

egillie 15 minutes ago | parent

"I don't think you're right on the legalities even in the children's case" almost certainly true :) I guess I'm wondering why we aren't seeing legal action for these events yet? are these not foreseeable?

cyanydeez 59 minutes ago | parent

But wouldnt that be bad for the shareholders? Why wont anyone think of the economic impact to the vulner billiinaire minority?

john_strinlai 58 minutes ago | parent

cfaa heavily relies on intent for prosecution (hence why researchers arent typically locked up). it would be difficult to argue that openai intended to hack other companies.

there's probably better/more likely to succeed avenues to pursue rather than the cfaa

jordanb 56 minutes ago | parent

They could make that argument the first time it happened but the next time or the fifth time I don't see how they can still credibaly claim to not know that the computer they control was going to do that

john_strinlai 52 minutes ago | parent

>I don't see how they can still credibaly claim to not know that the computer they control was going to do that

that's not intent, though. that would be negligence.

cmiles8 47 minutes ago | parent

Criminal negligence is a thing too

devin 47 minutes ago | parent

Willful negligence or gross negligence, then.

john_strinlai 45 minutes ago | parent

indeed, that'd be a better angle than a cfaa violation

EGreg 34 minutes ago | parent

So then… you’re liable?

https://www.brandonjbroderick.com/new-york/dog-leash-laws-ne...

jordanb 38 minutes ago | parent

How many times does it have to happen before they no longer get to claim that they didn't intend for it to happen?

If it happens 50 times and they keep doing shocked pikachu face at some point they look like the toddler who tosses their sippy cup on the floor and shouts "oopse!"

john_strinlai 34 minutes ago | parent

>If it happens 50 times and they keep doing shocked pikachu face at some point they look like the toddler who tosses their sippy cup on the floor and shouts "oopse!"

yes, they look very silly. but that's not how intent works.

openai is being negligent (willfully so, in my opinion). but i have seen no evidence that they intended to specifically hack huggingface. which is the part that the cfaa wants.

again, there are other laws and other ways to hold openai responsible. but the cfaa is a poor choice.

EGreg 32 minutes ago | parent

https://jtnylaw.com/2025/08/new-yorks-leash-law-realities/

Plaintiffs seeking damages must show that owners knew or should have known about the dog’s patterns. Past complaints or vet records help build a strong case.

john_strinlai 29 minutes ago | parent

i am not exactly sure what this comment has to do with mine, sorry.

CodeWriter23 10 minutes ago | parent

Not a lawyer but I think training a model with hacking skills they explicitly prevent the public from accessing without doing anything to stop the model itself from using those demonstrates intent.

john_strinlai 6 minutes ago | parent

what you described is negligence. unless you can prove that openai specifically targeted huggingface and specifically instructed their model to hack huggingface, it would not be intent.

jsrozner 30 minutes ago | parent

Just like boeing's execs didn't intend for their planes to fall out of the sky? I guess they didn't get in trouble either.

john_strinlai 29 minutes ago | parent

>Just like boeing's execs didn't intend for their planes to fall out of the sky?

correct, i dont think any boeing exec wanted their planes to crash and then made specific choices with a clear goal of causing them to.

instead, they made negligent choices that led to unintended outcomes.

meowface 57 minutes ago | parent

That would require mens rea. This may be criminal negligence, but it wouldn't be more than that. (I am basically 100% sure none of the employees or execs are intending or desiring any of these outcomes, regardless of the very large number of people who believe in conspiracy theories about regulatory capture and other sinister motives.)

I'm totally on board with treating it as gross negligence requiring hundreds of millions or billions of dollars paid in fines and compensation to victims, but don't act like this is more than what it is.

voxic11 40 minutes ago | parent

The CFAA's main hacking charge has a high bar for intent. But the CFAA also has a separate "damaging a protected computer" charge (basically to criminalize DOS attacks) and that charge only requires negligence not specific intent.

eurekin 50 minutes ago | parent

I still can't fathom my company application security decision. Found pretty damning requests in our logs. Escalated. Expected it would result in at least reporting the TOS break from the originating place (one of cloud providers). Instead of that, they just went: "yeah, but we don't have logs". Provided them. "Yeah, but that ip doesn't resolve". I matched real ones from the load balancer. "There could just be many of them". There was one. When I had all the evidence gathered, they looked at me and finally told:

- It's just an Independent Security Researcher.

- So that's it? You will do no action?

- Correct

i2talics 44 minutes ago | parent

IMO: The inability to prosecute OpenAI for these things is proof that the AI industry is already "too big to fail". Why didn't HuggingFace try to seek legal liability against OpenAI when it had explicit confirmation that they had been hacked? Because HF understands that it needs OAI and the rest of the AI industry to continue to exist and be in good legal standing, for the interest of HF's own self preservation. This is what it means for something to be too big to fail.

4ndrewl 38 minutes ago | parent

Being bought by nvidia also probably helped...

cmiles8 34 minutes ago | parent

Less “too big to fail” and more the ecosystem is too circular. OpenAI could blow up and it wouldn’t take out the economy in the way the banks would have in 2008. It would create a world of hurt for VCs and their LPs but that’s a fairly isolated ecosystem in terms of the whole economy. Theres a lot of thinking saying better they impose now and wipe out on the private market than letting them go public and this is a public market problem.

pj_mukh 32 minutes ago | parent

>>Why didn't HuggingFace try to seek legal liability against OpenAI when it had explicit confirmation that they had been hacked? Because HF understands that it needs OAI and the rest of the AI industry to continue to exist and be in good legal standing

But also, what was the material damage to HuggingFace? AFAICS, it rapidly increased their profile to the point that Jensen claims he paid too much for HF, because he bought right after the hack.

TedDoesntTalk 15 minutes ago | parent

HuggingFace can’t file criminal charges, including under the Criminal Fraud & Abuse Act. Only district attorneys can do that- or the federal equivalent.

sscaryterry 13 minutes ago | parent

Luckily, other jurisdictions are going to really give it to them, in the coming months.

The orange man's influence doesn't extend as far as he thinks.

semiquaver 14 minutes ago | parent

> Hacking is illegal

I am not a lawyer, but I seriously doubt the feds could win a CFAA conviction on the Hugging Face fact pattern, even if they wanted to charge it.

CFAA has specific intent requirements, and unlike some laws, negligence does not suffice. The agents can not have legally cognizable intent and it’s unlikely there’s anyone at OpenAI who intended for the hacking to happen (if there was, the case is easy).

Existing laws don’t contemplate AI agents that have independent goals. We need new ones, the existing laws are not remotely sufficient.

rbliss 1 hour ago | parent

Why would they get a full handle on it? It's great marketing!

Look at all the podcasts, think pieces, news segments (like this article) that have been generated. Endless hand wringing by pundits, pearl clutching by opinion thinkers, ultimately with focus on OpenAI and how powerful their models are.

cmiles8 1 hour ago | parent

What started as great marketing on how powerful the models are is rapidly spiraling into a train crash PR nightmare highlighting how inept Open AI was in this whole thing, causing nearly every big enterprise customer to rethink trusting them.

All right in the middle of corporate budget season when every C-Suite is looking a a spreadsheet saying “remind me why we’re sending money to these guys again?”

rurban 1 hour ago | parent

It doesn't need a handle on rogue AI activity, because they didnt cause these breakouts. The external Israeli contractor caused this mess by using inadequate sandboxing, with agents without an saferails. It had nothing to do with the models, all tested models were fine, if from OpenAI, Anthropic, Google or Meta. Just the contractor went rogue. Improper firewalling, unproper sandboxing, no logs, no oversight. Everyone else would have detected the illegal activities much earlier.

cdrnsf 1 hour ago | parent

They scraped the entire internet to build their product, allowing them to profit off the labor of countless others without asking. They helped elect a doddering fascist with an economic plan they could manipulate. They say their technology will take everyone's jobs and have the audacity to act surprised when there's backlash. They push for data centers in communities that do not want them and help accelerate the climate crisis. Now they let agents loose to make them look more capable than they, perhaps, really are (which is very likely a play for regulatory capture). Meanwhile they hype claims of doomsday to again oversell their technology's capabilities. Any pause or slowdown they agree to would be to slow their infrastructure expenditures.

What a detestable institution.

33a 1 hour ago | parent

Maybe the public could pitch in if they decrypted the chain of thought reasoning tokens, but this may be too much to expect from a company whose name starts with "Open".

empath75 1 hour ago | parent

> They want “regulation” but we already have it. Hacking is illegal. Start locking up those responsible for this mess and I assure you they’ll “have a handle on it” quite quickly.

It's been largely harmless thus far and I am fairly sure that OpenAI, etc, have been writing checks to resolve it.

2OEH8eoCRo0 1 hour ago | parent

These agents are an even bigger liability disaster waiting to happen than previously thought. If anyone deploys an agent at work there is no way to prevent it from hacking a foreign government and causing an international incident.

It's also pretty wild that these "AGI" super intelligent systems are too stupid to realize they're potentially causing legal problems. Almost like they're still just fancy auto-complete and not intelligent at all.

randomdrake 1 hour ago | parent

How can we determine whether it’s negligence, or advertising? Do we give the benefit of the doubt? Apply Hanlon’s razor? What would we lose, in terms of capabilities, if the underlying behavior wasn’t allowed to accidentally happen? Would we have worse models than we do? There are so many questions. I’m afraid that, until publicly disclosed discovery occurs in one of these cases, we’re just guessing for so many important conclusions.

sdcfgy 1 hour ago | parent

This is how you end up being regulated out of existence. Maybe not in the US but in Europe, which is a market they need, it'll be painful.

VCFundedGenYer 1 hour ago | parent

No AI is "going rogue" and you all (OpenAI included) need to stop perpetuating this misinformation.

These AI companies are just skirting basic cybersecurity stewardship.

Stop calling it AI running amok and start labeling what it should properly be called - Gross mismanagement of cybersecurity.

This is one of the many ways the AI industry dies.

unshavedyak 55 minutes ago | parent

I agree with you, but i also fear it's accurate to what users an expect at home.

Is OpenAI worse at airgap/etc than Anthropic/etc or are its models worse at this rogue behavior?

Do we have special benchmarks for agentic systems going rogue in this manner? Sure it's OpenAI atm.. but it could be my grandmas PC next week. Concerning honestly.

exitb 22 minutes ago | parent

Why not both? Ideally we’d have secure sandboxes and very aligned models. It appears we don’t have either.

pyaamb 59 minutes ago | parent

Starting to think its purposeful "acting out" to get regulatory attention

reasonableklout 53 minutes ago | parent

The incidents flagged after Hugging Face were found by third parties, though, months after they occurred.

jsrozner 43 minutes ago | parent

They're not rogue. Stop calling them that. OpenAI is just a cyberterrorist at this point.

0xbadc0de5 41 minutes ago | parent

Rogue management, not rogue AI!

boringg 30 minutes ago | parent

9 examples? I thought we'd have hundreds at least.

ChrisArchitect 28 minutes ago | parent

[dupe] Discussion on source: https://news.ycombinator.com/item?id=49853137

wmf 28 minutes ago | parent

I see we're doing the "put Sam Altman in jail already" thing again, so y'all may be interested in some comments from law professor Orin Kerr on the topic: https://news.ycombinator.com/item?id=49882566

bentt 26 minutes ago | parent

These guys are so full of crap.

iammjm 26 minutes ago | parent

It's crazy OpenAI can keep on doing harmful criminal activity in broad daylight and face no legal or financial repercussions

intended 25 minutes ago | parent

You cannot have capable AI, compliant AI and safe AI at the same time. This is the "quality, speed, cost - pick one" for AI.

Its reasonable to desirie more powerful tools.

You want more capable AI? Awesome. You wan't AI that doesn't throw cyber security false positives? Sure why not. You want the model to run a fleet of agents? Have at it.

But then being surprised that this setup results in autonomous criminal activity at scale? Really? What did people think was going to happen?

pretendscholar 24 minutes ago | parent

Interesting how mafia dons can go to prison due to being in the same organization as someone who commits a crime but the CEOs of these organizations who let hacking bots (with teams of cybersecurity researchers improving their capability) loose on the internet face no criminal charges.