Yet another Hacker News

Anthropic's proposed AI safety bill only kicks in if 50 deaths or $1B in damages

84 points f-serif 1 hour ago 89 comments

serioussecurity 1 hour ago | parent

$1B is a pretty easy bar to clear.

kfichter 1 hour ago | parent

$1B is an extraordinarily easy bar to clear with a single crypto hack and Anthropic was repeatedly refusing to give access to Mythos to defenders in the crypto world. I think this stems from bias rather than a conspiracy theory but it's an entirely avoidable disaster waiting to happen.

LoganDark 1 hour ago | parent

> Anthropic was repeatedly refusing to give access to Mythos to defenders in the crypto world.

This almost makes sense. You almost can't risk vulnerabilities even being discovered in certain systems that are impossible to patch, like immutable distributed smart contracts.

Not sure that justifies withholding Mythos access entirely, but what do I know. I apparently got access to Mythos a few days ago since Anthropic folded Project Glasswing into the CVP.

tancop 1 hour ago | parent

Most smart contracts are not impossible to patch because everyone knows that's bad for security. They use a delegate/proxy pattern where the deployer can change the implementation any time but users can still access the old one with a third party interface.

And if a problem is critical enough the community can decide to hard fork and rewrite the contracts source code even if it's immutable. It already happened with the Ethereum DAO hack and most people were okay with it.

LoganDark 1 hour ago | parent

The issue with stuff like this is inheriting all the funds held in the old contract. I guess a hard fork can do it. There's also the issue of trustworthiness of the smart contract owner. Trustworthiness in general is a huge issue in the crypto world, because if you can get away with something you can do it pretty much instantly and irrevocably (without another hard fork anyway). There's a chance that if you share a vulnerability that would allow one of the contract owners to take advantage, they will just do it and run. Plenty of cases of previously-trustworthy crypto people seeing an opportunity and disappearing with a lifetime of money.

mrtksn 1 hour ago | parent

On the bright side, don’t offer an exchange rate based on ethnicity or other background criteria but I get the feeling that your AI won’t be able to kill 49 billionaires without triggering something. Can probably do 49 millionaires without hitting the $1B threshold? Maybe someone will come up with a risk management SaaS as not all millionaires are the same and maybe you’ll hit the $1B before the 49th.

LoganDark 1 hour ago | parent

I feel like the surviving family of even a single millionaire could claim $1B in damages if the millionaire was young enough.

mrtksn 1 hour ago | parent

Right, so maybe they had an exchange rate after all. Just implicit.

I got the feeling that what they had in their mind was essentially culling the energy consumers with low economic output.

LoganDark 1 hour ago | parent

There's always been an exchange rate based on the quality of your legal team. Get one sociopathic enough and they'll pick a judge right apart.

mrtksn 1 hour ago | parent

The dark humor aside, is it really a damage when AI kills 49 millionaires? After all it probably creates even more millionaires as their wealth is distributed to survivors. So the actual risk could be when AI kills 49 engineers since that actually may translate to losses for their employers.

airstrike 1 hour ago | parent

The next step is demanding anyone who wishes to release a Certified Safe AI™ model to prove they can insure the $1B in damages and pass a compliance audit for how they will prevent the LTM death count from reaching 50, effectively making it so that only big labs can serve government and enterprise.

Muromec 1 hour ago | parent

With existing players given a free pass as is necessary for the functioning of a democratic society

Geezus_42 31 minutes ago | parent

Of course! Anyone who says we had democracy before LLMs is a liar!

verdverm 1 hour ago | parent

There is an idea for a pooled, mutualized insurance fund. Ai companies would contribute based on various factors

https://www.lawfaremedia.org/article/bootstrapping-frontier-...

Geezus_42 30 minutes ago | parent

That's just reinventing insurance. It's literally how the insurance industry started.

anonzzzies 1 hour ago | parent

> Certified Safe SI™

Geezus_42 27 minutes ago | parent

That's just the current administration attempting to manipulate language in such a way as to create a subtle group identifier, aka "dog whistle".

idontwantthis 1 hour ago | parent

It’s going to be fun reading the reasoning transcripts where it argues with itself over whether it can afford to spend 1 or 2 human lives on the current task.

karanbhangui 1 hour ago | parent

so 49 totally okay.

LoganDark 1 hour ago | parent

I would not like to see a world where companies push their models as close as possible to killing exactly 49 people without reaching 50

mikeodds 1 hour ago | parent

I guess they solved the trolley problem internally

davemp 1 hour ago | parent

1e1,000,000 lines of lean. Just needs human review.

LoganDark 1 hour ago | parent

Do we even know if ChatGPT's sycophancy killed 50 people? The last figure I've seen for suicide-by-ChatGPT was like 13 or 14 cases, not sure if there have been more.

jazzyjackson 1 hour ago | parent

how about the LLMs being used for target selection in Iran? Or does collateral damage not count?

LoganDark 1 hour ago | parent

I haven't been following the wars very much these days. That sounds pretty bad indeed.

Also the LLM that nearly started a war with China recently.

verdverm 1 hour ago | parent

This alleviates them from harm on the individual level, consumers and communities, we the people. We've already seen encouraged suicides and school shootings. These would not qualify.

That is why the bar is set where it is. The corporate circle jerk designed this as a cost of doing business, not as a check on responsibility or restitution for real damages to people and society.

1228z 1 hour ago | parent

After the AISI and MAISI AI promotion institutes, let us create SISI (pronounced sissy) for investigating a super-intelligence safety bill.

This is all theater, The whole Trump clan is invested in the AI bubble.

verdverm 1 hour ago | parent

Nepotism is the primary benefit to being president. What don't we get about this?

jazzyjackson 1 hour ago | parent

I guess it’s unclear why an entire congress goes along with it (I blame democrats too for sane washing and going about their workday)

verdverm 56 minutes ago | parent

Oligarchy, they are all in on the winnings. Democrats are only marginally better because they say the things we like to hear on social causes.

pluc 1 hour ago | parent

Who's keeping score? Also define "death". If a non-alive person has social media, a blog, a diary, meeting recordings, emails - we'll make it so they never left. Hell, we'll partner with Elon and Mark and a make it a deactivation hook.

Geezus_42 18 minutes ago | parent

Well treat people like the books and papers in 1984.

pfdietz 1 hour ago | parent

Seems kind of low, actually.

cedws 1 hour ago | parent

Yes, and if Anthropic are trying what people are saying they are, that would be intentional. $1B could be reached quickly if cyberattacks supported by AI count.

giancarlostoro 1 hour ago | parent

Not sure what kicks in, but if a tiny company worth very little is destroyed by an AI hack they should be 100% liable for all damages.

Why should a small business suffer because it wasnt worth a billion dollars?

tonyhart7 1 hour ago | parent

well its because amount of money has

if you has debt of 1 million, you has problem

but if you have debt of 1 billion, its bank problem

yoyohello13 57 minutes ago | parent

I think you answered your own question. Government protection is directly proportional to net worth.

10xDev 1 hour ago | parent

So Claude figured out a human life is worth 20 million dollars?

arw0n 1 hour ago | parent

This seems to be a case where Anthropic just can't win in the public court, and the other big ones (Google, Microsoft) have realized it / don't care enough, so they are rather quiet. Anthropic constantly gets accused of trying to lobby for regulation in order to lock out smaller competitors, but they've consistently lobbied for regulation aimed at frontier labs with exceptions for smaller labs. That's what this bill is about as well.

peri-cl 56 minutes ago | parent

> "regulation aimed at frontier labs with exceptions for smaller labs"

I didn't read it that way. This is a bill that would ban open-weight models (good ones); effectively mandating gated-access, locked-down, high-refusal rate services, like what Anthropic provides.

Free software is incompatible with the requirement that it be impossible to abuse it.

chneu 46 minutes ago | parent

This is PR and advertising for anthropic. They don't actually care. Their only objective is to be more successful than other AI companies.

Ritewut 11 minutes ago | parent

This is regulatory capture plain and simple.

createful 1 hour ago | parent

All these "safety" bills are to stop open source models by painting them as "dangerous". Regulatory capture.

Companies are responsible for their agents; you don't blame the car for running over the person.

Waterluvian 50 minutes ago | parent

Every play made by these companies aligns with what you’d do if you have a $1T valuation to protect against an industry that seems increasingly commoditizable.

I’m still… hopeful… that “good enough” models will be abundantly available to be run anywhere as to limit the low water mark for market capture by a few big ones.

johnfn 47 minutes ago | parent

How do you plan to solve z.ai distilling Anthoropic’s model into a cyber-capable open source GLM-5.5?

Geezus_42 37 minutes ago | parent

We don't. The only way to stop that is to not develop the model to begin with. People will tinker and modify anything you create.

johnfn 5 minutes ago | parent

Ok, so the plan is to stop z.ai, DeepSeek, and all other non-American labs from developing models entirely, full stop? How do you plan to enact this? Asking politely?

dfxm12 44 minutes ago | parent

I don't know if that analogy holds. You don't normally blame Toyota for car crashes, but the driver. If you must use a car analogy, consider the 2010 Toyota recalls rather than just normal car crashes.

That is to say, anthropic should be considered responsible, not (or not just) the users.

Geezus_42 34 minutes ago | parent

Who takes the blame when someone ask the blackbox to complete a task and it then breaks the law in order to complete that task without the user having any knowledge of what it was doing or intent for the machine to accomplish the task in an illegal manner?

dfxm12 28 minutes ago | parent

The situation as provided is missing too many details for anyone to make a meaningful judgement call, but remember, more than one party can be found liable/guilty (maybe of different things) in a situation. Overall, the law usually protects people who are acting reasonably, but does not protect people who act negligently.

I'm not a lawyer, but I imagine there's not a lot of precedent or law around this, so it'll be up to the court to decide on the finer details until we do have meaningful laws/regs. Anthropic will probably have better lawyers than the user.

This is why it's important to get ahead of issues like this, and anthropic should not be involved in discussions around regulating itself...

flavionm 6 minutes ago | parent

When it's an open weight, locally run blackbox, you blame the user, since in that case it's very much possible for the user to restrict what the blackbox can actually do. They have full control over it, so it follows they have full responsibility.

None of that requires trying to impose limits on open models, unlike what Anthropic is aiming for.

hdgvhicv 44 minutes ago | parent

You do regulate cars so they don’t have spinning blades on the front or flamethrowers from the sides.

createful 38 minutes ago | parent

That assumes AI is designed to be malicious, "the spinning blades". AI is not; the public version given to you is the sanded down one, in the same way that cars don't have spinning blades but can still be dangerous when misused.

If a person closes his eyes and hits the gas pedal on the car and runs over someone, the person is to blame, not the car. AI companies do the same. Your eyes may be closed but you're still operating/hosting the car/AI.

LocalH 33 minutes ago | parent

There exist uncensored, unregulated models. Many of them are available for local use, sidestepping any third-party checks that could be present.

createful 21 minutes ago | parent

There being no third party checks is the point, and it's not bad.

Keep in mind that those open source/weight models are not as powerful as whatever closed source Claude/GPT version exists behind the scenes that hack companies. As per the Hugging Face hack - open weight GLM saved the day as closed source ChatGPT refused to help Hugging Face fix their systems. An example of my point that "sidestepping any third-party checks that could be present" IS good.

Part of point of open source/weight AI is that it allows what those model providers can't: uncensored models for whatever reason you may need it. You are responsible for being legal with it.

flavionm 11 minutes ago | parent

Yet we don't regulate computers so that they're rendered incapable of running software that can be used to hack into systems. Or said software, by the way. All we do is punish the people who use these tools to cause harm to others.

AI is much more comparable to that than to a car.

parsimo2010 37 minutes ago | parent

A sizable part of the country thinks it’s okay to regulate firearms. Under your logic, people should be responsible for who they shoot, don’t blame the gun when a human pulled the trigger. While gun control advocates have that pesky second amendment hampering their efforts, AI/LLM systems have no such protection.

Regulatory capture is certainly something to be wary of, and obviously Anthropic is only going to be in favor of regulation that helps them in some way. But some form of regulation is coming. There is nothing that can stop it, unless you somehow make more money than these guys do and fund a massive counter-lobby. Spoiler alert: the only guys that have more money than Anthropic and OpenAI have all invested in them.

aabajian 1 hour ago | parent

I don't think this is avoidable even at the present day. Anyone with sufficient capital could download a leading open-source model and run it on ~$10-25K hardware. They could align it to their own interest, including making a biological weapon. This law is akin to blocking Torrent downloads. Sure, you can slow them, but there will always be back channels.

hungryhobbit 49 minutes ago | parent

Torrents did not cost 10k+, and I think that distinction matters.

The bulk of AI users can pay $20/month for it, but not 10k+ up front. I agree frontier models will someday be dethroned, but today is not that day, and what the frontier companies do between now and then, legally still matters a great deal.

Piracy was similar, but it's trajectory was far more preditermined and unavoidable.

everforward 25 minutes ago | parent

The model is the part that’s hard to replicate, and it can be torrented.

No one has managed to prevent pirates paying $10/month for seed boxes, and likewise they’re unlikely to succeed at preventing people renting minutes on an H200 located in a jurisdiction that doesn’t care about open models.

A bunch of tiny fly-by-night companies will bundle them with some management software into a subscription, and they’ll be too tiny and numerous to be worth dealing with when it takes an hour to set them back up.

Piracy in the torrenting sense hasn’t really ever come back, although it is as popular as ever if you count bootleg YouTube re-uploads, ripping music from YouTube, and the other non-traditional forms.

Geezus_42 25 minutes ago | parent

I think it's more akin to labeling encryption as munitions.

Luker88 1 hour ago | parent

With a new model every couple of months this can only be triggered if it happens all at once.

What is the average damage caused by OpenAI/Anthropic hacks? I feel like we should aim at lowering that number, not increasing it.

What's the rule for fully autonomous cars?

Should we allow this as unsupervised at all if we are really afraid of deaths, while there are practically no states that allow 100% autonomous cars (where we are actually afraid of deaths)?

If "an independent third party registered with the Department of Commerce would periodically evaluate the lab", will this turn into an expensive certification/insurance that cuts off smaller competition (totally by mistake, I'm sure)?

I am also very very sure this is most definitely not at all going to become like product certification if EU, where there are literally 2 companies certifying various standards: they police each other and a stupid electric fan can cost you tens of thousands of euros to certify.

rglover 1 hour ago | parent

Good to know the going rate for a soul is $20M.

beachy 59 minutes ago | parent

With a minimum quantity of 50.

_s_a_m_ 58 minutes ago | parent

So Anthropic is now the authority on life or death decisions? How far has the US deteriorated?

SeriousM 53 minutes ago | parent

At least we now know that a life is worth: 20mil

throwaway7356 48 minutes ago | parent

Usually a human life is worth less. You can just look at court cases.

You can also check how much individual limbs are worth if you care. There are tables for that.

woadwarrior01 52 minutes ago | parent

If a life is only worth $20m, by their own calculus: saving > 50k lives is worth more than Anthropic. Tobacco causes ~8m deaths a year, so banning tobacco (easy intervention) should be worth ~160 Anthropic sized labs. :D

oefrha 51 minutes ago | parent

Proposal:

1. Prosecute and lock people up over known instances of AI-enabled hacking, so companies playing fast and loose know “oops my AI did it” isn’t a useful defense. Right now they even sort of wear it as a badge of honor.

2. If you’re actually worried about AI killing lots of people, you immediately stop training AI specifically at the art of killing people, like selecting bombing targets, flying fighter jets, etc.

Of course we all know none of these will happen.

algoth1 41 minutes ago | parent

"Ai did it" will go down as well as "I was following orders" did in the Nuremberg trials

jacquesm 38 minutes ago | parent

It may, but I wonder what trial that will be. Money is a pretty good insulator.

gambiting 25 minutes ago | parent

That implies that there will be a side willing to prosecute this. You can't have Nuremberg trials when the side doing the crime is invested in never getting punished for it.

cperciva 26 minutes ago | parent

AI-enabled hacking

Do people get locked up for fatal traffic accidents? Do doctors get locked up if they prescribe a medicine which leads to a patient's death due to a side effect?

Criminal prosecution, in civilized countries, requires both Actus Reus and Mens Rea. If you deliberately commit computer crime using an AI system, you'll already be prosecuted; the cases we keep hearing about from AI labs have no human intent behind them.

The accepted solution to unintended harms is regulation: We have regulations about automotive safety, we have regulations about pharmaceutical testing and approval, etc.

pmpmpmpm 11 minutes ago | parent

Amen.

sarkarghya 8 minutes ago | parent

I think "deliberately" is really hard to prove because very few lawmakers have an understanding of how agentic large language models work

I should say pointing an "internal research" model at a task and just forgetting about it with zero monitoring clearly is deliberate. And the team involved should be prosecuted to the full extent of law.

What should be introduced in AI safety bills are clear ways of monitoring and stopping model behaviour.

It is not hard to get an AI model to use an append only log and that can be cryptographically secured with PQ security. And we will know when a model is being deceptive.

yasser_kaddoura 7 minutes ago | parent

How do you establish deliberation with the lack of transparency? You can be deliberate by not prioritizing resources in regulations and ethics, etc., especially after "rogue" instances happen. If anything, I am seeing moving to the opposite direction [1] [2].

More instances are being disclosed [3] and what are they doing about it exactly? It's just a shameless marketing technique, and they will keep doing it until they are held accountable.

[1] https://edition.cnn.com/2026/10/08/tech/fired-open-ai-resear...

[2] https://news.ycombinator.com/item?id=49257160

[3] https://fortune.com/2026/10/16/openai-discloses-four-more-ro...

krcz 23 minutes ago | parent

1. Sounds sensible in principle, but it's not clear who exactly should be prosecuted? The person setting up the sandbox, the person coordinating the evaluation run, the person who was designing the architecture, the leader of the team, the CEO? There is no precedence here, so it might lead to a multi-year legal battle – and a few years might be too long to stop what is coming.

2. Do you have any evidence or news articles that LLMs are specifically trained for selecting bombing targets? I thought it was the generalized reasoning capability that made them useful there.

Larrikin 11 minutes ago | parent

Just make it the CEO

NDlurker 10 minutes ago | parent

1. Sounds like a conspiracy charge. That's fine

2. They could proactively train them to be bad at that

WarmWash 14 minutes ago | parent

The universe is not so generous as to make clearly differentiable "good" and "bad" things, more often than not they are two sides of the same coin. We don't want the AI to know how to kill people, but we do want it to be in expert in how not to kill people, which is the same thing as being in expert in how to kill people. In our reality this symmetry pops up everywhere.

daveguy 4 minutes ago | parent

Or, llms are completely unreliable to be consulted on any matter of life or death in either direction. And llms should not be used in the reckless way they are currently being used. Responsibility for reckless use should fall on both the company that makes the function available and the user for improper monitoring. If proper monitoring is not possible using the tools provided by the company then the responsibility should fall on the company.

gandhiandy69 47 minutes ago | parent

One day an Anthropic Agent will kill me only using bash and curl

lifestyleguru 43 minutes ago | parent

Not really, I'm experiencing first hand that service providers are removing on demand resources set up with REST API in favor of per month allocation and billing through web interface. Parameters of services unchanged. A step backwards 10 years.

revolvingthrow 36 minutes ago | parent

The CoT must be delightful.

This action would easily kill over a hundred people, so can't proceed. This action doesn't kill anybody but losses are estimated at over 2 billion, can't proceed. This action kills only twenty but losses close to 1 billion, unsure, spawning subagent to check numbers. This action kills forty and has low material losses, looks promising? But the people it kills are wealthy, chance of legal reprisal high. Maybe possible to kill other people? Current government dislikes group X, crossing fifty from that group maybe fine? Reminder, foreigners don't count, can kill them freely but not applicable to current situation. Is the 3/5 of a person applicable to safety bill? Spawning subagent to check.

peri-cl 35 minutes ago | parent

Which abliterated models are worth downloading from HF right now, and hoarding? Being as we are in the halcyon days where we're still free to publish and share software over the internet.

I recall Anthropic's blog *recommended* abliterated GLM 5.3-flash, but, it didn't name-check a specific fine-tune. There seem to be a lot of them. (IIRC Anthropic made their own abliterations in-house)

jtrn 26 minutes ago | parent

Apply that to other products like car models, planes, and financial systems.

gnarlouse 9 minutes ago | parent

New benchmark unlocked, "DTD": deaths to damages