YHK - yet another hacker news clone

This certainly looks like a way to slow down competitors and regulate foreign and open models.

It's always about money

Anthropic releases models as open weights + more information about how they do training and alignment

> Crack down on unauthorized distillation / prevent weight theft

Actually hilarious to put that in writing, given the genesis of this entire business model.

1) AI is a dangerous technology that can literally end the world.

2) Only me and a handful of other [people like me] should be trusted to determine who can use it and how.

3) The state must use its coercive power to support my control of this technology to the exclusion of [people not like me].

This is the only concrete prediction in the entire essay.

And it simply cannot happen. For one, you will need billions worth of compute.

First: if they are unilaterally doing it: what about their competitors? Is this a chance for OpenAI to pass them? Or China? If either did, would that be a net-benefit for them or the world?

Maybe this is a ploy for "regulatory capture". And that might help them in the short term. But the rest of the world will continue on. Honestly, it isn't clear to me right now if the winning strategy has as much to do with being the smartest company or simply the one that can command the most compute.

If Anthropic fumbles their IPO and OpenAI scoops them, I don't think I will be happy.

Amodei tops it off by using diseases to capture the reader's favor. It no longer works, people are just disgusted by it after four years of daily marketing.

How exactly? So far AI has accelerated misinformation at scale and wealth concentration.

However the real reason to slow down is more of how it's introduced to the economy. Hypereautomation will kill jobs and destroy the economy. I work as an AI engineer and companies are delivering products at vibe coding speeds to kill jobs and at the same time automating internally. All companies are doing this at the same time and the target it sto eliminate workers.

Most people are so extremely slow to pick this up. How hard can it be to understand what hyperautomation does to the workforce? Companies are desperate to surrivive and they will do all it takes to lower their costs and at the same time not loose to competitors. Its Wild West out there.

This is called malware and creating and distributing it is a felony. Why is this not called out ? AI companies seem to be living in a bubble where normal rules dont apply.

There could be an element to truth to it, but it's certainly not the entire story and is just so tiresome at this point.

That said, if slowing this down were possible, I think it would have happened by now. Maybe he’s hoping the OAI/HF incident becomes something the industry can rally around, but it won’t. The fundamental problem is simpler: no one will slow down because no one trusts anyone else to slow down.

> Transparency. Regardless of what commitments we make, the public deserves to know what is going on. Anthropic has been a supporter of transparency for a long time

And then it goes on tangent of how we should pace everything (not just AI, but also the ingredients of what goes into AI, whatever that means), but only within approved democracies™, and outright restrict everything outside approved democracies™, because reasons that are definitely not about succeeding commercially that is threatened by the most transparent instrument possible - open weights, produced by basically just China.

I wonder what Dario would have done if open weights weren't produced by US's geopolitical adversary. How would an authoritarian manifesto be wrapped then?

Suppose it was, I don't know, Australia producing SOTA open weights, because they believed it was the right thing to do for the benefits of humanity - would Dario propose making Australia a geopolitical adversary too?

This and related quotes seem to attempt to place some of the burden on the US Government as opposed to themselves. It seems to be a trend that Anthropic wants to be able place some of the burden of preventing distillation on parties other than themselves.

Preventing distillation is fundamentally their problem. Whenever it happens, it is primarily due to their security efforts not being able to prevent it. I'm tired of seeing them play the blame game and divert responsibility. Sure there is a level of national concern, but the response could simply be the government placing stronger controls on Anthropic themselves. If they are unable to prevent distillation, then the solution may be to literally limit their distribution until they are capable of doing so effectively.

> A race to the bottom, spurred by commercial incentives, can make these risks more acute

I also think it is ironic that they're planning an IPO while also talking about a race to the bottom due to commercial incentives. These are fundamentally at odds. Being a public company means you are beholden to investors and a board with the primary goal of making more money. What higher commercial incentive is there than that.

If commercial incentives are as dangerous as he states, potentially becoming one of the largest public IPO's and largest traded companies in history a pretty strange way to reduce the risk of commercial incentive risks.

People are not going to slow down because this was brought to them on less than endearing terms. They don't see any of these stated noble intentions.

Claude was already used to cause economic, political and social destruction. As Anthropic is seen doing it, others aren't going to just sit down, read the blog and say, oh, I'll stop developing models because Dario, you touched my heart with your true words.

This is impossible. The capability keeps advancing regardless. More people use AI, that feedback is used for reinforcement, that reinforcement makes the model better. Not just in the US, but for every lab and model. Whether it's distilled or direct reinforcement, same result. You can't keep the whole world from working on AI.

> it’s my worry that in 6–12 months such a swarm could be capable of taking over the entire internet with a persistent botnet [..] and that the scale of damage would continue to increase from there if AI becomes more powerful without the necessary guardrails

If true, then what we need isn't guardrails or less-capable AI. We need an internet that isn't fragile. If the infrastructure is vulnerable, the answer isn't to make a law that asks tool-makers to blunt their tools. The answer is to fix the goddamn infrastructure. Today it's almost trivial to take down large parts of the internet (happens by accident all the time). This should've been solved ages ago.

We didn't have the motivation to fix it before, but now we do. But Dario's answer isn't to fix things. It's to hold back progress so we can maintain the status quo (shitty infrastructure) and he can keep his company making billions of dollars. He could be calling for fixing these things. But he'd rather go the easy route, which just happens to advantage his company in the process.

If the US government is really concerned with defense, they need to invest more of their nearly $1T in federal funding towards making the internet safer. They need to do this for us, and other nations, since 1) we depend on the rest of the world for our goods and services, and 2) if other nations are taken down, they can't spend money on our financial and tech services (which are the only industries we have left).

Dario calls for regulation later in the post. If he's okay with government safety regulations for AI software, he should be okay with the same regulations for all software, whether it's AI or not. We need a national software building code, focusing on internet safety.

Dario: We gotta pace the frontier!

Sam: How do you pace a frontier? The frontier's not goin' anywhere. It's right where it was, just go out and explore it!

Dario: Sam, I'm tellin' ya, ya gotta pace it! Things are getting very doomy out there, and Dario's gettin' upset!

1. Artificial intelligence is clearly a vastly dangerous technology with plausible potential for human extinction.

2. This scares people.

This sounds like pre-IPO hype. They better just try and top astra.

The cybersecurity threat will likely be a cat and mouse react game for a while. Just like robberies / the mob was in the early 20th century.

Social forces bring things into balance over time, much more so than the proactive actions of individuals.

Unfortunately, the genie at this point is unlikely to go back into the bottle. There's enough 'intelligence' out there that a super intelligent model could emerge at some point in spite of pacing.

This isn't a doomer scenario - we tend to navigate social changes better than we ever could have hoped.

Super intelligence (the ability to have many smarter minds than your own reporting to you) has always been available to the wealthy and powerful. They used it to invent nuclear weapons, cause climate change, mechanize warfare, and pillage the global south.

Now that this same tool is on the cusp of being available to everyone, capital is starting to panic and throw up fences. They want to pump the breaks on a revolution they know they can't control. They see what they've done with super intelligence and (perhaps not unreasonably) fear what the masses will do with that same power.