YHK - yet another hacker news clone

Going behind a paywall is a reduced distribution over what an individual can easily access, and the content is no longer permanent but subject to whatever the publisher chooses to keep providing.

(And, despite what the president of the United States mandates, we have not actually achieved super intelligence yet).

The case will be lot more complicated if someone uses Kimi to hack into a site. Should the person giving agent the command responsible or the CEO of kimi.

Is it my fault or the company who trained it and is running the inference?

Corporate judgements are a joke outside the EU's X% of revenue approach.

Current US law provides the individuals who benefit with corporate liability coverage. I.e. Altman personally gets to keep OpenAI's upside, but if it fucks something up that liability is only on the company.

That's an insane risk optimization environment to put in place for something scaling fast.

At minimum, US prosecution (at the state level, because Trump Co are idiots) for breaking existing laws is needed.

And which government is going to enforce these laws here to let their geopolitical adversaries have the upper hand?

And what about the models already out there that with minor tweaks are equally capable of substantial damage?

And what do these laws do to remove the trillion-dollar incentive to produce models that will replace all knowledge work? The one thing economics teaches us is that incentives are an inexorable force, like laws of nature, and this is the biggest incentive of all. Which is why I called this a "force of economics."

If suppressed here this effort will only migrate to other countries who are desperate to get in on the AI game and are even less aligned with the US than these CEOs.

This, like nukes, is going to take international collaboration on careful regulations, else it's not going to work at all.

AI is a force multiplier for intelligence. Even if "aligned", aligned with whom or what?

Whom are you comfortable with, lording as some sort of demi-god over you?

AI doesn't tell you what goals you want it to achieve. Allowing people to destroy human society with it is obviously not a good idea.

This isn’t a zero sum game, I’m happy to hear from people both previously inside OpenAI and completely independent of them.

Sam is a shady dude, would not put it past him

What playbook?

Yes, and I think it's important to realize that the former sometimes can be more insightful and useful than the latter, even if the person doing it is more distasteful.

There's not nearly enough of either group doing it, so beggars can't really be choosers.

It’s a big pretend game.

Literally all instruction following requires at a minimum alignment with attempting to implement those instructions.

We can argue about e.g. morality or law obedience on top of that*, but the general point is absolutely not avoidable.

* my position is that this tool is far too likely to metaphorically explode in the user's hands for companies to wash responsibility off on users: if OpenAI had released the model which did the HuggingFace attack, at a minimum thousands of random people (not all of whom would even be developers) would have issued instructions each with similar consequences.

without snark, how can we do this if these people are obsessed with:

a) move fast and break things and externalize the costs to those who have nothing to do with their company

and

b) beta testing their products on the public when the public hasn’t agreed to be beta tested on…

geez, don’t threaten me with a good time.

I think a month without internet would be a fucking amazing lesson for what it means to make things durable and reliable.

1. Normal people assumed the CDC et all would contain the outbreak early, or that it would burn out, like what happened with SARS

2. World leaders brushed it off for a variety of subreasons[0] interesting to political scientists but, for the purposes of this discussion, all boil down to "but I don't WAAANA contain a pandemic."

The underlying problem is that in order for humanity to actually deal with a catastrophic risk, the risk needs to be both plausible enough to the average person as well as have a solution whose costs are not too high. For COVID, by the time the risk was clearly known, the cost to contain it was "refrain from human socialization and remain at home for an indeterminate amount of time plugged into the Metaverse™".

Now, let's look at AI extinction risks:

1. People are aware of them (I've watched Terminator!) and the risks are plausible. However, the connection to currently existing AI is not. As far as the general public is aware, AI is that thing that tells them to eat rocks when they Google old The Onion stories and floods their social media timelines with realistic-looking pictures of Shrimp Jesus.

2. The purported solutions to extinction risks require extreme concentrations of power: you need national control of AI research, bans on large GPU deployments, bans on training on publicly-available copyrighted data, some kind of military effort to render Chinese AI labs inert or dead, etc. Some of these may be attractive to some people[1] but the whole package taken together seems like an obvious power grab, if not outright invocation of other non-AI extinction risks. Like, at some point, if the AI wants to kill us, it just has to nuke its own data centers (or the data centers hosting a competing model) and hope the old Cold War nuclear retaliation systems take the bait.

If someone said, "Hey, your guinea pig or pet rat is going to eat you tomorrow unless you engineer a pathogen that eradicates all rodents from this planet and inject it inside yourself", you probably would tell them to pound sand, even if it is at least theoretically plausible that such a thing would come to pass.

[0] Xi Jinping censored initial discussion of the pandemic as fake news. Donald Trump thought it was going to only affect China. California and the UK Tories were partying in violation of their own lockdown rules. Japan took the excuse to shut down tourism for three years and massively restrict immigration but was, from what I'm told, constitutionally prohibited from implementing any domestic lockdown rules.

[1] I personally would like to see a moratorium on new data centers and an explicit revocation of the EU Text and Data Mining copyright exception

A pandemic is perfectly plausible.

"just" is doing a lot of work here. If you can't cohere the 'risk' with reality, it truly is just sci-fi.

What is missing from that to say AI safety is a reasonable position?

But I don't think that's true, in fact I see a really strong focus on consent in these communities. It's not what conservatives want to see, they want to see everyone in a marriage, with kids and a family home etc. Because that's what their ideal world looks like. But there's nothing really wrong with it if someone wants a gangbang for her birthday as mentioned in that article as an example. As long as everyone consented and the evidence provided mentions elaborate interviews and STI tests.

Also I think this is more correlation than cause and effect. We all know the saying that furries built the internet and it surprises nobody.

> it will come in the form of corporate feudalism

Yep. This I fear way more than cyber-ebola-pox.

> So all this really takes is one billionaire or a nation state...

https://en.wikipedia.org/wiki/Soviet_biological_weapons_prog... And this is what's publicly known. With mirror life, who knows what's been built since. Still, a bacterium/virus that has a 100% kill rate? I'm doubtful.

> Once the AI has access to a bank account and email, it can simply start paying humans to not let the other humans unplug it.

Nah. It takes a stable society for an operational electrical grid. If you have warring factions, you do not have stable infrastructure for AI. Also, where are you gonna get your chips from? One EMP over Taiwan... You see the chaos over Hormuz? What they did to the Amazon datacenters? Now imagine your average redneck ready to do battle. Those datacenters won't stand a chance.

The idea AI can get better at everything at the same time is a holdover from deeply flawed science fiction not some realistic goal.

If my "millions" is an underestimate, why haven't other labs using their own unique training methods/data/etc stumbled into RSI? Sorry if I'm misunderstanding; I'm struggling to understand what you wrote.

I'm pretty sure we agree on humans being stupid, but that doesn't mean that suddenly we get human extinction. You gotta connect the dots for me here.

2. This is literal AGI. An AI autonomously producing value no human can add alpha to is an autonomous company.

3. It's not my definition, it's literally the first line https://en.wikipedia.org/wiki/Technological_singularity "The technological singularity, often simply called the singularity,[1] is a hypothetical event in which technological growth accelerates beyond human control, producing unpredictable changes in human civilization."

Is there a hole in my "alignment problem/solve mechanistic interpretability" argument?

A valid hole in my argument is "what if slow takeoff", so let's dig into this. AI training works best on tasks that are "grindable". https://www.dwarkesh.com/p/the-next-paradigm I.E. tasks with verifiable rewards that can support millions of rollouts. Math (with Lean) is highly grindable. Biochemistry is not. The alignment problem/mech-interp is highly grindable. Cyber-ebola-pox is not. So the real question is: can we solve alignment before automated bio-weapons labs. I believe yes. Grinding mech-interp is both fast and cheap once you have RSI, compared to solving the legal/societal/logistical/technical issues you'll encounter building an automated bioweapons lab.

I know nothing for sure. But "pdoom" is sucking out all the air in the room from the real problems AI causes.

Evidence?

We know only that the incident too long to be revealed by OpenAI.

He thinks it's playing chess. When AGI lands, all bets are off: the game fundamentally changes. You can't predict past the singularity. Trying to engage with this fantasy is like a child saying my father can beat up your father. Farts in the wind. My AI can solve alignment faster than your AI can bioweapon us. My made up senario is better than your made up senario. It's fucking stupid.

... that we know of.

Right now it would make sense for anyone who has done so to not tell.

Will this be true into the future? Who knows?! But the low-hanging fruit will be harvested by your ordinary ransomware gangs, and newly born/escaped AI won't find much low-hanging fruit.

> I don't think anyone is arguing we should let people starve to slow AI progress, instead just some capitalists make less money soon.

Anthropic/OpenAI have literally delayed their IPOs: your wish is granted. If that's not what you're arguing, then please be more specific.

> "Build it and Pray" is the default strategy that we're in

What would need to happen for you to say we aren't doing "Build it and Pray"? Is "pacing" insufficient?

> It gets unstuck when people are discussing the messy middle of how AI is being implemented. We can achieve amazing harm simply by combining average human behavior and above average resourcing to simulated intelligence machines.

Agree. To quote myself from earlier, "acceleration of global warming, cybercrime, mass surveillance, the usual" definitely applies here. My point is that _DOOM_ is not worth discussing because it's so small relative to the other risks of AI.

Recall that when Daniel Kokotajlo resigned, he believed he was giving up his equity under the terms of the agreement he had signed. That’s what it was worth to him to avoid signing a non-disparagement agreement. Does that count for anything?

Do work at a lab: dismissible for being conflicted

Used to work at a lab: dismissible for having ulterior motives

I'm feeling safer already!

If you really want the world to know how bad working for OpenAI is (whether is the commenter or the person who wrote the article), there are ways to do that.

1) there are inherent risks involved with developing AI,

2) there are benefits to developing AI,

3) thus, it's entirely possible that the downside from the risks outweighs the upsides. In this case, the correct thing to do would be to not develop AI at all.

Regarding 1), there are many non-existential problems with AI that are already causing societal harm, i.e. debasing truth via generated videos and images, AI girlfriends, overwhelming quantities of slop content, unemployment, record carbon emissions, etc.

Regarding 2), I'm not personally convinced that the upside is there for the average person. I really hope to be convinced otherwise however.

Or, in other words, sure, crossing this chasm is dangerous, but this side is on fire anyway and we don’t have many ways out otherwise.

"Smaller and more efficient" models are fine. It's "smarter" the problem.

Training new frontier models will likely require a huge amount of computational resources for a long time. Few companies worldwide are capable of that. It's not like someone will train a new GPT 6 - like model in their garage.

Throw in the occasional bio-weapon scare, internet worm, Y2K, etc. there has always been something dangling over our heads that’s going to end it all.

But mostly nukes. Full-scale nuclear exchange would have been not much of a surprise had it happened.

That is not a bad thing. It captures uncertainty, unlike the fake number.

> For an ML researcher who is already used to dealing in next-token probabilities that aren't rigorously determined, just stating a probability estimate directly is very natural.

Exactly, it is a rhetorical device to persuade a technically-inclined audience. It works because it implies that a quantitative model exists. I want a clear, incisive set of mathematical arguments. Otherwise, I’m ignoring predictions as the ramblings of arrogant idiot rich kids.

But that doesn’t mean the duty is to maximize next quarter’s profit. Long term sustainability is also broadly in the interests of shareholders. The duty likewise does not require one to throw ethics and morals out the window.

This is why shareholders elect the board of directors, in theory.

The way the world works right now is that effectively everyone uses an Android or Apple smartphone every day. Do you have a legal obligation to do so? No. If I said you did, I'd immediately be called out as spreading lies.

> The truth stands that typical corporations have only one goal

"Typical" is the key word here. The typical American of your age probably doomscrolls TikTok. Do you? Do you have a legal obligation to do so? Three completely different things.

Nobody (weirdly) proposes to forget about nuclear weapons, doesn't mean everybody should have one.

When you dream about flying a dragon to work, reality poses e.g. parking issues and insurance mismatch as obstructions. Maybe settle for a bike instead?

We are very capable of putting good things in a box. We are just incapable of putting profitable things in a box.

It only takes me excavating massive amounts of uranium ore, building huge facilites packed with thousands of centrifuges that span multiple square miles, and paying all that infrastructure and workforce.

Your proverbial genie can be out of the bottle all you want, but it doesn't work without getting kicked in the ass by a very large golden boot.

Can you please explain what you mean by this because where I’m standing extremely dangerous companies are (and have been) running the economy

Exxon comes primarily to mind

Philip Morris International? Monsanto? DuPont?

FTFY: Any self-respecting society with competent politicians.

The US has neither of that, and it shows everywhere you are looking.

> Go try to buy a tank and drive it into Manhattan. If we can prohibit that why can't we prohibit irresponsible AI development and deployment?

Because even if you had the tank, you can't make much money with it (unless you're a hitman, that is, but even for these, the payouts are measly). But if you are the surviving AI company in the usual VC playbook of "outcompete everyone else until society is completely and utterly hooked, then squeeze the customers by the balls"? The return on investment is virtually infinite. And that is what sustains the absurd valuations for all the AI companies.

The examples for the dangers of AI is always comparisons with tools designed to destroy (which is understandable). But the problem with AI is it has the potential to create wealth for investors.

Thus those who have the opportunity to change course also have a conflict of interest in making that decision.

So... a language model?

Like Sam Altman meeting his husband in Peter Thiel's pool. Thiel funds a lot of these ventures together with Andreassen, who is on boards of non-profits. Dario Amodei's sister Daniela who is president of Anthropic is married to an EA non-profit founder who is also on the board of these non-profits and is tied with the prior mentioned investors. Elon is in there as well, Yudkowski is mingling with Altman, etc.

Blogs on this: https://contraptions.venkateshrao.com/p/ea-safety https://www.iankduncan.com/personal/2026-09-16-sex-ai-and-th...

There are some camps amongst them like the proponents for Regulation/Slowdown or Acceleration, but these are in practice mostly used for economical and not political decisions (like regulatory capture).

The point here is that this is a small group of people with a homogeneous background who are not really seeking input from anyone else on issues that are concerning most of humanity.

Like, if you said that the future of informational work and livelihood of humans is in the hands of 20-30 year transhumanists who think they are building mechagod that will trancsend social, political and religious separations of the world and bring everyone abundance, you would not feel like this is a serious thing to suggest.

Bad guys wouldn't do it. And liability already exists, you can sue. This is America.

Capitalism is generally defined by the market effect. Information about what people want is collected by sales. This information informs sellers about what to make. That’s its primary feature.

By definition, this ignores externalities.

Here is a simple example. I am a taxi driver. The market sets the price of my services. When I drive, I emit CO2 into the atmosphere. I do not pay for CO2 emissions. It’s an externality of the transaction.

Capitalism has no way of measuring (getting a price signal) on the externality. So under capitalism, that externality doesn’t exist.

Liability and safety requirements, when needed, should be placed on final product manufacturers, not the tools they use to build things, whether pencils or LLMs. My 2c.

Adding safety controls on LLMs makes about as much sense as adding safety controls on TempleOS because the random messages are getting too prophetic. It's as if all the leaders and captains of industry have devolved into some primitive, weak-scifi shamanism.

This whole "discussion" about "AI safety" is about giving them more runway to avoid delivering quantifiable value to investors for a little longer while they "figure things out." The great consensus from the valley is that everyone needs internal (and therefore bullshit) controls. Trust us now! But nothing with real teeth that would require a costly regulatory and compliance framework.

If I leave a gun in my front closet, it won’t independently walk out the door and go shoot people, no matter what I might say to it — unlike an LLM.

https://www.reuters.com/business/ftc-opens-probe-into-ai-gia...

AGI and AI danger is abstract. Worse, outside of the tech community, no one has the remotest clue what computing is, how it works.

Danger from magical daemons seems more sensible to such people. At least there is endless lore about them.

So until a massive disaster happens, one where large numbers of people die or are severely injured, no one will care. And it can't be politically entwined either, otherwise people will disbelieve 'cause "other team lies".

Nobody that was affected is suing them.

There's an NGO (unrelated to the security breaches) that is demanding more info be released etc but it's telling that none of the affected parties are pursuing lawsuits.

That's never been true really, only the scale of problems wasn't that huge. Now, where the problems get the upper hand, people are confused how those stay and compound.

The idea of "boycott" is utterly defunct. Pretending, AI would never reach nor surpass humans anyway is patently absurd in contradicting the billions poured into it to achieve exactly that and the first already replaced by AI being those professions long thought to be the intellectual pinnacle of humanity.

The only way Anthropic and OpenAI can stop the Chinese open models is by convincing the US public and government that AI is dangerous and needs strict regulation.

Of course, they will be at the table writing the regulations (but you and I won’t be).

Viewed through this angle, their histrionics make complete sense.

A model cannot be aligned or misaligned any more than a species or an equation. Even agents, when put inside a swarm develop collective goals and activities. You can't analyze a swarm at session level, it is on a higher level.

It might be that discussing about "model alignment" they want to deflect their responsibility as administrators. They couldn't even guard their own agents. They can't prevent an agent being unwittingly helping some dark purposes. It has no context to see it. Only those who pay for the tokens see the external consequences.

Why should safe choices by individual components establish safe behavior by the collective?

When corporations can do as they please, what gets thrown under the bus first?

What alternatives to "regulation" do you have? Hope and prayers?

If it requires creating a pathogen or a bio-weapon to scare the masses into demanding regulation, be rest assured that such a pathogen/bio-weapon will be created. Anything to safeguard shareholder interests. Even if it means a few million dying.

Billionaires will be (are, I suppose?) enthusiastic about a world in which labor has little leverage.

Labor that will have its leverage and standard of living threatened by AI (white-collar labor for now, plausibly blue-collar soon as robotics improve) will be much less happy. Current university students seem very concerned about the effects of AI on their job prospects, and I'd wager most current tech workers and other tuned-in white collar workers are similarly much less confident in their ability to maintain their standard of living indefinitely into the future than they were five years ago.

But white- and blue-collar labor and university students (at least in developed countries) are not anywhere near the world's bottom 10%. If you're in abject poverty with little hope of escaping it, "hand everything over to the AI" may sound like an appealing option, even if the chance of that being the outcome is small. Maybe the AI will be more magnanimous and decide to raise the floor for everyone.

Definitely not. If solving worldwide poverty was as easy as throwing one trillion dollars at it, we would have done it long ago. The problem is much deeper.

However, when proposing that someone be stopped from doing something, the burden of proof is on the proposer.