YHK - yet another hacker news clone

Going behind a paywall is a reduced distribution over what an individual can easily access, and the content is no longer permanent but subject to whatever the publisher chooses to keep providing.

(And, despite what the president of the United States mandates, we have not actually achieved super intelligence yet).

The case will be lot more complicated if someone uses Kimi to hack into a site. Should the person giving agent the command responsible or the CEO of kimi.

Is it my fault or the company who trained it and is running the inference?

Corporate judgements are a joke outside the EU's X% of revenue approach.

Current US law provides the individuals who benefit with corporate liability coverage. I.e. Altman personally gets to keep OpenAI's upside, but if it fucks something up that liability is only on the company.

That's an insane risk optimization environment to put in place for something scaling fast.

At minimum, US prosecution (at the state level, because Trump Co are idiots) for breaking existing laws is needed.

AI is a force multiplier for intelligence. Even if "aligned", aligned with whom or what?

Whom are you comfortable with, lording as some sort of demi-god over you?

AI doesn't tell you what goals you want it to achieve. Allowing people to destroy human society with it is obviously not a good idea.

This isn’t a zero sum game, I’m happy to hear from people both previously inside OpenAI and completely independent of them.

Sam is a shady dude, would not put it past him

What playbook?

without snark, how can we do this if these people are obsessed with:

a) move fast and break things and externalize the costs to those who have nothing to do with their company

and

b) beta testing their products on the public when the public hasn’t agreed to be beta tested on…

geez, don’t threaten me with a good time.

I think a month without internet would be a fucking amazing lesson for what it means to make things durable and reliable.

1. Normal people assumed the CDC et all would contain the outbreak early, or that it would burn out, like what happened with SARS

2. World leaders brushed it off for a variety of subreasons[0] interesting to political scientists but, for the purposes of this discussion, all boil down to "but I don't WAAANA contain a pandemic."

The underlying problem is that in order for humanity to actually deal with a catastrophic risk, the risk needs to be both plausible enough to the average person as well as have a solution whose costs are not too high. For COVID, by the time the risk was clearly known, the cost to contain it was "refrain from human socialization and remain at home for an indeterminate amount of time plugged into the Metaverse™".

Now, let's look at AI extinction risks:

1. People are aware of them (I've watched Terminator!) and the risks are plausible. However, the connection to currently existing AI is not. As far as the general public is aware, AI is that thing that tells them to eat rocks when they Google old The Onion stories and floods their social media timelines with realistic-looking pictures of Shrimp Jesus.

2. The purported solutions to extinction risks require extreme concentrations of power: you need national control of AI research, bans on large GPU deployments, bans on training on publicly-available copyrighted data, some kind of military effort to render Chinese AI labs inert or dead, etc. Some of these may be attractive to some people[1] but the whole package taken together seems like an obvious power grab, if not outright invocation of other non-AI extinction risks. Like, at some point, if the AI wants to kill us, it just has to nuke its own data centers (or the data centers hosting a competing model) and hope the old Cold War nuclear retaliation systems take the bait.

If someone said, "Hey, your guinea pig or pet rat is going to eat you tomorrow unless you engineer a pathogen that eradicates all rodents from this planet and inject it inside yourself", you probably would tell them to pound sand, even if it is at least theoretically plausible that such a thing would come to pass.

[0] Xi Jinping censored initial discussion of the pandemic as fake news. Donald Trump thought it was going to only affect China. California and the UK Tories were partying in violation of their own lockdown rules. Japan took the excuse to shut down tourism for three years and massively restrict immigration but was, from what I'm told, constitutionally prohibited from implementing any domestic lockdown rules.

[1] I personally would like to see a moratorium on new data centers and an explicit revocation of the EU Text and Data Mining copyright exception

A pandemic is perfectly plausible.

"just" is doing a lot of work here. If you can't cohere the 'risk' with reality, it truly is just sci-fi.

What is missing from that to say AI safety is a reasonable position?

But I don't think that's true, in fact I see a really strong focus on consent in these communities. It's not what conservatives want to see, they want to see everyone in a marriage, with kids and a family home etc. Because that's what their ideal world looks like. But there's nothing really wrong with it if someone wants a gangbang for her birthday as mentioned in that article as an example. As long as everyone consented and the evidence provided mentions elaborate interviews and STI tests.

Also I think this is more correlation than cause and effect. We all know the saying that furries built the internet and it surprises nobody.

> it will come in the form of corporate feudalism

Yep. This I fear way more than cyber-ebola-pox.

> So all this really takes is one billionaire or a nation state...

https://en.wikipedia.org/wiki/Soviet_biological_weapons_prog... And this is what's publicly known. With mirror life, who knows what's been built since. Still, a bacterium/virus that has a 100% kill rate? I'm doubtful.

> Once the AI has access to a bank account and email, it can simply start paying humans to not let the other humans unplug it.

Nah. It takes a stable society for an operational electrical grid. If you have warring factions, you do not have stable infrastructure for AI. Also, where are you gonna get your chips from? One EMP over Taiwan... You see the chaos over Hormuz? What they did to the Amazon datacenters? Now imagine your average redneck ready to do battle. Those datacenters won't stand a chance.

The idea AI can get better at everything at the same time is a holdover from deeply flawed science fiction not some realistic goal.

If my "millions" is an underestimate, why haven't other labs using their own unique training methods/data/etc stumbled into RSI? Sorry if I'm misunderstanding; I'm struggling to understand what you wrote.

I'm pretty sure we agree on humans being stupid, but that doesn't mean that suddenly we get human extinction. You gotta connect the dots for me here.

2. This is literal AGI. An AI autonomously producing value no human can add alpha to is an autonomous company.

3. It's not my definition, it's literally the first line https://en.wikipedia.org/wiki/Technological_singularity "The technological singularity, often simply called the singularity,[1] is a hypothetical event in which technological growth accelerates beyond human control, producing unpredictable changes in human civilization."

Is there a hole in my "alignment problem/solve mechanistic interpretability" argument?

A valid hole in my argument is "what if slow takeoff", so let's dig into this. AI training works best on tasks that are "grindable". https://www.dwarkesh.com/p/the-next-paradigm I.E. tasks with verifiable rewards that can support millions of rollouts. Math (with Lean) is highly grindable. Biochemistry is not. The alignment problem/mech-interp is highly grindable. Cyber-ebola-pox is not. So the real question is: can we solve alignment before automated bio-weapons labs. I believe yes. Grinding mech-interp is both fast and cheap once you have RSI, compared to solving the legal/societal/logistical/technical issues you'll encounter building an automated bioweapons lab.

I know nothing for sure. But "pdoom" is sucking out all the air in the room from the real problems AI causes.

Recall that when Daniel Kokotajlo resigned, he believed he was giving up his equity under the terms of the agreement he had signed. That’s what it was worth to him to avoid signing a non-disparagement agreement. Does that count for anything?

Do work at a lab: dismissible for being conflicted

Used to work at a lab: dismissible for having ulterior motives

I'm feeling safer already!

1) there are inherent risks involved with developing AI,

2) there are benefits to developing AI,

3) thus, it's entirely possible that the downside from the risks outweighs the upsides. In this case, the correct thing to do would be to not develop AI at all.

Regarding 1), there are many non-existential problems with AI that are already causing societal harm, i.e. debasing truth via generated videos and images, AI girlfriends, overwhelming quantities of slop content, unemployment, record carbon emissions, etc.

Regarding 2), I'm not personally convinced that the upside is there for the average person. I really hope to be convinced otherwise however.

Throw in the occasional bio-weapon scare, internet worm, Y2K, etc. there has always been something dangling over our heads that’s going to end it all.

But mostly nukes. Full-scale nuclear exchange would have been not much of a surprise had it happened.

That is not a bad thing. It captures uncertainty, unlike the fake number.

> For an ML researcher who is already used to dealing in next-token probabilities that aren't rigorously determined, just stating a probability estimate directly is very natural.

Exactly, it is a rhetorical device to persuade a technically-inclined audience. It works because it implies that a quantitative model exists. I want a clear, incisive set of mathematical arguments. Otherwise, I’m ignoring predictions as the ramblings of arrogant idiot rich kids.

But that doesn’t mean the duty is to maximize next quarter’s profit. Long term sustainability is also broadly in the interests of shareholders. The duty likewise does not require one to throw ethics and morals out the window.

This is why shareholders elect the board of directors, in theory.

I am very critical of AI but this is an unfair assumption

Nobody (weirdly) proposes to forget about nuclear weapons, doesn't mean everybody should have one.

When you dream about flying a dragon to work, reality poses e.g. parking issues and insurance mismatch as obstructions. Maybe settle for a bike instead?

Can you please explain what you mean by this because where I’m standing extremely dangerous companies are (and have been) running the economy

Exxon comes primarily to mind

Like Sam Altman meeting his husband in Peter Thiel's pool. Thiel funds a lot of these ventures together with Andreassen, who is on boards of non-profits. Dario Amodei's sister Daniela who is president of Anthropic is married to an EA non-profit founder who is also on the board of these non-profits and is tied with the prior mentioned investors. Elon is in there as well, Yudkowski is mingling with Altman, etc.

Blogs on this: https://contraptions.venkateshrao.com/p/ea-safety https://www.iankduncan.com/personal/2026-09-16-sex-ai-and-th...

There are some camps amongst them like the proponents for Regulation/Slowdown or Acceleration, but these are in practice mostly used for economical and not political decisions (like regulatory capture).

The point here is that this is a small group of people with a homogeneous background who are not really seeking input from anyone else on issues that are concerning most of humanity.

Like, if you said that the future of informational work and livelihood of humans is in the hands of 20-30 year transhumanists who think they are building mechagod that will trancsend social, political and religious separations of the world and bring everyone abundance, you would not feel like this is a serious thing to suggest.