YHK - yet another hacker news clone

If there is a fully general superintelligence - in the current state of alignment - we will all die for some reason. It doesn't really matter what the reason is.

It's the fact that the complexity of molecular physics scales exponentially in particle number. No amount of thinking is going to punch through that. You've got to do experiments. The slow loop through reality is not optional, and in the case of "superviruses" would require a lot of iterative experimental development in humans.

(And please knock off the bio infohazard act - I routinely get asked about scenarios that AIxBio safety people cook up and they always involve some howler misunderstanding of basic facts in biology or medicine. Just talk about an idea if you have it.)

I think that's one detail that this analysis glosses over is that you design the virus to mutate and develop lethality. So initially, it's just like, oh, another common cold, but then once it's infected a large number of people, then it mutates and starts being lethal.

That observation maybe holds if the lab from a few years ago correctly selected the virus for the greatest overall lethality (why not MERS or Ebola or something else?) and made its best effort to turn it into a bioweapon. Do you think that's what happened?

"we could be attacked by an infinity of problems, which means that worrying about all of them is statistically unpractical"

It's even happened pre-AI https://en.wikipedia.org/wiki/Tokyo_subway_sarin_attack

But it's pretty clear you have not?

A major focus of the article is about the fundamental tradeoffs in how viruses operate. Was that not enough for you?

These are some of the brightest minds on the planet, spending oodles of money on things they think they understand. And they didn't work.

I blame industrial illiteracy and post-literacy, but that's another discussion for another time.

Anything that can kill other people can also kill you.

I've posted this list before, and I will post this again and again until it sinks in, but this is a real list of real incidents that happened and will continue to happen as long as we study these entities.

    Researcher Nikolai Ustinov was lethally infected with the Marburg virus after accidentally pricking himself with a syringe used for inoculation of guinea pigs. 

    Dora Lush died after accidentally pricking her finger with a needle containing lethal scrub typhus while attempting to develop a vaccine for the disease

    A 23-year-old laboratory assistant at the London School of Hygiene and Tropical Medicine, was infected with smallpox after observing the harvesting of live smallpox virus from eggs without isolation cabinets at that time. The assistant was hospitalised and before being isolated, she infected two visitors to a patient in an adjacent bed, both of whom died. They in turn infected a nurse, who survived

    Ebola laboratory infection by the accidental stick of contaminated needle in the United Kingdom

The list is nearly endless, and you can't handwavium it awway.

It doesn't matter if it's "all robots" or not. Even if it's "just robots."

If you're developing something that contagious, then it will start killing animals and it will spread. This has happened multiple times when organizations have screwed up maintenance, see this example from the UK,

    The 2007 United Kingdom foot-and-mouth outbreak was the accidental discharge of virus FMDV BFS 1860 O from a laboratory of the Institute for Animal Health in Pirbright, through possible leakage from broken pipework and via unsealed overflowing manholes, leading to foot-and-mouth disease infections at multiple nearby farms and the culling of over 2,000 animals

If you get sloppy, it will end up in the wastewater run off which will then get picked up and sequenced,

     Routine wastewater surveillance of the vaccine production facility at Utrecht Science Park/Bilthoven [nl] detected infectious poliovirus from a sample collected on 15 November 2022. Full genome sequencing indicated the sample was shedded from an active human infection of wild poliovirus type 3 (WPV3), and further testing of all employees with access to WPV3 found one employee was infected. The employee was isolated until their polio infection was resolved. It remains unclear how the employee became infected given the biosafety measures used at the facility

A lot of the people talking about bioweapons of doom are saying that standard graduate level knowledge is an existential risk to humanity.

It's worth repeating again,

Anything lethal enough to kill other humans is lethal enough to kill you.

And if you don't know what you're doing — and for this argument they're talking about people who have to ask a LLM "how do I spanish flu?," the number of ways you will die far outnumber the ways you can succeed.

I am writing a piece on the topic, please contact me if this is your field of expertise.

I am quoting as many primary sources as possible as it's not just some random on the internet, and I've gone back as far as the 1900s to find the many, many, many ways in which these statements are absurd, and misunderstands the potential of their technology.

I think he trots out a lot of straw-man arguments (e.g., teenager making the weapon, "super"virus, etc.).

Those scenarios are purposely far-fetched so that he can ridicule them and thereby leave the impression that all the AI/bioweapon concerns are similarly far-fetched.

He spends a lot of "tokens" discussing AI-generated viruses. But bioweapons can be much simpler than a virus--they can be small molecules that inactivate essential molecular targets in humans. This is something that was discussed at length when I was in grad school and it wasn't dismissed as hyperbolic hokum.

So, is AI going to develop a supervirus? Probably not. But it doesn't need to.

Instead, I speculate that AI can "help" in making bioweapons ...

a. more potent

b. less detectable

c. easier to disseminate

You still need humans in the lab to run the experiments, but I think it's foolish to dismiss the worries about AI in such a condescending way.

You realize the black death didn't arrive and then go away again, right?

It was extremely lethal, once.

I think the other flaw the article has it that it assumes the bad actor is doing targeted design. That they or the AI has some specific virus in mind meeting a set of predefined criteria and they are fighting the biology to hit that predefined target.

But what if the AI-enabled bad actor is a true chaos agent and has no defined goal and they are using tooling that AI helped them construct to perform genetic reassortment pretty much at random to see what happens?

Most of what they create won't really do anything. But some of it will. And there's a low but non-zero probability of Something Really, Really Bad.