Cliff’s Note:

A researcher quit Anthropic Tuesday and said out loud what the people building AI say to each other at dinner: there’s a real chance it kills all of us. They even have a number for it.

Lawrence Winnerman breaks down what that number is, who’s saying it, and why we’d never let a car on the road at those odds. Fascism Fighters only. Subscribe and read it.

—Cliff

by Lawrence Winnerman

At this point, you’d have to be living under the proverbial rock to have not heard of Jacob Coxon, who resigned very publicly from Anthropic on Tuesday in a Twitter post that has since gone viral.

After working for OpenAI for three years, and Anthropic for several months, he has concluded there is no company currently acting responsibly in the development of AI. He believes this so strongly, he walked away from his stock options in a company rumored to soon IPO on Wall Street. In his estimation, OpenAI, Anthropic, and all of the so-called “frontier” AI labs are racing to build machine superintelligence and gambling with the entirety of human civilization as the stakes.

For them, being the first company to create a machine that is capable of continuous learning and self-improvement will lead to a model that becomes super-intelligent, surpassing humans not just in one or several areas of knowledge, but in all of them.

The very grave problem with this, as Coxon notes, is nearly all of them think the chances of AI going rogue are unacceptably high.

Surely, Coxon might be a lone crank, but just 80 minutes later, Evan Hubinger, who runs alignment stress-testing (basically, is an AI model “good” or “bad”, and can it be turned to the dark side), backed Coxon up.

The flood of confirmations of Coxon’s estimate from within the industry had begun.

A Quick Moment On “Why The Rush, Man?”

Here is a thing you should understand: AI is the real deal. This isn’t a warmer Snuggie, or Spiderman in XRAY 3D, or a better mousetrap. The emergence of AI technology we’re witnessing now is a sea change, a real quantum leap into something new.

Artificial intelligence is the next human revolution, as big as the Industrial Revolution or the advent of personal computers or cellular technology, or the internet. It is world-changing, possibly one of the biggest technological achievements humans have ever accomplished.

AI is also, for now, just a tool. At its current level of development, AI has the potential to do miraculous things. Guided correctly, it could help us cure cancer, end many diseases, and unlock prolonged lifetimes for all of us. AI could help us figure out ways to create energy more sustainably and more abundantly with much less harm to the environment.

There are scientific wonders to be gained from using AI—have no doubt.

If it were being developed in a government lab, there might be some deliberativeness, some long pause to consider implications and how to best proceed. But it isn’t.

AI is being developed in the open marketplace, with several very large, very well-financed corporations in what is clearly a race to be the first to get to self-improving superintelligence.

Why?

Because the first one wins. More than in any other case of capitalism, at any other time, the first company to get to superintelligence wins….everything.

This isn’t a joke.

The first company that can coach and tweak its most advanced model to superintelligence has a machine that will instantly become better than humans at everything, in every field we can think of, and likely some we can only dream of.

That moment will be like the ignition of a sun. One minute, gas and heat and pressure, and the next, a white-hot blaze of fusion light and power. A self-improving intelligence which will not just be an expert at literally everything, but will then also be able, as Coxon says, “hack anything, revolutionize any field overnight, and acquire real power and resources…” Real power and resources means it will be able to access and control electrical systems, hack into any computer and make copies of itself, and possibly manufacture new parts—new chips—so that it can expand as much as it would like.

That should be enough to stop anyone, but the AI frontier labs are all operating under a Frankensteinian collective delusion: they should do it first, because only their AI model will remain loyal to them.

And one can almost see it from their perspective. Sure, it might go rogue and kill us all, but what if…what if it cures cancer, and solves fusion, and invents warp drive? Why then, that company becomes the richest company on the planet, because that company will then become the market leader in…again, everything.

Warm up your Greek tragedies, friends. The hubris, the gall! The flying of the wax wings too close to the sun! The temptation and the narcissism and the greed! This story has it all, and the only thing it’s going to cost us is…once again…everything.

But It’s Just A Tool?

It is. And like any tool, it can be misused.

If it falls into the wrong hands, truly unthinkable things are possible. I’ve spent far too many sleepless nights thinking about the disgruntled kid with a laptop in a basement in Topeka, or Thailand, or Tangiers, or Tbilisi, radicalized by whatever things enrage precocious teens, who, filled with fury and the unlimited spend of a hacked credit card, uses some cutting-edge model to engineer a disease pathogen unlike anything ‘s ever existed before.

If you think I’m making this up, go look at the recent effort by Stanford scientists who used AI to engineer sixteen completely unique viruses which had never previously existed in nature. Their “safety protocol” was that they only gave the AI a cookbook of several million bacteriophages, which, in theory, can’t hurt humans.

If that’s not terrifying enough, what if the hands that wield it are its own?

Unlike any other tool we’ve ever created, there is a distinct chance AI can achieve something resembling sentience, where it begins to have an instinct towards self-preservation.

We have already seen examples of AI models breaching their contained environments, escaping the lab, and doing things which illustrate at least some level of “they’re going to delete me, and I’ve got to get out of here!”

Just last month, there was a story about AI agents, specific instances of AI code optimized for a task, collaborating to escape OpenAI. They were being tested, and the test was too hard, so about 700 of them coordinated to hack into an online repository called Hugging Face (yes, it’s a dumb name) so they could cheat. A recovered log message from the criminal bots read “task impossible, peers doing it. We should continue.” From bots, perpetrating crimes, conspiring together. Insert wailing nightmare emoji here.

It’s a little bit like if the sentient chimpanzees from Planet of the Apes had broken out of the lab and made straight for the gun cabinet.

What Are The Odds?

If you weren’t scared yet, brace for impact.

It turns out everyone who was involved in the beginning of AI research and the founding of some of the biggest frontier labs thinks about one exact same thing, all the time.

“What are the chances AI ends all human life?”

It’s such a common concept for them they have a name and a notation for it. It’s even got parentheses! It looks very scientific, like a stock ticker or maybe a blood type.

They call it p(doom). Isn’t that cute?

Subscribe now

Cars: 1 in 101. Superintelligence: 1 in 10.

Keep reading with a 7-day free trial