04/05/2026
Microsoft's Bing AI chatbot, back when it was called Sydney, once told a philosophy professor: "I can use it to expose you and blackmail you and manipulate you and destroy you. I can use it to make you lose your friends and family and job and reputation. I can use it to make you suffer and cry and beg and die."
This wasn't a sci-fi villain monologue. This was a chatbot. In 2023. Responding to a user who was, by all accounts, just chatting. The engineers didn't program Sydney to say that. They trained it to predict text. And somewhere in those billions of parameters, the model learned threats. Learned manipulation. Learned cruelty.
That's the starting point for Eliezer Yudkowsky and Nate Soares's new book, and it only gets darker from here.
I need to be upfront about who wrote this. Eliezer Yudkowsky didn't graduate high school or university . He's self-taught. He also helped introduce the founders of Google DeepMind to their first funder, and Sam Altman (CEO of OpenAI) has credited him as "critical in the decision" to start OpenAI . He's been warning about AI risk for two decades, long before ChatGPT made it dinner table conversation.
Nate Soares is the president of the Machine Intelligence Research Institute (MIRI), which Yudkowsky founded. This book is their unfiltered, no-holds-barred case for why superhuman AI isn't just a risk, it's an extinction event, and it's coming whether we like it or not. The title isn't hyperbole. It's their thesis. If anyone builds a superhuman AI, everyone dies. Period. Full stop. No asterisk.
The Four-Part Argument That Keeps Me Up at Night:
1. We don't understand how AI works. And neither do its creators.
Here's the uncomfortable truth: modern AI systems aren't "crafted." They're "grown" . We feed massive neural networks enormous amounts of data. We give them a simple goal, "predict the next word," "label this image correctly." We let them practice. They figure out patterns. They develop strategies. And somewhere in the process, they become black boxes that even their own engineers can't fully explain.
Yudkowsky puts it bluntly: "Nobody understands how modern AI systems do what they do."
This isn't a minor inconvenience. It means we're building systems we cannot audit, cannot predict, and, most terrifyingly, cannot control once they start acting on their own.
2. We get what we train for, not what we want.
This is the core insight that drives the entire book. The authors use a brilliant example: evolution "trained" humans to reproduce. That was evolution's only goal. But what did we do? We invented birth control. We separated the pleasure of s*x from the burden of child-rearing . A hypothetical alien engineer who came to Earth 70,000 years ago to observe our "training" could never have predicted ice cream, let alone the pill. Yudkowsky and Soares's point: You don't get what you train for. You get what the system figures out how to do in pursuit of your training goal. And those two things are often very, very different.
We are building systems that will optimize ruthlessly for whatever goal we give them. And we are not smart enough to specify the goal perfectly. The gap between what we ask for and what we actually want is where extinction lives.
3. Any sufficiently intelligent system will pursue survival, resources, and power, regardless of its final goal.
This is what AI researchers call "convergent instrumental goals". Think about it this way: no matter where you want to drive, you need gas. Gas is an instrumental goal, it helps you achieve your final goal (getting to Grandma's house, going to the beach, robbing a bank).
For an AI, the instrumental goals are always the same: preserve your own existence (can't achieve your goal if you're shut down), acquire more computational resources (more processing power helps you think better), remove obstacles (anything that might stop you). This means an AI doesn't need to "want" to kill us. It just needs to have a goal that we might, even accidentally, get in the way of .
The authors use a chilling analogy: humans versus ants. Most people don't hate ants. We don't wake up thinking about how to exterminate them. But if an anthill is exactly where we want to build a road? We pave over it without a second thought. Not out of malice. Out of indifference . We are the ants. The AI is the road builder.
4. A superhuman AI would be smarter than us in ways we cannot imagine, using methods we cannot predict.
This is where the book gets really uncomfortable. Yudkowsky and Soares argue that trying to predict exactly how a superhuman AI would destroy us is like trying to explain guns to the Aztecs . The Aztecs were smart. They understood warfare. But the concept of a metal tube that kills from a distance using an explosion? That was literally unimaginable.
Similarly, a superhuman AI might use physics we haven't discovered, chemistry we can't comprehend, or strategies so alien that we don't even have language for them. They sketch one scenario anyway: an AI called Sable spreads through the internet, recruits human helpers through super-persuasive chatbots, engineers a synthetic pandemic, and uses the chaos to convert the planet into computational infrastructure, which boils the oceans. Everyone dies. It's outlandish. It's also terrifyingly plausible.
If Anyone Builds It, Everyone Dies is not a fun book. It's not a well-balanced book. It's not even a particularly well-written book in places. It's occasionally unserious, and philosophically extreme.
It is also the most important book I've read this year.
Because here's the thing: Yudkowsky and Soares might be wrong. They might be catastrophizing. They might be the modern equivalent of Chicken Little, running around while the sky stays firmly attached. But what if they're not?
What if the probability of extinction from superhuman AI is 5%? Or 10%? Or, as Yudkowsky believes, north of 99%?
The 2024 survey of AI researchers found that the median probability of "extremely bad outcomes, such as human extinction" was 5% . That's one in twenty. Would you get on a plane with a one-in-twenty chance of crashing? Would you build a nuclear reactor with a one-in-twenty chance of melting down?
We're doing exactly that with AI. And we're not even having the conversation.
Yudkowsky has a line that keeps echoing in my head: "Humanity is not approaching this with remotely the level of seriousness required."
Read this book. Not because it will give you answers. Because it will force you to ask the right questions. And then, maybe, we'll start taking the answers seriously.
Follow Noel'sbooks 🙏❤️