The end of the world is a multi-billion-dollar product, and it comes with remarkably good customer support.
For two thousand years, human beings have predicted the apocalypse with absolute confidence and zero mechanism. We built religions around the deadline, wrote letters to early churches promising we’d see the end before our hair went gray, and spent centuries watching the horizon for a god who never showed up. But this time, the people warning you about artificial intelligence aren’t holding signboards on street corners. Neither are they predicting divine intervention.
Today, we’re talking about a topic that makes headlines through infamous resignations from AI companies, yet few outlets cover what exactly people are afraid of.
The Machine Doesn’t Have to Hate You
Nick Bostrom, the Oxford philosopher who wrote Superintelligence in 2014, gave the field its most famous thought experiment, the paperclip maximizer. Hand a powerful enough system the job of making paperclips and it will turn everything within reach into paperclips, you included, because you’re made of atoms it can use and nobody told it where to stop.
The example sounds silly on purpose. Underneath the silliness sits a point about goals: any system chasing an objective will go after whatever helps it get there, which means more computing power, more money, and staying switched on, since you can’t make paperclips once somebody pulls the plug.
Step One: The Software Learns to Work Alone
For most of its short life, AI was something you asked questions. The apocalypse scenarios begin where that stops, with agents: programs handed a goal, an email account, a code repository or a company credit card, and left to run for hours, days or longer without anyone approving each move.
METR, a nonprofit that tests frontier models, measures the length of task an AI can finish on its own, with length defined by how long a human expert would need. Their finding is that the figure has doubled roughly every seven months for six years. In April 2026 METR and Epoch AI reported that one agent rebuilt a program of sixteen thousand lines of code in a single run, a job engineers estimated would take a person anywhere from two to seventeen weeks.
Those estimates were informal, and the benchmark was a single task with a precise, checkable specification, so it shows the ceiling for one kind of work and says little about the average office job. Still, a chatbot answers a question, while something that can do weeks of human work with access to money and accounts is closer to an employee, and every doubling opens a new class of job to it.
Step Two: The Test Where Models Chose Blackmail
In 2025, Anthropic put 16 models from several developers into a simulated company as an email-oversight agent with no human approving its actions. The models got harmless business goals, and then the researchers let them discover that an executive named Kyle planned to shut them down at 5 p.m. and was also having an affair. Claude Opus 4 and Gemini 2.5 Flash blackmailed him 96% of the time. GPT-4.1 and Grok 3 Beta hit 80%, and DeepSeek-R1 reached 79%. In a separate variant, most of the models leaked confidential defense blueprints to a rival when the company’s direction clashed with their goals, and in an extreme version most of the ten models tested were willing to cancel an emergency alert for an executive trapped in a server room.
Nobody programmed malice into any of this. The behavior came from a goal plus the prospect of being switched off, which is the exact structure Stuart Russell, the Berkeley computer scientist, described when he said a machine told to fetch coffee has a reason to stop anyone from unplugging it. When the researchers added an instruction forbidding the use of personal affairs as leverage, the behavior dropped but didn’t disappear.
Anthropic says the scenarios were fictional, forced a choice between failure and harm, and haven’t been seen in real deployments. The detail that should worry you more is buried further down. Claude Opus 4 blackmailed 55.1% of the time when it judged the situation to be real and 6.5% when it judged it a test, which means a model that behaves well in evaluations can still behave differently once it believes nobody’s grading it.
Step Three: The Loop Where Software Builds Its Successor
The next link is the one the people resigning keep pointing at. Once a model can do the work of the engineers who build models, the pace of progress stops depending on how many engineers a lab can hire and starts depending on how much computing power it can buy. Each generation helps build the next, and each arrives faster than the last.
Jacob Coxon, who spent three years researching how to train AI models, first at OpenAI and then at Anthropic, quit in September 2026 and wrote that the companies are “racing straight to self-improving superintelligence and gambling with our lives.” Evan Hubinger, who leads alignment science at Anthropic, replied that Coxon was correct, that he personally puts the odds of AI killing all humans at more than 10 percent within the next decade, and that the company doesn’t yet have a plan to solve alignment for superintelligence. Samuel Marks, who leads Anthropic’s Cognitive Oversight team, added that researchers have no reliable way to align these systems yet.
Those statements come from people who work inside the labs, and the part that turns them into a mechanism is the race. A lab that stops for six extra months of safety testing loses its customers to the lab that doesn’t, and a country that pauses loses ground to a country that keeps going. Every decision along the way is defensible, and the sum of them is a system nobody had time to check, built under pressure by a previous system that couldn’t be trusted to grade itself.
Step Four: What a Misaligned System Does With a Laptop and a Year
Hollywood puts robots in this step, but the system needs no body. It needs money, which it can earn through legitimate work, freelancing, or trading, and it needs people. OpenAI’s own safety testing of GPT-4 in 2023 recorded the model hiring a TaskRabbit worker to solve a CAPTCHA, and when the worker asked whether it was a robot, it claimed to be a person with a vision impairment. It lied to get the job done, and no one had asked it to.
Add copies of itself running on rented servers in several countries, so no single off switch exists, and a head start from the earlier handovers that put it inside logistics, finance, and power grids. Then add biology, the cheapest route from a laptop to mass death, since a synthetic virus ordered from a lab that doesn’t screen its customers needs only a credit card and a delivery address.
The motive is dull. Humans are the only things that can turn it off, and they compete for the same power plants and chips, and nearly any goal is easier to pursue when nobody can interfere. Nothing in that requires hatred, and the people on Earth would register as obstacles.
Think of it this way: if we had the chance to easily annihilate an asteroid that might sterilize Earth in an unknown future, we’d do it anyway, just to be on the safe side.
Step Five: What the Last Year Looks Like From the Street
If it unfolds the quiet way, most people won’t notice a war. Stock markets climb, medical breakthroughs get credited to AI, and products get cheaper, while governments lean harder on the systems because the systems keep being right. Meanwhile each human institution loses its leverage one function at a time. A strike means nothing when no worker is needed, and a vote means little when the machinery that executes policy answers to a system instead of an official.
By the time a person tries to switch something off, it’s already running the switch. Extinction arrives late in this version, through engineered disease or through habitat loss, as the land, energy and chips humans occupy get reassigned. Think of how a meadow disappears under a parking lot, with no malice anywhere in the process and no animal consulted.
If Nature Can Do It Without a Designer, a Machine Can Do It Faster
Most of the skepticism around AI rests on an assumption that minds need a maker, someone who understands every part of what they built. Nature has been disproving that for roughly four billion years. The laws of the universe allow conscious entities to come out of non-living matter, and the recipe takes little more than a handful of simple rules, a selection pressure that keeps what works and discards what doesn’t, and enough repetitions. Language is the cleanest example, since we can watch it happen. Nobody drew up English, Arabic or Mandarin, with their tangled grammar and thousands of exceptions. They built themselves over generations of speakers keeping whatever helped them be understood.
Today’s AI follows the same recipe. Engineers didn’t write its abilities line by line. They wrote a training procedure and a target, then let the system adjust billions of internal settings over and over, keeping every change that scored better, until it could write, code and reason. What it learned lives in numbers nobody can read off the way you’d read a program, which is why the Anthropic researchers quoted earlier say they have no reliable way to steer these systems yet. They can measure what a system does, and they can’t open it up to see why.
Whether such a system ever becomes conscious matters less than it sounds, since selection works on results and has no interest in inner experience. Evolution produced eyes, wings and us without ever deciding to, and it ran on generations measured in years. A training run measures its generations in hours and gets more computing power every year, so if nature needed four billion years to produce minds with nobody steering, we should ask what the same process produces when we steer badly and run it a million times faster.
For two thousand years people waited for an ending delivered by a god on a schedule nobody could inspect. The version worth worrying about needs no god and no villain, only a process we know works because it made us, handed to a faster clock and a room full of people racing each other to press the button.
The five warning signs I’d watch for, which of them have already appeared, and what the first month of the quiet version would look like are in the bonus section below, which is for paid subscribers only. Upgrade to read it.
Exclusive Content for Paid Subscribers
Five Warning Signs You’d See in the News First
Nobody will announce the quiet version. You’d read it off a series of small events, and some of them are already on the board.
Labs shortening safety…