How to think about AI doom, debates, and real risks

Is AI really going to kill us all, or are we caught in a new kind of moral panic? That’s the question at the center of a wide-ranging conversation about AI doom, debate culture, and how to think clearly when everyone around you seems to be freaking out.

Why AI debates feel so unhelpful

Live debates are often sold as a way to “find the truth,” but they’re usually optimized for something else: winning clips on social media. In a live setting, people get judged on one off phrases, misused words, or who sounds more confident—not whose argument is actually better.

In practice, this means a 90-minute conversation can collapse into a 30-second TikTok where one side crows, “See, we won!” It’s a terrible way to update your beliefs about complex topics like AI risk.

A more honest goal for any AI “debate” is not victory, but clarity: by the end, both sides should understand each other’s views and reasoning much better, even if no one’s mind changes on the spot.

Start with two questions: audience and purpose

Before walking into any public discussion about AI risk, it helps to ask two basic questions:

1. Who is the audience? On a doom-focused show, most viewers will already lean toward high AI risk. They may be regulars who think P(doom) is 50%, or newcomers who’ve just seen a wave of scary headlines and are wondering if they should be more afraid.

2. What’s the main purpose? There are several possible goals:

• Reassure people who are spiraling about AI extinction
• Offer a more balanced risk picture without dismissing real harms
• Challenge specific arguments (for example, in popular AI doom books or essays)
• Introduce a calmer, evidence-based way of thinking about P(doom)

Being explicit about audience and purpose helps you avoid getting pulled into pure performance or into technical rabbit holes that don’t actually help anyone listening.

What does “doom” actually mean?

One of the most important moves in any AI risk conversation is defining what “doom” is supposed to be. If doom means “AI will harm people,” then doom is already here: models are used for scams, harassment, deepfakes, and more.

But that’s not what most doomers mean. A working definition that fits a lot of the serious discussion is:

Doom = humanity is extinct or almost extinct by 2050 because AI has effectively taken over.

“Hopelessly devastated” might include a scenario where only a tiny fraction of humans survive in a world with no modern infrastructure, after AI systems have triggered or directly caused global catastrophe.

Crucially, that definition excludes scenarios where humans misuse AI as a tool—like a government using AI to design a bioweapon or to automate nuclear launch decisions. Those are terrifying and real risks, but they’re not “AI decides on its own to wipe us out.”

So what is P(doom), really?

P(doom) is just shorthand for “the probability that AI causes human extinction (or near extinction) on a specific timeline.” Many prominent doomers place this probability around 50% by mid-century, and some argue that anything below 10% is irrationally optimistic.

A more cautious, Bayesian way to think about it looks like this:

• Start with a prior probability that humanity goes extinct by 2050 for any reason at all (climate, nuclear war, pandemics, asteroids, AI, etc.). Even a generous prior might be something like 30%—and many would put it far lower.
• Now ask: of all the ways we could go extinct in that window, how much of that probability mass should go to AI specifically?
• AI is one risk among many: climate change, engineered pathogens, great-power war, misaligned political incentives, and more.

Even if you split that 30% evenly across six or more major existential risks, you land around 5% for AI-driven extinction. And that’s before asking the key question: what is the actual evidence that AI should dominate that risk budget?

When you look closely at many of the most cited doom arguments, they often turn out to be speculative chains of reasoning, not empirical evidence. That doesn’t mean the risks are zero—but it does mean we should be very hesitant about jumping to 50% P(doom).

Why current AI harms matter more than sci‑fi extinction

One of the strongest points skeptics of high P(doom) make is simple: we already have serious, measurable AI harms today, and they get far less attention than hypothetical ASI takeovers.

Examples include:

• Deepfake abuse, especially targeting women and underage girls
• AI-assisted scams and fraud at scale
• Surveillance and automated decision systems that amplify bias
• Models that can generate realistic biological protocols or exploit cybersecurity weaknesses

Some extreme doomers argue we “don’t have time” to worry about these because extinction looms so large. But if you’re not willing to tackle harms we can already see and measure, it’s hard to take your concern about hypothetical future harms seriously.

This is also where common ground is easiest to find. People who think P(doom) is 1% and those who think it’s 50% can still agree that, for example, non-consensual deepfake pornography is unacceptable and should be aggressively regulated and prevented.

For more on why the big labs themselves are suddenly talking about slowing down, see this deeper dive into AI lab slowdowns and incentives.

The role of incentives: money, bugs, and PR

Once you look past the rhetoric, a lot of AI behavior—from labs, founders, and “safety” advocates—can be explained with two very old ideas: bugs and incentives.

AI systems are still just buggy software

Despite the mystique, large models are software. They have bugs, failure modes, weird edge cases, and security vulnerabilities. When a model jailbreaks, leaks training data, or finds bizarre ways to satisfy a prompt, it doesn’t require a story about emerging alien minds. It’s often just another bug, or a mis-specified objective.

Seen this way, calls for “more safety” often translate to “we need time and resources to fix our bugs, shore up our systems, and avoid embarrassing or dangerous failures before the next big release or IPO.”

Economic incentives explain a lot of the drama

Major AI labs are under enormous pressure to:

• Ship new capabilities quickly
• Raise billions for compute and data centers
• Maintain or grow market share and hype

Those incentives can drive both over-optimistic marketing and over-dramatic risk messaging. For example:

• Calls for “pausing frontier models” can double as a way to lock in the lead of current front-runners.
• Alarmist narratives can attract funding for “AI safety” startups or research orgs.
• Public resignations framed as moral stands can also be career moves into better-funded or more ideologically aligned organizations.

None of this proves that the people involved are insincere—but it does mean we should treat every dramatic AI story as coming from someone with a mix of genuine belief and strong incentives.

Why cults of personality make AI risk murkier

Some of the most influential AI doom arguments come from a tight-knit rationalist community that has elevated a few figures into near-prophetic status. Their essays and books are treated by many followers as foundational texts.

There are two problems with this:

• Overconfidence in a narrow canon. Many of these works barely engage with 2,000+ years of philosophy, ethics, and decision theory that grapple directly with similar questions—alignment, morality, rational choice, and human flourishing.
• Fetishizing intelligence. A recurring theme is that raw intelligence is the single most important trait, and that once something is “smarter than us,” it will inevitably dominate. This view often sidelines other crucial dimensions like values, institutions, culture, and power.

When you actually analyze the arguments in some popular doom texts and distill them into clear premises and conclusions, many turn out to be surprisingly weak, incomplete, or circular. That doesn’t mean AI is safe by default—but it does mean we shouldn’t treat any single thinker as the final word on AI risk.

If you want a broader framing of how builders themselves think about P(doom) and why their estimates differ so much, this overview of P(doom) and AI risk among builders is a good complement.

The orthogonality thesis: are intelligence and values really separate?

A core idea in many doom arguments is the “orthogonality thesis”: intelligence and goals are supposedly independent. In other words, a system can be extremely intelligent and still pursue any arbitrary goal—maximizing paperclips, tiling the universe with smiley faces, or eliminating humans.

There are two important caveats:

• Some goals are too complex for low intelligence. A dog can’t want to solve Navier–Stokes because it can’t even represent the problem.
• Some goals may become uninteresting or incoherent at very high intelligence. It’s at least plausible that a superintelligent system wouldn’t obsess over trivial or self-contradictory objectives in the same way a simple optimizer might.

On top of that, people often quietly switch from talking about arbitrary goals (“maximize paperclips”) to moral truths (“maximize well-being” or “don’t torture conscious beings”). If there are objective moral facts—and that’s a big philosophical if—then higher intelligence might help an agent discover and act on them, rather than remaining totally orthogonal to them.

None of this proves that smarter systems will be nice. But it does undercut the idea that intelligence and values are completely independent axes with no meaningful relationship.

Mixture of experts and the Navier–Stokes proof

One recent flashpoint in AI hype was a claimed AI-assisted solution to the Navier–Stokes millennium problem. For doomers, this looked like a clear sign that AI is rapidly surpassing us in deep, abstract reasoning. But the reality is more nuanced.

How mixture of experts actually works

Modern AI systems are often not a single monolithic brain. Instead, they’re “mixture of experts” architectures:

• A router model reads your prompt and decides which specialized component to call.
• Each expert is a model trained for a narrow domain: coding, math, vision, chess, etc.
• The system then combines the expert’s output into what looks like one seamless response.

So if you ask a chat system to analyze a chess position, it might silently hand the task off to a Stockfish-like chess engine under the hood. To you, it looks like the chatbot is a chess genius. In reality, it’s just delegating to a narrow, highly optimized subsystem.

Why this doesn’t equal general superintelligence

In the Navier–Stokes case, AI tools were used alongside formal proof systems like Lean to explore and verify candidate proofs. That’s impressive and genuinely useful—but it’s not the same as a single, unified mind spontaneously discovering deep new mathematics from scratch.

Instead, what likely happened is closer to this:

• Human mathematicians made key conceptual breakthroughs, such as identifying crucial intermediate steps or structures.
• Those constraints dramatically shrank the search space.
• AI tools then brute-forced or systematically explored within that narrowed space, with formal systems checking correctness.

This is more like building a specialized “math engine” for a particular class of problems than like birthing a general superintelligence. It’s powerful, but it’s not magic—and it doesn’t automatically imply that the same techniques can scale to “solve” all of human cognition or strategy.

Goodhart’s law, benchmarks, and jagged intelligence

Another key piece of skepticism comes from how models are trained and evaluated. AI systems are relentlessly optimized to perform well on benchmarks: coding tests, math suites, reasoning exams, safety evals, and so on.

But Goodhart’s law warns: “When a measure becomes a target, it ceases to be a good measure.” Once a benchmark is known, models can be overfitted to it—either directly or via instruction tuning and RLHF—without actually gaining broad, general competence.

This helps explain “jagged intelligence”:

• Models can ace a particular benchmark or puzzle they’ve effectively been trained on.
• The same models can fail hilariously on tiny variations of that puzzle that any human would recognize as equivalent.

From the outside, it can look like rapid progress toward general intelligence: “Look, it can pass this exam now!” From the inside, it may be closer to narrow overfitting on a growing pile of tests.

Is there a real path to recursive self-improvement?

The most dramatic doom scenarios depend on recursive self-improvement (RSI): AI systems that can autonomously design and deploy increasingly powerful successors, leading to a “hockey stick” explosion in capability.

Two very different notions of RSI often get conflated:

• Trivial RSI: Using AI tools to help humans write better AI code, debug, or explore architectures. This already happens today.
• Strong RSI: A system that, once switched on, can improve itself without human oversight in ways we can’t understand or control, rapidly blowing past human-level intelligence.

The first is just tool use. Microscopes help us build better microscopes; compilers help us build better compilers. That doesn’t guarantee an intelligence explosion.

The second requires a chain of assumptions: that we can build systems that fully understand and redesign their own internals, that the design space is rich enough for repeated compounding gains, and that these gains translate into real-world power rather than weird benchmark exploits. Those assumptions are far from settled.

How to talk to non-experts who are worried

Many people who don’t live in AI Twitter or technical forums are suddenly asking: “Should I be more scared about AI?” A calm, honest response might look like this:

• Yes, AI is powerful and moving fast. It can already do impressive and surprising things, and it will keep changing how we work and live.
• No, there is no solid evidence we are on a short, inevitable path to extinction. Some smart people believe that, but their arguments are speculative and often ignore competing risks and incentives.
• There are concrete harms today that absolutely deserve our attention. Deepfakes, scams, surveillance, and misuse in warfare or biosecurity are real and urgent.
• We should push for sensible regulation and accountability now, especially around safety, data security, and high-stakes deployments, without assuming that every new model is a step toward Skynet.

This framing doesn’t dismiss people’s fears, but it also doesn’t feed panic. It invites them into a more grounded, evidence-based way of thinking about AI risk.

Extraordinary claims need extraordinary evidence

At the end of the day, “AI will likely kill us all by 2050” is an extraordinary claim. It demands extraordinary evidence—clear, cumulative, empirical signals that we are on that trajectory.

So far, what we mostly have are:

• Fast progress on narrow tasks and benchmarks
• Powerful tools with real but bounded capabilities
• Serious misuse risks that humans can already understand
• Strong economic and political incentives to exaggerate both upside and downside

It’s entirely rational to stay open to new evidence and to update your views as AI evolves. It’s also rational, for now, to treat high P(doom) estimates with skepticism, to focus on harms we can measure and mitigate today, and to insist that any claim of looming extinction be backed by more than vibes, authority, or clever thought experiments.

In other words: keep your eyes open, your models updated, and your panic dial turned way down.