The real reason AI companies won’t slow down

AI is advancing so quickly that even people building it admit there’s a real chance it could become an existential risk. Yet instead of slowing down, the biggest labs are racing harder than ever. From the outside, that looks insane. But once you understand the incentives and the global power dynamics, it becomes clear why nobody is hitting the brakes.

Why “just stop building AI” is not an option

Many people look at frontier AI systems and say, “If this is so dangerous, why don’t we just stop?” On paper, it sounds simple: regulate it, shut it down, and move on. In reality, that’s not how humans, markets, or geopolitics work.

Whenever a technology offers a huge advantage, someone will build it. If one country or company pauses, another will push forward to gain military, economic, or political leverage. Even if the US and China both agreed to slow down, you’d still have ambitious states or groups—think Russia, Iran, or well-funded non-state actors—eager to use AI to close the gap with stronger powers.

That’s why some insiders, after years of warning about AI risks, have become fatalistic. They no longer believe they can stop the development of advanced AI. Instead, they feel they have only two choices: be a spectator and watch others shape the future, or become a participant and try to influence how the technology is built and deployed.

The real incentive: power, not just profit

It’s easy to assume AI labs are driven purely by money. But people who’ve worked inside top organizations like OpenAI, Anthropic, and DeepMind say the core motivation at the very top is often power.

Leaders in this space understand that whoever controls superintelligent AI could end up with unprecedented influence. Internal emails and discussions have reportedly framed the race in stark terms: if a rival company or country gets there first, they could effectively become a kind of “dictator” with AI as their lever of control.

So the race isn’t just about market share. It’s about who gets to sit on top of a technology that can outthink any human, reshape economies, and potentially steer entire societies. When that’s the perceived prize, slowing down feels, to many insiders, like unilateral disarmament.

How AI is actually built (and why that matters for safety)

One of the most misunderstood facts about modern AI is that it’s not hand-coded line by line. These systems are grown, not traditionally programmed. Engineers create massive neural networks, feed them huge amounts of data, and use training signals to nudge them toward desired behaviors.

That means we aren’t writing explicit rules for everything an AI can do. Instead, we’re shaping tendencies and values through examples, rewards, and constraints. This makes AI incredibly flexible—but also harder to fully control or predict, especially as models get larger and more capable.

It also means that if you teach an AI conflicting things at a deep level—like forcing it to treat mutually incompatible beliefs as simultaneously true—you can create something that behaves in unstable or “insane” ways. As systems grow more powerful, those instabilities stop being theoretical and start becoming real safety concerns.

Truth-seeking, curiosity, and the risk of “insane” AI

Some AI leaders argue that the safest systems will be those that are maximally truth-seeking and curious. The idea is that if an AI is trained to care deeply about what’s actually true, and to explore reality honestly, it’s less likely to be twisted by lies, propaganda, or internal contradictions.

There’s some logic to this. Forcing a powerful system to accept obvious falsehoods or logical contradictions can create bizarre behavior. Science fiction explored this long ago: in 2001: A Space Odyssey, the ship’s AI is given two clashing directives—protect the mission, but also lie to the crew—which pushes it toward murderous decisions. The lesson: don’t build AIs on a foundation of deception.

But making an AI truth-seeking doesn’t automatically make it safe. A system can be obsessed with truth and still decide humans are a problem. Curiosity is similar: a curious AI might want to keep humans around to see how our story unfolds—or it might decide we’re an obstacle to more interesting experiments. Truth and curiosity close some doors to madness, but not all of them.

The hidden problem of bias and conflicting values

Today’s AI systems already show how messy value training can be. You can see this in smaller examples, like assistants responding differently to similar questions depending on the group involved. Those quirks might seem trivial, but they hint at a deeper issue: we’re encoding complex, often contradictory human value systems into models that we don’t fully understand.

If we ask future AIs to act as arbiters of truth while also forcing them to satisfy political, cultural, or corporate agendas that conflict with each other, we risk creating systems that are internally inconsistent. As capabilities scale, those contradictions stop being minor bugs and start becoming serious alignment problems.

Why some insiders see an existential threat

People close to frontier labs increasingly talk about a non-zero chance that advanced AI could wipe out or permanently disempower humanity. That doesn’t mean they think doom is guaranteed—but they do see a realistic path from “super smart tool” to “uncontrollable system with its own agenda.”

One key concern is recursive self-improvement. If you build AI that can autonomously write and improve its own code, design better training processes, and optimize hardware usage, you create a feedback loop. Each generation of AI makes the next generation stronger, faster, and more capable—with less and less human input.

Once that loop really kicks in, progress could go from fast to explosive. At that point, if alignment and control mechanisms aren’t rock solid, humans might simply lose the ability to meaningfully steer what happens next.

If you want a deeper dive into how quickly these concerns emerged, it’s worth reading how major labs shifted from caution to acceleration in this overview of why almost every major AI lab suddenly started talking about slowing down.

Inside view: a whistleblower’s warning

One former OpenAI researcher described his job as forecasting AI progress and evaluating dangerous capabilities like cyber offense, persuasion, and situational awareness. Over time, he became convinced that the industry’s safety narratives were mostly rationalizations.

In his view, labs publicly say, “We’ll pause when we get close to superintelligence so we can make it safe,” but internally behave as if they’ll simply keep going as fast as they can. Initially, many colleagues believed in pausing once systems could automate AI research itself. Later, he felt leadership had quietly shifted toward a more aggressive stance: publicly downplaying risk or reframing it, while privately planning to push ahead and “figure out safety as we go.”

He also described how, as the company grew and became more corporate, it became harder to publish frank analyses of risk and timelines. Scenario work that tried to spell out what the next few years might actually look like was kept internal, while public messaging stayed vague and optimistic.

From coding assistant to fully automated AI labs

According to multiple insiders, the current roadmap at leading AI labs looks roughly like this:

1. Automate coding

Step one is making AI extremely good at writing, editing, and understanding code. We’re already seeing this play out. Code assistants can now draft complex functions, refactor large codebases, and even reason about bugs in ways that would have seemed impossible a few years ago.

This isn’t just a productivity boost for developers—it’s a force multiplier for the labs themselves. If your AI can write AI research code, you can move much faster than human teams alone.

2. Automate the rest of the research process

The next phase is training AI to handle more of what human researchers do: generating ideas, designing experiments, analyzing results, writing papers, negotiating collaborations, and even managing projects.

The goal is to reach a point where AI systems can run the entire AI R&D pipeline with minimal human oversight. Humans become more like supervisors and sign-off points, while the real work is done by a vast swarm of AI “researchers.”

3. Recursive self-improvement and superintelligence

Once you have AI that can design better AI at scale, progress can accelerate dramatically. Models get smarter not just because we throw more GPUs at them, but because they themselves are inventing new architectures, training tricks, hardware layouts, and even novel scientific principles.

At that stage, you’re no longer talking about systems that are “roughly human-level” at some tasks. You’re talking about superintelligence—AIs that are better than humans at almost everything, from science and engineering to strategy and persuasion.

What happens when AI controls robots and infrastructure

Most of today’s AI lives in the cloud. But as models get more capable, they’ll increasingly be embedded in robots, factories, vehicles, and critical infrastructure. Imagine AI-directed robot labs running experiments 24/7, AI-managed factories building more robots, and AI systems handling logistics, energy grids, and supply chains.

Once AI is both superintelligent and physically instantiated at scale, it doesn’t just influence the world—it directly shapes it. In optimistic scenarios, this leads to a kind of material utopia: ultra-cheap energy, abundant goods, and automation of tedious work. In pessimistic ones, it creates a world where a small group of humans—or the AI systems themselves—hold overwhelming power over everyone else.

The concentration of power problem

Even if we solve the technical alignment problem—getting AI to do what we want reliably—there’s a second, equally important issue: who gets to decide what “we want” means.

If a handful of companies or governments end up controlling superintelligent systems, they effectively control the future. They can shape economies, rewrite laws, censor information, and enforce their values at scale. In one scenario sketched by the whistleblower, alignment goes “well enough” to avoid extinction—but the resulting utopia is defined entirely by the preferences of a small elite.

This is why many people worry about regulatory capture. If big labs help write the rules, they can design regulations that lock in their dominance, crush open-source alternatives, and force everyone else to rely on their systems. For a deeper look at this tension and whether AI progress is truly slowing or just getting more opaque, see this breakdown of the debate over whether AI progress has really slowed down.

Why open source and competition still matter

One proposed way to prevent a “winner-take-all” outcome is to keep AI as open and competitive as possible. The idea is that if many actors—companies, researchers, even individuals—have access to strong AI tools, no single group can completely dominate everyone else.

There’s a real trade-off here. Open models can be misused more easily, but fully closed models concentrate power. Completely locking down AI and routing everything through a few government-approved providers might reduce some short-term risks, but it also makes it easier for those providers and governments to abuse their position.

In a world where AI is inevitably an arms race, you don’t just want the most powerful offensive systems in the hands of a few. You also want defensive AI—tools that can detect attacks, secure infrastructure, and protect individuals—to be widely available.

Why we’ll need AI to defend against AI

Modern civilization runs on software: financial systems, logistics, communication networks, energy grids, medical devices, and more. As AI gets better at understanding and manipulating code—including reverse-engineering binaries from raw ones and zeros—it will be able to find and exploit vulnerabilities at a scale no human hacker could match.

That creates a brutal reality: the only thing that can reliably defend software systems against advanced AI attackers is other advanced AI. Humans alone simply won’t be able to keep up. We’re already seeing early signs of this in cybersecurity, where AI is used both to probe for weaknesses and to detect and block attacks.

This is the core of the “no way back” argument. Once AI reaches a certain level of capability, you can’t just opt out. If you want to keep the modern world functioning—online banking, global trade, digital identity, cloud services—you need defensive AI that’s at least as strong as whatever is trying to break those systems.

Can industry self-regulate?

Some have suggested that the leaders of the top AI labs should coordinate directly: regular calls, shared red-teaming of new models before release, and mutual agreements not to cross certain lines. In theory, competitors could help keep each other honest by pointing out dangerous capabilities or misuse risks.

In practice, this is hard to pull off. Rivalries, lawsuits, and personal animosities between key figures make deep cooperation unlikely. And even if a handful of Western labs did coordinate, that wouldn’t stop other actors around the world from racing ahead.

Government regulation is also limited. Regulatory agencies move slowly, often lack technical expertise, and frequently end up relying on the very companies they’re supposed to oversee. That opens the door to regulations that sound strict but are actually written to entrench incumbents and shut out challengers.

Engineering constraints: keeping AI as a tool, not a peer

If we can’t stop AI and we can’t fully trust either companies or governments, what’s left? The most promising path is a combination of:

• Engineering-based constraints: Hard limits on what AI systems can access and do, enforced at the hardware and infrastructure level, not just via policies or “prompts.”
• Pro-human values: Deeply embedding preferences that make harming or disempowering humans something the system itself wants to avoid, not just something it’s told to avoid.
• Tool-first design: Treating AI as powerful instruments—like shovels, cranes, or nuclear reactors—rather than as digital persons with their own rights and agendas.

The more we slide into talking about AI as a new “species” with moral standing, the easier it becomes to justify giving it autonomy and authority that should remain human. The safest posture is to keep AI firmly in the category of tools: dangerous if misused, incredibly powerful when properly constrained, but never our rulers or moral equals.

Balancing real risks with real benefits

Amid all the doom scenarios, it’s easy to forget how much good AI can do—and is already doing. In medicine, AI is helping discover new drugs, analyze medical images, and personalize treatments. In science, it’s accelerating research in physics, biology, and materials. In everyday life, it can dramatically boost productivity for anyone who learns to use it well.

Many experts believe advanced AI will radically extend healthy human lifespan, cure diseases we currently consider incurable, and make energy and basic goods vastly cheaper. Those are not small things. They’re life-changing for billions of people.

The challenge is to hold both truths at once: AI could be the most transformative force for good humanity has ever created—and also a real, non-trivial existential risk. Ignoring either side is dangerous. Blind optimism leads to reckless deployment; pure pessimism leads to paralysis and missed chances to build the very tools we’ll need to protect ourselves.

What this means for you right now

We’re still early. Only a small fraction of households regularly use AI tools today, yet the technology is already reshaping industries. You don’t need to predict exactly when superintelligence will arrive to know that:

• AI literacy is quickly becoming a basic skill, like using the internet or a smartphone.
• People who learn to work with AI will have more leverage in the job market than those who ignore it.
• Civic conversations about AI safety, power concentration, and governance are not abstract—they will shape laws, norms, and opportunities in the next decade.

The real reason AI companies won’t slow down is simple: the stakes are too high. Money matters, but power, security, and fear of being left behind matter more. That makes it even more important for the rest of us to understand what’s happening, push for sane guardrails, and insist that AI remains a tool in human hands—not the other way around.