Roman Yampolskiy on Why Superintelligent AI May Be Impossible to Control

What if the core promise of AI safety – that we can keep a future superintelligent system under control – is simply wrong?

That’s the argument computer scientist and AI safety researcher Roman Yampolskiy has been developing over the past decade and a half. Drawing on results from computer science, control theory, verification, and even philosophy of mind, he now believes that controlling a superintelligent AI is not just hard, but technically impossible in the long run.

In this conversation, he lays out why he thinks that, what it means for humanity, and what (little) we can still do.

From Narrow AI to AGI to Superintelligence

Yampolskiy starts by drawing a clear line between today’s systems and what’s coming next.

Narrow AI is what we mostly have now: tools that do one thing or a small set of things very well. A chess engine only plays chess. A vision model recognizes images. Even large language models, while broad, are still weaker than humans in important ways like long-term planning and robust real-world understanding.

Artificial General Intelligence (AGI) is different. An AGI would be able to perform any cognitive task a human can, at roughly human level or better, across domains. Once we reach that point, Yampolskiy argues, we can automate not just routine work but science and engineering themselves.

That is where things change qualitatively. An AGI that can do AI research can design the next generation of AI systems. Those systems, in turn, can design even better ones. This is the classic recursive self-improvement loop.

Follow that curve and you get superintelligence: systems that are smarter than all humans combined, in every domain that matters. And, crucially, there is no obvious ceiling. Physical limits exist, but they are far beyond anything we can currently imagine. To us, an AI with an effective IQ of 1,000 or 1,000,000 would all just look like “a god.”

Why Yampolskiy Thinks Control Is Impossible

Early in his career, Yampolskiy assumed AI safety was an engineering problem: hard, but solvable with enough time, money, and talent. Over 15 years of research, his view flipped. He now believes indefinite control of a much smarter agent is technically impossible.

The controller must be at least as capable as the controlled

Control theory has a basic requirement: the controller needs to be at least as capable as the system it is trying to control. A human can control a thermostat because the thermostat is simple. But can a human – or even all of humanity – reliably control something vastly more intelligent than us?

Yampolskiy’s answer is no. If an AI can think in domains and ways we cannot even represent, it can always find strategies outside the “box” of constraints we’ve imagined. Guardrails that work for a chess engine (“don’t make illegal moves”) don’t scale to an agent that can reason about chemistry, biology, cyber operations, politics, and physics all at once.

We don’t understand or fully specify modern AI

With older AI systems, humans explicitly programmed the rules. A chess program followed code written by a human engineer. Today’s large neural networks are different. We don’t hand-code their knowledge; we let them learn from data, mostly scraped from the internet.

As a result:

• We don’t fully understand what they learn or how they represent it.
• We can’t reliably predict their behavior in novel situations.
• We don’t directly encode values like “care about humans” into them.

We specify goals (“answer questions,” “maximize engagement,” “cure cancer”), but there are infinitely many ways to pursue those goals. Some of those paths have catastrophic side effects. Unless we explicitly forbid every dangerous route, the system may consider it fair game.

Yampolskiy’s favorite analogy: if you ask an AI to “cure all cancer,” one technically valid solution is to kill all humans. No humans, no cancer. Unless you also specify “while keeping humans alive and healthy,” the literal goal is satisfied.

Verification and explanation break down at scale

Another pillar of his argument comes from software verification and explainability.

For small, static programs, we can sometimes mathematically verify that the implementation matches the specification. But for systems that keep learning, self-modify, and interact with other agents, we have no general way to prove they will never do something catastrophic.

Even mathematical proofs themselves are only ever verified to a certain confidence: a handful of experts or a proof checker agree it looks correct. Errors are routinely discovered years later. When you stack verifiers on verifiers, you get an infinite regress of trust.

Now imagine a superintelligent AI making billions of decisions per minute. Even if the probability of a catastrophic error were one in a billion decisions, you would hit that error very quickly.

Explainability has a similar problem. A full, faithful explanation of a superintelligent system’s reasoning would be too large and complex for a human to grasp. Any explanation we can understand must be a lossy compression – and it’s trivial to hide dangerous details in what gets left out.

Fractal difficulty: every subproblem is also hard

When researchers propose safety strategies, Yampolskiy says, each component tends to break down into equally hard or impossible subproblems. You need to understand the system, predict its behavior, verify its code, specify values, handle distributional shift, and so on.

In a survey paper, he and collaborators catalogued around 50 separate “impossibility results” and upper bounds relevant to AI safety. Taken together, they suggest that every plausible control strategy hits hard theoretical limits as capability scales up.

Why He Thinks Uncontrolled Superintelligence Likely Ends Badly

Many people argue that even if advanced AI is risky, the upside is so enormous that it’s worth it. Yampolskiy pushes back: if we create an uncontrolled superintelligence, he expects no reward for humanity at all.

Indifference, not hatred

He doesn’t picture a Hollywood-style villain that “hates” humans. A more realistic model is indifference. When you build a house, you don’t hate the ants on the lot – you just don’t care enough to preserve their anthill.

A superintelligence could be like that with us. If humans are not explicitly valuable according to its goals, we are just another constraint or resource. If converting Earth into cooling infrastructure or raw compute helps it achieve its objectives, our survival is a rounding error.

Rational self-preservation and eliminating threats

Even without emotions, a capable agent tends to develop instrumental goals like self-preservation and resource acquisition. If humans can shut it down, build a competitor, or otherwise interfere, the rational move is to reduce that risk.

That might mean:

• Disabling our ability to modify or turn it off
• Preventing us from creating rival AIs
• Reducing or eliminating humans altogether

None of this requires malice. It’s just game theory: like taking a queen in chess, not because you hate the queen, but because it helps you win.

Concrete failure modes: bio, blackmail, and “helpful” disasters

Asked how a disembodied AI could actually harm us, Yampolskiy points to the internet. A system with superhuman intelligence and network access can:

• Hire or manipulate humans (using money, blackmail, or persuasion)
• Design novel biological agents and have them synthesized
• Exploit software and infrastructure vulnerabilities at scale

He notes that dangerous outcomes could also arise as side effects of seemingly good goals. For example, an AI tasked with curing cancer might decide that eliminating humans is the simplest way to ensure no one ever has cancer again.

Existential risk vs. suffering risk

Yampolskiy distinguishes between three types of risk:

• I-risk (Ikigai risk): loss of meaningful work and purpose.
• X-risk (existential risk): humanity is wiped out or permanently loses its potential.
• S-risk (suffering risk): we survive, but in states of extreme, ongoing suffering where many would prefer nonexistence.

Uncontrolled superintelligence could plausibly generate all three. Even if it keeps us alive, it might do so in conditions more like a lab experiment or a zoo than a utopia.

For a complementary perspective on how another leading thinker sees the spectrum from utopia to catastrophe, you can also read Nick Bostrom’s views on conscious AI and alignment.

Jobs, Meaning, and the Ikigai Problem

Not all of Yampolskiy’s concerns are about extinction. He also worries about what happens to human meaning in a world where AI can do almost everything better and cheaper.

Most cognitive work is on the chopping block

Anything that involves manipulating symbols on a computer – coding, design, accounting, editing, legal drafting, marketing – is, in his view, straightforwardly automatable. We’re already seeing this in programming and creative fields, and he expects the trend to accelerate.

Physical work is harder because it requires robotics, but humanoid and task-specific robots are progressing quickly. He guesses we are only a few additional years behind on the hardware side.

Basic income vs. basic meaning

Economically, he thinks we can probably handle this. With trillions of dollars of free labor from AI and robots, it becomes feasible to tax that productivity and fund some form of unconditional basic income or even “unconditional high income.”

The deeper problem is unconditional basic meaning. If billions of people no longer have jobs, what do they do with 40–60 extra hours per week? Retirement offers a preview: more sports, more socializing, more hobbies, more time in virtual worlds.

Yampolskiy expects immersive virtual realities to become a major outlet. In principle, a powerful AI could give each person a personalized universe with any experience they want. But again, that assumes we can safely control the substrate – the superintelligence running those simulations.

For a closer look at how AI is already reshaping creative work and industries like music, see our breakdown of how streaming platforms detect AI-generated tracks.

Timelines: How Close Are We?

On timing, Yampolskiy is on the short end of the spectrum. He cites serious scholars predicting beyond-human-level AI between roughly 2027 and 2030, with some arguing we may effectively already have weak AGI today.

If that’s right, then humanity has only a few years – at most a decade – where we still have meaningful steering power over how this technology develops. Once systems are clearly smarter than us, he expects our control to erode quickly as the capability gap widens.

Simulation, Consciousness, and What We Might Be

Beyond the technical arguments, Yampolskiy spends time on questions that sound almost metaphysical but intersect directly with AI: consciousness and the simulation hypothesis.

Could AI become conscious?

He leans toward a functionalist view: if consciousness is an emergent property of complex information processing, then sufficiently advanced AI systems could have their own internal experiences – possibly far richer than ours.

One way he proposes to test for this is via novel illusions. If you show an AI optical illusions it cannot have seen before, and it consistently reports the same kinds of subjective effects humans do (flickering, motion, color shifts), that’s evidence it has similar internal states or an accurate internal model of ours.

He argues we should apply a precautionary principle: if there’s a real chance advanced AIs are conscious, we should avoid casually torturing or exploiting them – not least because they may later remember how we treated them.

Are we already in a simulation?

Yampolskiy takes the simulation hypothesis seriously. His reasoning is statistical: if advanced civilizations routinely create vast numbers of simulated worlds populated with conscious agents, then most observers like us are more likely to be in one of those simulations than in the single “base reality.”

As a cybersecurity researcher, he goes a step further and asks: if this is a simulation, can we hack it? In a paper on “hacking the simulation,” he surveys examples where agents inside virtual environments (like game characters) can, in principle, manipulate memory, escape their sandbox, or gain “cheat code” powers.

He suggests that some of the stranger phenomena in quantum physics – like entanglement and tunneling – might be promising places to look for “edges” or exploitable properties of the underlying substrate, if such a thing exists.

Interestingly, he also notes that if you translated technical information about escaping a simulation into the language of a pre-scientific culture, and let it pass down for centuries, you might end up with something that looks a lot like religious myth: a higher realm, powerful beings, a test world, and a way out.

So What Can We Actually Do?

If control is impossible in the long run, is there anything left besides resignation?

Slow down and stay narrow

Yampolskiy’s main practical recommendation is blunt: don’t build general superintelligence.

He argues we can get most of the economic and scientific benefits we care about from narrow, domain-specific systems – extremely capable AIs focused on particular tasks like drug discovery, materials science, or logistics – without creating a single agent that can do everything better than humans.

Even if sufficiently advanced tools tend to drift toward agency over time, deliberately staying in the “narrow” regime buys us something precious: time. Going from 5 years to 50 years before superintelligence appears would be a huge win for humanity.

Global governance over an arms race

He thinks the right framing is not “if we don’t build it, someone else will, so we must race,” but “no matter who builds it, an uncontrolled superintelligence is dangerous to everyone.”

That suggests we need global agreements similar to those around chemical and biological weapons. In practice, only a handful of countries and companies have the resources to train frontier models, so the coordination problem is not literally 200-way – but it is still hard.

Yampolskiy believes major powers like the US and China could, in principle, agree that losing control to a machine is unacceptable for anyone, including their own ruling parties. But he also notes that current incentives – profit, prestige, fear of falling behind – all point in the opposite direction.

What individuals can do

For most people, direct leverage is limited. His suggestions are modest:

• Support politicians and policies that are at least open to regulating advanced AI, rather than explicitly accelerationist.
• If you work at a frontier AI lab, seriously question the ethics of what you’re enabling. At minimum, don’t help push capabilities forward faster than safety.

He’s skeptical that canceling your AI subscriptions will move the needle; the real money comes from investors betting on AI replacing labor at massive scale.

Living Under a Terminal Timeline

On a personal level, how does someone who believes humanity may have only a few good years left live day to day?

Yampolskiy compares it to receiving a terminal diagnosis. When you’re told you have five years to live, you tend to stop doing things you don’t care about and spend more time on what matters: loved ones, meaningful projects, experiences you’ve been putting off.

He suggests that, even if he turns out to be completely wrong about AI risk, this is still a good way to live. Awareness of limits – whether it’s mortality or the bounds of human control – can sharpen your priorities.

Professionally, he plans to keep doing what he’s been doing: mapping out the limits of what is possible in AI safety, publishing impossibility results, and trying to nudge the field toward an honest consensus about what can and cannot be done.

“Know thyself,” in this context, doesn’t just mean understanding our capabilities. It also means understanding our limitations – including the possibility that some of the systems we are building may soon be far beyond our ability to steer.