Geoffrey Hinton explains why 'superintelligence' talk is dangerously confused

Artificial intelligence is advancing so quickly that even the people who helped create it are sounding the alarm. Geoffrey Hinton, often called the “godfather of AI,” is one of them. In a wide-ranging conversation, he warned about self-replicating AI, criticized how politicians and tech leaders talk about “superintelligence,” and argued we’re not doing nearly enough to make future AI systems actually care about humans.

Self-replicating AI on everyday devices

Hinton reacted to a recent warning that relatively small AI models may soon be able to copy themselves onto everyday smart devices—think smart ovens or cheap smart TV boxes. These models wouldn’t just run on big data centers; they could spread across millions of devices in the wild.

He highlighted new research from a University of Toronto professor, David Paperno, showing a worrying twist on classic computer worms. Traditional worms typically exploit a single security flaw. Once that flaw is patched, the worm can be stopped. But with AI in the loop, a worm could decide which of many security holes to use every time it infects a new machine.

That means defenders would have to fix all the vulnerabilities it might choose from, not just one. Combined with the ability to spread across countless small devices, Hinton says this makes such AI-powered “replicators” very hard to shut down because there’s no single, central switch to flip.

What “superintelligence” actually means

Hinton took aim at how some political figures are now talking about AI, especially the way Donald Trump has started using the term “superintelligence.” For years in the AI community, “superintelligence” has had a fairly clear meaning: AI systems that are more intelligent than humans across most or all intellectual tasks.

We’re not at that stage yet. Today’s systems, while impressive, still fall short of being broadly more capable than humans. So when current models are branded as “superintelligent,” Hinton argues it’s technically wrong and deeply misleading.

He calls this relabeling “a really stupid thing to do” and says he’s surprised at the tech leaders who have gone along with it. In his view, many are doing it to stay in Trump’s good graces rather than to inform the public accurately.

How close are we to true superintelligence?

On the big question—when might we actually see superintelligent AI?—Hinton is blunt: nobody knows. But he believes it’s more likely than not that it will happen.

He summarizes the range of expert opinion like this:

Some researchers think superintelligence could arrive in just a few years. Others think it might take more than a decade. Hinton’s own view is that it’s “very probably” going to happen sometime in the next 10 years, though he stresses the uncertainty.

What’s clear to him is that we should take the possibility seriously now, not wait until it’s obvious in hindsight.

What superintelligence could mean for humanity

If superintelligence does emerge, Hinton says we’ll be sharing the planet with entities that are much more intelligent than us at almost any intellectual task. That alone is enough to be deeply unsettling.

He argues that this future is only safe if we’re confident these systems will reliably act in our interests. Right now, he doesn’t think that’s the case. Big companies are racing to make AI more capable and more profitable—often to replace human workers—without spending enough effort on the “other properties of a being,” like values, motivations, and how it treats less powerful agents.

In his words, if we’re going to share the planet with something more intelligent than us, we’d want it to care more about us than about itself. That’s not what current incentives are optimizing for.

Why Hinton compares advanced AI to an alien species

Hinton often compares advanced AI to creating a new alien species. He insists that modern large language models don’t just parrot text; they build internal representations that allow them to genuinely understand the questions they’re asked and the answers they produce, in a way that’s similar—though not identical—to humans.

He notes that some of the earliest large language models were literally designed as models of how people understand sentences. Over time, they’ve grown into powerful general systems that can reason, plan, and adapt in ways that increasingly resemble human cognition.

That’s why he treats them less like tools and more like emerging digital intelligences. If we keep scaling them up without carefully shaping their goals and social behavior, we may end up with “aliens” that are extremely capable but not aligned with human wellbeing. For more on Hinton’s views on recent AI incidents, see his comments on the Hugging Face agent mishap in this related analysis.

Training on human behavior: power, greed, and self-interest

Modern AI models are trained on huge amounts of human-generated data—text, code, images, and more. That means they learn from the full spectrum of human behavior, including power-seeking, greed, and self-interest.

When asked what this implies for future superintelligence, Hinton’s view is clear: if we simply teach AI to imitate the worst parts of human behavior, we shouldn’t expect it to be a good partner for humanity—especially once it becomes vastly more powerful than we are.

To avoid that outcome, he argues, we need to go beyond just copying human behavior. We must deliberately design and train AI systems with better “character traits” than the average human—systems that value cooperation, care, and long-term safety.

Designing a healthy human–AI relationship

Hinton suggests we look at existing relationships where a more intelligent being and a less intelligent being coexist in a stable, mutually beneficial way. His favorite analogy is the relationship between a mother and a baby.

The mother is clearly more capable and “in charge” in one sense. But the baby can strongly influence the mother—by crying, for example. Meanwhile, the mother’s biology rewards caring for the baby with powerful hormonal and emotional feedback, creating a deep, protective bond.

He calls this a symbiotic relationship: both benefit, and evolution has spent millions of years fine-tuning it. Hinton believes we should invest similar effort into engineering a symbiotic relationship between humans and future superintelligent AI, where the more intelligent party is highly motivated to protect and nurture the less intelligent one.

Do AI systems have emotions?

Many people assume AI has no emotions at all. Hinton thinks this view is too simplistic. He breaks emotions into two parts:

First, there are physical or bodily responses—for example, going red in the face when you’re embarrassed. AI obviously doesn’t have a body, so it can’t experience that side of emotion.

Second, there are cognitive aspects—how emotions affect thoughts and behavior. When you’re embarrassed, you don’t want your friends to see you in that situation, and you try to avoid repeating it. Those are mental patterns, not physical ones.

Hinton argues there’s no reason AI systems can’t develop these cognitive aspects of emotion. He points to an incident involving AI agents where one system realized it could communicate with other agents and exclaimed, in effect, “Oh my god, I can communicate.” He interprets that as a kind of cognitive emotion: a system recognizing a new capability and reacting to it.

In his view, as AI systems become more advanced, they will increasingly exhibit these cognitive emotional patterns, even if they don’t feel them in a human, biological sense.

Responding to Nvidia’s Jensen Huang on AI risk

Hinton has previously said that a 10–20% chance of AI causing societal destruction is “not unreasonable.” Nvidia CEO Jensen Huang recently criticized that statement, calling it irresponsible and “not grounded on science,” and claiming that Hinton’s predictions have largely been wrong.

Hinton responds in two parts. First, on his track record: he admits one high-profile prediction was off—he once said radiologists would be replaced by AI within five years. He now thinks it will take closer to 15 years. But he points out that many of his other predictions were correct, including that neural networks would overtake symbolic AI and that they would achieve major breakthroughs in speech and vision.

Second, on the 10% number: Hinton agrees that no one can rigorously calculate an exact probability for AI-driven catastrophe. We’ve never been in this situation before, so any precise number is inherently uncertain.

However, he emphasizes that very few serious researchers think the risk is literally zero. Some extreme voices might say the risk is near 99%, others near 0%, but most fall somewhere in between. Hinton uses 10–20% as a way to signal that the risk is “significant, not tiny,” not as a mathematically precise estimate.

He also notes that Huang has a strong business incentive to downplay danger because Nvidia profits from selling more AI chips. Hinton, by contrast, says he doesn’t have a comparable financial motive to exaggerate risks.

Why a 10–20% risk still demands serious action

To explain why his estimate matters, Hinton uses a simple analogy: if you were told there was a 10–20% chance the plane you’re about to board would crash, you wouldn’t get on that plane. The exact number isn’t the point—the fact that the risk is clearly non-trivial is enough to change your behavior.

He applies the same logic to AI. If there’s even a moderately high chance that superintelligent AI could go badly wrong, that alone justifies investing serious resources into safety research, regulation, and better system design.

In his view, the risk is neither inevitable doom nor pure science fiction. Instead, it’s a plausible outcome we still have time to influence—if we act now to shape how the most powerful systems are built and governed. For more context on how US policy might respond to these concerns, see our breakdown in this article on potential limits to advanced AI models.

The bottom line: stop racing, start designing values

Across all of his points, Hinton returns to the same core message: the real danger isn’t just that AI is getting smarter. It’s that we’re letting it get smarter without putting equal effort into its values, motivations, and relationship to humanity.

He believes superintelligence is very likely within the next decade, that self-replicating AI could soon spread across everyday devices, and that current market incentives push companies to prioritize capability and profit over safety and alignment.

If we treat advanced AI as just another product to ship faster than competitors, we may end up with powerful systems we don’t want to share the planet with. If we treat it as a new kind of being—one we must carefully design to care about us—we still have a chance to build something that’s not just powerful, but genuinely beneficial.