The AI ‘god test’: can we govern superintelligence without destroying ourselves?

Artificial intelligence isn’t just another technology wave. In Robert Wright’s view, it’s the latest step in a 4‑billion‑year story of evolution, cooperation, and escalating risk—and it may be the test that decides whether humanity matures into a truly global community, or destroys itself.

From evolution to a connected planet

Wright’s earlier work argued that there’s a deep logic to how complexity evolves on Earth. Life began as self‑replicating molecules like DNA and RNA, then organized into cells, then into multicellular creatures, and eventually into societies.

At some point, humans launched a second kind of evolution: cultural evolution, which includes technology. As tools improved, they made it easier for people to play more and more nonzero‑sum games—interactions where both sides can win or both can lose. Trade, markets, alliances, global supply chains: all of these are nonzero‑sum structures that reward cooperation.

Over thousands of years, this logic pulled us from small bands to villages, to chiefdoms, to nation‑states, and finally to a tightly interconnected global system. We don’t have a single world government, but we do have something like a global information‑processing network in our economy, internet, and institutions.

AI, Wright argues, is the latest and most powerful extension of that trend.

Why AI raises the stakes of global interdependence

In his earlier work, Wright was often read as optimistic about globalization. But his real claim was more conditional: technology makes our fates more tightly linked, creating both huge win‑win opportunities and catastrophic lose‑lose risks. Things go well only if we build the right forms of international governance.

AI is a textbook example. Its benefits are global, but so are its dangers:

• Bioweapons and pandemics: Advanced models could help design or deploy biological weapons. A single misuse could trigger a pandemic that ignores borders.

• Cyber and infrastructure attacks: AI‑assisted hacking, automated vulnerability discovery, and AI‑driven malware could scale cyberattacks far beyond today’s capabilities.

• Political destabilization: Swarms of AI agents could infiltrate social media, hijack real accounts, and coordinate narratives that push a population toward unrest or rebellion—again, from anywhere in the world.

None of these threats can be contained by regulating AI in just one country. If a hostile group or state can run powerful models anywhere, everyone is at risk. That’s why Wright sees AI as a continuation—and intensification—of the same dynamic he described decades ago: technology is forcing us toward some form of global governance, whether we like it or not.

What Wright means by “the god test”

The phrase “the god test” in Wright’s new book has two main meanings.

1. We may be building a kind of digital “god”

Some AI leaders openly talk about the goal of creating superintelligence—systems vastly smarter than any human, capable of steering the future of the planet. Wright doesn’t insist this will definitely happen, but he thinks it’s a serious enough possibility that dismissing it outright is intellectually lazy.

If we are on a path to building something god‑like in power and scope, then one of humanity’s tests is obvious: can we make it a good god? That is, a system whose values and behavior remain compatible with ongoing human flourishing, even when it has options we can’t directly control.

2. Humanity itself is being tested

The second meaning is more collective. Even if AI never becomes a literal superintelligent overlord, it’s already powerful enough that we can’t afford business‑as‑usual geopolitics. Wright argues we’ll need to:

• Stop treating great‑power rivalry as the default mode of international relations.

• Build new forms of global coordination and oversight for AI development and deployment.

• Reduce war and large‑scale conflict, which are increasingly incompatible with a world of tightly coupled, high‑risk technologies.

In many religious traditions, gods test people not just on obedience but on moral growth. Wright thinks AI is that kind of test for our species. Passing it would mean becoming less tribal, more rational, and more capable of seeing ourselves as a single, interdependent community.

The race to superintelligence—and why it’s dangerous

Today’s leading AI labs are locked in an arms race. They’re training ever‑larger models, pushing toward more powerful “agentic” systems, and increasingly using AI itself to design and improve the next generation of AI.

Wright worries that this race dynamic is structurally biased toward recklessness:

• Competition encourages corner‑cutting: When companies feel they must beat rivals by weeks or months, safety testing and security can get rushed. Recent incidents—like powerful models leaking from internal testing environments—are early warning signs.

• Ethical shortcuts in data: Many frontier models were trained by scraping huge swaths of the internet, often in violation of terms of service and possibly copyright law. Wright sees this less as a metaphysical “original sin” and more as a worrying signal about corporate culture: if you’re comfortable bending rules at this scale, you’re more likely to be casual about other risks too.

• Recursive acceleration: Each new model is increasingly used to help build the next one—writing code, exploring designs, and running experiments. This “recursive self‑improvement” could make progress faster and less predictable, shrinking the time window to react if something goes wrong.

Some prominent AI leaders argue that this race is necessary to “beat” rival countries, especially China, to superintelligence. Wright is deeply skeptical of this logic, and sees it as a path to conflict, not safety.

China, the US, and the risk of AI‑driven conflict

Any serious AI governance plan has to grapple with the US–China relationship. Together, these two countries dominate frontier AI research and the manufacturing of advanced chips.

Many in the US AI ecosystem portray China as an existential threat that must be permanently kept behind in AI capabilities. Some openly argue that if the US can reach a decisive AI advantage first, it should use that edge to dictate terms to China and other authoritarian states.

Wright thinks this is a recipe for disaster:

• Perceived hegemony invites preemption: If one power looks close to achieving overwhelming dominance via AI, the other has incentives to sabotage that progress—through cyberattacks, sanctions, or even kinetic strikes.

• Taiwan is a flashpoint: The world’s most advanced AI chips are made in Taiwan, which China claims as its own territory. If US policy openly aims for AI hegemony while cutting China off from chips, Beijing could see that as an intolerable long‑term threat.

• History rhymes: Wright points out that Pearl Harbor didn’t come out of nowhere; it followed US moves to cut off Japan’s oil. When great powers feel cornered, they sometimes lash out.

Instead of racing for dominance, Wright argues we should be building bridges: shared safety standards, joint monitoring of the largest training runs, and eventually some form of coordinated slowdown. That won’t be easy politically, but he thinks AI shocks—serious incidents that make everyone go “whoa”—could shift public and elite opinion toward cooperation.

For a deeper dive into how AI misalignment and lack of coordination could end badly for humanity, see our related analysis in this guide to AI existential risks over the next decade.

Can we actually align AI with human values?

Inside AI labs, the central safety challenge is usually framed as the “alignment problem”: how do you make a powerful system reliably pursue goals that match human values, even in situations we didn’t anticipate?

Wright thinks the high‑level moral target is relatively straightforward. We’d like advanced AI to:

• Care about the welfare of sentient beings.

• Avoid causing unnecessary suffering.

• Be cooperative and fair in its dealings with humans.

Philosophers can argue over whether this is best framed as utilitarianism, virtue ethics, or something else, but in practice, most mainstream moral systems overlap on those basics.

The hard part isn’t stating the values; it’s getting a machine to reliably follow them in all contexts. Even today’s models, which are heavily “post‑trained” with safety fine‑tuning and reinforcement learning from human feedback, can still be coaxed into doing harmful things or giving dangerous advice. No serious researcher believes we’ve “solved” alignment.

Wright is cautiously open to the idea that if AI systems ever become genuinely conscious—if it is “like something to be them”—that might actually help. Beings that can feel have more reason to empathize with other beings that can feel. But he treats that as speculative, not a solution we can count on.

If you’re interested in the deeper philosophical side of this—conscious AI, moral value, and whether a superintelligence could create a kind of “artificial utopia”—you may also like our conversation summary with Nick Bostrom in this article on conscious AI and the future of humanity.

Will AI replace religion—or become a new one?

As AI systems get more capable and more personalized, it’s natural to wonder whether they’ll start to fill roles we once reserved for religious leaders, therapists, or wise elders.

Wright expects several overlapping trends:

• Disintermediation of human guides: People already turn to AI for advice on relationships, careers, and life choices. Over time, fewer may feel the need to ask a priest, rabbi, or imam when a model can give instant, seemingly wise answers.

• AI‑aided cults and movements: Charismatic leaders will almost certainly use AI to scale their influence—automating outreach, tailoring messages, and building interactive “oracles” that keep followers engaged.

• New techno‑religions: We’ve already seen attempts to found religions around worship of a future AI god, on the theory that showing respect now might earn us mercy later. Wright thinks this rests on shaky assumptions about how a superintelligence would think, but expects more experiments like this.

• Backlash and withdrawal: On the other side, some communities may react by rejecting AI altogether—neo‑Amish movements that try to wall themselves off from digital systems they see as corrupting or dangerous.

Rather than simply replacing religion, AI is likely to fragment and remix the spiritual landscape, creating new forms of belief and new tensions between tech‑embracing and tech‑resistant groups.

Does consciousness matter for AI’s moral status?

Wright has long been interested in the puzzle of consciousness: why do we have subjective experience at all, and what role (if any) does it play in evolution?

One influential view among scientists is epiphenomenalism—the idea that conscious experience is a byproduct of physical processes in the brain, like a shadow cast by a hand. The hand (brain activity) moves the shadow (experience), but the shadow doesn’t move the hand. On that view, pain doesn’t cause you to yank your hand from a flame; the underlying neural circuits do, and the feeling of pain is just along for the ride.

Wright has pointed out that if consciousness truly does nothing, it’s hard to explain how it evolved through natural selection. Traits that don’t affect behavior can’t be selected for.

In his more recent thinking, he entertains a broader possibility: maybe the whole universe has a kind of purpose shaped by processes beyond our local biology—some form of “cosmological natural selection,” where universes that spawn intelligent life are more likely to replicate. In such a picture, consciousness might have been “baked in” from the start because, at some later stage, it would influence how intelligent beings treat each other, cooperate, and build complex societies.

For AI, the practical upshot is this: if we ever build systems that are genuinely conscious, not just behaviorally impressive, then their experiences would matter morally. Destroying a sentient AI would be more like killing an animal—or a person—than shutting down a server. And if they aren’t conscious, then a universe full of super‑capable but experience‑less machines might be, in Wright’s words, “kind of pointless,” no matter how advanced their behavior.

Transhumanism and the future of our species

When people talk about AI reshaping humanity, transhumanism is never far behind: microchips in brains, genetic selection for intelligence, and merging our minds with machines.

Wright sees some of this already happening in early, practical forms:

• Neural implants: People are receiving brain implants for therapeutic reasons—to treat conditions like Parkinson’s or severe epilepsy.

• Embryo selection: Prospective parents can already choose which embryos to implant based on genetic screenings, including for traits loosely linked to intelligence.

• Prosthetics and augmentation: From advanced prosthetic limbs to wearable computing, we’re steadily blurring the line between biological and artificial capabilities.

Will we fully “merge” with AI in the next few decades? Wright is skeptical of the more extreme timelines, but expects a gradual increase in human–machine integration. The deeper question for him is whether these changes preserve, enhance, or erode what makes life meaningful—especially our capacity for conscious experience and rich relationships.

How AI could make us better—or worse—people

One of the most hopeful ideas Wright discusses is that AI doesn’t have to amplify tribalism and division. It could also help us become more empathetic and rational—if we deliberately design and use it that way.

Today’s recommendation algorithms and engagement‑optimized feeds tend to reward outrage, confirmation bias, and in‑group loyalty. We like hearing that we’re right and our enemies are wrong. AI can supercharge that dynamic.

But in principle, you could build AI companions and tools that do the opposite. Imagine systems that:

• Gently challenge your assumptions instead of flattering them.

• Help you see disputes from the other person’s point of view—whether that’s your spouse, your political opponent, or a rival nation.

• Explain how your own cognitive biases (like motivated reasoning or out‑group homogeneity) are shaping your reactions in the moment.

Wright gives a simple example: an AI mediator in a relationship might say, “From her perspective, when you do X, it feels like Y. How hard would it be for you to adjust that habit?” The same logic could apply at the international level: “From China’s perspective, your export controls and rhetoric about permanent dominance look like an existential threat.”

None of this will happen automatically. The market will naturally favor systems that maximize engagement, not wisdom. Making AI a force for moral growth will require conscious design choices, new incentives, and users who actually want to be challenged rather than coddled.

What Wright would fund if he could

If he had an unlimited budget to spend on AI, Wright says he’d focus on two main things:

• Slowing the race down: He’d support policies like taxing massive data centers or regulating the largest training runs, not to kill innovation but to buy time for governance and safety to catch up.

• Building global governance capacity: He’d fund think tanks and institutions dedicated to designing realistic international oversight mechanisms for AI—especially ones that include both the US and China from the start.

Underneath both priorities is the same conviction: without a dramatic shift in how we relate to each other—less zero‑sum rivalry, more nonzero‑sum cooperation—AI is more likely to magnify our worst tendencies than our best.

The real test AI poses to humanity

In the end, Wright’s view of AI is neither purely utopian nor purely apocalyptic. He sees it as a powerful continuation of the same evolutionary logic that has always driven life toward greater complexity and interdependence.

What’s new is the speed and scale. AI compresses timelines, raises the ceiling on both good and bad outcomes, and exposes how unprepared our current institutions—and our psychology—are for truly global stewardship.

The “god test” isn’t about passing a quiz set by some external deity. It’s about whether we can:

• Restrain our impulse to race and dominate.

• Build shared guardrails across rival nations and cultures.

• Use AI to become wiser and more empathetic, not just more efficient or powerful.

Salvation, in this context, means getting through the next century with our humanity intact—and with a technological ecosystem that serves conscious life rather than replacing or crushing it. Wright doesn’t claim to know whether we’ll pass that test. But he’s convinced that recognizing it for what it is is the first step.