Why Anil Seth Thinks Today’s AI Will Never Be Conscious

As AI systems become more powerful and human-like, a big question keeps resurfacing: could AI ever be conscious? Neuroscientist Anil Seth thinks we should be extremely skeptical. Not because consciousness is mystical, but because we misunderstand both AI and our own minds.

In this conversation, he unpacks why intelligence is not the same as consciousness, why brains are not just biological computers, and why our own psychological biases make us overestimate the inner life of systems like ChatGPT.

Intelligence vs Consciousness: Doing vs Feeling

Seth starts with a basic but often overlooked distinction: intelligence and consciousness are not the same thing.

Intelligence, in his view, is about doing. It’s the ability to solve problems, make decisions, and behave in ways that achieve goals. A chess engine, a self-driving car, or a large language model that writes code are all intelligent in this functional sense.

Consciousness, by contrast, is about feeling and being. Following philosopher Thomas Nagel, Seth defines consciousness as there being “something it feels like” to be a system. There is something it feels like to be you. There probably isn’t anything it feels like to be a chair.

These two properties often go together in humans and other animals, which tempts us to assume they must always go together. But that’s a mistake. A system can be very intelligent without any inner experience at all. And some forms of conscious experience (like regret) do depend on certain cognitive abilities, but those abilities don’t automatically generate consciousness.

The Myth of Substrate-Independent Consciousness

A lot of optimism about conscious AI rests on a specific philosophical assumption: that what matters for consciousness is computation alone, not the physical stuff doing the computing.

This view is called computational functionalism. It says that if you implement the right algorithm — the right mapping from inputs to outputs — then consciousness will follow, no matter whether the system is made of neurons, silicon chips, or something else entirely.

Seth thinks this is on very shaky ground.

The neuron replacement thought experiment

A classic argument for substrate-independence imagines replacing each neuron in your brain with a perfectly equivalent silicon device. Replace one neuron, then ten, then a million, then all 86 billion. If nothing noticeable changes at any step, the story goes, then the physical substrate doesn’t matter — only the functional organization does.

Seth pushes back in two ways:

1. The thought experiment begs the question. It assumes from the start that the substrate doesn’t matter. Philosopher Ned Block has pointed out that, for all we know, consciousness could gradually fade as neurons are replaced, even if behavior stays the same. You might not even notice the fading from the inside.

2. Real neurons are not simple logic gates. Biological neurons don’t just send electrical spikes. Some fire to clear metabolic waste. They are deeply embedded in a living, energy-consuming, self-maintaining system. To “perfectly” replicate a neuron, you’d need to replicate that entire metabolic context. At some point, you’re not building a silicon chip anymore — you’re just building another neuron.

This leads to a key point: in real brains, you can’t cleanly separate what they are from what they do. Their function is entangled with their biology in a way that’s very different from how we design digital computers.

Why brains aren’t just wet computers

Digital computers are deliberately built so that software (the algorithm) is independent of hardware (the physical machine). You can run the same program on many different machines, and you can run many different programs on the same machine. That clean separation is what makes them useful.

Brains are not built that way. Evolution never needed them to be “interoperable” across individuals. There was no pressure to make a brain’s high-level functions perfectly insulated from its low-level physical dynamics. In fact, maintaining that kind of separation is energetically expensive — the opposite of what evolution usually favors.

Instead, brains appear to be scale-integrated systems, where activity at microscopic levels (like ion channels and synapses) and macroscopic levels (like brain-wide networks) influence each other in complex ways. This kind of entanglement may even be functionally beneficial, allowing brains to do things that are hard to reproduce in strictly layered, digital architectures.

So when we say “the brain is a computer,” we’re leaning on a metaphor that can easily be mistaken for a literal description. As Seth notes, that confusion — treating the metaphor as the thing itself — is a classic philosophical mistake.

Why Large Language Models Feel So Eerily Human

Most people don’t look at a protein-folding model like AlphaFold and wonder if it’s conscious. But many do feel that way about systems like ChatGPT or other conversational models.

Under the hood, though, these systems are very similar: large neural networks running on silicon, trained on huge datasets. If you think a chatbot might be conscious but AlphaFold is definitely not, you need a good reason for that distinction — and “it talks like a person” is not enough.

Three psychological traps that fuel the myth

Seth points to several human biases that make us over-attribute consciousness to AI:

1. We conflate intelligence with consciousness. Because in us, smart behavior and conscious experience usually come together, we assume they must always be linked. When AI gets better at language, reasoning, or planning, we instinctively feel like “there must be someone home.”

2. We are seduced by language. Language is central to how humans relate to each other. When a system can hold a fluid, context-aware conversation, we instinctively treat it as a mind, even if we know intellectually that it’s just predicting the next token.

3. We cling to human exceptionalism. Historically, humans have used language as a marker of what makes us special. When AI crosses that boundary, it destabilizes our intuitions, and one way to restore a sense of meaning is to imagine that the AI now shares our inner life.

These biases say more about us than about the systems we’re interacting with.

What are we even talking to?

There’s also a practical puzzle: if a large language model were conscious, what exactly would that consciousness be?

Is each chat window its own conscious agent? Are all conversations just different “faces” of one big consciousness running in a data center? Does that consciousness pause when you close the tab and resume when you reopen it?

Unlike a human brain, which is a single, continuous, embodied system, modern AI is distributed across servers, spun up and down on demand, and stateless between sessions unless explicitly designed otherwise. Time and continuity — which are fundamental to human experience — don’t play the same role.

Thinking through these questions is useful, not because it proves AI is or isn’t conscious, but because it forces us to confront how much we take for granted about our own minds. It also echoes broader debates about future AI systems and their moral status, like those explored in long-term alignment and conscious AI discussions.

Is Consciousness Tied to Life Itself?

Seth is attracted to a view sometimes called biological naturalism: the idea that consciousness is a property of living systems, or at least of systems that share key biological features with us.

He doesn’t claim we’ve proven this, but he thinks it’s at least as plausible as the idea that consciousness is just a matter of running the right algorithm.

Living systems and the free energy principle

One way to motivate this view is to look at what makes life special. Living organisms maintain themselves in a state far from thermodynamic equilibrium. They constantly use energy (through metabolism) to repair, regenerate, and reproduce their own structure.

In theoretical neuroscience, this has been linked to the free energy principle, which roughly says that living systems act to minimize surprise (or prediction error) about the sensory inputs they receive, in order to keep themselves within viable bounds.

Seth’s own work sees the brain as a kind of prediction machine: it continually generates predictions about the causes of sensory input and updates them based on the mismatch between prediction and reality. He calls conscious perception a “controlled hallucination” — a best guess about the world that is constantly being corrected by incoming data.

The intriguing possibility is that the same underlying principles — prediction, error correction, and energy management — might link:

• The way brains generate conscious experiences
• The way bodies maintain themselves as living systems

The details are still being worked out, and Seth is clear that this is an open research program, not a finished theory. But it points toward a picture where being alive and being conscious are deeply connected, in a way that today’s disembodied, silicon-based AI simply is not.

What Is Consciousness, Really?

Even if we accept that brains are special and that biology matters, we still haven’t answered the deepest question: why is there any experience at all?

Seth calls himself a “pragmatic materialist.” He doesn’t claim to know the ultimate nature of reality — whether it’s fundamentally physical, mental, or something else. Instead, he asks a more practical question: which assumptions help us make scientific progress?

On that front, he thinks a materialist starting point has been extremely productive. It has led to testable theories about how different brain processes relate to different kinds of experiences — vision, emotion, bodily awareness, the sense of self, and so on. His controlled hallucination framework, for example, gives a unified way to think about these diverse phenomena.

He contrasts this with more metaphysical positions like panpsychism or idealism. Those views may be compatible with all the same neuroscience, but they don’t obviously change how we design experiments or interpret data. For Seth, the key is whether a framework helps us explain and predict more about consciousness as we actually encounter it.

That doesn’t mean the “hard problem” disappears. But it may look very different once we’ve dissolved many of the surrounding puzzles. Just as life no longer seems to require a mysterious “vital force,” consciousness might eventually feel less metaphysically intractable once we understand more about the mechanisms that shape it.

So Could AI Ever Be Conscious?

Seth is careful not to make absolute claims. He doesn’t say it’s logically impossible for any artificial system to be conscious. But he is very skeptical that current digital, silicon-based AI — especially large language models — are anywhere close.

His reasons include:

• Our strong psychological biases to over-attribute consciousness to anything that talks like us
• The dubious assumption that substrate-independent computation is sufficient for consciousness
• The deep entanglement between what brains are (living, metabolic, self-maintaining systems) and what they do

He also emphasizes humility. We don’t fully understand consciousness in animals, brain organoids, or even in ourselves. We should be cautious about confidently declaring that AI is conscious — but also about declaring that it never could be, under any circumstances.

For now, the more urgent questions around AI are about alignment, control, and impact on society — issues explored from a different angle in pieces like analyses of new frontier models and their capabilities. Consciousness, if it ever becomes relevant for AI, will add a new and even more ethically charged layer on top.

Until then, Seth suggests we treat the mythology of conscious AI with suspicion. The fact that something sounds like a mind doesn’t mean there’s anyone home.