Should we really fear AI that can improve itself?

Anthropic recently released a report with a dramatic title: “When AI Builds Itself.” Paired with an animation of machines replicating like cells in a petri dish, it quickly fueled headlines about AI racing toward a point where it can improve itself beyond human control.

The report raises the specter of “recursive self-improvement” – AI systems autonomously designing and developing their own successors. That sounds like the first step toward an AI singularity. But if you look closely at the data and how today’s AI coding tools actually work, the story is much less apocalyptic and much more grounded in productivity tools and software engineering.

What Anthropic is really worried about

The report describes a trend where more parts of AI development are being delegated to AI systems themselves. Given enough compute and enough delegation, Anthropic suggests this could eventually lead to systems that fully autonomously design and build their own successors – recursive self-improvement (RSI).

They frame this as a historic shift: AI that can build AI could unlock huge benefits in science, healthcare, and more, but also raise the risk that humans lose control over increasingly capable systems.

Many headlines zoomed in on a passage that sounded like a call for a global pause on AI development. But the actual wording is more conditional. Anthropic says slowing AI progress would likely be good if it could be done effectively and globally. If not, they argue, slowing down alone could just let less cautious actors catch up, making everyone less safe. In other words: “We’d slow down if everyone else did too. Otherwise we keep going.”

The data behind the “AI builds itself” narrative

Underneath the dramatic framing, the report is mostly about something more mundane and concrete: AI-assisted software development. Anthropic shares a few key charts to support its concerns about RSI.

1. More code per engineer

One chart tracks “code contributed per quarter per person.” For several years, output is relatively flat. Then, starting in late 2025 and into 2026, the amount of code per engineer jumps sharply.

The report notes that lines of code is a crude metric – more code doesn’t always mean better code – but they take the spike as evidence of acceleration once AI coding tools were introduced.

2. AI success on coding tasks

Another chart shows “cloud code session success rate” on different types of programming problems: trivial, routine, substantial, and open-ended tasks. The data starts in fall 2025, when more mature AI coding tools (sometimes called “coding harnesses”) became available.

For the hardest, open-ended problems – like diagnosing why a complex system is crashing – success rates rise from the low 20% range to around 70% as newer models (such as Claude Mythos and Claude Opus 4.7) are introduced in 2026.

The key detail: you don’t see earlier data because, before these coding harnesses existed, you couldn’t realistically ask an LLM to autonomously tackle multi-step debugging or planning. The graph is essentially showing that once proper tooling arrived, performance on coding tasks improved a lot in a short time.

3. When AI outperforms human programmers

A third chart asks: “Where researcher went wrong: could Claude have done better?” Anthropic looks at transcripts of human programmers working through technical problems. They identify moments where the human takes a wrong turn – investigating the wrong cause or following an unproductive path.

They then feed the transcript up to that decision point into an AI coding tool and ask: “What should we do next?” If the AI suggests the correct next step where the human went wrong, that counts as a “win” for the AI.

Across model generations, this “AI does better than the human at the next step” rate rises from around 50% to roughly 59–64%. That’s a meaningful improvement, but not a leap into omniscience.

What these charts actually show

Put together, the charts all point to the same underlying story: once the major AI labs got serious about building dedicated software development tools on top of large language models (LLMs), programmer productivity on many coding tasks went up.

These tools combine two pieces:

• A large language model (like Claude Mythos or Opus 4.7) that’s good at understanding and generating code.

• A “coding harness” – a traditional, human-written program that orchestrates the work: calling the LLM, running code, parsing outputs, and coordinating multi-step workflows.

Once these harnesses arrived and were tuned, AI systems became much better at things like:

• Generating boilerplate code

• Suggesting fixes for bugs

• Planning multi-step debugging or refactoring tasks

This is a big deal for developer productivity, and it’s exactly the type of use case where LLMs shine. But it’s a stretch to jump from “we’re shipping more code with AI help” to “AI is on the verge of recursively improving itself beyond our control.”

Faster coding does not equal smarter AI

The first key reality check: speeding up software development doesn’t automatically make AI fundamentally smarter or more capable in the scientific sense.

Breakthroughs in AI – the kind that unlock new capabilities – typically come from new ideas, not just more code. For modern generative AI, three major conceptual advances paved the way:

• Deep learning via backpropagation. Work by Geoff Hinton and others showed that neural networks with many layers could be effectively trained using calculus-based gradient methods. This enabled the “deep” in deep learning.

• Attention and transformers. Google researchers introduced the transformer architecture, which uses attention mechanisms to selectively focus on different parts of the input. This allowed models to generate coherent text based on long, complex inputs.

• Scaling laws. Researchers at OpenAI, led by Jared Kaplan, systematically explored what happens when you massively scale model size and training compute beyond traditional expectations. The result: bigger models trained on more data kept getting better.

None of these breakthroughs depended on writing code faster or having fewer bugs in the codebase. They were conceptual and mathematical insights about architectures, training, and scaling.

AI doesn’t become more intelligent just because you can refactor your codebase more quickly. Productivity tools help engineers implement ideas faster, but they don’t generate the core scientific ideas themselves. So even if AI coding tools double or triple developer throughput, that doesn’t mean we’re automatically accelerating toward uncontrollable superintelligence.

If you’re interested in practical ways to use these tools without losing control of your own projects, it’s worth looking at workflows for AI-assisted development, like those discussed in a complete workflow for AI-assisted coding without losing control of your codebase.

How AI coding tools actually work (and why they’re controllable)

The second reality check: the systems Anthropic is measuring are not mysterious alien brains. They’re straightforward combinations of predictable software and probabilistic text generators.

At a high level, a modern AI coding tool looks like this:

• The coding harness. This is a normal program written by humans. It contains explicit logic: conditionals, pattern matching, tool calls, and hard-coded handling for common cases. It decides when to call the LLM, how to interpret its output, and what actions to take next.

• The LLM. This is the model (Claude, GPT, etc.) that takes in text prompts and returns text outputs. It doesn’t run tools, change files, or execute code on its own; it just produces tokens.

The harness is fully transparent and deterministic. We know exactly what it can and can’t do because we wrote it. If you don’t want it to access a particular tool, API, or directory, you simply never give it that ability. No matter what the LLM suggests in its text output, the harness controls what actually happens.

LLMs themselves are unpredictable in the sense that they sample from probability distributions over possible next tokens. Ask the same question multiple times and you might get slightly different answers. But that unpredictability is confined to the text they produce – not to the overall behavior of the system, which is governed by the harness.

So when people talk about AI coding tools as if they’re black boxes that might suddenly decide to rewrite their own core logic or “go rogue,” they’re skipping over this architecture. The LLM can’t unilaterally change the harness. The harness doesn’t magically develop goals or intentions. It just follows the rules we gave it.

Are we actually close to recursive self-improvement?

Recursive self-improvement would require more than “AI helps write code”:

• Systems would need to understand their own architecture at a deep, mechanistic level.

• They’d need to propose meaningful design changes that actually improve capabilities, not just refactor or optimize existing code.

• They’d need the ability to implement, test, and deploy those changes with minimal human oversight.

• They’d need some form of goal-directed behavior that pushes them toward self-improvement as an objective.

Today’s coding tools are nowhere near this. They’re powerful autocomplete and automation layers wrapped in carefully constrained software. They can help humans explore ideas faster, but they don’t independently invent and validate new architectures in the way that past human researchers did.

This doesn’t mean we should ignore long-term alignment or control problems. There are serious researchers who believe superintelligence could be fundamentally uncontrollable; for example, see the arguments in Roman Yampolskiy’s perspective on why we might not be able to control superintelligence. But it does mean we should be precise about what current systems can and can’t do, rather than treating every productivity gain as evidence that RSI is around the corner.

More apps, not necessarily more value

A useful way to think about the current moment is to look at what’s happening in software more broadly. A recent analysis of iOS apps shows that after AI coding tools became widely available in 2025, the number of new apps released per month jumped significantly.

At the same time, the number of apps with significant usage stayed flat or even declined. In other words, AI made it easier to ship more software, but it didn’t automatically produce more software that people actually want or need.

This pattern likely generalizes: AI tools can flood the world with more content, more code, more prototypes – but turning that raw output into something genuinely valuable still requires focus, judgment, and human insight. The bottleneck shifts from “can we build it?” to “is this worth building, and does it solve a real problem?”

Where this leaves us

Anthropic’s report taps into real long-term questions about how far we should delegate critical work to AI systems, especially as they become more capable. But the concrete data they present is mostly about something narrower: AI-assisted coding tools getting better at helping humans write and debug software.

From that data, a more grounded picture emerges:

• AI coding harnesses plus LLMs are making developers faster and more capable on many tasks.

• These systems are built on fully controllable, human-written orchestration code.

• The main drivers of AI capability jumps are still new ideas and architectures, not just more lines of code.

• More output (apps, code, content) doesn’t automatically mean more value or more danger.

We should absolutely care about AI’s trajectory, governance, and long-term risks. But we should also be careful not to let every productivity chart turn into a sci-fi story about imminent loss of control. For now, tools like Cloud Code, Codex-style assistants, or Cursor are powerful helpers – not autonomous agents plotting their own upgrades.

The challenge ahead is less about stopping AI from secretly rewriting itself, and more about using these tools productively, safely, and responsibly in the systems we design.