Former Anthropic researcher doubles down on AI extinction warning

Artificial intelligence safety is no longer just a Silicon Valley debate. It’s now playing out in city halls and the White House, with former AI researchers warning that companies are moving too fast and governments still scrambling to catch up.

Whistleblowers warn NYC lawmakers about losing control of AI

At a landmark hearing before the New York City Council, AI researchers, whistleblowers, and representatives from major companies like OpenAI and Google were called to testify on the risks of rapidly advancing AI systems.

One of the most striking voices was Jacob Coxen, a former Anthropic employee who has become a prominent critic of how frontier AI is being developed. Coxen has previously warned that AI could pose an existential threat to humanity by 2030, and in his testimony he doubled down on that concern.

Why a former Anthropic researcher is scared of current AI progress

Coxen highlighted two core reasons he believes AI development is on a dangerous path:

1. We don’t really know how to control advanced AI systems. He argued that today’s most capable models are not fully understood by their creators. He described them less as machines we simply "build" and more as systems we "grow"—complex, emergent, and only partially steerable. That lack of control, he suggested, becomes more serious as the systems get more powerful.

2. AI capabilities are improving at an accelerating pace. Coxen warned that next-generation models could be dramatically more capable than today’s systems, and that we may be approaching a tipping point where AI can start to meaningfully improve its own capabilities. Once systems can help design, train, or optimize newer versions of themselves, the speed and scale of progress could jump again.

His bottom line for lawmakers: if development continues at its current pace without much stronger safeguards, humanity could lose control of advanced AI systems.

Calls to slow down and increase transparency

Coxen wasn’t alone. Daniel Kokotajlo, a former OpenAI researcher who resigned a few years ago, joined him in calling for stronger safety measures.

Their main demands included:

Slowing the pace of frontier AI development. Both argued that companies should not rush to deploy ever more powerful models without first proving they can be kept safe and aligned with human values.

Greater transparency from AI labs. Kokotajlo said companies need to be more open about what their systems can do, what risks they see, and how they are testing for dangerous behaviors. Without that transparency, regulators and the public are effectively being asked to trust corporate assurances alone.

These concerns echo a broader shift inside the industry, where even some major labs have begun publicly arguing for slower, more cautious development. For more background on this trend, see why almost every major AI lab suddenly wants to slow down.

City council frustration over vague answers from AI companies

New York City Council Speaker Julie Menin pressed company representatives on how seriously they take catastrophic risks from AI. At one point, she asked them to quantify what a worst-case scenario might look like.

Their responses, which reportedly failed to offer clear numbers or concrete scenarios, drew laughter from people in the room. Menin concluded that the companies were not able—or not willing—to meaningfully quantify catastrophic AI risk.

That exchange underscored a growing tension: researchers and whistleblowers warn of extreme downside risks, while companies often speak in broad terms about safety without providing the level of detail lawmakers are starting to demand. This power struggle over who defines and manages AI risk is already reshaping the policy conversation, as explored in the new AI panic and the fight over who controls the future.

Trump’s new “super intelligence force” enters the scene

While New York City wrestles with local guardrails, the federal government is moving in a different direction. President Trump announced a new initiative he calls the “super intelligence force,” aimed at coordinating how the federal government engages with the AI ecosystem.

According to a post on Truth Social, the task force will coordinate with:

• Consumers and public interest groups
• Religious organizations
• Critical infrastructure providers (like energy, transportation, and telecom)
• "Super intelligence" companies developing advanced AI

The director of national intelligence, Jay Clayton, has been tapped to lead the effort alongside other senior officials. The group’s first major job: produce a comprehensive report within 120 days on the opportunities and risks of AI, and outline the administration’s preferred approach.

Will the federal task force replace stronger AI regulation?

Early signs suggest the Trump administration may see this task force as a way to avoid or limit more aggressive tech regulation.

The expected path looks something like this:

1. Publish a broad AI report. The task force will likely map out how the administration views AI’s benefits and risks, and recommend a light-touch regulatory framework.

2. Use the report as a blueprint. That blueprint could become the basis for very narrow federal rules or voluntary standards, rather than sweeping regulation. How far it goes will depend heavily on how Congress responds—and whether Democrats or Republicans control it.

3. Set the starting point for future negotiations. Even if the report doesn’t have legal force on its own, it could serve as the Trump administration’s opening position in longer battles over AI law and oversight.

Growing friction between federal, state, and local AI rules

States and cities aren’t waiting for Washington. Places like California, home to many of the world’s biggest AI labs, have already started passing their own AI regulations and consumer protections.

The Trump administration has previously signaled that it wants a single federal framework to preempt a patchwork of state rules. But so far, that federal framework doesn’t really exist, and the administration hasn’t aggressively challenged state laws in court or through funding battles.

Once the super intelligence force publishes its report, that may change. A federal blueprint—especially one favoring limited regulation—could give the administration more political and legal ammunition to push back against stricter state or city-level rules like those New York is now considering.

What this means for the future of AI governance

The hearing in New York and the new federal task force highlight a few key trends:

Safety concerns are moving into mainstream politics. Warnings about losing control of AI, once mostly discussed in research circles, are now shaping city policy debates and national strategy.

Whistleblowers are becoming central voices. Former insiders from labs like Anthropic and OpenAI are pushing for slower development and stronger guardrails, often in direct conflict with their former employers.

Regulatory power is up for grabs. Cities, states, and the federal government are all trying to define who gets to set the rules for AI—and how strict those rules will be.

As frontier AI systems become more capable, the clash between rapid innovation and cautious oversight is only going to intensify. Whether policymakers side more with whistleblowers or with large AI companies could determine how much control we actually have over the next generation of AI.