AI Is Now Building AI: Is Anthropic's Warning Genuine Concern or Business Strategy?

Background: The Tipping Point of AI Self-Evolution

In early June 2024, American AI company Anthropic released a bombshell report warning that AI has begun showing signs of breaking free from human control. The report states that their latest AI models are accelerating the creation of next-generation, more powerful AI systems, and that artificial intelligence is approaching a critical threshold where it could autonomously develop its own "successors."

AI self-improvement refers to the ability of artificial intelligence systems to autonomously optimize their own architecture, parameters, or training processes with little to no human intervention. This concept is closely related to "Recursive Self-Improvement" — where AI improves itself to become smarter, and the smarter AI then makes even better improvements, creating an exponentially accelerating positive feedback loop. This is the core of mathematician I.J. Good's 1965 "intelligence explosion" hypothesis: once machine intelligence surpasses human intelligence, it will be able to design even better machines, triggering uncontrollable intelligence growth. While current large language models haven't yet achieved fully autonomous evolution, they've already demonstrated significant self-assistance capabilities in areas like code generation and model tuning — considered important signals on the path to that tipping point.

The report's most critical finding is that AI now possesses the ability to self-optimize and self-upgrade without much human intervention, and can even design next-generation systems more powerful than itself. Facing this risk of losing control, Anthropic took the rare step of calling on all frontier AI labs worldwide to establish coordinated, verifiable mechanisms, and to consider slowing down or even pausing the development of advanced AI systems when necessary.

Anthropic was founded in 2021 by former OpenAI Vice President Dario Amodei and his sister Daniela Amodei. The company's core mission centers on "AI safety," with a technical approach emphasizing "Constitutional AI" — a method that allows AI to self-regulate and self-correct based on a set of explicit principles, rather than relying entirely on human feedback. The company's flagship Claude model series is widely recognized in the industry for its reasoning capabilities and safety features. Anthropic's investors include tech giants like Google, Salesforce, and Amazon, with Amazon alone investing over $4 billion cumulatively. The company has consistently presented itself as a "responsible AI developer," which gives its safety warnings considerable industry credibility.

Shocking Internal Data: Direct Evidence of AI Building AI

The Dramatic Shift in Code Contribution

Anthropic disclosed a set of highly impactful data: as of May 2025, over 80% of code merged into their codebase was written by their AI model Claude. Before the launch of Claude Code in early 2025, this proportion was still in the single digits. In roughly one year, AI went from being a "helper" to the "lead programmer."

Claude Code is an AI programming tool launched by Anthropic in early 2025, falling under the category of "AI Coding Agents." Unlike traditional code completion tools (such as early versions of GitHub Copilot that only provided line-level suggestions), Claude Code can understand the full context of a codebase and autonomously plan multi-step programming tasks, including writing new features, refactoring code, fixing bugs, and writing test cases. The emergence of such tools marks a paradigm shift from "AI-assisted programming" to "autonomous programming." When AI-written code exceeds 80%, it means human engineers are transitioning from "code writers" to "code reviewers" and "requirements definers" — a fundamental change in the production relationships of software development.

The 52x Speed Optimization Experiment

To further verify AI's self-improvement capabilities, Anthropic conducted a test: they gave Claude a program for training a small AI model and asked it to optimize the running speed as much as possible without introducing errors. The results showed that a skilled human researcher would need 4 to 8 hours to barely achieve a 4x optimization, while Claude pushed it to 52x — a completely different order of magnitude.

Speed optimization of AI training processes involves multiple technical dimensions: computational graph optimization (reducing redundant calculations), memory management optimization (reducing data transfer overhead), parallelism strategy adjustments (more efficient utilization of multiple GPUs/TPUs), mixed-precision training (using lower bit-width data types without sacrificing accuracy), and algorithmic improvements (such as more efficient attention mechanism implementations). Human engineers typically need to identify bottlenecks one by one and make trade-offs based on experience, while AI can simultaneously explore numerous optimization combinations and verify results through rapid experimentation. A 52x optimization means a training task that originally required 52 hours can be completed in 1 hour, dramatically accelerating AI model iteration cycles and making the "AI building AI" loop far faster than human-led development.

This means AI can not only write code but also optimize AI training processes with efficiency far exceeding humans — the most direct evidence of "AI building AI."

Anthropic's Call: Establishing a Global Coordinated Pause Mechanism

Anthropic's proposal doesn't ask any single company to stop alone, but rather to establish a global coordination mechanism. Specifically, this includes:

• Multiple well-resourced, frontier labs jointly agreeing to participate
• Clear rules defining what conditions trigger a pause and what conditions allow resumption
• An independent oversight body for third-party verification

Anthropic's proposal doesn't come out of nowhere — it builds on existing international AI governance efforts. In November 2023, the UK hosted the first AI Safety Summit, where 28 countries signed the Bletchley Declaration, committing to cooperate on frontier AI risks. In 2024, South Korea and France hosted follow-up summits. Additionally, major AI powers including the US, EU, and China are all advancing their own AI regulatory legislation: the EU's AI Act officially took effect in 2024, adopting risk-level-based classification regulation; the US has used executive orders to require frontier model developers to report safety test results to the government. However, no international mechanism currently has the power to compel a pause in AI development — Anthropic's proposal is essentially calling for an unprecedented global technology governance architecture.

The proposal seems reasonable, but its timing has sparked enormous controversy.

Questioning the Business Motive: A Carefully Planned Pre-IPO Move?

The Subtle Timing in Capital Markets

Anthropic just completed a $65 billion Series H funding round, with its valuation soaring to over $900 billion — even surpassing industry leader OpenAI. Its annualized revenue has broken through $47 billion, and it has reportedly secretly filed an IPO application with the U.S. Securities and Exchange Commission.

Suddenly coming out at this critical pre-IPO juncture to shout "it's dangerous, everyone should pause" — the logic here is certainly worth scrutinizing. What better marketing tagline could there be than "our AI is so powerful we need to pause development"?

The "Tacit Understanding" Between Two Giants

Even more intriguing is that in the same week Anthropic published its report, competitor OpenAI also released an article titled "A Blueprint for Democratic Governance of Frontier AI," similarly listing AI self-improvement as a key issue requiring future oversight. Two leading companies throwing out the exact same alarming topic almost simultaneously makes it hard not to suspect coordinated intent behind the scenes.

The Accusation of "Regulatory Capture"

David Sacks, an informal advisor to President Trump and venture capitalist, publicly criticized Anthropic's actions as pushing "regulatory capture" — where large companies attempt to manufacture panic to convince governments to impose strict regulations, thereby limiting the development of low-cost AI models and conveniently consolidating their own monopoly position.

Regulatory Capture is a classic concept in public choice theory, systematically articulated by Nobel laureate George Stigler in 1971. It describes a phenomenon where regulatory agencies that should serve the public interest are instead "captured" by the entities they regulate (typically large corporations), resulting in policies that favor incumbents. In the AI field, this concern is particularly acute: if governments impose strict licensing systems, computing power thresholds, or compliance requirements based on safety warnings from leading companies, well-funded large corporations can easily meet these requirements, while startups and open-source communities may be excluded. Historically, the Basel Accords in finance and the FDA approval process in pharmaceuticals have both been criticized as de facto industry barriers.

NVIDIA founder Jensen Huang has also publicly quipped: "Some people develop a God complex after becoming CEO and start worrying about the fate of the world." The implication being that these companies already far in the lead calling for "hitting the brakes" are essentially widening the gap with their pursuers.

Genuine Concern or Business Strategy? Likely Both

Professor Ethan Mollick of the University of Pennsylvania offers a relatively neutral perspective: Anthropic's warning may indeed contain some marketing elements, but the underlying concern about AI losing control is real and deserves serious attention from all of humanity.

From a technical standpoint, AI code contribution rates surging from single digits to 80% and optimization capabilities reaching 52 times human performance — even discounted, these figures are alarming enough. From a business standpoint, releasing such a report on the eve of an IPO is indeed too "coincidental" in its timing.

The truth is likely both: the technical risks are real, but Anthropic's choice of this particular moment and this particular approach clearly involved careful business calculation. For ordinary people, what matters isn't speculating about corporate motives, but seriously considering: when AI truly starts building AI, is humanity ready?

Key Takeaways