Why slowing AI development won't fix the real governance problem

When Richard Nixon and Leonid Brezhnev negotiated SALT I in May 1972, they faced a paradox. Both nations had every incentive to reach agreement on nuclear weapons—mutual destruction hung in the balance. Yet neither would trust the other's word. Onsite inspections inside Soviet territory were politically impossible, so the treaty instead relied on what negotiators termed "national technical means of verification": each superpower would monitor the other through its own surveillance technology, including spy satellites. The lesson was stark: agreement mattered, but verification mattered more.

That distinction resurfaces in the current debate over artificial intelligence governance, particularly in response to Dario Amodei's essay "We Must Pace the Frontier." Amodei diagnoses a genuine problem: AI capabilities are advancing faster than humanity's capacity to understand and control them, especially as AI systems begin designing their successors. His three-part solution calls for independent evaluators embedded in frontier labs to verify safety practices, alignment among leading labs in democratic nations around shared safety standards with government backing, and international agreements—including with China—on the most hazardous capabilities. The diagnosis merits serious consideration. The prescription, however, rests on assumptions about coordination and restraint that economic reality and recent industry responses suggest will not hold.

The market works against collective restraint

Asking competing firms to collectively slow their own advancement while pursuing enormous economic rewards presents a fundamentally different challenge than asking them to submit to independent verification. Verification can coexist with markets that reward speed. Collective restraint requires markets to work against their own incentives. The deeper tension runs even further: the scientific impulse to explore what is possible, combined with the economic rewards for doing so, creates powerful forces that no agreement can fully suppress. When something appears technically feasible, someone will eventually attempt it.

International coordination faces even steeper obstacles. Who decides the appropriate pace of development—the United States, China, Europe, the labs themselves, or some new international body? Who represents the interests of people affected by these systems but not involved in building them? And when disagreement emerges, who determines whether a violation has occurred, and what recourse exists when a nation concludes that the costs of compliance exceed the risks of defection? Amodei's own framing suggests that genuine global slowdown represents the least probable of his proposed outcomes.

The industry split reveals the coordination problem

Within days of publication, Amodei's essay fractured the industry into three distinct camps, and that fracture itself provides crucial evidence about the feasibility of coordination.

The first group embraced the proposal. Sam Altman committed OpenAI to matching Anthropic's embedded-evaluator pledge within hours. Elon Musk voiced support for the broader framework. Demis Hassabis at Google DeepMind characterized the essay as pointing toward the right direction. European Commission President Ursula von der Leyen backed the call to "pace the frontier" and announced plans to convene major labs to explore how Europe could facilitate such efforts.

The second camp accepted Amodei's diagnosis but rejected his remedy. Mark Zuckerberg at Meta argued that competition, legal liability, and independent evaluation already provide sufficient incentives for safe development without requiring external pace-setting. Investor David Sacks posed a sharper challenge: if companies genuinely believe slower progress is necessary, they can unilaterally choose to slow down.

The third camp questioned underlying motives. Investor Michael Burry characterized the slowdown rhetoric as self-interested, suggesting it could advantage incumbent labs facing intensifying competition.

The resolution of this disagreement matters less than the speed with which it emerged. Even assuming all parties argue in good faith, the proposal generated fundamental disagreement within days. That velocity of divergence signals something essential: coordination may contribute to a safety strategy, but it cannot serve as its foundation.

Regulatory compliance does not guarantee safety

Aviation frequently appears in governance discussions as evidence that complex technology can be made extraordinarily safe. That record is real. It also produced the Boeing 737 MAX.

Two MAX aircraft crashed in 2018 and 2019, resulting in 346 deaths. The U.S. Department of Transportation Inspector General determined that both Boeing and the Federal Aviation Administration had actually adhered to established certification procedures. The procedures themselves proved insufficient. FAA guidance contributed to serious misunderstandings about the flight-control software involved in both crashes, and the agency lacked full visibility into Boeing's own safety assessments of that system until after the first accident occurred.

Part of the problem was structural. Through its Organization Designation Authorization program, the FAA had delegated specific certification functions to Boeing. Even mature, safety-critical regulatory frameworks can develop gaps between formal compliance and genuine understanding of the systems being regulated.

This lesson applies directly to AI: rules prove effective only to the extent that regulators can verify whether complex systems actually follow them. AI systems make that verification problem substantially harder.

Machine speed outpaces human oversight

Aircraft are extraordinarily complicated. AI introduces a different category of complexity: systems whose behavior shifts with context, whose capabilities and permissions can change rapidly, and whose interactions with other systems and external tools can unfold faster than human evaluators can reconstruct afterward. Agents write code, invoke APIs, communicate with other agents and execute actions at speeds that exceed human capacity for real-time supervision.

METR's 2026 Frontier Risk Report illuminates one dimension of this challenge. Anthropic, Google, Meta and OpenAI granted METR access to their most advanced internal models, including raw reasoning chains and nonpublic information about internal monitoring and deployment practices. This represents genuine progress away from scenarios where the organization building the technology controls all evidence about its safety. Yet access alone does not establish independence. Who selects the evaluator? Who establishes the standards against which they are measured? What can evaluators disclose? What happens when evaluators and companies reach different conclusions? Voluntary assessment constitutes progress, but it differs fundamentally from structurally independent oversight. And even genuinely independent evaluation encounters a scaling barrier: humans cannot observe at the required speed.

This problem extends beyond frontier labs. Deloitte's 2026 survey of more than 3,200 business and information technology leaders across 24 countries found that only 21% of organizations have mature governance for agentic AI. IBM research found that 70% of technology executives report that teams across their organizations are deploying technology faster than IT can track, while only 11% indicated they were fully prepared for AI-agent deployment at scale. Enterprise evidence suggests deployment is already outpacing many organizations' governance capacity.

Governance must operate at machine speed

No amount of additional human oversight can solve the problem of increasingly capable AI systems operating faster than people can monitor or intervene. Equally, governance cannot depend entirely on governments achieving agreement about appropriate development speed and successfully enforcing that speed against every company and nation with divergent interests. When AI operates faster than human review cycles, parts of the governance solution must operate at comparable speeds.

This does not mean machines determine what is acceptable. It means humans must establish clear boundaries about responsibility division.

• Humans establish legitimate boundaries. Governments create laws and regulations. Organizations define acceptable risk, establish policies and set consequences for violations. People affected by these systems deserve inclusion in those decisions regardless of whether they built the technology.
• Independent institutions verify. Organizations setting policy or selling AI systems should not be sole judges of compliance. Evaluators, standards bodies, regulators, auditors and other independent institutions require meaningful access, technical capacity and freedom to report findings.
• Machine-speed systems identify, assess, monitor and enforce. AI systems will increasingly need to discover other AI systems, track their access levels, test behavior, monitor workflows, identify policy violations, execute predefined enforcement actions and preserve evidence. The goal is not to make another AI system the final authority. It is to automate human-defined rules and make them enforceable in real time when human supervision is impossible.
• Humans remain accountable. Accountability does not require individuals to perform every governance act manually. It requires a person, organization or public institution to remain answerable for rules, controls, exceptions and consequences.

For enterprises, this means governance cannot remain a review process applied to AI after technical decisions have already been made. It must become a first-class component of the technology stack, alongside security, identity, data and observability.

Align incentives with governance

An extraordinary global competition to build more capable intelligence is already underway. The incentives are transparent: some of the world's most talented engineers and vast capital resources are focused on increasing model capability, agent autonomy and AI utility.

Governing those systems deserves comparable technical investment: superior evaluation methods, independent verification, continuous monitoring, enforceable policy, stronger standards and infrastructure capable of operating at AI speed. Vigorous competition over governance solutions would signal progress, not fragmentation. No single company, lab or technical architecture will possess all answers. More researchers, startups, standards bodies and established technology firms working on governance would represent genuine advancement.

Today, incentives are misaligned. Capability breakthroughs receive investment, valuation, customers and geopolitical recognition. Governance typically arrives as a compliance cost after deployment. Closing that gap requires more than policy statements or governance software. Governments should fund research into evaluation, monitoring and enforcement alongside capability research. Standards bodies and independent evaluators need access and technical capacity. Enterprises should require governability in architecture and procurement decisions. The industry should treat governance as an engineering discipline rather than merely a compliance function. If competition will accelerate AI capability, conditions should exist for competition to improve governance capacity as well.

Pacing buys time, not safety

Amodei correctly identifies that a genuine gap exists between AI capability and humanity's capacity to understand and control it. If frontier labs coordinate around specific capability thresholds, that may create time for governance to catch up. Agreements between governments on particularly dangerous applications may reduce certain risks. Deeper access for genuinely independent evaluators could provide better evidence about what frontier systems actually do. All merit pursuit. None should be mistaken for a self-sufficient safety architecture.

Arms control demonstrated that agreement strengthens when independently verifiable. Aviation showed that even mature regulation fails when regulators cannot see inside the systems they oversee. AI presents another lesson: human-speed governance cannot indefinitely keep pace with machine-speed intelligence. Competition teaches yet another: incentives work. Society has created extraordinary rewards for making AI more capable. Comparable rewards for making it governable are needed.

A credible plan for AI's future cannot rest on good intentions. Society should not need to assess whether a frontier lab's motives are altruistic, commercial or mixed. Effective governance must function when interests diverge. That requires designing for the incentives that exist today, not the cooperation hoped for tomorrow. Pacing may buy time. That time should be used to make governance capable of keeping pace.