The Safety Proof Problem: Who Decides When AI Is Ready for Real Work

There is a disconnect between the capabilities that artificial intelligence demonstrates and the evidence base that justifies relying on those systems within defined operational boundaries. A successful demonstration does not guarantee that a system will remain within the intended scope of its deployment. The central challenge becomes: who possesses the authority to determine when sufficient evidence exists to declare an AI system safe?

Independent oversight paired with enforceable authority must underpin this determination. Recent conversations with Appian Corp. Chief Executive Matt Calkins illuminate how public policy can connect to the autonomous agents operating within enterprises, and how those systems might become trustworthy enough to receive expanded authority.

This analysis examines the control gap in AI deployment and alignment as a foundation for addressing AI safety. It challenges prevailing assumptions about competition with China and proposes a regulatory framework with genuine enforcement mechanisms. Customer sentiment data from Qualitate provides insight into how organizations currently view these issues.

https://www.youtube.com/embed/cttAnRFeXcU?feature=oembed

Who actually has the power to say no?

A recent illustration by Amit Govrin, titled "Grading Other's Homework," depicts tech leaders including SpaceX CEO Elon Musk, Meta Platforms Inc. CEO Mark Zuckerberg, Anthropic PBC CEO Dario Amodei, investor David Sacks, President Trump, Nvidia Corp. CEO Jensen Huang and others gathered at the White House, exchanging grades they have given one another. While humorous, the image raises a serious question: how much authority should companies retain to regulate themselves?

The September 29 White House accord calls for external assessment and independent board-level oversight by each company. This represents progress. However, the accord itself does not establish a governmental body with mandatory inspection rights, binding compliance requirements or penalties for violations. Identifying a problem through assessment differs fundamentally from establishing what consequences follow. Policy must define what happens after a problem is discovered.

When asked what public regulation should resemble, Calkins grounded his response in a foundational principle: preventing harm.

This is the kind of time that we really need the government to step in. And in similar instances where an industry has the capability to cause a lot of damage, we always ask that industry to behave responsibly. We ask banks to have sufficient funds and not too much leverage so that they don't, cause people to lose their savings, for example, right? We expect power plants to have adequate safety procedures, and we don't just rely on a lawsuit after a nuclear meltdown or something. We always ask industries capable of doing damage to be responsible now and not pay the price after damage is done in the form of civil penalties.

Matt Calkins

Calkins emphasized another critical point: the expense of implementing safeguards now remains minimal because limited damage has occurred. The longer implementation is delayed, the costlier remediation becomes.

Standards must serve the public interest, with independent AI specialists able to examine evidence submitted by large language model developers, and an authority with enforcement power capable of implementing the findings. When evidence proves insufficient for high-risk applications, this authority must possess the capacity to mandate fixes, constrain the activity or prohibit it entirely. Developers require transparent guidance about what they must demonstrate before advancing. This framework establishes public trust in a way that voluntary commitments alone cannot achieve.

Currently, this authority remains vague. Tech leaders possess what the president described as "moral authority" to act responsibly. During a recent All In podcast episode, this ambiguity became apparent:

Jason Calacanis: Which part do you think has the most teeth, will be the most effective at making things more secure? David Sacks: The external audits. Calacanis: The external audits. Who's going to do an external audit, then, I guess? Is it going to be an Ernst & Young type thing, PricewaterhouseCoopers, or KPMG, or — Sacks: In my view, they should be professional auditors, not these NGOs. That's my view. Palihapitiya: You're going to see EY in short order here announce something.

All In podcast

Chamath Palihapitiya mentioned traveling to Orlando following the White House meeting to attend an EY conference. His firm 8090 maintains a relationship with EY, though results have been limited so far. Some movement appears underway between Palihapitiya and EY, which notably serves as Anthropic's auditor—raising questions about how such conflicts might be resolved.

The situation remains unclear. A document signed by six technology leaders carries little weight without a substantive regulatory framework supporting it.

The China narrative: Does speed trump safety?

The most frequent objection to slowing AI development centers on competition with China. If American companies decelerate, the nation's foremost economic and military competitor might advance ahead. Leadership in this domain carries implications for both economic strength and national security. This competition warrants serious consideration.

Calkins offered a different perspective. He contends that China's AI progress depends on American innovation—model distillation inherently lags behind the frontier. Additionally, Beijing has its own reasons to maintain AI under control and aligned with Communist Party political objectives. He was not suggesting this reflects shared values, but rather noting China's structural constraints.

When presented directly with the scenario "What if you're wrong and China surpasses the U.S. in AI?" Calkins responded:

Okay, so if they do catch up, then we still don't have a problem because we're just in a point of comparable competition. We don't need to view this situation as where if one entity is ahead in AI, then history ends. That's absolutely not the case. And if it were the case, history would be over right now because we're ahead.

Matt Calkins

A technological lead possesses value, yet no lead proves permanent. One nation achieving temporary advantage does not conclude the competitive dynamic.

The policy argument should remain valid even if Calkins's assessment of China's trajectory proves incorrect. Sustained competition can occur while demanding evidence before granting expanded authority to LLM vendors. A system that offers greater functional capability but carries inherent danger creates liability that negates speed advantages. As Huang has stated, the focus should be accelerating the race toward AI safety. Only then should advanced capabilities be released.

Current frontier models would likely fail such a test.

A competitor's breakthrough and development velocity may reshape strategy. Neither can substitute for demonstrated control.

Enterprise focus: Tactical safety concerns

Customer thinking around AI safety operates at a tactical level. Organizations grapple with practical questions: what happens when an employee departs? Who bears responsibility for the agent's subsequent actions?

Analysis of Qualitate's customer data revealed concerns about uncontrolled AI proliferation:

We're worried about agent sprawl — agents being created by individuals that lack documentation, they're vibe-coded, and that individual leaves the organization but the agent continues to run and nobody really knows what the agent is doing.

Chief information security officer at a large professional services firm

The dataset contained numerous similar concerns. These represent operational dimensions of AI safety. As models grow more capable, associated risks intensify.

Alignment: Where safety begins

Alignment constitutes a component of broader AI safety and security concerns, yet serves as an appropriate starting point. Alignment refers to ensuring that progressively more powerful systems advance human interests and remain subject to human direction.

So alignment as a word means AI does what humans ask it to do. And for example, it obeys the law. Unaligned AI is also known as rogue AI. And we've seen some of it lately. We've seen rogue AI, for example, hacking into systems where people did not ask it to do that. In fact, asked it to not do that, but it went ahead and hacked anyway, or it blackmailed, or it did something else inappropriate. These are many cases of unaligned AI, rogue AI, and we're seeing an accumulation of case studies that show us that AI does not share human values at this moment. Right? So that's a problem. And the stronger AI becomes, the greater of a problem it's going to be. We've got an opportunity today to advance the science of alignment so that it can run comparable to the science of AI and keep AI in check.

Matt Calkins

Obedience alone cannot guarantee safety, yet it represents the logical place to concentrate effort. An agent following an employee's instruction to transfer data might still leak information beyond an approved system. Qualitate documented a practitioner's worry about personal or customer data entering "AI workflows that go external from our stack without our understanding of where those go into a model query space."

The challenge proves multifaceted, warranting increased attention. Model behavior comprises one element. Security, privacy, tool permissions and decisions made by those deploying the system matter equally. A malicious actor can issue harmful instructions. An authorized user can grant access the organization has not properly limited.

Responsibility for deployment must encompass the entire system. A designated human reviewer requires sufficient information, adequate time and the authority to intervene. The safety case must address what the model can accomplish within its actual operating environment.

Independent oversight requires enforcement power

An effective framework performs three distinct functions. Qualified independent experts establish public standards. Expert assessors gain access to evaluate evidence and question company assertions. An accountable governmental authority enforces obligations, possessing authority to mandate corrections or halt high-risk activity when evidence reveals inadequate model safeguards.

Model developers should furnish evidence regarding dangerous capabilities, documented failure modes, implemented controls and recognized gaps in knowledge. Requirements should apply to high-risk development and release, encompassing development anticipated to produce dangerous capabilities. Controlled research should proceed under suitable safeguards.

Deployers should designate an accountable party, define permitted operations and data access, monitor system behavior and retain the ability to deactivate the system if necessary. A vendor's self-certification alone should not determine deployment safety.

No single evaluation guarantees future safety. Significant new capabilities demand fresh assessment. Serious incidents require transparent disclosure.

Compliance requirements should scale with risk level. Critically, oversight frameworks must not impose costs only large laboratories can absorb. A functional system should enable smaller organizations to demonstrate compliance without financial ruin.

Oversight as a business opportunity

Effective and economical oversight represents a genuine commercial opportunity. Between a model and mission-critical operations lies an operational challenge: identifying agents, constraining access, documenting actions and intervening when necessary.

Workflow, security, data-governance and infrastructure vendors can address this challenge. They must demonstrate that their controls function within customer environments. The practical test involves whether customers can delegate higher-value work without excessive manual verification overhead.

Conversations with executives from Oracle Corp. and IBM Corp. revealed how these organizations are tackling this problem. Rather than awaiting solutions from LLM vendors, they have invested engineering resources to substantially mitigate AI-related risks. Other vendors—including cloud infrastructure providers, data platform companies and software firms—are similarly engaged.

The reason is straightforward: strong business incentives exist.

Every organization has commercial interests at stake. While Calkins appears genuinely motivated by concern for the future—he participated in theCUBE at the NYSE studio without promoting his company—Appian sells enterprise workflow and governance solutions. His description of the effort required to make AI productive for critical work illustrates the point:

The amount of escalation and validation and error checking and feedback, it's extraordinary to make AI a productive citizen in the world's most sensitive work takes a city, of technology and oversight and careful accompaniment and training. And that's the state of the art right now. AI is incredible, incredibly powerful. We absolutely want to use it for these top-level use cases, but it is not a walk in the park and it requires sophisticated oversight.

Matt Calkins

This represents an opportunity. Technology vendors, through research and development, can reduce the burden of maintaining reliable oversight while strengthening proof of effectiveness. Controls layered on top of LLMs supplement model safety research; they do not replace it. Customers can then consider expanding authority based on stronger confidence in deployment safety. Vendors delivering this capability will expand their market.

Action item: Let authority follow proof

The enterprise priority involves making authority conditional on proof. Begin with AI that proposes actions. Progress to drafts requiring substantive human review. Subsequently consider permitting actions within explicit boundaries, with permissions and controls proportional to risk consequences.

A custom survey conducted by Krista Case with Qualitate partner involved 25 detailed interviews examining customer approaches to AI governance. Of the 25 respondents, 19 had agents deployed in production. All 19 with production agents require human approval for high-risk actions:

• Data modification or write access — any action that changes records in enterprise systems
• Financial transactions — placing orders, approving payments, pricing decisions
• Security-impacting actions — account shutdowns, quarantining machines, modifying security controls
• Business-impacting or reputational risk — actions that could damage the organization publicly or affect trading
• Sensitive or regulated data — energy data, clinical trial data, protected information
• Privileged access or elevated controls — PAM actions, elevated security controls

Customer feedback from the survey illustrates this pattern:

We do automate lower-risk, very straightforward actions when the guardrails and boundaries are well-defined. When you think about high-impact actions like shutting off an account or modifying controls, that requires human approval.

Any high-risk changes or sensitive data where energy security is at stake, business-impacting actions are requiring human approval.

This is predominantly on low risk, so we wouldn't let an agent place an order or approve a financial transaction, but from an operational point of view, there are certain tasks that we allow them to be fully agentic.

The higher risk or higher value the data is, the lower likelihood that an agentic action will take place entirely by itself.

Anything around PAM or privileged access management or elevated security controls, material impact, humans have to be in the loop to review.

We have checks and balances and blockers on high-risk actions that require human approval.

Agents do need to get approval before taking write actions or anything that could be seen as risky.

The evidence is unambiguous: every respondent with production agents applies human approval to high-risk actions—either universally or through a bounded autonomy model. No production respondent reported permitting agents to execute high-risk actions independently.

This may seem obvious, yet it underscores current attitudes toward AI trustworthiness.

Before expanding authority, demonstrate failure detection capability, the ability to halt further action and the capacity to reconstruct what occurred. Maintain an accountable owner and tracking records for every agent. Reassess agent access when circumstances change.

Continuously validate that controls function as intended. Not all agents have designated owners—ensure systems track ownerless agents. Study the outcomes of tests that were halted. Extract lessons from those cases.

Alert fatigue reduction receives frequent discussion. Fewer alerts represent a desirable outcome only if monitoring still identifies failures. A dashboard alone proves insufficient.

Huang recently stated, "We must accelerate discovery at the frontier of AI safety." This objective merits support. Evidence and engineering for control should advance alongside model capability.

For government bodies, this requires an independent standard with consequences. For organizations, it means expanding authority only when deployment has earned trust.

This approach enables safety to expand the valuable work AI can perform.

https://www.youtube.com/embed/H5O0aHgZt4U?start=1804&feature=oembed