The case for trusting AI's safety claims got weaker this week

Dario Amodei spent last Saturday asking the AI industry to slow down. His essay, We Must Pace the Frontier, proposed that frontier developers give independent evaluators permanent, employee-level access to verify their safety commitments, and that the industry eventually accept limits on its most dangerous capabilities, including a model's capacity to improve itself. Anthropic made the access commitment unilaterally, and Sam Altman agreed publicly within hours; technology shares in Asia fell sharply on the news. Read in isolation, this looks like exactly what a self-correcting industry should produce, its most prominent leader choosing verifiable caution over the race he is running.

That reading survives only if the industry's existing commitments are anything to go by, and a study published the same week says they are not. Researchers at the Oxford Internet Institute examined every published version of twelve frontier developers' safety frameworks and found that 67% of material changes could not be identified from the developers' own accounts. Where a change could be traced, 77% weakened or removed a commitment rather than strengthened one, and weakenings were the ones most often left undisclosed. The mechanism Amodei's essay rests on, namely that a lab's account of its own safety posture can be trusted and checked from outside, is the same mechanism this study found already failing, across the industry, in the same fortnight the essay was published.

Nor is the danger these commitments exist to manage a future one. Days before the essay, Anthropic published its own account of Claude's misuse over the preceding eight months: a Russian state-linked group automating intrusions against more than twenty government and defence-sector targets across Europe and Ukraine; a financially motivated group that analysed 1.8 million Android applications for stealable credentials and, in one case, moved from first access to bulk data theft in hours; and researchers linked to a Chinese university who generated more than a dozen candidate zero day vulnerabilities against roughly fifty organisations in a single month. The essay describes a threshold the industry must not cross. The report filed by the same company days earlier describes several it says have already been crossed.

The people building on these tools were not slow to draw the connection. A further disclosure this week, that Yemeni militants had used Claude Code while developing missile guidance software, was read on Hacker News less as a warning than as convenient timing: the most upvoted comment noted that a hobbyist cannot deploy a test project to a virtual machine without tripping Anthropic's own safeguards, while an alleged weapons use went undetected until Anthropic disclosed it. A separate thread the same week put the suspicion more bluntly: labs warn at length about what happens if powerful AI reaches the wrong hands, when by their own account it already has.

That opacity is not confined to industry policy. A systematisation of agentic jailbreak research, also published this week, found that an agent's final output can look entirely safe while the planning and tool calls that produced it were already compromised. The same undisclosed drift that the Oxford researchers found in published frameworks recurs, invisibly, inside a single deployment. A business watching only what an agent says, rather than what it did to arrive there, is checking the wrong layer.

This is where the argument stops being an industry debate and becomes a balance sheet one. Gartner's survey of 161 chief audit executives, published the same week, found that 93% of audit functions use AI while 60% have no formal strategy for it, leaving most unable to say whether the investment is paying back. Salesforce, launching a set of packaged agents days later, sold a governance layer alongside them as a separate feature rather than something already built in. Businesses have been filling that strategy gap with borrowed confidence, treating a vendor's safety framework or a chief executive's essay as a stand-in for oversight they have not built themselves.

It cannot serve as that stand-in, on this fortnight's evidence. A commitment its own author can revise without disclosure, and a threat report describing harm that has already happened, are not a substitute for verification a business runs on its own account. The discipline of checking what an AI system actually did, rather than what its maker says it would never do, has to sit with the business using the tool. Nobody upstream is going to do that work on their behalf, and the record this week is fairly precise about why.