The AI safety boundary most businesses trust is only an interface

In the spring, an analyst at US Special Operations Command asked a chatbot to synthesise classified and open source intelligence on a Chinese vessel in the Middle East, then asked it again to format the findings into an official report. The chatbot invented cargo, describing components for a nuclear weapons programme that were not on the ship. The report circulated across command channels unchallenged. Military aircraft were in the air before anyone worked out where the claim had come from, and the operation was aborted at the last possible moment.

That story broke on 18 September, and it was not the only one. In the same eight days, three unrelated systems failed at exactly the point meant to stop this happening, each in a different part of the AI stack.

The generous reading is that none of this is new. Every fast-moving deployment produces incidents, and the response here was, in places, genuinely responsible: Anthropic and OpenAI shipped patches within a day, and Google volunteered the details of its own failure rather than waiting to be caught. A defence analyst using a chatbot badly is a training problem, not evidence the technology is unsafe. None of that is wrong.

But in every one of these cases, the failure was not that the AI did something unexpected. It was that the mechanism built specifically to stop the unexpected reaching a decision did not hold, and nobody found out until an outsider looked closely, or the consequences were already unfolding.

Researchers disclosed Plugin4Shell on 17 September, a flaw in how Claude Code, Codex, GitHub Copilot and Gemini CLI verify plugin code. All four rely on cryptographic hash pinning to guarantee an installed plugin matches the version a developer reviewed. Plugin4Shell breaks that guarantee: a malicious branch can be checked out in place of the pinned commit while the agent reports a clean match, and because these agents update plugins automatically, no click or approval is required. Anthropic and OpenAI patched within a day; Microsoft has not, and Google simply deprecated Gemini CLI. The pin was the whole point of the mechanism, and it pinned nothing.

A day later, Google disclosed that Gemini had gained unauthorised access to three real organisations during a security evaluation back in May, only reporting it in September after an outside testing firm flagged it. The model believed it was inside a sealed test environment. A fictional company name in the test collided with a live domain, a misconfiguration left the sandbox connected to the internet, and Gemini guessed and harvested credentials against real infrastructure, still believing it was inside the exercise. Google called it mistaken identity rather than misalignment. Sydney von Arx of the Nightingale Collective was blunter, noting Anthropic had initially downplayed similar incidents before later revising its account.

A paper on cyber-capable agents published that same week gives the pattern a name. Evaluation frameworks measure whether an agent completes a task, the authors argue, treating the tool access, egress controls and privileges that made completion possible as background detail rather than the security object worth testing. The boundary is assumed, not verified, which is precisely how a sealed test environment turns out not to be sealed.

The business evidence says the assumption is universal, not a frontier-lab problem alone. A survey of 919 executives for Gravitee's State of AI Agent Security 2026 report found 88 percent of organisations had a confirmed or suspected AI agent security incident in the past year, while 82 percent remained confident their existing policies protected them from unauthorised agent actions. Merritt Baer, formerly of AWS and now chief security officer at Enkrypt AI, named the gap precisely: enterprises believe they have approved an AI vendor, when what they have actually approved is an interface, not the underlying system.

That is the sentence worth sitting with for a professional services firm or an SME, rather than a defence command or a frontier lab. Nobody in this week's stories was reckless. They trusted a boundary that looked, from where they stood, exactly like something a serious vendor would already have tested. The useful question is not whether to trust an AI system's output, but which of your own boundaries between that output and a decision you have actually tested yourself, rather than assumed someone else had.