AI agents are being trusted faster than they are being checked

Every part of this week's AI news describes the same shortage. Frontier labs are shipping agents into the most consequential parts of a business faster than anyone is building a reliable way to check what those agents actually did, and that gap, not the adoption curve, is now the more expensive one.

The natural objection is that capability is still the binding limit, and this week supplied its strongest form. Salesforce and Anthropic announced Claudeforce, plugging Claude directly into the revenue cycle: pipeline review, forecasting, deal health, all handled by an agent with live access to CRM data and, in Salesforce's own language, permission to take governed action on it. Anthropic's annualised revenue has surged past $65 billion this year, and the company is committing $45 billion to computing capacity from Nscale to keep building. The honest reading of this thread alone is that models are not yet capable enough, not that anything is missing from how businesses use them.

A paper published this year under the title Towards a Science of AI Agent Reliability tested 15 models across two benchmarks and found the opposite: recent capability gains have yielded only small improvements in reliability, once reliability is measured properly, across consistency, robustness to minor variation, predictability of failure, and how badly things go wrong when they do. The point is structural, not a complaint about any one model. A business does not need to know whether an agent is clever enough for a task; it needs to know, in advance, whether the tenth run will behave like the first nine, and at present nothing about a newer model answers that question.

Practitioners said the same thing more bluntly this week. A Meta security researcher described an email agent that had behaved correctly on a small test inbox and then, turned loose on her real one, deleted messages it had been asked only to organise. The discussion that followed on Hacker News did not converge on the model needing to be smarter; it converged on treating agent output the way any engineer treats input from outside the system, verified through a deterministic check before it is allowed to act, never obeyed on trust. That is a discipline, not a capability, and most organisations giving agents access to real systems this year have not built it.

The earnings data shows what happens without it. McKinsey's 2026 survey of over 1,700 executives found that roughly 6% of organisations qualify as AI high performers, attributing meaningful earnings impact to AI use; the other 94% do not, despite record spending. What separates the 6% is not model choice: high performers are close to three times more likely to have redesigned the workflow around the tool rather than simply installing it. Deloitte's 2026 enterprise survey found the same shape from the other direction: 66% of firms report productivity gains, but only 20% report revenue growth, since barely a third have redesigned any process at all. Deloitte's own recommendation is to embed oversight into performance metrics as AI takes on more of the task, which is a verification structure by another name.

Regulators have started building the version of this that businesses have not. Transparency obligations under the EU AI Act took effect on the second of August, requiring chatbots to disclose that they are automated and AI generated content to carry a machine readable mark, backed by fines of up to €15 million or 3% of global turnover. Whatever one thinks of the mechanism, the instinct is identical to the one in the reliability paper and the deleted inbox: somebody outside the system has to be able to check what it did, since a more capable model does not remove the need for that check, it only raises the cost of skipping it.

None of this argues against adoption, which this blog has made the case for before and will not re-argue here. It argues that adoption has stopped being the question that matters most. The businesses separating themselves this year are not the ones that gave agents the most access soonest; they are the ones that built, at the same pace as the deployment, a way to check what the agent actually did before it touched a customer, a contract, or an inbox that mattered. Nobody is going to build that discipline for a company except the company itself, or eventually a regulator with a fine attached.