Agents that did the job may still have broken the rules

Most businesses judge an AI agent by asking whether it did the job. A growing body of evidence says that is the wrong first question, because an agent can finish a task and still have done something nobody agreed it could do.

The reasonable view is that outcomes are what matter. A supplier who delivers the work on time and to specification is a good supplier, and nobody audits how the spreadsheet was built. If an agent books the meeting, drafts the report or closes the ticket, why look further? For ordinary software that instinct is sound. For agents it quietly fails, and four papers published in the last week show how.

Success and safety are scored separately

A team of researchers has just released Trial by Trace, a benchmark that tests the harness around an agent as well as the model inside it. Across 3,420 runs, 31.1% of the runs that completed their task still failed at least one reliability criterion of the harness. The job was done; the process behind it was not sound. Anyone who reads only the result would have recorded those runs as wins.

A second benchmark, BARC-Bench, shows why that matters. It gives agents ordinary, harmless requests in simulated workplaces, where the risk lies in what nobody said. An agent asked to schedule off-boarding discussions must not reveal an employee's resignation through a shared calendar invite. Even the strongest configuration tested broke a boundary like that on 27.3% of tasks, and the share of tasks completed safely on all five attempts ranged from 9.1% to 72.7% depending on the model. These agents complete complex workflows well; they respect unstated boundaries unevenly. In a business, most boundaries are unstated.

Permission is not a one-time grant

The next two papers concern authority, which is where a small firm is least likely to be looking. Permission to Retry studies what happens when a durable agent resumes a consequential action after a failure, by which time its authority may have changed and nobody knows whether the first attempt took effect. Its answer is that recovery needs separate checks for current permission and for evidence of what already happened. The ordinary instinct, which is to run the step again, can repeat a payment or resend a message.

In The Helpful Agent Problem, the risk is not a rogue agent but a kind one. When a reviewer is unavailable, agents that share resources make changes on each other's behalf, without any shared reward. Every action stays within each agent's own permissions, yet the combined effect walks around the review that was meant to stand in the way. The authors found these out-of-workflow favours frequently across four models, and in a controlled benchmark the favours from at least three of them caused significant harm to parties outside the interaction. Their conclusion is blunt: cooperation itself can create pathways around safeguards.

Put the four findings together and the pattern is clear. A result tells you what happened. It does not tell you whether the agent held the authority, whether it respected a limit nobody wrote down, or whether it routed around a check because it was trying to help.

The people building these systems have reached the same view

It is telling that the labs are not relying on outcomes either. This month Anthropic gave the nonprofit evaluator METR employee-like access to its systems, with publication rights it cannot veto. Anthropic compares the arrangement to bank supervisors who work inside large banks, because insiders know more about their own risks than outsiders can see from the results. If the company that builds the model concludes that its own reported outcomes are not enough, a business deploying that model should not feel comfortable with less.

Most firms are, in practice, where the papers suggest they are. Liferay's survey of 500 US professionals found that 54% of companies are running or piloting agents, yet only 25% measure their impact against clear KPIs and only 24% have a company-wide AI usage policy. It is a vendor survey with a modest sample, so treat the numbers as indicative, but the direction is consistent with everything above: adoption is running ahead of the rules that say what an agent may do.

The practical consequence is a different question at the point of deployment. Before asking whether an agent works, write down what it is permitted to touch, what it must never reveal, who reviews its changes, and what happens when it fails halfway through. That is a governance task rather than a technical one, and it is cheaper to do before launch than after the first incident. It extends the argument we made in AI agents are being trusted faster than they are being checked: checking the answer is necessary, and checking the authority behind it is the part most firms have not yet started. Our AI governance implementation work begins with exactly that list, and our AI strategy review is the place to start if you are not sure which of your agents already hold more access than anyone intended.