AI reasoning is going dark just as the law demands it

Something has changed in how the most capable AI models reason, and it happened quietly. OpenAI's newest model, reported under the name Astra, uses a technique called recurrent depth, in which the same layer of the network processes a query's hidden state repeatedly, rather than writing its reasoning out as a sequence of readable words. The answer that arrives looks the same as before. What is different is that a meaningful share of the thinking behind it no longer exists anywhere a human, a regulator, or a court could read it.

OpenAI's own account of the decision is reasonable and deserves to be stated before it is argued with. Chief scientist Jakub Pachocki has said Astra's effective reasoning depth has been kept close to that of GPT-4, and that the company continues to prioritise chain-of-thought monitoring precisely because it understands what is at stake in giving it up. On this account the architecture was chosen for efficiency, not obscurity; the loop count is a dial OpenAI has deliberately left low, and nothing published about Astra suggests the model reasons entirely in latent space today. Read generously, this is a faster way to spend compute, managed by people who take the risk seriously.

The difficulty with a dial is that somebody eventually turns it, and the people best placed to judge that possibility are the least reassured by it. When the reporting broke, Redwood Research's Buck Shlegeris said he was extremely concerned, and his colleague Ryan Greenblatt asked the more pointed question of whether the industry now has a readily available route to much higher opaque reasoning depth, not whether one exists in theory. On the LessWrong thread that followed, the disagreement was rarely about the current model; even researchers offering reassurance conceded the trajectory, noting that OpenAI's own job postings already refer candidly to the potential loss of monitorability this class of architecture invites. Competitive pressure does the rest. Nobody expects a rival lab to leave its loop count low once a faster model is one setting away.

This is not a new worry, only a newly urgent one. A paper on chain-of-thought monitorability, circulated among safety researchers months before Astra existed, made the underlying argument in its abstract: verbalised reasoning is a fragile safety mechanism that depends on models continuing to think in language, and a shift toward continuous latent representations would eliminate that mechanism outright, not merely weaken it. The paper was written as a warning about a hypothetical shift. Astra is the first widely reported model where the hypothetical has a name.

England's legal system is, at the same moment, building the opposite assumption into how it intends to hold AI to account. The UK Jurisdiction Taskforce's statement on liability for AI harms, published in July, concluded that existing negligence law is flexible enough to cover most of what AI will do wrong, and went further: a professional could in time be found negligent not for using AI but for failing to, once competent peers routinely do. That rests on being able to establish, case by case, what a system did and why, closely enough to show it fell short of a reasonable standard. The statement acknowledges the opacity of AI systems as a live complication for causation, then largely sets it aside as no different in kind from problems English law has solved before. It may be right about that. It is also being written at the exact moment the reasoning it will need to inspect is becoming less available to inspect, not more.

For the professional-services firms and SMEs this affects, the two shifts arrive as one problem, not two abstract ones. Deloitte's most recent survey of AI in the enterprise found that only one company in five has a governance model mature enough to oversee autonomous agents at all, long before those agents are reasoning in loops nobody outside the lab can read. A rising standard of care and a shrinking evidentiary trail are heading toward each other, and most organisations adopting agentic AI this year have no way to notice the collision until a court, or a client, asks them to explain something they cannot.

None of this argues for waiting on AI, a case this publication has made before on different grounds. It argues for treating the audit trail as a procurement question rather than an engineering afterthought: whether a vendor's model can produce its reasoning in a form fit to be shown to someone, not whether the model is impressive. That is a duller question than whether an architecture is exciting. For the businesses that will eventually be asked to account for what their agents did, it is the one that will matter.