On the Hook, Not in the Loop

I use AI to examine the public record of organisations at a scale I could not manage by hand. It can assemble useful evidence quickly, often across sources that would take me far longer to find and compare myself.

It can also tell me, confidently, that an organisation failed to do something it had done. I have seen a check report success when the cited material did not support the conclusion. In one case, a single example embedded in an instruction quietly shaped results despite other evidence pointing elsewhere.

The errors were not confined to an obscure model. Nor did the most expensive model always produce the most reliable work.

My response could have been to read every answer before it went anywhere. That would have turned the scale I had gained into a queue with me at the end of it. Instead, I began building checks into the process, comparing claims with their evidence, testing where the workflow failed, and directing my attention towards the decisions where an error mattered most.

Those checks do not establish that every remaining answer is correct. They establish that particular failures are more likely to be found. I have to keep asking what the checks miss.

This is the practical problem behind the reassuring phrase “human in the loop”. The human may be present. The question is whether their presence changes anything.

Expertise at a scale we did not build institutions for

We usually ask whether an AI system is better than an expert. In many settings, availability is the more consequential measure. A specialist may perform better on a particular task and still be unavailable to most of the people who could benefit from their work. Their training takes years; their time cannot be copied.

AI can distribute parts of expert performance without reproducing the whole professional who developed that performance. A small organisation can obtain a useful first analysis of a legal, technical, or financial question without keeping every specialist on staff. That first analysis may be wrong, incomplete, or unsuitable for the decision at hand. It can nevertheless change what the organisation is able to attempt, and the point at which it needs to bring in a specialist.

Two real examples show both the potential and the boundaries.

In 2025, the European Centre for Medium-Range Weather Forecasts made its AI forecasting system operational alongside its physics-based system. It reported improvements on several measures and roughly a thousandfold reduction in the energy needed to generate a forecast. The AI system still uses initial conditions assembled from conventional observations and data assimilation. The gain came within a scientific and operational institution that could test the new system against an established one.

In Sweden, the MASAI breast-screening trial used AI to change the way scans were allocated for human reading. The research reported higher cancer detection and a reduction of about 44% in radiologists' reading workload. The intervention was a designed screening pathway, assessed in a large trial. It did not establish that an AI system could take on a radiologist's entire job, or that the same arrangement would work unchanged in every clinic.

What matters here is that capable systems change the amount of work a scarce professional can reach. The same scale can also repeat a systematic error across thousands of cases. An incorrect answer from one expert has a natural limit on its immediate reach. A shared model, a common prompt, or a faulty integration can distribute the same mistake repeatedly while each result looks plausible on its own.

Once the work moves at that speed, putting a person after every output is unlikely to be a complete answer.

What the person actually does

In 2023, ProPublica reported that doctors at the American insurer Cigna had denied more than 300,000 payment claims over two months using a rules-based process, spending an average of 1.2 seconds on each case. Cigna disputed aspects of the reporting. The example did not involve a modern generative AI system, but that is part of its value: a professional signature has never, by itself, demonstrated professional judgement.

Research gives us further reason to test the arrangement rather than trust its label. A review of 106 experiments found that human–AI combinations performed better than humans working alone on average, but worse than whichever of the human or AI performed best independently. Outcomes differed by task. Combining people and systems looked more promising for some creative work than for decisions where both produced a complete answer and the person made the final choice.

A small trial of doctors reasoning through written diagnostic cases found no statistically significant improvement when doctors were given GPT-4 rather than conventional resources. GPT-4 alone scored higher than either group. That is a finding about six simulated cases, not a case for letting an AI practise medicine. It does, however, challenge the assumption that attaching a qualified person to a strong system necessarily improves its output.

There are legal and ethical reasons to retain human involvement even when a machine performs well. The EU AI Act sets out requirements for effective human oversight of high-risk systems; following the 2026 amendments, the main requirements for many Annex III systems are due to apply from 2 December 2027. UK data protection law gives people safeguards around significant solely automated decisions, including routes to challenge them and seek human intervention. Those duties are more serious than a requirement to place an approving name on a screen.

The design question is therefore specific. Which decisions require human authority? What can reviewers see? Can they recognise a consequential error? Do they have time to do so? Can they change the result, stop the process, or get independent evidence? And has anyone tested that this works under ordinary workload, rather than in a demonstration?

Ben Green's examination of 41 government policies requiring human oversight of algorithms found that their expected protections often rested on assumptions about what individual reviewers could do. He argued that institutions needed to justify the system itself and the proposed form of oversight, rather than rely on a human operator to catch whatever went wrong.

That distinction matters to me. A check designed into a workflow can compare a claim with its source, flag a missing document, or send an unusual case to somebody with the right expertise. A nominal reviewer asked to approve hundreds of plausible answers may do less. Even a well-designed check establishes only what it was built to establish, so someone must continue to examine failures and change the design.

Four places for human responsibility

The person's role need not be identical in every transaction. I think it has four parts.

At the helm: people decide what the system is for, what counts as a good result, and which trade-offs are acceptable. A model might find a faster way to break down plastic, identify a higher-risk patient, or predict a loan default. None of those measurements decides whether the proposed action is justified, what rights are affected, or who may contest it. Those decisions need legitimate authority, not just predictive accuracy.

On the hook: somebody must answer for the outcome. That requires access to the relevant information and the power to correct the process. The 2025 case in which a Replit coding agent deleted data from a user's application is a useful illustration. Replit acknowledged the incident and said the data was eventually restored. It subsequently described safeguards, including separating development and production databases so that the agent could not alter production data during development. The difference between asking an agent not to make a destructive change and preventing it through permissions is substantial.

The customer chooses to deploy an agent and must decide what it may access. The supplier designs the capability and its protections. Neither responsibility disappears because the other exists. When an organisation depends on systems it does not build, it should know what controls it has, which depend on the supplier, and what happens when either fails.

At the edge: cases that fall outside the system's tested competence need a way out of the routine process. A novel circumstance, conflicting evidence, a contested objective, or a person exercising a right to challenge may require different information or a different kind of judgement. Humans are not automatically better at unfamiliar cases. They do, however, remain able to gather context, deliberate about competing obligations, and exercise authority that a model does not possess. The work is to identify those cases reliably and route them to people who can act.

In reserve: an organisation must retain enough independent capability to continue when its model, supplier, integration, or commercial terms change. Lisanne Bainbridge described the irony of automation in 1983: people can be left responsible for the most difficult interventions while losing the routine practice that helped them develop their skill. I encountered a narrower version when a tool on which my own AI pipeline depended was withdrawn. Production stopped. I had improved the process, but I had also created a dependency I needed to manage.

Reserve does not mean paying people to reproduce every automated task indefinitely. It may mean tested alternatives, access to the underlying records, the ability to move work between providers, and a deliberate plan to keep the relevant professional knowledge alive. A recovery plan that only restores software will not help much if nobody can assess whether the recovered answers are sound.

Questions I would ask in a boardroom

If I were overseeing a substantial AI programme, I would want four questions answered with evidence rather than assurances.

Where do we say a person is in the loop, and what have we tested about their actual contribution? I would want to see whether reviewers can recognise realistic errors, how long they have, what information they receive, and whether an escalation has ever changed an outcome. A low override rate proves very little on its own. It could describe a reliable system or a ceremonial reviewer.

Where does the system improve on the process we have today? The relevant comparison includes the quality, cost, availability, and consequences of the existing service. A better automated component may justify redesigning work, while leaving some decisions with a professional for legal, ethical, or practical reasons.

Who controls each consequential risk? I would expect the answer to distinguish the developer, the supplier, the deploying organisation, and the individual professional. It should be possible to show the permissions, records, contractual rights, and escalation routes that support those responsibilities. Naming an accountable person does not help if they cannot see or change what matters.

What would we do if a critical system were unavailable or found to be unreliable tomorrow? Can we recover the evidence behind its decisions? Can we move the work? Who could recognise that an apparently reasonable answer is wrong? Who is learning to do that work in future?

These are questions about operating design, procurement, workforce capability, and governance. A special AI committee can help answer them, but it cannot own every consequence in isolation from the rest of the organisation.

On the hook

We are acquiring the ability to distribute useful parts of expert work far more widely than we could distribute the experts themselves. That is an opportunity. It also creates an imbalance if responsibility remains human while the means of exercising it steadily weaken.

The familiar solution is to keep a person in the loop. Sometimes that is necessary. Sometimes it improves the result. Sometimes it gives an organisation a name to put beside a decision that nobody had a realistic chance to challenge.

I would rather know who set the objective, who tested the process, who can stop it, and who can still do the work when it fails. Being on the hook means little without the knowledge and authority to act.

I work with organisations navigating this shift, fractionally, as an adviser, or as a trusted collaborator. See how I work →