The Hidden Governance Failure Behind Human in the Loop

Human in the Loop is a widely adopted governance control for AI. Too often, though, the presence of a human creates confidence without creating capability.

There is a growing assumption that Human in the Loop (HITL) for AI systems is achieved once a named individual has been added to an approval workflow. In my experience, the big question is whether that individual has actually been equipped to exercise independent judgement.

When an AI-enabled decision causes harm in a regulated organisation, regulators are unlikely to start by asking who approved it. They will want to understand how the decision was reached, what controls existed around it, and whether those controls gave the reviewer a realistic chance of catching the problem. That distinction matters, because assigning accountability is not the same thing as enabling judgement.

Human in the Loop has become one of the most widely adopted governance controls for AI, and it’s easy to see why: an organisation identifies a reviewer, updates its governance documentation, and records that human oversight is in place. On paper, the control looks complete. Too often, though, the presence of a human creates confidence without creating capability — the reviewer becomes accountable for decisions they were never properly equipped to evaluate. The weakness here isn’t the individual; it’s the way the control was designed around them.

When Oversight Exists Only on Paper

I encountered this during a transformation programme involving access to sensitive information (details below are anonymised and illustrative). An individual had responsibility for reviewing approvals that had already been granted, their role was to confirm access remained appropriate and, where necessary, withdraw approvals that should no longer exist. On the surface it looked like a robust governance control: responsibilities were clearly defined, and the authority to revoke access had been formally assigned.

What was missing was the information needed to actually exercise that authority. The reports he received contained only a limited set of data fields, and several of the key data points needed to determine whether access remained valid had simply been omitted. She had no way of establishing whether access should already have expired, whether the original approval was still appropriate, or whether it had been granted using the correct criteria in the first place. She challenged the reporting, pointing out that meaningful oversight was impossible without sufficient evidence to support an informed decision. She was right, and this wasn’t a failure of the reviewer. It was a failure of governance design.

The lesson applies just as directly to AI. Assigning a person to review an AI-generated recommendation doesn’t automatically create effective oversight. Human judgement can only exist where people have the information, context, authority, and time needed to reach an independent conclusion.

This is Not a New Problem

Failures in Human in the Loop didn’t begin with generative AI. The Dutch childcare benefits scandal is one of the clearest examples of what goes wrong when they’re overlooked.

Between 2005 and 2019, the Dutch Tax and Customs Administration used algorithmic risk profiling to identify suspected welfare fraud. Tens of thousands of families were wrongly accused, and many were required to repay childcare benefits in full, often tens of thousands of euros, creating severe financial hardship for people who had done nothing wrong.

Human reviewers were involved throughout the process, but their presence did not prevent the harm. They were unable to interrogate how the algorithm reached its conclusions and were not adequately equipped to challenge the outcomes it produced. Oversight existed in principle, but not in practice.

I should underline that this wasn’t a generative AI system. It was a traditional algorithm, which makes the lesson even more relevant today. If organisations struggled to design effective human oversight for relatively simple automated systems, the challenge only grows as AI systems become more sophisticated.

Why Human in the Loop Breaks Down

The problem is rarely that people fail to take their responsibilities seriously. More often, organisations design controls around completing a process rather than supporting good judgement.

Research has consistently shown that people place greater trust in automated recommendations when they’re working under time pressure or have limited experience, and high review volumes compound the problem by reducing concentration and increasing the likelihood that recommendations get accepted without real scrutiny. Meanwhile, organisations tend to measure the things that are easy to count such as approvals completed, queues cleared, or turnaround times, and rarely measure how often reviewers challenge a recommendation, request additional evidence, or overturn an automated decision.

The result is a process that looks efficient on a dashboard while the quality of oversight slowly deteriorates underneath it. Adding another reviewer rarely fixes this. Designing a better control does.

What Meaningful Human in the Loop Requires

Article 14 of the EU AI Act — the provision governing human oversight of high-risk AI systems — provides a practical benchmark for what meaningful oversight looks like: reviewers should be able to understand the system’s output, decide whether or not to rely on it, override its recommendations where appropriate, and intervene when necessary. None of those capabilities can be achieved through training alone. They have to be built into the operating model itself.

In practice, effective Human in the Loop depends on five conditions. The reviewer must understand exactly what decision they are accountable for, and must have sufficient information to reach an independent judgement on it. They need adequate time to consider the evidence in front of them, and the ability to challenge, pause, or escalate a decision without fear that doing so will be seen as poor performance. Finally, the reasoning behind their decision should be recorded, not just the fact that an approval took place.

Remove any one of these conditions, and the effectiveness of the control starts to weaken.

What Organisations Should Review

Rather than adding another layer of oversight, organisations should first determine whether the controls they already have genuinely enable independent judgement. That means asking whether reviewers receive enough information to make informed decisions, and measuring how often automated recommendations are actually challenged rather than simply how quickly they’re approved. It means checking whether people feel able to question an AI recommendation without being seen as slowing delivery, and making sure approval records capture the reasoning behind important decisions rather than just names, timestamps, and electronic sign-offs.

The objective isn’t to demonstrate that a person participated in the process. It’s to demonstrate that meaningful judgement took place.

If You are the Human in the Loop

If you have been appointed as the Human in the Loop, you should ask yourself a few direct questions. Do you understand exactly what you are expected to assess? Do you have sufficient information to make an independent judgement, rather than simply endorsing what the system has already concluded? Can you actually challenge or overturn the recommendation if you disagree with it, or would doing so raise eyebrows? Are you given enough time to review the evidence properly, rather than working through a queue at speed? And if you had to explain this decision to an auditor or regulator six months from now, would you be able to justify it?

If the answer to any of these is no, the weakness may not lie with you. It may lie with the design of the governance control itself.

As AI becomes more deeply embedded within regulated organisations, Human in the Loop will remain an essential safeguard. But its effectiveness will never depend solely on the presence of a person, but evidence that the review was designed, resourced, and governed in a way that allows that person’s judgement to actually be exercised. That is the hidden governance failure behind Human in the Loop, and it’s one every organisation should address before its next regulatory review, not after it.