Forty years of research suggest a simple principle: accountability should follow authority, not proximity to failure. An AI-generated research synthesis.
When artificial intelligence causes harm, organizations instinctively look for the human closest to the mistake.
Who approved it? Who reviewed it? Who clicked the button?
That is often the wrong place to look.
Four decades of research on automation, human error, computer systems, and algorithmic decision making point toward a harder conclusion: accountability should follow authority, not proximity to failure.
If an employee is expected to answer for an AI decision, that employee needs the information, time, expertise, and authority to influence it. If those things sit somewhere else in the organization, accountability needs to sit there too.
AI makes this problem more urgent, but it did not create it.
The important question is not simply who touched the decision last. It is who decided that the machine should be allowed to make the decision, under what conditions, with what safeguards, and who accepted the consequences if it was wrong.
That changes both how we investigate AI failures and how we should govern AI before they happen.
Automation changed human error. It did not eliminate it.
The foundation for this argument predates modern AI.
Lisanne Bainbridge’s Ironies of Automation (1983) identified a paradox that still matters. As machines take over routine work, humans increasingly monitor them and step in when something unusual happens. But automation can weaken the attention, experience, and situational awareness people need to intervene successfully.
James Reason’s Human Error (1990) attacked the same problem from another direction. Accidents that appear to have been caused by a person often have deeper causes in system design, inadequate defenses, conflicting incentives, missing information, or interactions among individually reasonable decisions.
Sidney Dekker later put the implication plainly: identifying human error should begin an investigation, not end it.
Consider an employee expected to review 300 AI generated decisions during a shift.
The employee technically has final approval. But if each decision receives only 30 seconds of review, the underlying evidence is unavailable, the system is presented as highly reliable, and the employee is measured on throughput, “human approval” tells us very little about who actually controlled the outcome.
The employee participated in the process.
That does not mean the employee exercised meaningful judgment over it.
This distinction matters because proximity to an error is not evidence of responsibility for it.
AI makes responsibility harder to locate
Helen Nissenbaum observed in Accountability in a Computerized Society (1996) that computerized systems distribute responsibility. Designers build them. Developers implement them. Managers decide how they will be used. Operators work with them. Organizations establish their objectives and incentives.
When something goes wrong, responsibility can disappear into that chain.
Andreas Matthias later called one version of this the “responsibility gap.” Traditional ideas of responsibility generally assume some relationship among knowledge, control, and consequences. Learning systems complicate that relationship because they can behave in ways their designers and operators did not specifically predict.
That is a real problem, but it also introduces a dangerous escape hatch.
Not being able to predict a particular failure is not the same as having no responsibility for creating the conditions under which it occurred.
An executive may have no idea which of a million AI decisions will be wrong. The executive can still be responsible for deciding that the system may make those decisions, determining what authority it receives, accepting a known level of uncertainty, and approving the controls surrounding it.
There are therefore at least two distinct questions:
- Who was responsible for the action?
- Who was responsible for the conditions that allowed the action?
As AI becomes more autonomous, the second question becomes more important.
A machine can work correctly and still cause harm
Traditional thinking also tends to equate failure with malfunction.
AI breaks that assumption.
An algorithm can perform exactly as designed and still produce unfair, inaccurate, or damaging outcomes.
Danielle Citron and Frank Pasquale’s work on algorithmic scoring, followed by Cathy O’Neil’s Weapons of Math Destruction, shifted attention toward this problem. The issue is not always a defective model. Sometimes the objective, data, performance measure, or application of the model creates the harm.
If the system is successfully optimizing the wrong objective, finding the developer responsible for a particular line of code solves very little.
The organization chose the objective. It accepted the data. It determined what constituted acceptable performance. It decided where the system could operate.
Responsibility does not disappear into the algorithm. It moves upstream toward the decisions that gave the algorithm its purpose and authority.
It also moves outward toward the person affected.
Accountability cannot consist solely of finding someone to blame after an incident. People affected by consequential AI decisions need practical ways to challenge those decisions, correct errors, and obtain remedies.
An organization cannot credibly claim accountability for an AI system if the people affected by it cannot challenge its decisions.
A human in the loop is not necessarily human control
This is where AI governance can create the appearance of accountability without the substance.
The standard answer to concerns about automated decisions is that a human will remain “in the loop.”
That sounds reassuring. Sometimes it is. Sometimes it merely provides someone to blame.
Madeleine Clare Elish’s concept of a “moral crumple zone” describes the problem well. A human operator can absorb responsibility when an automated system fails, even though that person exercised little practical control over the system.
The relevant distinction is not whether a human participated.
It is whether the human meaningfully controlled the decision.
Work by Filippo Santoni de Sio and Jeroen van den Hoven provides a useful foundation for thinking about meaningful human control. For organizations, I would reduce the idea to two rules:
Responsibility without meaningful authority is scapegoating.
Authority without accountability is a governance failure.
Adding another approval step fixes neither.
Accountability and blame are different questions
When something goes wrong, organizations often collapse four questions into one:
- Cause: What produced the failure?
- Blame: Who, if anyone, deserves blame?
- Accountability: Who must answer for the outcome?
- Remedy: Who must correct the harm and prevent it from happening again?
The answers do not have to be the same.
Work by Santoni de Sio and Giulio Mecacci on different responsibility gaps is particularly useful here. It helps separate individual culpability from broader obligations to explain what happened and take action.
Consider an AI system that incorrectly denies someone a service.
It may be genuinely difficult to identify one person who deserves blame. Perhaps the system behaved unexpectedly and the people involved acted reasonably given what they knew.
The organization still has to fix the decision.
The person harmed should not have to solve the organization’s philosophical problem before receiving a remedy.
That moves accountability away from punishment and toward institutional capability.
Can the organization reconstruct what happened? Can it explain the decision? Can someone challenge it? Can the organization correct it? Can it change the system so the same failure is less likely to recur?
The absence of an obvious villain does not eliminate those obligations.
Agents move responsibility toward delegation
Generative AI complicates authorship. Autonomous agents complicate authority.
An agent can increasingly select tools, take actions, communicate with customers, execute transactions, or coordinate with other agents. Requiring human approval for every action can eventually defeat the point of using an autonomous agent.
The consequential human decisions therefore happen earlier.
What objective does the agent receive? What permissions does it have? What is it prohibited from doing? How much uncertainty is acceptable? When must it stop and ask for human intervention?
Recent work on AI delegation pushes directly on this issue: unpredictability does not necessarily erase responsibility for the decision to delegate.
That argument has limits. Some outcomes may genuinely be unforeseeable. Developers, deployers, executives, managers, and users may have very different knowledge and authority. Moral blame, institutional accountability, and legal liability may not land in the same place.
But the direction is clear:
As machines exercise more operational autonomy, human responsibility moves toward the decisions that define that autonomy.
Governance has to move with it.
Start the investigation one decision earlier
Consider a simple case.
An AI system generates an inaccurate financial claim. An employee reviews it, approves it, and sends it to a customer.
The obvious question is:
Why didn’t the employee catch the error?
Ask it.
Then ask the more important questions.
Why was this task delegated to AI? What was known about the system’s reliability? What evidence could the employee see? Could the employee realistically verify the answer? How much time was available for review? Who determined that these controls were sufficient? What risk did the organization knowingly accept? What could have stopped the error from reaching the customer? Who now has the authority and obligation to fix the harm?
Those questions do not absolve the employee.
They prevent the employee from becoming the entire explanation.
That is the difference between investigating an error and investigating the system that produced it.
What organizations should do
The research points toward five things organizations should establish before deploying consequential AI.
1. Decision ownership
Name who authorizes the delegation, sets the objectives, accepts the risk, and has the authority to stop the system.
2. Meaningful control
If a human is responsible for a decision, that person needs enough information, time, competence, and authority to change it. Otherwise the control exists mainly on a process diagram.
3. Evidence and traceability
Preserve enough information about instructions, inputs, permissions, decisions, and actions to reconstruct consequential outcomes.
4. Redress
Give people a practical way to challenge important automated decisions, correct errors, and obtain remedies without first determining which technical component failed.
5. Learning
Use incidents, near misses, and unexpected behavior to change models, controls, permissions, processes, or deployment decisions. Finding someone to blame while leaving the system unchanged is not learning.
These controls should match the consequence.
An AI system drafting an internal meeting summary should not be governed like an autonomous agent authorized to transfer money.
The standard is not perfect prediction.
The standard is defensible delegation.
An organization should be able to explain why it gave an AI system a particular authority, what risks it accepted, how it constrained those risks, and what it committed to do if the system was wrong.
Accountability should follow authority
AI does not remove human responsibility.
It changes where we need to look for it.
Research on human error tells us not to stop at the person closest to an accident. Computer ethics shows how technology can diffuse responsibility. Research on algorithmic harm demonstrates that systems can operate correctly and still cause damage. Work on meaningful human control shows why ceremonial oversight is insufficient.
AI agents push that logic one step further.
The more operational authority we give machines, the more consequential the human decisions surrounding that authority become.
What objective did we give the machine?
What authority?
What constraints?
What risks did we accept?
Who could stop it?
What happens when it is wrong?
Those are human decisions.
So the central question in AI accountability is not whether a human was somewhere in the loop.
It is whether the humans we intend to hold accountable had meaningful authority over what happened.
Accountability should follow authority, not proximity to failure.