We build confidence in most parts of a system the same way. Equipment is qualified. Instruments are calibrated. A process is validated. Software is tested against its requirements.

People do not work that way.

We can train them, confirm competence, add a second person to verify critical work. That raises assurance. It never makes human performance certain, which is why we build controls around it.

But that is only one side of the person. The same human who can make the mistake is often the only one who can catch what the system was never built to see.

A machine can stay well inside its limits while an experienced operator hears that it is running differently. A record can carry every required field while a reviewer senses the story does not add up.

In those moments, the human is not a source of variability. The human is the control.

When you automate the routine, what you leave for the person is everything that could not be reduced to a rule. You have not simplified their job. You have concentrated it into its hardest part, and often stopped preparing them for it.

I have watched a capable operation automate a process, run it well for months, and slowly lose the ability to perform that same process by hand.

The manual method was still the official backup. On paper, the fallback existed. But the person expected to execute it had not run the process in a long time, because the automation had. The skill decayed while everything looked fine.

When the manual path was finally needed, the operator was no longer an expert at it. They were a novice, starting back up the curve almost from zero, at the exact moment the situation called for fluency.

Some organizations see this coming and decide, rationally, to wait for the automation to be repaired rather than run by hand. The people can no longer carry it. The dependence has become total, and everyone has agreed not to look at it.

The earlier decision is where the trap is really set. When a new process is designed, the choice to start fully automated almost always wins, because starting it up manually first and then transitioning is messy and slow. So it goes live automated from day one, and the human never learns to run it at all. They supervised the build. They trained the system. And now they cannot operate without it.

We created the dependence at the moment of design, and called it efficiency.

Lisanne Bainbridge named this in 1983, in a paper on the ironies of automation that has only grown more relevant. The more you automate, she observed, the more demanding the human's leftover role becomes, and the less practiced they are to meet it. The skills that fade during smooth automated running are the very skills the failure will demand.

The former expert, in the moment of crisis, has become the novice.

She was writing about industrial plants and flight decks. She could have been describing a manufacturing floor today.

The failures here are quiet. A few patterns worth watching:

The record is trusted past its limits. A completed field, a signature, an audit trail tell us what was entered and when. They do not tell us what a person actually saw, understood, or intended.

The manual skill erodes unseen. An operation can look fully in control on every dashboard while the human ability to step in quietly disappears. Nothing on the slide shows the skill is gone until the day it is needed.

Pressure rewards dependence. Schedule, output, release, and inspection readiness all push toward leaning harder on the automation and asking fewer questions about what happens when it stops.

Over-reliance becomes a compliance risk. As AI moves into operations, the FDA has signaled that leaning on it as a substitute for human expert judgment and quality oversight is not acceptable. Human readiness is becoming something to demonstrate, not assume.

If the human remains part of the control system, that dependence deserves the same seriousness as every other critical element.

Keep the skill alive. A backup no one can execute is not a backup.

Design for the human you are keeping. Give the person what they need to perform reliably: the right information at the point of use, an interface that makes the correct action easy, enough time and staffing to do the work as intended.

Match authority to responsibility. It is not credible to say quality is everyone's job and then make it hard for someone to stop the line, question an instruction, or escalate what they saw.

Read repeated error as a message about the system. When the same problem keeps recurring and the answer is to retrain the same people again, the retraining may be telling you more about the design than about the operators.

"The real question is not whether machines think but whether men do."
— B.F. Skinner, Contingencies of Reinforcement, 1969

Reflect: Think about a process your organization has automated and run smoothly for a long time.

If the automation failed tomorrow and you had to run it by hand, who could actually do it today, without relearning it from the start? And when did that person last fly solo?

If you are not sure of the answer, you have found the dependence worth examining.

In a process that could be fully automated, the human would not be needed. For some processes, someday, that may be exactly right. The danger is reaching for it carelessly, removing the person's practice while still depending on their judgment, keeping the human in name and losing the human in fact.

Automation removes work. That is its value, and it is real.

But removing work is not the same as removing the human, and not the same as keeping that human ready.

As systems grow more capable, and as AI begins to act inside operations rather than only inform them, the temptation will be to treat whatever human remains as enough, simply because they are there.

Can the person we kept still see what the system cannot, still decide when it counts, and still do the work when the automation stops? That readiness does not survive on its own. It has to be designed, protected, and practiced, with the same seriousness we give everything else we call critical.

Leading through clarity,
Jose

AI is moving fast and most people are falling behind. 

The Rundown AI keeps you ahead of the curve. 

It's a free AI newsletter that keeps you up-to-date on the latest AI news, and teaches you how to apply it in just 5 minutes a day.

Plus, complete the quiz after signing up and they’ll recommend the best AI tools, guides, and courses — tailored to your needs.

Sign up to start learning.