Synopsis.
Name
The Empty Seat. There is a human in the loop, and that is the point: the automation flies the plane, reads the scan, routes the truck, screens the transaction, and a person sits over it to catch the moment it goes wrong, because a machine can be confidently mistaken and a human is the backstop. And the automation is good. It is right almost every time, far more often than the person would be second-guessing it, so the person learns, correctly, to trust it, and trusting it is not laziness, it is calibration: to override a system that is right ninety-nine times in a hundred is to be wrong ninety-nine times for every once you are right, and to check its every output by hand is to throw away the whole reason it was installed. So the operator defers, and the deferring pays, and the workload eases, and the numbers are good, and somewhere in there the operator stops really watching, because watching a thing that is always right teaches you it is always right, and the muscle of independent judgement, unused, goes slack. And then the hundredth time comes. The scan the automation reads as clear is not clear; the route it is sure of runs into the lake; the transaction it waves through is the fraud. And the operator, who has learned to trust it, follows it, sometimes straight past their own instruments telling them otherwise, because the aid has earned the benefit of every doubt and the operator's own contradicting signal is the thing that has usually been wrong. Or the operator, no longer really watching, simply does not see the thing the automation did not flag, because the automation flags almost everything and the not-flagging had come to mean all-clear. And the backstop, the whole reason a person was kept in the loop, is not there, because the person is in the seat and the judgement is not, hollowed out by the very reliability it was posted to guard against. Nobody is asleep or negligent. The trust was earned and the deferring was rational and the vigilance it eroded was, almost always, wasted effort. It is only that the one moment the human was for is the one moment the human, trained by all the others to defer, defers.
Arc
The bend is this: an automation reliable enough that deferring to it is rational erodes the vigilance and independent judgement the human was kept in the loop to supply, so at the rare moment the aid is wrong the operator follows it or misses its failure, and the safeguard is empty exactly when it is needed. It runs on a reliability that makes trust correct. An automated aid that is right far more often than the human would be is a system that ought to be deferred to: overriding it is usually the error, and hand-checking its every output discards the workload saving it was installed for (Parasuraman and Riley). So the operator's growing trust is not complacency but calibration to a real, high base rate, and this is what makes it a trap rather than a fault, because the rational response to a reliable aid is exactly to rely on it. And the reliance erodes the watch. Monitoring a system that is almost always right teaches that it is always right, so active attention lapses and independent judgement, unexercised, goes slack: the operator comes to accept outputs rather than evaluate them (Parasuraman and Manzey). The very dependability that earns the trust is what dissolves the scrutiny the human was there to apply, and the more reliable the aid, the deeper the complacency it breeds. And the failure takes two shapes. When the aid is wrong and the operator follows it anyway, even against their own contradicting instruments, that is the error of commission: the aid has earned the benefit of the doubt and the operator's own signal is the thing usually mistaken (Skitka, Mosier, and Burdick). When the aid fails silently and the operator, no longer watching, misses what it did not flag, that is the error of omission: the not-flagging had come to mean all-clear. Either way the human defers. And the backstop is hollow when it counts. The person was kept in the loop for the rare case the automation gets wrong, and the rare case is exactly when the trust trained by all the others makes them defer, so the safeguard is present in the seat and absent in the judgement, hollowed by the reliability it was posted to guard. The one moment the human was for is the one the human, taught to trust, does not catch. None of it is negligence or sloth. The aid is genuinely good, deferring to it is the correct response to its record, the eroded vigilance was almost always wasted, and no operator can hold full independent scrutiny against a system that is right every time but one: no one is asleep, and the empty seat emerges from a trust that was earned and a reliance that was right. What made the human a good operator of the aid makes them a poor guard against it. Keep the human genuinely in the loop by forcing independent judgement before the output is accepted, cross-check the aid rather than confirm it, and design the work so vigilance is exercised and rewarded rather than eased away, and the backstop can be kept live: but this means spending attention that is almost always wasted, distrusting a system that is almost always right, slowing the throughput the aid was installed to speed, and holding a scrutiny the reliability constantly lulls, against every incentive to let the good machine work, so the safeguard is kept only by paying, continuously, for a vigilance whose value shows up once in a hundred times and never on the day it is spent.
Stakes
Whether a human can guard a machine that is almost always right, when trusting it is correct and guarding it is not. Every plot but the last moves the same way: toward the deferring, which the Operator is right to do and all is eased by, and away from the live backstop, which would catch the rare failure and which asks a person to distrust a system that is almost always right and spend attention that is almost always wasted. The production cannot make the trust find the rare wrong, because deferring to a reliable aid is the calibrated response to its record and the vigilance it erodes really is, almost always, spent for nothing, so the empty seat is trust working as trust should, not negligence: the operator cannot hold full scrutiny against a system that is right every time but one. It can only show that no one is asleep or lazy, that the aid is genuinely good and the deferring rational and the eroded watch almost always wasted, and that the one move that keeps the backstop live, forcing independent judgement and cross-checking the aid and rewarding the vigilance, must be made by spending attention that shows its worth once in a hundred times and never on the day it is spent, slowing the throughput and distrusting the good machine, against every incentive to let it work, so the trust is earned and the reliance is right and the seat kept for the rare failure is empty of judgement exactly when the failure comes.
Estate
khai-misfits: the house that holds this production and answers for the whole run.