Too good to fail? The surprising way a high-performing system can hurt you.
Translated by Juan Carlos Enciso of original for Cassie Kozyrkov (edited by Carlos Secada)
Imagine two (human) workers:
- Careless Daniel: is a constant disappointment to you, performing well on assigned tasks 70% of the time and producing utter embarrassment the rest of the time. Watching Daniel make 10 tries is more than enough to elicit an “oh my gosh” response from you.
- Carlos Reliable is another story. You have seen Carlos in action more than a hundred times and he has always impressed you.
Here comes the million dollar question. Which worker is most dangerous for your company?
On a high-stakes task, the answer might be Trustworthy Carlos… but maybe not for the first reason that comes to mind.
In another article, I have pointed out that ultra-reliable workers can be dangerous when the decision maker is deranged. These workers simply “follow orders”, even if they are terrible, so they can amplify incompetence (or meanness). But this is not the logic that I am going to follow here, since you have already heard me argue about it. Let’s look at this from a different angle.
Assuming the project is a wonderful idea that will make the world a better place if done right, is it still the best option?
The thing is that you You know that you shouldn’t trust It’s obvious to you. You hope he’s wrong… and that’s why you’re not going to bet the house on (Right?). You are not going to let incompetence catch you by surprise, so you will find alternatives. You will be wise enough to test the inevitable mistake.
You will also make sure to keep an eye on all things, so you will supervise Careless Daniel very rigorously. But Carlos? you confías in Carlos Reliable. Why control or build safety nets? Carlos is impeccable, right?
Carlos is not perfect. You just haven’t seen it fail yet – more data is needed to see the breakout point. The fact is that you haven’t had a chance to properly assess how catastrophic Carlos’ collapse could be.
Overconfidence is a problem. When a system is obviously flawed, it plans around its errors. You don’t trust perfect execution.
If leaders do not understand that there is a crucial difference between what bueno and the perfectthey can turn the blessing of having a good worker into the curse of working with one who offers superior performance.
The problem is that you think you’ve examined Carlos thoroughly, but you haven’t. It takes more than 100 attempts to learn the anatomy of a bug. When you scale up operations, you’re going to run into some kind of nasty problem.
Although the advice in this article is valid for human workers, it is even more urgent for Artificial Intelligence (AI) systems and other scalable solutions. One of the most dangerous things about math and data based solutions is that non-experts put too much trust in them. Don’t be that pushover who believes there is perfection in complex tasks.
When you increase the scale, you will come across the long tail.
It is better to assume that nothing is perfect. Even the most secure systems can fail… especially if you give them too many chances.
As reliability engineers often say: “when you increase the scale, you will meet the tails long”
Even if your system has been carefully tested and turned out to be 99.99% good, that doesn’t mean it’s perfect. Unfortunately, if you’re not careful, you can mentally round that percentage up to 100%. In other words, you will discard the possibility of errors because the probability it is low. That’s another way that a high performing system can be more dangerous than a low performing one… unless you do something about it.
Don’t rule out the *possibility* of observing bugs when their *chance* is lowered.
What makes Carlos Reliable dangerous is not superlative performance, but overconfidence.
So what is the solution? How to take advantage of all the benefits of excellence without exposing yourself to risks? Very easy. Build safety nets for Carlos Trustworthy as if you were dealing with Daniel Careless Then you’ll get the best of all worlds.
Just because you haven’t noticed an error yet doesn’t mean your system is perfect. Plan for failures and create safety nets!
Whether the task is performed by humans or by machines, never underestimate the importance of safety nets. Indulging in a false sense of security over seemingly flawless performance is bad leadership.
Relying on perfection is dangerous. Think of perfection as a bonus, but never trust it.
Instead, ask yourself unpleasant questions like and yes. What if your best surgeon gets sick while he’s working? What if the machine that monitors a patient’s vital signs fails? What if the driver is too tired to pay attention to the road? What if the facial recognition system of an automated border control misidentifies someone? What if the human checking a passport is wrong? What happens next?
Whenever I come across unfortunate AI apps that turn my stomach, the part that gives me goosebumps is the creators’ blissful ignorance of bugs; it is rarely the automation itself. Sometimes this kind of ignorance borders on the criminal.
Errors *will occur.*
The question to ask about errors is not: “will happen?” Because they will. Rather, one must ask:
- What safety nets are in place to protect people from the consequences of those mistakes?
- If the whole system fails – safety nets and all – what is the plan to fix things?
If there is no plan to prevent and remedy the damage, be prepared for the disaster ahead. Whoever is in charge of that project is worse than incompetent. It is a threat to society. Don’t be that person.
If a mistake becomes so catastrophic that failure becomes intolerable, then don’t automate the task and don’t let human workers do it either. Or, if there’s something in your ethics that says it’s more acceptable for failure to come from a human worker than from a machine (the crux of many AV debates), then use a keep-the-human-informed approach.
Better is not the same as perfect.
But whatever you do, remember that mistakes are possible. Humans make mistakes, AI systems too. Even if the deployed AI system makes fewer errors than the human alternative, remember that fewer is not the same as none. Better is not the same as perfect.
As long as the tasks are complex or the inputs are varied, errors will occur.
Believing in the myth of perfection can have dire consequences, so don’t let mathematical thinking get in the way of common sense. Whenever the tasks are complex or the inputs are varied, will produce mistakes.
If there is no plan to deal with a mistake, the result can be catastrophic! It may affect you much more than a bad executor’s mistake for the same reason you forgot to plan for it.
So, if you are prudent, you will go for the best system but build safety nets as if it were the worst.
Learn more about data science and artificial intelligence in Spanish here.