When Should AI Hand Off to a Human? The 15% Rule

The human-in-the-loop AI model is based on a few key principles: maintain human oversight when the AI does not know the answer, and involve a human when a case is exceptional, involves a sensitive decision, or has consequences that cannot easily be reversed.

Another way to think about this approach is the 15% rule. The case in a workflow require human judgment is around 15%; that proportion should be accounted for when designing the system. Rather than trying to automate every situation, the goal should be to identify the cases that need human involvement and ensure they are handled appropriately.

What Is Human-in-the-Loop AI?

Human in the loop AI is an approach that incorporates humans into certain points on the AI process. 

The AI can extract information from the data, answer questions, classify queries, and automatically recommend actions or perform steps in a task. If it meets a case outside of it, it stops and requests permission or delegates the task to a person.

For instance, an ecommerce support agent can answer to a customer's query on delivery times on his own. If a customer asks for a refund that is outside of the scope of the policy, the AI can then record the information of the order and clearly explain the problem, and then the decision will be made by a human support agent.

Why Not Automate 100% of the Workflow?

Imagine an Artificial Intelligence (AI) Customer Service agent.

“Where is my order” is easy. The ai can look up that information and give the customer an answer. But when it comes to “Why did you suspend my account?” “Can you please reactivate my account”. These could be many things maybe their account got banned for violating policy.

It could have been suspended at some point or even something that you would need to speak with someone about. By having an effective human in the loop, they will know what they can handle and what needs to be directed to someone else. 

What's the 15% Rule? 

The 15% rule simply divides your workflow into two different parts: 

• Ideally, there should be around 85% of the repeatable tasks, which is one where the AI can operate within a very restricted set of rules.  
• Exceptions and higher-risk tasks that may be best dealt with by human judgment make up 15%. The focus is to plan for handoffs and not allow them to fail. 

Not sure where your 15% sits? Velocity’s AI enablement audit maps which tasks to automate and where people must stay in the loop. Book a free consultation. 

When Should AI Hand Off to a Human? 

This type of escalation can be incorporated into the workflow in several different situations. 

When Confidence Is Too Low 

If there is no enough reliable information to complete the task at hand the AI should not assume an answer to continue the dialogue. 

Let's say a customer is asking you if a product will work on a previous model machine. But you don't know that from the product information what would you do? 

The best answer is to accept the reality that it is limited and to refer it to someone that can certify that. 

Human-in-the-loop AI agents can also include confidence thresholds to automatically pass the case to the human agents if it is considered a routine case or to involve human agents if the case is not certain. 

When the Action Has Serious Consequences 

The standard appointment reminder approval process should not be the same as the one used for the large customer contract cancellation process. 

If a human approval point is intended, it could be important information that is lost, sensitive to customer information, payment, or major refunds. An important question to consider is whether this can be easily fixed if the AI makes a mistake. 

When the Customer Is Already Irritated  

Another important thing to have is when the AI knows that it will create friction by continuing to converse. Maybe the customer has typed what there problem is. They ask for a person or they are having a unusual issue with the product. It may not be worth typing back at them. 

The agent will still be able to help, though, by summarizing the conversation before transferring it. This is why we need an important second step in the human-in-the-loop AI workflow: don't require the human to build from scratch. 

A Handoff Should include Context 

Now imagine interacting with an AI agent for 10 minutes and then being transferred to an employee who asks you for clarification on everything again. 

That creates unnecessary frustration.  

The AI should convey the appropriate context to the task. Including:  

• It could be what the user wants 
• What is already known 
• What attempts the system has made  
• Why it has escalated 
• What it hasn't decided upon yet  

But all that is about to change when the AI takes over all the routine tasks and then steps in to do the work that needs judgment. 

Avoid Relying on Humans as a Safety Net for Bad Automation 

Increasing the number of staff in the escalation queue will not make a difference if half of the time your team becomes involved in the AI conversation. 

Regularly review escalation patterns. If humans keep asking the same question, see if there is a data source or rule that can assist the AI to safely answer the question.  

However, when it has to be processed manually, perhaps it should have a threshold at which it will be able to process refunds automatically. Human-in-the-loop AI should help you determine where a human touch is needed and where AI may be. 

Human Oversight Should Change as the System Improves 

If something has high control over it there is a good chance it will be your first workflow when working with AI. You may find yourself looking at more responses and acting on them once you start using it in your daily life. 

Ongoing AI maintenance and support helps you can expand on what you can automate. 

You can continue to automate more things that come through your workflow. If you notice certain requests aren't giving you desired results you can add another approval process. 

The Best AI Knows When to Stop  

Good design often combines both approaches for various steps in the same process.  Include people when it has to be ambiguous, at risk, sensitive, and when there is really judgment to be made. The 15% Rule: Don't plan for complete automation of your system; here we can start with 15%.   Organize it so that the majority, if not all, of the straightforward issues may be resolved promptly, and the more complex situations reach a human in a situation where action can be taken.  It's this that makes human-in-the-loop AI relevant in real life. The person isn’t there because automation failed. The human is in the picture because there is a point where automation must stop.