The Threshold Is an Amount, Not a Percentage

have the same line ESCALATION_THRESHOLD = 0.90. In addition, the agent is active. Below it, one has a ping.

It's easy. It is readable. It allows everyone to feel that their independence is in control.

But it is not the right solution.

Correction is not a better number. The correction notes that the increase limit was never a percentage in the first place. It's a price.

Start with the wrong question

Many groups ask the question of ability. Can the agent write SQL? Can it issue a refund? If so, let it work.

But being able to do something and deciding to do it alone are different issues. The second is actually about the decision itself: How much does a mistake cost, and what does it cost to involve someone instead?

That is the real choice. If the agent does, he risks the cost of the error. When you grow, you pay for someone's time, whether the agent was right or not.

Put numbers on it and the comparison is almost embarrassingly easy. One thought first: one gets well inflated tickets. We will get comfortable with that later. In the meantime, do it yourself if:

(1 - p) * cost_of_error  <  cost_of_escalation

the more likely the agent is right. Reorganized, upload anytime:

p  <  1 - (cost_of_escalation / cost_of_error)

That number on the right is the actual limit. Note where it depends. Not a model. It is not a policy from the workplace. It's just a ratio between two costs. If errors are cheap, the margin is low and the agent can work more often. When mistakes are costly, the stakes are high, and even a confident agent must ask for help.

A fixed cutoff assumes that the ratio is the same for all decisions the agent will face. That's rare.

This method is based on Chow's rule of rejection from 1970, which is the basis of modern learning-reversal research. Old ideas are old when they are right.

Two tickets

The easiest way to show it: the receipt of a support ticket. An agent reads incoming tickets and resolves or forwards them to a person. The numbers here are for the sake of argument, but they are cumulative.

General refund request. Small fee, clear policy. If the agent fails, cleaning is a follow-up ticket and interest credit. Call it £15. The burn that develops is about three minutes of the technician's time. Call that £4.

The threshold is 1 minus 4/15. About 0.73.

So if the agent is 90% sure, it should work clearly. An increase would use £4 to avoid only £1.50 of expected error. If you say it out loud, no one will make that trade, but a fixed cutoff automatically does it throughout the day.

Now it is possible to take account. A customer reports a password change that he did not perform. If the agent takes this for granted, the negative is not a liability. Fraud loss, repair, legal exposure, non-returning customer. Put a solid £2,000 on it. The climb still costs £4.

The new threshold is 1 minus 4/2000, or 0.998. Basically, this means you have to go up every time. With 90% confidence, letting the agent go it alone risks £200 in expected costs, compared to just £4 in upside. It is fifty times cheaper to transfer it.

Same agent. Same 90%. Opposite correct answers. The 90% rule says it does in both cases, and in the second it quietly burns £196 per ticket to save someone three minutes.

That comment often comes up in conversation.

One loyal ally. The example assumes that the agent knows which ticket it owns. In actual testing, that's uncertainty. If the agent is 90% sure that the ticket is a normal return, the remaining 10% can be a sneaky take. So measure the cost of a mistake by what the ticket is, not what it is. When in doubt, price it as the worst category that makes sense. Tickets that look cheap are where expensive mistakes are hiding.

Catching up

All of those calculations depend on one assumption. If the agent says 90%, it is correct about 90% of the time.

That's an estimate, and it's not the same thing as confidence. Confidence is a number that the model produces. Estimating whether a number means anything.

Feed that into law and watch it break. If 0.95 actually means 0.70, your expected error cost is six times what the arithmetic calls for. The margin you have carefully priced is a myth. You can no longer manage the risk. He cleans it up with a number that looks like it might be but isn't one.

So before any of this, check the number. Make all decisions with confidence. A bucket with a belt of confidence, each stage of the decision. Compare the mean versus the observed accuracy for each bucket. If the agent says 90% and hits 78%, you've got your fix: put one on the other before anything touches the limit. Isotonic descent or Platt scaling will do the trick.

Two details bite here. Agents tend to rate worse in the rare, high-end category, because they are rare. That's the class that carries your extreme limit, so the global rating hides a significant failure. And the map moves as the ticket mix changes, so rescale it accordingly. Process is important under practice. Until you measure the gap, there is no limit to earning your trust.

Law by code

Here it is like an educational toy. A few lines to make the layout visible, not something I built or posted.

def should_act_autonomously(stated_confidence: float,
                            cost_of_error: float,
                            cost_of_escalation: float,
                            calibrate) -> bool:
    """
    Teaching toy: act alone or escalate, by expected cost.

    `calibrate` maps stated confidence to a measured
    probability of being correct, built from logged
    decisions and realised outcomes.
    """
    p_correct = calibrate(stated_confidence)

    expected_cost_of_acting = (1 - p_correct) * cost_of_error
    expected_cost_of_asking = cost_of_escalation

    return expected_cost_of_acting < expected_cost_of_asking

There is nothing good within the work. Most important is how you choose the three inputs. This is where your effort should go, rather than arguing that the magic number should be 0.85 or 0.92.

Where the numbers go wrong

Two things make the installation very difficult.

Rising costs aren't just about salary minutes. Increases pile up in the queue, and the queue gets crowded. At worst, over-indulgence trains reviewers to rubber-stamp. Send someone forty custom returns an hour and within a week they've stopped reading them. A person who approves everything is not an administrator. It's a tradition. So the marginal increase costs more than the appearance, and the gap increases in volume.

And remember the thought we made earlier: one gets it right. Usually they don't. Give the expert an error estimate and the rule becomes an action where (1 − p) × error cost is less than incremental cost and h × error cost. Usually uh is small enough to ignore. In ambiguous tickets it is not, and it reverses the limit towards the agent. That surprises people.

None of these problems change the way of working; they just touch the numbers. That is the advantage of using the numerical method.

You are doing this for your agent

The process is short.

First, group the agent's decisions into classes where the error costs are approximately the same. For example, put returns in one class and security events in another. If you can't measure the cost of a mistake in a classroom, it's not ready to be independent yet. In fact, finding this out is an important part of the process.

Then set the price for each class. Talk to the people who clean up the mess, not just the people who build the agent. Finance and support ops know these numbers better than developers, and are usually happy that someone finally asks.

Also, estimate the cost of mounting, including the effects of compaction and rubber sealing.

Use adjusted probabilities in your rule, not raw scores.

The last step: find a threshold for each class in the cost estimate, and let it go when the cost goes. A new pattern of fraud raises the price of a security error, so that the margin increases as well. No meeting is necessary, because it is taken instead of being ordered.

Finally, your agent will have several parameters, not just one. There will be more autonomy where mistakes are cheap, and less where mistakes are expensive. The model and comparison remain the same.

And that's the honest answer to the question everyone starts with. “How confident should my agent be before it can work alone?” it cannot be answered, because it is not specified. “How much does a wrong call cost here, and how much does it cost?” it can be. Set the limit as a value. The percentage adjusts itself.