Escalation by Consequence: A Threshold Design Note

The system we'd shipped had a confidence threshold: above a certain number, act automatically; below it, send it to a reviewer. Standard design, and for the first month it looked like it was working exactly as intended — reviewers were catching genuine edge cases, the automation rate was climbing steadily as the model improved, everyone was happy.

Somewhere around week six, someone on the ops side mentioned, almost in passing, that the review queue had gotten quiet. We pulled the numbers. Override rate had drifted from a healthy few percent down to essentially zero over about three weeks. Our first reaction, honestly, was to read this as a win — the model got better, fewer cases needed a human, the system was maturing the way it was supposed to.

It took an uncomfortable conversation with one of the reviewers to change our minds. She said, more or less, that she'd stopped really looking. Not out of laziness — the queue had become almost entirely cases the model was highly confident about, and after enough consecutive "yep, that's right" clicks, her attention had adjusted to match the actual information content of the task, which was close to zero. She wasn't wrong to adjust. We'd built a system that trained her to stop paying attention, and then congratulated ourselves when the override rate reflected that.

The fix we landed on wasn't more review — more review, generically, just gives you the same erosion problem on a longer timeline. What we actually needed was to stop treating confidence as the only thing that decides whether a case needs a human, and start treating consequence as a separate variable entirely. A high-confidence decision on something reversible and low-stakes genuinely doesn't need a person looking at it. A high-confidence decision on something expensive or hard to undo does, regardless of how sure the model is — maybe especially when the model is very sure, because that's exactly the case where a confident, wrong pattern-match slides through unchecked.

We rebuilt the routing around both axes instead of one, and split high-consequence cases into their own fast-review lane so a reviewer confirming a confident, high-stakes decision could do it in seconds rather than it feeling like full manual processing. Override rate went back up, which was the point — not because the model got worse, but because the queue started actually containing the cases where a second look mattered, instead of a random sample that happened to converge toward nothing.

What stuck with us afterward wasn't the fix, it was how long the wrong signal looked like a good one. Nobody was hiding anything, nobody made a bad call in the moment — the metric that was supposed to tell us something had gone quiet was structurally incapable of telling us that on its own, and it took a person noticing something felt off before the number meant anything again.