Glass-Box Explainability That Holds Up at Audit

We'd spent a good chunk of the project on explainability. SHAP values, waterfall charts per prediction, a clean internal tool where anyone on the team could pull up a decision and see exactly which features pushed it up or down. We were proud of it. We walked into the model risk review expecting it to be the easy part of the conversation.

It was not the easy part.

The reviewer looked at the waterfall chart for a declined application, nodded politely, and asked a question we hadn't prepared for: "If I re-ran this explanation next week, would I get the same numbers?" We didn't actually know — the explanation method involved sampling, and we'd never fixed the seed or documented that we needed to. That was the first small crack. The second one was bigger: she asked us to point to the specific policy factor — debt-to-income, payment history, whatever the credit policy actually named — that the decision turned on. We had feature contributions in SHAP units. We did not have an answer in the vocabulary her policy was written in, and translating on the spot, live, in the room, was not a good moment.

We left with homework, which is the polite version of "come back when this actually works for us."

What we built afterward was smaller than what we'd already built, which surprised us. We kept the SHAP tooling exactly as it was — it's genuinely the right tool for a data scientist debugging a model, and nothing about the review changed that. What we added was a second, much simpler layer sitting next to it: a direct mapping from the model's decision to the actual policy factors the credit committee is accountable for, stated in their language, not ours. It didn't need to be sophisticated. It needed to be something a non-technical reviewer could independently check against the written policy without anyone in the room translating for them.

We also went back and fixed the reproducibility problem, which took less time than the meeting where we found it. Seed the sampling, log the seed with the decision, done.

The thing we'd underestimated going in was that "explainable" isn't one property — it's at least two different questions wearing the same word. Why did the model decide this specific case is a different question from what does the model generally weight, and does that create a pattern the organization should worry about. We had tooling for the first question and had quietly assumed it also answered the second, and nobody in that room let us get away with the assumption. In hindsight, that was the useful part of the meeting, even though it didn't feel like it walking out.