Decision-Level Tracing with OpenTelemetry

A compliance officer asked us why an automated approval had fired at 2:14am on a Sunday. We had the service logs. We had the request and response payloads. What we didn't have was the decision itself — which pieces of evidence the system had weighed, in what order, against which version of the policy, and why the weighing had come out the way it did. We could reconstruct an approximate answer by cross-referencing three log streams and a database snapshot, which took an engineer most of an afternoon, for one decision, after the fact. That afternoon is the reason we now instrument decisions as first-class spans, not as a side effect of instrumenting services.

Most teams' observability stacks are built around the service call as the unit of work: a span for the HTTP request, a span for the database query, a span for the model invocation. That's the right unit for debugging latency and errors. It's the wrong unit for answering "why did the system do this," because a single decision — approve this claim, escalate this transaction, suppress this alert — is usually assembled from several service calls, and the decision's own reasoning doesn't live in any one of them. It lives in the code that combines their outputs, which is exactly the code that traditional service-level tracing doesn't touch.

Treating a decision as a span, not a log line

The fix we've converged on is mechanically simple and organizationally underused: wrap each individual automated decision in its own OpenTelemetry span, as a child of the request trace but a sibling to the service-call spans it draws on, and attach the decision's actual reasoning as structured span attributes rather than as unstructured log text.

Concretely, that span carries: the policy or model version that produced the decision, the specific evidence inputs it considered (not "the request," but the actual fields it read and their values at decision time), the decision output and its confidence or score, and — critically — a reference to every child span that fed it, so the evidence chain is walkable rather than merely present. Because it's a real OpenTelemetry span, it inherits everything the tracing infrastructure already gives you for free: it's queryable by trace ID alongside the rest of the request, it shows up in the same waterfall view your engineers already use for latency debugging, and it survives the transition from "thing we built for compliance" to "thing engineers actually open when something looks wrong," because it's sitting in the same tool they already have open.

What this buys you that a compliance log doesn't

A hand-maintained compliance log answers "what did the system decide." A decision span answers "why," and it answers it at the granularity a regulator, an auditor, or your own incident-response engineer actually needs: you can query for every decision that used a specific stale evidence field, across every trace, in the time it takes to run a trace query — not by grepping application logs and hoping the field name didn't change between versions. You can pull the full evidence chain for a single contested decision in seconds instead of an afternoon, because the chain is the trace, not a manual reconstruction. And you get this for every decision the system makes, not just the ones someone thought to add explicit audit logging for in advance, because the instrumentation lives at the decision layer itself rather than being bolted onto specific code paths after the fact.

Where this gets genuinely hard

The honest complications are cardinality and cost, and teams that skip past them end up with a tracing bill nobody budgeted for. Decision spans carry more structured attribute data than a typical service span, and at volume — tens of thousands of decisions a day, each with a dozen evidence attributes — that's real storage and real query load on your tracing backend. We handle this with tiered retention: full-fidelity decision spans for a rolling window (30 to 90 days, driven by the actual regulatory retention requirement rather than a round number picked for convenience), and a compacted summary — decision outcome, policy version, and a content hash of the full evidence set — retained indefinitely so a contested decision from two years ago can still be verified against a rehydrated record even after the full trace has aged out.

The other real difficulty is get buy-in from engineering teams who see this as compliance overhead bolted onto their trace budget. It stops being a hard sell the first time an engineer uses a decision span to debug a production issue that had nothing to do with audit — tracing down why a specific customer got an unexpected outcome, using the same span they'd have needed to build anyway for compliance. Once that happens once, decision-level tracing stops being the compliance team's tool and becomes everyone's default way of asking "why did this happen," which is the actual goal: not a parallel audit system that engineers tolerate, but the same tracing infrastructure they already trust, extended to cover the layer that actually explains outcomes.