Event-Driven Agent Meshes on Kafka

The system we inherited had eleven agents calling each other directly — a planner agent invoking a retrieval agent invoking a validation agent, each call a synchronous request waiting on the next one to finish before it could return. It worked, in the demo. In production, a single slow downstream agent — a validation step hitting a rate-limited third-party API — stalled the entire chain, and because every call was synchronous, the failure propagated backward through all eleven agents with no record of where in the chain the request had actually died. Debugging meant grepping through eleven separate log files hoping the timestamps lined up. We rebuilt it as an event-driven mesh over Kafka, and the rebuild changed more than the failure mode — it changed what kinds of systems we could confidently promise to operate.

Why synchronous agent-to-agent calls don't scale past a handful of agents

Direct calls make sense for two or three agents with a fixed, known call graph. Past that, the call graph itself becomes the liability: every agent needs to know the network address of every agent it might call, every timeout has to be tuned against the worst-case latency of everything downstream of it, and adding a twelfth agent means touching the code of whichever agents now need to call it. None of that is specific to agents — it's the same argument against synchronous service-to-service calls that pushed most backend architectures toward message buses a decade ago. Agent systems are just arriving at the same conclusion later, because early multi-agent frameworks modeled agent communication as function calls, which made direct calls the path of least resistance regardless of whether it was the right shape.

What the mesh actually looks like

Every agent in the rebuilt system publishes events to and consumes events from Kafka topics scoped by intent, not by which agent produced them — a validation.requested topic, a validation.completed topic, a plan.revised topic — so an agent doesn't need to know which specific agent will act on an event it publishes, only what event it's publishing. The planner agent publishes a plan.step.ready event and moves on; it doesn't block waiting for a response, because the response arrives as its own event on a topic the planner separately subscribes to, correlated back to the original plan by an ID carried through every event in that plan's lineage. That correlation ID, propagated as an OpenTelemetry trace context attached to every message, is what makes the mesh debuggable — we can pull every event across every agent for a single failed plan and see the entire causal chain in one trace, ordered by actual event time, instead of reconciling eleven log files by hand.

Reliability changes shape, it doesn't just improve

The most important shift wasn't fewer failures — it was that failures stopped cascading. When the rate-limited validation agent slows down under the event-driven design, its consumer lag grows and Kafka holds the backlog; the planner agent isn't blocked, downstream agents waiting on validation results simply see their events arrive later, and nothing times out and retries in a way that multiplies load on an already-struggling service. Backpressure becomes visible as a metric — consumer lag on the validation.requested topic — instead of invisible until it manifests as a stack of timeout errors. We added circuit-breaking at the topic level: past a defined lag threshold, upstream agents stop publishing new validation requests and route to a degraded-but-available fallback path, a policy we could never have expressed cleanly in the synchronous version because there was no single place backpressure was observable.

The cost of this is real and worth naming

Event-driven meshes trade one kind of complexity for another. Eventual consistency across agent state means you need explicit reconciliation logic somewhere — we keep a plan-state store, updated only by consuming the mesh's own events, that's the single source of truth for "what has this plan actually accomplished so far," because reconstructing that from the event stream on every read is wasteful and reconstructing it inconsistently across agents is worse. And Kafka itself is now a dependency every agent needs to be operated against, with its own capacity planning and its own failure modes. We wouldn't build this for three agents with a fixed call graph — direct calls are simpler and that simplicity is worth keeping until the agent count or the reliability requirements outgrow it. We'd build it for anything past that, every time.