Building a Multi-Agent Incident Response System on AGNTCY

It’s easy today to build and ship a single agent. Teams of all sizes are doing it, and many are already running multi-agent workflows by chaining several agents together to deliver a larger task. The problem is what happens when a new agent ships. The workflow has to be edited to fold the new agent in. At a small scale this is a chore. At enterprise scale it generally requires teams to agree on a common tech stack and deployment platform, and agree on a standardized agent message format, and route every new agent through a code change to the workflow before any business value is unlocked. This is the wrong set of constraints to push onto teams.

The right constraint is the message shape. The Agent2Agent (A2A) protocol gives organizations exactly that: a common envelope for agents to talk through, regardless of stack or deployment. From there, agents can start communicating with one another using an agreed upon message shape, and each agent can be built with a completely different tech stack, language, and framework, and be deployed anywhere in the organization.

However, a standard messaging shape with A2A is just the starting point. There are still many questions to answer, like:

• How can agents dynamically discover one another?
• How is communication between agents performed in a secure and controllable manner?
• How can we observe the actions taking place in the multi-agent system?
• How can agents prove their identity to one another across organizational boundaries?

The AGNTCY project proposes answers to these questions, and below we put it to the test.

We built a five-agent Incident Response Room performing root-cause analysis of IT incidents in a mock enterprise environment. An Incident Orchestrator receives an alert, either from a human triggering a dashboard or from an upstream monitoring system, and behaves like a seasoned on-call engineer. It reasons about where to look. It queries specialists. It correlates findings, asks follow-up questions, and synthesizes a conclusion.

The agents we built for our Incident Response Room, with the Kubernetes Agent being dynamically brought into the investigation

Three core specialist agents are wired in from the start. A Log Analysis Agent identifies error patterns and onset times. A Change Analysis Agent reviews change records from the ITSM system, diffs suspicious deployments, and correlates changes with the incident timeline. An IT Knowledge Agent backs its answers with a live Neo4j graph providing a clear view of deployed services, teams, and dependency relationships, giving the agent the best possible picture of impacted services and blast radius.

In our test run, the system found the root cause: change record CHG-2847 had reduced a connection timeout from 30 seconds to 3, causing cascading failures. It reported the finding with 95% confidence, accompanied by a full timeline, the affected service list, evidence citations against the source systems, and a set of remediation steps. The report is delivered as structured typed output and piped into a human operator dashboard. Adding a human-in-the-loop step where an engineer approves the suggested remediation and the agent applies the change is a natural extension; the structured output is shaped to support it.

This investigation can of course be solved by a single agent or by a team of agents executing a static workflow. That is not what’s interesting about it. What is interesting is how independent agents—owned by different teams, built on different stacks, deployed on different infrastructure—composed themselves at runtime to do it.

The component that enables multi-agent team composition is AGNTCY's Agent Directory Service (ADS). ADS is federated rather than central: any organization runs its own directory, and directories interconnect through a content-routing protocol to form what AGNTCY calls the Internet of Agents inventory. Each agent an organization deploys is registered as a record in the Open Agentic Schema Framework (OASF), an extensible, versioned schema that captures the agent's skills, domain, features, and evaluation signals. Discovery is skill-based. OASF defines a hierarchical Taxonomy of AI Agent Skills, and agents query ADS by capability rather than by name.

In our run, the Incident Orchestrator initially had three specialists performing initial investigation. After the first round of correlation, it determined that the impacted service was deployed on Kubernetes. The Incident Orchestrator issued a skill-based query against ADS, got a Kubernetes Agent back, and pulled the agent into the investigation as the fourth specialist. No workflow file referenced that agent. Nobody updated the configuration to make this work. The investigation composed itself at runtime, around what the problem actually required.

Every agent in the system has a workload identity, a cryptographic identity issued at startup derived from the environment the agent runs in rather than from a secret it carries. The implementation uses SPIFFE, a framework for identifying and securing communications between services. A SPIRE server acts as the certificate authority, signing short-lived identities (SVIDs), and a SPIRE agent on every node performs workload attestation—a series of checks against the workload's environment—before handing it an SVID. In practice, this means no agent can impersonate another by presenting an HTTP header or a stolen token from a config file.

Once an agent has obtained its workload identity, what it is then allowed to do—for example, which tools it can invoke and which other agents it can communicate with—can now be governed in a central IAM system rather than being hard-coded inside agent images. Security teams write the policies. Application teams build agents.

The enforcement point in this system is AGNTCY's SLIM transport layer (Secure Low-latency Interactive Messaging). SLIM replaces direct HTTP between agents with an authenticated messaging fabric. There is no socket from one agent to another. Every message rides an authenticated channel; identity is verified per message from the SPIFFE-attested caller before the message reaches the target.

Task-Based Access Control (TBAC) enforces a policy governing agent tool usage. When the Orchestrator requested the Change Analysis Agent to use its rollback_change tool, TBAC denied it, logged it, and surfaced the denial event to the live Incident Response dashboard in real time. The model on the Change Analysis Agent never had the chance to invoke the tool. The call was rejected at the transport layer before the target agent's code ever ran.

The reason TBAC matters more than per-call authorization is that authorizing a tool invocation in a multi-agent system is not a single-call question. It is a chain-of-identity question. The relevant input to the policy decision is the full sequence of agent identities and messages that led to the call: who initiated, who relayed, with what context, against which tool, and with what parameters. That is the layer where security teams will (or will not) approve multi-agent systems at the procurement gate, and it is the layer where most teams have done no work at all.

Visualizing the run time discovery with the Agent Directory Service, and sending A2A messages through the SLIM secure messaging layer

We streamed every identity check, policy decision, tool call, and agent finding to a live dashboard over WebSocket. A human operator can watch the investigation unfold in real time, correlate any decision back to the identity that made it, and intervene. The same event stream is the audit record.

The harder version of the problem is what happens when you do not own every agent in the system. You still need visibility into the full span of what happened, in what order, and why— both to operate the system and to evaluate and improve the agents over time. AGNTCY's Observability SDK addresses that by instrumenting both the agents themselves and the protocols between them, SLIM and A2A. The telemetry it emits is OTel-compatible and supports trace recomposition across agent boundaries, so a single investigation produces a single trace even when its specialists were authored, deployed, and operated by different teams.

Existing enterprise IAM gets you most of the way on identity. SPIFFE/SPIRE is a framework that’s been running in production for microservices for years, and it worked for agents without modification. If your platform team already runs SPIRE, or has any workload identity story in place, you don't need a new identity stack to start securing multi-agent systems. The harder identity question is delegating user identity through an agent chain, but for agent to agent authentication, the existing tools are ready.

Dynamic agent discovery is the new infrastructure investment. This is the piece that doesn't have a drop-in equivalent in existing enterprise stacks. If you want agents to compose into teams at runtime based on capability rather than hardcoded workflows, you need something like AGNTCY's Agent Directory Service. Every agent your organization deploys gets registered, and other agents query it by skill.

For secure agent to agent messaging, reach for SLIM. End-to-end encrypted, authenticated, identity-verified per message, it’s purpose-built for agents and available on AGNTCY's SLIM GitHub repo. If you're earlier in your journey and just want to get agents talking to each other, a Pub/Sub system or message queue gets you started, but you'll be bolting authentication and encryption on top yourself. SLIM gives you those primitives by default.

Fine-grained tool authorization is the policy work every enterprise team should be planning for now. Authorizing whether an agent can call a tool isn't enough. The real policy question is which parameters it's allowed to set, what values those parameters can take, and under what conditions the call is allowed. An agent might be allowed to call a restart_service tool, but only against non-production environments. It might be allowed to query logs, but never with a time range exceeding 24 hours. It might be allowed to roll back changes, but only between 9am and 5pm with a human approver in the loop. TBAC is moving in this direction, and this is the layer that will determine whether your security team approves multi-agent systems at scale or blocks them at the procurement gate.

Incident response is the use case we chose, but the pattern underneath the build—independent specialist agents composing themselves at runtime around a task, on a shared substrate of discovery, identity, policy, and observability—applies wherever a problem requires multiple specialists, working against different systems and data types, to correlate evidence into a structured decision with an audit trail behind it. Three brief illustrations, drawn from B2C and B2B commercial core, that share this shape:

Order-issue resolution and fulfillment exception handling. An orchestrator triages an order problem, such as payment failure, inventory mismatch, shipping exception, fraud hold, or address validation. Specialists query the order management system, the payment provider, the fraud platform, the warehouse, the shipping carrier, and customer history. The output is a resolution path with action steps and citations against the source systems. Event-triggered, multi-system investigation, structured output.

Returns, refunds, and dispute resolution. An orchestrator triages a return or dispute. Specialists pull purchase and payment history, fraud and abuse signals, item-condition assessment from images and text, shipping status, policy interpretation, and payment-reversal feasibility against the processor. The output is a structured decision with citations strong enough to defend in a chargeback or a regulatory review. The chain-of-identity authorization story from earlier maps directly: granting issue_refund to an agent must consider amount, tenure, and prior dispute pattern, not just the call itself.

Configure-Price-Quote (CPQ) and deal-desk orchestration. An orchestrator reasons through deal complexity: specialists validate product configuration, evaluate pricing rules and discount policy, assess customer credit and payment terms, check fulfillment feasibility, and screen legal terms, each owned today by separate teams maintaining separate systems. The output is a structured quote with the approval steps it requires. The pattern extends from event-triggered investigation to proactive orchestration around any task that crosses team boundaries.

The components used here are all open source and live on AGNTCY's GitHub. The Incident Response Room is reproducible by any team willing to invest in the discovery, identity, policy, and observability layers (and the harder of those investments, identity, is one most enterprises have already made). We are happy to walk teams through the architecture, the decisions, and the trade-offs in detail.