Is my service healthy? One question, one agent, one answer

There is a question every engineer dreads during an incident: "Is the service healthy?"

It sounds simple. It is not. Answering it properly means switching between dashboards, firing off multiple queries, correlating latency spikes with error logs, and checking whether downstream dependencies are contributing to the problem, all under pressure and often at an inconvenient hour.

What if that entire investigation collapsed into a single conversation?

That is what we set out to build: an AI agent that behaves like a knowledgeable SRE sitting beside you. It knows which questions to ask, knows how to query your APM data, and returns a clear Red, Yellow, or Green health verdict with context and recommendations. We call it the APM Service Health Monitor, and it runs entirely on Elastic.

Why APM health is a derived signal, not a single metric

Here is the thing about APM data: you already have everything you need. Latency percentiles, error rates, throughput, and span-level dependency traces are all sitting in your traces-* indices, indexed and ready.

The missing piece is not data. It is the reasoning layer that connects those signals into a coherent picture.

A p95 latency of 890 ms means very little on its own. But a p95 latency of 890 ms that runs 38% above the 24-hour baseline, alongside a 6% error rate on your postgres-primary dependency? That is a story. That is a Red, and it tells an engineer exactly where to look.

The APM Service Health Monitor encodes that reasoning (threshold logic, trend comparison, and dependency blast-radius analysis) into an Elastic AI agent that runs it on demand, for any service, anytime.

APM Service Health Monitor agent at the centre, fanning out to five ES|QL tools querying traces-*

How does the APM Service Health Monitor architecture work?

The agent is registered in Elastic Agent Builder as a chat-type agent wired to five ES|QL-powered tools. Each tool does one thing precisely. The agent orchestrates, compares, and reasons across all five.

POST kbn:/api/agent_builder/agents
{
  "id": "apm_service_health_agent",
  "type": "chat",
  "name": "APM Service Health Monitor",
  ...
}

The APM Service Health Monitor sits at the center, fanning out to five ES|QL tools that query traces-*.

Here is how the tool layer is composed.

Tool 1: apm_metrics_overview_tool, the 24-hour baseline

Every health assessment starts with a baseline. This tool computes the aggregate picture for a service over the last 24 hours (average latency, p95 and p99 latency, error rate, and throughput) in a single ES|QL query:

FROM traces-*
| WHERE service.name == ?service
| WHERE transaction.type == "request"
| WHERE @timestamp >= NOW() - 24 hours
| EVAL is_error = CASE(event.outcome == "failure", 1, 0)
| STATS
    avg_latency_ms  = AVG(transaction.duration.us / 1000),
    p95_latency_ms  = PERCENTILE(transaction.duration.us / 1000, 95),
    p99_latency_ms  = PERCENTILE(transaction.duration.us / 1000, 99),
    error_rate      = 100.0 * SUM(is_error) / COUNT(*),
    throughput_rps  = COUNT(*) / (24*3600)

This snapshot becomes the anchor. Every trend tool that follows compares its latest readings back to these numbers, so the agent always has a reference point, not just a raw value.

Tool 2: apm_latency_trend_tool, performance over time

Averages hide inflection points. The latency trend tool buckets p95, p75, and average latency into 5-minute intervals across the last 24 hours, giving the agent a time-series view of performance:

| STATS
    avg_latency_ms = AVG(transaction.duration.us / 1000),
    p95_latency_ms = PERCENTILE(transaction.duration.us / 1000, 95),
    p75_latency_ms = PERCENTILE(transaction.duration.us / 1000, 75)
  BY time_bucket = DATE_TRUNC(5 minutes, @timestamp)
| SORT time_bucket ASC

The agent takes the most recent 5-minute bucket's p95_latency_ms and compares it against the p95_latency_ms from apm_metrics_overview_tool, the 24-hour rolling average. That percentage difference is what gets scored Green, Yellow, or Red.

Tool 3: apm_error_trend_tool, failure shape detection

Error rate is directional. A gradual rise from 0.5% to 1.2% tells a different story than a sudden spike to 8%. The error trend tool captures this shape with 5-minute buckets:

| EVAL is_error = CASE(event.outcome == "failure", 1, 0)
| STATS
    total_requests = COUNT(*),
    error_count    = SUM(is_error),
    error_rate     = 100.0 * SUM(is_error) / COUNT(*)
  BY time_bucket = DATE_TRUNC(5 minutes, @timestamp)

The agent reads the most recent bucket's error_rate from this trend and scores it directly against the fixed thresholds (below 1% Green, 1 to 5% Yellow, above 5% Red).

Tool 4: apm_throughput_trend_tool, traffic as a health signal

Throughput is underrated as a health indicator. A 40% drop in requests per second is itself an incident. It can mean a deployment regression, a misconfigured load balancer, or a silent upstream failure.

| STATS requests_count = COUNT(*)
  BY time_bucket = DATE_TRUNC(5 minutes, @timestamp)
| SORT time_bucket ASC

The agent normalizes each 5-minute bucket's request count to a per-second rate and compares it against throughput_rps from apm_metrics_overview_tool.. The same 10% and 30% drift thresholds apply. A service that is suspiciously quiet is flagged just as quickly as one that is on fire.

Tool 5: apm_dependency_health_tool, the blast-radius view

The apm_dependency_health_tool is often the one that finds what the others cannot surface directly. A service can look internally healthy while a downstream database or external API silently accumulates failures. This tool maps every dependency via span.destination.service.resource and computes its error rate from span data:

FROM traces-*
| WHERE processor.event == "span"
  AND span.destination.service.resource IS NOT NULL
| EVAL is_error = CASE(event.outcome == "failure", 1, 0)
| STATS
    total_calls  = COUNT(*),
    failed_calls = SUM(is_error),
    error_rate   = 100.0 * SUM(is_error) / COUNT(*)
  BY dependency_name = span.destination.service.resource
| SORT error_rate DESC

Unlike the other four tools, dependency health is scored independently. Each dependency's error_rate is compared directly against fixed thresholds (1% or below Green, above 1% Yellow, above 5% Red), with no baseline comparison needed.

How the health scoring works

Once all five tools return their data, the agent computes a health state for each metric in a single reasoning step. There is no polling loop and no intermediate storage. Here is exactly what gets compared to what:

The overall verdict follows a single rule: the worst individual metric wins.

Any metric is Red                → Overall = 🔴 Red
Any metric is Yellow (no Red)   → Overall = 🟡 Yellow
All metrics are Green            → Overall = 🟢 Green

The Red/Yellow/Green scoring logic lives in the agent's natural-language instructions. It is readable, auditable, and adjustable without touching code or redeploying infrastructure.

Four metric columns (Latency, Error Rate, Throughput, Dependencies) converging into Red/Yellow/Green outcome states

Deploying the APM Service Health Monitor in Agent Builder

Registration follows Elastic's standard Agent Builder API pattern, tested against Elasticsearch 9.3 and Kibana 9.3. Register the tools first, then wire the agent to all five tool IDs:

# Step 1: Register the five ES|QL tools
POST kbn:/api/agent_builder/tools   # apm_metrics_overview_tool
POST kbn:/api/agent_builder/tools   # apm_latency_trend_tool
POST kbn:/api/agent_builder/tools   # apm_error_trend_tool
POST kbn:/api/agent_builder/tools   # apm_throughput_trend_tool
POST kbn:/api/agent_builder/tools   # apm_dependency_health_tool
 
# Step 2: Register the agent with tools wired in
POST kbn:/api/agent_builder/agents  # apm_service_health_agent
 
# Step 3: Verify the agent is live
GET kbn:/api/agent_builder/agents/apm_service_health_agent

Each tool is parameterized with a service binding, so the same agent serves every service in your fleet with no per-service configuration. The queries target traces-* with cross-cluster wildcard support, so a single deployment covers multi-cluster environments out of the box.

Try it: ask the agent about your service

Once deployed, open the agent in Kibana and type this:

What is the health of my checkout-service in the last 24 hours?

The agent fans out to all five tools, computes the metric states, applies the health logic in one pass, and responds with a structured report.

Kibana chat card: checkout-service · Last 24h · per-metric table with Red/Yellow/Green status pills · Overall: 🔴 Red

The agent identified the root cause without any manual correlation: postgres-primary is failing 6.1% of its calls, and that is cascading directly into the p95 latency spike. No dashboard pivoting, no manual ES|QL. One prompt, full situational awareness.

You can continue the conversation with follow-up questions in the same session:

• Which other services depend on postgres-primary?
• How does this compare to yesterday's health?
• Show me the error trend for the last 6 hours only.

The agent invokes the appropriate tools for each follow-up, keeping the full context of the original health assessment in view.

Why Agent Builder and ES|QL are the right stack for this

A few deliberate choices made this design work cleanly.

ES|QL as the query layer

ES|QL's pipe-based syntax makes each tool query readable, testable, and independently verifiable. The DATE_TRUNC bucketing for trends, the PERCENTILE aggregations for latency, and the span.destination.service.resource grouping for dependencies are precise, auditable queries. Run any of them directly in Kibana Dev Tools and you will see exactly what the agent sees.

Narrow, stateless tools

Each tool does one thing and returns structured data. The agent provides the orchestration and reasoning. Adding a new metric dimension means registering one new tool and updating the agent instructions. Nothing else changes.

Instructions as a runbook

The agent's health logic is expressed in natural language inside its configuration. Anyone on your team can read it, adjust it without a code deploy, and audit it in full from the Agent Builder UI in Kibana.

What's next for the APM Service Health Monitor

The APM Service Health Monitor is a foundation, not a ceiling. Natural extensions include:

• Alert-triggered health checks: wire the agent to fire automatically when an anomaly detection rule triggers, attaching the health summary directly to the alert notification.

• Deployment correlation: integrate change event data so the agent can identify whether a Red status started after a specific deployment.

• SLO-aware thresholds: replace fixed percentage thresholds with per-service error budget consumption, so Red, Yellow, and Green reflect actual business impact against defined SLOs.

• Cross-service traversal: extend the dependency tool to recursively assess upstream and downstream services, building a full topology view from a single query.

Requirements and deployment

Requirements:

• Elasticsearch 9.3 and Kibana 9.3 (tested version).
• Agent Builder enabled in Kibana.
• APM data flowing into traces-* indices via any Elastic APM agent.
• Elastic Enterprise license is required to utilize Agent Builder and ES|QL cross-cluster search.

The full agent and tool definitions are available in this repository. Once your traces are indexed, all five ES|QL tools work against your data with no modification, parameterized by service name at query time, covering your entire service fleet from a single deployment.

The next time someone asks "is the service healthy?", you will have a precise, data-backed answer before the question finishes echoing in the incident channel.

Questions, extensions, or feedback? Join the conversation in the Elastic community forums.

Full deployment reference

Everything you need to deploy the APM Service Health Monitor, in execution order. Run the five tool registrations first, then the agent.

Tool 1: Dependency health

POST kbn:/api/agent_builder/tools
{
  "id": "apm_dependency_health_tool",
  "type": "esql",
  "description": "Evaluates the health of external dependencies (DBs, APIs, caches)...",
  "configuration": {
    "query": "FROM traces-*\n| WHERE service.name == ?service\n| WHERE processor.event == \"span\"\n  AND span.destination.service.resource IS NOT NULL\n| WHERE @timestamp >= NOW() - 24 hours\n| EVAL is_error = CASE(event.outcome == \"failure\", 1, 0)\n| STATS\n    total_calls = COUNT(*),\n    failed_calls = SUM(is_error),\n    error_rate = 100.0 * SUM(is_error)/COUNT(*)\n  BY dependency_name = span.destination.service.resource\n| SORT error_rate DESC | LIMIT 100"
  }
}

Tool 2: Error trend

POST kbn:/api/agent_builder/tools
{
  "id": "apm_error_trend_tool",
  "type": "esql",
  "description": "Tracks 5-minute bucketed error rates for the service over 24 hours...",
  "configuration": {
    "query": "FROM traces-*\n| WHERE service.name == ?service\n| WHERE transaction.type == \"request\"\n| WHERE @timestamp >= NOW() - 24 hours\n| EVAL is_error = CASE(event.outcome == \"failure\", 1, 0)\n| STATS\n    total_requests = COUNT(*),\n    error_count = SUM(is_error),\n    error_rate = 100.0 * SUM(is_error)/COUNT(*)\n  BY time_bucket = DATE_TRUNC(5 minutes, @timestamp)\n| SORT time_bucket ASC | LIMIT 1000"
  }
}

Tool 3: Latency trend

POST kbn:/api/agent_builder/tools
{
  "id": "apm_latency_trend_tool",
  "type": "esql",
  "description": "Provides 5-minute bucketed latency trends (avg, p95, p75) for last 24 hours...",
  "configuration": {
    "query": "FROM traces-*\n| WHERE service.name == ?service\n| WHERE transaction.type == \"request\"\n| WHERE @timestamp >= NOW() - 24 hours\n| STATS\n    avg_latency_ms = AVG(transaction.duration.us / 1000),\n    p95_latency_ms = PERCENTILE(transaction.duration.us / 1000, 95),\n    p75_latency_ms = PERCENTILE(transaction.duration.us / 1000, 75)\n  BY time_bucket = DATE_TRUNC(5 minutes, @timestamp)\n| SORT time_bucket ASC | LIMIT 1000"
  }
}

Tool 4: Metrics overview

POST kbn:/api/agent_builder/tools
{
  "id": "apm_metrics_overview_tool",
  "type": "esql",
  "description": "Aggregates key service metrics over last 24 hours: avg latency, p95/p99, error rate, throughput...",
  "configuration": {
    "query": "FROM traces-*\n| WHERE service.name == ?service\n| WHERE transaction.type == \"request\"\n| WHERE @timestamp >= NOW() - 24 hours\n| EVAL is_error = CASE(event.outcome == \"failure\", 1, 0)\n| STATS\n    avg_latency_ms = AVG(transaction.duration.us / 1000),\n    p95_latency_ms = PERCENTILE(transaction.duration.us / 1000, 95),\n    p99_latency_ms = PERCENTILE(transaction.duration.us / 1000, 99),\n    error_rate = 100.0 * SUM(is_error) / COUNT(*),\n    throughput_rps = COUNT(*) / (24*3600)"
  }
}

Tool 5: Throughput trend

POST kbn:/api/agent_builder/tools
{
  "id": "apm_throughput_trend_tool",
  "type": "esql",
  "description": "Provides service throughput trends in 5-minute intervals over 24 hours...",
  "configuration": {
    "query": "FROM traces-*\n| WHERE service.name == ?service\n| WHERE transaction.type == \"request\"\n| WHERE @timestamp >= NOW() - 24 hours\n| STATS requests_count = COUNT(*)\n  BY time_bucket = DATE_TRUNC(5 minutes, @timestamp)\n| SORT time_bucket ASC | LIMIT 1000"
  }
}

Agent: APM service health monitor

POST kbn:/api/agent_builder/agents
{
  "id": "apm_service_health_agent",
  "type": "chat",
  "name": "APM Service Health Monitor",
  "description": "Provides Red/Yellow/Green health status and trends for services over 24 hours.",
  "labels": ["apm", "health", "service", "monitoring"],
  "avatar_color": "#4CAF50",
  "configuration": {
    "instructions": "You are the Service Health Agent...",
    "tools": [{
      "tool_ids": [
        "apm_dependency_health_tool",
        "apm_error_trend_tool",
        "apm_latency_trend_tool",
        "apm_metrics_overview_tool",
        "apm_throughput_trend_tool"
      ]
    }]
  }
}