Prerequisites for Elastic Agent Builder MCP tools

• An Elastic Cloud deployment with Elastic Stack 9.3+ (or Elastic Cloud Serverless) with Agent Builder enabled.
• An APM-ingested service. If your cluster does not already have APM data, the companion notebook includes instructions to generate synthetic traffic using elastic/apm-integration-testing with the opbeans-node demo app.
• An MCP-compatible client: Claude Code, Cursor, or VS Code with an MCP extension.
• Basic familiarity with ES|QL syntax.
• Node.js 18+ (for the mcp-remote bridge).

If you are new to MCP or need to set up the Elastic MCP server for the first time, check out Connect Agent Builder tools to any AI agent with Elastic MCP server for the full setup walkthrough. This article assumes the MCP server is already configured.

The problem: why developers fly blind

A developer is about to merge a pull request. The change looks simple: increasing the timeout for the downstream recommendations service call from 2 seconds to 5 seconds. But before hitting the merge button, a question lingers: is the service healthy enough to absorb this change right now?

To answer that question today, the developer has two options:

1. Check dashboards manually. Open the APM UI, look at error rates, scan latency charts, find the SLO page, and look for recent deploys. This takes 10-15 minutes and requires knowing what to look for and how to interpret it.
2. Ask an SRE. Ping the platform team on Slack: "Hey, is checkout healthy? I want to merge something." This creates an interruption, adds latency to the decision, and doesn't scale.

The core problem is not the data. Elastic already collects everything: traces, metrics, error logs, deploy markers, and SLO budgets. The problem is that correlating multiple signals requires mental overhead and domain knowledge that most developers don't have.

An SRE knows that a p99 spike after a deploy is normal for 5 minutes, that an error rate under 0.5% is acceptable during a release window, and that merging when the SLO budget is below 20% is risky. That knowledge lives in runbooks, tribal memory, and experience.

What if the platform engineer could encode that knowledge into tools that any developer can query from their editor?

How MCP tools in Elastic Agent Builder encode your runbook

The key insight is this: a tool is not just a query; it is a query plus interpretation. A dashboard shows you a p99 of 450ms. A well-designed tool tells you "p99 is 450ms, which is within normal range for this service, and has been stable since the last deploy 2 hours ago."

The difference is that the tool description carries the domain knowledge. When a platform engineer creates a tool in Agent Builder, they write descriptions like: "Error rate above 1% typically indicates a regression. If this coincides with a recent deploy, the deploy is the likely cause." That description becomes part of the context the AI agent uses when reasoning across multiple tool results.

This is what we mean by contextual AI: the AI agent does not just fetch data; it reasons over it using the interpretation rules that the platform team encoded.

Here is the architecture:

The platform engineer authors the tools once. Every developer on the team benefits from their own editor, without needing to learn ES|QL or understand APM data models.

Setting up the Elastic Agent Builder sample environment

The full end-to-end setup (traffic generation with opbeans-node, deploy annotations, SLO creation, and the three Agent Builder tools) is available as a runnable notebook at this repository: notebook.ipynb. The sections below focus on the ES|QL queries and tool descriptions: the why behind each tool, not the mechanics of posting them.

Building Tool 1: get_endpoint_health

This tool answers the question: "How is this endpoint performing right now?" It returns error rate, latency percentiles (p50, p95, p99), and throughput for a given service and endpoint within a time window.

Here is the full tool configuration as created in Agent Builder:

{
  "id": "get_endpoint_health",
  "type": "esql",
  "description": "Returns the current health of a service endpoint: error rate, latency percentiles (p50/p95/p99), and throughput. Use this tool to assess whether a service is healthy before making changes. Interpretation guide: error rate below 0.5% is healthy, 0.5-1% is elevated (check for recent deploys), above 1% indicates a problem. For latency, compare p99 against the service baseline: checkout is typically under 500ms, product-search under 200ms. A sudden p99 spike within 15 minutes of a deploy suggests the deploy caused a regression.",
  "tags": ["apm", "reliability", "health"],
  "configuration": {
    "query": "FROM traces-apm-* | WHERE service.name == ?serviceName AND @timestamp >= NOW() - ?timeWindow AND transaction.duration.us IS NOT NULL | STATS total_transactions = COUNT(*), error_count = SUM(CASE(event.outcome == \"failure\", 1, 0)), p50_latency_ms = PERCENTILE(transaction.duration.us, 50) / 1000, p95_latency_ms = PERCENTILE(transaction.duration.us, 95) / 1000, p99_latency_ms = PERCENTILE(transaction.duration.us, 99) / 1000 BY service.name | EVAL error_rate_pct = ROUND(error_count / total_transactions * 100, 2) | EVAL throughput_per_min = ROUND(total_transactions / ?windowMinutes, 1)",
    "params": {
      "serviceName": {
        "type": "keyword",
        "description": "The APM service name to check (e.g., opbeans-node)"
      },
      "timeWindow": {
        "type": "keyword",
        "description": "Time window to analyze, in ES|QL duration format (e.g., 30 minutes, 1 hour, 6 hours)"
      },
      "windowMinutes": {
        "type": "integer",
        "description": "Time window in minutes, used to calculate throughput per minute"
      }
    }
  }
}

The query uses the traces-apm-* data stream, which contains raw transaction data. We filter with transaction.duration.us IS NOT NULL to select only transaction events (excluding spans). Using traces-apm-* is more portable than the pre-aggregated metrics-apm.transaction.1m-* stream, which only populates after sustained traffic.

Notice the description field. It is not just "returns health metrics." It includes interpretation rules: what error rate thresholds mean, what latency baselines look like, and how to correlate spikes with deploys. This is the runbook encoded in the tool.

Building Tool 2: get_recent_deploys

This tool answers: "What has been deployed recently?" Deploy history is a critical context because most production issues correlate with code changes. The agent needs this to reason about whether current metrics are normal or reflect a recent deployment.

Deploy annotations are stored in the observability-annotations index. Here is the full tool configuration:

{
  "id": "get_recent_deploys",
  "type": "esql",
  "description": "Returns the deployment history for a service over the last 24 hours, including version numbers, timestamps, and deploy messages. Use this tool to understand the deployment timeline when assessing service health. Key patterns: if a deploy happened within the last 15 minutes, elevated error rates or latency may be expected (warm-up period). If metrics degraded immediately after a deploy, the deploy is the likely cause. Multiple deploys in a short window (under 2 hours) increase risk because it becomes harder to isolate which change caused an issue.",
  "tags": ["apm", "deploys", "change-tracking"],
  "configuration": {
    "query": "FROM observability-annotations | WHERE service.name == ?serviceName AND @timestamp >= NOW() - 24 hours | SORT @timestamp DESC | KEEP @timestamp, service.version, service.environment, message | LIMIT 10",
    "params": {
      "serviceName": {
        "type": "keyword",
        "description": "The APM service name to check deploy history for"
      }
    }
  }
}

Again, the description encodes domain knowledge: the 15-minute warm-up window, the correlation between deploys and metric changes, and the risk of multiple rapid deploys. This is how a platform engineer transfers their intuition into something an AI agent can reason with.

Building Tool 3: get_slo_status

This tool answers: "How much error budget do we have left?" SLO budget is the platform team's quantified way of expressing risk tolerance. If the budget is nearly spent, even a small change could cause a violation.

Unlike the previous tools that query APM data, this one queries the internal SLO indices where Elastic stores pre-computed SLI data. The query calculates the current burn rate, that is, how fast the service is consuming error budget relative to the allowed threshold:

{
  "id": "get_slo_status",
  "type": "esql",
  "description": "Returns the current SLO burn rate for a service over the last hour. The response includes: SLI value (current performance), error budget target, and burn rate percentage. The burn rate tells you how fast the service is consuming error budget relative to the allowed threshold. Interpretation: a burn rate below 100% means the service is consuming budget slower than the limit (sustainable). Between 100-200%, the service is burning budget faster than planned (proceed with caution). Above 200%, the service is burning budget at double the allowed rate (delay non-critical changes). Above 500%, investigate immediately. Note: this measures the current burn rate over the last hour, not cumulative budget consumption over the full SLO window. A temporarily high burn rate does not mean the overall budget is exhausted.",
  "tags": ["slo", "reliability", "budget"],
  "configuration": {
    "query": "FROM .slo-observability.sli-v* | WHERE slo.id == ?sloId AND @timestamp >= NOW() - 1 hour | STATS sli_value = AVG(slo.numerator) / AVG(slo.denominator) BY slo.id, slo.name | EVAL error_budget_target = 0.995 | EVAL burn_rate_pct = ROUND((1 - sli_value) / (1 - error_budget_target) * 100, 1)",
    "params": {
      "sloId": {
        "type": "keyword",
        "description": "The SLO identifier. Use the SLO ID for the service you are evaluating."
      }
    }
  }
}

Note on the SLI index: the version suffix in .slo-observability.sli-v* depends on your Stack release (e.g., v3.6 in Stack 9.3). Verify with GET _cat/indices/.slo-observability.*?v and adjust the pattern if your cluster uses a different version.

The burn rate interpretation rules in the description are the most valuable part. A raw number like "burn rate 85%" means nothing to a developer without context. The tool description translates that into actionable guidance: "below 100% means sustainable, above 200% means delay non-critical changes."

Connecting to your editor via MCP

With all three tools created in Agent Builder, they are automatically available through the MCP server endpoint. Configure your MCP client to connect.

Claude Code configuration

Add the Elastic MCP server to your Claude Code settings:

{
  "mcpServers": {
    "elastic-agent-builder": {
      "command": "npx",
      "args": [
        "mcp-remote",
        "https://your-kibana-url/api/agent_builder/mcp",
        "--header",
        "Authorization: ApiKey your-base64-api-key"
      ]
    }
  }
}

Cursor configuration

For Cursor, add the server in Settings > MCP Servers:

{
  "mcpServers": {
    "elastic-agent-builder": {
      "command": "npx",
      "args": [
        "mcp-remote",
        "https://your-kibana-url/api/agent_builder/mcp",
        "--header",
        "Authorization: ApiKey your-base64-api-key"
      ]
    }
  }
}

Once connected, your editor's AI agent will discover all three tools automatically. You can verify by asking: "What Elastic tools do you have available?" The agent should list get_endpoint_health, get_recent_deploys, and get_slo_status.

API key permissions: the API key needs the feature_agentBuilder.read Kibana privilege and read access to the relevant indices (traces-apm.*, observability-annotations, .slo-observability.*). For production use, set the key expiry to 30-90 days and follow the principle of least privilege.

The scenario: "Is it safe to merge this PR?"

A developer on the team has a pull request that increases the timeout for the downstream recommendations service call from 2 seconds to 5 seconds in opbeans-node. Before merging, they ask the agent:

Developer: "I'm about to merge PR #42, which increases the recommendations service timeout from 2s to 5s in opbeans-node. Is it safe to merge right now?"

The agent begins its multi-signal reasoning chain. Here is what happens.

Step 1: the agent calls get_endpoint_health

The agent checks the current health of the service:

Step 2: the agent calls get_recent_deploys

Next, it checks for recent deployments:

Step 3: the agent calls get_slo_status

The agent's response

After correlating all three results, the agent produces a recommendation:

The agent pulled the current p99, checked recent deploys, and read the SLO burn rate. It combined those signals with the timeout change in the PR, flagged the merge as risky, and recommended next steps.

Conclusion: when to use MCP tools instead of pinging an SRE

With Elasticsearch, Agent Builder, and MCP, a developer can answer questions like "is it safe to merge this PR?" from inside their editor, in seconds, without pinging an SRE. Elasticsearch holds the signals: traces, deploy markers, and SLO budgets. Agent Builder is where the platform team encodes how to read those signals: the thresholds, the warm-up windows, the correlation rules. MCP is what carries those tools into the developer's editor.

The query pulls the data. The description tells the agent how to read it. The platform engineer writes the runbook once, and every developer on the team gets to use it.

Next steps: extend Elastic Agent Builder MCP tools to CI/CD

• Explore the Elastic Agent Builder documentation for more tool types and configuration options.
• See Agentic CI/CD: Kubernetes Deployment Gates with Elastic MCP Server for extending this pattern into your CI/CD pipeline.
• Check out Agent Skills for Elastic Observability for a complementary approach using pre-packaged observability skills.

Why do we need alerts?

Logs, metrics, and traces are just the base to build a complete monitoring solution for Kubernetes clusters. Their main goal is to provide debugging information and historical evidence for the infrastructure.

While out-of-the-box dashboards, infrastructure topology, and logs exploration through Kibana are already quite handy to perform ad-hoc analyses, adding notifications and active monitoring of infrastructure allows users to deal with problems detected as early as possible and even proactively take actions to prevent their Kubernetes environments from facing even more serious issues.

How can this be achieved?

By building alerts on top of their infrastructure, users can leverage the data and effectively correlate it to a specific notification, creating a wide range of possibilities to dynamically monitor and observe their Kubernetes cluster.

In this blog post, we will explore how users can leverage Elasticsearch’s search powers to define alerting rules in order to be notified when a specific condition occurs.

SLIs, alerts, and SLOs: Why are they important for SREs?

For site reliability engineers (SREs), the incident response time is tightly coupled with the success of everyday work. Monitoring, alerting, and actions will help to discover, resolve, or prevent issues in their systems.

• An SLA (Service Level Agreement) is an agreement you create with your users to specify the level of service they can expect.
• An SLO (Service Level Objective) is an agreement within an SLA about a specific metric like uptime or response time.
• An SLI (Service Level Indicator) measures compliance with an SLO.

SREs’ day-to-day tasks and projects are driven by SLOs. By ensuring that SLOs are defended in the short term and that they can be maintained in the medium to long term, we lay the basis of a stable working infrastructure.

Having said this, identifying the high-level categories of SLOs is crucial in order to organize the work of an SRE. Then in each category of SLOs, SREs will need the corresponding SLIs that can cover the most important cases of their system under observation. Therefore, the decision of which SLIs we will need demands additional knowledge of the underlying system infrastructure.

One widely used approach to categorize SLIs and SLOs is the Four Golden Signals method. The categories defined are Latency, Traffic, Errors, and Saturation.

A more specific approach is the The RED method developed by Tom Wilkie, who was an SRE at Google and used the Four Golden Signals. The RED method drops the saturation category because this one is mainly used for more advanced cases — and people remember better things that come in threes.

Focusing on Kubernetes infrastructure operators, we will consider the following groups of infrastructure SLIs/SLOs:

• Group 1: Latency of control plane (apiserver,
• Group 2: Resource utilization of the nodes/pods (how much cpu, memory, etc. is consumed)
• Group 3: Errors (errors on logs or events or error count from components, network, etc.)

Creating alerts for a Kubernetes cluster

Now that we have a complete outline of our goal to define alerts based on SLIs/SLOs, we will dive into defining the proper alerting. Alerts can be built using Kibana.

See Elastic documentation.

In this blog, we will define more complex alerts based on complex Elasticsearch queries provided by Watcher’s functionality. Read more about Watcher and how to properly use it in addition to the examples in this blog.

Latency alerts

For this kind of alert, we want to define the basic SLOs for a Kubernetes control plane, which will ensure that the basic control plane components can service the end users without an issue. For instance, facing high latencies in queries against the Kubernetes API Server is enough of a signal that action needs to be taken.

Resource saturation

The next group of alerting will be resource utilization. Node’s CPU utilization or changes in Node’s condition is something critical for a cluster to ensure the smooth servicing of the workloads provisioned to run the applications that end users will interact with.

Error detection

Last but not least, we will define alerts based on specific errors like the network error rate or Pods’ failures like the OOMKilled situation. It’s a very useful indicator for SRE teams to either detect issues on the infrastructure level or just be able to notify developer teams about problematic workloads. One example that we will examine later is having an application running as a Pod and constantly getting restarted because it hits its memory limit. In that case, the owners of this application will need to get notified to act properly.

From Kubernetes data to Elasticsearch queries

Having a solid plan about the alerts that we want to implement, it's time to explore the data we have collected from the Kubernetes cluster and stored in Elasticsearch. For this we will consult the list of the available data fields that are ingested using the Elastic Agent Kubernetes integration (the full list of fields can be found here). Using these fields we can create various alerts like:

• Node CPU utilization
• Node Memory utilization
• BW utilization
• Pod restarts
• Pod CPU/memory utilization

CPU utilization alert

Our first example will use the CPU utilization fields to calculate the Node’s CPU utilization and create an alert. For this alert, we leverage the metrics:

kubernetes.node.cpu.usage.nanocores
kubernetes.node.cpu.capacity.cores.

The following calculation (nodeUsage / 1000000000 ) /nodeCap grouped by node name will give us the CPU utilization of our cluster’s nodes.

The Watcher definition that implements this query can be created with the following API call to Elasticsearch:

curl -X PUT "https://elastic:changeme@localhost:9200/_watcher/watch/Node-CPU-Usage?pretty" -k -H 'Content-Type: application/json' -d'
{
  "trigger": {
    "schedule": {
      "interval": "10m"
    }
  },
  "input": {
    "search": {
      "request": {
        "body": {
          "size": 0,
          "query": {
            "bool": {
              "must": [
                {
                  "range": {
                    "@timestamp": {
                      "gte": "now-10m",
                      "lte": "now",
                      "format": "strict_date_optional_time"
                    }
                  }
                },
                {
                  "bool": {
                    "must": [
                      {
                        "query_string": {
                          "query": "data_stream.dataset: kubernetes.node OR data_stream.dataset: kubernetes.state_node",
                          "analyze_wildcard": true
                        }
                      }
                    ],
                    "filter": [],
                    "should": [],
                    "must_not": []
                  }
                }
              ],
              "filter": [],
              "should": [],
              "must_not": []
            }
          },
          "aggs": {
            "nodes": {
              "terms": {
                "field": "kubernetes.node.name",
                "size": "10000",
                "order": {
                  "_key": "asc"
                }
              },
              "aggs": {
                "nodeUsage": {
                  "max": {
                    "field": "kubernetes.node.cpu.usage.nanocores"
                  }
                },
                "nodeCap": {
                  "max": {
                    "field": "kubernetes.node.cpu.capacity.cores"
                  }
                },
                "nodeCPUUsagePCT": {
                  "bucket_script": {
                    "buckets_path": {
                      "nodeUsage": "nodeUsage",
                      "nodeCap": "nodeCap"
                    },
                    "script": {
                      "source": "( params.nodeUsage / 1000000000 ) / params.nodeCap",
                      "lang": "painless",
                      "params": {
                        "_interval": 10000
                      }
                    },
                    "gap_policy": "skip"
                  }
                }
              }
            }
          }
        },
        "indices": [
          "metrics-kubernetes*"
        ]
      }
    }
  },
  "condition": {
    "array_compare": {
      "ctx.payload.aggregations.nodes.buckets": {
        "path": "nodeCPUUsagePCT.value",
        "gte": {
          "value": 80
        }
      }
    }
  },
  "actions": {
    "log_hits": {
      "foreach": "ctx.payload.aggregations.nodes.buckets",
      "max_iterations": 500,
      "logging": {
        "text": "Kubernetes node found with high CPU usage: {{ctx.payload.key}} -> {{ctx.payload.nodeCPUUsagePCT.value}}"
      }
    }
  },
  "metadata": {
    "xpack": {
      "type": "json"
    },
    "name": "Node CPU Usage"
  }
}

OOMKilled Pods detection and alerting

Another Watcher that we will explore is the one that detects Pods that have been restarted due to an OOMKilled error. This error is quite common in Kubernetes workloads and is useful to detect this early on to inform the team that owns this workload, so they can either investigate issues that could cause memory leaks or just consider increasing the required resources for the workload itself.

This information can be retrieved from a query like the following:

kubernetes.container.status.last_terminated_reason: OOMKilled

Here is how we can create the respective Watcher with an API call:

curl -X PUT "https://elastic:changeme@localhost:9200/_watcher/watch/Pod-Terminated-OOMKilled?pretty" -k -H 'Content-Type: application/json' -d'
{
  "trigger": {
    "schedule": {
      "interval": "1m"
    }
  },
  "input": {
    "search": {
      "request": {
        "search_type": "query_then_fetch",
        "indices": [
          "*"
        ],
        "rest_total_hits_as_int": true,
        "body": {
          "size": 0,
          "query": {
            "bool": {
              "must": [
                {
                  "range": {
                    "@timestamp": {
                      "gte": "now-1m",
                      "lte": "now",
                      "format": "strict_date_optional_time"
                    }
                  }
                },
                {
                  "bool": {
                    "must": [
                      {
                        "query_string": {
                          "query": "data_stream.dataset: kubernetes.state_container",
                          "analyze_wildcard": true
                        }
                      },
                      {
                        "exists": {
                          "field": "kubernetes.container.status.last_terminated_reason"
                        }
                      },
                      {
                        "query_string": {
                          "query": "kubernetes.container.status.last_terminated_reason: OOMKilled",
                          "analyze_wildcard": true
                        }
                      }
                    ],
                    "filter": [],
                    "should": [],
                    "must_not": []
                  }
                }
              ],
              "filter": [],
              "should": [],
              "must_not": []
            }
          },
          "aggs": {
            "pods": {
              "terms": {
                "field": "kubernetes.pod.name",
                "order": {
                  "_key": "asc"
                }
              }
            }
          }
        }
      }
    }
  },
  "condition": {
    "array_compare": {
      "ctx.payload.aggregations.pods.buckets": {
        "path": "doc_count",
        "gte": {
          "value": 1,
          "quantifier": "some"
        }
      }
    }
  },
  "actions": {
    "ping_slack": {
      "foreach": "ctx.payload.aggregations.pods.buckets",
      "max_iterations": 500,
      "webhook": {
        "method": "POST",
        "url": "https://hooks.slack.com/services/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",
        "body": "{\"channel\": \"#k8s-alerts\", \"username\": \"k8s-cluster-alerting\", \"text\": \"Pod {{ctx.payload.key}} was terminated with status OOMKilled.\"}"
      }
    }
  },
  "metadata": {
    "xpack": {
      "type": "json"
    },
    "name": "Pod Terminated OOMKilled"
  }
}

From Kubernetes data to alerts summary

So far we saw how we can start from plain Kubernetes fields, use them in ES queries, and build Watchers and alerts on top of them.

One can explore more possible data combinations and build queries and alerts following the examples we provided here. A full list of alerts is available, as well as a basic scripted way of installing them.

Of course, these examples come with simple actions defined that only log messages into the Elasticsearch logs. However, one can use more advanced and useful outputs like Slack’s webhooks:

"actions": {
    "ping_slack": {
      "foreach": "ctx.payload.aggregations.pods.buckets",
      "max_iterations": 500,
      "webhook": {
        "method": "POST",
        "url": "https://hooks.slack.com/services/T04SW3JHXasdfasdfasdfasdfasdf",
        "body": "{\"channel\": \"#k8s-alerts\", \"username\": \"k8s-cluster-alerting\", \"text\": \"Pod {{ctx.payload.key}} was terminated with status OOMKilled.\"}"
      }
    }
  }

The result would be a Slack message like the following:

Next steps

In our next steps, we would like to make these alerts part of our Kubernetes integration, which would mean that the predefined alerts would be installed when users install or enable the Kubernetes integration. At the same time, we plan to implement some of these as Kibana’s native SLIs, providing the option to our users to quickly define SLOs on top of the SLIs through a nice user interface. If you’re interested to learn more about these, follow the public GitHub issues for more information and feel free to provide your feedback:

• https://github.com/elastic/package-spec/issues/484
• https://github.com/elastic/kibana/issues/150050

For those who are eager to start using Kubernetes alerting today, here is what you need to do:

1. Make sure that you have an Elastic cluster up and running. The fastest way to deploy your cluster is to spin up a free trial of Elasticsearch Service.
2. Install the latest Elastic Agent on your Kubernetes cluster following the respective documentation.
3. Install our provided alerts that can be found at https://github.com/elastic/integrations/tree/main/packages/kubernetes/docs or at https://github.com/elastic/k8s-integration-infra/tree/main/scripts/alerting.

Of course, if you have any questions, remember that we are always happy to help on the Discuss forums.

The Kubernetes OpenTelemetry Assets package now ships four Kubernetes SLO templates for Deployments, StatefulSets, DaemonSets, and Jobs on those OTel metrics. If you already followed Part 1 and have the dashboards and alert rules, create an SLO from a template and you get SLI, remaining budget, and burn rate without new instrumentation.

The diagram above extends the stack from Part 1. The Kubernetes OpenTelemetry Assets package (kubernetes_otel) 2.3.0 includes:

• Dashboards designed for drill-down (Part 1)
• Alert rule templates that fire on known bad states (Part 1)
• ML anomaly detection jobs with workload baselines (Part 1)
• SLO templates for rolling 30-day budgets (this post)

All four use the same OTel metrics. Burn rate alerts on an SLO send you back into Overview, Workloads, and Deployment Details when the number alone is not enough.

Why Kubernetes observability needs SLO monitoring alongside alerts

Part 1 built the reactive stack for the engineer who gets paged at 3 AM. SLOs serve the planning conversation on a 30-day horizon: Are we meeting our reliability commitments? They give platform and engineering leaders a number for prioritisation: how much error budget remains and which workload is burning it fastest. The table later in this post maps each SLO template to its Part 1 alert counterpart.

The SLO templates in this post are part of the Kubernetes OpenTelemetry Assets package (kubernetes_otel). Install the Kubernetes OpenTelemetry Assets package and confirm your cluster is already sending Kubernetes metrics through OpenTelemetry (the same pipeline from Part 1). No additional instrumentation is required.

Four SLO templates for Kubernetes Deployments, StatefulSets, DaemonSets and Jobs

In Integrations → Kubernetes OpenTelemetry → Assets, enable any of the four templates below. Names match Kibana; each includes the [Kubernetes OTel] prefix in the UI.

Each is a timeslice-metric SLO: Elastic marks every five-minute window good or bad, then rolls those results into a 30-day rolling objective per namespace and workload.

Reliability is scored at two levels. Each five-minute slice gets one verdict: Elastic aggregates OTel metrics in that window, evaluates the template equation, and compares the result to the metric threshold. For Deployments, that is sum(available) / sum(desired) >= 1. At a ~30-second OTel scrape cadence, that is roughly ten measurements per slice, and the slice passes or fails on the aggregated result. The SLO target (99.5% or 99.0%) is the share of slices that must pass across the rolling window. Over 30 days at five-minute slices, that is 8,640 possible slices per workload (30 × 24 × 12). After you create an SLO from a template, the SLO detail view shows how many slices passed and how much error budget remains.

At 99.5%, a workload can miss roughly 43 of those slices (~3.6 hours of bad slices) before breach. At 99.0%, about 86 slices (~7.2 hours).

How to set SLO targets by Kubernetes workload type

We picked defaults per workload type, not one number for the whole cluster.

Deployments and StatefulSets at 99.5%: We considered 99.9% (~43 minutes per month), which fits a single critical API or a formal SLA buffer. For a default integration template across many Deployments, 99.5% (~3.6 hours) leaves room for normal rollout churn: a 20-minute bad image tag is roughly half the monthly budget at 99.9%, but a small fraction at 99.5%. Tune per workload; payment paths often warrant 99.9% or higher.

DaemonSets and Jobs at 99.0%: We considered 99.5% for DaemonSets, but node additions, replacements, and rolling updates often leave ready_nodes below desired_scheduled_nodes for several minutes per event. At 99.5%, that normal platform churn would burn error budget on infrastructure agents (log collectors, monitoring, CNI) as if they were user-facing outages. 99.0% (~7.2 hours) absorbs that lifecycle noise. Jobs get 99.0% for a different reason: a failed ETL run usually hurts data freshness, not live request availability, and failures can sit unnoticed until downstream teams see stale reports.

Deployment replica availability (99.5%)

Metric:     sum(k8s.deployment.available) / sum(k8s.deployment.desired) >= 1
Target:     99.5% of 5-minute timeslices over 30 days
Group by:   resource.attributes.k8s.namespace.name + resource.attributes.k8s.deployment.name

When available < desired, the application runs fewer healthy replicas than configured. Failed rollouts, crash loops, and node loss all show up here. 99.5% leaves roughly 3.6 hours of degradation per deployment per month before breach.

Grouping by namespace and deployment name creates one SLO per workload. A cluster-wide average would let a healthy frontend mask a burning checkout. Linked dashboards (Overview and Workloads) connect the SLO view to investigation context in one click; from Workloads you drill into Deployment Detail for the failing deployment.

StatefulSet replica availability (99.5%)

Metric:     sum(k8s.statefulset.ready_pods) / sum(k8s.statefulset.desired_pods) >= 1
Target:     99.5% of 5-minute timeslices over 30 days
Group by:   resource.attributes.k8s.namespace.name + resource.attributes.k8s.statefulset.name

When ready_pods < desired_pods, the StatefulSet reports fewer Ready replicas than configured. Ordered rollouts, stuck pods, and node loss show up here too. Rollouts proceed in order, and each pod keeps its name and volume, so a missing replica can stay below desired longer than a stateless pod would.

Grouping by namespace and StatefulSet name avoids a healthy workload masking another that is burning the SLO budget. 

DaemonSet scheduling availability (99.0%)

Metric:     sum(k8s.daemonset.ready_nodes) / sum(k8s.daemonset.desired_scheduled_nodes) >= 1
Target:     99.0% of 5-minute timeslices over 30 days
Group by:   resource.attributes.k8s.namespace.name + resource.attributes.k8s.daemonset.name

DaemonSets run node-level infrastructure: log collectors, monitoring agents, security agents, and network plugins. When ready_nodes < desired_scheduled_nodes, an eligible node lacks a Ready pod, which can leave that node without logs or metrics from that agent. Rolling updates and new nodes drive most gaps; pods that never become Ready show the same signal. Cordoned nodes often still run DaemonSet pods. 99.0% (~7.2 hours per month) reflects that churn. 

Group by namespace and DaemonSet name so a healthy fluentd does not mask a broken node-exporter on the same SLO budget.

Job completion success rate (99.0%)

Metric:     max(k8s.job.failed_pods) <= 0
Target:     99.0% of 5-minute timeslices over 30 days
Group by:   resource.attributes.k8s.namespace.name + resource.attributes.k8s.job.name

Jobs cover batch workloads: ETL pipelines, backups, database migrations, and scheduled reports. When failed_pods > 0, at least one pod created by the Job reached the Failed phase. Application errors, timeouts, and missing dependencies drive many failures; when retries reach the configured backoffLimit, Kubernetes marks the Job as Failed. Missed runs often surface as stale or delayed data, not as a serving outage. 99.0% (~7.2 hours per month) reflects that occasional batch failure is less time-sensitive than a Deployment or StatefulSet breach. 

How do SLOs and alerts work together in Kubernetes observability?

The SLO templates and the alert rules from Part 1 serve different people asking different questions at different times.

Alert rules answer: Is something broken right now? They fire within minutes, page the on-call engineer, and expect immediate action.

SLO templates answer: Are we meeting our reliability commitments over time? They accumulate signal across weeks and turn prioritisation debates into a number tied to remaining budget.

From incident to error budget burn: a Kubernetes walkthrough

A deployment drops from 3/3 to 2/3 available replicas during a rolling update. The new pod fails its readiness probe. Here is what happened in our test cluster, from dashboard signal through alert, SLO impact, and root cause.

Rollout begins. The Deployment dashboard shows available: 2, desired: 3. The Part 1 Deployment unavailable replicas rule has a 5-minute grace period, so on-call is not paged yet during a short rollout gap. The Deployment Detail view for web-frontend shows available replicas dropping while desired stays at 3. The Deployment replicas over time chart marks where the rollout started to fail.

Alert fires, then root cause. After the grace period, the alert rule triggers: Deployment unavailable replicas. The on-call engineer opens the Workloads dashboard, finds web-frontend at available: 2, desired: 3, and drills into Deployment Detail. The replicas-over-time chart confirms when availability dropped. 

In Discover, filter Kubernetes events for that pod with k8s.object.name: "web-frontend-796fcd55b9-jmlkh". The event stream shows ImagePullBackOff and Back-off pulling image "nginx:nonexistent-tag-999". The rollout references an image tag that does not exist.

Rollback and recovery. The engineer runs kubectl rollout undo deployment/web-frontend. Replicas return to 3/3.

How two rollouts consumed 88% of a 30-day error budget

The rollback fixed availability. The SLO still counted the day's failures.

Two rollout failures left web-frontend with 39 failed timeslices where available < desired. That consumed 88.0% of the 30-day error budget. The SLI still read 99.56%, above the 99.5% target, but only 12% of the monthly allowance remained.

The burn rate alert fired next, even though replicas were healthy again. Over the past day the deployment consumed budget at 26× the rate a 99.5% SLO can sustain long term. Each failed 5-minute timeslice uses roughly 2.3% of the monthly budget (about 43 failures allowed per 30 days). Thirty-nine failures across two rollouts is worth a reliability review, not a one-line postmortem. The burn rate alert often matters more than the raw SLI mid-month because it fires while you still have budget left to spend deliberately.

Try it yourself: trigger an error budget burn on a test Deployment

If you already have Kubernetes OpenTelemetry Assets installed, the SLO templates live under Integrations → Kubernetes OpenTelemetry → Assets.

Create a Deployment replica availability SLO for the deployment you use below. Open the SLO and note the baseline: current SLI, remaining error budget, and existing timeslice history.

Create an isolated namespace and a small deployment so the exercise does not affect production workloads. Wait a few minutes for the OTel collector to scrape metrics before you create the SLO.

kubectl create namespace blog-demo
kubectl create deployment web-frontend --namespace blog-demo --image=nginx:latest --replicas=3

Trigger a bad rollout with a non-existent image tag:

kubectl get deployment web-frontend -n blog-demo
kubectl set image deployment/web-frontend nginx=nginx:nonexistent-tag-999 --namespace blog-demo

Within a few minutes a new pod enters ImagePullBackOff, available replicas drop below desired, and the SLO records failed timeslices. Roll back to recover:

kubectl rollout undo deployment/web-frontend -n blog-demo

Refresh the SLO view. You should see new failed timeslices in the 30-day history and a reduction in remaining error budget.

One failed timeslice consumes about 2.3% of the monthly error budget at 99.5%. Repeat that across deployments in a week and the burn rate alert becomes the prioritisation signal.

When you are done, delete the test namespace with:

kubectl delete namespace blog-demo

What's next: from SLO monitoring to agentic remediation

Alerts tell you replicas dropped. SLOs tell you how much monthly budget that cost. In the walkthrough above, the same ImagePullBackOff showed up in Deployment Detail, the unavailable-replicas alert, and failed timeslices on the replica-availability SLO, all from the OTel pipeline you installed in Part 1. The SLI still read 99.56% while 88% of the monthly error budget was gone.

Part 1 closed by previewing Agentic Investigations: investigation workflows that run when an alert fires, with skills, tools, and MCP views. This post adds the SLO layer on those same metrics so you can quantify reliability debt before automating runbooks. A follow-up post will cover that agentic workflow and propose remediations you review before applying.

Which remediations would you trust a workflow to suggest on a Kubernetes incident, and which would you keep manual? Join the Elastic Community discussion.

Since version 8.8, we have a built in functionality to calculate SLOs — check out our guide!

Foundations of calculating an SLA

There are many ways to calculate and measure an SLA. The most important part is the definition of the SLA, and as a consultant, I’ve seen many different ways. Some examples include:

• Count of HTTP 2xx must be above 98% of all HTTP status
• Response time of successful HTTP 2xx requests must be below x milliseconds
• Synthetic monitor must be up at least 99%
• 95% of all batch transactions from the billing service need to complete within 4 seconds

Depending on the origin of the data, calculating the SLA can be easier or more difficult. For uptime (Synthetic Monitoring), we automatically provide SLA values and offer out-of-the-box alerts to simply define alert when availability below 98% for the last 1 hour.

I personally recommend using Elastic Synthetic Monitoring whenever possible to monitor service performance. Running HTTP requests and verifying the answers from the service, or doing fully fledged browser monitors and clicking through the website as a real user does, ensures a better understanding of the health of your service.

Sometimes this is impossible because you want to calculate the uptime of a specific Windows Service that does not offer any TCP port or HTTP interaction. Here the caveat applies that just because the service is running, it does not necessarily imply that the service is working fine.

Transforms to the rescue

We have identified our important service. In our case, it is the Steam Client Helper. There are two ways to solve this.

Lens formula

You can use Lens and formula (for a deep dive into formulas, check out this blog). Use the Search bar to filter down the data you want. Then use the formula option in Lens. We are dividing all counts of records with Running as a state and dividing it by the overall count of records. This is a nice solution when there is a need to calculate quickly and on the fly.

count(kql='windows.service.state: "Running" ')/count()

Using the formula posted above as the bar chart's vertical axis calculates the uptime percentage. We use an annotation to mark why there is a dip and why this service was below the threshold. The annotation is set to reboot, which indicates a reboot happening, and thus, the service was down for a moment. Lastly, we add a reference line and set this to our defined threshold at 98%. This ensures that a quick look at the visualization allows our eyes to gauge if we are above or below the threshold.

Transform

What if I am not interested in just one service, but there are multiple services needed for your SLA? That is where Transforms can solve this problem. Furthermore, the second issue is that this data is only available inside the Lens. Therefore, we cannot create any alerts on this.

Go to Transforms and create a pivot transform.

1. Add the following filter to narrow it to only services data sets: data_stream.dataset: "windows.service". If you are interested in a specific service, you can always add it to the search bar if you want to know if a specific remote management service is up in your entire fleet!

2. Select data histogram(@timestamp) and set it to your chosen unit. By default, the Elastic Agent only collects service states every 60 seconds. I am going with 1 hour.

3. Select agent.name and windows.service.name as well.

4. Now we need to define an aggregation type. We will use a value_count of windows.service.state. That just counts how many records have this value.

5. Rename the value_count to total_count.

6. Add value_count for windows.service.state a second time and use the pencil icon to edit it to terms, which aggregates for running.

7. This opens up a sub-aggregation. Once again, select value_count(windows.service.state) and rename it to values.

8. Now, the preview shows us the count of records with any states and the count of running.

9. Here comes the tricky part. We need to write some custom aggregations to calculate the percentage of uptime. Click on the copy icon next to the edit JSON config.

10. In a new tab, go to Dev Tools. Paste what you have in the clipboard.

11. Press the play button or use the keyboard shortcut ctrl+enter/cmd+enter and run it. This will create a preview of what the data looks like. It should give you the same information as in the table preview.

12. Now, we need to calculate the percentage of up, which means doing a bucket script where we divide running.values by total_count, just like we did in the Lens visualization. Suppose you name the columns differently or use more than a single value. In that case, you will need to adapt accordingly.

"availability": {
        "bucket_script": {
          "buckets_path": {
            "up": "running>values",
            "total": "total_count"
          },
          "script": "params.up/params.total"
        }
      }

13. This is the entire transform for me:

POST _transform/_preview
{
  "source": {
    "index": [
      "metrics-*"
    ]
  },
  "pivot": {
    "group_by": {
      "@timestamp": {
        "date_histogram": {
          "field": "@timestamp",
          "calendar_interval": "1h"
        }
      },
      "agent.name": {
        "terms": {
          "field": "agent.name"
        }
      },
      "windows.service.name": {
        "terms": {
          "field": "windows.service.name"
        }
      }
    },
    "aggregations": {
      "total_count": {
        "value_count": {
          "field": "windows.service.state"
        }
      },
      "running": {
        "filter": {
          "term": {
            "windows.service.state": "Running"
          }
        },
        "aggs": {
          "values": {
            "value_count": {
              "field": "windows.service.state"
            }
          }
        }
      },
      "availability": {
        "bucket_script": {
          "buckets_path": {
            "up": "running>values",
            "total": "total_count"
          },
          "script": "params.up/params.total"
        }
      }
    }
  }
}

14. The preview in Dev Tools should work and be complete. Otherwise, you must debug any errors. Most of the time, it is the bucket script and the path to the values. You might have called it up instead of running. This is what the preview looks like for me.

{
  "running": {
    "values": 1
  },
  "agent": {
    "name": "AnnalenasMac"
  },
  "@timestamp": "2021-12-07T19:00:00.000Z",
  "total_count": 1,
  "availability": 1,
  "windows": {
    "service": {
      "name": "InstallService"
    }
  }
},

15. Now we only paste the bucket script into the transform creation UI after selecting Edit JSON. It looks like this:

16. Give your transform a name, set the destination index, and run it continuously. When selecting this, please also make sure not to use @timestamp. Instead, opt for event.ingested. Our documentation explains this in detail.

17. Click next and create and start. This can take a bit, so don’t worry.

To summarize, we have now created a pivot transform using a bucket script aggregation to calculate the running time of a service in percentage. There is a caveat because Elastic Agent, per default, only collects the every 60 seconds the services state. It can be that a service is up exactly when collected and down a few seconds later. If it is that important and no other monitoring possibilities, such as Elastic Synthetics are possible, you might want to reduce the collection time on the Agent side to get the services state every 30 seconds, 45 seconds. Depending on how important your thresholds are, you can create multiple policies having different collection times. This ensures that a super important server might collect the services state every 10 seconds because you need as much granularity and insurance for the correctness of the metric. For normal workstations where you just want to know if your remote access solution is up the majority of the time, you might not mind having a single metric every 60 seconds.

After you have created the transform, one additional feature you get is that the data is stored in an index, similar to in Elasticsearch. When you just do the visualization, the metric is calculated for this visualization only and not available anywhere else. Since this is now data, you can create a threshold alert to your favorite connection (Slack, Teams, Service Now, Mail, and so many more to choose from).

Visualizing the transformed data

The transform created a data view called windows-service. The first thing we want to do is change the format of the availability field to a percentage. This automatically tells Lens that this needs to be formatted as a percentage field, so you don’t need to select it manually as well as do calculations. Furthermore, in Discover, instead of seeing 0.5 you see 50%. Isn’t that cool? This is also possible for durations, like event.duration if you have it as nanoseconds! No more calculations on the fly and thinking if you need to divide by 1,000 or 1,000,000.

We get this view by using a simple Lens visualization with a timestamp on the vertical axis with the minimum interval for 1 day and an average of availability. Don’t worry — the other data will be populated once the transformation finishes. We can add a reference line using the value 0.98 because our target is 98% uptime of the service.

Summary

This blog post covered the steps needed to calculate the SLA for a specific data set in Elastic Observability, as well as how to visualize it. Using this calculation method opens the door to a lot of interesting use cases. You can change the bucket script and start calculating the number of sales, and the average basket size. Interested in learning more about Elastic Synthetics? Read our documentation or check out our free Synthetic Monitoring Quick Start training.

Site reliability engineers (SREs) face the challenging task of sifting through vast quantities of data, not just from the applications themselves but also from the underlying infrastructure. In addition to data analysis, they are responsible for ensuring the effective use and development of operational tools. The growing volume of data, the daily resolution of issues, and the continuous evolution of tools and processes can detract from the focus on business performance.

Elastic Observability offers a solution to this challenge. It enables SREs to integrate and examine all telemetry data (logs, metrics, traces, and profiling) in conjunction with business metrics. This comprehensive approach to data analysis fosters operational excellence, boosts productivity, and yields critical insights, all of which are integral to maintaining high-performing applications in a demanding digital environment.

To help manage operations and business metrics, Elastic Observability's SLO (Service Level Objectives) feature was introduced in 8.12. This feature enables setting measurable performance targets for services, such as availability, latency, traffic, errors, and saturation or define your own. Key components include:

• Defining and monitoring SLIs (Service Level Indicators)

• Monitoring error budgets indicating permissible performance shortfalls

• Alerting on burn rates showing error budget consumption

Users can monitor SLOs in real-time with dashboards, track historical performance, and receive alerts for potential issues. Additionally, SLO dashboard panels offer customized visualizations.

Service Level Objectives (SLOs) are generally available for our Platinum and Enterprise subscription customers.

In this blog, we will outline the following:

• What are SLOs? A Google SRE perspective

• Several scenarios of defining and managing SLOs

Service Level Objective overview

Service Level Objectives (SLOs) are a crucial component for Site Reliability Engineering (SRE), as detailed in Google's SRE Handbook. They provide a framework for quantifying and managing the reliability of a service. The key elements of SLOs include:

• Service Level Indicators (SLIs): These are carefully selected metrics, such as uptime, latency, throughput, error rates, or other important metrics, that represent the aspects of the service and are important from an operations or business perspective. Hence, an SLI is a measure of the service level provided (latency, uptime, etc.), and it is defined as a ratio of good over total events, with a range between 0% and 100%.

• Service Level Objective (SLO): An SLO is the target value for a service level measured as a percentage by an SLI. Above the threshold, the service is compliant. As an example, if we want to use service availability as an SLI, with the number of successful responses at 99.9%, then any time the number of failed responses is > .1%, the SLO will be out of compliance.

• Error budget: This represents the threshold of acceptable errors, balancing the need for reliability with practical limits. It is defined as 100% minus the SLO quantity of errors that is tolerated.

• Burn rate: This concept relates to how quickly the service is consuming its error budget, which is the acceptable threshold for unreliability agreed upon by the service providers and its users.

Understanding these concepts and effectively implementing them is essential for maintaining a balance between innovation and reliability in service delivery. For more detailed information, you can refer to Google's SRE Handbook.

One main thing to remember is that SLO monitoring is not incident monitoring. SLO monitoring is a proactive, strategic approach designed to ensure that services meet established performance standards and user expectations. It involves tracking Service Level Objectives, error budgets, and the overall reliability of a service over time. This predictive method helps in preventing issues that could impact users and aligns service performance with business objectives.

In contrast, incident monitoring is a reactive process focused on detecting, responding to, and mitigating service incidents as they occur. It aims to address unexpected disruptions or failures in real time, minimizing downtime and impact on service. This includes monitoring system health, errors, and response times during incidents, with a focus on rapid response to minimize disruption and preserve the service's reputation.

Elastic®’s SLO capability is based directly off the Google SRE Handbook. All the definitions and semantics are utilized as described in Google’s SRE handbook. Hence users can perform the following on SLOs in Elastic:

• Define an SLO on an SLI such as KQL (log based query), service availability, service latency, custom metric, histogram metric, or a timeslice metric. Additionally, set the appropriate threshold.

• Utilize occurrence versus time slice based budgeting. Occurrences is the number of good events over the number of total events to compute the SLO. Timeslices break the overall time window into slammer slices of a defined duration and compute the number of good slices over the total slices to compute the SLO. Timeslice targets are more accurate and useful when calculating things like a service’s SLO when trying to meet agreed upon customer targets.

• Manage all the SLOs in a singular location.

• Trigger alerts from the defined SLO, whether the SLI is off, burn rate is used up, or the error rate is X.

• Create unique service level dashboards with SLO information for a more comprehensive view of the service.

SREs need to be able to manage business metrics.

SLOs based on logs: NGINX availability

Defining SLOs does not always mean metrics need to be used. Logs are a rich form of information, even when they have metrics embedded in them. Hence it’s useful to understand your business and operations status based on logs.

Elastic allows you to create an SLO based on specific fields in the log message, which don’t have to be metrics. A simple example is a simple multi-tier app that has a web server layer (nginx), a processing layer, and a database layer.

Let’s say that your processing layer is managing a significant number of requests. You want to ensure that the service is up properly. The best way is to ensure that all http.response.status_code are less than 500. Anything less ensures the service is up and any errors (like 404) are all user or client errors versus server errors.

If we use Discover in Elastic, we see that there are close to 2M log messages over a seven-day time frame.

Additionally, the number of messages with http.response.status_code > 500 is minimal, like 17K.

Rather than creating an alert, we can create an SLO with this query:

We chose to use occurrences as the budgeting method to keep things simple.

Once defined, we can see how well our SLO is performing over a seven-day time frame. We can see not only the SLO, but also the burn rate, the historical SLI, and error budget, and any specific alerts against the SLO.

Not only do we get information about the violation, but we also get:

• Historical SLI (7 days)

• Error budget burn down

• Good vs. bad events (24 hours)

We can see how we’ve easily burned through our error budget.

Hence something must be going on with nginx. To investigate, all we need to do is utilize the AI Assistant, and use its natural language interface to ask questions to help analyze the situation.

Let’s use Elastic’s AI Assistant to analyze the breakdown of http.response.status_code across all the logs from the past seven days. This helps us understand how many 50X errors we are getting.

As we can see, the number of 502s is minimal compared to the number of overall messages, but it is affecting our SLO.

However, it seems like Nginx is having an issue. In order to reduce the issue, we also ask the AI Assistant how to work on this error. Specifically, we ask if there is an internal runbook the SRE team has created.

AI Assistant gets a runbook the team has added to its knowledge base. I can now analyze and try to resolve or reduce the issue with nginx.

While this is a simple example, there are an endless number of possibilities that can be defined based on KQL. Some other simple examples:

• 99% of requests occur under 200ms

• 99% of log message are not errors

Application SLOs: OpenTelemetry demo cartservice

A common application developers and SREs use to learn about OpenTelemetry and test out Observability features is the OpenTelemetry demo.

This demo has feature flags to simulate issues. With Elastic’s alerting and SLO capability, you can also determine how well the entire application is performing and how well your customer experience is holding up when these feature flags are used.

Elastic supports OpenTelemetry by taking OTLP directly with no need for an Elastic specific agent. You can send in OpenTelemetry data directly from the application (through OTel libraries) and through the collector.

We’ve brought up the OpenTelemetry demo on a K8S cluster (AWS EKS) and turned on the cartservice feature flag. This inserts errors into the cartservice. We’ve also created two SLOs to monitor the cartservice’s availability and latency.

We can see that the cartservice’s availability is violated. As we drill down, we see that there aren’t as many successful transactions, which is affecting the SLO.

As we drill into the service, we can see in Elastic APM that there is a higher than normal failure rate of about 5.5% for the emptyCart service.

We can investigate this further in APM, but that is a discussion for another blog. Stay tuned to see how we can use Elastic’s machine learning, AIOps, and AI Assistant to understand the issue.

Conclusion

SLOs allow you to set clear, measurable targets for your service performance, based on factors like availability, response times, error rates, and other key metrics. Hopefully with the overview we’ve provided in this blog, you can see that:

• SLOs can be based on logs. In Elastic, you can use KQL to essentially find and filter on specific logs and log fields to monitor and trigger SLOs.

• AI Assistant is a valuable, easy-to-use capability to analyze, troubleshoot, and even potentially resolve SLO issues.

• APM Service based SLOs are easy to create and manage with integration to Elastic APM. We also use OTel telemetry to help monitor SLOs.

For more information on SLOs in Elastic, check out Elastic documentation and the following resources:

• What’s new in Elastic Observability 8.12

• Introducing the Elastic AI Assistant

• Elastic OpenTelemetry support

Ready to get started? Sign up for Elastic Cloud and try out the features and capabilities I’ve outlined above to get the most value and visibility out of your SLOs.

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

In this blog post, we may have used or referred to third party generative AI tools, which are owned and operated by their respective owners. Elastic does not have any control over the third party tools and we have no responsibility or liability for their content, operation or use, nor for any loss or damage that may arise from your use of such tools. Please exercise caution when using AI tools with personal, sensitive or confidential information. Any data you submit may be used for AI training or other purposes. There is no guarantee that information you provide will be kept secure or confidential. You should familiarize yourself with the privacy practices and terms of use of any generative AI tools prior to use.

Elastic, Elasticsearch, ESRE, Elasticsearch Relevance Engine and associated marks are trademarks, logos or registered trademarks of Elasticsearch N.V. in the United States and other countries. All other company and product names are trademarks, logos or registered trademarks of their respective owners.