Indeed, an expired certificate is not just a minor technical glitch. It is a direct hit on your most critical systems:

• Your CI/CD pipeline grinds to a halt because it can not trust the internal image registry.

• Your Single Sign-On (SSO) system fails, locking all your internal users out.

• Your external clients see scary browser warnings, shattering user trust and forcing support tickets.

• Your SLOs burn due to services not being able to communicate with one another.

In Kubernetes, certificates are usually dynamically generated and auto-renewed by tools like cert-manager. In more unlucky scenarios, certificates might be tucked away inside Secrets and ConfigMaps, leading to challenges while inventorying them. It is neither hard nor unheard of to have a dozen critical certificates and no centralized way to know when they are about to expire.

Additionally, only monitoring the certificates for external Load Balancers might lead to huge internal risks, since many certificates never get exposed to external users.

In this blog post, we will guide you through establishing comprehensive, cluster-wide certificate monitoring using the OpenTelemetry Collector, the x509-certificate-exporter, and Elastic Observability.

Classical approach: HTTP monitoring

The classical approach to monitor TLS certificate expiration in the Elastic Observability is by treating it like any other service availability check. Historically, this was accomplished using Heartbeat or, more recently, Elastic Observability's Synthetics. These tools perform an external check against a public HTTPS endpoint and automatically extract the certificate's validity dates, allowing you to configure a Synthetics TLS certificate rule in Kibana to trigger an alert when expiration is within a specified threshold (e.g., 30 days).

While effective for external-facing services, this "classical" approach has two major shortcomings when dealing with Kubernetes:

• It only works for certificates exposed via HTTP(S), meaning you cannot use this for internal services, databases, or message queues using other protocols. In other words, this won't work to monitor common, critical TLS certificates such as Kafka's.

• The monitoring agent must have network access to the endpoint. In a segmented or private Kubernetes environment, deploying agents with the necessary access often introduces unnecessary complexity or security risks.

To gain true cluster-wide visibility, we need to inspect the certificates at their source: inside Kubernetes Secrets or ConfigMaps.

A Kubernetes-native approach: monitor Secrets and ConfigMaps

Monitoring TLS certificate expiration directly within Kubernetes Secrets and ConfigMaps is the only reliable way to gain visibility into internal, non-HTTP-exposed certificates, such as those used for service meshes, internal registries, or databases. In this section, we will use the OpenTelemetry Collector to monitor certificate expiration.

The OpenTelemetry Collector provides a mechanism to read up-to-date information from the Kubernetes API, including Secrets, via the k8sobjects receiver. However, this receiver only fetches raw TLS certificate resource data, which the OpenTelemetry Transformation Language (OTTL) can not properly parse. Therefore, we need to use a dedicated exporter to collect the certificate data and expose the results in a digestible format.

The industry-standard solution

As mentioned above, simply reading certificate information from the Kubernetes API is not a feasible solution. We will therefore use a specialized, lightweight exporter (specifically, the popular x509-certificate-exporter) to collect TLS certificate data and expose the results, allowing the OpenTelemetry Collector's Prometheus receiver to seamlessly scrape the data and send it to Elastic Observability. This approach immediately and easily enables us to monitor both certificates generated by cert-manager and self-managed ones, such as the ones created for ECK.

A fully working configuration example and a script to set up a complete local development environment is available here. Feel free to use it to follow along as you read through this guide and try out the examples. Please note that, while this repository uses the Elastic Distribution of OpenTelemetry (EDOT), it can be easily adapted to use the OpenTelemetry Collector.

Helm Chart Configuration

We configured the x509-certificate-exporter with the official Helm Chart and used the following minimal configuration:

secretsExporter:
  secretTypes:
  - type: kubernetes.io/tls
    key: tls.crt
  # For ECK that uses different secret types
  - type: Opaque
    key: tls.crt
  - type: Opaque
    key: ca.crt
  configMapKeys:
  - tls.crt
  - ca.crt

# Create a service to have a stable endpoint for scraping metrics
service:
  create: true
  # -- TCP port to expose the Service on
  port: 9793

# Disable prometheus service monitor and prometheus rules
prometheusServiceMonitor:
  create: false
prometheusRules:
  create: false

We refer to the reference values.yaml to get insights in the plethora of configuration options.

OpenTelemetry Collector Configuration

Afterward, we configured the OpenTelemetry Collector to scrape the metrics from the service:

prometheus/cert-expiration:
  config:
    scrape_configs:
      - job_name: "cert-expiration"
        scrape_interval: 60m
        static_configs:
          - targets:
              - "x509-certificate-exporter.monitoring.svc.cluster.local:9793"

We deliberately used a long scrape interval of 60 minutes, because certificate expiration is a low-frequency concern.

Visualizing the data in Kibana

Once the data is ingested, we can explore it using Discover. We can select the metrics-* Data View and search for our data with the filter data_stream.dataset : "prometheusreceiver.otel".

An example document looks like the following:

{
  "@timestamp": "2025-12-19T09:43:45.317Z",
  "_metric_names_hash": "7d113f55b70019d9",
  "attributes": {
    "issuer_CN": "tls-cert.example.com",
    "issuer_O": "TLS Cert",
    "secret_key": "tls.crt",
    "secret_name": "tls-cert-secret",
    "secret_namespace": "test-certs",
    "serial_number": "250887723804527203192865532237673843132727735771",
    "subject_CN": "tls-cert.example.com",
    "subject_O": "TLS Cert"
  },
  "data_stream": {
    "dataset": "prometheusreceiver.otel",
    "namespace": "default",
    "type": "metrics"
  },
  "metrics": {
    "x509_cert_expired": 0,
    "x509_cert_not_after": 1768488242,
    "x509_cert_not_before": 1765896242
  },
  "resource": {
    "attributes": {
      "server.address": "x509-certificate-exporter.monitoring.svc.cluster.local",
      "server.port": "9793",
      "service.instance.id": "x509-certificate-exporter.monitoring.svc.cluster.local:9793",
      "service.name": "cert-expiration",
      "url.scheme": "http"
    }
  },
  "scope": {
    "name": "github.com/open-telemetry/opentelemetry-collector-contrib/receiver/prometheusreceiver",
    "version": "9.2.2"
  }
}

The core metric reported by the x509-certificate-exporter is x509_cert_not_after that represent the Unix Epoch timestamp (in seconds) of the certificate's expiration date. This metric has some attributes associated with it. In the case of Secrets, the following attributes are relevant:

• secret_namespace: The namespace of the Secret containing the certificate.
• secret_name: The name of the Secret containing the certificate.
• secret_key: The specific key within the Secret where the certificate is stored.

In the case of ConfigMaps, we can infer the attributes of interest from the filepath attribute.

Finally, we can leverage ES|QL to compute the remaining days until expiration. In the following examples, we will use the TS command, which is optimized and recommended for interacting with time-series data.

For Secrets:

TS metrics-*
| WHERE metrics.x509_cert_not_after is not NULL
| STATS expiration_date = MAX(LAST_OVER_TIME(metrics.x509_cert_not_after)) by attributes.secret_namespace, attributes.secret_name, attributes.secret_key
| EVAL remaining_days = DATE_DIFF("days", NOW(), TO_DATETIME (1000 * expiration_date))
| EVAL expiration_date = TO_DATETIME(1000 * expiration_date)
| SORT expiration_date ASC

And for ConfigMaps:

TS metrics-*
| WHERE metrics.x509_cert_not_after IS NOT NULL
| WHERE attributes.filepath IS NOT NULL
| DISSECT attributes.filepath "k8s/%{namespace}/%{configmap}"
| WHERE configmap != "kube-root-ca.crt" // Filter out the Kubernetes API server certificate's signing CA
| STATS expiration_date = MAX(LAST_OVER_TIME(metrics.x509_cert_not_after)) by namespace, configmap, filename
| EVAL remaining_days = DATE_DIFF("days", NOW(), TO_DATETIME (1000 * expiration_date))
| EVAL expiration_date = TO_DATETIME(1000 * expiration_date)
| SORT expiration_date ASC

Based on these core queries, we can easily build a dashboard that shows the remaining days until expiration for all the certificates in the cluster:

and create alerts about certificates that are about to expire by adding a condition after the query:

WHERE remaining_days < 30

Conclusion

In this blog post, we explored how to monitor TLS certificate expiration within a Kubernetes cluster using the OpenTelemetry Collector. We discussed the limitations of traditional HTTP-based monitoring approaches and introduced a Kubernetes-native solution leveraging the x509-certificate-exporter to extract certificate expiration data directly from Kubernetes Secrets and ConfigMaps. This method provides comprehensive visibility into all certificates used within the cluster, including those not exposed via HTTP(S).

For the sake of simplicity, we just focused on monitoring certificate expiration with the OpenTelemetry Collector on Kubernetes. However, this approach can be easily applied with classical Elastic Agent by leveraging the Prometheus input package (read more on how to use input packages here) and can be also extended to monitor certificates on virtual machines or bare-metal servers by deploying the x509-certificate-exporter there.

Finally, is worth knowing that Elastic Observability, offers an officially supported distribution of the OpenTelemetry Collector, called Elastic Distributions of OpenTelemetry (EDOT).

If you are an Elastic user, you could consider using EDOT Collector to monitor certificates with OpenTelemetry: since it is supported by Elastic Observability, it will be easier to manage and keep up to date. Alternatively you can use upstream OTel compnents also.

What's next?

Now that Elastic supports Rule Templates and OpenTelemetry content packs, our near-term objective is to contribute to the integration repository to make the setup of certificate monitoring even easier for our users. Stay tuned for more updates on this!

Check out other resources on Elastic's OpenTelemetry

Elastic's OTLP EndPoint

Elastic's EDOT PHP Contribution

Opentelemetry SDK Central Management with EDOT

Also sign up for Elastic Cloud and try out your application with OpenTelemetry in Elastic

Prerequisites for Elastic integration alerting rule templates

Elastic Stack 9.4.0 or later.

Alerting rule templates have been available since 9.2.1, under the integration Assets tab. This article covers three things that need 9.4.0: the dedicated Alerting tab, idle data streams rules, and the NGINX OpenTelemetry Assets package, which is in technical preview.

Step 1: Send NGINX logs and metrics to Elasticsearch with OpenTelemetry

First, get NGINX metrics and logs into Elasticsearch.

Enable the NGINX stub_status module and make the access and error logs readable by the collector. Then, configure an EDOT or upstream OpenTelemetry Collector with the nginx and filelog receivers to export metrics and logs to Elasticsearch.

The integration setup has the full receiver and pipeline configuration.

If you want to reproduce this example, you can use the companion repository.

Both signals matter for alerting, and each group of templates reads a different data stream:

• The log-based templates (4xx and 5xx error rates, error log spike) query logs-nginx.access.otel-* and logs-nginx.error.otel-*, which come from the filelog receiver.
• The metric-based templates (active connections, dropped connections) query metrics-nginxreceiver.otel-*, which comes from the nginx receiver.

This is easy to get wrong: the Fleet Nginx (OpenTelemetry) input package collects stub_status metrics only. Its companion for logs is the classic Nginx integration, which writes ECS-based nginx.access and nginx.error datasets, not the *.otel-* data streams the log-based templates query. If you rely on that pairing alone, the log-based rules have nothing to evaluate and silently never fire. Run the filelog receiver too, not just the nginx receiver.

Step 2: Install the NGINX OpenTelemetry Assets integration

NGINX OpenTelemetry Assets is a content-only package. It ships the dashboards, alerting rule templates, and SLO templates, but it does not collect data itself. The data comes from the collector you set up in Step 1.

You don't need to install it by hand. Once the NGINX OTel data from Step 1 starts arriving, Elastic detects it and installs the Assets package for you, which takes a minute or two. Confirm it under Management > Integrations > Installed integrations, where NGINX OpenTelemetry Assets should appear.

Step 3: Create Elasticsearch alert rules from a rule template

Open the integration and select the Alerting tab.

This package ships six templates: high 4xx and 5xx error rates, high active connections, an error log spike, dropped connections, and a generic High error rate by service template that points at a placeholder logs-myservicereceiver.otel-* index for you to repoint and rename. The five NGINX rules run ES|QL every minute and group results by host.name, so an alert points at the host with the problem. The generic template groups by service.name instead, since it is meant to be repointed at whichever service you choose.

Select a template, for example [Nginx OTel] High 5xx error rate. Kibana opens a prefilled Create rule form built on an Elasticsearch query rule. It runs the template's ES|QL on a schedule and alerts when the query returns rows.

The query looks like this:

FROM logs-nginx.access.otel-*
// Flag each access log entry as a server error (5xx) or not
| EVAL is_5xx = CASE(http.response.status_code >= 500, 1, 0)
// Aggregate total requests and 5xx count per NGINX host
| STATS total = COUNT(*), errors_5xx = SUM(is_5xx) BY host.name
// Minimum sample size to avoid noisy low-traffic hosts
| WHERE total > 50
// Calculate 5xx error rate as a percentage
| EVAL error_rate_pct = ROUND(TO_DOUBLE(errors_5xx) / TO_DOUBLE(total) * 100.0, 2)
// Alert threshold: adjust to tune sensitivity
| WHERE error_rate_pct > 5.0
| SORT error_rate_pct DESC
| LIMIT 10

It counts requests and 5xx responses per host, keeps hosts with enough traffic to matter, and returns those above five percent.

Three things to get right while the form is open:

• Send data first. ES|QL validates column names against the indices that exist when the query runs. Open a template before any NGINX data has been ingested and the editor reports Unknown column "http.response.status_code" and the form shows errors. Once data is flowing (Step 1), the same query validates and the error clears, so collect data before you create the rule.
• Set the time field to @timestamp.
• Leave "Create an alert for each row" selected. Because the query groups by host.name, this makes every affected host raise its own alert.

Add a connector and an action so the alert reaches Slack, email, or PagerDuty, then save and enable the rule.

Step 4: Tune alerting rule template thresholds in ES|QL

The thresholds are starting points, so confirm them against your own traffic. The threshold lives in the ES|QL WHERE clause.

To make the rule stricter, change error_rate_pct > 5.0 to error_rate_pct > 2.0. To require more traffic before it fires, raise total > 50. Use Test query in the rule form to confirm the edited query parses and returns rows before you save.

Three more settings are worth a look:

• Time window: the look-back period the query runs over. A shorter window reacts faster but is noisier on bursty traffic.
• Rule schedule: how often the query runs, every minute by default.
• Alert delay: the number of consecutive runs the condition must hold before an alert is created, which filters out single-run blips.

How do you detect idle data streams in Elasticsearch?

Threshold rules only fire while data keeps arriving. When an agent goes offline or an output breaks, the data stops, and a threshold rule has nothing to evaluate.

Many Elastic integrations include a dynamically generated idle data streams template for exactly this case. It is named [{Integration name}] Idle data streams and appears in the same Alerting tab, though it is generated automatically rather than bundled with the integration. It alerts when no data is written to any of the integration's data stream patterns within a set period.

The NGINX OpenTelemetry packages do not include an idle data streams template, which is why no such template appears in the Alerting tab from Step 3. The end of this section covers what to do instead.

The default period is 24 hours, which is usually too long. A production service can go quiet for most of a day before you hear about it.

When you create the rule, drop the period to match how fast you need to know. Fifteen minutes to one hour works for a critical service. For a batch job, set a period comfortably longer than its run interval, so the quiet gaps between runs do not trigger it.

So why is this example left out? The template is generated from the data stream patterns an integration defines, and it is not generated for input-only packages. A content-only package like NGINX OpenTelemetry Assets defines no data streams of its own either. To catch silence in a collector-based setup like this, recreate the rule by hand with an Elasticsearch query rule. Use the query DSL or KQL variant rather than ES|QL, because an ES|QL rule fires on returned rows and so cannot alert on the absence of data. Point it at the OTel data streams (logs-nginx.access.otel-*, or metrics-nginxreceiver.otel-*) and set the condition to fire when the number of matching documents is below 1 over a window of, say, the last 15 minutes. That reproduces what an idle data streams template does, scoped to the data streams your collector writes.

Get started with Elastic integration alerting rule templates

Alerting rule templates turn alert setup into a few steps: send data, install the integration, create a rule from a template, and adjust the threshold. Treat the bundled thresholds as defaults to confirm, not numbers to trust blindly. And where an idle data streams template is available, reduce its 24-hour default so you find out quickly when a service goes silent.

Resources

• Companion repository, to generate the NGINX demo data used here
• Alerting rule templates
• NGINX OpenTelemetry Assets integration
• End-to-end tracing for NGINX with OpenTelemetry
• Elasticsearch query rule
• Elastic Agent built-in alerts, for monitoring the agents themselves

Here's what's new:

Improved Related Alert Grouping with Relevance Scoring & Reasoning

We've enhanced our related alert detection to go beyond surface-level correlations. Alerts are now grouped based on a relevance score that reflects the strength of their relationship across dimensions like:

• Shared entities or resources (e.g. same host, pod, or service)
• Temporal proximity (alerts firing within a suspiciously short window)
• Signal similarity (e.g. spikes in logs, metrics, and traces that point to the same failure mode)

More importantly, we now show the why. You'll see why an alert is grouped, whether it's sharing the same Kubernetes pod, has similar log patterns, or was triggered by the same upstream anomaly. This gives users confidence in the grouping logic and accelerates root cause analysis.

Link Dashboards to Alert Rules and Get Smart Suggestions

You can now link dashboards directly to your alert rules, giving responders an instant visual lens into the metrics or logs that matter most for that alert. No more scrambling to remember which dashboard to check — just click and go.

And we've made this smarter too: Elastic will now suggest relevant dashboards based on the alert's source, rule logic, or monitored entities, helping users land on the right view without needing to configure anything upfront.

Investigation Guides Embedded Into Alerts

Every alert can now be configured with an investigation guide, a set of pre-configured, context-aware instructions or next steps tailored to the alert. Think of it as a playbook that's embedded right where and when you need it.

Use it to:

• Document your team's runbooks and standard triage steps or link to existing runbooks
• Guide junior engineers or on-call responders through unfamiliar territory
• Automate the first few steps of root cause analysis

Why This Matters

These changes are all about reducing time to detect (MTTD) and time to resolve (MTTR). By:

• Grouping alerts more intelligently (and transparently)
• Giving you the dashboards you need, when you need them
• Embedding action-oriented guides in every alert

We're bringing you closer to a truly streamlined incident response workflow; No swivel-chairing, no guesswork, just clarity.

Additionally, look at some of our other articles on Elastic Observability Labs related to analysis:

• Using the AI Assistant in Elastic Observability to Accelerate Root Cause Analysis

• All of the log analytics features in Elastic Observability

• Our latest on OpenTelemetry support in Elastic Observability

The Kubernetes OpenTelemetry Assets package now ships four Kubernetes SLO templates for Deployments, StatefulSets, DaemonSets, and Jobs on those OTel metrics. If you already followed Part 1 and have the dashboards and alert rules, create an SLO from a template and you get SLI, remaining budget, and burn rate without new instrumentation.

The diagram above extends the stack from Part 1. The Kubernetes OpenTelemetry Assets package (kubernetes_otel) 2.3.0 includes:

• Dashboards designed for drill-down (Part 1)
• Alert rule templates that fire on known bad states (Part 1)
• ML anomaly detection jobs with workload baselines (Part 1)
• SLO templates for rolling 30-day budgets (this post)

All four use the same OTel metrics. Burn rate alerts on an SLO send you back into Overview, Workloads, and Deployment Details when the number alone is not enough.

Why Kubernetes observability needs SLO monitoring alongside alerts

Part 1 built the reactive stack for the engineer who gets paged at 3 AM. SLOs serve the planning conversation on a 30-day horizon: Are we meeting our reliability commitments? They give platform and engineering leaders a number for prioritisation: how much error budget remains and which workload is burning it fastest. The table later in this post maps each SLO template to its Part 1 alert counterpart.

The SLO templates in this post are part of the Kubernetes OpenTelemetry Assets package (kubernetes_otel). Install the Kubernetes OpenTelemetry Assets package and confirm your cluster is already sending Kubernetes metrics through OpenTelemetry (the same pipeline from Part 1). No additional instrumentation is required.

Four SLO templates for Kubernetes Deployments, StatefulSets, DaemonSets and Jobs

In Integrations → Kubernetes OpenTelemetry → Assets, enable any of the four templates below. Names match Kibana; each includes the [Kubernetes OTel] prefix in the UI.

Each is a timeslice-metric SLO: Elastic marks every five-minute window good or bad, then rolls those results into a 30-day rolling objective per namespace and workload.

Reliability is scored at two levels. Each five-minute slice gets one verdict: Elastic aggregates OTel metrics in that window, evaluates the template equation, and compares the result to the metric threshold. For Deployments, that is sum(available) / sum(desired) >= 1. At a ~30-second OTel scrape cadence, that is roughly ten measurements per slice, and the slice passes or fails on the aggregated result. The SLO target (99.5% or 99.0%) is the share of slices that must pass across the rolling window. Over 30 days at five-minute slices, that is 8,640 possible slices per workload (30 × 24 × 12). After you create an SLO from a template, the SLO detail view shows how many slices passed and how much error budget remains.

At 99.5%, a workload can miss roughly 43 of those slices (~3.6 hours of bad slices) before breach. At 99.0%, about 86 slices (~7.2 hours).

How to set SLO targets by Kubernetes workload type

We picked defaults per workload type, not one number for the whole cluster.

Deployments and StatefulSets at 99.5%: We considered 99.9% (~43 minutes per month), which fits a single critical API or a formal SLA buffer. For a default integration template across many Deployments, 99.5% (~3.6 hours) leaves room for normal rollout churn: a 20-minute bad image tag is roughly half the monthly budget at 99.9%, but a small fraction at 99.5%. Tune per workload; payment paths often warrant 99.9% or higher.

DaemonSets and Jobs at 99.0%: We considered 99.5% for DaemonSets, but node additions, replacements, and rolling updates often leave ready_nodes below desired_scheduled_nodes for several minutes per event. At 99.5%, that normal platform churn would burn error budget on infrastructure agents (log collectors, monitoring, CNI) as if they were user-facing outages. 99.0% (~7.2 hours) absorbs that lifecycle noise. Jobs get 99.0% for a different reason: a failed ETL run usually hurts data freshness, not live request availability, and failures can sit unnoticed until downstream teams see stale reports.

Deployment replica availability (99.5%)

Metric:     sum(k8s.deployment.available) / sum(k8s.deployment.desired) >= 1
Target:     99.5% of 5-minute timeslices over 30 days
Group by:   resource.attributes.k8s.namespace.name + resource.attributes.k8s.deployment.name

When available < desired, the application runs fewer healthy replicas than configured. Failed rollouts, crash loops, and node loss all show up here. 99.5% leaves roughly 3.6 hours of degradation per deployment per month before breach.

Grouping by namespace and deployment name creates one SLO per workload. A cluster-wide average would let a healthy frontend mask a burning checkout. Linked dashboards (Overview and Workloads) connect the SLO view to investigation context in one click; from Workloads you drill into Deployment Detail for the failing deployment.

StatefulSet replica availability (99.5%)

Metric:     sum(k8s.statefulset.ready_pods) / sum(k8s.statefulset.desired_pods) >= 1
Target:     99.5% of 5-minute timeslices over 30 days
Group by:   resource.attributes.k8s.namespace.name + resource.attributes.k8s.statefulset.name

When ready_pods < desired_pods, the StatefulSet reports fewer Ready replicas than configured. Ordered rollouts, stuck pods, and node loss show up here too. Rollouts proceed in order, and each pod keeps its name and volume, so a missing replica can stay below desired longer than a stateless pod would.

Grouping by namespace and StatefulSet name avoids a healthy workload masking another that is burning the SLO budget. 

DaemonSet scheduling availability (99.0%)

Metric:     sum(k8s.daemonset.ready_nodes) / sum(k8s.daemonset.desired_scheduled_nodes) >= 1
Target:     99.0% of 5-minute timeslices over 30 days
Group by:   resource.attributes.k8s.namespace.name + resource.attributes.k8s.daemonset.name

DaemonSets run node-level infrastructure: log collectors, monitoring agents, security agents, and network plugins. When ready_nodes < desired_scheduled_nodes, an eligible node lacks a Ready pod, which can leave that node without logs or metrics from that agent. Rolling updates and new nodes drive most gaps; pods that never become Ready show the same signal. Cordoned nodes often still run DaemonSet pods. 99.0% (~7.2 hours per month) reflects that churn. 

Group by namespace and DaemonSet name so a healthy fluentd does not mask a broken node-exporter on the same SLO budget.

Job completion success rate (99.0%)

Metric:     max(k8s.job.failed_pods) <= 0
Target:     99.0% of 5-minute timeslices over 30 days
Group by:   resource.attributes.k8s.namespace.name + resource.attributes.k8s.job.name

Jobs cover batch workloads: ETL pipelines, backups, database migrations, and scheduled reports. When failed_pods > 0, at least one pod created by the Job reached the Failed phase. Application errors, timeouts, and missing dependencies drive many failures; when retries reach the configured backoffLimit, Kubernetes marks the Job as Failed. Missed runs often surface as stale or delayed data, not as a serving outage. 99.0% (~7.2 hours per month) reflects that occasional batch failure is less time-sensitive than a Deployment or StatefulSet breach. 

How do SLOs and alerts work together in Kubernetes observability?

The SLO templates and the alert rules from Part 1 serve different people asking different questions at different times.

Alert rules answer: Is something broken right now? They fire within minutes, page the on-call engineer, and expect immediate action.

SLO templates answer: Are we meeting our reliability commitments over time? They accumulate signal across weeks and turn prioritisation debates into a number tied to remaining budget.

From incident to error budget burn: a Kubernetes walkthrough

A deployment drops from 3/3 to 2/3 available replicas during a rolling update. The new pod fails its readiness probe. Here is what happened in our test cluster, from dashboard signal through alert, SLO impact, and root cause.

Rollout begins. The Deployment dashboard shows available: 2, desired: 3. The Part 1 Deployment unavailable replicas rule has a 5-minute grace period, so on-call is not paged yet during a short rollout gap. The Deployment Detail view for web-frontend shows available replicas dropping while desired stays at 3. The Deployment replicas over time chart marks where the rollout started to fail.

Alert fires, then root cause. After the grace period, the alert rule triggers: Deployment unavailable replicas. The on-call engineer opens the Workloads dashboard, finds web-frontend at available: 2, desired: 3, and drills into Deployment Detail. The replicas-over-time chart confirms when availability dropped. 

In Discover, filter Kubernetes events for that pod with k8s.object.name: "web-frontend-796fcd55b9-jmlkh". The event stream shows ImagePullBackOff and Back-off pulling image "nginx:nonexistent-tag-999". The rollout references an image tag that does not exist.

Rollback and recovery. The engineer runs kubectl rollout undo deployment/web-frontend. Replicas return to 3/3.

How two rollouts consumed 88% of a 30-day error budget

The rollback fixed availability. The SLO still counted the day's failures.

Two rollout failures left web-frontend with 39 failed timeslices where available < desired. That consumed 88.0% of the 30-day error budget. The SLI still read 99.56%, above the 99.5% target, but only 12% of the monthly allowance remained.

The burn rate alert fired next, even though replicas were healthy again. Over the past day the deployment consumed budget at 26× the rate a 99.5% SLO can sustain long term. Each failed 5-minute timeslice uses roughly 2.3% of the monthly budget (about 43 failures allowed per 30 days). Thirty-nine failures across two rollouts is worth a reliability review, not a one-line postmortem. The burn rate alert often matters more than the raw SLI mid-month because it fires while you still have budget left to spend deliberately.

Try it yourself: trigger an error budget burn on a test Deployment

If you already have Kubernetes OpenTelemetry Assets installed, the SLO templates live under Integrations → Kubernetes OpenTelemetry → Assets.

Create a Deployment replica availability SLO for the deployment you use below. Open the SLO and note the baseline: current SLI, remaining error budget, and existing timeslice history.

Create an isolated namespace and a small deployment so the exercise does not affect production workloads. Wait a few minutes for the OTel collector to scrape metrics before you create the SLO.

kubectl create namespace blog-demo
kubectl create deployment web-frontend --namespace blog-demo --image=nginx:latest --replicas=3

Trigger a bad rollout with a non-existent image tag:

kubectl get deployment web-frontend -n blog-demo
kubectl set image deployment/web-frontend nginx=nginx:nonexistent-tag-999 --namespace blog-demo

Within a few minutes a new pod enters ImagePullBackOff, available replicas drop below desired, and the SLO records failed timeslices. Roll back to recover:

kubectl rollout undo deployment/web-frontend -n blog-demo

Refresh the SLO view. You should see new failed timeslices in the 30-day history and a reduction in remaining error budget.

One failed timeslice consumes about 2.3% of the monthly error budget at 99.5%. Repeat that across deployments in a week and the burn rate alert becomes the prioritisation signal.

When you are done, delete the test namespace with:

kubectl delete namespace blog-demo

What's next: from SLO monitoring to agentic remediation

Alerts tell you replicas dropped. SLOs tell you how much monthly budget that cost. In the walkthrough above, the same ImagePullBackOff showed up in Deployment Detail, the unavailable-replicas alert, and failed timeslices on the replica-availability SLO, all from the OTel pipeline you installed in Part 1. The SLI still read 99.56% while 88% of the monthly error budget was gone.

Part 1 closed by previewing Agentic Investigations: investigation workflows that run when an alert fires, with skills, tools, and MCP views. This post adds the SLO layer on those same metrics so you can quantify reliability debt before automating runbooks. A follow-up post will cover that agentic workflow and propose remediations you review before applying.

Which remediations would you trust a workflow to suggest on a Kubernetes incident, and which would you keep manual? Join the Elastic Community discussion.

Why use Elastic Observability with Proxmox?

Here at Elastic, we are passionate about efficiently managing and monitoring infrastructure and applications. Many of us have fun playing with home labs, oftentimes running Proxmox VE, a powerful open-source virtualization platform used to run virtual machines and Linux Containers (LXCs) with ease. While Proxmox provides robust tools for managing virtualized resources, gaining deep insights into the performance and health of your LXCs, VMs, and hosts requires a comprehensive monitoring solution. This blog post will guide you through leveraging the power of Elastic Observability, in conjunction with Elastic Agent, to effectively monitor your Proxmox VE deployment, ensuring optimal performance and proactive issue resolution thanks to Kibana Alerts.

The homelab setup

Our homelab setup centers around an Intel N100 mini PC, serving as the host for Proxmox VE. This setup is simple and minimal, yet effective for showcasing a few interesting capabilities. On top of this mini PC, we run several Linux Containers (LXCs) for various services, along with a dedicated virtual machine for Home Assistant.

Elastic Agent installation and configuration

Before beginning, it is worth noting that there are numerous ways to install and configure the Elastic Agent. For the sake of simplicity, we will showcase a setup in which only one instance of the Elastic Agent is running on the host machine. The Elastic Agent reports to an Elastic Cloud Observability deployment and is managed via Fleet, which makes it tremendously easy to upgrade and re-configure it whenever needed.

Diving into the host

Kibana offers various panes that make it nice and easy to learn about a system's health at a quick glance.

As a first step, let's take a look at the Infrastructure > Hosts page in Kibana:

Here we can see various information about our Proxmox VE host (i.e. the mini PC). The top processes running on it are presented, including processes running in LXCs such as pia-daemon. We can also see a kvm process, specifically running a Home Assistant virtual machine, and a Proxmox pve-firewall process.

Let's now take a look at Universal Profiling > Flamegraph. This graph shows how much CPU time is consumed by different stack traces from processes running on the host system. You can drill down into specific processes using the search bar at the top. For instance, you can filter by kvm to only see information regarding this specific process.

The Observability AI Assistant

All the Kibana panes we visited so far have proved to be highly interesting, but they struggle to answer urgent questions such as:

• did anything happen in our mini PC recently?
• was there any significant change in functionality?
• is there any precious information hidden among the thousands of data points collected?

The Elastic Observability AI Assistant helps us by answering these questions in natural language. By default, on Elastic Cloud, it uses the Elastic-managed LLM connector, which means users do not need to configure anything to get started with it. It just works!

Let's go to the Observability > AI Assistant pane in Kibana and let's try to ask a generic prompt such as: "please give me an overview of the health of my prox host".

Let's then wait a minute so that it can dig into the data... et voilà, here comes lots of relevant information in the form of graphs and natural language explanations. The Observability AI Assistant understood our question, went through all the data for our Proxmox host, ran data analytics on it, and reported back in a matter of seconds!

Alerting upon disruption with Kibana Alerts

As a final step, let's try to define a Kibana Alert to help us understand whether our host is overloaded. Let's head to Observability > Alerts > Rules and create a new rule. We will create a Custom Threshold rule that will fire if CPU usage for the host is higher than 80% on average for the last 15 minutes. Kibana will send us an email in case the rule fires. The rule is also configured to fire if no data appears for the last 15 minutes, which is extremely helpful as it would imply the presence of some issues to be debugged: broken network or no electricity in the house, a faulty Agent deployment, or even a hardware issue with the mini PC.

Conclusion

In this blog post we showcased how to effectively use the Elastic Stack to monitor Proxmox VE deployments. If you would like to try out such a setup first-hand, you are more than welcome to enjoy Elastic Cloud's 14-days free trial.

In future blog posts, we will investigate how to dig deeper into LXCs and VMs to gather even more information from our home lab and create more tailored alerts. Stay tuned!