Two bad rollouts on one Deployment burned 88% of a 30-day error budget in a day and fired a 26X burn-rate alert while the SLI still read 99.56%. That is the gap Part 1 alert rules cannot close on their own: they page when replicas drop; they do not tell you how much monthly reliability budget the incident cost.
The Kubernetes OpenTelemetry Assets package now ships four Kubernetes SLO templates for Deployments, StatefulSets, DaemonSets, and Jobs on those OTel metrics. If you already followed Part 1 and have the dashboards and alert rules, create an SLO from a template and you get SLI, remaining budget, and burn rate without new instrumentation.
The diagram above extends the stack from Part 1. The Kubernetes OpenTelemetry Assets package (kubernetes_otel) 2.3.0 includes:
- Dashboards designed for drill-down (Part 1)
- Alert rule templates that fire on known bad states (Part 1)
- ML anomaly detection jobs with workload baselines (Part 1)
- SLO templates for rolling 30-day budgets (this post)
All four use the same OTel metrics. Burn rate alerts on an SLO send you back into Overview, Workloads, and Deployment Details when the number alone is not enough.
Why Kubernetes observability needs SLO monitoring alongside alerts
Part 1 built the reactive stack for the engineer who gets paged at 3 AM. SLOs serve the planning conversation on a 30-day horizon: Are we meeting our reliability commitments? They give platform and engineering leaders a number for prioritisation: how much error budget remains and which workload is burning it fastest. The table later in this post maps each SLO template to its Part 1 alert counterpart.
The SLO templates in this post are part of the Kubernetes OpenTelemetry Assets package (kubernetes_otel). Install the Kubernetes OpenTelemetry Assets package and confirm your cluster is already sending Kubernetes metrics through OpenTelemetry (the same pipeline from Part 1). No additional instrumentation is required.
Four SLO templates for Kubernetes Deployments, StatefulSets, DaemonSets and Jobs
In Integrations → Kubernetes OpenTelemetry → Assets, enable any of the four templates below. Names match Kibana; each includes the [Kubernetes OTel] prefix in the UI.
| Template | Rolling objective | Package description |
|---|---|---|
| Deployment Replica Availability 99.5% Rolling 30 Days | 99.5% / 30d | Tracks Deployment availability from OTel metrics: 99.5% of intervals should have each Deployment at its desired replica count. When k8s.deployment.available < k8s.deployment.desired, the workload has fewer healthy replicas than configured. |
| StatefulSet Replica Availability 99.5% Rolling 30 Days | 99.5% / 30d | Same pattern for StatefulSets, where pod identity and ordering matter for databases, queues, and caches. |
| DaemonSet Scheduling Availability 99.0% Rolling 30 Days | 99.0% / 30d | Tracks whether each DaemonSet runs on all eligible nodes. Covers node-level agents such as log collectors, monitoring, security, and CNI plugins. |
| Job Completion Success Rate 99.0% Rolling 30 Days | 99.0% / 30d | Tracks batch Jobs (ETL, backups, pipelines, scheduled tasks) completing without failed pods over the rolling window. |
Each is a timeslice-metric SLO: Elastic marks every five-minute window good or bad, then rolls those results into a 30-day rolling objective per namespace and workload.
Reliability is scored at two levels. Each five-minute slice gets one verdict: Elastic aggregates OTel metrics in that window, evaluates the template equation, and compares the result to the metric threshold. For Deployments, that is sum(available) / sum(desired) >= 1. At a ~30-second OTel scrape cadence, that is roughly ten measurements per slice, and the slice passes or fails on the aggregated result. The SLO target (99.5% or 99.0%) is the share of slices that must pass across the rolling window. Over 30 days at five-minute slices, that is 8,640 possible slices per workload (30 × 24 × 12). After you create an SLO from a template, the SLO detail view shows how many slices passed and how much error budget remains.
At 99.5%, a workload can miss roughly 43 of those slices (~3.6 hours of bad slices) before breach. At 99.0%, about 86 slices (~7.2 hours).
How to set SLO targets by Kubernetes workload type
We picked defaults per workload type, not one number for the whole cluster.
Deployments and StatefulSets at 99.5%: We considered 99.9% (~43 minutes per month), which fits a single critical API or a formal SLA buffer. For a default integration template across many Deployments, 99.5% (~3.6 hours) leaves room for normal rollout churn: a 20-minute bad image tag is roughly half the monthly budget at 99.9%, but a small fraction at 99.5%. Tune per workload; payment paths often warrant 99.9% or higher.
DaemonSets and Jobs at 99.0%: We considered 99.5% for DaemonSets, but node additions, replacements, and rolling updates often leave ready_nodes below desired_scheduled_nodes for several minutes per event. At 99.5%, that normal platform churn would burn error budget on infrastructure agents (log collectors, monitoring, CNI) as if they were user-facing outages. 99.0% (~7.2 hours) absorbs that lifecycle noise. Jobs get 99.0% for a different reason: a failed ETL run usually hurts data freshness, not live request availability, and failures can sit unnoticed until downstream teams see stale reports.
Deployment replica availability (99.5%)
Metric: sum(k8s.deployment.available) / sum(k8s.deployment.desired) >= 1
Target: 99.5% of 5-minute timeslices over 30 days
Group by: resource.attributes.k8s.namespace.name + resource.attributes.k8s.deployment.name
When available < desired, the application runs fewer healthy replicas than configured. Failed rollouts, crash loops, and node loss all show up here. 99.5% leaves roughly 3.6 hours of degradation per deployment per month before breach.
Grouping by namespace and deployment name creates one SLO per workload. A cluster-wide average would let a healthy frontend mask a burning checkout. Linked dashboards (Overview and Workloads) connect the SLO view to investigation context in one click; from Workloads you drill into Deployment Detail for the failing deployment.
StatefulSet replica availability (99.5%)
Metric: sum(k8s.statefulset.ready_pods) / sum(k8s.statefulset.desired_pods) >= 1
Target: 99.5% of 5-minute timeslices over 30 days
Group by: resource.attributes.k8s.namespace.name + resource.attributes.k8s.statefulset.name
When ready_pods < desired_pods, the StatefulSet reports fewer Ready replicas than configured. Ordered rollouts, stuck pods, and node loss show up here too. Rollouts proceed in order, and each pod keeps its name and volume, so a missing replica can stay below desired longer than a stateless pod would.
Grouping by namespace and StatefulSet name avoids a healthy workload masking another that is burning the SLO budget.
DaemonSet scheduling availability (99.0%)
Metric: sum(k8s.daemonset.ready_nodes) / sum(k8s.daemonset.desired_scheduled_nodes) >= 1
Target: 99.0% of 5-minute timeslices over 30 days
Group by: resource.attributes.k8s.namespace.name + resource.attributes.k8s.daemonset.name
DaemonSets run node-level infrastructure: log collectors, monitoring agents, security agents, and network plugins. When ready_nodes < desired_scheduled_nodes, an eligible node lacks a Ready pod, which can leave that node without logs or metrics from that agent. Rolling updates and new nodes drive most gaps; pods that never become Ready show the same signal. Cordoned nodes often still run DaemonSet pods. 99.0% (~7.2 hours per month) reflects that churn.
Group by namespace and DaemonSet name so a healthy fluentd does not mask a broken node-exporter on the same SLO budget.
Job completion success rate (99.0%)
Metric: max(k8s.job.failed_pods) <= 0
Target: 99.0% of 5-minute timeslices over 30 days
Group by: resource.attributes.k8s.namespace.name + resource.attributes.k8s.job.name
Jobs cover batch workloads: ETL pipelines, backups, database migrations, and scheduled reports. When failed_pods > 0, at least one pod created by the Job reached the Failed phase. Application errors, timeouts, and missing dependencies drive many failures; when retries reach the configured backoffLimit, Kubernetes marks the Job as Failed. Missed runs often surface as stale or delayed data, not as a serving outage. 99.0% (~7.2 hours per month) reflects that occasional batch failure is less time-sensitive than a Deployment or StatefulSet breach.
How do SLOs and alerts work together in Kubernetes observability?
The SLO templates and the alert rules from Part 1 serve different people asking different questions at different times.
| SLO Template | Alert rule (Part 1) | Failure consequence | Monthly budget (30d) |
|---|---|---|---|
| Deployment Replica Availability | Deployment below the desired replicas | Reduced throughput, degraded UX | ~3.6 hours at 99.5% |
| StatefulSet Replica Availability | No dedicated rule. Covered by CrashLoopBackOff / OOMKilled at pod level | Split-brain risk, degraded durability | ~3.6 hours at 99.5% |
| DaemonSet Scheduling Availability | Pod stuck in Pending / node disk pressure | Blind spots: unmonitored nodes and gaps in node-level coverage | ~7.2 hours at 99.0% |
| Job Completion Success Rate | CrashLoopBackOff / OOMKilled | Stale or incomplete data | ~7.2 hours at 99.0% |
Alert rules answer: Is something broken right now? They fire within minutes, page the on-call engineer, and expect immediate action.
SLO templates answer: Are we meeting our reliability commitments over time? They accumulate signal across weeks and turn prioritisation debates into a number tied to remaining budget.
From incident to error budget burn: a Kubernetes walkthrough
A deployment drops from 3/3 to 2/3 available replicas during a rolling update. The new pod fails its readiness probe. Here is what happened in our test cluster, from dashboard signal through alert, SLO impact, and root cause.
Rollout begins. The Deployment dashboard shows available: 2, desired: 3. The Part 1 Deployment unavailable replicas rule has a 5-minute grace period, so on-call is not paged yet during a short rollout gap. The Deployment Detail view for web-frontend shows available replicas dropping while desired stays at 3. The Deployment replicas over time chart marks where the rollout started to fail.
Alert fires, then root cause. After the grace period, the alert rule triggers: Deployment unavailable replicas. The on-call engineer opens the Workloads dashboard, finds web-frontend at available: 2, desired: 3, and drills into Deployment Detail. The replicas-over-time chart confirms when availability dropped.
In Discover, filter Kubernetes events for that pod with k8s.object.name: "web-frontend-796fcd55b9-jmlkh". The event stream shows ImagePullBackOff and Back-off pulling image "nginx:nonexistent-tag-999". The rollout references an image tag that does not exist.
Rollback and recovery. The engineer runs kubectl rollout undo deployment/web-frontend. Replicas return to 3/3.
How two rollouts consumed 88% of a 30-day error budget
The rollback fixed availability. The SLO still counted the day's failures.
Two rollout failures left web-frontend with 39 failed timeslices where available < desired. That consumed 88.0% of the 30-day error budget. The SLI still read 99.56%, above the 99.5% target, but only 12% of the monthly allowance remained.
The burn rate alert fired next, even though replicas were healthy again. Over the past day the deployment consumed budget at 26× the rate a 99.5% SLO can sustain long term. Each failed 5-minute timeslice uses roughly 2.3% of the monthly budget (about 43 failures allowed per 30 days). Thirty-nine failures across two rollouts is worth a reliability review, not a one-line postmortem. The burn rate alert often matters more than the raw SLI mid-month because it fires while you still have budget left to spend deliberately.
Try it yourself: trigger an error budget burn on a test Deployment
If you already have Kubernetes OpenTelemetry Assets installed, the SLO templates live under Integrations → Kubernetes OpenTelemetry → Assets.
Create a Deployment replica availability SLO for the deployment you use below. Open the SLO and note the baseline: current SLI, remaining error budget, and existing timeslice history.
Create an isolated namespace and a small deployment so the exercise does not affect production workloads. Wait a few minutes for the OTel collector to scrape metrics before you create the SLO.
kubectl create namespace blog-demo
kubectl create deployment web-frontend --namespace blog-demo --image=nginx:latest --replicas=3
Trigger a bad rollout with a non-existent image tag:
kubectl get deployment web-frontend -n blog-demo
kubectl set image deployment/web-frontend nginx=nginx:nonexistent-tag-999 --namespace blog-demo
Within a few minutes a new pod enters ImagePullBackOff, available replicas drop below desired, and the SLO records failed timeslices. Roll back to recover:
kubectl rollout undo deployment/web-frontend -n blog-demo
Refresh the SLO view. You should see new failed timeslices in the 30-day history and a reduction in remaining error budget.
One failed timeslice consumes about 2.3% of the monthly error budget at 99.5%. Repeat that across deployments in a week and the burn rate alert becomes the prioritisation signal.
When you are done, delete the test namespace with:
kubectl delete namespace blog-demo
What's next: from SLO monitoring to agentic remediation
Alerts tell you replicas dropped. SLOs tell you how much monthly budget that cost. In the walkthrough above, the same ImagePullBackOff showed up in Deployment Detail, the unavailable-replicas alert, and failed timeslices on the replica-availability SLO, all from the OTel pipeline you installed in Part 1. The SLI still read 99.56% while 88% of the monthly error budget was gone.
Part 1 closed by previewing Agentic Investigations: investigation workflows that run when an alert fires, with skills, tools, and MCP views. This post adds the SLO layer on those same metrics so you can quantify reliability debt before automating runbooks. A follow-up post will cover that agentic workflow and propose remediations you review before applying.
Which remediations would you trust a workflow to suggest on a Kubernetes incident, and which would you keep manual? Join the Elastic Community discussion.