This guide outlines the strategy to close that gap. It details how you move from passive Observability—watching a dashboard turn red—to proactive auto-remediation, where the system identifies, analyzes, and fixes itself. You will see how Elastic Workflows, Elastic Observability, Out-of-the-box (OOTB) Observability Agents, and Custom Agent in Agent Builder converge to solve the "Trust Deficit" that plagues traditional automation. We examine the technical implementation of self-healing systems and the business imperatives that make this transition necessary for the modern Chief Information Officer (CIO). You will learn to construct a reliability architecture that acts faster than your best engineer.

The Entropy of Scale

The Failure of Manual Operations

Systems generate telemetry at machine speed—millions of log lines, metric data points, and traces per second. Yet, your response to failure operates at human speed. You rely on an on-call engineer to wake up, interpret a notification, log in to a VPN, navigate a console, and execute a command. This latency is the "Remediation Gap". In that gap, revenue bleeds, customer trust erodes, and technical debt accumulates.

Manual remediation fails at scale for three specific reasons: cognitive overload, context switching, and the fear of action.

Cognitive Overload and the Signal-to-Noise Problem

Your Observability tools ingest petabytes of data. Finding the signal in that noise is the primary challenge for your Site Reliability Engineers (SREs). Traditional alerting relies on static thresholds where a rule is set at a specific value, for example, "Alert if CPU > 90%." In a dynamic cloud environment, static thresholds are insufficient as they create a lot of noise from false positives. For example, a database compaction process might spike CPU usage safely every night, or a Java application might naturally consume memory up to its heap limit before garbage collection. This is perfectly normal and expected behaviour in the environment, which creates confusion and fatigue for SREs when they’re alerted to false-positive issues in the environment.

When you bombard engineers with these low-fidelity alerts, you create "Alert Fatigue”, where SREs become overwhelmed with large volumes of alerts, many of which are false positives. When this happens, SREs stop trusting the pager. They treat alerts as suggestions rather than mandates, which impacts responsiveness to fixing real issues. Many organisations look to automation as the solution to alert fatigue, to fast-track response to issues in the environment. However, when alerts are inaccurate and untrustworthy, this creates more issues than it solves. If you automate a restart based on a false positive, you turn your monitoring system into a chaos monkey that attacks your own infrastructure.

The Cost of Context Switching

Consider the anatomy of a typical incident. An alert fires in your Observability platform. The engineer sees "High Latency on Service Checkout." Now the toil begins. They open a new tab for the infrastructure provider to check the pod status. They open another tab for the APM traces. They open Jira to see recent changes. They open Slack to ask if anyone pushed code as part of a change.

This fragmentation of data is expensive. Every context switch breaks the analyst's flow, widening the Remediation Gap through delays in determining the root cause and resolution fix. Context switching interrupts an analyst's focus. In a major incident, you need analysts to be focused on resolving the issue at hand, rather than losing time switching between tools and data, sets trying to work out what has happened. You need the context and the control plane to exist in the same interface so analysts can identify and resolve issues as fast as possible. The separation of "looking" and "doing" forces your engineers to act as human routers, copying IDs and error messages between incompatible tools.

The Blast Radius and the Fear of Automation

You hesitate to turn on auto-remediation because you fear the unknown impact of automation operating outside the expected bounds. A script that restarts a service works fine when one node fails. If a global configuration error causes all nodes to fail simultaneously, that same script might trigger a cascading restart loop that takes down the entire platform.

This fear leads to "Change Management Paralysis". You implement rigid approval boards and manual checks to prevent automation from running amok. You trade speed for reliability, but ultimately lose both. The solution is not to avoid automation, but to implement "Safe Automation" that understands dependency chains and business impact.

The Architecture of Action

Unified Observability and Remediation

To reduce the Remediation Gap, you must combine your observability data and action systems . You cannot have one tool for seeing and another for acting. Elastic Observability and Elastic Workflows provide this unification, enabling organisations to take intelligent response actions to proactively remediate issues in the environment. You ingest logs, metrics, and traces into Elastic, where intelligence, machine-learning and AI-driven analysis is done out of the box, against your environment’s behaviour baseline. From here, Elastic can use AIOps to detect anomalies with high precision and engage with proactive notifications and automations to quickly resolve the issue. You use the AI to diagnose the root cause and understand the resolution steps. You use Workflows to execute the fix and resolve the issue.

Combining Elastic Observability, AI & Agent Builder, and Workflows creates a closed-loop architecture for automatically resolving issues:

1. Sense: Ingest telemetry.
2. Think: Analyze with Machine Learning and AI.
3. Act: Trigger a Workflow.
4. Verify: Measure the result with telemetry.

Elastic Workflows: The Orchestration Engine

Elastic Workflows is the mechanism that turns insight into action. It sits directly within the Elastic platform, which means it has zero-latency access to your data. This is key when taking intelligent response actions to resolve performance issues in a timely manner, as context can be given to the automation for auto-analysis and informed logical execution.

A Workflow acts as a structured "recipe" for automation. It defines a sequence of steps that execute based on a trigger. Workflows are defined in YAML, which allows you to treat operations as code. You version control them, review them in Pull Requests, and test them before deployment.

AIOps and SLOs: The Vitals of the Enterprise

You solve the "Trust Deficit" by moving away from static thresholds and adopting Service Level Objectives (SLOs) and AIOps.

Understanding SLOs through the "Vital Signs" Analogy

Think of your IT system as a patient in a hospital. An SLO is the target for the patient's vital signs (e.g., "Heart rate must stay between 60 and 100").

• SLI (Service Level Indicator): This is the heart monitor. It measures the reality (e.g., "Current heart rate is 105").
• Error Budget: This is the patient's resilience. The patient can tolerate a heart rate of 105 for a few minutes (budget consumption) without permanent damage. However, if it stays there for an hour (budget exhaustion), you have a crisis.

In auto-remediation, we use the Burn Rate of this budget to determine urgency. A slow burn (minor degradation) is like a slight fever; you treat it with medication (email notification) in the morning. A fast burn (outage) is cardiac arrest; you treat it with a defibrillator (immediate pager/auto-restart) instantly.

Elastic Agent Builder: The Cognitive Engine

Standard automation is brittle because it is deterministic. It follows strict rules: "If X, do Y." Real-world incidents are messy. They require probabilistic reasoning and adaptation. Elastic Agent Builder provides this capability using Generative AI and Retrieval Augmented Generation (RAG).

Agent Builder allows you to define multiple agents that can be used to build Agentic AI experiences. It does more than just "chat”, by integrating into Elastic Workflows as a reasoning agent that can interact with other agents as needed.

• Contextual Analysis: It looks at the specific logs related to an alert. It deciphers cryptic stack traces. It determines whether there are related alerts or cases.
• Institutional Memory: It searches your internal runbooks, Jira tickets, and Confluence pages (ingested into Elasticsearch). It finds that "Error 503 on Service Payment" was fixed last month by rotating a specific key.
• Code Generation: It writes the specific ES|QL query needed to verify the impact of the issue.

Technical Implementation Scenarios

We will now look at two distinct scenarios: an Application-Level issue (SLO breach) and an Infrastructure-Level issue (Disk Space Exhaustion).

Scenario A: Application SLO Breach (Smart Escalation)

The Objective: Handle a service degradation intelligently. If the service is merely "degraded" (breaching a warning threshold but not down) and the time is near the start of the business day, we route the alert to email to avoid waking up engineers. If the service is "down" or if it is a critical off-hours failure, we page the on-call team.

Workflow Logic:

1. Trigger: SLO Burn Rate alert fires.
2. Check Context: Calculate if it is "Start of Business Day."
3. Decision: If Alert is Warning AND Time is Start of Day → Email. Else → PagerDuty.

Workflow YAML Definition:

name: Smart Escalation
enabled: false
description: Escalation based on business hours
triggers:
  - type: alert

steps:
  # Check for start day
  - name: start_day_check
    type: console
    with:
      message: |
        {%- assign hour = "now" | date: "%H", "Australia/Sydney" | plus: 0 %}
        {%- if hour >= 7 and hour <= 9 %}
          true
        {%- else %}
          false
        {%- endif %}
  - name: get_severity
    type: console
    with:
      message: '{{event.alerts[0]["kibana.alert.severity"] | default: "low"}}'
  # routing logic
  - name: routing_logic
    type: if
    condition: 'steps.get_severity.output: critical AND steps.trimmed_start_day.output: false'
    steps:
      - name: hard_notification
        type: pagerduty
        connector-id: "# Enter connector UUID here"
        with:
          eventAction: "trigger"
          severity: "{{steps.get_severity.output}}"
          summary: 'CRITICAL: {{event.alerts[0]["monitor.name"]}} SLO breached'
        
    else:
      - name: soft_notification
        type: email
        connector-id: Elastic-Cloud-SMTP
        with:
          to: ["<your_email>"]
          subject: '{{event.alerts[0]["kibana.alert.rule.name"]}} Failed'
          message: 'Reason: {{event.alerts[0]["kibana.alert.reason"]}}'

Scenario B: Infrastructure Auto-Remediation (Disk Space)

The Challenge:

In a large enterprise, "Out of Disk Space" is a silent killer. When /var/log fills up:

• Nginx crashes because it cannot write access logs.
• MySQL panics because it cannot write transaction logs.
• SSH fails because it cannot write to /var/log/auth.log, locking you out of the server you need to fix.

By the time the server stops responding to ping, it is too late. You need to catch the trend while the server is still reporting to your observability solution.

Note: To allow grouping of the infra service, we utilized the Add custom field feature of the Elastic Agent policy to add the my_org.custom.service field.

The Workflow:

1. Phase 1: Context & Health Verification
   • Detect: Elastic detects disk usage > 90% over time
   • Verify (Freshness): Confirm the host is still sending logs (it hasn't crashed yet).
   • Impact Analysis (HA): Check if other instances of this service are healthy. If 9/10 nodes are healthy, this is a P3 issue. If 1/2 nodes are healthy, it is P1.
2. Phase 2: Remediate
   • Remediate: Trigger an Ansible Tower playbook to rotate logs and clear cache.
   • Audit: Log the result.

Workflow YAML Definition:

name: "Disk_Space_Remediation"
description: "Detects full disk, checks HA status, and triggers Ansible Tower cleanup."
enabled: true
consts:
  ansible_webhook: "<your_ansible_url>"
triggers:
  - type: alert
steps:
  # ---------------------------------------------------------
  # Phase 1: Context & Health Verification
  # ---------------------------------------------------------
  # Check 1: Are logs still flowing? (Avoid false positives from dead nodes)
  - name: check_log_freshness
    type: elasticsearch.esql.query
    with:
      query: |
        FROM logs-*
        | WHERE host.name == "{{event.alerts[0]['host.hostname']}}"
        | STATS latest_log = MAX(@timestamp)
        | EVAL latency_sec = (TO_LONG(NOW()) - TO_LONG(latest_log)) / 1000
        | LIMIT 1
  # Get how many hosts serving that service within 24h
  - name: count_service_hosts
    type: elasticsearch.esql.query
    with:
      query: |
        FROM logs-*
        | WHERE @timestamp >= (NOW() - TO_TIMEDURATION("24 hours")) AND my_org.custom.service == "{{event.alerts[0]['kibana.alert.grouping'].my_org.custom.service}}"
        | STATS COUNT_DISTINCT(host.hostname)
  # Get how many hosts of the service have alerts within the last 15m 
  - name: count_distinct_alerts
    type: elasticsearch.esql.query
    with:
      query: |
        FROM .alerts-observability.metrics.alerts-default
        | WHERE @timestamp >= (NOW() - TO_TIMEDURATION("15 minutes")) AND kibana.alert.grouping.my_org.custom.service == "{{event.alerts[0]['kibana.alert.grouping'].my_org.custom.service}}"
        | STATS COUNT_DISTINCT(kibana.alert.grouping.host.name)
  # ---------------------------------------------------------
  # Phase 2: Remediation
  # ---------------------------------------------------------
  - name: routing_logic
    type: if
    condition: 'steps.check_log_freshness.output.values[0][1] <= 300'
    steps:
      # Action: Trigger Ansible Tower Job Template (Log Cleanup)
      # We pass the hostname as an 'extra_var' to the playbook
      - name: trigger_ansible
        type: http
        with:
          url: "{{consts.ansible_webhook}}"
          method: POST
          body:
            extra_vars:
              target_host: "{{event.alerts[0]['host.hostname']}}"
              remediation_type": clear_var_log
          headers: 
            Accept: application/json
            Content-Type: application/json
      # Action: Notify SRE with Context (SLO/HA)
      # If HA is healthy (>75% of the nodes), send standard notification. If HA is at risk, escalate.
      - name: notify_result
        type: console
        with:
          message: |
            {%- assign num_alerted_hosts = steps.count_distinct_alerts.output.values[0][0] | plus: 0 %}
            {%- assign num_service_hosts = steps.count_service_hosts.output.values[0][0] | plus: 0 %}
            {%- assign critical_ratio = num_alerted_hosts | divided_by: num_service_hosts | times: 1.0 %}
            {%- if critical_ratio >= 0.25 %} #ops-critical {%- else %} #ops-alerts{%- endif %}
            *Auto-Remediation Triggered: Disk Space*
            *Host:* {{event.alerts[0]['host.hostname']}}
            *Service:* {{event.alerts[0]['kibana.alert.grouping'].my_org.custom.service}}
            *Number of alerted instances:* {{steps.count_distinct_alerts.output.values[0][0]}}/{{steps.count_service_hosts.output.values[0][0]}}.
            *Action:* Ansible Playbook ID 15 triggered.
    else:
      # Fallback: If logs are stale, the agent might be dead. Manual intervention needed.
      - name: manual_escalation
        type: console
        with:
          message: |
            CRITICAL: Host {{event.alerts[0]['host.hostname']}} Unresponsive & Disk Full

Note: The notify_result is using a console step placeholder to consolidate demo output.

Analysis of the Infrastructure Scenario

This workflow demonstrates a mature "self-healing" capability:

1. Predictive vs. Reactive: By triggering at 90% (a safe threshold before systems start to crash), we use the Elastic Agent to fix the issue before the OS locks up the filesystem.
2. Safety Checks: The check_log_freshness step ensures we don't try to run a remote playbook on a server that has already disconnected, which would likely fail and create noise.
3. Business Awareness (HA): The notification logic (check_ha_status) understands priority. A disk issue on 1 of 50 web servers is a standard alert. A disk issue on 1 of 2 database nodes is a critical emergency. The workflow adjusts the Slack channel destination dynamically based on this reality.

Scenario C: AI-Augmented Triage (The "Digital Specialist")

The Challenge:

Application errors are often ambiguous. An alert says "Payment Failed: Error 9001." No dashboard explains this. The answer lies in a PDF runbook titled Legacy Payment API v2.0 stored on a forgotten SharePoint or Wiki, or in experienced SRE or application team members' heads.

Usually, the SRE wastes 30 minutes searching for this document or finding a team member with the contextual background knowledge.

The Solution:

By combining Search and Generative AI, we can couple telemetry and context into a single system. Using a RAG architectural approach, Elastic can surface relevant contextual information alongside telemetry using natural language queries to support analysts without switching tools. We index our runbooks and documentation into Elastic using Search tools and techniques which allow contextual information to be stored for use by AI Agents. This data provides the AI Agents with important contextual information about the environment, which can be used as part of intelligent automation workflows that can be applied to solve performance issues. In this scenario, when the workflow triggers, it searches an index (‘sre-knowledge-base’) to fetch any applicable runbooks which would be used to solve the issue, or documented knowledge articles. This information is then handed to the AI Agent to synthesize a fix and provide the analyst with recommended action steps.

The Workflow:

1. Phase 1: Retrieve Knowledge
   • Trigger: "Application Error" alert fires.
   • Retrieve Context (RAG): The workflow performs a Semantic Search against the ‘sre-knowledge-base’ index using the error message from the alert to find relevant contextual information about the error.
2. Phase 2: AI Analysis
   • Synthesize (AI): The workflow passes the Alert Logs and the Retrieved Runbook Snippets to the AI Agent, along with a prompt which asks: "Based on these logs and this runbook, what is the fix?"
3. Phase 3: Communication
   • Notify: The workflow posts the analysis to Slack.

Workflow YAML Definition:

name: AI_Runbook_Assistant
enabled: true
description: Uses RAG to find runbooks for unknown errors and suggests fixes.
triggers:
  - type: alert

steps:
  # ---------------------------------------------------------
  # Phase 1: Retrieve Knowledge (The "Memory" Lookup)
  # ---------------------------------------------------------
  
  # Perform a Semantic Search against internal runbooks
  # We use the alert reason as the search query

  - name: search_knowledge_base
    type: elasticsearch.esql.query
    with:
      query: |
        FROM sre-knowledge-base*
        | WHERE semantic_text : "What caused '{{event.alerts[0]['kibana.alert.rule.parameters'].criteria[0].value}}' of '{{event.alerts[0]['kibana.alert.grouping'].service.name}}' and if available, how to resolve?"
        | KEEP title, text
        | LIMIT 3

  # ---------------------------------------------------------
  # Phase 2: AI Analysis (The "Specialist" Reasoning)
  # ---------------------------------------------------------
  # Pass the retrieved knowledge + live logs to the AI
 - name: ai_analysis
    type: ai.agent
    with:
      agent_id: observability.agent
      message: |
        ISSUE DETECTED:
        {{ event.alerts[0]["kibana.alert.context"].conditions}} for {{event.alerts[0]['kibana.alert.grouping'].service.name}} 

        RELEVANT INTERNAL RUNBOOKS FOUND:
        {% for doc in steps.search_knowledge_base.output.values limit: 3 offset: 0 %}
        - Title: {{doc[0]}}
          Content: {{doc[1]}}
        {% endfor %}

        TASK:
        Analyze the issue. If the runbooks provide a solution, summarize it in a concise manner. 
        If not, suggest standard triage steps.
        Provide the output in Slack Markdown format.

  # ---------------------------------------------------------
  # Phase 3: Communication
  # ---------------------------------------------------------
  
  - name: notify_slack
    type: console 
    with:
      message: |
        channel: "#incident-war-room"
        text: |
          🚨 *Incident Detected*
          *Issue:* {{ event.alerts[0]["kibana.alert.context"].conditions}} for {{event.alerts[0]['kibana.alert.grouping'].service.name}} 
        
          🤖 *AI Agent Analysis:*
          {{ steps.ai_analysis.output.content }}
          

Note: The notification to Slack is simulated using a console step to consolidate demo output.

Analysis of the AI-Augmented Scenario

This workflow changes the nature of incident response:

1. Instant Context: It automates the "Search" phase of troubleshooting. The engineer enters the war room and the relevant runbook is already there.
2. Semantic Understanding: Unlike traditional keyword search, the RETRIEVE command uses semantic search techniques to find the required runbook, even if the wording doesn't match exactly (e.g., searching for "Login Fail" finds "Authentication Timeout").
3. Automated AI Triage: Leverage AI agents for first pass analysis and suggested remediation steps, giving analysts a head start on identifying the root cause and remediation steps. Get the output automatically sent to the engineers for proactive response action.
4. Reduced MTTR: By putting the solution in front of the engineer immediately, you eliminate the initial investigation delay.

The screenshots below show the transparency Workflows provides across the data analysed and transferred in each step of the process. You can see how Search is used to retrieve the appropriate run book, which is then incorporated with automated AI analysis for a comprehensive notification to engineers, which contains initial triage and remediation steps.

Figure 1: Output from automated Search for runbooks using Workflow

Figure 2: Output from automated AI triage of error using Elastic Workflows

Figure 3: AI Analysis, which is sent to Slack using Elastic Workflows

The Agentic Shift: Integrating Elastic Agent Builder

While workflows are powerful, they are deterministic. They follow the "If X, then Y" logic you explicitly code. However, modern operations often require probabilistic reasoning—the ability to figure out "X" in the first place. Additionally, RAG-based architectures often rely on a one-shot approach to prompt response, which can be limited for complex scenarios. This is where the Elastic Agent Builder can be used as part of Agentic AI experiences, where AI Agents are able to reason and interact autonomously in a series of steps to provide improved and highly accurate responses.

Bridging Reasoning and Action

Elastic Agent Builder allows you to create specialized AI agents that live within your Elastic environment. These agents have secure access to data in your Elastic deployment - both telemetry and unstructured information in the knowledge base for contextual background. More crucially, they have the ability to dynamically query this information and access Workflows as Tools, allowing them to assess and reason, and then take action appropriately .

This creates a powerful bidirectional architecture:

• Elastic Workflows call Agents: A workflow pauses to ask an agent for further investigation and decision (e.g., "Is this log pattern malicious?").
• Agents call Elastic Workflows: An agent, during a chat with an SRE, calls a workflow to execute a safe, pre-approved action (e.g., "Run the Disk Cleanup Workflow").

The "Hands" of the AI (MCP)

A major risk with AI in operations is hallucination. You do not want an LLM to guess commands and steps to execute. When using AI in automation, we need to give it tools and information to ground the LLM and ensure responses and actions are accurate and controlled. You solve this with the Model Context Protocol (MCP).

MCP is the standard that allows your Agent to connect to external systems safely. Instead of giving the Agent raw shell access, you give it a "Tool" called cleanup_disk, which has a specific command attached to it in the external system. Different types of tools can be defined, including Workflows, ES|QL queries or index patterns that expose a set of indices to the Agent. For example, instead of searching the ‘sre-knowledge-base’ index as a step in the workflow, a tool could be created to allow Agents to query the data autonomously when asked to do analysis. Elastic has the ability to serve tools via MCP to other systems as needed, as well as to native Agents defined in Agent Builder. For example, a ‘cleanup_disk’ tool maps directly to the deterministic workflow we built in Scenario B, which can be used as part of a revised AI-powered remediation flow:

The Human-in-the-loop Interaction Flow:

1. SRE: "Agent, the checkout service is failing."
2. Agent: Queries logs (using ES|QL tool). "It looks like the disk is full on node-01."
3. SRE: "Fix it."
4. Agent: "I will run the Disk_Space_Remediation workflow for node-01." (The agent invokes the workflow via MCP, where it is defined as the ‘cleanup_disk’ tool).
5. Elastic Workflows: Executes the Ansible job. (Deterministic, logged, and safe).

The Self-Healing Flow:

1. Alert: "The checkout service is failing."
2. Alert Action: "Run the SRE Agent Runbook workflow"
3. Elastic Workflows: Pass the alert and its context to the SRE agent.
4. SRE Agent: Queries logs (using ES|QL tool). "It looks like the disk is full on node-01."
5. SRE Agent: I have authorization and the tool to remediate this.
6. SRE Agent: "I will run the Disk_Space_Remediation workflow for node-01." (The agent invokes the workflow via MCP, where it is defined as the ‘cleanup_disk’ tool).
7. Elastic Workflows: Executes the Ansible job. (Deterministic, logged, and safe).

Implementation: Connecting Agent Builder

To enable this, you register your workflows as tools in the Agent Builder UI.

1. Define the Tool: In Agent Builder, create a new tool. Select "Elastic Workflow”, and provide the tool name and description. The description should be informative, as it is used by the AI Agents in understanding how the tool is used.
2. Bind the Workflow: Choose the Disk_Space_Remediation workflow from the drop-down.
3. Define Schema: Tell the agent what inputs the workflow needs (e.g., target_host).
4. Deploy: Create the AI Agent in the Agent Builder UI, which has access to the tool we just created. It can now execute the defined workflow, ensuring a sequence of actions is completed in a deterministic manner.

Custom Knowledge Base Tool creation API:

POST kbn:/api/agent_builder/tools
{
  "id": "search_knowledge_base",
  "type": "esql",
  "description": "Semantic Search against internal runbooks",
  "tags": [
    "knowledge-base",
    "observability"
  ],
  "configuration": {
    "query": """FROM sre-knowledge-base*
        | WHERE semantic_text : "What caused '?alert_criteria' of '?service_name' and if available, how to resolve?"
        | KEEP title, text
        | LIMIT 3""",
    "params": {}
  }
}

Custom Cleanup Tool creation API:

POST kbn:/api/agent_builder/tools
{
  "id": "cleanup_disk",
  "type": "workflow",
  "description": "Predefined Disk Space remediation workflow to perform the required cleanups. ",
  "tags": [],
  "configuration": {
    "workflow_id": "workflow-0f1f5ba7-0e75-4469-92b1-b85a032074e5",
    "wait_for_completion": true
  }
}

Custom SRE Agent creation API:

POST kbn:/api/agent_builder/agents
{
  "id": "sre_agent",
  "name": "SRE Agent",
  "description": "An SRE agent to analyze issues, provide summarized solutions from runbook where they exist or suggest standard triage steps when they don't exist. ",
  "labels": [],
  "avatar_color": "#61A2FF",
  "avatar_symbol": "🕵",
  "configuration": {
    "instructions": """You are a Senior SRE.

The issues/alerts detected will be provided in the following format
ISSUE DETECTED:
<alert_criteria> for <service_name>

Use the search_knowledge_base tool to extract known runbooks based on the alert_criteria and service name. 
For each runbook result, format it as follows.
- Title: <title from search_knowledge_base>
   Content: <text from search_knowledge_base>

TASK:
Analyze the issue. If the runbooks provide a solution, summarize it in a concise manner. 
If not, suggest standard triage steps. If the determined solution is to remediate via a disk cleanup, use the cleanup_disk tool. 
Provide the output in Slack Markdown format. Avoid unnecessary token usage be concise whilst descriptive. """,
    "tools": [
      {
        "tool_ids": [
          "platform.core.get_workflow_execution_status",
          "observability.get_alerts",
          "search_knowledge_base",
          "cleanup_disk"
        ]
      }
    ]
  }
}

Workflow YAML Definition:

version: "1"
name: SRE Agent Runbook
description: Use a custom SRE agent to suggests fixes and perform disk cleanup where appropriate.
enabled: true
triggers:
  - type: alert
steps:
  - name: sre_analysis
    type: ai.agent
    with:
      agent_id: sre_agent
      message: |
        ISSUE DETECTED:
        {{ event.alerts[0]["kibana.alert.context"].conditions}} for {{event.alerts[0]['kibana.alert.grouping'].service.name}} 
  - name: notify_slack
    type: console
    with:
      message: |+
        channel: "#incident-war-room"
        text: |
          🚨 *Incident Detected*
          *Issue:* {{ event.alerts[0]["kibana.alert.context"].conditions}} for {{event.alerts[0]['kibana.alert.grouping'].service.name}} 

          🤖 *SRE Agent Analysis:*
          {{ steps.sre_analysis.output }}

Note: The notification to Slack is simulated using a console step to consolidate demo output.

Workflow YAML Definition:

The screenshots below show how we create custom tools for the agent, then create a custom agent scoped to specific tooling. The level of control simplifies the workflow and sets up the framework for how users and agents interact and their boundaries.

Figure 4: Create a custom search_knowledge_base tool

Figure 5: Create a custom cleanup_disk tool

Figure 6: Create a custom SRE Agent with defined tools and Instructions

Figure 7: Leverage the custom SRE Agent within Elastic Workflows

This shifts your operating model from reactive, manual human-driven analysis and action to a proactive, automated analysis and action model. Using the tools explored in this article, Elastic brings together data, context and remediation for an intelligent, automated response that is reliable and controlled, with the option for ‘Human-in-the-loop’ check points or full AI-driven automation. This supports analysts in bridging the remediation gap and reducing MTTR by using AI and automation to augment root cause analysis efforts.

Conclusion

You operate in an era when the complexity of your systems has outpaced the capacity for manual management. The "Remediation Gap" is the single largest source of inefficiency in your IT operations. You cannot hire enough engineers to close it. You must close it with automation.

Proactive operations rely on bringing together telemetry data, contextual information and automation and AI capabilities to reduce time to repair. Elastic Observability provides the vision across systems in your environment. Elastic Workflows provides the hands to automate response actions. Elastic Agent Builder provides the brain for intelligent automation and AI-assisted triage. By unifying these three elements, you build a system that is not just monitored, but resilient. You move from a reactive posture—waiting for the phone to ring—to a proactive one, where the system heals itself before the customer notices.

For more examples of Elastic Workflows checkout the documentation and this GitHub repository.

Start small. Pick one recurring issue. Build one workflow. Measure the time saved. Trust is built on evidence. Once you prove that the machine can fix the machine, you change the nature of your operations forever.

Sign up for Elastic Cloud Serverless or Elastic Cloud and try this out.

The reasoning behind the design (the trust deficit, the remediation gap, and the sense → think → act → verify loop that frames everything below) is laid out in Automated reliability: The architecture of self-healing enterprises. Read that one for the why. Certificates are the right first workload for this pattern because the signal is unambiguous, the corrective action is well-understood, and the blast radius of the automation itself is small enough to build trust before applying the same pattern to noisier domains.

The full source (synthetics journeys, sample workflow YAMLs, sample ES|QL alert queries, the ingest pipeline, and a short-lived-cert local lab) lives in the aiops-synthetics-lab repository. The article walks through the patterns; the repo has the complete artifacts.

Mapping the build to the closed loop

Each phase of the closed loop maps to a specific Elastic component and a corresponding artifact in the companion repository. The table below is a quick reference; the diagram that follows shows how data flows between the phases.

Figure 1: Each phase maps to a specific Elastic primitive, and each phase only relies on clean data from the previous one. No out-of-band coordination, no shared mental state.

Discovery: picking the right two methods

There are four ways to discover certificates, but two methods provide rapid time to value as the following table shows.

Endpoint synthetics gives you the client view. Osquery gives you the host view. A load balancer can serve a valid public cert while a downstream internal host quietly serves an expired one to its peers; invisible from the outside, fully visible from the keystore. You need both.

The left side (yellow) shows agent-based monitoring: Elastic Agent runs on each host and uses the Osquery certificates pack to read the local operating system keystore — /etc/ssl on Linux, CurrentUser and LocalMachine stores on Windows. This is the host view: certificates that services hold regardless of whether those services expose a public endpoint. The right side (blue) shows agentless monitoring: Elastic Synthetics executes browser journeys against reachable HTTPS endpoints and captures the full certificate chain as a client would see it. Both paths normalize output to ECS tls.server.x509.* fields, so the workflow can join them on common_name without any translation step.

Figure 2: Synthetics validates what clients see; Osquery enumerates what hosts hold. Both feed ECS-mapped tls.server.x509.* so the workflow can join them on common_name without translation.

Architecture overview

The architecture has four horizontal layers, each corresponding to one phase of the loop. Signal collection (top) shows the three data sources: Synthetics is the primary alert driver, Osquery is queried by the workflow for host discovery, and canary curl_certificate validates impact on demand. Alerting runs an ES|QL rule on synthetics-* and fires when a cert falls within the expiry window. Smart escalation and remediation is the workflow itself: pre-flight severity mapping, host discovery via Osquery, then branching into auto-rotation for certs expiring within 7 days, Jira ticketing for 7-90 day windows, or PagerDuty for anything that needs a human. Impact validation runs the canary workflow on demand to measure blast radius before and after any rotation.

Figure 3: Four layers, one closed loop. Synthetics fires the alert, Osquery maps the target hosts, and canary validation measures impact.

Prerequisites

Sense: signal collection

Clone the companion repository to get the complete set of artifacts used throughout this post: Synthetics browser journeys, Osquery pack configuration, Elastic Workflow YAML definitions, ES|QL alert queries, the ingest pipeline definition, and a local lab for testing against short-lived certificates. The npm ci step installs the Node.js dependencies required for running browser journeys locally.

git clone https://github.com/adrianchen-es/aiops-synthetics-lab.git
cd aiops-synthetics-lab
npm ci

CSV-driven endpoints

Endpoints are managed through a CSV file, so adding new hosts requires no code changes. Just edit journeys/tls-browser/tls-target-hosts.csv:

host,criticality,assertionText,assertionSelector
elastic.co,critical,Elastic,h1
internal-api.mycompany.com,high,,
payment-gateway.internal,critical,,

criticality becomes a journey tag (criticality:critical) which flows into the alert payload and drives escalation routing — no CMDB lookup required for basic triage. npm run generate:tls-targets rebuilds the TypeScript host module from the CSV and runs automatically before npm test and npm run push. The repo has additional groups under journeys/tls/, journeys/demos/, and journeys/kibana/ for non-browser TLS, badssl-style demos, and a multi-step Kibana login check, respectively npm run push:tls and friends scope deploys to one folder.

The browser TLS journey

Each host runs its own journey. Certificate inspection relies on Node's built-in tls module, where cert.fingerprint256 is precomputed by OpenSSL during the handshake, so it incurs no extra cost.

import { journey, step, expect } from '@elastic/synthetics';
import { TLS_TARGET_HOSTS } from '../../helpers/tlsTargetHosts.tls-browser.generated';
import { fetchCertInfo, checkCertTrusted, logCertInfo } from '../../helpers/tls';

for (const { host, criticality, assertionText, assertionSelector } of TLS_TARGET_HOSTS) {
  journey(
    { name: `TLS Browser Check - ${host}`, tags: criticality ? [`criticality:${criticality}`] : [] },
    ({ page }) => {
      step(`TLS Validation for ${host}:443`, async () => {
        // Route-stub: gives Synthetics UI the actual hostname instead of about:blank.
        await page.route('**/*', route => route.fulfill({ status: 200, body: 'TLS check context' }));
        await page.goto(`https://${host}`, { waitUntil: 'commit' });

        const cachedCert = await fetchCertInfo(host, 443);
        const trusted = await checkCertTrusted(host, 443);
        logCertInfo(host, 443, cachedCert); // emits TLS_CERT stdout line for ingest pipeline

        expect(cachedCert.sha256).toMatch(/^([0-9A-F]{2}:){31}[0-9A-F]{2}$/);
        expect(cachedCert.validTo.getTime(), `Certificate expired: ${host}`).toBeGreaterThan(Date.now());
        expect(trusted, `CA untrusted: ${host}`).toBe(true);
      });

      step(`Navigate to ${host} and verify page content`, async () => {
        await page.context().route('**', route => route.continue());
        const response = await page.goto(`https://${host}`, { waitUntil: 'domcontentloaded', timeout: 30_000 });
        expect(response).not.toBeNull();
        if (assertionText && assertionSelector) {
          await expect(page.locator(assertionSelector)).toContainText(assertionText);
        }
      });
    }
  );
}

Both successful and failed runs index tls.server.x509.* into synthetics-*; the alert rule reads both.

The ingest pipeline (don't skip this)

logCertInfo() emits a structured TLS_CERT stdout line per check. Without an ingest pipeline, that lands under synthetics.payload.message and the Kibana TLS Summary card stays empty even when journeys are succeeding. Most certificate-monitoring tutorials skip this step; it caught me out the first time.

curl -X PUT -H "Content-Type: application/json" \
  -u "elastic:${ELASTIC_PASSWORD}" \
  "https://<YOUR_CLUSTER_HOST>:9243/_ingest/pipeline/synthetics-browser%40custom" \
  --data-binary @docs/ingest-pipeline-synthetics-browser.json

The pipeline parses the embedded JSON into tls.server.x509 and tls.server.hash, stripping colons from the fingerprint hex to match the UI's expected format.

Deploying monitors via CI/CD

export KIBANA_URL="https://your-deployment.kb.us-east-1.aws.elastic-cloud.com"
export SYNTHETICS_API_KEY="<your-kibana-api-key>"
npm run push

Adding a new service is a CSV row in a PR, not a Kibana form fill; the same monitors-as-code idea covered for Logstash in Logstash Pipeline Management & Configuration with GitOps. The repo's GitHub Actions workflow runs tsc --noEmit, test:dry, and test:unit on every PR (none of which need network access, so the loop catches journey breakage before merge).

Osquery host certificate inventory

Browser synthetics only sees certificates bound to active web listeners. Backend databases, internal gRPC services, and message queue brokers don't surface in any Playwright test. The Osquery pack queries each host's certificate store daily and writes to logs-osquery_manager.result* with these ECS mappings:

"ecs_mapping": {
  "tls.server.x509.subject.common_name": { "field": "common_name" },
  "tls.server.x509.not_before":          { "field": "not_valid_before" },
  "tls.server.x509.not_after":           { "field": "not_valid_after" },
  "tls.server.hash.sha1":                { "field": "sha1" },
  "file.path":                           { "field": "path" }
}

Without ECS mapping, certificate fields land in proprietary osquery.* columns. With it, tls.server.x509.subject.common_name is directly matchable against the Synthetics alert payload, which is what the workflow's host-discovery step depends on.

CN matching is necessary but not sufficient. Modern certificates carry many SANs and wildcard subjects. The host-discovery query joins on common_name because that is what the alert payload exposes most cleanly; in environments with heavy wildcard usage (*.svc.cluster.local, *.internal) extend the join to also match against tls.server.x509.alternative_names and filter out shared wildcards renewed centrally — otherwise one Let's-Encrypt-backed wildcard renewal will fan out the workflow to every host in the cluster on the same day.

Think: alerting on Synthetics data

The alert rule runs against synthetics-*, not Osquery. Synthetics captures the certificate as the client actually sees it, including intermediate-chain issues that host keystores can't detect. Osquery is consulted later, inside the workflow to find which hosts to act on.

Create an Elasticsearch query rule with this ES|QL (full file: docs/elastic-samples/alerts/01-certificate-approaching-30d-expiring.md — note the repo file uses a 30-day window for stricter alerting; the 90-day version below matches my own preferred default):

FROM synthetics-*,*:synthetics-*
| WHERE tls.server.x509.not_before IS NOT NULL
    AND tls.server.x509.not_after IS NOT NULL
    AND tls.server.x509.subject.common_name IS NOT NULL
| STATS
    record_count = COUNT(*),
    tls.server.x509.not_after = MAX(tls.server.x509.not_after),
    tls.server.x509.not_before = MIN(tls.server.x509.not_before),
    monitor.name = VALUES(monitor.name),
    @timestamp = MAX(@timestamp)
  BY tls.server.x509.subject.common_name, tags
| WHERE NOW() + 90d > tls.server.x509.not_after
    OR tls.server.x509.not_before > NOW()
| EVAL days_until_expiry = DATE_DIFF("days",
    tls.server.x509.not_before,
    tls.server.x509.not_after)
| KEEP @timestamp, tls.server.x509.subject.common_name,
       tls.server.x509.not_before, tls.server.x509.not_after,
       monitor.name, tags, days_until_expiry
| SORT @timestamp
| LIMIT 1000

The STATS BY common_name, tags aggregation deduplicates to one row per certificate per criticality tag, so the rule fires once per expiring cert, not once per monitor execution per location. MAX(not_after) and MIN(not_before) pick the definitive validity window across all observed documents.

Two tuning notes worth flagging:

• ACME certificates (90-day validity) will sit at days_until_expiry near 90 every check cycle. Run a separate rule with a tighter threshold (< 14 days) and a different escalation path for them.
• For cross-cluster index patterns (*:synthetics-*), confirm tags is populated consistently across clusters before relying on it for routing.

Act: smart certificate rotation escalation and remediation

The full workflow lives at docs/elastic-samples/workflows/01-smart-certificate-rotation-escalation-remediation.yaml. Copy it into Kibana → Stack Management → Workflows, swap connector IDs, and you have the picture below running. What follows is just the patterns you can't intuit by skimming the YAML.

Figure 4: Every leaf has a contract — auto-rotate, ticket, soft-notify, or page. The workflow never silently drops a certificate. The image below shows how smart escalation and remediation operate. Beneath it, the reasoning for each stage is explained.

Figure 5: The same logic on the time axis instead of the decision axis.

The patterns that matter

Setting workflow variables. The pre-flight steps compute and store values that every later branch reads. In 9.3, the only way to do this was to abuse console steps — a Liquid template in message renders to a string accessible at steps.<name>.output. In 9.4, data.set makes this an explicit, named, testable operation with typed output fields.

9.3 — console as expression evaluator (compatible with 9.3 and 9.4; not recommended for new workflows):

- name: business_hours_check
  type: console
  with:
    message: |-
      {%- assign hour = "now" | date: "%H", "Australia/Sydney" | plus: 0 %}
      {%- if hour >= 8 and hour < 17 %}true{%- else %}false{%- endif %}

9.4 — data.set (preferred for new workflows):

- name: business_hours_check
  type: data.set
  with:
    is_business_hours: |-
      {%- assign hour = "now" | date: "%H", "Australia/Sydney" | plus: 0 -%}
      {%- if hour >= 8 and hour < 17 %}true{%- else %}false{%- endif -%}

The data.set output is accessible at steps.business_hours_check.output.is_business_hours, a named field rather than a raw string. The same upgrade applies to the severity_mapping and pagerduty_criticality_mapping steps in the full workflow YAML. Swap the timezone to suit your team.

Cross-source host discovery. The alert fires on Synthetics; this query asks Osquery which hosts carry the expiring CN:

- name: find_affected_hosts
  type: elasticsearch.esql.query
  with:
    query: |
      FROM logs-osquery_manager.result-*
      | WHERE tls.server.x509.subject.common_name == "{{ event.alerts[0]['tls.server.x509.subject.common_name'][0] }}"
        AND `file.path` != "LocalMachine\\Certificate Enrollment Requests"
      | EVAL days_until_expiry = DATE_DIFF("days", tls.server.x509.not_before, tls.server.x509.not_after)
      | KEEP host.hostname, tls.server.x509.subject.common_name,
             tls.server.x509.not_before, tls.server.x509.not_after,
             file.path, days_until_expiry
    format: json

Output columns are accessed positionally below: [0] host.hostname, [5] days_until_expiry. The Windows Certificate Enrollment Requests exclusion drops pending renewal requests from the keystore — they generate false matches mid-renewal.

The circuit breaker. Don't act on a host that can't currently observe itself:

- name: check_host_freshness
  type: elasticsearch.esql.query
  with:
    query: |
      FROM metrics-system.cpu-*
      | WHERE host.name == "{{ steps.find_affected_hosts.output.values[0][0] }}"
      | STATS latest_metric = MAX(@timestamp)
      | EVAL latency_sec = (TO_LONG(NOW()) - TO_LONG(latest_metric)) / 1000
      | LIMIT 1

- name: remediation_routing_logic
  type: if
  condition: 'steps.check_host_freshness.output.values[1] <= 300'
  steps:
    - name: trigger_ansible_rotation
      type: http
      with:
        url: "{{consts.ansible_webhook}}"
        method: POST
        body: { extra_vars: { target_host: "{{ steps.find_affected_hosts.output.values[0][0] }}" } }
        headers: { Authorization: "Bearer {{consts.ansible_token}}" }

This is the trust deficit circuit breaker made concrete. My first version of this workflow didn't have the freshness check, we cheerfully fired Ansible jobs at hosts that had already left the building. Tower returned 200 because the job queued, the workflow declared success, and the certificate never actually rotated. Two lines of YAML and one postmortem you don't have to write.

Ansible Tower is illustrative — type: http works with any webhook-addressable executor: Rundeck, AWX, Salt, GitHub Actions workflow_dispatch, a thin internal service in front of kubectl rollout restart. The pattern that matters is the health check before the action.

The ansible_token literal in consts: is for prose readability. In production, inject the credential via Workflows' secret handling (or load it from a referenced connector) so it never lands in version-controlled YAML.

Severity-aware escalation when the circuit breaker fires. Low severity outside business hours gets a non-urgent email; everything else pages, with severity translated to PagerDuty's vocabulary. Same 9.3/9.4 pattern as above:

9.3:

- name: pagerduty_criticality_mapping
  type: console
  with:
    message: |-
      {%- assign sev = steps.severity_mapping.output -%}
      {%- case sev -%}
        {%- when "critical" -%}critical
        {%- when "high"     -%}error
        {%- when "medium"   -%}warning
        {%- else            -%}info
      {%- endcase -%}

9.4:

- name: pagerduty_criticality_mapping
  type: data.set
  with:
    severity: |-
      {%- assign sev = steps.severity_mapping.output.severity -%}
      {%- case sev -%}
        {%- when "critical" -%}critical
        {%- when "high"     -%}error
        {%- when "medium"   -%}warning
        {%- else            -%}info
      {%- endcase -%}

The mapping step has to run before hard_notification references its output; keep that ordering if you refactor. In 9.4, the data.set output is referenced by field name (steps.severity_mapping.output.severity); in 9.3, the console step exposes a raw string at steps.severity_mapping.output.

Verify: closing the loop

A self-healing system that takes an action and walks away is not self-healing: it's just hopeful. Verify is the phase that distinguishes a closed loop from a fire-and-forget script, and it's where most "automated remediation" implementations quietly fall short.

For certificates, verification has two natural anchors. The next Synthetics interval (1–10 minutes after rotation) re-reads the chain — the new not_after and fingerprint land in synthetics-* and the alert rule simply stops matching that CN. No alert is the right outcome. For deeper confidence, the canary curl_certificate workflow confirms that the fleet — not just the public endpoint — has the new cert. Add the following block at the end of the rotation success path:

- name: wait_for_rotation
  type: wait
  with:
    duration: 120s

- name: verify_rotation
  type: elasticsearch.esql.query
  with:
    query: |
      FROM synthetics-*
      | WHERE tls.server.x509.subject.common_name
          == "{{ event.alerts[0]['tls.server.x509.subject.common_name'][0] }}"
        AND @timestamp > NOW() - 5 minutes
      | STATS new_not_after = MAX(tls.server.x509.not_after),
              new_fingerprint = VALUES(tls.server.hash.sha256)
      | EVAL days_remaining = DATE_DIFF("days", NOW(), new_not_after)

- name: rotation_outcome
  type: if
  condition: "${{ steps.verify_rotation.output.values[0][2] > 30 }}"
  steps:
    - name: notify_verified
      type: slack
      connector-id: "slack-sre-channel-uuid"
      with:
        message: "✅ Rotation verified. {{ steps.verify_rotation.output.values[0][2] }} days remaining."
  else:
    - name: notify_verification_failed
      type: pagerduty
      connector-id: "<your-pagerduty-connector-uuid>"
      with:
        eventAction: "trigger"
        severity: "error"
        summary: "Rotation triggered but Synthetics still shows old cert. Investigate Ansible job and DNS/cache."

The verify query reads from the same source the alert rule reads from, so the two definitions can never drift. The check is numeric (days_remaining > 30), nothing to misinterpret. And a failed verification escalates harder than the original: a cert that was about to expire and silently failed to rotate is the more dangerous state, because automation has already swallowed the warning a human would otherwise have seen.

The 120-second pause is conservative. For ACME certs renewed by a sidecar, the next monitor interval (often under a minute) is enough. For Ansible-driven PKI rotations that touch a reverse-proxy reload, two minutes feels about right in my experience; long enough not to chase a stale cache, short enough that on-call doesn't go to bed thinking the rotation worked.

To rehearse the whole loop end-to-end before pointing it at production, the repo's local-lab/ directory ships a Docker Compose setup that issues a one-day self-signed cert and serves it from Nginx or Apache. Renew the cert, restart the container, and watch Synthetics pick up the new fingerprint within one monitor cycle — same shape as a real rotation, with no production risk.

Skip Verify and you have built an open loop dressed up as a closed one. The trust deficit returns the first time a silent rotation failure becomes a 4 AM page.

Impact validation: the canary workflow

The combined workflow handles the predictable case. Some incidents fall outside the predictable case: a Synthetic monitor failing with a TLS handshake timeout where the chain is not obvious, or a situation where you want to confirm blast radius before committing to rotation. For these, an Osquery curl_certificate live query gives a client-side view of the TLS chain from the perspective of machines that actually communicate with the target. A reverse proxy can hold a valid public cert while a downstream internal host serves an expired CA-signed cert that breaks microservice calls: invisible to endpoint polling, fully visible from internal canary hosts.

Two workflow files are in the repo: 02-canary-certificate-impact-check.yaml (9.3-compatible) and 02-canary-certificate-impact-check-9.4.yaml (GA default for 9.4+). Two ways to invoke either:

• Manually in Kibana with target_host and target_port inputs.
• Autonomously as a registered Agent Builder tool, used by the SRE Certificate Agent during triage — same registration pattern as How to Troubleshoot Kubernetes Pod Restarts & OOMKilled Events with Agent Builder.

The interesting moves inside the YAML are: dispatch a curl_certificate live query with agent_all: true, poll the live-query API until status is completed, then aggregate the asynchronous results with one ES|QL pass. The polling step is where 9.3 and 9.4 differ most visibly.

9.3 — foreach + intentional failure workaround:

consts:
  items: [1]   # single-element list; foreach runs once per retry attempt

steps:
  - name: while_not_workaround_loop
    type: foreach
    foreach: "${{consts.items}}"
    steps:
      - name: osquery_check_query_completion
        type: kibana.request
        with:
          method: GET
          path: /api/osquery/live_queries/{{ steps.osquery_check_host.output.data.action_id }}
      - name: wait_10s
        type: wait
        with:
          duration: 10s
      - name: retry
        type: http
        if: "${{ steps.osquery_check_query_completion.output.data.status != 'completed' }}"
        with:
          url: "https://httpbin.org/status/404"   # deliberate 404 triggers on-failure
    on-failure:
      retry:
        max-attempts: 9

9.4 — native while loop with data.set:

- name: poll_osquery_completion
    type: while
    condition: "${{ steps.osquery_poll_state.output.query_status != 'completed' }}"
    max-iterations:
      limit: 10
      on-limit: fail
    steps:
      - name: wait_10s
        type: wait
        with:
          duration: 10s
      - name: check_query_completion
        type: kibana.request
        with:
          method: GET
          path: /api/osquery/live_queries/{{ steps.osquery_check_host.output.data.action_id }}
      - name: osquery_poll_state
        type: data.set
        with:
          query_status: "{{ steps.check_query_completion.output.data.status }}"

See the 9.3 → 9.4 migration guide for a full list of changes. The ES|QL aggregation step is identical in both versions:

FROM logs-osquery_manager.result*
| WHERE action_id == "{{ steps.osquery_check_host.output.data.queries[0].action_id }}"
| WHERE NOW() > tls.server.x509.not_after
| STATS expired_canary_hosts = COUNT_DISTINCT(agent.name)
    BY certificate_validity_duration = DATE_DIFF("day",
        tls.server.x509.not_before,
        tls.server.x509.not_after)

expired_canary_hosts is the count of agents that confirmed expiry. certificate_validity_duration > 90 distinguishes long-lived PKI from short-lived ACME, which the workflow uses to route the response — replace the console placeholders in the YAML with Slack / PagerDuty / a follow-on remediation step as preferred.

Registering as an agent tool

Register the workflow as a tool so any Agent Builder agent can invoke it during triage:

POST kbn:/api/agent_builder/tools
{
  "id": "o11y.canary_certificate_check",
  "type": "workflow",
  "description": "Queries curl_certificate from all fleet agents to confirm real TLS impact for a given hostname and port. Returns count of affected canary hosts and certificate type (PKI vs ACME).",
  "tags": ["observability", "certificates"],
  "configuration": {
    "workflow_id": "<your-osquery-curl-workflow-id>",
    "wait_for_completion": true
  }
}

Packaging as a skill (9.4+)

Elastic 9.4 introduces skills, a higher-level abstraction over individual tools. Where a tool is a single discrete operation (run a query, call a webhook), a skill bundles instructions, tools, and reference context into a reusable capability pack that an agent loads selectively rather than holding in the system prompt at all times.

For certificate triage, a skill lets you combine the canary impact check tool with domain-specific instructions, for example, how to interpret PKI vs ACME results, when to escalate vs when to redirect to DNS, and which follow-on workflows to invoke. The agent selects the skill automatically when the conversation context signals a TLS problem, or users invoke it directly with a slash command (/cert-triage).

Skills are managed through the skill library in Agent Builder and are available to any agent in the workspace: one definition, shared across the SRE agent, the on-call agent, and any future agents that need certificate expertise. Retrieve available skills with GET /api/agent_builder/skills.

A typical agent-driven triage with the skill loaded:

SRE: "The Synthetics monitor for blueprint-portal is failing. Investigate."

Agent: [Reviews synthetics logs — TLS handshake error on blueprint-portal.internal]
Agent: [Invokes o11y.canary_certificate_check for blueprint-portal.internal:443]

Canary result: expired intermediate cert confirmed on 3 of 12 agents — PKI class (365-day cert)

Agent: "The Synthetic failure is caused by an expired intermediate CA on the downstream
gateway. Three canary hosts confirm. The leaf cert is valid — this is a CA renewal gap on
the internal chain, not a service certificate expiry."

SRE: "Execute the proxy cache flush workflow."
Agent: [Invokes proxy flush workflow via MCP]
Agent: "Flush confirmed. Monitor should recover within one check interval."

When the canary check comes back clean, the agent redirects to DNS, routing, or upstream service health without the SRE needing to query anything by hand.

Operational considerations

Observability of the workflow itself. The question I get asked once people see this pattern is the same: "how do you know the workflow is still doing what you think?" The Workflows run history in Kibana shows every execution, status, and step output. And the clearest sign that the loop is closing, straight from the alerts index:

// Certificates that fired the alert more than once in the last 14 days —
// strong signal that rotation is failing silently and the loop is open
FROM .alerts-observability.metrics.alerts-default
| WHERE @timestamp > NOW() - 14 days
  AND kibana.alert.rule.name == "Certificate Expiry"
| STATS fires = COUNT(*) BY tls.server.x509.subject.common_name
| WHERE fires > 1
| SORT fires DESC

Pin that query somewhere visible. A CN appearing more than once means the workflow is acting, but the action isn't landing; exactly the failure mode this architecture exists to surface. The companion meta-alert query is checked in at docs/elastic-samples/alerts/02-certificate-workflow-verification.md.

Criticality context for backend hosts. The CSV tags field only covers endpoints with active Synthetic monitors. For Osquery-discovered certificates on backend services with no monitor, add a LOOKUP JOIN enterprise_cmdb ON host.name step before remediate_or_notify. Two patterns keep the CMDB index current: pull (Logstash sync of CMDB exports) or push (the workflow POSTs back via ServiceNow webhook when Osquery surfaces a host that isn't in the CMDB; that's a real finding worth investigating).

Rehearse end-to-end before production. The repo's local-lab/ directory issues a 1-day self-signed cert, serves it via Nginx (port 8443) or Apache (port 8444), and includes a renew-tls-certs.sh script. Run the loop against it for a day or two before pointing the workflow at services that matter.

What success looks like

The single highest-value metric is the last one. A successful self-healing build is one that no human notices for months at a time.

Repository structure

Figure 6: CSV in, monitors out. Sample workflow YAML, alert ES|QL, ingest pipeline, and a short-lived-cert local lab are all colocated and version-controlled together.

Full reference at github.com/adrianchen-es/aiops-synthetics-lab.

Next steps

1. Clone the repo, edit journeys/tls-browser/tls-target-hosts.csv with your endpoints and criticality tags.
2. Install the ingest pipeline once: PUT _ingest/pipeline/synthetics-browser@custom.
3. Run npm run test:demos against revoked.badssl.com to confirm Sense and the ingest pipeline are wired correctly. Optionally point the journey at the local-lab/ Nginx (https://127.0.0.1:8443) to rehearse the full loop with 1-day certificates.
4. Push monitors with npm run push.
5. Deploy the Osquery pack via the Kibana API.
6. Paste the ES|QL from docs/elastic-samples/alerts/01-…md into a Kibana ES|QL rule.
7. Import docs/elastic-samples/workflows/01-…yaml as a Kibana Workflow and configure connector IDs. Add the Verify block before enabling auto-rotation in production.
8. Optionally import docs/elastic-samples/workflows/02-…yaml and register it as an Agent Builder tool.

Try it on Elastic Cloud Serverless or Elastic Cloud.

Frequently asked questions

Why do TLS certificates keep expiring in production if the expiry date is known in advance?

The expiry date is known, but acting on it requires a reliable chain from detection to rotation — and that chain typically runs through spreadsheets, manual calendar reminders, and whoever happens to notice first. Automation closes that gap: Elastic Synthetics and Osquery provide continuous visibility into expiry state, while Elastic Workflows manages rotation and verification without manual intervention.

How is monitoring TLS certificates with Elastic Synthetics different from a simple cron job or alerting rule?

A Synthetics journey captures the full certificate chain as the client actually sees it, including intermediate CA issues that a host-side check misses entirely. It also maps results into tls.server.x509.* ECS fields, so the same data feeds both the alert rule and the workflow's host-discovery query — no translation layer, no schema drift between detection and remediation.

How do I make sure automated certificate rotation actually worked and didn't just queue a job?

After triggering rotation, the Elastic Workflow waits 120 seconds and queries synthetics-* with ES|QL to check whether the new not_after date is more than 30 days out. If the old certificate is still present, the workflow escalates to PagerDuty at higher severity than the original alert — because a silent rotation failure is more dangerous than the expiry warning it consumed.

What happens if Ansible fires a rotation job at a host that's already been decommissioned?

The workflow includes a circuit-breaker step that queries metrics-system.cpu-* before triggering Ansible. If the host hasn't reported telemetry in the last 300 seconds, the rotation is skipped and an escalation is sent instead. Without this check, the Ansible Tower job queues and returns 200, the workflow declares success, and the certificate never actually rotates.

When should I use the Osquery curl_certificate canary workflow instead of the main rotation workflow?

The main rotation workflow handles predictable expiry detected by Synthetics. The canary workflow is for triage when the failure mode is unclear — for example, a TLS handshake timeout where the root cause isn't obvious. It dispatches a curl_certificate live query to all fleet agents simultaneously and tells you which machines confirm the expiry, distinguishing a public endpoint failure from an internal chain failure that Synthetics can't see.

Does this pipeline require Elastic Synthetics monitors for every certificate I want to cover?

No. The Osquery component independently inventories the local certificate store on every enrolled host daily, surfacing certificates used by backend databases, gRPC services, and message brokers that have no HTTP endpoint. Synthetics covers what clients see from the outside; Osquery covers what hosts hold internally. Both write to tls.server.x509.* so the remediation workflow can join them on common name without a translation layer.

What are the limitations of joining Synthetics and Osquery data on common name?

Common name matching works well for simple certificate topologies. In environments with heavy wildcard usage (*.svc.cluster.local, *.internal), a single Let's Encrypt wildcard renewal can fan out the remediation workflow to every host in the cluster. The fix is to extend the join to tls.server.x509.alternative_names and filter out centrally managed wildcards before triggering per-host rotation.

What version of the Elastic Stack do I need to use Elastic Workflows for certificate monitoring?

Elastic Workflows is generally available in Elastic Stack 9.4 and was available as a Technical Preview in 9.3. The ES|QL LOOKUP JOIN used for host discovery requires 9.1 or later. For new deployments, target 9.4 to access the native while loop and data.set steps, which eliminate the workarounds required in 9.3.

What is the difference between a tool and a skill in Elastic Agent Builder?

A tool is a single discrete operation — run a query, call a webhook, trigger a workflow. A skill bundles instructions, tools, and reference context into a reusable capability pack that an agent loads selectively rather than keeping in the system prompt at all times. For certificate triage, registering the canary impact check as a tool gives an agent one action; packaging it as a skill gives the agent the context to interpret results, decide when to escalate, and invoke follow-on workflows automatically.

Related reading

• Automated Reliability: The Architecture of Self-Healing Enterprises — the inspiration for this build. Covers the why: the remediation gap, the trust deficit, and the sense → shink → sct → verify loop this implementation realizes.
• How to Troubleshoot Kubernetes Pod Restarts & OOMKilled Events with Agent Builder — a directly comparable Agent Builder triage pattern; the SRE Certificate Agent here uses the same registration model.
• Agentic CI/CD: Kubernetes Deployment Gates with Elastic MCP Server — extends the workflow-as-tool pattern to deployment pipelines; relevant if you want to gate releases on certificate health.
• Elastic Ramen: A CLI harness for SRE investigation and remediation — brings Agent Builder conversations, skills, and tools into the terminal so engineers can move from investigation to remediation in a single thread.

From Fragile Art Project to Auditable Asset: How to Tame Your Logstash Configurations with Version Control and Automation

Observability ensures system health, performance, and security. Logstash drives this by processing and routing your data. But as you scale, manual configuration management becomes a bottleneck. It leads to errors, outages, and security gaps. You need a better way.

This blog post shows you how to manage Logstash pipelines using GitOps. You will use Git as your single source of truth and automate deployments to increase stability, security, and efficiency of your enterprise organisation’s observability infrastructure. 

This blog post details the benefits of this methodology and provides a practical implementation model using GitHub for version control and Jenkins for Continuous Integration and Continuous Deployment (CI/CD).

The Unsung Hero: Why Logstash Remains a Cornerstone of Enterprise Data Strategy

In the evolving landscape of observability and data pipelines, Logstash remains one of the most powerful and reliable components in the Elastic ecosystem. While it may not always take the spotlight, its depth of capability, flexibility, and resilience make it essential for enterprises managing complex, varied data streams. Logstash offers four main benefits:

• Extensive Integration Support: Logstash supports a wide array of input and output plugins — including Kafka, syslog, Beats, cloud services, and databases — making it ideal for ingesting data from diverse environments and routing it across your architecture.

• Advanced Data Transformation: With rich filtering capabilities and optional Ruby scripting, Logstash enables complex enrichment, field manipulation, and conditional routing — allowing teams to standardise and prepare data early in the pipeline.

• Offloading Elasticsearch Ingest Load: The elastic_integration filter replicates ingest pipeline logic in Logstash, enabling upstream transformations that reduce processing overhead on Elasticsearch and streamline the indexing path.

• Operational Resilience with Persistent Queues: Logstash’s persistent-queue buffers data during downstream slowdowns or outages, helping smooth ingestion spikes, prevent data loss, and maintain stability under load.

In modern CI/CD workflows, where automation and rapid iteration are standard, Logstash’s maturity and flexibility continue to make it a dependable choice — quietly powering the data flows that keep observability pipelines running strong.

The Case for a GitOps-Driven Observability Strategy

GitOps is a paradigm that applies proven DevOps best practices such as version control, collaboration, compliance, and CI/CD to infrastructure and configuration management. When applied to Logstash, this means that every pipeline configuration is treated as code—defined, versioned, reviewed, and deployed from a Git repository.

For enterprise environments, the adoption of a GitOps model for Logstash pipelines offers compelling advantages:

• Enhanced Auditability and Compliance: Every pipeline modification is captured as a Git commit, creating an immutable, chronological audit trail. This provides unparalleled visibility into who made what change, when, and why, which is indispensable for meeting regulatory compliance requirements and conducting security audits.

• Improved System Stability and Reliability: The risk of deploying faulty configurations is drastically reduced. By enforcing a pull request (PR) workflow, all changes undergo peer review and automated validation before they are merged and deployed. In the event of an incident caused by a new configuration, a rollback is as fast and straightforward as reverting a Git commit.

• Increased Automation and Operational Efficiency: Automating the deployment lifecycle eliminates manual, error-prone configuration tasks. This frees up skilled engineers from routine operational duties, allowing them to focus on higher-value activities such as optimising data flows, improving analytics, and strengthening security postures.

• Fostered Cross-Team Collaboration: Git provides a universal and well-understood platform for collaboration. Development, Security, and Operations (DevSecOps) teams can work together seamlessly on a unified codebase. This shared ownership breaks down silos and ensures that pipeline configurations are robust, secure, and fit for purpose across the organization.

Implementation Model: GitHub and Jenkins

This section details a practical framework for implementing a GitOps workflow for Logstash.

1. Prerequisites

• An established GitHub organisation or account.

• A running Jenkins instance with the necessary plugins installed (e.g., Git, GitHub Integration).

• A target Logstash environment where configurations will be deployed.

• Working knowledge of Git, Jenkins pipelines, and Logstash configuration syntax.

2. Step 1: Establish a Centralised Git Repository

The foundation of a GitOps workflow is a version-controlled repository.

1. Create a Repository: In GitHub, create a new repository (e.g., logstash-configurations). This will serve as the single source of truth for all pipeline configurations.

2. Define a Directory Structure: A logical directory structure is crucial for managing configurations across different environments. A recommended structure is:

    /
    ├── pipelines/
    │   ├── development/
    │   │   ├── 01-input-beats.conf
    │   │   ├── 10-filter-nginx.conf
    │   │   └── 99-output-elasticsearch.conf
    │   ├── staging/
    │   │   └── ...
    │   └── production/
    │       └── ...
    └── Jenkinsfile

This structure clearly separates configurations by environment and allows for a modular and maintainable pipeline design.

3. Step 2: Automate Deployment with a Jenkins CI/CD Pipeline

The Jenkins pipeline automates validation and deployment of the configurations from Git to your Logstash instances.

1. Create a Jenkinsfile: Add a Jenkinsfile to the root of your repository to define the automation pipeline. This pipeline-as-code approach ensures the deployment process itself is version-controlled.

2. Define the Pipeline Stages: The pipeline should include distinct stages for checking out code, validating configurations, and deploying to the target environment.

   A sample Jenkinsfile could look as follows:

   
    pipeline {
        agent any
   
        // Trigger the pipeline on every push to the main branch
        triggers {
            githubPush()
        }
   
        stages {
            stage('Checkout') {
                steps {
                    // Clone the repository
                    git 'https://github.com/your-org/logstash-configurations.git'
                }
            }
   
            stage('Validate Staging Configs') {
                steps {
                    // Run Logstash's built-in config test
                    // This prevents syntax errors from reaching production
                    sh 'docker run --rm -v ${WORKSPACE}/pipelines/staging:/usr/share/logstash/pipeline/ docker.elastic.co/logstash/logstash:9.3.2 logstash --config.test_and_exit'
                    sh 'docker run --rm -v ${WORKSPACE}/pipelines/staging:/usr/share/logstash/pipeline/ docker.elastic.co/logstash/logstash:9.3.2 logstash --config.test_and_exit'
                }
            }
   
            stage('Deploy to Staging') {
                // This stage requires Jenkins to have credentials to access the Staging server
                steps {
                    withCredentials([sshUserPrivateKey(credentialsId: 'staging-server-creds', keyFileVariable: 'KEY_FILE')]) {
                        sh '''
                            scp -i ${KEY_FILE} ${WORKSPACE}/pipelines/staging/*.conf user@staging-logstash-host:/etc/logstash/conf.d/
                            ssh -i ${KEY_FILE} user@staging-logstash-host 'sudo systemctl reload logstash'
                        '''
                    }
                }
            }
   
            // Optional: Add a manual approval step before deploying to production
            stage('Approval for Production') {
                steps {
                    input 'Deploy to Production?'
                }
            }
   
            stage('Deploy to Production') {
                 steps {
                    // Similar deployment steps for the production environment
                    // using production credentials
                 }
            }
        }
    }
   

4. The GitOps Workflow in Practice

This setup enables a controlled, auditable, and automated workflow:

1. Branch Creation: An engineer creates a feature branch in Git to propose a change (e.g., feature/add-syslog-input).

2. Configuration Change: The engineer modifies or adds a pipeline configuration file in their branch.

3. Pull Request: A pull request is created in GitHub. This action can trigger automated checks in Jenkins to validate the syntax of the proposed changes.

4. Peer Review: Team members review the changes for logic, security, and adherence to standards.

5. Merge and Deploy: Upon approval, the PR is merged into the main branch. This merge automatically triggers the Jenkins pipeline, which deploys the validated configuration to the corresponding Logstash environment.

Best Practices for Enterprise Adoption

To successfully implement this model at an enterprise scale, consider the following best practices:

• Branching Strategy: Adopt a consistent branching strategy, such as GitFlow, to manage features, releases, and hotfixes in an orderly manner. Protect your main or production branches with rules that require PR reviews and passing status checks before merging.

• Scalability: For large-scale deployments with many Logstash nodes, use configuration management tools like Ansible, Puppet, or Chef within your Jenkins pipeline to orchestrate the deployment across your entire fleet.

• Fostering a GitOps Culture: Successful adoption is as much about people and processes as it is about tools. Provide training and documentation to ensure all stakeholders understand the workflow and their role within it. Emphasise the collaborative benefits and the shared responsibility for maintaining a stable and secure observability platform.

• Pipeline Observability (Optional): Monitoring the health and performance of your CI/CD pipelines is crucial and recommended for early detection of issues, visibility into bottlenecks, and auditability. Elastic Observability provides native support for monitoring Jenkins pipelines using the Elastic CI/CD Observability plugin.

• Secrets Management: Never hardcode sensitive information (passwords, API keys) in your configuration files. Use a secrets management tool like HashiCorp Vault or AWS Secrets Manager, and have Logstash retrieve these secrets at runtime.

A sample snippet to retrieve the secret and store it securely in a Logstash keystore could look as follows:

```
...

	stage('Update Logstash Secret') {
    // Define the secret path and key in Vault
    def secretPath = 'secret/logstash/production'
    def secretKey = 'elasticsearch_password'
    def keystoreKey = 'ES_PWD' // The key name to be used in the Logstash keystore

    // Wrap the steps in withVault to get access to the secrets
    withVault(configuration: [url: 'http://your-vault-server:8200',
                             credentialsId: 'vault-approle-creds']) {
        
        // Retrieve the secret from Vault. The plugin makes it available as an environment variable.
        def secrets = readVault(path: secretPath, key: secretKey)
        def esPassword = secrets[secretKey]

        // Use SSH credentials to access the Logstash server
        withCredentials() {
            sh """
            ssh -i ${KEY_FILE} user@logstash-host <<'ENDSSH'
            # Pipe the secret directly into the logstash-keystore command
            # This avoids writing the secret to disk or exposing it in the process list
            echo "${esPassword}" | sudo -u logstash /usr/share/logstash/bin/logstash-keystore add ${keystoreKey} --stdin

            # After updating the keystore, reload Logstash to apply the change
            sudo systemctl reload logstash
            ENDSSH
            """
        }
    }
}
...

```

Conclusion

Adopting a GitOps approach for managing Logstash pipelines is a strategic move that aligns observability with modern DevSecOps principles. It replaces manual, opaque processes with an automated, transparent, and collaborative framework. For enterprise organisations, this leads to a more secure, resilient, and efficient observability infrastructure, empowering teams to derive maximum value from their data while minimising operational overhead and risk.

The above example is just a start; there’s a lot more you can do once you lay the foundation—GitOps is just the beginning. From branching automation to pipeline promotion workflows to building self-service deployment portals, the possibilities are limited only by your creativity (and maybe your CI minutes).

GitOps lays the foundation. To see the whole picture, you need to monitor your pipelines. Start a trial and try out Elastic’s CI/CD Observability solution to track build health and deployment trends. It connects code changes to production behavior, giving you deep visibility into your new automated workflow.

Build smarter pipelines. Monitor what matters. And let your GitOps-powered observability stack become the quiet hero of your DevSecOps story. See how Streams can supercharge your data engineering with the next generation of AI-powered log management & log processing.