From recommendation to remediation in 4 stages: human-in-the-loop automation with Elastic Workflows

An approve button with no evidence behind it is not a control. Human-in-the-loop automation binds five things into one inspectable record: the evidence you observed, the action you propose, the person who decided, the deadline they had, and what actually executed.

In this article, you'll build that gate with Elastic Workflows, reusing the control plane from the companion article Build an SRE Control Plane with Agent Builder and Workflows. This time the incident is narrow enough to act on: a stale pricing cache on a single worker, with a structured approval step sitting between the recommendation and the remediation. Nothing here touches production, so you can run the approved branch and the declined branch and compare what each one leaves in the execution record. That record is how incident response automation earns autonomy one incident class at a time.

Prerequisites

• Elastic Stack 9.4+

What an SRE control plane needs before you add approval

This article builds on Build an SRE Control Plane with Agent Builder and Workflows, which defines the control-plane pattern used here: telemetry, investigation context, policy, and a set of known actions, wired together. Agent Builder reasons over logs, traces, metrics, alerts, runbooks, and previous cases. Elastic Workflows executes a defined sequence with explicit inputs and permissions. Human-in-the-loop, or HITL, connects those two responsibilities at exactly the point where evidence becomes action.

A recommendation and a remediation have very different failure modes. A weak recommendation wastes an engineer's time. A weak remediation changes production. This is why the approval gate belongs inside the execution model.

The five bindings that make an approval gate a reliability control

A useful gate preserves five properties. Remove any one of them and the gate gets weaker.

• Evidence binding: The reviewer sees the logs, alert details, enrichment, or agent rationale that produced the proposal. An "Approve" button without evidence only asks a person to accept the automation's confidence.
• Action binding: The request names the exact bounded action, target, parameters, and expected effect. A request without an explicit target can authorize far more than the reviewer intended.
• Identity binding: The record shows who approved or declined the request, and when. Without it, you have an outcome but no accountability.
• Time binding: The decision has a deadline, and stale requests fail closed instead of running later without context. An approval without a timeout can outlive the incident evidence that justified it.
• Outcome binding: The execution record shows what ran, what was skipped, and whether post-action verification passed.

The form still matters, but the form is only the human-facing part of a much larger control. Approval design is reliability engineering.

The four stages from AI recommendation to auto remediation

Treat autonomy as a sequence of operational states rather than a product toggle.

The important transition is from stage 1 to stage 2. That is where the system stops being an advisor and gains an execution path. That path should be deterministic even when an AI agent contributed to the investigation: the agent summarizes evidence and recommends an action, while the workflow owns the pause, the structured decision, the branch, and the execution record.

Build the human-in-the-loop automation gate in Elastic Workflows

Elastic Workflows gives you the waitForInput step for this. When execution reaches that step, the workflow stops in the WAITING_FOR_INPUT state and waits for a person. The reviewer answers a small form in the Kibana execution view (or through the resume API), and whatever they submit becomes available to later steps at steps.<step_name>.output.

The workflow pulls the last 30 minutes of checkout-api failure logs, hands them to Agent Builder for a root-cause analysis, and writes that analysis into an Observability case. Then it stops and asks one question: should we clear the pricing cache on checkout-worker-07? If you approve, it records the simulated action, runs a verification query, and appends the result to the case. If you decline, it records the decision and runs nothing. Either way, production is untouched.

name: obs-labs-checkout-control-plane-hitl
description: Human approval gate for the OpenTelemetry-grounded checkout control plane.
enabled: false
tags:
  - sre-control-plane
  - agent-builder
  - human-in-the-loop
  - workflows
  - opentelemetry

settings:
  timeout: "30m"

triggers:
  - type: manual

consts:
  incident_id: "obs-labs-checkout-hitl-20260719"

steps:
  - name: collect_evidence
    type: elasticsearch.search
    with:
      index: "logs-*"
      size: 10
      query:
        bool:
          filter:
            - range:
                "@timestamp":
                  gte: "now-30m"
            - term:
                "service.name": "checkout-api"
            - term:
                "attributes.incident.id": "{{ consts.incident_id }}"
            - match_phrase:
                "body.text":
                  query: "checkout failed: stale pricing cache"

  - name: rca_analysis
    type: ai.agent
    agent-id: elastic-ai-agent
    create-conversation: true
    with:
      message: |
        Investigate the checkout-api incident identified by {{ consts.incident_id }}.
        The workflow evidence query found {{ steps.collect_evidence.output.hits.total.value }} matching OpenTelemetry log events in the last 30 minutes.
        Search logs, traces, and metrics for service.name checkout-api and incident.id {{ consts.incident_id }}.
        Identify the affected worker, deployment version, error type, HTTP status, and latency evidence.
        Return the likely cause, supporting evidence, confidence, and whether the bounded simulated action below is consistent with the evidence.
        Proposed simulated action: simulate clearing the pricing cache on checkout-worker-07.
        This workflow must not change production.

  - name: case_title
    type: ai.agent
    agent-id: elastic-ai-agent
    with:
      conversation_id: "{{ steps.rca_analysis.output.conversation_id }}"
      message: "Produce a short case title for this checkout incident. Output only the title."

  - name: case_description
    type: ai.agent
    agent-id: elastic-ai-agent
    with:
      conversation_id: "{{ steps.rca_analysis.output.conversation_id }}"
      message: "Produce a concise case description grounded in the OpenTelemetry evidence. Include the incident ID and say that remediation requires human approval. Output only the description."

  - name: create_case
    type: cases.createCase
    with:
      title: "{{ steps.case_title.output.message }}"
      description: "{{ steps.case_description.output.message }}"
      owner: "observability"
      severity: "medium"
      tags:
        - sre-control-plane
        - agent-builder
        - human-in-the-loop
        - opentelemetry

  - name: add_agent_analysis
    type: cases.addComment
    with:
      case_id: "{{ steps.create_case.output.case.id }}"
      comment: |
        ## Agent Builder RCA and proposed action

        Evidence query matches: {{ steps.collect_evidence.output.hits.total.value }}

        {{ steps.rca_analysis.output.message }}

        Proposed simulated action: simulate clearing the pricing cache on checkout-worker-07.
        No action has run yet.

        Agent conversation: {{ kibanaUrl }}/app/agent_builder/conversations/{{ steps.rca_analysis.output.conversation_id }}

  - name: review
    type: waitForInput
    with:
      message: |
        Approve the bounded checkout response?

        Incident: {{ consts.incident_id }}
        Evidence: {{ steps.collect_evidence.output.hits.total.value }} matching checkout failure logs in the last 30 minutes.
        Target: checkout-worker-07
        Action: simulate clearing the pricing cache on this worker only.
        Expected effect: subsequent checkout requests no longer use the stale cache.
        Blast radius: one worker. This simulated step does not change production.

        Review the Agent Builder analysis in case {{ steps.create_case.output.case.id }} before deciding.
      schema:
        type: object
        properties:
          approved:
            type: boolean
            title: "Approve the simulated cache clear"
          notes:
            type: string
            title: "Reviewer notes"
        required:
          - approved

  - name: approved_action
    type: console
    if: "steps.review.output.approved : true"
    with:
      message: "Approved. Simulated cache clear recorded for checkout-worker-07. Reviewer notes: {{ steps.review.output.notes }}"

  - name: verify_after_approval
    type: elasticsearch.search
    if: "steps.review.output.approved : true"
    with:
      index: "logs-*"
      size: 0
      query:
        bool:
          filter:
            - range:
                "@timestamp":
                  gte: "now-5m"
            - term:
                "service.name": "checkout-api"
            - term:
                "attributes.incident.id": "{{ consts.incident_id }}"
            - match_phrase:
                "body.text":
                  query: "checkout failed: stale pricing cache"

  - name: record_approved
    type: cases.addComment
    if: "steps.review.output.approved : true"
    with:
      case_id: "{{ steps.create_case.output.case.id }}"
      comment: |
        ## Human decision: approved

        Reviewer notes: {{ steps.review.output.notes }}
        Simulated action target: checkout-worker-07
        Verification query matches in the last 5 minutes: {{ steps.verify_after_approval.output.hits.total.value }}

        This walkthrough did not change production.

  - name: record_declined
    type: cases.addComment
    if: "steps.review.output.approved : false"
    with:
      case_id: "{{ steps.create_case.output.case.id }}"
      comment: |
        ## Human decision: declined

        Reviewer notes: {{ steps.review.output.notes }}
        No action ran.

  - name: declined_console
    type: console
    if: "steps.review.output.approved : false"
    with:
      message: "Declined. No action executed. Reviewer notes: {{ steps.review.output.notes }}"

The collect_evidence and rca_analysis steps keep the same evidence-then-reasoning sequence as the first article, and the workflow writes that analysis into the case before it ever reaches the waitForInput boundary. By the time a human is asked to decide, the reasoning is already durable and linkable.

The approval form asks for one boolean and accepts optional notes.

Only the approved branch records the simulated cache clear, runs a bounded verification query, and appends the result to the case. The declined branch records the decision and runs nothing.

The approval request carries the resulting count into the decision, while the linked case retains the full Agent Builder analysis. Treat that count as evidence for the reviewer, not as a root-cause claim or a performance measurement.

Replace the simulated action with real auto remediation

The walkthrough keeps the approved branch as a console step so you can run both branches safely. In production you replace only that step with a narrowly scoped external action, and leave the case, timeout, verification, and decline path exactly as they are.

You have two options, depending on how the target system is reached.

Call an internal remediation API with an HTTP connector

Configure an HTTP connector in Kibana with the base URL, authentication, and any encrypted headers, then reference it by connector-id. Secrets stay in the connector, never in the workflow YAML.

  - name: clear_pricing_cache
    type: http
    connector-id: "checkout-remediation-api"
    if: "steps.review.output.approved : true"
    with:
      path: "/v1/cache/pricing/purge"
      method: "POST"
      body:
        worker: "checkout-worker-07"
        incident_id: "{{ consts.incident_id }}"

Hand off to Jira, Slack or PagerDuty

Kibana connectors are available as workflow steps, so the approved branch can open a jira ticket for a change-managed action, post to slack for the on-call channel, or page through PagerDuty, all using credentials your team already manages centrally.

  - name: notify_oncall
    type: slack
    connector-id: "sre-oncall-channel"
    if: "steps.review.output.approved : true"
    with:
      message: "Approved by {{ steps.review.output.notes }}: pricing cache cleared on checkout-worker-07 for {{ consts.incident_id }}."

Whichever you choose, keep the action bound.

How to design a human-in-the-loop automation gate

The workflow above shows the mechanics. Getting the gate right is a design problem: where the pause goes, what the request tells the reviewer, what happens when nobody answers, and what the execution record has to preserve. Each one maps back to one of the five bindings.

Where should the approval step go in an incident response workflow?

The best place for waitForInput is immediately before the first step that increases impact. Don't pause before gathering evidence, because the workflow can usually search, enrich, classify, and open a draft case without touching the affected service. And don't pause after the remediation, because that just asks a person to ratify something that already happened.

Before anyone enables the workflow, a reviewer can inspect the evidence query, the form schema, the timeout, the branch condition, and the action itself.

What a good approval request tells the reviewer

An on-call engineer should not have to reconstruct the investigation from five other screens. Lead with the decision and include only the evidence needed to make it. A strong approval request answers these questions, in this order:

1. What exactly am I deciding?
2. What telemetry supports the proposal?
3. What target and parameters will the action use?
4. What is the expected effect and blast radius?
5. What happens if I decline or do nothing?

One required decision plus optional notes is usually enough. If the reviewer has to type in service names, host identifiers, environment, and action parameters, the proposal wasn't specific enough before the pause.

Paused executions stay discoverable in history and can be resumed by any authorized reviewer, which brings up a queueing requirement. Once a team has more than a few paused executions, reviewers need an inbox or an equivalent filtered view showing pending decisions, age, owner, severity, and target. Otherwise, a safe pause quietly becomes an invisible backlog.

What happens if nobody approves in time

waitForInput has no default timeout; the execution waits indefinitely. The settings.timeout field caps the entire execution, including time spent waiting for input, and in this workflow the 30-minute value limits how long the proposal stays actionable after the evidence was collected.

Choose that value from the incident and the action; a traffic shift during an active outage may need a decision within minutes. A maintenance approval may stay valid for hours. Confirm the timeout behavior on the exact Elastic version you operate, and add an external escalation or cancellation path if what you observe doesn't meet your fail-closed requirement.

Whatever you do, don't convert silence into approval.

What the audit record must capture

The execution history should answer four questions without anyone digging through chat history:

• What evidence did the workflow collect?
• What exact input did the reviewer submit?
• Which branch ran?
• What did the action and verification steps return?

A paused execution shows the evidence step and the decision still pending. An approved execution adds the action branch and the verification output to that same view. A declined execution preserves the same evidence and the same decision while skipping every action step. Keeping the walkthrough action simulated lets you inspect both branches without changing production.

For longer-lived incident context, push it into the case. Add the evidence summary, reviewer notes, action result, and verification result as comments, and the case becomes the durable record that outlives the execution.

How to know when a workflow is ready to run without approval

Don't remove the approval gate because a handful of runs succeeded. Review enough executions to understand both the normal and the exceptional paths, and measure at least these outcomes:

Autonomous execution is reasonable only when the incident class, target selection, action, verification, permissions, timeout, and rollback path are all narrow and repeatable. Even then, keep the same workflow structure. Automation should bypass the human wait for the proven path, not bypass evidence collection, authorization, verification, or audit records. Route anything uncertain back to human review.

Conclusion

The approval gate is a small amount of YAML, but it changes what the automation is. waitForInput turns the decision into a structured input the branch logic depends on, so the action, the verification, and the case comment all exist because a named person answered a specific question at a specific time. Placing that pause right before the first step that increases impact, and capping it with settings.timeout, is what makes it a reliability control instead of a confirmation dialog.

Taking this to production is a smaller change than it looks: the simulated console step becomes an http, slack, or jira connector step, and everything else stays as it is. From there you earn autonomy one incident class at a time, letting acceptance rate, approval age, action success, verification success, and rollback rate tell you when a path is proven enough to run without the wait.

Where to start with human-in-the-loop automation

Start with one recurring operational signal and one reversible response. Build the evidence query first, then add a recommendation that names the target and the expected effect, then insert waitForInput immediately before the action and run the workflow in a lab or case-only mode. Review the execution history with SREs, developers, support, security, and the product owners of the affected service.

The approval gate is not the destination. It is the mechanism that lets a team move toward narrow autonomy without giving up evidence, accountability, or control.

• Human-in-the-loop workflow guide

• waitForInput flow-control reference

• External systems and apps steps

• Kibana connectors reference

• Workflow monitoring guide