Data action steps

Data action steps are explicit, testable data transformations. They do the things you could do inline with Liquid templating, but with their own named output, an execution log, and an attachment point for error handling.

Use a data step when the transformation has any complexity: multi-field mapping, grouping, JSON parsing, regex extraction, or deduplication. For small inline transforms (take a field, uppercase it, fall back to a default), use Liquid templating instead.

Most data steps place the source of the data to transform at the step top level, not inside with. with holds the transformation configuration. The exceptions are data.set, which has no separate source field, and where noted below.

Source arrays usually need the raw-value form ${{ ... }} so they pass as an array rather than as a string:

items: "${{ steps.search.output.hits.hits }}"
		

See Pass data and handle errors for the {{ }} vs. ${{ }} distinction.

Variables and mapping data.set · data.map

Filter, find, and aggregate data.filter · data.find · data.aggregate

Combine and de-duplicate data.concat · data.dedupe

JSON data.parseJson · data.stringifyJson

Regex data.regexExtract · data.regexReplace

Set named variables in the workflow context. Subsequent steps reference them as {{ variables.<name> }}. Unlike the other data steps, data.set has no top-level source field; values go directly inside with.

- name: init
  type: data.set
  with:
    service_name: "checkout"
    threshold: 70
		

Later steps can read {{ variables.service_name }} and {{ variables.threshold }}.

Transform each element of an array into a new shape. items is a top-level step field; the per-element mapping goes under with.fields. Inside the mapping, item refers to the current array element.

- name: normalize_alerts
  type: data.map
  items: "${{ steps.search.output.hits.hits }}"
  with:
    fields:
      id: "{{ item._id }}"
      severity: "{{ item._source.kibana.alert.severity }}"
      host: "{{ item._source.host.name }}"
		

Keep only the elements of an array that match a Kibana Query Language (KQL) predicate. Inside the condition, item.<field> refers to element fields and index refers to the zero-based position.

Output is always an array of matched items.

- name: only_critical
  type: data.filter
  items: "${{ steps.search.output.hits.hits }}"
  with:
    condition: "item._source.kibana.alert.severity : critical"
    limit: 10
		

Return the first element of an array that matches a KQL predicate. Output is { item, index }, or null when nothing matches (unless errorIfEmpty is set).

- name: first_critical
  type: data.find
  items: "${{ event.alerts }}"
  with:
    condition: "item.kibana.alert.severity : critical"
    errorIfEmpty: false
		

Group a collection by one or more keys and compute metrics per group. Supported operations: count, sum, avg, min, max.

Basic grouping:

- name: count_by_host
  type: data.aggregate
  items: "${{ event.alerts }}"
  with:
    group_by: ["host.name"]
    metrics:
      - name: "count"
        operation: "count"
    order: "desc"
    order_by: "count"
    limit: 10
		

With bucketed aggregation:

- name: age_distribution
  type: data.aggregate
  items: "${{ steps.fetch_users.output }}"
  with:
    group_by: ["department"]
    metrics:
      - name: "count"
        operation: "count"
    order: "desc"
    buckets:
      field: "age"
      ranges:
        - to: 30
          label: "junior"
        - from: 30
          to: 50
          label: "mid"
        - from: 50
          label: "senior"
		

Concatenate arrays. arrays is the top-level source field.

- name: combine_lists
  type: data.concat
  arrays:
    - "${{ steps.list_a.output }}"
    - "${{ steps.list_b.output }}"
  with:
    dedupe: true
		

Remove duplicates from an array. The uniqueness fields go in with.keys.

- name: unique_hosts
  type: data.dedupe
  items: "${{ event.alerts }}"
  strategy: "keep_first"
  with:
    keys: ["host.name"]
		

Parse a JSON string into an object.

- name: parse_payload
  type: data.parseJson
  source: "${{ steps.http_call.output.body }}"
		

Serialize an object to a JSON string.

- name: to_string
  type: data.stringifyJson
  source: "${{ steps.search.output }}"
  with:
    pretty: true
		

Extract matches from a string with a regular expression. fields maps output field names to capture group references ($1, $2, or named groups).

- name: pull_user_id
  type: data.regexExtract
  source: "${{ steps.log.output.message }}"
  errorIfNoMatch: false
  with:
    pattern: "user_id=(?<id>[a-f0-9]+)"
    fields:
      user_id: "id"
    flags: "i"
		

Replace regex matches in a string.

- name: redact_ssn
  type: data.regexReplace
  source: "${{ steps.log.output.message }}"
  with:
    pattern: "\\b\\d{3}-\\d{2}-\\d{4}\\b"
    replacement: "[REDACTED]"
    flags: "g"
		

• Templating engine: Liquid filters for small inline transformations.
• Pass data and handle errors: The {{ }} vs. ${{ }} distinction and error-handling strategies.
• Foreach step: Pair with data steps for per-item work after filtering or mapping.