When running Elastic Agent in some environments, you might not know all the input configuration details up front. To solve this problem, the input configuration accepts variables and conditions that get evaluated at runtime using information from the running environment. Similar to autodiscovery, these capabilities allow you to apply configurations dynamically.

Let’s consider a unique agent policy that is deployed on two machines: a Linux machine named "linux-app" and a Windows machine named "winapp". Notice that the configuration has some variable references: ${host.name} and ${host.platform}:

inputs:
 - type: logfile
   streams:
    - paths: /var/log/${host.name}/another.log
      condition: ${host.platform} == "linux"
    - path: c:/service/app.log
      condition: ${host.platform} == "windows"

At runtime, Elastic Agent resolves variables and evaluates the conditions based on values provided by the environment, generating two possible input configurations.

On the Windows machine:

inputs:
 - type: logfile
   streams:
    - path: c:/service/app.log

On the Linux machine:

inputs:
 - type: logfile
   streams:
    - paths: /var/log/linux-app/another.log

Using variable substitution along with conditions allows you to create concise, but flexible input configurations that adapt to their deployed environment.

Variable substitutionedit

The syntax for variable substitution is ${var}, where var is the name of a variable defined by a provider. A provider defines key/value pairs that are used for variable substitution and conditions.

Elastic Agent supports a variety of providers, such as host and local, that supply variables to Elastic Agent. For example, earlier you saw ${host.name} used to resolve the path to the host’s log file based on the {host.platform} value. Both of these values were provided by the host provider.

All providers are enabled by default when Elastic Agent starts. If a provider cannot be configured, its variables are ignored.

See Providers for more detail.

The following agent policy uses a custom key named foo to resolve a value defined by a local provider:

inputs:
 - type: logfile
   streams:
    - paths: /var/log/${foo}/another.log

providers:
  local:
    vars:
      foo: bar

The policy generated by this configuration looks like this:

inputs:
 - type: logfile
   streams:
    - paths: /var/log/bar/another.log

When an input uses a variable substitution that is not present in the current key/value mappings being evaluated, the input is removed in the result.

For example, this agent policy uses an unknown key:

inputs:
  - type: logfile
    path: "/var/log/foo"
  - type: logfile
    path: "${ unknown.key }"

The policy generated by this configuration looks like this:

inputs:
  - type: logfile
    path: "/var/log/foo"

Alternative variables and constantsedit

Variable substitution can also define alternative variables or a constant.

To define a constant, use either ' or ". When a constant is reached during variable evaluation, any remaining variables are ignored, so a constant should be the last entry in the substitution.

To define alternatives, use | followed by the next variable or constant. The power comes from allowing the input to define the preference order of the substitution by chaining multiple variables together.

For example, the following agent policy chains together multiple variables to set the log path based on information provided by the running container environment. The constant /var/log/other is used to end of the path, which is common to both providers:

inputs:
  - type: logfile
    path: "/var/log/foo"
  - type: logfile
    path: "${docker.paths.log|kubernetes.container.paths.log|'/var/log/other'}"

Providersedit

Providers supply the key-value pairs that are used for variable substitution and conditionals. Each provider’s keys are automatically prefixed with the name of the provider in the context of the Elastic Agent.

For example, a provider named foo provides {"key1": "value1", "key2": "value2"}, the key-value pairs are placed in {"foo" : {"key1": "value1", "key2": "value2"}}. To reference the keys, use {{foo.key1}} and {{foo.key2}}.

Provider configurationedit

The provider configuration is specified under the top-level providers key in the elastic-agent.yml configuration. All registered providers are enabled by default. If a provider cannot connect, no mappings are produced.

The following example shows two providers (local and local_dynamic) that supply custom keys:

providers:
  local:
    vars:
      foo: bar
  local_dynamic:
    vars:
      - item: key1
      - item: key2

Explicitly disable a provider by setting enabled: false. All providers are prefixed without name collisions. The name of the provider is in the key in the configuration.

providers:
  docker:
    enabled: false

Elastic Agent supports two broad types of providers: context and dynamic.

Context providersedit

Context providers give the current context of the running Elastic Agent, for example, agent information (ID, version), host information (hostname, IP addresses), and environment information (environment variables).

They can only provide a single key-value mapping. Think of them as singletons; an update of a key-value mapping results in a re-evaluation of the entire configuration. These providers are normally very static, but not required. A value can change which results in re-evaluation.

Context providers use the Elastic Common Schema (ECS) naming to ensure consistency and understanding throughout documentation and projects.

Elastic Agent supports the following context providers:

Localedit

Provides custom keys to use as variables. For example:

providers:
  local:
    vars:
      foo: bar

Agent provideredit

Provides information about the Elastic Agent. The available keys are:

Host provideredit

Provides information about the current host. The available keys are:

Env Provideredit

Provides access to the environment variables as key-value pairs.

For example, set the variable foo:

foo=bar elastic-agent run

The environment variable can be referenced as ${env.foo}.

Kubernetes Secrets Provideredit

Provides access to the Kubernetes Secrets API.

The provider needs a kubeconfig file to establish connection to the Kubernetes API. It can automatically reach the API if it’s run in an inCluster environment (Elastic Agent runs as pod).

providers.kubernetes_secrets:
  #kube_config: /Users/elastic-agent/.kube/config

Reference the Kubernetes Secrets variable as ${kubernetes_secrets.default.somesecret.value}, where default is the namespace of the Secret, somesecret is the name of the Secret and value the field of the Secret to access.

If you run agent on Kubernetes, the proper rule in the ClusterRole is required to privide access to the Elastic Agent pod in the Secrets API:

- apiGroups: [""]
  resources:
    - secrets
  verbs: ["get"]

Kubernetes LeaderElection Provideredit

Provides the option to enable leaderelection between a set of Elastic Agents running on Kubernetes. Only one Elastic Agent at a time will be the holder of the leader lock and based on this, configurations can be enabled with the condition that the Elastic Agent holds the leadership. This is useful in cases where the Elastic Agent between a set of Elastic Agents collects cluster wide metrics for the Kubernetes cluster, such as the kube-state-metrics endpoint.

Provider needs a kubeconfig file to establish a connection to Kubernetes API. It can automatically reach the API if it’s running in an inCluster environment (Elastic Agent runs as Pod).

providers.kubernetes_leaderelection:
  #kube_config: /Users/elastic-agent/.kube/config
  #leader_lease: agent-k8s-leader-lock

The available key is:

Use conditions based on the kubernetes_leaderelection.leader key to leverage the leaderelection provider and enable specific inputs only when the Elastic Agent holds the leadership lock. The below example enables the state_container metricset only when the leadership lock is acquired:

- data_stream:
    dataset: kubernetes.state_container
    type: metrics
  metricsets:
    - state_container
  add_metadata: true
  hosts:
    - 'kube-state-metrics:8080'
  period: 10s
  condition: ${kubernetes_leaderelection.leader} == true

Dynamic Providersedit

Dynamic providers give an array of multiple key-value mappings. Each key-value mapping is combined with the previous context provider’s key and value mapping which provides a new unique mapping that is used to generate a configuration.

Local dynamic provideredit

Define multiple key-value pairs to generate multiple configurations.

For example, the following Elastic Agent policy defines a local dynamic provider that defines three values for item:

inputs:
 - type: logfile
   streams:
     - paths: "/var/${local_dynamic.my_var}/app.log"

providers:
  local_dynamic:
    items:
      - vars:
          my_var: key1
      - vars:
          my_var: key2
      - vars:
          my_var: key3

The configuration generated by this policy looks like:

inputs:
 - type: logfile
   streams:
     - paths: "/var/key1/app.log"
 - type: logfile
   streams:
     - paths: "/var/key2/app.log"
 - type: logfile
   streams:
   - paths: "/var/key3/app.log"

Docker Provideredit

Provides inventory information from Docker. The available keys are:

For example, the Docker provider provides the following inventory:

[
    {
       "id": "1",
       "mapping:": {"id": "1", "paths": {"log": "/var/log/containers/1.log"}},
       "processors": {"add_fields": {"container.name": "my-container"}}
    },
    {
        "id": "2",
        "mapping": {"id": "2", "paths": {"log": "/var/log/containers/2.log"}},
        "processors": {"add_fields": {"container.name": "other-container"}}
    }
]

Elastic Agent automatically prefixes the result with docker:

---
[
    {"docker": {"id": "1", "paths": {"log": "/var/log/containers/1.log"}}},
    {"docker": {"id": "2", "paths": {"log": "/var/log/containers/2.log"}},
]
---

To set the log path dynamically in the configuration, use a variable in the Elastic Agent policy to return path information from the provider:

inputs:
  - type: logfile
    path: "${docker.paths.log}"

The policy generated by this configuration looks like:

inputs:
  - type: logfile
    path: "/var/log/containers/1.log"
    processors:
      - add_fields:
          container.name: my-container
  - type: logfile
    path: "/var/log/containers/2.log"
    processors:
      - add_fields:
          container.name: other-container

Kubernetes Provideredit

Provides inventory information from Kubernetes. The available keys are:

Fox example, if the Kubernetes provider provides the following inventory:

[
    {
       "id": "1",
       "mapping:": {"namespace": "kube-system", "pod": {"name": "kube-controllermanger"}},
       "processors": {"add_fields": {"container.name": "my-container"}}
    },
    {
        "id": "2",
        "mapping:": {"namespace": "kube-system", "pod": {"name": "kube-scheduler"}},
        "processors": {"add_fields": {"kuberentes.namespace": "kube-system", "kubernetes.pod": {"name": "kube-scheduler"}}
    }
]

Elastic Agent automatically prefixes the result with kuberentes:

---
[
    {"kubernetes": {"id": "1", "namespace": "kube-system", "pod": {"name": "kube-controllermanger"}},
    {"kubernetes": {"id": "2", "namespace": "kube-system", "pod": {"name": "kube-scheduler"}},
]
---

providers.kubernetes:
  node: ${NODE_NAME}
  scope: node
  #kube_config: /Users/elastic-agent/.kube/config
  #sync_period: 600
  #cleanup_timeout: 60

To set the target host dynamically only for a targeted Pod based on its labels, use a variable in the Elastic Agent policy to return path information from the provider:

- data_stream:
      dataset: kubernetes.scheduler
      type: metrics
  metricsets:
    - scheduler
  hosts:
    - '${kubernetes.pod.ip}:10251'
  period: 10s
  condition: ${kubernetes.pod.labels.component} == 'kube-scheduler'

The policy generated by this configuration looks like:

- hosts:
  - 172.18.0.4:10251
  metricsets:
  - scheduler
  module: kubernetes
  period: 10s
  processors:
  - add_fields:
    fields:
      namespace: kube-system
      pod:
        ip: 172.18.0.4
        labels:
          component: kube-scheduler
          tier: control-plane
        name: kube-scheduler-kind-control-plane
        uid: 6da04645-04b4-4cb2-b203-2ad58abc6cdf
    target: kubernetes

To set the log path of Pods dynamically in the configuration, use a variable in the Elastic Agent policy to return path information from the provider:

streams:
  - data_stream:
      dataset: generic
    symlinks: true
    paths:
      - /var/log/containers/*${kubernetes.container.id}.log

The policy generated by this configuration looks like:

- paths:
  - /var/log/containers/*c957652eca53594ce496c7b237d19f05be339ebfe281b99ce1c0a0401e48ce3a.log
  processors:
  - add_fields:
    fields:
      container:
        id: c957652eca53594ce496c7b237d19f05be339ebfe281b99ce1c0a0401e48ce3a
        image: k8s.gcr.io/kube-apiserver:v1.18.2
        name: kube-apiserver
        runtime: containerd
      namespace: kube-system
      pod:
        ip: 172.18.0.4
        labels:
          component: kube-apiserver
          tier: control-plane
        name: kube-apiserver-kind-control-plane
        uid: f8743f90-50a4-4ef8-9fe9-78c245eb8bf3
    target: kubernetes
  symlinks: true

Conditionsedit

A condition is a boolean expression that you can specify in your agent policy to control whether a configuration is applied to the running Elastic Agent. You can set a condition on inputs, streams, or even processors.

In this example, the input is applied if the host platform is Linux:

inputs:
  - type: logfile
    streams:
      - paths:
         - /var/log/syslog
    condition: ${host.platform} == 'linux'

In this example, the stream is applied if the host platform is not Windows:

inputs:
  - type: system/metrics
    streams:
      - metricset: load
        data_stream.dataset: system.cpu
        condition: ${host.platform} != 'windows'

In this example, the processor is applied if the host platform is not Windows:

inputs:
  - type: system/metrics
    streams:
      - metricset: load
        data_stream.dataset: system.cpu
    processors:
      - add_fields:
          fields:
            platform: ${host.platform}
          to: host
        condition: ${host.platform} != 'windows'

Condition syntaxedit

The conditions supported by Elastic Agent are based on EQL's boolean syntax, but add support for variables from providers and functions to manipulate the values.

Supported operators:

Functions:

Types:

Condition examplesedit

Run only when a specific label is included.

arrayContains(${docker.labels}, 'monitor')

Skip on Linux platform or macOS.

${host.platform} != "linux" and ${host.platform} != "darwin"

Run only for specific labels.

arrayContains(${docker.labels}, 'monitor') or arrayContains(${docker.label}, 'production')

Function referenceedit

The condition syntax supports the following functions.

addedit

add(Number, Number) Number

Usage:

add(1, 2) == 3
add(5, ${foo}) >= 5

arrayContainsedit

arrayContains(Array, String) Boolean

Usage:

arrayContains(${docker.labels}, 'monitor')

concatedit

concat(String, String) String

Usage:

concat("foo", "bar") == "foobar"
concat(${var1}, ${var2}) != "foobar"

divideedit

divide(Number, Number) Number

Usage:

divide(25, 5) > 0
divide(${var1}, ${var2}) > 7

endsWithedit

endsWith(String, String) Boolean

Usage:

endsWith("hello world", "hello") == true
endsWith(${var1}, "hello") != true

hasKeyedit

hasKey(Dictionary, String) Boolean

Usage:

hasKey(${host}, "platform")

indexOfedit

indexOf(String, String, Number?) Number

Usage:

indexOf("hello", "llo") == 2
indexOf(${var1}, "hello") >= 0

lengthedit

length(Array|Dictionary|string)

Usage:

length("foobar") > 2
length(${docker.labels}) > 0
length(${host}) > 2

matchedit

match(String, Regexp) boolean

Usage:

match("hello world", "^hello") == true
match(${var1}, "world$") == true

moduloedit

modulo(number, number) Number

Usage:

modulo(25, 5) > 0
modulo(${var1}, ${var2}) == 0

multiplyedit

multiply(Number, Number) Number

Usage:

multiply(5, 5) == 25
multiple(${var1}, ${var2}) > x

numberedit

number(String) Integer

Usage:

number("42") == 42
number(${var1}) == 42

startsWithedit

startsWith(String, String) Boolean

Usage:

startsWith("hello world", "hello") == true
startsWith(${var1}, "hello") != true

stringedit

string(Number) String

Usage:

string(42) == "42"
string(${var1}) == "42"

stringContainsedit

stringContains(String, String) Boolean

Usage:

stringContains("hello world", "hello") == true
stringContains(${var1}, "hello") != true

subtractedit

subtract(Number, Number) Number

Usage:

subtract(5, 1) == 4
subtract(${foo}, 2) != 2

Debuggingedit

To debug configurations that include variable substitution and conditions, use the inspect command. This command shows the configuration that’s generated after variables are replaced and conditions are applied.

First run the Elastic Agent. For this example, we’ll use the following agent policy:

outputs:
  default:
    type: elasticsearch
    hosts: [127.0.0.1:9200]
    username: elastic
    password: changeme

providers:
  local_dynamic:
    items:
      - vars:
          key: value1
        processors:
          - add_fields:
              fields:
                custom: match1
              target: dynamic
      - vars:
          key: value2
        processors:
          - add_fields:
              fields:
                custom: match2
              target: dynamic
      - vars:
          key: value3
        processors:
          - add_fields:
              fields:
                custom: match3
              target: dynamic

inputs:
  - type: logfile
    enabled: true
    streams:
      - paths:
          - /var/log/{{local_dynamic.key}}

Then run elastic-agent inspect to see the generated configuration. For example:

$ ./elastic-agent inspect output -o default
[default] filebeat:
filebeat:
  inputs:
  - index: logs-generic-default
    paths:
    - /var/log/value1
    processors:
    - add_fields:
        fields:
          custom: match1
        target: dynamic
    - add_fields:
        fields:
          dataset: generic
          namespace: default
          type: logs
        target: data_stream
    - add_fields:
        fields:
          dataset: generic
        target: event
    type: log
  - index: logs-generic-default
    paths:
    - /var/log/value2
    processors:
    - add_fields:
        fields:
          custom: match2
        target: dynamic
    - add_fields:
        fields:
          dataset: generic
          namespace: default
          type: logs
        target: data_stream
    - add_fields:
        fields:
          dataset: generic
        target: event
    type: log
  - index: logs-generic-default
    paths:
    - /var/log/value3
    processors:
    - add_fields:
        fields:
          custom: match3
        target: dynamic
    - add_fields:
        fields:
          dataset: generic
          namespace: default
          type: logs
        target: data_stream
    - add_fields:
        fields:
          dataset: generic
        target: event
    type: log
output:
  elasticsearch:
    hosts:
    - 127.0.0.1:9200
    password: changeme
    username: elastic

---
[default] FLEET_MONITORING:
output:
  elasticsearch:
    hosts:
    - 127.0.0.1:9200
    password: changeme
    type: elasticsearch
    username: elastic
programs:
- filebeat

---

Capabilitiesedit

Capabilities are a set of rules that can define the behavior of Elastic Agent on specific machines. When capabilities are specified, the input configuration is overridden, and any details that are not used are subject to the capabilities' definition. When Elastic Agent starts, it locates the capabilities.yml file next to its configuration file. Please note, Elastic Agent will not reload the definition after it starts.

This example definition configuration enables the system/metrics input and denies every other input. Rules are applied in the order they are defined, with the first matching rule being applied. The * wildcard is also supported.

version: 0.0.1
capabilities:
- rule: allow
  input: system/metrics
- rule: deny
  input: "*"

Let’s use the following configuration and the capabilities defined above.

inputs:
	- type: logfile
	  streams:
	  	- paths: []{a.error,b.access}
	- type: system/logs
	  streams:
	  	- paths: []{a,b}
	- type: system/metrics
	  streams:
	  	- metricset: cpu
	  	- metricset: memory

The resulting configuration is this.

inputs:
	- type: system/metrics
	  streams:
	  	- metricset: cpu
	  	- metricset: memory

In this example, the stream is applied if the host platform is not Windows:

inputs:
  - type: system/metrics
    streams:
      - metricset: load
        data_stream.dataset: system.cpu
        condition: ${host.platform} != 'windows'

In this example, the processor is applied if the host platform is not Windows:

inputs:
  - type: system/metrics
    streams:
      - metricset: load
        data_stream.dataset: system.cpu
    processors:
      - add_fields:
          fields:
            platform: ${host.platform}
          to: host
        condition: ${host.platform} != 'windows'

Supported capabilitiesedit

You can use capabilities to modify the congestion of outputs, inputs or upgrade the capability of Elastic Agent.

Inputsedit

For inputs, you can the input key containing an expression.

version: 0.0.1
capabilities:
- rule: allow
  input: system/*

Outputsedit

For outputs, you can use the output key containing an expression.

version: 0.0.1
capabilities:
- rule: deny
  output: kafka

Upgradeedit

For an upgrade, you can use the output key containing an expression that allows a specific upgrade to version 8.0.0. ${version} is the target version.

version: 0.0.1
capabilities:
- rule: allow
  upgrade: "${version} == '8.0.0'"

With a versioned EQL, conditions can be used, as well as a * wildcard. This allows upgrades just for bug fixes.

version: 0.0.1
capabilities:
- rule: allow
  upgrade: "${version} == '8.0.*'"

Upgrade capability definition

This upgrade definition allows upgrades just for minor version changes.

version: 0.0.1
capabilities:
- rule: allow
  upgrade: "${version} == '8.*.*'"

Part of the upgrade action is also a source_uri address that specifies the repository from where artifacts should be downloaded.

To restrict to HTTPS, you can use the following.

version: 0.0.1
capabilities:
- rule: allow
  upgrade: "startsWith(${source_uri}, 'https')"

Upgrade capability is blocking. This means that if an action does not meet the first upgrade definition, we’re not proceeding with the evaluation.