Configure the Elasticsearch outputedit

When you specify Elasticsearch for the output, APM Server sends the transactions directly to Elasticsearch by using the Elasticsearch HTTP API.

Example configuration:

output.elasticsearch:
  hosts: ["http://localhost:9200"]
  index: "apm-server-%{[beat.version]}-%{+yyyy.MM.dd}"
  ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]
  ssl.certificate: "/etc/pki/client/cert.pem"
  ssl.key: "/etc/pki/client/cert.key"

To enable SSL, just add https to all URLs defined under hosts.

output.elasticsearch:
  hosts: ["https://localhost:9200"]
  username: "admin"
  password: "s3cr3t"

If the Elasticsearch nodes are defined by IP:PORT, then add protocol: https to the yaml file.

output.elasticsearch:
  hosts: ["localhost"]
  protocol: "https"
  username: "admin"
  password: "s3cr3t"

Compatibilityedit

This output works with all compatible versions of Elasticsearch. See "Supported Beats Versions" in the Elastic Support Matrix.

Configuration optionsedit

You can specify the following options in the elasticsearch section of the apm-server.yml config file:

enablededit

The enabled config is a boolean setting to enable or disable the output. If set to false, the output is disabled.

The default value is true.

hostsedit

The list of Elasticsearch nodes to connect to. The events are distributed to these nodes in round robin order. If one node becomes unreachable, the event is automatically sent to another node. Each Elasticsearch node can be defined as a URL or IP:PORT. For example: http://192.15.3.2, https://es.found.io:9230 or 192.24.3.2:9300. If no port is specified, 9200 is used.

Note

When a node is defined as an IP:PORT, the scheme and path are taken from the protocol and path config options.

output.elasticsearch:
  hosts: ["10.45.3.2:9220", "10.45.3.1:9230"]
  protocol: https
  path: /elasticsearch

In the previous example, the Elasticsearch nodes are available at https://10.45.3.2:9220/elasticsearch and https://10.45.3.1:9230/elasticsearch.

compression_leveledit

The gzip compression level. Setting this value to 0 disables compression. The compression level must be in the range of 1 (best speed) to 9 (best compression).

Increasing the compression level will reduce the network usage but will increase the cpu usage.

The default value is 0.

workeredit

The number of workers per configured host publishing events to Elasticsearch. This is best used with load balancing mode enabled. Example: If you have 2 hosts and 3 workers, in total 6 workers are started (3 for each host).

usernameedit

The basic authentication username for connecting to Elasticsearch.

passwordedit

The basic authentication password for connecting to Elasticsearch.

parametersedit

Dictionary of HTTP parameters to pass within the url with index operations.

protocoledit

The name of the protocol Elasticsearch is reachable on. The options are: http or https. The default is http. However, if you specify a URL for hosts, the value of protocol is overridden by whatever scheme you specify in the URL.

pathedit

An HTTP path prefix that is prepended to the HTTP API calls. This is useful for the cases where Elasticsearch listens behind an HTTP reverse proxy that exports the API under a custom prefix.

headersedit

Custom HTTP headers to add to each request created by the Elasticsearch output. Example:

output.elasticsearch.headers:
  X-My-Header: Header contents

It is generally possible to specify multiple header values for the same header name by separating them with a comma.

proxy_urledit

The URL of the proxy to use when connecting to the Elasticsearch servers. The value may be either a complete URL or a "host[:port]", in which case the "http" scheme is assumed. If a value is not specified through the configuration file then proxy environment variables are used. See the golang documentation for more information about the environment variables.

indexedit

The index name to write events to. The default is "apm-server-%{[beat.version]}-%{+yyyy.MM.dd}" (for example, "apm-server-6.2.2-2017.04.26"). If you change this setting, you also need to configure the setup.template.name and setup.template.pattern options (see Load the Elasticsearch index template). If you are using the pre-built Kibana dashboards, you also need to set the setup.dashboards.index option (see Load the Kibana dashboards).

indicesedit

Array of index selector rules supporting conditionals, format string based field access and name mappings. The first rule matching will be used to set the index for the event to be published. If indices is missing or no rule matches, the index field will be used.

Rule settings:

index: The index format string to use. If the fields used are missing, the rule fails.

mapping: Dictionary mapping index names to new names

default: Default string value if mapping does not find a match.

when: Condition which must succeed in order to execute the current rule.

Examples elasticsearch output with indices:

output.elasticsearch:
  hosts: ["http://localhost:9200"]
  index: "logs-%{[beat.version]}-%{+yyyy.MM.dd}"
  indices:
    - index: "critical-%{[beat.version]}-%{+yyyy.MM.dd}"
      when.contains:
        message: "CRITICAL"
    - index: "error-%{[beat.version]}-%{+yyyy.MM.dd}"
      when.contains:
        message: "ERR"

max_retriesedit

The number of times to retry publishing an event after a publishing failure. After the specified number of retries, the events are typically dropped.