When you specify Elasticsearch for the output, the Beat sends the transactions directly to Elasticsearch by using the Elasticsearch HTTP API.
output: elasticsearch: # The Elasticsearch cluster hosts: ["http://localhost:9200"] # Comment this option if you don't want to store the topology in # Elasticsearch. The default is false. # This option makes sense only for Packetbeat # save_topology: false # Optional index name. The default is "topbeat" and generates # [topbeat-]YYYY.MM.DD keys. index: "topbeat" # tls configuration. By default is off. tls: # List of root certificates for HTTPS server verifications certificate_authorities: ["/etc/pki/root/ca.pem"] # Certificate for TLS client authentication certificate: "/etc/pki/client/cert.pem" # Client Certificate Key certificatekey: "/etc/pki/client/cert.key"
To enable SSL, just add
https to all URLs defined under hosts.
output: elasticsearch: # The Elasticsearch cluster hosts: ["https://localhost:9200"] # Comment this option if you don't want to store the topology in # Elasticsearch. The default is false. # This option makes sense only for Packetbeat # save_topology: false # HTTP basic auth username: "admin" password: "s3cr3t"
If the Elasticsearch nodes are defined by
IP:PORT, then add
protocol: https to the yaml file.
output: elasticsearch: # The Elasticsearch cluster hosts: ["localhost"] # Optional http or https. Default is http protocol: "https" # Comment this option if you don't want to store the topology in # Elasticsearch. The default is false. # This option makes sense only for Packetbeat # save_topology: false # HTTP basic auth username: "admin" password: "s3cr3t"
You can specify the following options in the
The list of Elasticsearch nodes to connect to. The events are distributed to
these nodes in round robin order. If one node becomes unreachable, the event is
automatically sent to another node. Each Elasticsearch node can be defined as a
If no port is specified,
9200 is used.
output: elasticsearch: # The Elasticsearch cluster hosts: ["10.45.3.2:9220", "10.45.3.1:9230"] # Optional http or https. Default is http protocol: https # HTTP Path at which each Elasticsearch server lives path: /elasticsearch
In the previous example, the Elasticsearch nodes are available at
The number of workers per configured host publishing events to Elasticsearch. This is best used with load balancing mode enabled. Example: If you have 2 hosts and 3 workers, in total 6 workers are started (3 for each host).
The host of the Elasticsearch server. This option is deprecated because it is replaced by hosts.
The port of the Elasticsearch server. This option is deprecated because it is replaced by hosts.
The basic authentication username for connecting to Elasticsearch.
The basic authentication password for connecting to Elasticsearch.
The name of the protocol Elasticsearch is reachable on. The options are:
https. The default is
http. However, if you specify a URL for
hosts, the value of
protocol is overridden by whatever scheme you
specify in the URL.
An HTTP path prefix that is prepended to the HTTP API calls. This is useful for the cases where Elasticsearch listens behind an HTTP reverse proxy that exports the API under a custom prefix.
The URL of the proxy to use when connecting to the Elasticsearch servers. The value may be either a complete URL or a "host[:port]", in which case the "http" scheme is assumed. If a value is not specified through the configuration file then proxy environment variables are used. See the golang documentation for more information about the environment variables.
The index root name to write events to. The default is the Beat name. For example "topbeat" generates "[topbeat-]YYYY.MM.DD" indexes (for example, "topbeat-2015.04.26").
The index template to use for setting mappings in Elasticsearch. By default, template loading is disabled. If you leave this option disabled, you must load the template by running the load script that is provided with Topbeat.
If you enable the
template config option, you can adjust the following settings to load your
own template or overwrite an existing one:
The name of the template. The default is
- The path to the template file.
- A boolean that specifies whether to overwrite the existing template. The default is false.
output: elasticsearch: hosts: ["localhost:9200"] name: "topbeat" path: "topbeat.template.json" overwrite: false
The number of times to retry publishing an event after a publishing failure.
After the specified number of retries, the events are typically dropped.
Some Beats, such as Filebeat, ignore the
max_retries setting and retry until all
events are published.
max_retries to a value less than 0 to retry until all events are published.
The default is 3.
The maximum number of events to bulk in a single Elasticsearch bulk API index request. The default is 50.
If the Beat sends single events, the events are collected into batches. If the Beat publishes
a large batch of events (larger than the value specified by
bulk_max_size), the batch is
Specifying a larger batch size can improve performance by lowering the overhead of sending events. However big batch sizes can also increase processing times, which might result in API errors, killed connections, timed-out publishing requests, and, ultimately, lower throughput.
bulk_max_size to values less than or equal to 0 disables buffering in libbeat. When buffering is disabled,
Beats that publish single events (such as Packetbeat and Topbeat) send each event directly to
Elasticsearch. Beats that publish data in batches (such as Filebeat) send events in batches based on the
The http request timeout in seconds for the Elasticsearch request. The default is 90.
The number of seconds to wait for new events between two bulk API index requests.
bulk_max_size is reached before this interval expires, additional bulk index
requests are made.
A Boolean that specifies whether the topology is kept in Elasticsearch. The default is false.
This option is relevant for Packetbeat only.
The time to live in seconds for the topology information that is stored in Elasticsearch. The default is 15 seconds.
Configuration options for TLS parameters like the certificate authority to use
for HTTPS-based connections. If the
tls section is missing, the host CAs are used for HTTPS connections to
See TLS Options for more information.
Intro to Kibana
ELK for Logs & Metrics