elasticsearch

This plugin is the recommended method of storing logs in Elasticsearch. If you plan on using the Kibana web interface, you’ll want to use this output.

This output only speaks the HTTP protocol. HTTP is the preferred protocol for interacting with Elasticsearch as of Logstash 2.0. We strongly encourage the use of HTTP over the node protocol for a number of reasons. HTTP is only marginally slower, yet far easier to administer and work with. When using the HTTP protocol one may upgrade Elasticsearch versions without having to upgrade Logstash in lock-step. For those still wishing to use the node or transport protocols please see the logstash-output-elasticsearch_java plugin.

You can learn more about Elasticsearch at https://www.elastic.co/products/elasticsearch

Retry Policy

This plugin uses the Elasticsearch bulk API to optimize its imports into Elasticsearch. These requests may experience either partial or total failures. Events are retried if they fail due to either a network error or the status codes 429 (the server is busy), 409 (Version Conflict), or 503 (temporary overloading/maintenance).

The retry policy’s logic can be described as follows:

  • Block and retry all events in the bulk response that experience transient network exceptions until a successful submission is received by Elasticsearch.
  • Retry the subset of sent events which resulted in ES errors of a retryable nature.
  • Events which returned retryable error codes will be pushed onto a separate queue for retrying events. Events in this queue will be retried a maximum of 5 times by default (configurable through :max_retries). The size of this queue is capped by the value set in :retry_max_items.
  • Events from the retry queue are submitted again when the queue reaches its max size or when the max interval time is reached. The max interval time is configurable via :retry_max_interval.
  • Events which are not retryable or have reached their max retry count are logged to stderr.

 

Synopsis

This plugin supports the following configuration options:

Required configuration options:

elasticsearch {
}

Available configuration options:

Setting Input type Required Default value

action

string, one of ["index", "delete", "create", "update"]

No

"index"

cacert

a valid filesystem path

No

codec

codec

No

"plain"

doc_as_upsert

boolean

No

false

document_id

string

No

document_type

string

No

flush_size

number

No

500

hosts

array

No

idle_flush_time

number

No

1

index

string

No

"logstash-%{+YYYY.MM.dd}"

keystore

a valid filesystem path

No

keystore_password

password

No

manage_template

boolean

No

true

max_retries

number

No

3

password

password

No

path

string

No

"/"

proxy

<<,>>

No

retry_max_interval

number

No

5

retry_max_items

number

No

5000

routing

string

No

sniffing

boolean

No

false

sniffing_delay

number

No

5

ssl

boolean

No

false

ssl_certificate_verification

boolean

No

true

template

a valid filesystem path

No

template_name

string

No

"logstash"

template_overwrite

boolean

No

false

timeout

number

No

truststore

a valid filesystem path

No

truststore_password

password

No

upsert

string

No

""

user

string

No

workers

number

No

1

Details

 

action

  • Value can be any of: index, delete, create, update
  • Default value is "index"

The Elasticsearch action to perform. Valid actions are:

  • index: indexes a document (an event from Logstash).
  • delete: deletes a document by id (An id is required for this action)
  • create: indexes a document, fails if a document by that id already exists in the index.
  • update: updates a document by id. Update has a special case where you can upsert — update a document if not already present. See the upsert option

For more details on actions, check out the Elasticsearch bulk API documentation

cacert

  • Value type is path
  • There is no default value for this setting.

The .cer or .pem file to validate the server’s certificate

codec

  • Value type is codec
  • Default value is "plain"

The codec used for output data. Output codecs are a convenient method for encoding your data before it leaves the output, without needing a separate filter in your Logstash pipeline.

doc_as_upsert

  • Value type is boolean
  • Default value is false

Enable doc_as_upsert for update mode. Create a new document with source if document_id doesn’t exist in Elasticsearch

document_id

  • Value type is string
  • There is no default value for this setting.

The document ID for the index. Useful for overwriting existing entries in Elasticsearch with the same ID.

document_type

  • Value type is string
  • There is no default value for this setting.

The document type to write events to. Generally you should try to write only similar events to the same type. String expansion %{foo} works here. Unless you set document_type, the event type will be used if it exists otherwise the document type will be assigned the value of logs

flush_size

  • Value type is number
  • Default value is 500

This plugin uses the bulk index API for improved indexing performance. To make efficient bulk API calls, we will buffer a certain number of events before flushing that out to Elasticsearch. This setting controls how many events will be buffered before sending a batch of events. Increasing the flush_size has an effect on Logstash’s heap size. Remember to also increase the heap size using LS_HEAP_SIZE if you are sending big documents or have increased the flush_size to a higher value.

hosts

  • Value type is array
  • There is no default value for this setting.

Sets the host(s) of the remote instance. If given an array it will load balance requests across the hosts specified in the hosts parameter. Remember the http protocol uses the http address (eg. 9200, not 9300). "127.0.0.1" ["127.0.0.1:9200","127.0.0.2:9200"] It is important to exclude dedicated master nodes from the hosts list to prevent LS from sending bulk requests to the master nodes. So this parameter should only reference either data or client nodes in Elasticsearch.

idle_flush_time

  • Value type is number
  • Default value is 1

The amount of time since last flush before a flush is forced.

This setting helps ensure slow event rates don’t get stuck in Logstash. For example, if your flush_size is 100, and you have received 10 events, and it has been more than idle_flush_time seconds since the last flush, Logstash will flush those 10 events automatically.

This helps keep both fast and slow log streams moving along in near-real-time.

index

  • Value type is string
  • Default value is "logstash-%{+YYYY.MM.dd}"

The index to write events to. This can be dynamic using the %{foo} syntax. The default value will partition your indices by day so you can more easily delete old data or only search specific date ranges. Indexes may not contain uppercase characters. For weekly indexes ISO 8601 format is recommended, eg. logstash-%{+xxxx.ww}

index_type (DEPRECATED)

  • DEPRECATED WARNING: This configuration item is deprecated and may not be available in future versions.
  • Value type is string
  • There is no default value for this setting.

The index type to write events to. Generally you should try to write only similar events to the same type. String expansion %{foo} works here.

Deprecated in favor of document_type field.

keystore

  • Value type is path
  • There is no default value for this setting.

The keystore used to present a certificate to the server. It can be either .jks or .p12

keystore_password

  • Value type is password
  • There is no default value for this setting.

Set the truststore password

manage_template

  • Value type is boolean
  • Default value is true

Starting in Logstash 1.3 (unless you set option manage_template to false) a default mapping template for Elasticsearch will be applied, if you do not already have one set to match the index pattern defined (default of logstash-%{+YYYY.MM.dd}), minus any variables. For example, in this case the template will be applied to all indices starting with logstash-*

If you have dynamic templating (e.g. creating indices based on field names) then you should set manage_template to false and use the REST API to upload your templates manually.

max_retries

  • Value type is number
  • Default value is 3

Set max retry for each event

password

  • Value type is password
  • There is no default value for this setting.

Password to authenticate to a secure Elasticsearch cluster

path

  • Value type is string
  • Default value is "/"

HTTP Path at which the Elasticsearch server lives. Use this if you must run Elasticsearch behind a proxy that remaps the root path for the Elasticsearch HTTP API lives.

proxy

<li> Value type is <<string,string>>
* There is no default value for this setting.

Set the address of a forward HTTP proxy. Can be either a string, such as http://localhost:123 or a hash in the form of {host: 'proxy.org' port: 80 scheme: 'http'}. Note, this is NOT a SOCKS proxy, but a plain HTTP proxy

retry_max_interval

  • Value type is number
  • Default value is 5

Set max interval between bulk retries

retry_max_items

  • Value type is number
  • Default value is 5000

Set retry policy for events that failed to send

routing

  • Value type is string
  • There is no default value for this setting.

A routing override to be applied to all processed events. This can be dynamic using the %{foo} syntax.

sniffing

  • Value type is boolean
  • Default value is false

This setting asks Elasticsearch for the list of all cluster nodes and adds them to the hosts list. Note: This will return ALL nodes with HTTP enabled (including master nodes!). If you use this with master nodes, you probably want to disable HTTP on them by setting http.enabled to false in their elasticsearch.yml. You can either use the sniffing option or manually enter multiple Elasticsearch hosts using the hosts paramater.

sniffing_delay

  • Value type is number
  • Default value is 5

How long to wait, in seconds, between sniffing attempts

ssl

  • Value type is boolean
  • Default value is false

Enable SSL/TLS secured communication to Elasticsearch cluster

ssl_certificate_verification

  • Value type is boolean
  • Default value is true

Option to validate the server’s certificate. Disabling this severely compromises security. For more information on disabling certificate verification please read https://www.cs.utexas.edu/~shmat/shmat_ccs12.pdf

template

  • Value type is path
  • There is no default value for this setting.

You can set the path to your own template here, if you so desire. If not set, the included template will be used.

template_name

  • Value type is string
  • Default value is "logstash"

This configuration option defines how the template is named inside Elasticsearch. Note that if you have used the template management features and subsequently change this, you will need to prune the old template manually, e.g.

curl -XDELETE <http://localhost:9200/_template/OldTemplateName?pretty>

where OldTemplateName is whatever the former setting was.

template_overwrite

  • Value type is boolean
  • Default value is false

Overwrite the current template with whatever is configured in the template and template_name directives.

timeout

  • Value type is number
  • There is no default value for this setting.

Set the timeout for network operations and requests sent Elasticsearch. If a timeout occurs, the request will be retried.

truststore

  • Value type is path
  • There is no default value for this setting.

The JKS truststore to validate the server’s certificate. Use either :truststore or :cacert

truststore_password

  • Value type is password
  • There is no default value for this setting.

Set the truststore password

upsert

  • Value type is string
  • Default value is ""

Set upsert content for update mode. Create a new document with this parameter as json string if document_id doesn’t exists

user

  • Value type is string
  • There is no default value for this setting.

Username to authenticate to a secure Elasticsearch cluster

workers

  • Value type is number
  • Default value is 1

The number of workers to use for this output. Note that this setting may not be useful for all outputs.