Curator 5 is out, with some amazing new features. It now supports both rollover and reindex, including reindex from remote! And the new period filter, too!

fondos-archivo.jpg

awi-core-archive-720x420.jpg

Announcing Curator 5

In <a href="https://www.elastic.co/blog/little-logstash-lessons-part-using-grok-mutate-type-data" target="_blank">Little Logstash Lessons Part 1</a>, you configured Logstash to parse your data and send it to Elasticsearch. &nbsp;Elasticsearch does a terrific job of guessing what your data types are, and how to handle them. &nbsp;But what if you know you’re going to have a lot of data, and you want to tune Elasticsearch to store that data as efficiently as possible? &nbsp;This has a lot of benefits, including reduced storage requirements, but it can also help reduce memory requirements for aggregations, and other large and complex queries. &nbsp;The way to do this is with a mapping template.<h4 dir="ltr">What are mappings?</h4> Simply put, mappings define how fields are interpreted by Elasticsearch. &nbsp;Elasticsearch is powerful because you can send data to it, and it will make very intelligent guesses as to what each field’s mapping should be. &nbsp;We refer to this as “schema-less” indexing, which is a terrific option if you’re just getting started with Elasticsearch. &nbsp;It’s also great in cases where you are continuously sending new data and you do not know what fields will be in it. As mentioned, however, there are great benefits in knowing what data is going to Elasticsearch, and “mapping,” or telling Elasticsearch exactly how that data should be treated. &nbsp;For example, if I send an IP address (like 8.8.8.8, which is one of Google’s DNS servers), it will appear as a string field in the JSON sent to Elasticsearch: <pre dir="ltr">{ “ip” : “8.8.8.8” } </pre> Even though the IP address is an IP to you and me, Elasticsearch will think of it as a string. &nbsp;For it to be more, you have to tell Elasticsearch how to treat the value, and the way to do that is with a mapping. If an IP is mapped as type ip in Elasticsearch, you can do a query, and filter by an IP range—you can even use CIDR notation—to include only IPs within a given subnet. While there is a lot of documentation about mappings themselves—and some pointers will be provided here—the primary goal of this post is to show you a relatively easy way to get Elasticsearch to create a mapping for you, and then take that and make a mapping template out of it. NOTE: For the purposes of this exercise, we will be using versions 5.3.0 of Logstash and Elasticsearch. &nbsp;If you are using an older version of Elasticsearch, particularly a 2.x version, your mapping will probably look very different from this one. <h4 dir="ltr">Get Logstash to send sample data to Elasticsearch</h4> Let’s try an example with apache access log data. <pre>input { stdin {} } filter { grok { match =&gt; { "message" =&gt; "%{COMBINEDAPACHELOG}" } remove_field =&gt; "message" } date { match =&gt; [ "timestamp", "dd/MMM/YYYY:HH:mm:ss Z" ] locale =&gt; en remove_field =&gt; ["timestamp"] } geoip { source =&gt; "clientip" } useragent { source =&gt; "agent" target =&gt; "useragent" } } output { elasticsearch { hosts =&gt; [ "127.0.0.1" ] index =&gt; "my_index" document_type =&gt; "mytype" } } </pre> With a setup like this, you are sending apache output data to a specifically named index. &nbsp;This index will only be temporary, so name it uniquely enough that it won’t be too similar to any other index name. &nbsp;The best part is, I can do this on my laptop. &nbsp;I don’t need to run this on a production node, or even a staging one. &nbsp;I can spin up a local instance of Elasticsearch and Logstash and get the results I need that way. Let’s send a few lines now: <pre dir="ltr">head -50 /path/to/apache/access_log | /usr/share/logstash/bin/logstash -f above_sample.conf </pre> And the output in the Elasticsearch node will look something like this: <pre dir="ltr">[2017-03-29T13:48:42,794][INFO ][o.e.c.m.MetaDataCreateIndexService] [iXsjSQY] [my_index] creating index, cause [auto(bulk api)], templates [], shards [5]/[1], mappings [] </pre><h4>Get and edit the mapping</h4> Now that we have an index, let’s see what our mapping looks like: <pre dir="ltr">curl -XGET http://127.0.0.1:9200/my_index/_mapping?pretty &gt; my_mapping.json </pre> Now if you edit my_mapping.json in your favorite text editor (something that won’t reformat the file, like a word processor), you’ll see something like this: <pre>{ "my_index" : { "mappings" : { "mytype" : { "properties" : { </pre> ...followed by a few hundred other lines of content. &nbsp;Let’s look at what we have here: Of course, my_index&nbsp;is the index name we specified, and mappings&nbsp;indicates that the following defines the mappings for the index. mytype&nbsp;is the document_type&nbsp;we specified in the elasticsearch output plugin, and properties&nbsp;is where all of the fields will be defined. As you can see, almost every field is of type text, and contains a secondary sub-field type of keyword.&nbsp;The keyword type is important because it treats the entire contents of the field (up to the ignore_above&nbsp;number of characters) as a single entity. &nbsp;This ensures that a user agent field containing operating system names like “Windows 10” are identified by Elasticsearch as “Windows 10” rather than have “Windows” and “10” separated. What about some of the other fields? Take a look at the nested fields in the geoip section: <pre> "geoip" : { "properties" : { "city_name" : { "type" : "text", "fields" : { "keyword" : { </pre> Do you see how there’s another properties&nbsp;block within geoip? &nbsp;It’s just like the one encapsulating all of the fields. &nbsp;Whenever you encounter a new level in a mapping definition, you’ll see another properties block. Within the geoip object, there are a few numeric fields, and an IP field: <pre> "dma_code" : { "type" : "long" }, "ip" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } }, "latitude" : { "type" : "float" }, "location" : { "type" : "float" }, "longitude" : { "type" : "float" }, </pre> At this point, I want to make a few important points about Logstash and grok, and their relation to Elasticsearch mapping. No matter how you may cast a field with grok (which supports int and float types), Elasticsearch will not know exactly what you intend unless you map it exactly. &nbsp;Logstash will render your log line as JSON, containing all of the fields you configured it to contain. &nbsp;Typically, this will contain strings and numbers. Numbers will be either integers (whole numbers) or floating point values (numbers with decimals). &nbsp;After that, it’s up to Elasticsearch to interpret these field types, either by having them explicitly mapped, or through the previously mentioned schema-less approach. We have already seen how well Elasticsearch handles strings and text values when using a schema-less approach. It also makes its best guess with numeric values. &nbsp;Generally, any integer value will turn up as a long. Values with decimals will show up as floats, or doubles. As already mentioned, you can save storage space as well as some overhead for calculations and aggregations if you map your values with the <a href="https://www.elastic.co/guide/en/elasticsearch/reference/5.3/number.html" target="_blank">smallest possible primitive value</a>.&nbsp;This is one of the primary benefits of creating a custom mapping. For instance, latitude and longitude can easily be represented by half_floats, and the dma_code can be represented as a short. Each of these changes reduces overhead in Elasticsearch. And what about our IP address example from the beginning? <pre> "ip" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } }, </pre> This ip field is being identified as text and keyword, and not as an IP. &nbsp;To map an IP field as type ip, replace that entire block with this: <pre dir="ltr"> "ip" : { "type" : "ip" }, </pre> Pretty simple, right? &nbsp;At this point, you may wonder why I am instructing you to only map the ip field in the geoip object, rather than the clientip field, or both. &nbsp;This is because in certain web server configurations, the log data that becomes the clientip field can contain an IPv4 address, an IPv6 address, or a host name. &nbsp;If you were to map the clientip field as type ip, and a log line with a host name were to be pushed to Elasticsearch it would result in an error, and that document would not be indexed due to a mapping conflict. The location field is also a special mapping type. It is an array of longitude and latitude, indicating a single point on a map. Elasticsearch stores these as mapping type geo_point, which enables users to do geographic searches and plot points on maps in Kibana. &nbsp;Map a geo_point like this: <pre dir="ltr"> "location" : { "type" : "geo_point" }, </pre> With all of these changes, the edited mapping for this section should appear as follows: <pre> "dma_code" : { "type" : "short" }, "ip" : { "type" : "ip" }, "latitude" : { "type" : "half_float" }, "location" : { "type" : "geo_point" }, "longitude" : { "type" : "half_float" }, </pre> The response field, which is outside the geoip block, is also a number, but it ended up stored as a string. &nbsp;This is because the grok subpattern that created this field does not cast it as an int or float. &nbsp;That doesn’t matter, though! Mappings can coerce a number stored as a string into a numeric mapping type. Since HTTP response codes will only ever have 3 digits, and the smallest mapping type to account for that is a short, it should look like this: <pre dir="ltr"> "response" : { "type" : "short" }, </pre><h4>Make the template</h4> So now we have a big mapping file with a few changes. &nbsp;How do we change this mapping file into a template? It's actually quite simple to turn a regular mapping into a mapping template!&nbsp; This is sample of what a mapping template might look like, including where to put your edited properties section: <pre>{ "template" : "my_index*", "version" : 50001, "settings" : { "index.refresh_interval" : "5s" }, "mappings" : { "_default_" : { YOUR PROPERTIES SECTION GOES HERE } } } </pre> An elasticsearch template at its most basic contains an index pattern to match, identified as template,&nbsp;and your default mappings. &nbsp;In this example, I’ve included a version, in case you want to track changes to your template, and a default refresh_interval of 5 seconds, which helps performance under higher indexing loads. &nbsp; To make this a complete template, you will need to&nbsp;replace&nbsp;a few lines from the my_mapping.json file. &nbsp;Copy that file to my_template.json, and open it in your editor. &nbsp;Then replace the 4 top lines, which should look like this: <pre>{ "my_index" : { "mappings" : { "mytype" : { </pre> With these lines: <pre>{ "template" : "my_index*", "version" : 50001, "settings" : { "index.refresh_interval" : "5s" }, "mappings" : { "_default_" : { </pre> The very next line should be: <pre dir="ltr"> "properties" : { </pre> IMPORTANT: We removed 4 opening curly braces and replaced them with 3. &nbsp;That means we have one too many closing curly braces at the end of our file, so scroll down and remove one of them. <pre> } } } } </pre> Should be: <pre> } } } </pre><h4>Save the template and upload it</h4> Save your file, and you’re ready to upload it: <pre dir="ltr">curl -XPUT http://localhost:9200/_template/my_index_template?pretty -d @my_template.json </pre> If your mapping is accurate, you will see: <pre>{ "acknowledged" : true } </pre><h4>Test the template</h4> So, now it’s time to test our template. Delete my_index from Elasticsearch by running the first line in the Kibana Console: <pre dir="ltr">DELETE /my_index </pre> or this line at the command-line: <pre>curl -XDELETE http://localhost:9200/my_index?pretty </pre> Re-run the Logstash sample command you ran earlier to test your template. When you check the mapping of my_index, you will see all of your changes! &nbsp; Happy Logstashing!<div> </div>

map_with_pins.jpg

cardinal_signs.jpg

​Little Logstash Lessons: Using Logstash to help create an Elasticsearch mapping template

I am very pleased to announce the release of <a href="https://www.elastic.co/guide/en/elasticsearch/client/curator/4.1/installation.html">Elasticsearch Curator 4.1</a>! &nbsp;There are some new features I’m excited to tell you about. <h3>Configuration Validation</h3>Curator 4.0 did not validate configuration options. &nbsp;As a result, some users had a bad experience —&nbsp;like having all of their indices selected, instead of a subset. &nbsp;So the first new feature in Curator 4.1 aims to prevent those sorts of situations with Configuration Validation. &nbsp;This was a bigger project than I’d originally anticipated, but I wanted to get it right. &nbsp;As a result, Curator will now fail to run if any option is configured outside of acceptable parameters. &nbsp;This does not prevent something from running if you set a 2 instead of a 3, for example, but other typos will be caught immediately. &nbsp;As always, it is recommended that you first test any new or changed configuration with the --dry-run flag for safety.<h3>Filter by Alias</h3>Curator 4.1 now allows you to <a href="https://www.elastic.co/guide/en/elasticsearch/client/curator/4.1/filtertype_alias.html">select indices based on their inclusion in an alias</a>. Or even <a href="https://www.elastic.co/guide/en/elasticsearch/client/curator/4.1/fe_aliases.html">multiple aliases</a>!<pre class="prettyprint">- filtertype: alias aliases: [ alias_one, alias_two ] exclude: False</pre><h3>Filter by Count</h3>You can now also <a href="https://www.elastic.co/guide/en/elasticsearch/client/curator/4.1/filtertype_count.html">filter by count</a>, or in other words, keep the most recent n indices or snapshots (or the earliest, you choose). &nbsp;It uses similar configuration to the space filter, in that you can specify to sort by alphanumeric (the default) or by age.<pre class="prettyprint">- filtertype: count count: 10 use_age: True source: creation_date</pre>This filter configuration will preserve the 10 most recent items by removing them from the actionable list. &nbsp;You would likely use this in conjunction with the pattern filter to keep only the n most recent items matching a given pattern, like this: <pre class="prettyprint">filters: - filtertype: pattern kind: prefix value: 'logstash-' - filtertype: count count: 10 use_age: True source: creation_date</pre>These filters will keep the 10 most recent indices starting with logstash-, based on the index creation_date, and perform the selected action on all other indices starting with logstash-.<h3>Environment Variables</h3>Also new in Curator 4.1 is the ability to use <a href="https://www.elastic.co/guide/en/elasticsearch/client/curator/4.1/envvars.html">environment variables</a> in both your <a href="https://www.elastic.co/guide/en/elasticsearch/client/curator/4.1/configfile.html">configuration</a> and <a href="https://www.elastic.co/guide/en/elasticsearch/client/curator/4.1/actionfile.html">action</a> files. &nbsp;This feature is experimental, so please use caution when testing it out. &nbsp;When in doubt, always use the --dry-run flag.To use an environment variable in a configuration file, use:<pre dir="ltr">${VAR}</pre>Where VAR is the name of the environment variable.Each variable reference is replaced at startup by the value of the environment variable. The replacement is case-sensitive and occurs while the YAML file is parsed, but before configuration schema validation. References to undefined variables are replaced by None unless you specify a default value. To specify a default value, use:<pre dir="ltr">${VAR:default_value}</pre>Where default_value is the value to use if the environment variable is undefined.Please note that environment variables are only available for values in YAML and not for keys.&nbsp;In other words, you can do this:<pre dir="ltr">key: ${ENVVAR}</pre>But not this:<pre class="prettyprint">${ENVVAR}: value</pre><h3>Conclusion</h3>These new features open the door to even more possibilities with Curator. I can’t wait to hear how you’re using them! As always, if you encounter an issue, you are always welcome to find help in our <a href="https://discuss.elastic.co/c/elasticsearch">discussion forum</a>, or submit any issue you may encounter at our <a href="https://github.com/elastic/curator/issues">GitHub repository</a>.Happy Curating!

New features in Curator 4.1

Improve the performance of the Redis input plugin with these tips and details.

Message queues are used in Logstash deployments to <a href="https://www.elastic.co/guide/en/logstash/current/deploying-and-scaling.html#deploying-message-queueing" target="_blank">control</a> a surge in events which may lead to slowdown in Elasticsearch or other downstream components. Redis is one of the technologies in use as a message queue. Due to its speed, ease of use, and low resource requirements it is also one of the most popular ways as well.
That&nbsp;is probably why it has been around for a very, very long time—since Logstash version 1.0.4, released more than five years ago!
With all of the recent changes to improve the Logstash pipeline—as well as improvements to the Redis plugin itself—you may not be getting all the performance the Logstash Redis input plugin is capable of delivering.
Logstash attempts to detect the number of cores in your system and set the number of pipeline workers to match. But pipeline workers only matter for filter and output plugins. &nbsp;It has no effect on input plugins. If you haven’t set `threads` in the Redis input plugin, you are only ingesting a fraction of what you could.
The test setup included the most recent stable version of Logstash (2.3.4) with Redis 3.0.7&nbsp;running on the same machine, which is a 2015 MacBook Pro, for reference. The Logstash configuration used is
<pre>input { 
 redis {
 host =&gt; "127.0.0.1"
 data_type =&gt; "list"
 key =&gt; "redis_test"
 # batch_count =&gt; 1
 # threads =&gt; 1
 }
}
output { stdout { codec =&gt; dots } }
</pre>Run this with bin/logstash -f some_config_file.config | pv -Wart &gt; dev/null
This test setup is only designed to test for maximum throughput. Actual performance numbers will vary widely depending on what other plugins are used.
By iterating through a few options for only the threads&nbsp;setting, and using all other settings at their defaults, we see results like these:
<ul>
	<li>threads =&gt; 1&nbsp;: 37.4K/sec</li>
	<li>threads =&gt; 2&nbsp;: 57K/sec</li>
	<li>threads =&gt; 4&nbsp;: 67K/sec</li>
	<li>threads =&gt; 8&nbsp;: 87.3K/sec</li>
</ul>Let’s run the same tests again, but with batch_count =&gt; 250, and add -b 250&nbsp;to our Logstash command line to take full advantage of that increase:
<ul>
	<li>threads =&gt; 1&nbsp;: 35.2K/sec</li>
	<li>threads =&gt; 2&nbsp;: 48.9K/sec</li>
	<li>threads =&gt; 4&nbsp;: 58.7K/sec</li>
	<li>threads =&gt; 8&nbsp;: 72.5K/sec</li>
</ul><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltc2a4980fa1466723/5f85fd80dcf0e74b7cbf58d0/blog-redis-logstash-1.png" title="Throughput: batch size vs threads" height="371" width="600" "="">
As you can see, increasing the default batch_count&nbsp;(which is 125), to 250 results in a decrease in performance.
It’s been common in the past for users to use a larger batch_count&nbsp;when working with Redis. &nbsp;This is no longer best practices, and will actually reduce performance. &nbsp;This is primarily due to changes in the Logstash pipeline architecture. Even an increase of batch_count&nbsp;to 250 results in a drop of performance from 89.5K/sec down to 72.5K/sec when threads =&gt; 8. The sweet spot for batch_count&nbsp;is right around 125, which is the default pipeline-batch-size&nbsp;(there’s a reason for that!).
<h4>Diving deeper</h4>Recent changes to the Redis input include turning on batch mode by default and using a <a href="https://github.com/logstash-plugins/logstash-input-redis/blob/df5bc905e41d36905363229038fbe263a34d17f6/lib/logstash/inputs/redis.rb#L141" target="_blank">Lua script</a> to execute the batch retrieval commands on the Redis server. Lua scripts also act as a transaction so we can take a batch and shorten the queue in one step. This forces other Redis connections to wait. Note, your Logstash Redis output plugin in the upstream Logstash config is one such connection.
Also, you should be aware that the manner in which events are added to Redis by the upstream Logstash can affect the performance of the downstream Logstash instance. For example, while preparing the results of the tests, we found that querying the list size with LLEN redis_test&nbsp;in redis_cli too often would affect the measurement. Try to understand these factors in your upstream Logstash instance:
<ul>
	<li>the rate that events are generated</li>
	<li>the size of the events, smallest; largest and average</li>
	<li>the batch_events size used the redis output</li>
</ul>You might want to look at using bigger batch_events sizes e.g. 500 on the upstream Redis output - this will buffer the events before calling Redis and doing this may give the downstream Redis input threads time to jump in to pull from Redis. However, if the event generation rate is high then this may not be as helpful - 50K/sec in batches of 500 means a batch buffer fills in 10 milliseconds.
We have seen that the performance of the Lua script suffers proportionately with larger batch sizes. This means that larger batch sizes on the Redis input will affect the performance of both of the Logstash instances on either side of Redis.
Generally, the goal of tuning the Redis input plugin config and the pipeline batch size and worker threads is to make sure that the input is able to keep pace with the filter/output stage. You should know what the average throughput of your filter/output stage is with the defaults. If, for example, your filter/output stage throughput is 35K/sec then there is little need to change the redis input worker count to 8 when one or two workers can fetch sufficient events per second.
One more source of variation is JSON encoding (1), decoding (2) and encoding (3) - yes, three times, 1 in the upstream Redis output, 2 in the downstream Redis input and 3 in the elasticsearch output. Events with big string values do take some time to decode so make sure your testing is using data sets that represent the size and distribution of size as close to your production data as you can make it.
<h4>Scaling throughput using multiple Logstash instance and one Redis server</h4>Many users seek to improve throughput by starting up multiple Logstash instances on different hardware all pointing to the same Redis server and list.
This does work and can work well, but you are advised to apply the same tuning tips to each instance. There is probably not much point in having multiple Redis inputs defined in any one config unless they are connecting to different Redis servers or a different list. Simply tune the threads of the one Redis input. Clearly, having, say, 3 Logstash instances pointing at the same Redis server will create some degree of Redis client connection contention - meaning that you should probably tune one instance and then retune all three when running together.
<h4>Conclusion</h4>In the not too distant future, we will be enhancing the Redis input with a JRuby wrapper around the popular and speedy Jedis Java library. &nbsp;This will bring the added bonus of <a href="https://github.com/logstash-plugins/logstash-input-redis/issues/10" target="_blank">supporting</a> Redis Cluster mode!
In the meanwhile, to give your Redis input performance a boost, try increasing the number of threads used. And be sure to measure and tune accordingly if you use something other than the default batch_count, or have multiple instances of Logstash pointing to a single Redis server.
If you have a question about Redis and Logstash, feel free to open a discussion topic on our <a href="https://discuss.elastic.co/c/logstash" target="_blank">forum</a>. You can also find us on IRC (#logstash), and on <a href="https://github.com/elastic/logstash" target="_blank">GitHub</a>.
 
Happy Logstashing!

blog-banner-release-logstash.png

blog-thumb-release-logstash.png

Just Enough Redis for Logstash

In this tutorial, you will learn how to create a centralized rsyslog server to store log files from multiple systems using Logstash & Elasticsearch.

logstash-ubuntu-full.jpg

logstash-ubuntu-loggy.jpg

How To Centralize Logs with Rsyslog, Logstash, and Elasticsearch on Ubuntu 14.04

<html><head></head><body>Hello, fellow Logstashers!
We&#x2019;ve been pleased with the community response to our recent 2.0 release. &#xA0;As this was a major release, there were some breaking changes, especially in conjunction with the Elasticsearch 2.0 release. &#xA0;One of these changes was that <a href="https://www.elastic.co/guide/en/elasticsearch/reference/2.0/breaking_20_mapping_changes.html#_field_names_may_not_contain_dots">Elasticsearch 2.0 does not support field names with a .&#xA0;(or dot character) in them</a>. &#xA0;A few Logstash plugins had to be changed to not use dotted fields to be compatible, like the metrics and elapsed filters. &#xA0;Unfortunately, many users have no control over the sources of their fields. &#xA0;This has resulted in a poor user experience where dotted fields existed. &#xA0;To address this issue, the <a href="https://github.com/logstash-plugins/logstash-filter-de_dot" target="_blank">de_dot</a>&#xA0;filter has been created. &#xA0;You can find the plugin documentation for it <a href="https://www.elastic.co/guide/en/logstash/master/plugins-filters-de_dot.html" target="_blank">here</a>.
You can install the de_dot filter using the plugin command:
<pre class="prettyprint">bin/plugin install logstash-filter-de_dot</pre>The de_dot filter will replace dots with underscores by default. &#xA0;It&#x2019;s as simple as that! &#xA0;If I had a dotted field called <code>baked.apple.pie</code>, the de_dot filter would change the field name to <code>baked_apple_pie</code>. &#xA0;You can also choose the separator with the <code>separator</code>&#xA0;configuration option. &#xA0;A separator does not have to be a single character, either. &#xA0;You can make a separator anything you want, except a dot. &#xA0;Please remember to change your queries and filters in Kibana to match whatever you change in the de_dot filter!
We also added an option to translate dots into nested fields. &#xA0;If you set <code>nested =&gt; true</code>&#xA0;in your configuration, Logstash will ignore the <code>separator</code>&#xA0;and attempt to convert dotted fields into nested fields. &#xA0;For example, if I had a dotted field called <code>top.level1.level2</code>, it would become <code>[top][level1][level2]</code>. &#xA0;The flexibility is yours to choose.
We&#x2019;ve provided a <code>fields</code>&#xA0;configuration option to allow you to specify which fields need to be processed. &#xA0;If no fields are provided, Logstash will check all&#xA0;of your fields for dots. &#xA0;The de-dot process can be a performance hit to your pipeline, so pre-specifying fields will prevent Logstash from having to check each for dots. &#xA0;Please note that you won&#x2019;t need to use a conditional to check for dots in field names, as the plugin does this for you.&#xA0; The idea behind specifying <code>fields</code>&#xA0;manually is to prevent Logstash from having to check every field, which can result in improved performance. &#xA0;If you are in a situation where you do not know all possible field names, it may be helpful to tag events which may&#xA0;have dotted fields, and put the de_dot filter within a conditional that checks for that tag.
The <code>fields</code>&#xA0;option also allows you to de_dot nested fields. &#xA0;The current release of the de_dot filter will only check the top level of an event for dotted fields. &#xA0;If you have a dotted field in a nested object, the de_dot filter cannot find it without help. &#xA0;In this case you must use the <code>fields</code>&#xA0;directive to specify a nested field. &#xA0;This can be done by using the field reference syntax: <code>fields =&gt; [ &quot;[top][level.1]&quot;, &quot;[top][level.2]&quot; ]</code>&#xA0;will find the nested dotted fields <code>level.1</code>&#xA0;and <code>level.2</code>. &#xA0;You can use either the <code>separator</code>&#xA0;or <code>nested</code>&#xA0;options to act on these fields.
We hope that the de_dot filter will help ease your transition to Logstash 2.0 and Elasticsearch 2.0. &#xA0;If you need help configuring the de_dot filter, please visit our discussion forum at <a href="https://discuss.elastic.co/c/logstash">https://discuss.elastic.co/c/logstash</a>. &#xA0;If you have suggestions or find an issue, please submit a ticket at <a href="https://github.com/logstash-plugins/logstash-filter-de_dot">https://github.com/logstash-plugins/logstash-filter-de_dot</a>.
Happy Logstashing!
</body></html>

filter.jpg

Introducing the de_dot filter

Have you ever wondered if Logstash was sending data to your outputs? There's a brand new way to check if Logstash has a "pulse." Introducing the <a href="http://github.com/logstash-plugins/logstash-input-heartbeat">heartbeat input plugin!</a> It’s bundled with Logstash 1.5 so you can start using it immediately!<h2>Why?</h2>Logstash currently has a single pipeline. All events generated by inputs travel through the filter block, and then out of Logstash through the output block. Even if you have multiple outputs and are separating events using conditionals all events pass through this single pipeline. If any one of your outputs backs up, the entire pipeline stops flowing. The heartbeat plugin takes advantage of this to help you know when the flow of events slows, or stops altogether.<h2>How?</h2>The heartbeat plugin sends a message at a definable interval. Here are the options available for the message configuration parameter:<ul><li>Any string value: The message field will contain the specified string value. &nbsp;If unset, the message field will contain the string value ok</li><li>epoch: Rather than a message field, this will result in a clock field which will contain the current epoch timestamp (UTC). If you are unfamiliar with this, it means the number of seconds elapsed since Jan 1, 1970.</li><li>sequence: Rather than a message field, this will result in a clock field which will contain a number. At start time, the sequence starts at zero and will increment each time your specified interval time has elapsed. Note that this means that if you restart Logstash, the counter resets to zero again.</li></ul><h2>Examples</h2>Be sure to assign a type to your heartbeat events. This will make it possible to conditionally act on these events later on.<h3>"ok" Message</h3>Perhaps you only want to know that Logstash is still sending messages. Your monitoring system can interpret an "ok" received within a time window as an indicator that everything is working. Your monitoring system would be responsible for tracking the time between "ok" messages.I can send the default "ok" message every 10 seconds like this:<pre class="prettyprint">input { heartbeat { interval =&gt; 10 type =&gt; "heartbeat" } # ... other input blocks go here } </pre> The events would look like this:<pre class="prettyprint">{"message":"ok","host":"example.com","@version":"1","@timestamp":"2015-03-18T17:05:24.696Z","type":"heartbeat"} {"message":"ok","host":"example.com","@version":"1","@timestamp":"2015-03-18T17:05:34.696Z","type":"heartbeat"} {"message":"ok","host":"example.com","@version":"1","@timestamp":"2015-03-18T17:05:44.695Z","type":"heartbeat"} </pre><h3>Epoch timestamp</h3>Perhaps your monitoring system uses unix timestamps to track event timing (like Zabbix, for example). If so, you can use the epoch timestamp in the clock field to calculate the difference between "now" and when Logstash generated the heartbeat event. You can calculate lag in this way. This may be especially useful if you inject the heartbeat before events go into a broker, or buffering system, like Redis, RabbitMQ, or Kafka. If the buffer begins to fill up, the time difference will become immediately apparent. You could use this to track the elapsed time--from event creation, to indexing--for your entire Logstash pipeline. This example will send the epoch timestamp in the clock field: <pre class="prettyprint">input { heartbeat { message =&gt; "epoch" interval =&gt; 10 type =&gt; "heartbeat" } # ... other input blocks go here }</pre> The events would look like this:<pre class="prettyprint">{"clock":1426698365,"host":"example.com","@version":"1","@timestamp":"2015-03-18T17:06:05.360Z","type":"heartbeat"} {"clock":1426698375,"host":"example.com","@version":"1","@timestamp":"2015-03-18T17:06:15.364Z","type":"heartbeat"} {"clock":1426698385,"host":"example.com","@version":"1","@timestamp":"2015-03-18T17:06:25.359Z","type":"heartbeat"} </pre><h3>Sequence of numbers</h3>This example makes it easy to immediately check if new events are occurring because the clock will continuously increase.<pre class="prettyprint">input { heartbeat { message =&gt; "sequence" interval =&gt; 10 type =&gt; "heartbeat" } # ... other input blocks go here }</pre> The events would look like this:<pre class="prettyprint">{"clock":1,"host":"example.com","@version":"1","@timestamp":"2015-03-18T17:08:13.024Z","type":"heartbeat"} {"clock":2,"host":"example.com","@version":"1","@timestamp":"2015-03-18T17:08:23.027Z","type":"heartbeat"} {"clock":3,"host":"example.com","@version":"1","@timestamp":"2015-03-18T17:08:33.029Z","type":"heartbeat"}</pre><h3>Output</h3>Now let's add a conditional to send this to our monitoring system, and not to our other outputs:<pre class="prettyprint">output { if [type] == "heartbeat" { # Define the output block for your monitoring system here } else { # ... other output blocks go here } } </pre> Of course, if you do want your heartbeat messages to be indexed alongside your log data, you are free to do so.<h2>Conclusion</h2>The new heartbeat plugin provides a simple, but effective way to monitor the&nbsp;availability of your Logstash instances right now. We have big plans for the future, though.In the future we plan to have a full API, complete with visibility into the pipeline, plugin performance, queue status, event throughput and so much more. &nbsp;We are super excited to bring these improvements to you!Happy Logstashing!

heartbeat-pulse-monitor.jpg

heartbeat-pulse-monitor-thumb.jpg

How to check Logstash's pulse

<article>
Logstash is an event processing pipeline, which features a rich ecosystem of plugins, allowing users to push data in, manipulate it, and then send it to various backends.

One of those plugins is <a href="/guide/en/logstash/current/plugins-filters-grok.html">grok</a>. Grok is currently the best for Logstash to parse unstructured log data and structure it so it can be best queried by Elasticsearch. <a href="/guide/en/logstash/current/plugins-filters-mutate.html">Mutate</a>, another popular plugin, allows the user to manipulate Logstash event data in many useful ways.

<h2>Why type my data?</h2>
Elasticsearch is so much more powerful than just its full-text search use case. It can also calculate <a href="/guide/en/elasticsearch/reference/1.4/search-aggregations-metrics-stats-aggregation.html">a variety of statistics</a> on numerical data in near real-time. <a href="/kibana">Kibana</a> can then take the results of these calculations and plot them in charts and dashboards. If the data is not typed properly, Elasticsearch will be unable to perform these calculations. Typing your data properly takes a little thought and effort, but yields amazing results!

<h3>JSON, Strings, and Numbers</h3>
All documents sent to Elasticsearch must be in JSON format, but Logstash takes care of transforming your data into JSON documents for you. When Elasticsearch receives a JSON document, it will do its best to try to guess what type of data each field contains. The list of core types and a thorough description of each one can be found <a href="/guide/en/elasticsearch/reference/current/mapping.html#mapping-type">here</a>.

If I were to send the following JSON document to Elasticsearch (patterned as a Logstash event):

<pre class="prettyprint lang-js">{
 "@timestamp": "2014-10-07T20:11:45.000Z",
 "@version": "1",
 "count": 2048,
 "average": 1523.33,
 "host": "elasticsearch.com"
}
</pre>
We can see 5 fields: <code>@timestamp</code>, <code>@version</code>, <code>count</code>, <code>average</code>, and <code>host</code>.

In the JSON I am sending, <code>@timestamp</code> and <code>host</code> are string fields, <code>count</code> and <code>average</code> are numeric fields, but <code>@version</code> is a strange hybrid. The value of <code>@version</code> is a number, but because it is inside double quotes <code>"</code> it means that it is considered a string within this JSON document.

If I were to send this document to Elasticsearch to be indexed:

<pre class="prettyprint lang-js">curl -XPOST localhost:9200/Logstash-2014.10.07/logs/1 -d '
{
 "@timestamp": "2014-10-07T20:11:45.000Z",
 "@version": "1",
 "count": 2048,
 "average": 1523.33,
 "host": "elasticsearch.com"
}
</pre>
and then check the mapping…

<pre class="prettyprint lang-js">curl localhost:9200/Logstash-2014.10.07/_mapping?pretty
{
 "Logstash-2014.10.07" : {
 "mappings" : {
 "logs" : {
 "properties" : {
 "@timestamp" : {
 "type" : "date",
 "format" : "dateOptionalTime"
 },
 "@version" : {
 "type" : "string"
 },
 "average" : {
 "type" : "double"
 },
 "count" : {
 "type" : "long"
 },
 "host" : {
 "type" : "string"
 }
 }
 }
 }
 }
}
</pre>
<code>average</code> is type <code>double</code>, <code>count</code> is type <code>long</code>, Elasticsearch successfully guessed that <code>@timestamp</code> is a date field, and <code>host</code> is type <code>string</code>.

Because <code>@version</code> was sent as a string, it remains type <code>string</code>.

It is important to understand that unless you type (or cast) your data accordingly, Logstash sends all values to Elasticsearch as strings.

<h2>Coercing a data type in Logstash</h2>
There are currently two ways to coerce Logstash to send numeric values: grok and mutate. You can also coerce types at the Elasticsearch level.

<h3>grok</h3>
Using grok to parse unstructured data into structured data can be a daunting task on its own. It can be downright confusing to tokenize numeric data into a field (let’s call it <code>num</code>) with the grok pattern <code>%{NUMBER:num}</code> only to find that Elasticsearch thinks <code>num</code> is a string field. Part of the confusion stems from the fact that grok treats the source data as a string since it is a regular expression engine. Because grok sees whatever is input as a string, without further information the output will also be a string. What we need is a way to tell grok and Logstash that the resulting value should be numeric.

<a href="/guide/en/logstash/current/plugins-filters-grok.html">The official documentation for grok</a> explains it like this:

<blockquote>
	Optionally you can add a data type conversion to your grok pattern. By default all semantics are saved as strings. If you wish to convert a semantic’s data type, for example change a string to an integer then suffix it with the target data type. For example <code>%{NUMBER:num:int}</code> which converts the <code>num</code> semantic from a string to an integer. Currently the only supported conversions are <code>int</code> and <code>float</code>.
</blockquote>
So, if I add <code>:int</code> into my grok field definition, suddenly my value will be cast as an integer. Caveat: The <code>NUMBER</code> grok pattern will also detect numbers with decimal values. If you cast a number with a decimal as <code>:int</code>, it will truncate the decimal value leaving only the whole number portion.

In that case, the number can be cast as <code>:float</code>, meaning floating-point value.

<h3>mutate</h3>
Like grok, mutate also allows you to type your fields. The mutate filter currently allows you to convert a field into an <code>integer</code>, a <code>float</code>, or a <code>string</code>. Using our previous example of field <code>num</code>, the configuration would look something like this:

<pre class="prettyprint lang-js">filter {
 mutate {
 convert =&gt; { "num" =&gt; "integer" }
 }
}
</pre>
As with grok, converting a value with a decimal to type <code>integer</code> will truncate the decimal portion, leaving only the whole number.

<h3>Checking our work</h3>
If I were to check the mapping in Elasticsearch after typing the field <code>num</code> to an integer, I would find something unusual.

<pre class="prettyprint lang-js">curl localhost:9200/Logstash-2014.10.07/_mapping?pretty
...
 "num" : {
 "type" : "long"
 }
...
</pre>
It says that it’s type <code>long</code>. But I typed it as <code>integer</code>! What happened?

The answer is in the JSON. Logstash can only send JSON to Elasticsearch. If you recall our first example in this post, we sent a document with the field <code>count</code> with a value of <code>2048</code>. Elasticsearch mapped that as type <code>long</code> also. We did the same in this example, but the field name was <code>num</code>. Unless you tell Elasticsearch what to expect, it can only make an educated guess how to interpret the JSON you send it.

<h2>Next steps: Typing in Elasticsearch</h2>
So how do we do typing in Elasticsearch? That will have to wait until the next issue of Little Logstash Lessons.

I hope you enjoyed today’s Little Logstash Lesson, where we learned how to re-type our numeric values inside Logstash! Stay tuned for our <a href="https://www.elastic.co/blog/logstash_lesson_elasticsearch_mapping" target="_blank">next lesson where we’ll learn how to take our typing to the next level in Elasticsearch.</a>

<aside id="post_tags">
</aside>
</article>

blackboard.jpg

Little Logstash Lessons - Part I: Using grok and mutate to type your data

<html><head></head><body>NOTE:This article now contains outdated information. Please reference our <a href="https://www.elastic.co/guide/en/elasticsearch/client/curator/current/index.html" rel="nofollow" target="_blank" data-saferedirecturl="https://www.google.com/url?hl=en&amp;q=https://www.elastic.co/guide/index.html&amp;source=gmail&amp;ust=1469220551020000&amp;usg=AFQjCNFp624gIIfRUwf0yPsUplf58IPh3Q">docs</a>, peruse our <a href="https://www.elastic.co/blog" rel="nofollow" target="_blank" data-saferedirecturl="https://www.google.com/url?hl=en&amp;q=https://www.elastic.co/blog&amp;source=gmail&amp;ust=1469220551020000&amp;usg=AFQjCNFkv2b4rzkS-_7DMuXXIydT2oVtXw">latest blogs</a>, and visit our <a href="https://discuss.elastic.co/c/elasticsearch" rel="nofollow" target="_blank" data-saferedirecturl="https://www.google.com/url?hl=en&amp;q=https://discuss.elastic.co/&amp;source=gmail&amp;ust=1469220551020000&amp;usg=AFQjCNFqokXwz24txOFrDbkZX3z-7vV5cg">forums</a> for the latest and greatest. Thank you.<h2>background</h2>A few years ago, I was managing an Elasticsearch, Logstash and Kibana (ELK) stack and needed a way to automatically clean up indices older than 30 days. After reading the API documentation and getting some help from the community in the #logstash and #elasticsearch IRC channels, I realized that this was fairly easy to set up with simple scripting and cron.
<pre class="prettyprint lang-bash">curl -XDELETE &apos;localhost:9200/logstash-2014.01.01?pretty&apos;
</pre>Sure, this works, and it&#x2019;s not terribly hard to generate dates, but I wanted something a bit more elegant.
<h2>In the beginning&#x2026;</h2>So, I started writing a script in python. It did the job with a cleaner command line and a single target number of days to keep, so I shared it with the larger community. Others polished it up and added new functionality. I wrote another script that also allowed me to optimize &#x2014; which really just means &#x201C;merge the segments in each shard until no more than n segments exist per shard &#x2014; old indices. These scripts have now been merged and enhanced to become a single, helpful tool to help manage your older indices like the fine works of art they are!
<h2>Introducing Curator</h2>Here are a few of the index operations you can do with Curator:
<ul>
	<li>Delete (by total space consumed or by date)</li>
	<li>Close</li>
	<li>Disable bloom filter cache</li>
	<li>Optimize (Lucene forceMerge)</li>
</ul><h2>Installing Curator</h2>As of this writing, Curator is at release 0.5.1 and works with versions up through 0.90.10. Curator should also should be compatible with Elasticsearch version 1.0 (which is still only at RC1). We&#x2019;ll be testing to ensure compatibility with each release.
It currently resides in a git repository. In the near future, it will be a pip installable package (it&#x2019;s python-based). Don&#x2019;t let that scare you away from using it, though! If you have python and pip installed on your machine, installation is as simple as:
<pre class="prettyprint lang-bash noescape">git clone https://github.com/elasticsearch/curator.git
pip install -r requirements.txt
</pre>After that, you should be able to run this:
<pre class="prettyprint lang-bash noescape">$ ./curator.py -v
curator.py 0.5.1
</pre><h3>How-To and examples</h3>Before we get to the examples, <a href="#arguments">you may want to look over the options</a> in context. The list is long (and included at the end of this post), but illustrative of how much control you have. Please note where defaults have been listed. If you like them, you do not need to specify those flags.
Now let&#x2019;s go through some simple examples to illustrate how Curator can make your ELK stack better, and even more responsive.
<h3>Delete</h3>Let&#x2019;s say you want to keep no more than 90 days of indices. The command is simple:
<pre class="prettyprint lang-bash noescape">$ curator.py --host my-elasticsearch -d 90
</pre>Here the -d specifies the number of days. Simple, right?
<h4>Delete by space</h4>This is a special case where you might want to delete indices in excess of some number of gigabytes (starting with the oldest):
<pre class="prettyprint lang-bash noescape">$ curator.py --host my-elasticsearch -C space -g 10024
</pre>Here you see that we specified curation (-C) by space, and a number of (-g) gigabytes (10024, or 10TB). The -g argument will accept a decimal value, e.g. 1.5, 0.5, etc.
Please note that you cannot combine delete by space with any of the other Curator options.
<h3>Close</h3>Closing an index is handled by the <a href="/guide/en/elasticsearch/reference/current/indices-open-close.html" title="Open / Close Index API" target="_blank">Open/Close Index API</a>:
<blockquote>The open and close index APIs allow to you close an index, and later on open it. A closed index has almost no overhead on the cluster (except for maintaining its metadata), and is blocked for read/write operations. A closed index can be opened which will then go through the normal recovery process.</blockquote>Closing an index means it&#x2019;s still there, but not searchable. Why is that useful?
Imagine that you have an obligation to keep indices for 90 days, but rarely if ever do you search through an index over 30 days old. In this case, you can close the indices which will save valuable resources (heap space, in particular). This means your cluster will have more memory for searches and indexing! And if you ever need the data in those indices, you can open them again with an API call and they&#x2019;ll be there again.
In such an event it may be wise to temporarily disable a scheduled execution of Curator to prevent your now-open indices from being closed again, prematurely.
<pre class="prettyprint lang-bash noescape">$ curator.py --host my-elasticsearch -c 30 -d 90
</pre>This builds on our previous example. This will close indices older than 30 days and delete indices older than 90 days. Still quite simple!
<h3>Disable Bloom Filters</h3><a href="https://github.com/elasticsearch/elasticsearch/issues/4525" title="Allow to enable / disable bloom filter loading on an index" target="_blank">This is a brand new feature not usable unless you&#x2019;re running Elasticsearch version 0.90.9 or better.</a>
Don&#x2019;t worry, the script checks to make sure your version is sufficient before attempting the operation.
What is a bloom filter? Why would I want it disabled?
The bloom filter is resources allocated to speed indexing operations. With time-series data this is still useful while indexing. Your index is probably not indexing new data 2 days after the date has rolled over. The bloom filter is then holding on to resources that the index no longer requires. With Curator we can free those resources!
<pre class="prettyprint lang-bash noescape">$ curator.py --host my-elasticsearch -b 2 -c 30 -d 90
</pre>Now we&#x2019;re freeing those bloom filter resources on indices older than 2 days (you could also do 1), closing indices older than 30 days and deleting indices older than 90 days.
<h3>Optimize, or rather forceMerge</h3>It&#x2019;s important to understand that when you see <a href="/guide/en/elasticsearch/reference/current/indices-optimize.html" title="Optimize" target="_blank">optimize</a> in the Elasticsearch API you&#x2019;re not seeing a command you need to run on a live index, or repeatedly on a &#x201C;cold&#x201D; index (meaning one that is no longer actively indexing). In fact, <a href="http://blog.trifork.com/2011/11/21/simon-says-optimize-is-bad-for-you/" title="SIMON SAYS: OPTIMIZE IS BAD FOR YOU&#x2026;" target="_blank">optimize was renamed to forceMerge inside Lucene</a> so people would not make an optimize call thinking it was improving their index. Merging segments in Elasticsearch can bring benefits, but understand the cost before you start optimizing all of your cold indices.
A forceMerge operation will try to reduce the segment count in each shard in your indices. Since each segment has an overhead, more segments means more resources used. That&#x2019;s good, right? Less resources?
It can be, but the amount of disk and network I/O to perform a significant merge operation can take a toll on your disks and your cluster&#x2019;s normal write operations. My advice here is to consider carefully if you need this. It can help make searches faster (by a few percentage points), and reduce resources. It can also make cluster recover faster as there are fewer segments to manage. It can take a very long time to optimize a single index &#x2014; perhaps an hour or more. As a result, increasing the timeout may be necessary, with a recommended value of no less than 3600 seconds (one hour). As with the warning labels on cleaning bottles, &#x201C;test in an inconspicuous place before using,&#x201D; you should test this during a time of reduced disk I/O and see if it improves operations and resources in a way that makes sense for your cluster and use-case. The default is to merge to 2 segments per shard, but you can override this default with the <code>--max_num_segments</code> flag.
Building on our previous example yet again,
<pre class="prettyprint lang-bash noescape">$ curator.py --host my-elasticsearch -b 2 -o 2 -c 30 -d 90
</pre>This will disable bloom filters for indices older than 2 days, &#x201C;optimize&#x201D; indices older than 2 days, close indices older than 30 days and delete indices older than 90 days.
<h3>Order of operations</h3>The script enforces the following order to prevent operations from colliding. Why optimize an index that&#x2019;s closed? Why close an index that&#x2019;s slated for deletion?
<ol>
	<li>Delete (by space or time)</li>
	<li>Close</li>
	<li>Disable bloom filters</li>
	<li>Optimize</li>
</ol><h3>Usage considerations</h3>In the most recent example, we performed all three operations in one command, but you may not want them all run serially.
<pre class="prettyprint lang-bash noescape">$ curator.py --host my-elasticsearch -b 2 -o 2 -c 30 -d 90
</pre>&#x2026;is functionally equivalent to&#x2026;
<pre class="prettyprint lang-bash noescape">$ curator.py --host my-elasticsearch -d 90
$ curator.py --host my-elasticsearch -c 30 
$ curator.py --host my-elasticsearch -b 2
$ curator.py --host my-elasticsearch -o 2
</pre>You could easily run these commands at different times, or with extra flags (especially the optimize run, which should have something like <code>--timeout 3600</code>).
You may also have indices with prefixes other than the default logstash- to crawl:
<pre class="prettyprint lang-bash noescape">$ curator.py --host my-elasticsearch --prefix logstash- -d 30
$ curator.py --host my-elasticsearch --prefix othername- -d 30
</pre><h2>Conclusion</h2>Curator helps you manage your time-series indices retention policies. With an abundance of configuration options you can easily manage your indices&#x2013;whether you have 1 node, or a hundred or more in your cluster. Feedback and contributions are welcome at <a href="https://github.com/elasticsearch/curator" title="Curator: Tending your time-series indices" target="_blank">https://github.com/elasticsearch/curator</a>!
<h5><a name="arguments">Help output</a> (all arguments and options displayed)</h5><pre class="prettyprint lang-sh">$ curator.py -h
usage: curator.py [-h] [-v] [--host HOST] [--port PORT] [-t TIMEOUT]
 [-p PREFIX] [-s SEPARATOR] [-C CURATION_STYLE]
 [-T TIME_UNIT] [-d DELETE_OLDER] [-c CLOSE_OLDER]
 [-b BLOOM_OLDER] [-g DISK_SPACE]
 [--max_num_segments MAX_NUM_SEGMENTS] [-o OPTIMIZE] [-n]
 [-D] [-l LOG_FILE]
Curator for Elasticsearch indices. Can delete (by space or time), close,
disable bloom filters and optimize (forceMerge) your indices.
optional arguments:
 -h, --help show this help message and exit
 -v, --version show program version number and exit
 --host HOST Elasticsearch host. Default: localhost
 --port PORT Elasticsearch port. Default: 9200
 -t TIMEOUT, --timeout TIMEOUT
 Elasticsearch timeout. Default: 30
 -p PREFIX, --prefix PREFIX
 Prefix for the indices. Indices that do not have this
 prefix are skipped. Default: logstash-
 -s SEPARATOR, --separator SEPARATOR
 Time unit separator. Default: .
 -C CURATION_STYLE, --curation-style CURATION_STYLE
 Curate indices by [time, space] Default: time
 -T TIME_UNIT, --time-unit TIME_UNIT
 Unit of time to reckon by: [days, hours] Default: days
 -d DELETE_OLDER, --delete DELETE_OLDER
 Delete indices older than n TIME_UNITs.
 -c CLOSE_OLDER, --close CLOSE_OLDER
 Close indices older than n TIME_UNITs.
 -b BLOOM_OLDER, --bloom BLOOM_OLDER
 Disable bloom filter for indices older than n
 TIME_UNITs.
 -g DISK_SPACE, --disk-space DISK_SPACE
 Delete indices beyond n GIGABYTES.
 --max_num_segments MAX_NUM_SEGMENTS
 Maximum number of segments, post-optimize. Default: 2
 -o OPTIMIZE, --optimize OPTIMIZE
 Optimize (Lucene forceMerge) indices older than n
 TIME_UNITs. Must increase timeout to stay connected
 throughout optimize operation, recommend no less than
 3600.
 -n, --dry-run If true, does not perform any changes to the
 Elasticsearch indices.
 -D, --debug Debug mode
 -l LOG_FILE, --logfile LOG_FILE
 log file
</pre></body></html>

Curator: Tending your time-series indices

Start free trial

Blog Footer CTA

Sign up for Elastic Cloud free trial

aaron-mildenstein.jpg

Aaron Mildenstein

The content on this page is not available in the selected language. As Elastic grows globally, we continue to support content in multiple languages.

The content on this page is not available in the selected language.

El contenido de esta página no está disponible en el idioma seleccionado. Elastic está trabajando para garantizar que el contenido esté disponible en varios idiomas. Gracias por tu paciencia mientras trabajamos.

By submitting you acknowledge that you've read and agree to our <a href="/legal/elastic-cloud-account-terms" target="_self">Terms of Service</a>, and you agree to receive the <a href="/legal/privacy-statement#how-we-use-the-information" target="_self">selected communications</a> at the contact details you provided above. See <a href="/legal/privacy-statement/" target="_self">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.

Communication Preferences - ONLY for /communication-preferences

Al enviar, confirma que ha leído y acepta nuestras <a href="/es/legal/elastic-cloud-account-terms" target="_self">Condiciones de servicio</a>, y que acepta recibir las <a href="/es/legal/privacy-statement#how-we-use-the-information" target="_self">comunicaciones seleccionadas</a> en los detalles de contacto que proporcionó anteriormente. Consulte la <a href="/es/legal/privacy-statement/" target="_self">Declaración de privacidad de Elastic</a> para obtener más información o para cancelar su suscripción en cualquier momento.

By submitting you acknowledge that you've read and agree to our <a href="/legal/serverless-preview-terms" target="_blank">Terms of Service</a>, and that Elastic may <a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">contact you</a> about our related products and services, using the details you provide above. See <a href="/legal/privacy-statement/" target="_blank">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.

Serverless GDPR Text

Al enviar, confirma que ha leído y acepta nuestras <a href="/es/legal/serverless-preview-terms" target="_self">Condiciones de servicio</a>, y que Elastic puede <a href="/es/legal/privacy-statement#how-we-use-the-information" target="_self">ponerse en contacto con usted</a> acerca de nuestros productos y servicios relacionados. Consulte la <a href="/es/legal/privacy-statement/" target="_self">Declaración de privacidad de Elastic</a> para obtener más información o para cancelar su suscripción en cualquier momento.

By submitting you agree to <a href="/agreements/global/cloud-services-monthly" rel="nofollow">Elastic Cloud Terms of Service</a>. Your personal data will be processed in accordance with <a href="/legal/privacy-statement">Elastic's Privacy Statement</a>.

Cloud Trial GDPR Text

Al enviar esta información, aceptas los <a href="/es/agreements/global/cloud-services-monthly" rel="nofollow">Términos de servicio de Elastic</a>. Los datos personales se procesarán de conformidad con la <a href="/es/legal/privacy-statement">declaración de privacidad de Elastic</a>.

By submitting you acknowledge that you've read and agree to our <a href="/legal/terms-of-use" target="_blank">Terms of Service</a>, and that Elastic may <a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">contact you</a> about our related products and services, using the details you provide above. See <a href="/legal/privacy-statement/" target="_blank">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.

Newsletter GDPR Text

Al enviar, confirma que ha leído y acepta nuestras <a href="/es/legal/terms-of-use" target="_self">Condiciones de servicio</a>, y que Elastic puede <a href="/es/legal/privacy-statement#how-we-use-the-information" target="_self">ponerse en contacto con usted</a> acerca de nuestros productos y servicios relacionados. Consulte la <a href="/es/legal/privacy-statement/" target="_self">Declaración de privacidad de Elastic</a> para obtener más información o para cancelar su suscripción en cualquier momento.

Hosted by

All Solutions

Follow us on Youtube

Watch now (no PT)

Articles by

Share by Email

Filters

All (no PT translation)

Contact information

See when this webinar starts in my time zone

Ver cuándo comienza este webinar en mi zona horaria

Thank you for your interest!

Register to Attend

Small image for

Register to Watch

Continue reading

Featured webinar

What to explore next...

Reset all

Load more press releases

Sign In to Attend

Share

Upcoming webinar

Learn more

Search Integrations

Share on Reddit

Share on Twitter

Share on Facebook

Load more news

View more posts

On-demand webinar

Follow us on Facebook

Author

Press Release

You'll also receive an email with related content.

You'll also receive an email with related content

Además, te enviaremos contenidos relevantes

Print

Contact Info

Overview

Video for

Read less

Register now

Share this story

See All Posts

See all top stories

View next

Can't make it? Register and we'll send you the recording. You'll also receive an email with related content

¿No puedes asistir? Regístrate y te enviaremos la grabación. También recibirás un correo electrónico con contenido relacionado.

Follow us on LinkedIn

Highlights

Explore similar demos

Follow us on Twitter

Share on LinkedIn

Load More

See more insights

Speakers

Close

Thank you for registering. We will send you a confirmation email soon.

Gracias por registrarte. Te enviaremos un correo electrónico de confirmación pronto.

Author

Articles by Aaron Mildenstein

Announcing Curator 5

​Little Logstash Lessons: Using Logstash to help create an Elasticsearch mapping template

New features in Curator 4.1

Just Enough Redis for Logstash

How To Centralize Logs with Rsyslog, Logstash, and Elasticsearch on Ubuntu 14.04

Introducing the de_dot filter

How to check Logstash's pulse

Little Logstash Lessons - Part I: Using grok and mutate to type your data

Curator: Tending your time-series indices

Sign up for Elastic Cloud free trial

Little Logstash Lessons: Using Logstash to help create an Elasticsearch mapping template