Most Logstash users that want to extract structured fields from an unstructured event data use the popular <a href="https://www.elastic.co/guide/en/logstash/5.0/plugins-filters-grok.html">Grok filter</a>. This filter uses regular expressions under the hood and is versatile to many forms of text. Over time, users tend to use Grok for all of their structured extraction needs. This, in many cases, is analogous to using a sledgehammer to crack a nut. Regular expressions are sledgehammers. They are extremely powerful, but they can be unforgiving -- a poorly built regex can slow down the Logstash pipeline when run on certain lines of text. The pipeline is designed to recover, but the overall performance becomes unpredictable leading to unhappy users.My colleague João Duarte, while <a href="https://www.elastic.co/blog/do-you-grok-grok">investigating</a> the performance characteristics of the Grok Filter noticed that match failures incur increasing performance penalties. Most of these issues boil down to <a href="https://en.wikipedia.org/wiki/ReDoS">ReDOS</a> problems, which essentially means that the regex engine descends into excessive backtracking, leading to a slower Logstash pipeline.Another colleague, <a href="https://github.com/ph">PH Pellerin</a>, has recently re-implemented the Beats Input to handle input data asynchronously that led to an impressive throughput increase. In Logstash processing, the slowest component slows down the entire pipeline, and typically filters are the heavy lifters. I thought it wouldn’t really help to have a couple of speedier inputs with a slow Grok filter for field extraction, and hence <a href="https://www.elastic.co/guide/en/logstash/5.0/plugins-filters-dissect.html">Dissect filter</a> was born.<h2>Design</h2>Essentially, field extraction in Dissect is a kind of 'split' operation and normally a split is done with one delimiter that is expected to be repeated in the source text. In contrast to Grok, which asks you to specify the regex patterns for fields of interest, Dissect asks you to specify the patterns of delimiters between the fields of interest. Dissect queues up the delimiters and searches the source text once from left to right, finding each successive delimiter in the queue. Different search algorithms are used for 1, 2 and 3+ byte delimiters.Dissect is written in Java with a very thin JRuby extension wrapper to expose the Java internals to Logstash's JRuby runtime environment. It also directly uses the rewritten Java Event code, released in Logstash 5.0. This is why Dissect is only compatible with Logstash 5.X.Dissect uses the bytes behind the JRuby UTF-8 Ruby String directly without the expensive conversion to a Java String beforehand. The delimiter searching will only remember where in the source bytes each delimiter was found. The Event field and value setting uses the found indices and lengths only once to create the strings to set into the Event.The format of the patterns of fields and delimiters is similar to Grok:"&lt;%{priority}&gt;%{syslog_timestamp} %{+syslog_timestamp} %{+syslog_timestamp} %{logsource} %{rest}".In the above example, the yellow highlight shows the delimiters and blue highlight the field names. I quickly realised that various field syntaxes were needed to differentiate fields of extraction. There are four at the time of writing: Normal, Skip, Append and Indirect.<h4>Normal field type</h4>This type simply adds the found value to the event using the specified text as the name of the field, e.g. %{priority}.<h4>Skip field type</h4>The pattern format requires you to specify each delimiter by putting them between } and %{ meaning there will be times when you need to specify fields but you don't need the values in the event - it should be ignored or skipped. To do this you can have an empty field e.g. %{} or a named skipped field which is prefixed with a question mark "?", e.g. %{?priority}. You might choose to name your skip fields to make the final dissect pattern more readable.<h4>Append field type</h4>For the same reason - needing to specify all delimiters, you might need to rebuild a split value. Taking the syslog date as an example, if the source text looks like this &lt;1&gt;Oct 26 20:21:22 alpha1 rest... - you will need to specify fields for the month, day and time. Maybe you want fields these as separate in your events but more likely you will want to rebuild these into one value. This is what an Append field does. For example, %{syslog_timestamp} %{+syslog_timestamp} %{+syslog_timestamp} will firstly have the month "Oct" value in the syslog_timestamp field but then the next +syslog_timestamp adds the day "26" value to "Oct" but with its prior delimiter in between - in this case a space, resulting in "Oct 26". This is repeated for last +syslog_timestamp so the final value for the field syslog_timestamp will be "Oct 26 20:21:22".Sometimes, you will want to rebuild the value in a different order - this is supported with an ordinal suffix "/&lt;int&gt;", e.g. %{+syslog_timestamp/1}. There is more info and examples on this in the <a href="https://www.elastic.co/guide/en/logstash/5.0/plugins-filters-dissect.html">docs</a>.<h4>Indirect field type</h4>Use the value of one field to be the key of another. This type is arguably the least useful type and the syntax looks strange because it leverages Skip fields. Please read up more on these in the <a href="https://www.elastic.co/guide/en/logstash/5.0/plugins-filters-dissect.html">docs</a>.<h4>Conversions</h4>To reduce the need to use a Mutate filter to convert an obvious floating point or integer number string into a real Float or Integer value in the event, you can specify a conversion. Please read the <a href="https://www.elastic.co/guide/en/logstash/5.0/plugins-filters-dissect.html">docs</a> for more information on this. This feature can be used on its own without a dissect directive.<h4>Example: Replace a Grok + CSV combo with one Dissect filter</h4>Extract fields for the <a href="https://www.paloaltonetworks.com/documentation/61/pan-os/pan-os/reports-and-logging/syslog-field-descriptions">Palo Alto Networks Firewall PAN OS 6 syslog output</a> for "threat" type messages. See the generator input config below for an example of a log line. It's a Syslog preamble with a CSV body <h5>Logstash Configs</h5> Input<pre class="prettyprint">input { generator { message =&gt; "&lt;1&gt;Oct 16 20:21:22 www1 1,2016/10/16 20:21:20,3,THREAT,SCAN,6,2016/10/16 20:21:20,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54" time =&gt; 120 } } </pre>The generator input has been modified to run for a fixed amount of time rather than a fixed amount of events. Grok + CSV<pre class="prettyprint">filter { if [message] =~ "THREAT," { grok { match =&gt; { "message" =&gt; "\A&lt;%{POSINT:priority}&gt;%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:sysloghost} %{DATA:pan_fut_use_01},%{DATA:pan_rec_time},%{DATA:pan_serial_number},%{DATA:pan_type},%{GREEDYDATA:message}\z" } overwrite =&gt; [ "message" ] } csv { separator =&gt; "," columns =&gt; ["pan_subtype", "pan_fut_use_02", "pan_gen_time", "pan_src_ip", "pan_dst_ip", "pan_nat_src_ip", "pan_nat_dst_ip", "pan_rule_name", "pan_src_user", "pan_dst_user", "pan_app", "pan_vsys", "pan_src_zone", "pan_dst_zone", "pan_ingress_intf", "pan_egress_intf", "pan_log_fwd_profile", "pan_fut_use_03", "pan_session_id", "pan_repeat_cnt", "pan_src_port", "pan_dst_port", "pan_nat_src_port", "pan_nat_dst_port", "pan_flags", "pan_prot", "pan_action", "pan_misc", "pan_threat_id", "pan_cat", "pan_severity", "pan_direction", "pan_seq_number", "pan_action_flags", "pan_src_location", "pan_dst_location", "pan_content_type", "pan_pcap_id", "pan_filedigest", "pan_cloud", "pan_user_agent", "pan_file_type", "pan_xff", "pan_referer", "pan_sender", "pan_subject", "pan_recipient", "pan_report_id", "pan_threat_extra1", "pan_threat_extra2"] } } } </pre>Dissect<pre class="prettyprint">filter { if [message] =~ "THREAT," { dissect { mapping =&gt; { message =&gt; "&lt;%{priority}&gt;%{syslog_timestamp} %{+syslog_timestamp} %{+syslog_timestamp} %{logsource} %{pan_fut_use_01},%{pan_rec_time},%{pan_serial_number},%{pan_type},%{pan_subtype},%{pan_fut_use_02},%{pan_gen_time},%{pan_src_ip},%{pan_dst_ip},%{pan_nat_src_ip},%{pan_nat_dst_ip},%{pan_rule_name},%{pan_src_user},%{pan_dst_user},%{pan_app},%{pan_vsys},%{pan_src_zone},%{pan_dst_zone},%{pan_ingress_intf},%{pan_egress_intf},%{pan_log_fwd_profile},%{pan_fut_use_03},%{pan_session_id},%{pan_repeat_cnt},%{pan_src_port},%{pan_dst_port},%{pan_nat_src_port},%{pan_nat_dst_port},%{pan_flags},%{pan_prot},%{pan_action},%{pan_misc},%{pan_threat_id},%{pan_cat},%{pan_severity},%{pan_direction},%{pan_seq_number},%{pan_action_flags},%{pan_src_location},%{pan_dst_location},%{pan_content_type},%{pan_pcap_id},%{pan_filedigest},%{pan_cloud},%{pan_user_agent},%{pan_file_type},%{pan_xff},%{pan_referer},%{pan_sender},%{pan_subject},%{pan_recipient},%{pan_report_id},%{pan_anymore}" } } } } </pre>Output<pre class="prettyprint">output { counter { warmup =&gt; 60 } } </pre>The output is a new output that I built - it captures the number of events and the start time of receiving the first event and the end time of last event and prints out the event count, duration, events per second and the microseconds per event. It is written to be thread-safe and be as lightweight as possible. A warmup delay before data capture of 60 seconds was specified to allow JRuby and the JVM to stabilise. <h2>Performance</h2>In these configurations, maximum throughput is when the filter/output, while still doing some work, takes events as fast as the input can generate them. Please note that this applies to the synchronous queue that sits between the inputs and the filter/outputs. The synchronous operation means that the single generator input thread must wait until a filter/output worker thread takes the event it is offering. In the near future we will release an asynchronous queue where this lock-step synchronicity will no longer apply. The filter sections were run on on Logstash 5.0.<h5>Events per second</h5><table><thead><tr style="background-color: rgb(242, 242, 242);"><td>Filters \ Workers </td><td>1 </td><td>2 </td><td>3 </td><td>4 </td><td>5 </td><td>6 </td></tr></thead><caption align="bottom">Table 1: Events per second for various filters with 1 to 6 workers run for 2 minutes with a 1 minute warmup</caption><tbody><tr><td>Grok + CSV </td><td>5003 </td><td>8769 </td><td>12697 </td><td>15615 </td><td>12467 </td><td>12895 </td></tr><tr style="background-color: rgb(242, 242, 242);"><td>CSV </td><td>7351 </td><td>12719 </td><td>18921 </td><td>21492 </td><td>21053 </td><td>16549 </td></tr><tr><td>Grok </td><td>14255 </td><td>27263 </td><td>40241 </td><td>45977 </td><td>46104 </td><td>46882 </td></tr><tr style="background-color: rgb(242, 242, 242);"><td>Dissect </td><td>16396 </td><td>32510 </td><td>50025 </td><td>53619 </td><td>53050 </td><td>52493 </td></tr><tr><td>Date </td><td>19759 </td><td>40290 </td><td>62775 </td><td>65445 </td><td>62854 </td><td>59488 </td></tr><tr style="background-color: rgb(242, 242, 242);"><td>Mutate (rename) </td><td>29248 </td><td>59630 </td><td>79114 </td><td>83195 </td><td>79745 </td><td>82988 </td></tr><tr><td>Noop </td><td>32031 </td><td>65574 </td><td>80645 </td><td>85616 </td><td>85397 </td><td>86881</td></tr></tbody></table> <table><caption align="bottom">Chart 1: A plot of average time in microseconds taken to process an event for various filters with 1 to 6 workers</caption><tbody><tr><td><img src="https://api.contentstack.io/v2/uploads/582ece22f04b53f6188f5461/download?uid=blt0ee3856bac533b98" data-sys-asset-uid="blt0ee3856bac533b98" alt="grokcsv-vs-dissect-perf.png" style="font-family: Arial, Helvetica, Verdana, Tahoma, sans-serif;font-size: 14px;color: rgb(116, 133, 144);float: left;margin: 0px;text-align: left;"/> </td></tr></tbody></table> <h5>Comparison</h5><table><thead><tr style="background-color: rgb(242, 242, 242);"><td>Filters \ Workers </td><td>1 </td><td>2 </td><td>3 </td><td>4 </td><td>5 </td><td>6 </td></tr></thead><caption align="bottom">Table 2: A closer look at the time taken for each approach to extract fields for the Palo Alto device syslog logs with 1 to 6 workers</caption><tbody><tr><td>Grok + CSV </td><td>199.89 </td><td>114.04 </td><td>78.76 </td><td>64.04 </td><td>80.21 </td><td>77.55 </td></tr><tr style="background-color: rgb(242, 242, 242);"><td>Dissect </td><td>60.99 </td><td>30.76 </td><td>19.99 </td><td>18.65 </td><td>18.85 </td><td>19.05 </td></tr><tr><td>Noop </td><td>31.22 </td><td>15.25 </td><td>12.40 </td><td>11.68 </td><td>11.71 </td><td>11.51</td></tr></tbody></table>From Table 2, it can be seen that Dissect out-performs Grok + CSV by some margin but with less certainty. See the important note below.<h5>Breakdown </h5><table><thead><tr style="background-color: rgb(242, 242, 242);"><td>Filters \ Workers </td><td>1 </td><td>2 </td><td>3 </td><td>4 </td><td>5 </td><td>6 </td></tr></thead><caption align="bottom">Table 3: A breakdown of time taken for CSV and Grok individually with 1 to 6 workers</caption><tbody><tr><td>CSV </td><td>136.03 </td><td>78.62 </td><td>52.85 </td><td>46.53 </td><td>47.50 </td><td>60.43 </td></tr><tr style="background-color: rgb(242, 242, 242);"><td>Grok </td><td>70.15 </td><td>36.68 </td><td>24.85 </td><td>21.75 </td><td>21.69 </td><td>21.33</td></tr></tbody></table>From the above tables and chart one can see that the CSV filter is responsible for the lower performance of the Grok + CSV configuration and that Grok when configured optimally, performs just fine for this data set. <h5>Other filters </h5><table><thead><tr style="background-color: rgb(242, 242, 242);"><td>Filters \ Workers </td><td>1 </td><td>2 </td><td>3 </td><td>4 </td><td>5 </td><td>6 </td></tr></thead><caption align="bottom">Table 4: Time taken for Date and Mutate with one rename directive with 1 to 6 workers</caption><tbody><tr><td>Date </td><td>50.61 </td><td>24.82 </td><td>15.93 </td><td>15.28 </td><td>15.91 </td><td>16.81 </td></tr><tr style="background-color: rgb(242, 242, 242);"><td>Mutate (rename) </td><td>34.19 </td><td>16.77 </td><td>12.64 </td><td>12.02 </td><td>12.54 </td><td>12.05</td></tr></tbody></table>Practical configurations might use other filters like the Mutate filter and the Date filter. Tables 3 and 4 show performance for these filters so you can get a feel for how latencies might build on each other. Also note that the performance for the Date filter above is for a single pattern that always matches and that latency will increase with multiple patterns when some dates in your data do not match the first pattern.Important: Be sure to guard a Dissect filter with an if field regex match condition to apply the correct dissection to the incoming text. It is more performant than two partial dissections with an “if field equals value” from a previous dissection. Without a conditional guard, Dissect will try to apply a mapping to a field and it may add some fields to an event but they would probably be not what was intended. Grok on the other hand inherently matches or fails. You should therefore be aware that, by using Dissect, you get better performance but less certainty of correct field extraction.<h4>Is Dissect a replacement for Grok?</h4>Yes and no. In some cases yes, but remember that Dissect is not as flexible as Grok. In cases where Dissect cannot entirely solve the problem, it can be used first to extract the more predictable fields and then the remaining irregular text is passed along to the Grok filter. As the reduced Grok patterns are more focussed and are being applied to smaller texts, Grok’s performance should be less variable.<h4>Is Dissect a replacement for CSV?</h4>Yes if performance is paramount. However, you may find the config syntax of the CSV filter more intuitive. In the near future, we will leverage some of the techniques from Dissect in the CSV filter to put its performance on a par with Dissect.Dissect is a new filter plugin for Logstash. It is performant enough to keep up with the fastest inputs and outputs. If you are using Logstash 5.X or planning to, evaluate Dissect to see for yourself whether it can improve your throughput and simplify your configs. You can simply install dissect by using bin/logstash-plugin install logstash-filter-dissect. We love feedback from our users, so please reach out to us on <a href="https://twitter.com/elastic">twitter</a> or on our <a href="https://discuss.elastic.co/c/logstash">forums</a>. Happy dissecting!Image - Chainsaw Art, by Arboriculturalist and chainsaw carver Jonny Stableford (JonnyChainsaw) By Mike Smith [CC BY 2.0 (http://creativecommons.org/licenses/by/2.0)], via Wikimedia Commons

Jonnychainsaw_during_a_chainsaw_art_demonstration_in_Scotland_big.jpg

Jonnychainsaw_during_a_chainsaw_art_demonstration_in_Scotland_23.jpg

Logstash Dude, where's my chainsaw? I need to dissect my logs

Improve the performance of the Redis input plugin with these tips and details.

Message queues are used in Logstash deployments to <a href="https://www.elastic.co/guide/en/logstash/current/deploying-and-scaling.html#deploying-message-queueing" target="_blank">control</a> a surge in events which may lead to slowdown in Elasticsearch or other downstream components. Redis is one of the technologies in use as a message queue. Due to its speed, ease of use, and low resource requirements it is also one of the most popular ways as well.
That&nbsp;is probably why it has been around for a very, very long time—since Logstash version 1.0.4, released more than five years ago!
With all of the recent changes to improve the Logstash pipeline—as well as improvements to the Redis plugin itself—you may not be getting all the performance the Logstash Redis input plugin is capable of delivering.
Logstash attempts to detect the number of cores in your system and set the number of pipeline workers to match. But pipeline workers only matter for filter and output plugins. &nbsp;It has no effect on input plugins. If you haven’t set `threads` in the Redis input plugin, you are only ingesting a fraction of what you could.
The test setup included the most recent stable version of Logstash (2.3.4) with Redis 3.0.7&nbsp;running on the same machine, which is a 2015 MacBook Pro, for reference. The Logstash configuration used is
<pre>input { 
 redis {
 host =&gt; "127.0.0.1"
 data_type =&gt; "list"
 key =&gt; "redis_test"
 # batch_count =&gt; 1
 # threads =&gt; 1
 }
}
output { stdout { codec =&gt; dots } }
</pre>Run this with bin/logstash -f some_config_file.config | pv -Wart &gt; dev/null
This test setup is only designed to test for maximum throughput. Actual performance numbers will vary widely depending on what other plugins are used.
By iterating through a few options for only the threads&nbsp;setting, and using all other settings at their defaults, we see results like these:
<ul>
	<li>threads =&gt; 1&nbsp;: 37.4K/sec</li>
	<li>threads =&gt; 2&nbsp;: 57K/sec</li>
	<li>threads =&gt; 4&nbsp;: 67K/sec</li>
	<li>threads =&gt; 8&nbsp;: 87.3K/sec</li>
</ul>Let’s run the same tests again, but with batch_count =&gt; 250, and add -b 250&nbsp;to our Logstash command line to take full advantage of that increase:
<ul>
	<li>threads =&gt; 1&nbsp;: 35.2K/sec</li>
	<li>threads =&gt; 2&nbsp;: 48.9K/sec</li>
	<li>threads =&gt; 4&nbsp;: 58.7K/sec</li>
	<li>threads =&gt; 8&nbsp;: 72.5K/sec</li>
</ul><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltc2a4980fa1466723/5f85fd80dcf0e74b7cbf58d0/blog-redis-logstash-1.png" title="Throughput: batch size vs threads" height="371" width="600" "="">
As you can see, increasing the default batch_count&nbsp;(which is 125), to 250 results in a decrease in performance.
It’s been common in the past for users to use a larger batch_count&nbsp;when working with Redis. &nbsp;This is no longer best practices, and will actually reduce performance. &nbsp;This is primarily due to changes in the Logstash pipeline architecture. Even an increase of batch_count&nbsp;to 250 results in a drop of performance from 89.5K/sec down to 72.5K/sec when threads =&gt; 8. The sweet spot for batch_count&nbsp;is right around 125, which is the default pipeline-batch-size&nbsp;(there’s a reason for that!).
<h4>Diving deeper</h4>Recent changes to the Redis input include turning on batch mode by default and using a <a href="https://github.com/logstash-plugins/logstash-input-redis/blob/df5bc905e41d36905363229038fbe263a34d17f6/lib/logstash/inputs/redis.rb#L141" target="_blank">Lua script</a> to execute the batch retrieval commands on the Redis server. Lua scripts also act as a transaction so we can take a batch and shorten the queue in one step. This forces other Redis connections to wait. Note, your Logstash Redis output plugin in the upstream Logstash config is one such connection.
Also, you should be aware that the manner in which events are added to Redis by the upstream Logstash can affect the performance of the downstream Logstash instance. For example, while preparing the results of the tests, we found that querying the list size with LLEN redis_test&nbsp;in redis_cli too often would affect the measurement. Try to understand these factors in your upstream Logstash instance:
<ul>
	<li>the rate that events are generated</li>
	<li>the size of the events, smallest; largest and average</li>
	<li>the batch_events size used the redis output</li>
</ul>You might want to look at using bigger batch_events sizes e.g. 500 on the upstream Redis output - this will buffer the events before calling Redis and doing this may give the downstream Redis input threads time to jump in to pull from Redis. However, if the event generation rate is high then this may not be as helpful - 50K/sec in batches of 500 means a batch buffer fills in 10 milliseconds.
We have seen that the performance of the Lua script suffers proportionately with larger batch sizes. This means that larger batch sizes on the Redis input will affect the performance of both of the Logstash instances on either side of Redis.
Generally, the goal of tuning the Redis input plugin config and the pipeline batch size and worker threads is to make sure that the input is able to keep pace with the filter/output stage. You should know what the average throughput of your filter/output stage is with the defaults. If, for example, your filter/output stage throughput is 35K/sec then there is little need to change the redis input worker count to 8 when one or two workers can fetch sufficient events per second.
One more source of variation is JSON encoding (1), decoding (2) and encoding (3) - yes, three times, 1 in the upstream Redis output, 2 in the downstream Redis input and 3 in the elasticsearch output. Events with big string values do take some time to decode so make sure your testing is using data sets that represent the size and distribution of size as close to your production data as you can make it.
<h4>Scaling throughput using multiple Logstash instance and one Redis server</h4>Many users seek to improve throughput by starting up multiple Logstash instances on different hardware all pointing to the same Redis server and list.
This does work and can work well, but you are advised to apply the same tuning tips to each instance. There is probably not much point in having multiple Redis inputs defined in any one config unless they are connecting to different Redis servers or a different list. Simply tune the threads of the one Redis input. Clearly, having, say, 3 Logstash instances pointing at the same Redis server will create some degree of Redis client connection contention - meaning that you should probably tune one instance and then retune all three when running together.
<h4>Conclusion</h4>In the not too distant future, we will be enhancing the Redis input with a JRuby wrapper around the popular and speedy Jedis Java library. &nbsp;This will bring the added bonus of <a href="https://github.com/logstash-plugins/logstash-input-redis/issues/10" target="_blank">supporting</a> Redis Cluster mode!
In the meanwhile, to give your Redis input performance a boost, try increasing the number of threads used. And be sure to measure and tune accordingly if you use something other than the default batch_count, or have multiple instances of Logstash pointing to a single Redis server.
If you have a question about Redis and Logstash, feel free to open a discussion topic on our <a href="https://discuss.elastic.co/c/logstash" target="_blank">forum</a>. You can also find us on IRC (#logstash), and on <a href="https://github.com/elastic/logstash" target="_blank">GitHub</a>.
 
Happy Logstashing!

blog-banner-release-logstash.png

blog-thumb-release-logstash.png

Just Enough Redis for Logstash

<html><head></head><body>In this post, we explore the popular file input, and its many use cases. We offer a peek into the evolution of this input and contrast it with alternatives like Filebeat.


<h3>Some History (before version 2.2.0)</h3>
The file input is a popular way in Logstash to ingest contents of a file. From the onset, it was designed to solve the tailing file use case where the file is being constantly updated. Many users have also tried to use it for uploading static file content into Logstash, and we&apos;ll explore this dichotomy in this post.

File input saves information of how far along in the file Logstash has processed so Logstash can resume when it is restarted. For this, it uses sincedb, a file-based persistent store that records the number of bytes read against the file inode (*nix) or identifier (windows).

Most of the real work behind the file input is done by the filewatch library, which is responsible for watching path globs and tailing files.

Filewatch has two phases of operation, discovery and processing. During the first discovery, the watch pattern glob detects files, and each file is put into an internal store. In later discovery phases, only newer files will be put in the store. In the first processing phase, reading: for each stored file, we open, seek to the last byte read position, and read it in 32K chunks. We extract each line from the chunk, send it to the file input, and record the line byte count to the sincedb. Note: The file stays open, and we compare its inode with what we have from the sincedb to decide whether we have seen the file before. In the subsequent processing phase, tailing: if the file has grown, it is read in chunks as above. Some problems with this are:

<ol>
	<li>The files are processed in the order returned by the operating system.</li>
	<li>On Logstash restart, from the little data stored in the sincedb, it is hard to know whether a sincedb entry maps accurately to a discovered file. Therefore sometimes we think that some portion of the file has been read when it is really new content or that none of the file was read when it has.</li>
	<li>All files remain open &#x2014; this uses up file handles and makes it harder to work with the files outside of Logstash.</li>
</ol>
The concept of identity mapping was introduced to the file and other inputs, to ensure that data from the same origin went to the same codec instance, so each time a new identity is seen (path or socket connection), we add a mapping to a clone of the codec specified. To avoid boundless growth of the map, we set the limit to 20,000 identities, because we did not imagine that there would be setups that have more than 20,000 active connections or files being actively tailed. Internally there is a mechanism to evict identities that have not been used for a while (1 hour). Eviction is non-destructive. However in the true tail setup, all the files may be &quot;hot&quot; and their identity (path) may be &quot;hot&quot; too &#x2014; meaning that the identity may never be evicted. However, if the number of discovered files is greater than 20,000 &#x2014; they are all opened &#x2014; then the hard limit will be reached before the eviction mechanism can do anything. 

If the number of open file handles hits the limit, then Logstash starts failing to read/write other files e.g., sincedb, templates, and sockets. 

In version 2.1.0 we implemented: 

<ul>
	<li>new config options: <code>ignore_older</code> and <code>close_older</code>.</li>
	<li>communication of file open and close activity from filewatch to the input to enable identity management in the identity mapper.</li>
</ul>
In this version the <code>ignore_older</code> behaviour bypasses reading and tailing if a file was last modified more than the setting seconds ago. The <code>close_older</code> behaviour is to close a file if it was last modified more than the setting seconds ago. We found two problems with this release: for <code>ignore_older</code>, files were opened then closed (because the open function did the sincedb update) but this consumed file handles unnecessarily and, for <code>close_older</code>, we should have used a <code>last_read</code> time to decide if a file can be closed.

<h3>File Input version 2.2.0</h3>
We introduced a tracking object to hold state on each file we discover and process. In discovery for each file path, a tracking object is stored. This object holds the current state and the previous state history as well as many other attributes e.g., path, inode, <code>bytes_read</code>, stat info, and some timestamps. Files satisfying the <code>ignore_older</code> setting are put into the &apos;ignored&apos; state during discovery and all others are put in the &apos;watched&apos; state. Files in the ignored state are not opened but are monitored for changes. 

We changed the <code>close_older</code> behaviour to use <code>last_read</code> time to decide if a file can be closed. Files in the closed state are monitored for changes.

We have added a <code>max_open_files</code> config option (default 4095). This is implemented as a &apos;sliding window&apos; of files in the active state across the list of files in the watched state.


<pre class="prettyprint">CCCCC|AAAAA|WWWWWWWWWW
CCCCCCCCC|AAAAA|WWWWWW
C = Closed, A = Active, W = watched (waiting)
</pre>

During the processing phase we now have the following steps: 

<ul>
	<li>check if any &apos;closed&apos; files have new content and change them to the &apos;watched&apos; state.</li>
	<li>check if any &apos;ignored&apos; files have new content and change them to the &apos;watched&apos; state.</li>
	<li>take some &apos;watched&apos; files and change them to the &apos;active&apos; state. The amount taken is enough to ensure that the number of &apos;active&apos; files is never more than the <code>max_open_files</code> setting. A warning is issued from time to time if the <code>max_open_files</code> limit is reached and there are files waiting. The file is opened and read in chunks as before.</li>
	<li>check each &apos;active&apos; file for: still readable (can stat) if not delete it is it closable?, change it to the &apos;closed&apos; state and evict its identity has it changed inode?, close and reopen, state stays &apos;active&apos; has it shrunk?, close and reopen, state stays &apos;active&apos; has it grown?, read more, state stays &apos;active&apos;</li>
</ul>
<h3>Multiple File Inputs</h3>
While it is true that multiple inputs will read files in parallel, inputs fight each other for access to the shared queue which is only drained as fast as the filters + outputs will go. However multiple file inputs are a good idea, if&#xA0;the files&#xA0;can be&#xA0;separated&#xA0;into different directories&#xA0;to overcome the 20K identity limit and to have data from the same time period processed together.

<h3>Option Tuning</h3>
<h4>Tailing Use Case</h4>
<ul>
	<li><code>max_open_files</code>: default 4095. Increase this if necessary, but be aware of the file handle limit and that each file input specified in your LS config will open this amount in parallel. If there are more files available to open than this setting, then some will have to wait until others become&#xA0;closable. However, if all the files are constantly getting new content and&#xA0;therefore may never become&#xA0;closable, then some files may never get opened.</li>
	<li><code>close_older</code>: default 1 hour. The appropriate setting of this value for tailing varies a bit. If there are some files that have long (5 seconds) update intervals, then a 2 second value will ensure that resources are freed. Be aware of the reliance the <code>max_files_open</code> setting has on the <code>close_older</code> setting when the number of files being tailed is greater than what is set for <code>max_files_open</code>.</li>
	<li><code>stat_interval</code>: default, 1 second. This is how long the discovery and processing loop waits before looping round. When the volume of files and data being tailed means that the processing stage takes a significant time, then it is not really necessary to wait before looping round &#x2014; therefore the <code>stat_interval</code> value can then be reduced to almost zero.</li>
	<li><code>discover_interval</code>: default: 15. This is, in reality, the number of times the processing part is done before a discovery is done. If it is known that the files first discovered will never change, then one can set the value quite large, e.g., 1000, meaning less time is spent on discovering nothing.</li>
</ul>
<h4>Read File Use Case (files are content complete)</h4>
Sometimes people want to process many, many thousands of files with sizes that vary from hundreds of bytes to gigabytes in size. The file input as it stands now can do this with a combination of <code>max_open_files</code> and <code>close_older</code> settings.

<ul>
	<li><code>max_open_files</code>: default 4095. It is important to know how many file inputs will be opening files on the server across all Logstash processes. In some ways, as it is known that the file will not be appended to, one can set this value and that of <code>close_older</code> quite low, say 64 and 0.1 seconds. So if one has 10 file inputs each on two logstash processes, using the default of 4095, theoretically there can be 81900 files open.&#xA0;Please note that&#xA0;<code>max_open_files</code> really means the number of files open at any time and not the maximum number of files that will ever be read.</li>
	<li><code>close_older</code>: default 1 hour. The value that you should set this to is very dependent on the size of the files you are processing and the value of <code>max_open_files</code>. For example if one has 100 1GB files to process, the&#xA0;closable&#xA0;check will only be done at the end of 100 GB of&#xA0;processing. However if one sets the <code>max_open_files</code> to 2, then the&#xA0;closable&#xA0;check will be done after 2 GB of data &#x2014; in this example a <code>close_older</code> of 1 second will work. &#xA0;In most read file cases you will set this very low, 1 second or less because the files are not being added to.</li>
</ul>
<h3>Filebeat</h3>
<a href="https://www.elastic.co/products/beats/filebeat">Filebeat</a> is a lightweight, open source shipper for log file data. As the next-generation Logstash Forwarder, Filebeat tails logs and quickly sends this information to Logstash for further parsing and enrichment or to Elasticsearch for centralized storage and analysis. 

Filebeat is really the way forward for the tailing use case.&#xA0; It is constantly evolving to support filtering, multiline, and more. You can find out more about Filebeat <a href="https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-getting-started.html">here</a>. 

In many cases it is preferable to install the lightweight Beats components on the machines that generate the logs. 

<h3>Future</h3>
What needs fixing in the file input?

The sincedb needs to store more information so better decisions can be made about whether a discovered file has really been seen before.

At the moment, filewatch communicates to the file input via method calls that certain actions, e.g., open, line(data), and close have taken place. The filewatch communicates the path (identity) only once when the file is opened &#x2014; meaning the file input must hold state, the path, while the file is read. It would be better if the filewatch passed a context object when making calls to the file input. The context object can hold the path, file size, read position, and any other info that establishes the provenance of the line of data.

Filewatch needs to have separate functionality optimised to the read use case. Work in this area has begun for a new input plugin code named &apos;<code>read_file</code>&apos;.

Features that filewatch will have (features common to the existing file input and the new <code>read_file</code> input):

<ul>
	<li>Support for prioritising the order of the files that are activated. Choose between oldest, newest, biggest, smallest, most read, etc.</li>
	<li>Enhanced sincedb that is actually the watched file store persisted.</li>
	<li>Breadth first processing of files &#x2014; striped reading of chunks from each file so that data from different files, but belonging to roughly the same time period, are more likely to be processed together.</li>
</ul>
Features that the Logstash <code>read_file</code> input plugin will have:

<ul>
	<li>close the file immediately without a <code>close_older</code> setting.</li>
	<li>better support the signaling of when the input is done with a file, maybe by updating the sincedb record, appending the path to a well known file, or writing to a socket.</li>
	<li>support for gzip decompression. The tricky bit with gzip decompression is that if Logstash&#xA0;restarts before the file is completely read, the behaviour will be different from a regular file &#x2014; it should restart at the beginning. It will also be a challenge to use breadth first processing of files if one of them is a gzip file. We should try to use streaming decompression because we need to prevent the whole file from being decompressed into memory before being read.</li>
</ul>

While we develop these features, we would love to hear about the read file use case to make sure that we are building the right thing. Please open an <a href="https://github.com/logstash-plugins/logstash-input-file/issues">issue</a> in GitHub with a short description of your setup and expectations. You could also find us on our <a href="https://discuss.elastic.co/c/logstash">forum</a> or <a href="https://twitter.com/elastic">tweet</a> us.</body></html>

banner-punch-cards-sewing-loom.jpg

punch-cards-sewing-loom.jpg

The evolving story about the Logstash File Input

Start free trial

Blog Footer CTA

Sign up for Elastic Cloud free trial

Guy Boertje

By submitting you acknowledge that you've read and agree to our <a href="/legal/elastic-cloud-account-terms" target="_self">Terms of Service</a>, and you agree to receive the <a href="/legal/privacy-statement#how-we-use-the-information" target="_self">selected communications</a> at the contact details you provided above. See <a href="/legal/privacy-statement/" target="_self">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.

Communication Preferences - ONLY for /communication-preferences

提交即表示您确认您已阅读并同意我们的<a href="/cn/legal/elastic-cloud-account-terms" target="_self">服务条款</a>，并且您同意通过您提供的上述联系方式接收<a href="/cn/legal/privacy-statement#how-we-use-the-information" target="_self">所选的通信</a>。如需了解更多信息，或想随时取消订阅，请参阅 <a href="/cn/legal/privacy-statement/" target="_self">Elastic 的隐私声明</a>。

By submitting you acknowledge that you've read and agree to our <a href="/legal/serverless-preview-terms" target="_blank">Terms of Service</a>, and that Elastic may <a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">contact you</a> about our related products and services, using the details you provide above. See <a href="/legal/privacy-statement/" target="_blank">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.

Serverless GDPR Text

提交即表示您确认您已阅读并同意我们的<a href="/cn/legal/serverless-preview-terms" target="_self">服务条款</a>，并且 Elastic 可以使用您提供的上述详细信息与您联系，以向您介绍我们相关的<a href="/cn/legal/privacy-statement#how-we-use-the-information" target="_self">产品和服务</a>。如需了解更多信息，或想随时取消订阅，请参阅 <a href="/cn/legal/privacy-statement/" target="_self">Elastic 的隐私声明</a>。

By submitting you agree to <a href="/agreements/global/cloud-services-monthly" rel="nofollow">Elastic Cloud Terms of Service</a>. Your personal data will be processed in accordance with <a href="/legal/privacy-statement">Elastic's Privacy Statement</a>.

Cloud Trial GDPR Text

提交即表示您同意 <a href="/cn/agreements/global/cloud-services-monthly" rel="nofollow">Elastic Cloud 服务条款</a>。我们将会按照 <a href="/cn/legal/privacy-statement">Elastic 的隐私声明</a>处理您的个人数据。

By submitting you acknowledge that you've read and agree to our <a href="/legal/terms-of-use" target="_blank">Terms of Service</a>, and that Elastic may <a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">contact you</a> about our related products and services, using the details you provide above. See <a href="/legal/privacy-statement/" target="_blank">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.

Newsletter GDPR Text

提交即表示您确认您已阅读并同意我们的<a href="/cn/legal/terms-of-use" target="_self">服务条款</a>，并且 Elastic 可以使用您提供的上述详细信息与您联系，以向您介绍我们相关的<a href="/cn/legal/privacy-statement#how-we-use-the-information" target="_self">产品和服务</a>。如需了解更多信息，或想随时取消订阅，请参阅 <a href="/cn/legal/privacy-statement/" target="_self">Elastic 的隐私声明</a>。

Explore similar demos

Overview

Speakers

Close

See more insights

The content on this page is not available in the selected language. As Elastic grows globally, we continue to support content in multiple languages.

The content on this page is not available in the selected language.

本页内容尚不支持所选语言。Elastic 正在不断努力，以实现对多种语言内容的支持。感谢您在此期间给予的耐心与陪伴！

Author

Articles by

Learn more

Watch now (no PT)

Watch now

See all top stories

All (no PT translation)

Contact information

Press Release

Share on Reddit

Share this story

Share by Email

Thank you for your interest!

Contact Info

Follow us on Youtube

Load More

Share on LinkedIn

Follow us on LinkedIn

Search Integrations

All Solutions

Video for

Follow us on Twitter

See when this webinar starts in my time zone

查看本次网络研讨会在我的时区的开始时间

Register to Watch

Register now

See All Posts

Can't make it? Register and we'll send you the recording. You'll also receive an email with related content

无法实时参加？注册后我们会给您发送录像。您还将收到相关内容的电子邮件。

Author

Articles by Guy Boertje

Logstash Dude, where's my chainsaw? I need to dissect my logs

Just Enough Redis for Logstash

The evolving story about the Logstash File Input

Sign up for Elastic Cloud free trial