We're excited to share the newest Logstash released with you. Meet version 6.6.0.

<html><head></head><body>Logstash 6.6.0 has been released with new features and enhancements. This release introduces the HTTP filter plugin, the Memcached filter plugin and the experimental Java Plugin API. 
 <h3 dir="ltr">Java Execution Engine</h3>The Java execution engine is GA in this release after months of being experimental. Since we <a href="https://www.elastic.co/blog/meet-the-new-logstash-java-execution-engine">last blogged about this feature</a> we have had feedback from the field that confirms higher performance, reduced memory usage, and lower config startup and reload times. You still need to opt-in to (turn on) Java execution and we have updated the Logstash documentation to explain this better. Java execution will become the default engine in 7.0.
 <h3 dir="ltr">Index Lifecycle Management &#xA0;(ILM)</h3>In 6.6.0, Elasticsearch introduces the Index Lifecycle Management feature. &#xA0;You can control how indices are managed as they age, and specify, for example, a maximum size or age before an index should be automatically rolled over into a new index. We updated the Elasticsearch output to suit. It can automatically create a default lifecycle policy that will automatically rollover an index after it reaches 50GB or is 30 days old. It can also work with more complex policies that have been previously created, which can perform more advanced lifecycle operations, such as deleting the index when it is no longer needed for retention, or moving the index to less performant hardware after a certain amount of time.
 <h3 dir="ltr">Worker Thread Naming</h3>We fixed a regression whereby the worker thread names were not being set in the same way that they were in 6.3. This made interpretation of &quot;hot threads&quot; and stalled shutdown reports really difficult, with this bug fix it is much easier to see that workers could be &quot;hot&quot; or stalled on shutdown.
 
<h3 dir="ltr">Native support for Java plugins</h3>This long awaited feature is released as experimental. Dan Hermann has written a <a href="/blog/previewing-native-support-for-java-plugins-in-logstash">detailed blog post</a>&#xA0;on this feature detailing how, as a Java developer, you can create Logstash plugins without needing to learn Ruby. With this API, Java developers should be able to write Logstash plugins using idiomatic Java code free of any dependencies on Ruby and its toolchain and ecosystem that seamlessly inter-operate with Ruby plugins and yield high performance in the Java execution engine.
 <h3 dir="ltr">HTTP Filter</h3>This is an enrichment filter, it allows you to invoke HTTP calls with parameters from an event to external APIs and have the response inserted into that event. Thanks go to Lucas Henning who created&#xA0;the logstash-filter-rest, upon which this filter is based.Here is an example calling the convert function on the <a href="https://www.neutrinoapi.com/">neutrinoapi</a> service.
 
<pre> http {
 url =&gt; &quot;https://neutrinoapi.com/convert&quot;
 body =&gt; {
 &quot;user-id&quot; =&gt; &quot;auser&quot;
 &quot;api-key&quot; =&gt; &quot;anapikey&quot;
 &quot;from-value&quot; =&gt; &quot;%{millimeters}&quot;
 &quot;from-type&quot; =&gt; &quot;mm&quot;
 &quot;to-type&quot; =&gt; &quot;inch&quot;
 }
 verb =&gt; &quot;POST&quot;
 body_format =&gt; &quot;json&quot;
 target_body =&gt; &quot;neutrinoapi_convert_mm_to_in&quot;
 add_tag =&gt; [&quot;from_api&quot;]
 id =&gt; &quot;http-neutrino-lookup&quot;
 }
 mutate {
 # needed because the neutrino response JSON object includes unneeded extra key/values
 copy =&gt; {&quot;[neutrinoapi_convert_mm_to_in][result]&quot; =&gt; &quot;[inches]&quot;}
 remove_field =&gt; [&quot;neutrinoapi_convert_mm_to_in&quot;]
 id =&gt; &quot;copy-to-inches&quot;
 }</pre>For more details on the plugin and usage instructions, check the <a href="/guide/en/logstash/6.6/plugins-filters-http.html">documentation page</a>. <h3 dir="ltr">Memcached Filter
</h3>This is the enrichment filter you have all been waiting for. Why? It enables K/V set as well as get, meaning, for example, you can have one pipeline extracting and loading memcached with key/values and another pipeline enriching events via the same keys.
An example of setting:
<pre> memcached {
 hosts =&gt; [&quot;localhost&quot;]
 namespace =&gt; &quot;neutrinoapi_convert&quot;
 set =&gt; {
 &quot;[neutrinoapi_convert_mm_to_in][result]&quot; =&gt; &quot;%{millimeters}mm&quot;
 }
 id =&gt; &quot;memcached-set&quot;
 }</pre> An example of getting:
<pre> memcached {
 hosts =&gt; [&quot;localhost&quot;]
 namespace =&gt; &quot;neutrinoapi_convert&quot;
 get =&gt; {
 &quot;%{millimeters}mm&quot; =&gt; &quot;[inches]&quot;
 }
 add_tag =&gt; [&quot;from_cache&quot;]
 id =&gt; &quot;memcached-get&quot;
 }</pre> For more details on the plugin and usage instructions, check the <a href="/guide/en/logstash/6.6/plugins-filters-memcached.html">documentation page</a>.
 <h3 dir="ltr">Other Plugin Improvements
</h3>Logstash 6.6.0 includes a number of improvements to some of our bundled plugins. Some highlights:
 <ul><li>The <a href="/guide/en/logstash/6.6/plugins-filters-aggregate.html">Aggregate filter</a> has some improvements such as custom timeouts, metrics, metadata additions in the code block.</li><li>The <a href="/guide/en/logstash/6.6/plugins-filters-elasticsearch.html">Elasticsearch filter</a> has improvements in the client connection.</li><li>The <a href="/guide/en/logstash/6.6/plugins-inputs-gelf.html">Gelf input</a>&#xA0;has had a general overhaul around shutdown, socket handling json parsing and code quality.</li><li>The <a href="/guide/en/logstash/6.6/plugins-inputs-kafka.html">Kafka input</a>&#xA0;and <a href="/guide/en/logstash/6.6/plugins-outputs-kafka.html">output</a>&#xA0;have version 2.1.0 of the Kafka Client and a new setting &quot;ssl.endpoint.identification.algorithm&quot;.</li><li>The <a href="/guide/en/logstash/6.6/plugins-outputs-elasticsearch.html">Elasticsearch output</a>&#xA0;now supports Index Lifecycle Management for Elasticsearch versions 6.6.0 and above running with a Basic License(or higher).&#xA0;</li><li>The <a href="/guide/en/logstash/6.6/plugins-outputs-tcp.html">Tcp output</a>&#xA0;no longer needs a certificate/key pair when enabling ssl</li></ul><h3> Download and Release Notes </h3>You can find the full list of the changes in the release notes.
 <ul>
	<li dir="ltr"><a href="/downloads/logstash">Download 6.6.0</a></li>
	<li dir="ltr"><a href="/guide/en/logstash/6.6/logstash-6-6-0.html">Release Notes for 6.6.0</a></li>
</ul> <h3 dir="ltr">Feedback</h3>If you want to try the new features added in Logstash 6.6.0, please <a href="/downloads/logstash">download</a> it, install it, and let us know what you think on Twitter (<a href="http://twitter.com/elastic">@elastic</a>) or in our <a href="https://discuss.elastic.co/c/logstash">forum</a>. Enjoy! 
</body></html>

blog-banner-generic-teal.png

logstash-blog-thumb.png

Logstash 6.6.0 Released

We're incredibly proud to announce the best Logstash ever with Logstash 6.2.0!

<html><head></head><body>Logstash 6.2.0 has been released, and it is bringing lots of new goodies. Please refer to the <a href="https://www.elastic.co/guide/en/logstash/6.2/logstash-6-2-0.html">release notes</a> for a complete list. You can download Logstash 6.2.0 from our <a href="https://www.elastic.co/downloads/logstash">downloads page</a>. If you are upgrading from Logstash 5.x, please take a minute to read about <a href="https://www.elastic.co/guide/en/logstash/6.0/breaking-changes.html">breaking changes</a>. Read on for more details on some big changes for Logstash 6.2.0. 
Logstash Keystore 
Tired of keeping passwords in plain text files? Logstash 6.2.0 has got you covered. We now include a new command line tool to help manage your passwords, API keys, and other configuration data that you want to keep from prying eyes. To get started, set the password that you want to use to protect the keystore using the &apos;LOGSTASH_KEYSTORE_PASS&apos; environment variable.
<pre>export LOGSTASH_KEYSTORE_PASS=myReallyStrongPasswordThatNoOneWillEverGuess
</pre>We highly recommend that you set the &apos;LOGSTASH_KEYSTORE_PASS&apos; environment variable. If you don&apos;t set the environment variable, we&#x2019;ll log a warning to notify you that your keystore is unprotected. First let&#x2019;s create the keystore:
<pre>bin/logstash-keystore create
</pre>Now that we have the keystore, we can add a secret into the store. In this case we&#x2019;ll store a password for the Elasticsearch output configuration:
<pre>bin/logstash-keystore add es_pswd
</pre>When you run the command above, you are prompted to type the secret value to be saved in the keystore. Now you can reference &apos;${es_pswd}&apos; in your Logstash configuration. Let&#x2019;s try it. Open your Logstash configuration and modify it as shown below. Notice how we&#x2019;re using the saved secret as the password value.
<pre>output{
 elasticsearch { 
 hosts =&gt; [&quot;http://127.0.0.1:9200&quot;]
 user =&gt; &quot;elastic&quot;
 password =&gt; &quot;${es_pswd}&quot; 
 }
}
</pre>When you restart Logstash, it replaces &#x2018;${es_pswd}&#x2019; with the secret value you set in the keystore. You can use the keystore for more than just passwords. Use it to store API keys, tokens, and other sensitive strings as needed. Use the following command to list the secrets:
<pre>bin/logstash-keystore list
</pre>Note: For security, the actual secret values are never shown. You can also use these secrets in your &#x2018;logstash.yml&#x2019; file. For example: 
<pre>xpack.management.elasticsearch.password: ${es_pswd}
</pre>Does the dollar sign curly bracket syntax looks familiar? It is the same syntax used to look up <a href="https://www.elastic.co/guide/en/logstash/current/environment-variables.html">environment variables</a>. If you are currently using environment variables to help protect passwords, upgrading to the keystore should be easy. You can use keystore and environment variables at the same time. Logstash prefers a keystore entry over an environment variable. 
To avoid any gotchas related to keystore location and password availability when Logstash is installed as a service, please refer to the <a href="https://www.elastic.co/guide/en/logstash/current/keystore.html">official documentation</a> for additional information. 
JDBC Static Filter
When we released <a href="https://www.elastic.co/blog/logstash-5-3-0-released">Logstash 5.3.0</a> we hinted that a static JDBC filter was coming. We are glad to announce that the <a href="https://www.elastic.co/guide/en/logstash/6.2/plugins-filters-jdbc_static.html">JDBC Static filter</a> is here in 6.2.0! This filter makes a different set of tradeoffs from its sibling, the JDBC Streaming filter, instead focusing on lightning fast lookup times for cacheable datasets.
The streaming filter performs a cached query to a remote database for event flowing through Logstash. Some lookups may be fast, and others may be slow, depending on what&#x2019;s in the LRU cache.
With the JDBC static filter, the data is loaded into a local in-memory Java DB instance. You design the local tables, columns, data-types and indexes, and control how the data is loaded into those fields. A scheduler can periodically reload the remote data. For more information, read the <a href="https://www.elastic.co/guide/en/logstash/6.2/plugins-filters-jdbc_static.html">official documentation</a>. 
Persistent Queue Improvements&#xA0;
Logstash 6.2 brings some significant improvements to the Persistent Queue (PQ). This release better aligns the behavior between the PQ and the Memory Queue (MQ).&#xA0;
The PQ implementation and the Memory Queue (MQ) implementation used different approaches to deque data up to now. The PQ favored minimizing the wait time for data to be available to form a batch while the MQ favored maximizing the batch sizes. In the case where outputs were faster than inputs, batch sizes would be much smaller on the PQ than on the MQ, leading to potential performance degradation when the PQ was used.&#xA0;This behavior was particularly problematic for the Elasticsearch output, because larger batches are much more efficient for Elasticsearch to write. With Logstash 6.2 we unified the behavior between the two queue types, favoring larger batch sizes with the PQ to bring it to parity with the MQ.&#xA0;
In addition to unifying behaviours, we changed some of the the default configuration settings. These changes are listed below.<ul>
	<li>`pipeline.batch.delay` has been changed from 5 to 50 milliseconds. Increasing the pipeline.batch.delay to favor larger batches. Generally this setting should not need to be tuned.</li>
	<li>`queue.page_capacity` has been changed from `250mb` to `64mb`. Having a smaller page size mitigates a known performance issue related to larger page sizes.</li>
	<li>JVM option `-XX:+DisableExplicitGC` has been removed. This was identified as having a negative effect on the PQ DirectByteBuffer implementation.</li>
</ul>For additional information see: <a href="https://github.com/elastic/logstash/pull/8702">https://github.com/elastic/logstash/pull/8702</a>, <a href="https://github.com/elastic/logstash/pull/8776">https://github.com/elastic/logstash/pull/8776</a>, <a href="https://github.com/elastic/logstash/pull/8797">https://github.com/elastic/logstash/pull/8797 </a>
Pipelines Listing UI in Monitoring
 In 6.0 we released the <a href="https://www.elastic.co/blog/logstash-pipeline-viewer-6-0">Pipeline Viewer</a> as part of X-Pack monitoring. To open a specific pipeline in the Pipeline Viewer, you had to first go through the Pipelines Listing. This view simply listed all the pipelines (and their versions) that you had running in the selected time window. Each pipeline was shown as a &#x201C;card.&#x201D;<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt02a08379bcd138be/5c30735c597189c90b0242e8/pipelines_listing_old.png" data-sys-asset-uid="blt02a08379bcd138be" alt="pipelines_listing_old.png">Well, we weren&apos;t entirely happy with this UI. In particular, if a pipeline had more than a handful of versions in the selected time window, we didn&apos;t have a nice way to present them all in a scalable manner. Also, the UI served a single purpose &#x2014; as a navigational tool. It allowed you to click a pipeline version to open it in the Pipeline Viewer. That&apos;s fine for a first cut, but a good UI would also draw your attention to pipelines that deserve investigation. Finally, the card layout was somewhat experimental and didn&apos;t offer intuitive affordances for operations such as sorting and filtering. So we set about redesigning the Pipelines Listing. After a few iterations and bit of hallway usability testing, we settled upon a new design.<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt894ae6877b7ede21/5c30735993d9791a70cd748d/pipelines_listing_new.png" data-sys-asset-uid="blt894ae6877b7ede21" alt="pipelines_listing_new.png">Notably in the new design, we got rid of the versions altogether as users found these confusing and overwhelming. Don&apos;t worry though; you can still navigate between different versions of a pipeline in the Pipeline Viewer. We also switched from a card layout to a more traditional table-of-rows layout to make sorting and filtering more intuitive. Finally, we added a couple of sparkline charts for each pipeline in the list, showing the performance of each pipeline over time. These charts help you quickly choose which pipeline to explore. </body></html>

blog-logstash-banner.jpg

blog-logstash-thumb.jpg

Logstash 6.2.0 Released

Introducing Logstash Dissect, an alternative way to extract fields from text

Jonnychainsaw_during_a_chainsaw_art_demonstration_in_Scotland_big.jpg

Jonnychainsaw_during_a_chainsaw_art_demonstration_in_Scotland_23.jpg

Logstash Dude, where's my chainsaw? I need to dissect my logs

Improve the performance of the Redis input plugin with these tips and details.

Message queues are used in Logstash deployments to <a href="https://www.elastic.co/guide/en/logstash/current/deploying-and-scaling.html#deploying-message-queueing" target="_blank">control</a> a surge in events which may lead to slowdown in Elasticsearch or other downstream components. Redis is one of the technologies in use as a message queue. Due to its speed, ease of use, and low resource requirements it is also one of the most popular ways as well.
That&nbsp;is probably why it has been around for a very, very long time—since Logstash version 1.0.4, released more than five years ago!
With all of the recent changes to improve the Logstash pipeline—as well as improvements to the Redis plugin itself—you may not be getting all the performance the Logstash Redis input plugin is capable of delivering.
Logstash attempts to detect the number of cores in your system and set the number of pipeline workers to match. But pipeline workers only matter for filter and output plugins. &nbsp;It has no effect on input plugins. If you haven’t set `threads` in the Redis input plugin, you are only ingesting a fraction of what you could.
The test setup included the most recent stable version of Logstash (2.3.4) with Redis 3.0.7&nbsp;running on the same machine, which is a 2015 MacBook Pro, for reference. The Logstash configuration used is
<pre>input { 
 redis {
 host =&gt; "127.0.0.1"
 data_type =&gt; "list"
 key =&gt; "redis_test"
 # batch_count =&gt; 1
 # threads =&gt; 1
 }
}
output { stdout { codec =&gt; dots } }
</pre>Run this with bin/logstash -f some_config_file.config | pv -Wart &gt; dev/null
This test setup is only designed to test for maximum throughput. Actual performance numbers will vary widely depending on what other plugins are used.
By iterating through a few options for only the threads&nbsp;setting, and using all other settings at their defaults, we see results like these:
<ul>
	<li>threads =&gt; 1&nbsp;: 37.4K/sec</li>
	<li>threads =&gt; 2&nbsp;: 57K/sec</li>
	<li>threads =&gt; 4&nbsp;: 67K/sec</li>
	<li>threads =&gt; 8&nbsp;: 87.3K/sec</li>
</ul>Let’s run the same tests again, but with batch_count =&gt; 250, and add -b 250&nbsp;to our Logstash command line to take full advantage of that increase:
<ul>
	<li>threads =&gt; 1&nbsp;: 35.2K/sec</li>
	<li>threads =&gt; 2&nbsp;: 48.9K/sec</li>
	<li>threads =&gt; 4&nbsp;: 58.7K/sec</li>
	<li>threads =&gt; 8&nbsp;: 72.5K/sec</li>
</ul><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltc2a4980fa1466723/5f85fd80dcf0e74b7cbf58d0/blog-redis-logstash-1.png" title="Throughput: batch size vs threads" height="371" width="600" "="">
As you can see, increasing the default batch_count&nbsp;(which is 125), to 250 results in a decrease in performance.
It’s been common in the past for users to use a larger batch_count&nbsp;when working with Redis. &nbsp;This is no longer best practices, and will actually reduce performance. &nbsp;This is primarily due to changes in the Logstash pipeline architecture. Even an increase of batch_count&nbsp;to 250 results in a drop of performance from 89.5K/sec down to 72.5K/sec when threads =&gt; 8. The sweet spot for batch_count&nbsp;is right around 125, which is the default pipeline-batch-size&nbsp;(there’s a reason for that!).
<h4>Diving deeper</h4>Recent changes to the Redis input include turning on batch mode by default and using a <a href="https://github.com/logstash-plugins/logstash-input-redis/blob/df5bc905e41d36905363229038fbe263a34d17f6/lib/logstash/inputs/redis.rb#L141" target="_blank">Lua script</a> to execute the batch retrieval commands on the Redis server. Lua scripts also act as a transaction so we can take a batch and shorten the queue in one step. This forces other Redis connections to wait. Note, your Logstash Redis output plugin in the upstream Logstash config is one such connection.
Also, you should be aware that the manner in which events are added to Redis by the upstream Logstash can affect the performance of the downstream Logstash instance. For example, while preparing the results of the tests, we found that querying the list size with LLEN redis_test&nbsp;in redis_cli too often would affect the measurement. Try to understand these factors in your upstream Logstash instance:
<ul>
	<li>the rate that events are generated</li>
	<li>the size of the events, smallest; largest and average</li>
	<li>the batch_events size used the redis output</li>
</ul>You might want to look at using bigger batch_events sizes e.g. 500 on the upstream Redis output - this will buffer the events before calling Redis and doing this may give the downstream Redis input threads time to jump in to pull from Redis. However, if the event generation rate is high then this may not be as helpful - 50K/sec in batches of 500 means a batch buffer fills in 10 milliseconds.
We have seen that the performance of the Lua script suffers proportionately with larger batch sizes. This means that larger batch sizes on the Redis input will affect the performance of both of the Logstash instances on either side of Redis.
Generally, the goal of tuning the Redis input plugin config and the pipeline batch size and worker threads is to make sure that the input is able to keep pace with the filter/output stage. You should know what the average throughput of your filter/output stage is with the defaults. If, for example, your filter/output stage throughput is 35K/sec then there is little need to change the redis input worker count to 8 when one or two workers can fetch sufficient events per second.
One more source of variation is JSON encoding (1), decoding (2) and encoding (3) - yes, three times, 1 in the upstream Redis output, 2 in the downstream Redis input and 3 in the elasticsearch output. Events with big string values do take some time to decode so make sure your testing is using data sets that represent the size and distribution of size as close to your production data as you can make it.
<h4>Scaling throughput using multiple Logstash instance and one Redis server</h4>Many users seek to improve throughput by starting up multiple Logstash instances on different hardware all pointing to the same Redis server and list.
This does work and can work well, but you are advised to apply the same tuning tips to each instance. There is probably not much point in having multiple Redis inputs defined in any one config unless they are connecting to different Redis servers or a different list. Simply tune the threads of the one Redis input. Clearly, having, say, 3 Logstash instances pointing at the same Redis server will create some degree of Redis client connection contention - meaning that you should probably tune one instance and then retune all three when running together.
<h4>Conclusion</h4>In the not too distant future, we will be enhancing the Redis input with a JRuby wrapper around the popular and speedy Jedis Java library. &nbsp;This will bring the added bonus of <a href="https://github.com/logstash-plugins/logstash-input-redis/issues/10" target="_blank">supporting</a> Redis Cluster mode!
In the meanwhile, to give your Redis input performance a boost, try increasing the number of threads used. And be sure to measure and tune accordingly if you use something other than the default batch_count, or have multiple instances of Logstash pointing to a single Redis server.
If you have a question about Redis and Logstash, feel free to open a discussion topic on our <a href="https://discuss.elastic.co/c/logstash" target="_blank">forum</a>. You can also find us on IRC (#logstash), and on <a href="https://github.com/elastic/logstash" target="_blank">GitHub</a>.
 
Happy Logstashing!

logstash-blog-banner.png

Just Enough Redis for Logstash

In this post, we explore the Logstash File Input, and its many use cases with a peek into its evolution and contrast it with alternatives like Filebeat

<html><head></head><body>In this post, we explore the popular file input, and its many use cases. We offer a peek into the evolution of this input and contrast it with alternatives like Filebeat.


<h3>Some History (before version 2.2.0)</h3>
The file input is a popular way in Logstash to ingest contents of a file. From the onset, it was designed to solve the tailing file use case where the file is being constantly updated. Many users have also tried to use it for uploading static file content into Logstash, and we&apos;ll explore this dichotomy in this post.

File input saves information of how far along in the file Logstash has processed so Logstash can resume when it is restarted. For this, it uses sincedb, a file-based persistent store that records the number of bytes read against the file inode (*nix) or identifier (windows).

Most of the real work behind the file input is done by the filewatch library, which is responsible for watching path globs and tailing files.

Filewatch has two phases of operation, discovery and processing. During the first discovery, the watch pattern glob detects files, and each file is put into an internal store. In later discovery phases, only newer files will be put in the store. In the first processing phase, reading: for each stored file, we open, seek to the last byte read position, and read it in 32K chunks. We extract each line from the chunk, send it to the file input, and record the line byte count to the sincedb. Note: The file stays open, and we compare its inode with what we have from the sincedb to decide whether we have seen the file before. In the subsequent processing phase, tailing: if the file has grown, it is read in chunks as above. Some problems with this are:

<ol>
	<li>The files are processed in the order returned by the operating system.</li>
	<li>On Logstash restart, from the little data stored in the sincedb, it is hard to know whether a sincedb entry maps accurately to a discovered file. Therefore sometimes we think that some portion of the file has been read when it is really new content or that none of the file was read when it has.</li>
	<li>All files remain open &#x2014; this uses up file handles and makes it harder to work with the files outside of Logstash.</li>
</ol>
The concept of identity mapping was introduced to the file and other inputs, to ensure that data from the same origin went to the same codec instance, so each time a new identity is seen (path or socket connection), we add a mapping to a clone of the codec specified. To avoid boundless growth of the map, we set the limit to 20,000 identities, because we did not imagine that there would be setups that have more than 20,000 active connections or files being actively tailed. Internally there is a mechanism to evict identities that have not been used for a while (1 hour). Eviction is non-destructive. However in the true tail setup, all the files may be &quot;hot&quot; and their identity (path) may be &quot;hot&quot; too &#x2014; meaning that the identity may never be evicted. However, if the number of discovered files is greater than 20,000 &#x2014; they are all opened &#x2014; then the hard limit will be reached before the eviction mechanism can do anything. 

If the number of open file handles hits the limit, then Logstash starts failing to read/write other files e.g., sincedb, templates, and sockets. 

In version 2.1.0 we implemented: 

<ul>
	<li>new config options: <code>ignore_older</code> and <code>close_older</code>.</li>
	<li>communication of file open and close activity from filewatch to the input to enable identity management in the identity mapper.</li>
</ul>
In this version the <code>ignore_older</code> behaviour bypasses reading and tailing if a file was last modified more than the setting seconds ago. The <code>close_older</code> behaviour is to close a file if it was last modified more than the setting seconds ago. We found two problems with this release: for <code>ignore_older</code>, files were opened then closed (because the open function did the sincedb update) but this consumed file handles unnecessarily and, for <code>close_older</code>, we should have used a <code>last_read</code> time to decide if a file can be closed.

<h3>File Input version 2.2.0</h3>
We introduced a tracking object to hold state on each file we discover and process. In discovery for each file path, a tracking object is stored. This object holds the current state and the previous state history as well as many other attributes e.g., path, inode, <code>bytes_read</code>, stat info, and some timestamps. Files satisfying the <code>ignore_older</code> setting are put into the &apos;ignored&apos; state during discovery and all others are put in the &apos;watched&apos; state. Files in the ignored state are not opened but are monitored for changes. 

We changed the <code>close_older</code> behaviour to use <code>last_read</code> time to decide if a file can be closed. Files in the closed state are monitored for changes.

We have added a <code>max_open_files</code> config option (default 4095). This is implemented as a &apos;sliding window&apos; of files in the active state across the list of files in the watched state.


<pre class="prettyprint">CCCCC|AAAAA|WWWWWWWWWW
CCCCCCCCC|AAAAA|WWWWWW
C = Closed, A = Active, W = watched (waiting)
</pre>

During the processing phase we now have the following steps: 

<ul>
	<li>check if any &apos;closed&apos; files have new content and change them to the &apos;watched&apos; state.</li>
	<li>check if any &apos;ignored&apos; files have new content and change them to the &apos;watched&apos; state.</li>
	<li>take some &apos;watched&apos; files and change them to the &apos;active&apos; state. The amount taken is enough to ensure that the number of &apos;active&apos; files is never more than the <code>max_open_files</code> setting. A warning is issued from time to time if the <code>max_open_files</code> limit is reached and there are files waiting. The file is opened and read in chunks as before.</li>
	<li>check each &apos;active&apos; file for: still readable (can stat) if not delete it is it closable?, change it to the &apos;closed&apos; state and evict its identity has it changed inode?, close and reopen, state stays &apos;active&apos; has it shrunk?, close and reopen, state stays &apos;active&apos; has it grown?, read more, state stays &apos;active&apos;</li>
</ul>
<h3>Multiple File Inputs</h3>
While it is true that multiple inputs will read files in parallel, inputs fight each other for access to the shared queue which is only drained as fast as the filters + outputs will go. However multiple file inputs are a good idea, if&#xA0;the files&#xA0;can be&#xA0;separated&#xA0;into different directories&#xA0;to overcome the 20K identity limit and to have data from the same time period processed together.

<h3>Option Tuning</h3>
<h4>Tailing Use Case</h4>
<ul>
	<li><code>max_open_files</code>: default 4095. Increase this if necessary, but be aware of the file handle limit and that each file input specified in your LS config will open this amount in parallel. If there are more files available to open than this setting, then some will have to wait until others become&#xA0;closable. However, if all the files are constantly getting new content and&#xA0;therefore may never become&#xA0;closable, then some files may never get opened.</li>
	<li><code>close_older</code>: default 1 hour. The appropriate setting of this value for tailing varies a bit. If there are some files that have long (5 seconds) update intervals, then a 2 second value will ensure that resources are freed. Be aware of the reliance the <code>max_files_open</code> setting has on the <code>close_older</code> setting when the number of files being tailed is greater than what is set for <code>max_files_open</code>.</li>
	<li><code>stat_interval</code>: default, 1 second. This is how long the discovery and processing loop waits before looping round. When the volume of files and data being tailed means that the processing stage takes a significant time, then it is not really necessary to wait before looping round &#x2014; therefore the <code>stat_interval</code> value can then be reduced to almost zero.</li>
	<li><code>discover_interval</code>: default: 15. This is, in reality, the number of times the processing part is done before a discovery is done. If it is known that the files first discovered will never change, then one can set the value quite large, e.g., 1000, meaning less time is spent on discovering nothing.</li>
</ul>
<h4>Read File Use Case (files are content complete)</h4>
Sometimes people want to process many, many thousands of files with sizes that vary from hundreds of bytes to gigabytes in size. The file input as it stands now can do this with a combination of <code>max_open_files</code> and <code>close_older</code> settings.

<ul>
	<li><code>max_open_files</code>: default 4095. It is important to know how many file inputs will be opening files on the server across all Logstash processes. In some ways, as it is known that the file will not be appended to, one can set this value and that of <code>close_older</code> quite low, say 64 and 0.1 seconds. So if one has 10 file inputs each on two logstash processes, using the default of 4095, theoretically there can be 81900 files open.&#xA0;Please note that&#xA0;<code>max_open_files</code> really means the number of files open at any time and not the maximum number of files that will ever be read.</li>
	<li><code>close_older</code>: default 1 hour. The value that you should set this to is very dependent on the size of the files you are processing and the value of <code>max_open_files</code>. For example if one has 100 1GB files to process, the&#xA0;closable&#xA0;check will only be done at the end of 100 GB of&#xA0;processing. However if one sets the <code>max_open_files</code> to 2, then the&#xA0;closable&#xA0;check will be done after 2 GB of data &#x2014; in this example a <code>close_older</code> of 1 second will work. &#xA0;In most read file cases you will set this very low, 1 second or less because the files are not being added to.</li>
</ul>
<h3>Filebeat</h3>
<a href="https://www.elastic.co/products/beats/filebeat">Filebeat</a> is a lightweight, open source shipper for log file data. As the next-generation Logstash Forwarder, Filebeat tails logs and quickly sends this information to Logstash for further parsing and enrichment or to Elasticsearch for centralized storage and analysis. 

Filebeat is really the way forward for the tailing use case.&#xA0; It is constantly evolving to support filtering, multiline, and more. You can find out more about Filebeat <a href="https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-getting-started.html">here</a>. 

In many cases it is preferable to install the lightweight Beats components on the machines that generate the logs. 

<h3>Future</h3>
What needs fixing in the file input?

The sincedb needs to store more information so better decisions can be made about whether a discovered file has really been seen before.

At the moment, filewatch communicates to the file input via method calls that certain actions, e.g., open, line(data), and close have taken place. The filewatch communicates the path (identity) only once when the file is opened &#x2014; meaning the file input must hold state, the path, while the file is read. It would be better if the filewatch passed a context object when making calls to the file input. The context object can hold the path, file size, read position, and any other info that establishes the provenance of the line of data.

Filewatch needs to have separate functionality optimised to the read use case. Work in this area has begun for a new input plugin code named &apos;<code>read_file</code>&apos;.

Features that filewatch will have (features common to the existing file input and the new <code>read_file</code> input):

<ul>
	<li>Support for prioritising the order of the files that are activated. Choose between oldest, newest, biggest, smallest, most read, etc.</li>
	<li>Enhanced sincedb that is actually the watched file store persisted.</li>
	<li>Breadth first processing of files &#x2014; striped reading of chunks from each file so that data from different files, but belonging to roughly the same time period, are more likely to be processed together.</li>
</ul>
Features that the Logstash <code>read_file</code> input plugin will have:

<ul>
	<li>close the file immediately without a <code>close_older</code> setting.</li>
	<li>better support the signaling of when the input is done with a file, maybe by updating the sincedb record, appending the path to a well known file, or writing to a socket.</li>
	<li>support for gzip decompression. The tricky bit with gzip decompression is that if Logstash&#xA0;restarts before the file is completely read, the behaviour will be different from a regular file &#x2014; it should restart at the beginning. It will also be a challenge to use breadth first processing of files if one of them is a gzip file. We should try to use streaming decompression because we need to prevent the whole file from being decompressed into memory before being read.</li>
</ul>

While we develop these features, we would love to hear about the read file use case to make sure that we are building the right thing. Please open an <a href="https://github.com/logstash-plugins/logstash-input-file/issues">issue</a> in GitHub with a short description of your setup and expectations. You could also find us on our <a href="https://discuss.elastic.co/c/logstash">forum</a> or <a href="https://twitter.com/elastic">tweet</a> us.</body></html>

Articles by Guy Boertje

Logstash 6.6.0 Released

Logstash 6.2.0 Released

Logstash Dude, where's my chainsaw? I need to dissect my logs

Just Enough Redis for Logstash

The evolving story about the Logstash File Input

产品

公司

资源