16 May 2018

Logstash Lines: SNMP input

By Monica Sarbu

Welcome to Logstash Lines! With these weekly series, we're keeping you up to date with what's new in Logstash, including the latest commits and releases.

Did you know that Logstash 6.2 is already available? Try it and let us know what you think.

Important Notes for Upgrades for Persistent Queue users

If you use the Logstash Persistent Queue we have an important message with regard to upgrading your Logstash instance. We’ve found a few different issues that can affect the way data is serialized in some versions of Logstash. We’ve made some fixes, but an unfortunate consequence of those fixes is that the queue data files may not work when upgrading between Logstash versions. This is something that we did not intend, and are striving not to repeat in the future.

When upgrading Logstash it is important that you either fully drain the queue before upgrading, or fully remove the queue data files. The location of the queue data files is determined by the path.queue setting in your logstash.yml. To drain the queue fully you must:

  1. Set queue.drain: true in your logstash.yml
  2. Start Logstash
  3. Issue a graceful shutdown to Logstash by sending it the SIGTERM signal. This will shut down all inputs and continue processing the pipeline until the queue is empty
  4. Perform the upgrade

We’re aiming to improve this situation in 6.3.0, which shouldn’t require these extra steps in updates from that point onward. Our goal is to provide a seamless experience, and we’re taking a close look at the root causes of this situation.

Logstash SNMP Input v0.1.0.beta1 is out!

We're happy to announce you can now perform SNMP polling with Logstash with the first beta version of SNMP input plugin, which has just been released.

The project is quite young but already has a big list of desirable features on its roadmap. This release is intended to collect feedback from those of you that value this capability and can, better than anyone else, help us shape the plugin into an SNMP polling tool you'd want to use.

To try it out, you can install it by running:

bin/logstash-plugin install --version="0.1.0.beta1" logstash-input-snmp

We currently target to bundle the SNMP input plugin with Logstash starting in version 6.4.

Concurrency issue resolved in DNS filter caching results in massive speed boost!

Our colleague Rob went through the effort of understanding and solving a severe limitation of the existing caching implementation in the DNS filter. This synchronized access to the hit cache caused a bottleneck in Logstash, dropping the event rate of a pipeline from 41k to 600 events per second with the introduction of the DNS filter. After applying his pull request #42 and properly sizing the hit cache for his particular use case, we were able to observe the throughput reach 40k eps, a mere 2% performance impact of performing DNS resolution once the cache warmed up.

Other changes

Repositories under: elastic/logstash-plugins

logstash-filter-dns - 3.0.10

  • Log timeouts as warn instead of error #43
  • Allow concurrent queries when cache enabled #42

logstash-filter-elasticsearch - 3.3.1

  • The filter now only calls filter_matched on events that actually matched. This fixes issues where all events would have success-related actions happened when no match had actually happened ( add_tag, add_field, remove_tag, remove_field) #99

logstash-input-google_pubsub - 1.2.0

  • Change to Java client #12
  • Add create_subscription setting. Fixes #9

logstash-input-http - 3.2.0

  • Add request_headers_target_field and remote_host_target_field configuration options with default to host and headers respectively #68
  • Sanitize content-type header with getMimeType #87
  • Move most message handling code to java #85
  • Fix: respond with correct http protocol version #84

logstash-input-s3 - 3.3.4

  • Log entry when bucket is empty #150

logstash-input-udp - 3.3.3

  • Work around jruby/jruby#5148 by cloning messages on jruby 9k, therefore resizing the underlying byte buffer #39

logstash-output-elasticsearch - 7.4.3

  • Avoid infinite loop by checking for zero body size #737
  • Add support for join based parent indexing (backport of #632) #686

logstash-output-loggly - 3.0.5

  • Get rid of a Ruby warning from using timeout - #24
  • Docs: Better directions for getting a key in Loggly & other cleanups. #26
  • Get rid of a few puts and reduce the noise at logging level info. There is no longer 1+ log/stdout line created for every event ingested. #26

Documentation

Repositor: elastic/logstash

Changes in 6.2:

  • Fix sections breaking changes 62 #9551