04 May 2018

Logstash Lines: Improvements to Plugins

By Monica Sarbu

Welcome to Logstash Lines! With these weekly series, we're keeping you up to date with what's new in Logstash, including the latest commits and releases.

Did you know that Logstash 6.2 is already available? Try it and let us know what you think.

Version 4.1.0 of the File input plugin brings new internals and new features!

In pull request #171, Guy Boertje massively improved the file input code structure, reduced external dependencies and brought in new features, of which we'd like to highlight:

  • New 'reading' mode: as opposed to 'tailing', this mode assumes files have ends and are not appended to;
  • Support for gzip compressed files (only in reading mode);
  • Support expiration of sincedb entries by configuring the `sincedb_clean_after` option

For more information, see the documentation for version 4.1.0.

HTTP Input version 3.1.0: goodbye Puma, hello Netty!

A large rewrite of this plugin's internals (#73) has brought more control over TLS/SSL settings, better handling of backpressure by limiting the number of inflight events over all connections and a modest 10% performance increase and lower memory usage. By building the HTTP layer on top of the Netty Java library this rewrite enables further improvements, as demonstrated in this currently open pull request that brings an extra 15 to 20% gain in performance.

Beta stage implementation of SNMP input is complete!

After a lot of demand for SNMP polling, this initial pull request by Colin Surprenant creates a snmp input plugin that can perform get/walk operations on multiple hosts and use mib files to translate OID to field names. Currently it only supports SNMPv2c over UDP but more is planned, see the plugin's roadmap for more information.

New filter: logstash-filter-bytes

Our own Shaunak has contributed a very useful filter that converts human descriptions of storage size to the canonical number of bytes they represent (e.g. "10 KB" is translated to 10240). See the documentation for more examples and uses.

Grok Debugger UI

Added syntax highlighting to the Grok Debugger UI: https://github.com/elastic/kibana/pull/18572

Documentation

Repository: elastic/logstash

Changes in master:

  • [DOCS] Fixes links to built-in users #9518
  • [DOCS] Removes X-Pack release notes and breaking changes #9509
  • Give an example of a single line Hash. #9505
  • [DOCS] Enables editing links for X-Pack pages #9500
  • [DOCS] Fixes broken link #9479
  • [DOCS] Adds new installation package details #9404

Repository: elastic/logstash-docs

Changes in 6.3:

  • Docs bump for 6.3.0 #521

Changes in versioned_plugin_docs:

  • Fix broken build #537
  • Fix broken links #536
  • Add branch attribute to resolve doc paths #535
  • auto generated update of versioned plugin documentation #533
  • auto generated update of versioned plugin documentation #532
  • auto generated update of versioned plugin documentation #531
  • auto generated update of versioned plugin documentation #530
  • Updated versioned plugin docs #522

Changes in master:

  • update docs for master #526
  • Add bytes filter to the plugin docs #517

Changes in 6.2:

  • update docs for 6.2 #529

Changes in 6.x:

  • update docs for 6.x #528

More changes

Logstash core

  • Fix deep cloning of Timestamp values: #9405

Logstash plugins

Generation of documentation for all versions of all plugins is now done automatically in CI (elastic/infra#4466).

logstash-codec-netflow - 3.13.1

  • Fixes exceptions due to concurrent access of IPFIX templates, see issue #134

logstash-codec-netflow - 3.13.0

  • Added support for Netflow 9 reduced-size encoding support
  • Added support for Barracuda IPFIX Extended Uniflow

logstash-input-google_pubsub - 1.1.0

  • Add additional attributes in the `[@metadata][pubsub_message]` field. Fixes #7