Logstash 5.6.0 is stacked with great features and important bug fixes. Here are the highlights:
Introducing Logstash modules
Make way for Logstash modules! Modules contain pre-packaged Logstash pipelines, Kibana dashboards and other metadata files to ease the set up of the Elastic Stack for certain data sources. The goal of these modules are to provide an end-to-end, getting started experience for users to start exploring data sources within minutes. This idea is based on the existing module feature in Filebeat and we've provided a similar user experience in Logstash as well. We are shipping two modules in the 5.6 release:
- Integrating with ArcSight: The Logstash ArcSight module is part of X-Pack Basic License. It enables you to easily integrate with your ArcSight data into Elasticsearch and visualize using Kibana. With a single command, the module taps directly into the Smart Connector or ArcSight Event Broker, parses and indexes the security events into Elasticsearch, and installs a suite of Kibana dashboards to get you exploring your data immediately. Please see this documentation to connect with your ArcSight data with the Elastic Stack, and start playing!
bin/logstash-plugin install x-pack bin/logstash --modules arcsight --setup -M "arcsight.var.inputs=smartconnector,eventbroker"
- Analyzing Netflow traffic: The Logstash Netflow module simplifies the collection, normalization, and visualization of network flow data. With a single command, the module parses network flow data, indexes the events into Elasticsearch, and installs a suite of Kibana dashboards to get you exploring your data immediately. This module supports Netflow version 5 and 9. Please see this documentation to index and visualize your flow data with the Elastic Stack!
bin/logstash --modules netflow --setup
Excited about this and want to see more modules for other data sources in the future? Let us know which modules we should build next by filling out this short survey. It will take less than 5 minutes, we promise.
Supporting string escapes in configuration
One of the oldest feature requests in LS — first reported in 2013 — is shipping with 5.6. Before this change, providing escaping quotes and control characters was not possible in a Logstash config. For example, "\n" was literally a backslash and lowercase n in the config — as a result, you couldn't use mutate filter to split strings with “\n”. Escape sequences such as this are now supported in the configuration. To keep backward compatibility with existing configurations, a feature flag (
config.support_escapes) is required to enable this in your Logstash deployment (#1645)
We spend significant time investigating performance issues and hot-spots in our pipeline. With these tests, we identified that since 5.4, the metrics infrastructure had added significant overhead to the throughput, in some cases up to 40%! One reason for this is that we're simply using the stats more with the introduction of the monitoring UI and the plugins recording more stats. In this release, we're bringing the performance back in line with pre-5.4 release. Overall, in this release, there is lower load average, less GC and higher throughput when running Logstash.
This release adds support for the free GeoIPLite2-ASN database from Maxmind to be able to look up ASN data out of the box.