Editor's Note (September 7, 2018): This post refers to X-Pack. Starting with the 6.3 release, the X-Pack code is now open and fully integrated as features into the Elastic Stack.
Dead Letter Queues
We are happy to introduce a popularly requested feature called dead letter queues. In message processing systems, a dead letter queue (DLQ) is a mechanism to route bad events to a different destination when they can't be processed. For Logstash, this means an ability to shunt poisoned or unsuccessful events in the running pipeline to a new destination for further processing. In Logstash's case, the new destination is a file based queue to stores these bad events, so you can re-process them. This feature allows the existing pipeline to continue processing events without getting stuck because of bad events.
As an initial implementation, the Elasticsearch Output is the only plugin that uses this feature. To use this feature, you'll have to enable it in the settings using dead_letter_queue.enable: true. Once enabled, events that fail to process due to mapping issues in the Elasticsearch Output are automatically moved to the DLQ on local disk. When it comes time for re-processing the data that’s been dead lettered, you simply spin up a separate LS pipeline with the new DLQ input plugin and the updated processing logic.
We are planning on iteratively expanding the scope of the DLQ from the plugin side and your input here is welcomed!
You know that joke about regular expressions — yep, regexes are hard! Grok makes using regexes easier in the context of extracting fields from unstructured data. It is one of our popular plugins! Even though Grok is much easier to work with than raw regexes, users often need help while constructing the grok expressions for their gnarly log data.
Enter Grok Debugger - a popular web UI to iteratively craft grok expressions. There are other excellent tools as well, like the Grok Constructor. Over the years, we've got requests from users to provide an option like this in Kibana. Also, many users didn't want to send their log data to an external website for security reasons. So, there you have it — under Dev Tools, there's now a Grok Debugger UI in 5.5! Just install x-pack on Kibana and it is free to use! Under the covers, the Grok Debugger uses the ingest node’s grok processor. Since ingest node’s Grok implementation and Logstash’s Grok plugin share the same Joni regular expression engine, any pattern you construct using this UI is portable across both products.
This is just an initial version and we plan to add more features!
GeoIP ASN enrichment
MaxMind recently introduced a free database (GeoIPLite2-ASN) that can be used to lookup ASN information from an IP address. ASN information is also available in the commercial GeoIP2-ISP database. Logstash now supports both these databases starting from GeoIP plugin version 4.2.0.
5.5.0 has plenty of bug fixes in Logstash core and plugins. The release notes have detailed information.