Brewing in Beats: Tomcat JMX monitoring and DNSSEC monitoring

By
Tudor Golubenco

Welcome to Brewing in Beats! With this series, we're keeping you up to date with all that's new in Beats, from the details of work in progress pull requests to releases and learning resources.

New community Beat: JmxProxyBeat

Created by Radovan Ondas, who also wrote Apachebeat and Elasticbeat in the past, JmxProxyBeat gets JMX metrics from Tomcat via the JMX Proxy Servlet. Great companion for Metricbeat :-).

DNSSEC support in Packetbeat

DNS monitoring is already one of the most popular use cases for Packetbeat. Thanks to this large community contribution, Packetbeat is now also able to understand EDNS and DNSSEC resource records.

Clean shutdown procedure

It used to be that you couldn’t stop the Beats if the output queue was not empty (for example, because Elasticsearch/Logstash was not available). After this fairly complex change, the Beats can signal to the publisher that they want to shutdown and that the outputs can drop the output queue if needed. Dropping the inflight messages is fine, because for the Beats that offer at-least-once-guarantees (Filebeat & Winlogbeat), the registry makes sure they will be resent after the Beat restart.

Besides solving the bug mention above, the clean shutdown semantics are one of the prerequisites for configuration reloading and running multiple Beats in a single binary.

Metricbeat, meet generic filtering

The new (since alpha1) libbeat functionality of generic filtering can now be used at module level in Metricbeat, making it possible to flexibly control which fields (metrics) are exported and which not.

Metricbeat system module

Metricbeat now borrows from Topbeat (every Beat is also a library!) the system module, so any host running it will report CPU/mem statistics.

Topbeat and Packetbeat now work on OpenBSD

We don’t yet consider OpenBSD a supported platform for Beats, but thanks to the efforts of Jasper Lievisse Adriaanse, a community member, Topbeat and Packetbeat can now be compiled and executed on OpenBSD. Jasper went as far as fixing libpcap on OpenBSD and provide a vagrant box to us so we can test OpenBSD more regularly. Filebeat was already working on BSDs via simple cross-compilation, because it's pure Go.

Load the Elasticsearch template on every connect

One of the known issues with 5.0.0-alpha1 was that the Beats only attempted to load the Elasticsearch template once, meaning that if they were started before Elasticsearch the template was not loaded. With this change, the template is loaded immediately after a successful connection is established. The error handling was also improved, so we can now make sure that we don’t insert any documents before the template is loaded.

Filebeat’s JSON decoder can now parse @timestamp fields

One of the bugs reported against alpha1 was that if a @timestamp field was supplied as JSON field, Filebeat would crash. This is because our output plugins expect the @timestamp field to have a particular type. Filebeat now makes sure that the timestamp and type (could also be affected by this) have the right types. Besides not crashing, a benefit of this is that the application can now provide its own timestamp to overwrite the one added by Filebeat.

Added path.logs option and log to files by default

The Beats used to log to syslog by default. With the introduction of the directory layout, and to be more similar with the other products in the stack, it makes more sense to log to files by default. So this is now the default. The files are automatically rotated and old files are removed.

When loading the default dashboards, you can now select the index name

Thanks to another contribution by Radovan, you can now rename the index names before loading the dashboards in Elasticsearch.

Other merges: