Welcome to Weekly Beats! With this series, we're keeping you up to date with all that's new in Beats, from the details of work in progress pull requests to releases and learning resources.
New community Beat: Nagioscheckbeat
There’s a new community Beat, written by Elastic’s Jay Greenberg, and it’s really interesting. Nagioscheckbeat runs Nagios checks and sends the results to Elasticsearch. I don’t think it was ever easier than this to scale and get analytics insights from all those Nagios scripts. Huge potential here, if you ask me. Thank you, Jay :-)
New community Beat: hsbeat (HotSpot VM metrics)
Hsbeat is another very interesting and promising community Beat. It collects performance data from the Java HotSpot VM by reading and decoding the binary performance logs created by the VM.
Filebeat to Logstash throughput improvements
Steffen continued to investigate the issue about low throughput performance when sending synchronously to Logstash, and adjusted the defaults for a significant (9x) increase in the Filebeat -> LS throughput.
These new libbeat defaults are at odds with the maximum memory usage of Packetbeat due to the multiplication of buffers (buffer size * bulk size), so there is another change that solves this issue by making the publisher know if the message contains one event or multiple (possibly thousands). This also improves the configurability of these buffers from the configuration file.
With these changes, Filebeat is at around 60% of the throughput achieved by the Logstash Forwarder. The difference seems to come from the JSON serialization, which we need for creating new features easier, and from making the protocol safer for back pressure situations. Steffen has ideas for closing and even reversing this gap, but they require larger code changes.
Metricbeat (name not final) design started
Nicolas put together a prototype-proposal, which stirred up very pragmatic conversations about what we want to achieve with Metricbeat, how it relates to other Beats (especially the existing community beats), and even how the implementation will look like. If you want to join the discussion, you can do so in this ticket, which also contains our current vision for Metricbeat.
Unified release process
To improve the way all Elastic projects are being built, tested, and released, a common interface needs to be implemented by all of them. We started executing on this interface, which included moving the beats-packer code into the main beats repository.
Topbeat exports total CPU times instead of user CPU times
Topbeat used to export user times for each process, but the Linux top command shows total times, which are arguably more important. So we adjusted topbeat to export total times instead.