Brewing in Beats: Mysqlbeat and HWsensorsbeat from the community
Welcome to Brewing in Beats! With this series, we're keeping you up to date with all that's new in Beats, from the details of work in progress pull requests to releases and learning resources.
Last week we’ve released Beats 1.2.2. The blog post is here.
New community Beat: HWsensorsbeat
Created by Jasper Lievisse Adriaanse, HWsensorsbeat reads HW sensors information through sysctl(3). It’s already part of OpenBSD ports: http://cvsweb.openbsd.org/cgi-bin/cvsweb/ports/sysutils/beats/hwsensorsbeat
New community Beat: Mysqlbeat
Created by Adi Ben-Dahan, Mysqlbeat is a fully customizable Beat for MySQL server. This beat can also ship the results of any query defined on the config file to Elasticsearch. It comes with a really cool Kibana dashboard.
Update Elasticsearch template for Beats
Starting with Elasticsearch 5.0.0-alpha1, string is replaced with text/keyword datatypes, text is for full text, and keyword for keyword identifiers. The string datatype will continue to work during the 5.x series, but it will be removed in 6.0.
We updated the Elasticsearch templates for all Beats, including the script that generates them, to use keyword datatype instead of string. In addition, the old Elasticsearch template is still available under *.template-es2x.json that works for Elasticsearch 2.x and 5.x versions.
Simplifying Filebeat state logic
As Filebeat gets more features, it is also important to refresh and simplify its architecture so that it can be maintained in the long therm. Nicolas spent some time last week to decouple and simplify the communication between the harversters, the prospectors, and the registrar.
Fixed a Filebeat JSON bug
If the decoded JSON contained “null” values this would travel through Filebeat and it could break the generic filtering code. This was fixed by hardening the normalization code that is executed before filtering is executed.
FIxed a Winlogbeat bug related to large events
Fixed panic that occurred when reading a large events on Windows Vista and newer. The bug occurred in an error recovery path so in order for the panic to occur, first there had to have been an error rendering the event as XML with the event message string. When that error occurs Winlogbeat tries to render the event as XML, but without the message string. If the XML was larger than half the buffer size a panic would occur.
Rename the filter configuration sections to filters
This is just a cosmetic change in libbeat, but it breaks the compatibility with 5.0.0-alpha1. The filters configuration section includes a list of filtering rules that are applied one by one in the order they are defined.
Generate a single template for Metricbeat
Each Metricbeat module exports fields whose type needs to be defined in Elasticsearch via a template. Instead of loading a template for each Metricbeat module, the pull request collects all exported fields and their datatypes from each Metricbeat module and generates a single template for Metricbeat.
Use ucfg to validate the Beats configuration files
The Beats are using the ucfg library, written by Steffen, to deal with configuration file management. As simple configuration errors are often the source of troubles, Steffen greatly improved the validation of the configuration files, which is now checked before the Beat starts.
Introduce namespacing in the Packetbeat configuration file
As we’re moving towards having all Beat specific settings namespaced with their Beat name (e.g. packetbeat.interfaces), Packetbeat accepts now packetbeat.interfaces and packetbeat.protocols instead of interfaces and protocols as it’s top level key. The change breaks compatibility with 5.0.0-alpha1.