19 12월 2016

Brewing in Beats: Filebeat modules for MySQL and Syslog

By Monica Sarbu

New community Beat: Udplogbeat

Al Lefebvre created Udplogbeat that receives the events via local UDP socket (in plain-text or JSON with ability to enforce schemas) to index  them in Elasticsearch. An interesting use case can be that Udplogbeat allows any application to easily log messages locally without writing them to disk. For example it can be used for applications that only support syslog logging.

This solution doesn’t guaranty that each log message is sent at least once as in case Udplogbeat is down for any given reason, log messages sent over UDP will not reach Elasticsearch.

Filebeat: Mysql and syslog modules (in progress)

Recently we started working on a prototype for Filebeat modules. Following fairly closely the model started in Metricbeat, a Filebeat module contains the out of the box configuration needed to read, parse and visualize data from various log files formats. This includes Ingest Node pipelines, Elasticsearch templates, Filebeat prospectors configurations, and Kibana dashboards. 

In the past week, we added two more sample modules, for MySQL and Syslog, to further test the assumptions made by the prototype. Work is currently being done to create automatic integration tests for these modules, then we can replace the prototype with the actual code.

Metricbeat: Couchbase support

You can now monitor the Couchbase database thanks to the community contribution of Daniel Paschke that added the Couchbase module to Metricbeat. It collects 3 different metricsets from Couchbase: cluster overview metrics, node metrics and bucket metrics.The Couchbase module is experimental and we are entitled to do any changes, including breaking changes.

Metricbeat: Prometheus support

Metricbeat is now able to monitor the Prometheus server itself. In the next steps, we are planning to fetch metrics from the Prometheus exporters or any application that offers a Prometheus endpoint and index them to Elasticsearch. In this case, you can use Metricbeat and Elasticsearch as an alternative to Prometheus.

The Prometheus module is marked as experimental.

Metricbeat: Export the number of CPU cores

The number of CPU cores is now exported as part of the system module in Metricbeat. Thanks Martin Scholz for the contribution!

Packaging improvements for the community Beats

Every community Beat can create deb/rpm/tar.gz for all platforms by simply calling make package. Cyrille Verrier has made several improvements to the community Beats packaging, allowing them to do their own versioning and license.

Fix for build files ownership

An annoying "feature" of our Docker based build system is that the temporary files and directory created by it are shown as owned by root on the host. Cyrille found a workaround that does a chown inside Docker as a last step.