Brewing in Beats: New community Beats for MongoDB and Amazon

Last week we released Beats 5.1.1 with support for Kafka and Docker monitoring in Metricbeat. For more details about the new features added, please read the blog post.

New community Beat: Mongobeat

Mongobeat, created by Scott Crespo,  discovers instances in a Mongo cluster and collects information from the commands db.stats() and db.serverStatus(). It provides additional information than the mongodb module in Metricbeat.

New community Beat: Amazonbeat

Alex Wormuth created Amazonbeat to read data about the Amazon products and index them to Elasticsearch. The Amazon prices can vary every day, if not every hour. So, an interesting use case would be to follow the price variation for a specific Amazon product, and maybe set an alert with Watcher when the price goes below a certain threshold.

Filebeat modules prototype

We have started working on the Filebeat modules, and we have the first prototype. Following fairly closely the model started in Metricbeat, a Filebeat module contains Ingest Node pipelines, Elasticsearch templates, Filebeat prospectors configurations, and Kibana dashboards that together are able to read, parse, and visualize data from various log files.

There’s already a fully working such module for Nginx (loadable via a prototype python script for now), and we plan to add many more such modules, for things like Mysql, Syslog, Mongodb, Kubernetes, and everything else under the sun. Modules are in a very early stage, but you can follow the progress on this meta ticket.

Metricbeat: Fetch docker stats in parallel

Fetching Docker container stats can be very slow with lots of containers, taking up to 2 seconds. A pull request is open on the Docker side to add support for bulk api, that would increase the performance and reduce the number of opened TPC connections. Until the feature is available in Docker, an workaround was introduced in the Docker module of Metricbeat to create the requests for getting the container stats in parallel. The main downside is that it opens a lots of TCP connections.

Filebeat: Add enabled configuration option for prospectors

To enable/disable a prospector in Filebeat, you had to uncomment/comment the entire configuration for the prospector. To make it easier, the enabled option was introduced in the prospector configuration. By default, the enabled configuration option is set to true.

Winlogbeat: XML unmarshall optimization

In the current implementation the data is read into a []byte encoded as UTF16, converted into []uint16 in order to use utf16.Decode() and then converted to a string. The XML was unmarshalled from the string.

The XML unmarshal process was optimized in Winlogbeat by converting the data directly to UFT8 and then saving the result into a reusable bytes.Buffer. The XML is then unmarshalled directly from the data in buffer.

New flags to import_dashboards

The import_dashboards script now accepts several new SSL flags to connect to Elasticsearch. You can pass a certificate authority in the -cacert argument, a client certificate under -cert argument and a certificate key in a -key argument. The client certificate together with the key must be in a PEM format.

In addition, you can allow insecure SSL connections by passing the -insecure option. By default, it’s false.