Brewing in Beats: HAProxy module in Metricbeat

Welcome to Brewing in Beats! With this series, we're keeping you up to date with all that's new in Beats, from the details of work in progress pull requests to releases and learning resources.

Metricbeat: new HAProxy module

Thanks to a community contribution by @hartfordfive, Metricbeat got an HAProxy module 🎉

Filebeat: The return of the Symlinks

Filebeat used to somehow unintentionally follow symlinks when opening files. So we deprecated them in 1.3 and were planning to remove the support for symlinks in 5.0. However, we’ve got some pretty strong pushback from Kubernetes users (K8s uses symlinks to map log files with useful file names), so we’ve listened and added back support for them. However, the symlink support is off by default and marked as experimental at this point.

Filebeat: close_removed and clean_removed are now on by default

This will keep the registry file from growing to quickly. For background: in 1.x the registry file was a map using the filename as a key. In 5.x the registry file is a list. A side effect of this change is that the registry file grows even if the same file names are reused (think of the usual syslog file names). By making these two options on by default, we’re making sure the registry file stays small.

Metricbeat: Improvements to the MySQL module

A community contribution by @claygorman adds several thread related metrics to the MySQL module in Metricbeat.

Store the version in the Elasticsearch templates

We now store the Beats version in the `_meta` section of the Elasticsearch templates we provide. This can help with troubleshooting and perhaps we could use it in the future to provide automatic mappings upgrades.

import_dashboards: automatically select the right Beat

Now that the import_dashboard scripts were rewritten in Go and download the beats-dashboards package when executed, we wanted to make sure that it does the “right thing” when called without any arguments. So we now adjust the binary at build time to have a different `-beat` argument depending on the package.

Packetbeat: Improve handling of HTTP messages larger than 10MB

Old implementation dropped messages larger than 10MB (hard coded), to protect against memory DoS. However, in some cases the transaction was still recorded, but the parsing of the headers was incomplete. This PR makes the support for large messages explicit, by adding a mode to the parser that "sees" the segments without storing them.