Today we are pleased to announce that Beats 5.3.0 was released. This is the latest stable version and it comes with Filebeat modules in Beta and dynamic configuration reloading for modules.
Up until this release, being able to successfully index logs, such as Apache access logs, required a variety of skills. The user was required to:
- have a good knowledge of Grok in order to parse the logs with Ingest node or Logstash
- be familiar with the Elasticsearch mapping template in order to set the type of each field
- create Kibana dashboards from scratch to visualize the data.
For common log formats, such as access logs, MySQL slow logs, or syslog messages, we found that our users frequently encountered similar issues when building the grok patterns, and they ended up building similar Kibana dashboards. This caused a lot of duplicated effort in our community that we felt could be reduced by sharing the grok patterns, Elasticsearch templates, and Kibana dashboards for specific services with the entire community.
Filebeat 5.3 comes with modules, similar to the modules in Metricbeat or Packetbeat. A typical module is composed of one or more filesets, one for each log type, that contain Ingest Node pipelines, Elasticsearch templates, Filebeat configuration files, and Kibana dashboards. Thus, monitoring the logs of common services gets as simple as running a single command, without the need to even know what grok is.
In the future, a module could also include machine learning configurations for anomaly detection and also Watcher configurations for alerts.
For the start, the Filebeat 5.3 release includes Beta modules for Apache2, MySQL, Nginx, and System.
But that’s not all! You can easily create your own Filebeat module to add support for another service. The exciting thing is that you don’t have to write any code for it, because modules contain only configuration.
Following the model of the metricset generator in Metricbeat, we created a generator for Filebeat filesets. It creates templates for most of the required files of a fileset, so most of the work for creating a new fileset is done for you. If we convinced you, give it a try and let us know what you think!
Dynamic configuration reloading for modules
Filebeat 5.3 release comes with dynamic configuration reloading for modules, which makes it possible to change any module configuration on the fly without restarting the Beat. With this feature, Metricbeat is able to dynamically reload its modules, Filebeat is able to reload the prospectors, and Heartbeat is able to reload the list of monitored targets.
You just need to define a configuration directory (by default
/etc/metricbeat/conf.d/ for Metricbeat) where new configuration files can be added, removed, or modified, and the Beat will automatically update its running configuration, and start or stop new modules accordingly. This feature is especially useful in container environments where one Metricbeat instance is used to monitor dynamic services in other containers on the same host, and the set of containers changes dynamically.
Dynamic configuration reloading for modules is a beta feature in Filebeat and experimental in Metricbeat and Heartbeat.
Export environment variables used to start a process
Metricbeat 5.3 is able to capture and index the environment variables that were used to start a process. This is a useful feature especially for monitoring Docker containers, where environment variables are often used for configuration. To avoid leaking sensitive data, by default, no environment variables are captured You can specify the list of environment variables to capture by configuring the
process.env.whitelist option under the system module . For example:
metricbeat.modules: - module: system metricsets: [process] process.env.whitelist: ['USER', 'PATH']
This feature is available on FreeBSD, Linux and macOS.