Brewing in Beats: Dynamic module configuration reloading

Metricbeat: Dynamic configuration reloading for modules

Until now, any change in the modules configuration required a Metricbeat restart. This change allows you to define a configuration directory where new files can be added, removed or modified and updates will automatically be processed by Metricbeat. New modules are started and stopped accordingly. This is especially useful in container environments where one Metricbeat instance is used to monitor all services in other containers on the same host, and the set of containers changes dynamically.

Metricbeat: Add Kafka consumer groups metricset

A new metricset is added to the Kafka module that exports the consumer group information, including the offsets. It also exports client to partition assignments (if available) showing the current consumer handling a partition. Like the producer metricset, it queries only the configured broker, and it reports only the groups currently managed by that broker.

Filebeat: Generate a new fileset

Following the model of the metricset generator in Metricbeat, this PR adds a generator for Filebeat modules. It creates templates for most of the required files of a fileset. Example run:

MODULE=apache FILESET=access make create-fileset

Filebeat: Apache2 module

Added another sample Filebeat module, this time for the Apache2 web server, able to read and parse access and error logs. It comes with an example Kibana dashboard.

All Beats: Make the decode_json_fields target field configurable

The decode_json_fields processor can be used for decoding nested JSON objects from other fields. With this change, you can configure the target field, under which to place the parsed JSON.

All Beats: Format the exported data

By default, the Beats are exporting the data in a JSON format regardless of the output. However, it can be sometimes more convenient to specify a more compact format, so with this PR opened by @treff7es you can specify a format string for the console, file, Kafka, or Redis outputs:

output.kafka:
    format: "%{@timestamp} %{message}"

Metricbeat: Export MySQL query counters

Thanks to @st1t, Metricbeat now exports counters for each MySQL query type. The values are available under the mysql.status.command field.

All Beats: Generate Makefile documentation

The Beats Makefile becomes more and more complex over time, and it becomes necessary to document its targets.  Thanks to @cyrilleverrier, you can run make help to list the available targets and a short description for each.

Metricbeat: Increase the field limit in the mapping

Starting with 5.0, Elasticsearch has a default limit of 1000 fields in the mapping. Because it creates mappings for all its modules, Metricbeat already is a bit over the limit. This PR raises the limit in our template files. In most deployments, these fields will not be used all at the same time, so the effects of increasing the limit shouldn’t be dangerous.