Brewing in Beats: Enrich events with Docker metadata

Welcome to Brewing in Beats! With this weekly series, we're keeping you up to date with what's new in Beats, including the latest commits and releases.

New processor: add_docker_metadata

Inspired by the add_kubernetes_metadata processor, this pull request adds the add_docker_metadata processor to enhance the events with the Docker metadata like name, image, and labels. The processor requires that the container ID to be present in the event fields, and queries the Docker API to get the metadata for that container. For example, the cgroups events generated by the system module in Metricbeat contain the container ID under system.process.cgroup.id, and you can configure the processor as following to enrich the events generated by each Docker container:

processors:
 - add_docker_metadata:
     match_fields: ["system.process.cgroup.id"]
     host: "unix:///var/run/docker.sock"

In case you want to connect to Docker over TLS, you need to configure a client and a CA certificate.

The processor is released in 6.0.0-alpha2.

Filebeat: Redis module

Build upon the Redis prospector, which was merged the week before, we now have a Filebeat Redis module with two filesets:

  • One for collecting and parsing the logs written on disk by Redis
  • One that uses the Redis prospector to interrogate the slow logs

A Kibana dashboard is included:

filebeat-redis.png

We currently plan to release this module with 6.0.

With this, Redis joins the select group of projects for which we have a Filebeat module to monitor its logs, a Metricbeat module to collect its metrics, and a Packetbeat module/protocol to parse its network traffic.

Metricbeat: New Rabbitmq module

Thanks to @amandahla, Metricbeat gets support for RabbitMQ. The module comes with the node metricset that collects metrics from RabbitMQ nodes by querying the HTTP API /api/nodes.

The module is marked as experimental and it will be released with 6.0.

Other changes

Repository: elastic/beats

Affecting all Beats

Changes in master:

  • Only attempt loading the ES template if ES output is enabled #4436
  • Add a permissions option to logging.files for all beats #4428
Filebeat

Changes in master:

  • Fix typo in the configuration files #4447
  • Nginx module: use first not private IP address as remote_ip #4417
Metricbeat

Changes in 5.x:

  • Fix type for HAProxy health.last field #4410

Changes in master:

  • Make IP port indexer constructor public #4434
  • Disable kubernetes event metricset by default #4431
Infrastructure

Changes in master:

  • Separate travis jobs in stages #4453
Documentation

Changes in 5.4:

  • Add docs for Docker images #4312,
  • Provide more detail on log levels #4405,
  • Add simple examples that illustrate multiline settings #4408
  • Bump docs version to 5.4.1 #4396

Changes in 5.3:

  • Bump docs version to 5.3.3 #4438

Changes in master:

  • Close changelog for 6.0.0-alpha2 #4455
  • Add `add_docker_metadata` processor docs #4448
  • Change input_type to type in config examples #4433
  • Remove duplicated content about contributing to beats #4424