Brewing in Beats: Add Docker metadata to logs

By
Monica Sarbu

Welcome to Brewing in Beats! With this weekly series, we're keeping you up to date with what's new in Beats, including the latest commits and releases.

Add Docker metadata to your Docker logs

In case you are using Filebeat to collect the logs from your Docker containers, this PR makes it simple to enhance the logs with the Docker metadata (container ID, name, image, labels). For that, you just need to configure the add_docker_metadata processor in Filebeat. 

processors:
   - add_docker_metadata: ~

When using the default logging driver (json driver) in Docker, the log files are placed under the path var/lib/docker/containers/<container_id>/*.log. Filebeat generates an event for each log line and places the name of the log file under the source field. With this change, the add_docker_metadata extracts the container ID from the source field, and queries the Docker API to fetch details about that Docker container.

Install audit rules with Metricbeat

This PR adds the ability to have Metricbeat install audit rules to the kernel. Metricbeat supports adding both file watch rules (-w) and syscall rules (-a or -A). The format for specifying rules is the same as with auditd. For example, both of these are supported. 

kernel.audit_rules: |
  -w /etc/passwd -p wa -k identity
  -a always,exit -F arch=b64 -S open -F exit=-EACCES -F key=access

Add random startup delays for each metricset

With this PR, Metricbeat starts each metricset at a slightly different time, via random delays. This avoids the thundering herd problem, where sometimes thousands of Beats are inserting documents in perfect sync, causing spikes in the ingestion rates.

Other changes

Repository: elastic/beats

Affecting all Beats

Changes in 5.4:

  • Remove empty properties from the template files #4488

Changes in master:

  • Fix import statement in beat generator #4520
  • vendor go-structform #4514
  • Rename `kubernetes` processor -> `add_kubernetes_metadata` #4473
Metricbeat

Changes in 5.4:

  • Update gosigar to fix Windows service timeout #4491

Changes in master:

  • fix docker.diskio.total #4507
  • Update gosigar to fix Windows service timeout #4491

Changes in 5.5:

  • Update gosigar to fix Windows service timeout #4491
Documentation

Changes in master:

  • Minor: add link to flows doc #4505

Changes in 5.4:

  • Document complex object support in env vars #4200
  • Minor: add link to flows doc #4505

Changes in 5.5:

  • Fix release state and update versions for 5.5 #4511

Repository: elastic/gosigar

Changes in master:

  • Fix hang during init on Windows if WMI is unavailable #74

Repository: elastic/go-libaudit

Changes in master:

  • Add rule flag parsing and generation of rule data #7