Brewing in Beats: Add Docker metadata to logs
Welcome to Brewing in Beats! With this weekly series, we're keeping you up to date with what's new in Beats, including the latest commits and releases.
Add Docker metadata to your Docker logs
In case you are using Filebeat to collect the logs from your Docker containers, this PR makes it simple to enhance the logs with the Docker metadata (container ID, name, image, labels). For that, you just need to configure the add_docker_metadata
processor in Filebeat.
processors: - add_docker_metadata: ~
When using the default logging driver (json driver) in Docker, the log files are placed under the path var/lib/docker/containers/<container_id>/*.log
. Filebeat generates an event for each log line and places the name of the log file under the source
field. With this change, the add_docker_metadata
extracts the container ID from the source
field, and queries the Docker API to fetch details about that Docker container.
Install audit rules with Metricbeat
This PR adds the ability to have Metricbeat install audit rules to the kernel. Metricbeat supports adding both file watch rules (-w
) and syscall rules (-a
or -A
). The format for specifying rules is the same as with auditd. For example, both of these are supported.
kernel.audit_rules: | -w /etc/passwd -p wa -k identity -a always,exit -F arch=b64 -S open -F exit=-EACCES -F key=access
Add random startup delays for each metricset
With this PR, Metricbeat starts each metricset at a slightly different time, via random delays. This avoids the thundering herd problem, where sometimes thousands of Beats are inserting documents in perfect sync, causing spikes in the ingestion rates.
Other changes
Repository: elastic/beats
Affecting all Beats
Changes in 5.4:
- Remove empty properties from the template files #4488
Changes in master:
- Fix import statement in beat generator #4520
- vendor go-structform #4514
- Rename `kubernetes` processor -> `add_kubernetes_metadata` #4473
Metricbeat
Changes in 5.4:
- Update gosigar to fix Windows service timeout #4491
Changes in master:
Changes in 5.5:
- Update gosigar to fix Windows service timeout #4491
Documentation
Changes in master:
- Minor: add link to flows doc #4505
Changes in 5.4:
Changes in 5.5:
- Fix release state and update versions for 5.5 #4511
Repository: elastic/gosigar
Changes in master:
- Fix hang during init on Windows if WMI is unavailable #74
Repository: elastic/go-libaudit
Changes in master:
- Add rule flag parsing and generation of rule data #7