10 October 2017

Brewing in Beats: Enhance data with Azure metadata

By Monica Sarbu

Welcome to Brewing in Beats! With this weekly series, we're keeping you up to date with what's new in Beats, including the latest commits and releases.

Enhance the event with Azure metadata

Thanks to @saada, the add_cloud_metadata processor gets support for Azure. If configured, it enhances each event with Azure metadata. This is useful in case you want to see what logs came from a specific Azure instance or what metrics are associated with it. This feature is scheduled to be released in 6.1.

Support for TLS renegotiation

TLS renegotiation is disabled by default in the Go standard library (and therefore in Beats), because it significantly complicates the state machine and has been the source of security issues in the past. However, it is sometimes needed, so this PR adds support for enabling TLS renegotiation. The setting is ssl.renegotiation and the options are never (default), once, and freely. The new setting can be used anywhere TLS is configured (the outputs, the http module in Metricbeat, Heartbeat, etc.). This feature will be available in 6.1 and is being ported to 5.6 as well.

Work in progress: Autodiscovery

We have started the work around autodiscovery in Beats. Once it’s finished, autodiscovery (the name still to be discussed) will allow for scenarios like:

  • If a Docker container is started using the Nginx image, automatically start the nginx module in Metricbeat/Filebeat configured with the right host/port.
  • If a Kubernetes pod has a particular label, use a predefined Metricbeat/Filebeat configuration for collecting the metrics and logs from that pod.
  • If the MySQL package is installed on a Linux system, automatically start and configure the mysql modules.

In general, auto discovery will allow the Beats to react and adapt to changes in the ever more dynamic infrastructures. See this PR for the general approach we take for the building blocks, as well as the first implementation for Docker.

This work is scheduled to be released in the 6.x time frame.

Participate to Hacktoberfest with Beats

You can participate in the fourth annual Hacktoberfest by contributing to the Beats open source project. Just start by looking at the GitHub issues tagged with the hacktoberfest label.

Screen Shot 2017-10-10 at 20.50.59.png

Other changes

Repository: elastic/beats

Affecting all Beats

Changes in master:

  • Only read go version from .go-version file #5367
  • Do not require template if index change and template disabled #5319
  • Fix data race accessing watched containers #5307

Changes in 6.0:

  • Do not require template if index change and template disabled #5319
  • Fix data race accessing watched containers #5307
  • Remove `-setup` flag deprecation message #5285
Metricbeat

Changes in master:

  • Switch to Kibana 6.0.0-rc1 and increase retry to 300 #5358
Packetbeat

Changes in 6.0:

  • Remove broken options runoptions.uid and runoptions.gid #5261
Filebeat

Changes in master:

  • Fix flaky Filebeat test on Windows #5314

Changes in 6.0:

  • Fix default paths in macOS for Redis 4.0.1 #5173
Auditbeat

Changes in 6.0:

  • Added the 5.x version of the Auditbeat dashboards #5222
Dashboards

Changes in master:

  • Fix RabbitMQ 5.x dashboard names #5359
Testing

Changes in master:

  • Increase Kibana timeout for tests #5316
  • Introduce jenkins_setup command to be reused #5315
  • Make cleanup more reliable #5303

Changes in 5.6:

  • Introduce jenkins_setup command to be reused #5315

Changes in 6.0:

  • Introduce jenkins_setup command to be reused #5315
  • Make cleanup more reliable #5303
Documentation

Changes in master:

  • Changelog cleanup in master #5322
  • Add queue settings to docs #4884

Changes in 5.6:

  • Bump docs version for 5.6.3 in the 5.6 branch #5370
  • Close changelog for 5.6.3 #5369

Changes in 6.0:

  • Add queue settings to docs #4884
  • Cleanup changelog in the 6.0 branch #5333