Brewing in Beats: Autodiscovery with Docker

By
Monica Sarbu

Autodiscovery - use Docker events to auto-configure Beats

The autodiscovery feature with the first provider (for Docker) has been merged. Autodiscovery allows the user to define different providers, that watch for system changes and emit events to a common bus. Then the autodiscovery module detects situations when there is something we can monitor and instantiates new modules for it.

The first provider watches for Docker events. It supports config mapping from container metadata to config templates, so new modules are created when a container starts. 

metricbeat.autodiscover:
  providers:
    - type: docker
      templates:
        - condition:
            equals.docker.container.image: redis
          config:
            - module: redis
              metricsets: ["info", "keyspace"]
              hosts: "${data.host}:${data.port}"

The above is an example configuration that instantiates the Metricbeat Redis module every time a new Redis container (defined by having the redis image) is started. Note that the connection information (host/ports) is filled in by the autodiscovery support via a template.

This feature will be released in Metricbeat and Filebeat 6.1.

Configure the number of routing shards in the Elasticsearch template

Elasticsearch 6.1 will have an API for splitting shards. To enable this feature, applications like Beats need to set the config option index.number_of_routing_shards at index time creation. The actual number of shards must be a factor of the number of routing shards.

This PR adds configuration option in Beats for the routing shards, with a default of 30. We chose 30 as it is a multiple of 1, 3 and 5, our current number of default shards in Beats and ES. 

The new configuration option will be present in Beats 6.1.

Packetbeat: support for reading TLS envelopes

This PR adds TLS protocol support to Packetbeat, which is one of the most anticipated Packetbeat features. It doesn’t mean decrypting traffic, but it parses the initial handshake and extracts data like ciphers supported by the client and the server, the client and server certificate chains, the subject alternative name (SAN), validity dates, raw certificates, and so on. This data is super valuable for debugging TLS issues and also for intrusion detection and auditing.

The implementation also comes with support for the ALPN extension to TLS, which allows Packetbeat to detect, for example, whether HTTP/2 or HTTP/1 are used as an application protocol on top of the TLS connection.

This feature will be released in Packetbeat 6.1.

Filebeat: Docker JSON-file prospector

This PR adds an (experimental) dedicated Filebeat prospector for Docker logs written by the default JSON logging driver. Filebeat could already read Docker logs via the log prospector with JSON decoding enabled, but this new prospector makes things easier for the user. It abstracts the format, so there is no need to manually configure JSON decoding. Here is an example config, which captures the logs from a single container specified by its ID: 

prospectors:
   - type: docker
     containers.ids:
       - c3ec7a0bd9640151a768663b7e78c115d5b1a7f87fba572666bacd8065893d41

It also parses the timestamp from the JSON file, something that wasn’t possible with Filebeat alone (it required Logstash or Ingest Node). 

This new prospector will be released with Filebeat 6.1.

Other changes

Repository: elastic/beats

Affecting all Beats

Changes in master:

  • Adjust generated CSV dependency file #5618
  • Fix console color output on Windows #5612
  • Optional pipeline and output metrics #5586
  • Add `Get` method to `cfgfile.Registry` #5581
Metricbeat

Changes in master:

  • Fix failing perfmon test #5607
  • Add Windows service dashboard #5603
  • Foundation for Logstash Metricbeat module #5540
Filebeat

Changes in master:

  • Remove spurious ID method #5583
  • Remove prospector validation as not needed anymore #5537
  • Add experimental Docker json-file prospector #5402
Heartbeat

Changes in master:

  • Fix visualization in Heartbeat dashboards #5564
Auditbeat

Changes in master:

  • Remove redundant log prefix for auditbeat module (#5533) #5539
Testing

Changes in master:

  • Fix generator tests #5622
  • Update to Golang 1.9.2 #5569
  • Improve test stability #5567
  • Test new Kafka setup for more stable builds #5549

Changes in 6.0:

  • Test new Kafka setup for more stable builds #5549
  • Fix packetbeat flaky tests #5200
  • Improve test stability #5567
Documentation

Changes in master:

  • Fix Logstash defaults in output documentation #5620
  • Add Docker docs for Auditbeat #5619
  • Update filters to processors #5616
  • Fix repo steps in the Beats platform ref #5602
  • Fix experimental and beta flags for features #5591
  • Flag experimental and beta modules in the docs #5590
  • Add link to ES docs about type removal #5588
  • Metricbeat docs fixes #5578

Changes in 5.6:

  • Clarify docs for system core percentages #5565
  • Update multiline.asciidoc #5356

Changes in 6.0:

  • Fix prerelease links in apt/yum package docs #5641
  • Backport to 6.0: Doc changes in #5588, #5590, #5591, #5602 #5624
  • Hot fix link that was using current. #5580
  • Backport to 6.0: #5513 and #5496 #5574
  • Backport to 6.0: Fix getting started (#5326) #5573
  • Set docs version to 6.0.0 and released #5561

Repository: elastic/gosigar

Changes in master:

  • Changes to support CPU usage metric in Beats #83