Beats 6.3.0 Released

We're pleased to release Beats 6.3.0. Enjoy!

• Download link
• Release notes

Here are the highlights:

Configure Beats via Kubernetes/Docker annotations

You probably got used to the fact that each 6.x minor release is making Kubernetes monitoring even nicer, so we couldn’t disappoint you with this one. The goodie that we have prepared in 6.3 is that you can now configure Beats not only via the Beats manifest files but also via Kubernetes annotations (or Docker labels) that are assigned to the monitored Pods.

To use this, enable Autodiscover with Hints enabled in the Beats configuration:

filebeat.autodiscover:
  providers:
    - type: kubernetes
      hints.enabled: true

Then, when creating a new deployment, you can use Kubernetes annotations to “hint” to the Beats how the new deployment should be monitored. For example, if you deploy an Nginx container, you can add the following Kubernetes annotations to it:

 annotations:
   co.elastic.logs/module: nginx
   co.elastic.logs/fileset: access
   co.elastic.metrics/module: nginx
   co.elastic.metrics/period: 10s
   co.elastic.metrics/hosts: "${data.host}:8080"

When Filebeat and Metricbeat receive the notification that a new Pod is created, they retrieve the annotations above and start the respective Nginx modules with the given configuration. Any Beats settings can be modified this way, and it’s a great way of keeping the logging and monitoring configuration near the application deployment configuration.

Filebeat Syslog input

Starting with 6.3, you can send logs to Filebeat using the syslog protocol, over UDP or TCP. The internal Syslog parser is a state machine built with Ragel, which makes it both efficient and flexible enough to deal with the usual variations from the Syslog standard. For the moment, RFC 3164 is supported. RFC 5424 is planned for a future release.

Here is a sample configuration:

filebeat.inputs:
- type: syslog
    protocol.tcp:
      # The host and port to receive the new event
      host: "localhost:9000"
      # Character used to split new message
      #line_delimiter: "\n"
      # Maximum size in bytes of the message received over TCP
      #max_message_size: 20MiB
      # The number of seconds of inactivity before a remote connection is closed.
      #timeout: 300s

Spooling to disk (Beta)

Starting with this release, all Beats get an optional disk queue where they can spool their events. Any event to be published will be written to the spool first and only forwarded to Elasticsearch or Logstash afterwards. With spooling to disk, events not yet published are still available after Beats have been restarted or in case outputs have been stalled for a long time due to network issues.

The spooling to disk feature can be used to provide “at least once” guarantees for Beats like Metricbeat and Auditbeat, that don’t read from a natural queue (like Filebeat does when tailing files).

This feature is currently in Beta, so please avoid it in production, but do try it on non-production systems and give us feedback on it.

Add host metadata

Everyone loves Beats processors like add_cloud_metadata or add_kubernetes_metadata. Add a line in your Filebeat or Metricbeat configs, and suddenly all events are populated with things like the instance type, region, kubernetes namespace, and so on. So we’ve asked ourselves why don’t we have something similar for non-cloud non-kubernetes hosts.

The new add_host_metadata processor adds the following fields to your event:

• host.name
• host.id
• host.architecture
• host.os.platform
• host.os.version
• host.os.family

These fields can be used in any filtering and aggregation in Kibana.

To support the new fields, we had to add a new object to the index template. This change may cause mapping conflicts for some Logstash users, even if you aren’t using the new processor. For more information, see the Beats breaking changes documentation. 

Wait, there’s more: IIS, Logstash, MongoDB, Etcd modules

The 6.3 release increases our collection of Filebeat and Metricbeat modules. Filebeat comes with the following new modules:

• MongoDB
• Logstash
• IIS