Beats 5.1.1 released

Today we are pleased to announce that Beats 5.1.1 was released. This is the latest stable version and it comes with a few goodies like support for monitoring Kafka and Docker containers with Metricbeat, and other good reasons to upgrade.

Latest stable release:

Docker module in Metricbeat

Starting with Beats 5.0.0, you are able to collect periodically metrics about your containers by reading cgroup data directly from /proc. To read the cgroup data, you need to mount the proc filesystem and the cgroup filesystem from the host inside of the container.

The benefit of collecting container metrics directly from cgroup is that it works for all types of container technologies you are using, not only for Docker containers, as Metricbeat reads directly from the cgroup pseudo filesystem provided by the Linux kernel. However, this approach alone is not able to get the container metadata like the Docker container name and labels.

More details about the cgroup data you can find in the blog post.

Another way to collect the information about your Docker containers is by interrogating periodically the Docker API. You can get similar details about your containers as by reading cgroup data, and includes:

  • Per container memory & CPU usage
  • Per container network stats like packets and bytes in/out on each network interface
  • Per container diskio stats like reads vs writes
  • General information like the command running in the container or the Docker image
  • Docker containers status like list of your containers, including their names and labels

With Beats 5.1.1, we are releasing a new Docker module in Metricbeat, that collects periodically information about your Docker containers by querying the Docker API. The Docker module is based on the code of a community Beat, Dockbeat, created by Ingensi and kindly migrated to a module in Metricbeat by Douae Jeouit.

The advantage of querying Docker API over reading the cgroup data is that it’s easier to set-up as typically the only required configuration is the address of the Docker server.

For now, the Docker module is marked as experimental, meaning that we reserve the right to do any breaking changes.

Docker container dashboards.png

Kafka module in Metricbeat

Beats 5.1.1 comes with support for monitoring the Kafka nodes thanks to a new Metricbeat module. The Kafka module is inspired from the community Beat, Kafkabeat, created by Dale McDiarmid from Elastic.

The Kafka module connects to the local Kafka node and reads periodically details about the partitions, like topic names, available event offsets, and replication status.

The Kafka module is experimental, and we're already looking into adding more Kafka metricsets to it.

New processor: add cloud metadata

You are now able to enrich each event generated by Beats with instance metadata from the machine’s hosting provider by configuring the add_cloud_metadata processor. When the Beat starts, it automatically detects the hosting provider and caches the instance metadata.

There are three cloud providers supported at the moment:

  • Amazon Elastic Compute (EC2)
  • Digital Ocean
  • Google Compute Engine (GCE)

If the add_cloud_metadata processor is configured, the event is enhanced with the `meta` field. Here is an example:

  "meta": {
    "cloud": {
      "availability_zone": "us-east-1c",
      "instance_id": "i-4e123456",
      "machine_type": "t2.medium",
      "provider": "ec2",
      "region": "us-east-1"

New processor: decode JSON from arbitrary fields

Filebeat can decode JSON encoded logs already since version 5.0, but it didn’t have so far a good way of handling more complex JSON-in-JSON situations. Such events can happen, for example, if one is using the Docker JSON logging driver (which wrapps every line in a JSON object) and the application also outputs in JSON.

Thanks to a community contribution by @suraj-soni, libbeat got a new processor, called decode_json_fields, that can decode JSON from an arbitrary field. This processor is useful in Filebeat, but can be used also, for example, in the Packetbeat configuration to decode the body of HTTP requests.

While at first sight the JSON processor is more general than the native JSON decoding we have in Filebeat, and it could replace it entirely, it’s important to note that the native JSON decoding is applied before the multiline processing. This means that if a multiline event is scattered across multiple JSON objects, Filebeat can first decode the JSON objects and then stick together the related lines.


If you want to make use of the new features added in Beats 5.1.1, please download the latest stable version, install it, and let us know what you think on Twitter (@elastic) or in our forum.

  • We're hiring

    Work for a global, distributed team where finding someone like you is just a Zoom meeting away. Flexible work with impact? Development opportunities from the start?