Brewing in Beats: Enrich events with Kubernetes metadata

作者
Monica Sarbu

Welcome to Brewing in Beats! With this weekly series, we're keeping you up to date with what's new in Beats, including the latest commits and releases.

Enrich each event with Kubernetes metadata

Thanks to Vijay Samuel, there is a community contribution to add support for Kubernetes as a processor for Beats. The Kubernetes processor allows Beats to enrich events with metadata coming from the Kubernetes Pod from which the event originated. By default, the Kubernetes processor adds the following metadata:

  • pod name
  • pod namespace
  • container name and labels

Depending on the Beat, the  Kubernetes processor can add additional information. For example for Filebeat, it takes the source fields, extracts the container ID, and uses it to retrieve metadata about the Pod from which the log message originated.

This will be released as Beta in 6.0.

Monitor Elasticsearch with Metricbeat

A new module is added to Metricbeat for monitoring Elasticsearch. It exports metrics about:

  • Node info (node metricset) by interrogating the Cluster API of Elasticsearch to get cluster nodes information. This metricset only fetches the data from the _local node, so it must run on each Elasticsearch node.
  • Node stats (node_stats metricset) by calling Cluster API of Elasticsearch to get the cluster nodes statistics. This metricset only fetches the data from the _local node, so it must run on each Elasticsearch node.

The Elasticsearch module in Metricbeat exports only a few basic metrics, and for a better monitoring experience, we recommend you to use X-Pack monitoring.

This will be released as Beta in 6.0.

New community Beat: Kafkabeat

Kafkabeat is built on top of libbeat infrastructure to read the streaming events stored in Kafka and send them to Elasticsearch. An option would be to use Kafkabeat in the following scenario: Filebeat >> Kafka >> Kafkabeat >> Elasticsearch.

Other changes

Repository: elastic/beats

Affecting all Beats

Changes in master:

  • Change error to error.message #3987

Metricbeat

Changes in 5.x:

  • Make system process metricset honor cpu_ticks config option #3776
  • Adding support for custom http headers and TLS for metricbeat modules #3945
  • Fixing panic on prometheus collector when label has , #3947

Changes in master:

  • Mark kubelet module as beta #3990

Packetbeat

Changes in 5.x:

  • Add DNS dashboard for Packetbeat #3883
  • Add DNS Tunneling dashboard for Packetbeat #3884

Winlogbeat

Changes in 5.x:

  • Fix Winlogbeat bug affecting include_xml #3943

Filebeat

Changes in 5.x:

  • Filebeat syslog module: support for lines without a program name #3944
  • Don't stop Filebeat when modules + logstash are used together #3929

Packaging

Changes in 5.x:

  • Bump version in the 5.x branch to 5.5.0 #4007

Infrastructure

Changes in master:

  • reviewdog: ignore underscores in package names #4015

Documentation

Changes in 5.0:

  • Remove yellow box redirecting users to master for dev docs #3958
  • Remove outdated content from packetbeat dev guide #3963

Changes in 5.3:

  • Update curl examples to include content-type #4020

Repository: elastic/gosigar

Changes in master:

  • Remove audit related code #70