Brewing in Beats: Monitor Kibana logs with Filebeat

Welcome to Brewing in Beats! With these weekly series, we're keeping you up to date with what's new in Beats, including the latest commits and releases.

Did you know that Beats 6.2 is already available? Try it and let us know what you think. If you are curious to see the Beats in action, check out the Getting Started with Beats webinar.

Filebeat module for Kibana logs

As part of our quest to make it easy to monitor the Elastic stack using the Elastic stack, we have merged a Filebeat fileset for the Kibana logs. The Kibana log is in the JSON format which makes decoding easy but at the same time it provides many fields which are not necessarily known in advance. To solve this issue the data is put under kibana.log.meta and the most common known fields are picked into kibana.log.* or directly put into the global field like message. This module doesn’t yet have dashboards, but we do plan to add them. The module will be released with version 6.4.

Filebeat: The TCP input accepts TLS connections

The recently added Filebeat TCP input can now receive events over a TLS connection. The input uses the same configuration conventions and code as the outputs that support TLS communication (Elasticsearch and Logstash).

The configuration looks like this:

  host: "localhost:9000"
  ssl.enabled: true
  ssl.verification_mode: full # default
  ssl.supported_protocols: [TLSv1.1]
  ssl.cipher_suites: []
  ssl.certificate_authorities: ["/etc/cacert"]
  ssl.certificate: /etc/mycert.crt
  ssl.key: /etc/mycert.key
  ssl.client_authentication: required

Note the ssl.client_authentication option, which can be used to authenticate the clients. Valid options are required, optional, and none.

This feature will be available in Filebeat 6.4.

Metricbeat: Kubernetes apiserver metricset

We have improved our existing Prometheus helper in Metricbeat to ease the process of implementing modules or metricsets fetching metrics in the Prometheus format (example). As the first use for it, we introduced a new Kubernetes apiserver metricset, to monitor Kubernetes API Server usage. The new metricset will be included in Metricbeat 6.4.

All Beats changes

Repository: elastic/beats

Metricbeat

Changes in master:

  • Elasticsearch pending tasks metricset #7144
  • Metricbeat: Add maxmemory to redis info metricset #7127
  • Move parts of metricset code to Elasticsearch module #7103
  • Add x-pack data for Elasticsearch shard metricset #7097
  • Unify default module configurations #6908
Filebeat

Changes in master:

  • Fix permissions of generated Filebeat filesets #7140
  • Fix permission on Filebeat Kibana module files #7136
  • Keep event when add_docker_metadata fails to extract container id #7133
  • Fix Filebeat module generator substitution #7121
  • Filebeat module fields.yml generator fix #7117
  • Add pod-uid support for add_kubernetes_metadata #7072
  • Features: Allow the TCP Input to receive events over a TLS connection #7056
  • Filebeat Kibana module log fileset #7052
  • Allow _ and - in Filebeat module name hints #6901
Auditbeat

Changes in master:

  • Don't instantiate MetricSets with the setup command #7112
Dashboards

Changes in master:

  • Update export_dashboards.go #7101
  • Fix default_field for metricbeat #7015
Testing

Changes in master:

  • Fix permissions on yaml files and add check #7137
  • Move filebeat module tests to parameterized tests #7116
  • Move common RabbitMQ testing code to its own package #7106
  • Use minikube on travis for K8S integration tests #7096
Packaging

Changes in master:

  • Remove arm packages from default packages to be built #7122
  • Fix snapshot packaging for packetbeat and winlogbeat #7120
  • Make beat.reference.yml optional. #7092