Brewing in Beats: Monitor Kibana logs with Filebeat
Welcome to Brewing in Beats! With these weekly series, we're keeping you up to date with what's new in Beats, including the latest commits and releases.
Did you know that Beats 6.2 is already available? Try it and let us know what you think. If you are curious to see the Beats in action, check out the Getting Started with Beats webinar.
Filebeat module for Kibana logs
As part of our quest to make it easy to monitor the Elastic stack using the Elastic stack, we have merged a Filebeat fileset for the Kibana logs. The Kibana log is in the JSON format which makes decoding easy but at the same time it provides many fields which are not necessarily known in advance. To solve this issue the data is put under kibana.log.meta
and the most common known fields are picked into kibana.log.*
or directly put into the global field like message
. This module doesn’t yet have dashboards, but we do plan to add them. The module will be released with version 6.4.
Filebeat: The TCP input accepts TLS connections
The recently added Filebeat TCP input can now receive events over a TLS connection. The input uses the same configuration conventions and code as the outputs that support TLS communication (Elasticsearch and Logstash).
The configuration looks like this:
host: "localhost:9000" ssl.enabled: true ssl.verification_mode: full # default ssl.supported_protocols: [TLSv1.1] ssl.cipher_suites: [] ssl.certificate_authorities: ["/etc/cacert"] ssl.certificate: /etc/mycert.crt ssl.key: /etc/mycert.key ssl.client_authentication: required
Note the ssl.client_authentication
option, which can be used to authenticate the clients. Valid options are required
, optional
, and none
.
This feature will be available in Filebeat 6.4.
Metricbeat: Kubernetes apiserver metricset
We have improved our existing Prometheus helper in Metricbeat to ease the process of implementing modules or metricsets fetching metrics in the Prometheus format (example). As the first use for it, we introduced a new Kubernetes apiserver metricset, to monitor Kubernetes API Server usage. The new metricset will be included in Metricbeat 6.4.
All Beats changes
Repository: elastic/beats
Metricbeat
Changes in master:
- Elasticsearch pending tasks metricset #7144
- Metricbeat: Add maxmemory to redis info metricset #7127
- Move parts of metricset code to Elasticsearch module #7103
- Add x-pack data for Elasticsearch shard metricset #7097
- Unify default module configurations #6908
Filebeat
Changes in master:
- Fix permissions of generated Filebeat filesets #7140
- Fix permission on Filebeat Kibana module files #7136
- Keep event when
add_docker_metadata
fails to extract container id #7133 - Fix Filebeat module generator substitution #7121
- Filebeat module fields.yml generator fix #7117
- Add pod-uid support for add_kubernetes_metadata #7072
- Features: Allow the TCP Input to receive events over a TLS connection #7056
- Filebeat Kibana module log fileset #7052
- Allow
_
and-
in Filebeat module name hints #6901
Auditbeat
Changes in master:
- Don't instantiate MetricSets with the setup command #7112
Dashboards
Changes in master:
Testing
Changes in master:
- Fix permissions on yaml files and add check #7137
- Move filebeat module tests to parameterized tests #7116
- Move common RabbitMQ testing code to its own package #7106
- Use minikube on travis for K8S integration tests #7096
Packaging
Changes in master: