Brewing in Beats: Central config, Journalbeat, Functionbeat, InfraOps UI, Logs UI

Welcome to Brewing in Beats! With these weekly series, we're keeping you up to date with what's new in Beats, including the latest commits and releases.

A lot of major features were merged in the last couple of weeks. The 6.5.0 release is going to have tons of new features and it will be a big milestone for our team.  

Central Config Management

Last week we merged Central Config Management for Beats. The idea is that a Beat needs to enroll with Central Config Management first. Once enrolled, one can assign tags to the Beat, with each tag having a multitude of configurations available. This way configurations are independent of the Beats and can be combined at will. The Beat makes use of auto-discovery to automatically load and apply configuration changes made in the UI.

Image from iOS.jpg

Unfortunately, we can not collect and display logs yet, but the UI signal errors if a Beat has not been available for some time or did signal an Error based on the last configuration update.

As of now only Filebeat and Metricbeat are supported. One can configure the Metricbeat Modules, Filebeat Modules and Inputs, and the Beat Output.

Screen Shot 2018-11-01 at 10.33.46 AM.png

Screen Shot 2018-11-01 at 10.35.55 AM.png

Journalbeat

Last week we merged Journalbeat into master. It is a new Beat for shipping journald logs. Journalbeat can tail logs written to journald files, directories or the local system journal. When reading from a directory or from the local journal, then journals and logs will be merged into one stream of logs.

Next to plain logs, journald provides additional meta data added to the logs. For example the executable name, process id, user id, priority, and facility. This allows Journalbeat users to configure rather a complex filtering of log messages via processors, reducing the number of logs that need to be published.

Journalbeat will be released as experimental in 6.5

Functionbeat

Last week we merged Functionbeat into master as well. Functionbeat provides support for configuring Beats based functions running in serverless environments. The initial release runs in AWS Lambda, collecting events based on triggers of: Cloudwatch Logs and SQS, Functionbeat also supports Kinesis API Gateway but doesn’t expose CLI function to deploy them.

Functionbeat will be released as Beta in 6.5.

InfraOps UI

Two weeks ago, we merged the InfraOps UI into Kibana master. InfraOps UI is a new Kibana app that shows you an overview of your infrastructure. The Kibana menu bar is enriched with two icons, one for InfraOps UI and one for Logs UI.

Below is a screenshot of the infrastructure overview, using a diagram that we call "waffle map". Each box in the waffle map can be a host/VM instance, a docker container, or a pod. The view comes with a search bar to filter the boxes that have a specific characteristic and with a "group by" functionality to group the boxes by certain fields like availability zone or the Kubernetes namespace. The color of each box represents the value of the selected metric, in the case of the screenshot, memory usage.

image (10).png

If you click on one of these boxes, you can get more details about the host itself as in the screenshot below. Here you can see not only  metrics like CPU usage, load, memory usage, and others, but also the logs that are coming from that host. If the host is a Kubernetes node, you can also see Kubernetes specific metrics.

image (16).png

InfraOps UI will be released as Beta in 6.5

Logs UI

Two weeks ago we merged Logs UI in master. Currently, our users are using the Kibana Discover page to visualize logs, but it lacks features like Infinite scrolling or live streaming that are making easier for the user to follow the logs. The new Logs UI is a dedicated Kibana app that comes to solve these issues.

image (12).png

The Logs UI will be released as Beta in 6.5

Kafka Metricbeat module to be released as GA

After merging the dashboard, we have decided that Metribceat Kafka module is ready to be released as GA.  

We have been also testing Filebeat with Kafka 2.0.0, so we can give a complete solution for this version starting on 6.5.0.

Support for Prometheus untyped metrics

Thanks to the contribution of @mvollman (#8681), Metricbeat will be able to collect untyped metrics from Prometheus exporters starting on 6.6.0.

All changes

Repository: elastic/beats

Metricbeat

Changes in master:

  • Unify initialization code in kafka metricsets #8705
  • Add support for untyped prometheus metrics #8681
  • Consistency in Elastic stack metricsets' code #8308

Changes in 6.x:

  • Unify initialization code in kafka metricsets #8705
  • Add support for untyped prometheus metrics #8681
Packetbeat

Changes in master:

  • Support parsing TLS 1.3 supported_versions extension (#8647) #8772
Filebeat

Changes in 6.x:

  • Reduce casting between []byte and string in CRI log processing #8424
Heartbeat

Changes in 6.5:

  • Fix failing heartbeat tests due to port allocations and timeouts #8738

Changes in master:

  • Fix failing heartbeat tests due to port allocations and timeouts #8738

Changes in 6.x:

  • Fix failing heartbeat tests due to port allocations and timeouts #8738
Functionbeat

Changes in master:

  • Use a goformation fork to go around MAX_PATH limitation #8859
Processors

Changes in master:

  • Dissect tag on parsing error #8751
Dashboards

Changes in 6.5:

  • mkdir for extracted dashboard files #8801

Changes in 6.4:

  • mkdir for extracted dashboard files #8801

Changes in master:

  • mkdir for extracted dashboard files #8801

Changes in 6.x:

  • mkdir for extracted dashboard files #8801
Testing

Changes in master:

  • Temporary disable x-pack/functionbeat on windows. #8765
Packaging

Changes in master:

  • Make sure that packages.yml use the same files definition #8836
Documentation

Changes in 6.5:

  • Fix heartbeat docs for config reloading #8839
  • Bump doc versions for 6.5 branch #8752

Changes in master:

  • Fix heartbeat docs for config reloading #8839
  • add NFS to shared protocol list #8274

Changes in 6.x:

  • Fix heartbeat docs for config reloading #8839

Repository: elastic/go-ucfg

Changes in master:

  • Prepare for v0.6.5 #123
  • Adding a ResolverNOOP #122