Brewing in Beats: Support for configuration reloading

Welcome to the weekly Brewing in Beats ! With this weekly series, we are keeping you up to date with all the changes in Beats, including the latest commits, releases and other learning resources.

New community Beat: Prombeat

Prombeat periodically scrapes time series from a Prometheus server and sends them to Elasticsearch directly or via Logstash. It uses the query interface of Prometheus to access the metrics stored in the Prometheus server.

Configuration reloading in libbeat

Dynamic configuration reloading was first introduced as a per-Beat implementation. Metricbeat is able to dynamically reload its modules, Filebeat is able to reload the prospectors, and Heartbeat is also able to reload the list of monitored targets. With this change, the reloading capabilities are abstracted into libbeat to be a reusable component.

Load Kibana dashboards with the -setup option

With this change, filebeat -setup loads the sample Kibana dashboards, being the rough equivalent of running: scripts/import_dashboards && filebeat. This works for all Beats, not only Filebeat. The import_dashboards script will continue to be shipped in the Beat package in 5.x, to avoid breaking compatibility. Code-wise, the two share most of the code. All flags available for the import_dashboard are available as configuration options. For example, to load the dashboards from a directory you can do:

filebeat -e -setup -E "dashboards.dir=_meta/kibana"

Or to load the snapshot version of the dashboards:

filebeat -e -setup -E "dashboards.snapshot=true"

Unify Beat generators

Generating a Beat is similar with generating a Beat based on Metricbeat, so we decided to unify them to use a single script instead of two different scripts. The templates used for generating the Beat are now available under the beats/generator directory.

Other features and fixes

  • Cleanup CEPH module #3455 #3509
  • Fix leaking go routine in docker stats fetching #3492
  • Add Df metricset for CEPH module #3499
  • Enable by default fetching cgroup metrics on Linux #3519
  • Gather module configuration fragments and add configuration samples for all modules  #3488
  • Use the Beat version in the Ingest Node pipeline ID #3516
  • Check if Ingest Node pipeline exists before loading #3522
  • Update Windows paths for chocolatey installed packages #3524
All Beats
  • Enhancements of the monitoring package #3478 #3510
  • Add support for long in dict-type #3515
  • Disable date_detection in index templates needed for dynamic metrics #3528

    • We're hiring

      Work for a global, distributed team where finding someone like you is just a Zoom meeting away. Flexible work with impact? Development opportunities from the start?