Brewing in Beats: New configuration layout

Welcome to Brewing in Beats! With this weekly series, we're keeping you up to date with what's new in Beats, including the latest commits and releases.

Move to a modules.d layout for the configuration files

This PR moves the per-module configuration in individual files inside a modules.d folder, and adds commands to list/enable/disable modules. As the number of modules in Metricbeat and Filebeat keeps growing, this makes the configuration more user friendly.

Example commands:

 
$ metricbeat modules list 
$ metricbeat modules enable redis 
$ metricbeat modules disable redis

The all-in-one configuration file is still supported and included in the packages (as beatname.reference.yml).

This functionality will be released in 6.0.0-beta1.

Test command

This PR adds a new test command, which enables checking that the Elasticsearch and Logstash outputs are correctly configured, and which is extensible for future tests. We hope that this will simplify troubleshooting for many of our users.

Here is an example output:

$ filebeat test output 
elasticsearch: https://61fc3e1983ebd7b13e9d6098e4651345.us-east-1.aws.found.io:9243...
  parse url... OK
  connection...
    parse host... OK
    dns lookup... OK
    addresses: 54.235.139.222, 54.221.244.80, 54.243.122.128, 54.204.28.14, 54.221.223.25, 54.243.118.44, 54.243.142.98, 54.235.122.205
    dial up... OK
  TLS...
    security: server's certificate chain verification is enabled
    handshake... OK
    TLS version: TLSv1.2
    dial up... OK
  talk to server... OK
  version: 5.5.0

This functionality will be released in 6.0.0-beta1.

Include Kibana dashboards in the individual Beats packages

Before this PR, the sample dashboards were packaged in their own zip file, and the Beats (or the import_dashboards program) were downloading them on the fly at load time. The URL was computed automatically, so the download was in most cases transparent for the user. However, for non-released version, the URL had to be manually specified, which sometimes caused issues. It was also a source of issues in case of installations without Internet access. So we’ve decided to simplify things by including the dashboards in the Beats packages themselves.

This functionality will be released in 6.0.0-beta1.

Configuration reloading for Filebeat modules

This PR adds support for live module configuration reloading, similar with what we have for the Metricbeat modules and Filebeat prospectors. A folder can be watched for configuration file changes, and the modules reloaded automatically in case a configuration is modified. This means one will be able to enable/disable modules (using the filebeat modules commands) without having to restart the Beat.

This functionality will be released in 6.0.0-beta1.

New Community Beat: Mqttbeat

Created by @nathan-K-, Mqttbeat can act as an MQTT broker and index and received message into Elasticsearch.

New Community Beat: Prometheusbeat

Create by @boernd, Prometheusbeat allows forwarding the metrics from a Prometheus instance to Elasticsearch for long term storage and visualization. It works by opening an HTTP endpoint, compatible with the remote_write functionality of Prometheus. While we have support for scrapping Prometheus endpoints in Metricbeat, this is different in the sense that it is meant to be used side by side with the Prometheus server.

Use UTC when computing index names

Fixed a bug where index names that contain dates may be incorrect when the system timezone is not in UTC.

Other changes

Repository: elastic/beats

Filebeat

Changes in 5.x:

  • New ML configurations and dashboards for the Filebeat Nginx module #4609

Changes in master:

  • Update ML datafeed settings #4676
  • Fix race in filebeat shutdown with timeouts #4666
  • Move filebeat to new publisher pipeline #4644
  • New ML configurations and dashboards for the Filebeat Nginx module #4609
  • Cleanup scan config options. Mark it experimental. #4564
Documentation

Changes in 5.4:

  • Backport to 5.4: #4530 #4547 #4628 #4630

Changes in 5.3:

  • Backport to 5.3: #4530 #4542 #4547 #4628 #3581 #4632

Changes in master:

  • Forward port to master: #4542 and #3581 #4634
  • Fix config option name #4628

Changes in 5.5:

  • Backport to 5.5: #4530 #4542 #4547 #4628 #4617 #4629
Affecting all Beats

Changes in master:

  • Add more info to debug Kibana client error #4669
  • Added 'result' field to Elasticsearch QueryResult #4668
  • Unified Pipeline/output metrics #4663
  • Normalize times to common.Time with UTC time zone #4658
  • Update broker/pipeline setup #4650
  • Remove yacc source file from vendor #4649
  • Publisher pipeline enhancements #4641
  • Log elasticsearch connect error #4639
  • Remove the HTTP endpoint from configuration #4615
  • Add auth support in Kibana #4597
  • Assume no xpack installed on 400 answer #4593
Auditbeat

Changes in master:

  • Auditbeat file module improvements #4636
  • Generate Auditbeat module config from template #4582
    Testing

    Changes in master:

    • Disable logging for module command tests #4645
    • Non-xpack snaphosts for testing #4592
    Dashboards

    Changes in master:

    • Include Kibana dashboards in the individual Beats packages #4675
    • Create Kibana index after checking ES version #4662
    • New beats-dashboards snapshot URL #4616
    • Save a single dashboard per json file #4614
    • Add make import-dashboards #4613
    • Don't save in git the index-pattern as it's generated #4607
    • Fix the path of the Kibana dashboards 6.x #4602
    Packaging

    Changes in master:

    • Remove os.path.realpath from deploy script #4640
    • Rename heartbeat packages #4601
    Winlogbeat

    Changes in master:

    • Removed validation of top-level config keys from Winlogbeat #4657
    Infrastructure

    Changes in master:

    • Add support for keyword in dynamic fields #4606
    Metricbeat

    Changes in master:

    • Add tests to verify module.d file perms and owner #4653
    • Remove clean target from Metricbeat #4646
    • correct format for memory module in dockerbeat #4619
    • Use the noxpack version of the env for Metricbeat #4605
    • Perfmon wildcard queries #4502
    • Used memory field corrected (#4461) #4462
      Packetbeat

      Changes in master:

      • Delete pfring code #4608
      • Use packetbeat protocols to publish beat.Event to new pipeline #4599