Brewing in Beats: Test command for Metricbeat modules

Welcome to Brewing in Beats! With this weekly series, we're keeping you up to date with what's new in Beats, including the latest commits and releases.

Test command for Metricbeat modules

This PR adds a new test command for the Metricbeat modules. It goes over all the configured modules and executes a fetch for each, reporting the result back to the user, or any error that happened during the fetch.

$ ./metricbeat test modules
system...
  process_summary...OK
    result: 
    {
     "_namespace": "process.summary",
     "idle": 0,
     "running": 1,
     "sleeping": 236,
     "stopped": 0,
     "total": 238,
     "unknown": 1,
     "zombie": 0
    }
nginx...
  stubstatus...
    error... ERROR error making http request: Get http://127.0.0.1/server-status: dial tcp 127.0.0.1:80: getsockopt: connection refused

This functionality will be released in 6.0.0-beta1.

Load dashboards via Kibana API in version 5.6

In the 5.x versions, the Beat loads the dashboards directly in the Elasticsearch .kibana index, and uses the import_dashboards script for that (available in the Beat package). Starting with 6.0 version, if the Kibana version is greater than 5.6, then the Beat uses the Kibana API to load the dashboards, otherwise it loads them directly to Elasticsearch.

The PR adds the option in the 5.6 version to load the dashboards via the Kibana API from the import_dashboards script, if requested by the user. This would make it possible for Beats 5.6 to import the dashboards in the future Kibana 6.x versions. The default is still to insert the dashboards directly in Elasticsearch, but one can provide a Kibana URL, in which case the Kibana API is used.

This functionality will be released only in 5.6.

Other changes

Repository: elastic/beats

Affecting all Beats

Changes in 5.x:

  • Normalize times to common.Time with UTC time zone #4658

Changes in master:

  • Add pattern option in the fields.yml #4731

Changes in 5.5:

  • Normalize times to common.Time with UTC time zone #4658
Metricbeat

Changes in master:

  • Update metricbeat to use new publisher API #4699
Filebeat

Changes in 5.x:

  • Change grok pattern to fetch correct IP from X-Forwarded-For list #4351
  • Nginx module: use first not private IP address as remote_ip #4417
  • Update ML datafeed settings #4676

Changes in master:

  • Added FBM Redis sample configuration files #4718
Winlogbeat

Changes in master:

  • Fallback on LevelRaw If the Level is not in the RenderingInfo section of the event #4257
  • Move Winlogbeat to new publisher pipeline #4690
Auditbeat

Changes in master:

  • Mark audit metricsets as beta #4727
Dashboards

    Changes in master:

    • Few fixes for load dashboards via Kibana API #4721
    Machine learning jobs

    Changes in 5.x:

    • Check ML job against a minimal version of ES #4681

    Changes in master:

    • Check ML job against a minimal version of ES #4681
    Testing

    Changes in master:

    • Use the beta1-snapshot in the testing env #4711
    • Improve Auditbeat file test on Windows #4698

    Changes in 5.5:

    • Update testing env to 5.5.1 #4722
    Infrastructure

    Changes in master:

    • Generate xunit report for system test #4686
    Documentation

    Changes in 5.x:

    • Remove invalid links to releases #4707

    Changes in master:

    Changes in 5.5:

    • Close changelog for 5.5.1 #4720

    Repository: elastic/go-libaudit

    Changes in master:

    • Apply hex decoding to CWD #11