Editor's Note (September 7, 2018): This post refers to X-Pack. Starting with the 6.3 release, the X-Pack code is now open and fully integrated as features into the Elastic Stack.
Today we are pleased to announce the Beats 5.6 release, the latest stable release of the 5.x series.
Before going through the highlights, let’s get those handy links out of the way:
Load Machine Learning jobs from Filebeat modules
Filebeat 5.6 release comes with the support to load Machine Learning (ML) job configurations directly from the Filebeat modules. This means that besides Kibana dashboards and Elasticsearch Ingest Node pipelines, Filebeat modules can also contain ML configurations to be used for anomaly detection.
Survey: With this feature, we are further enriching the modules experience for our community. Which Filebeat & Metricbeat modules should we target next to include preconfigured machine learning jobs? Let us know by filling this short survey. It won't take more than 5 minutes, we promise.
As part of this release, Filebeat comes with five ML job configurations which were already tuned to work perfectly with the data collected by the Nginx module. If you have Xpack installed and ML enabled, simply run:
filebeat -e -modules=nginx -setup
And you will see the following job configurations created for you:
- Detect unusual visitor rate
- Detect unusual response code rates
- Detect low request rate
- Detect unusual IP address - high distinct count of urls
- Detect unusual IP addresses - high request rates
After you let Filebeat ingest some data, simply click the Run button on the ML jobs, and BOOM: anomaly detection for your access logs.
And that’s not all! For any incident detected by ML, you can drill down in dedicated Kibana dashboards which are ready made for you to investigate these incidents.
Preparing for a smooth transition to 6.0
If you are on a 5.x version of Beats, you will need to upgrade to 5.6 before making the jump to 6.0. This is because we’ve made some changes to 5.6 to smoothen the upgrade experience.
_all.norms setting is no longer disabled in the Beats 5.6 mapping templates. This is because Elasticsearch is removing support for the
_all field in 6.0, and having settings for it in the template is problematic at upgrade time.
Kibana 6.0 will use a new internal format and a new API for importing and exporting dashboards. To make sure Beats 5.6 will work well with Kibana 6.0, we added support to import the dashboards via the new API in this release already. For this, you can run the `import_dashboards` script with the
./scripts/import_dashboards -kibana http://localhost:5601
If Kibana 5.x is used, the dashboards are imported directly into Elasticsearch.