As datasets increase in size and complexity, the human effort required to inspect dashboards or maintain rules for spotting infrastructure problems, cyber attacks, or business issues becomes impractical. Elastic machine learning features such as anomaly detection and outlier detection make it easier to notice suspicious activities with minimal human interference.
If you have a basic license, you can use the Data Visualizer to learn more about your data. In particular, if your data is stored in Elasticsearch and contains a time field, you can use the Data Visualizer to identify possible fields for anomaly detection:
[experimental] This functionality is experimental and may be changed or removed completely in a future release. Elastic will take a best effort approach to fix any issues, but experimental features are not subject to the support SLA of official GA features. You can also upload a CSV, NDJSON, or log file (up to 100 MB in size). The Data Visualizer identifies the file format and field mappings. You can then optionally import that data into an Elasticsearch index.
The Elastic machine learning anomaly detection feature automatically models the normal behavior of your time series data — learning trends, periodicity, and more — in real time to identify anomalies, streamline root cause analysis, and reduce false positives. Anomaly detection runs in and scales with Elasticsearch, and includes an intuitive UI on the Kibana Machine Learning page for creating anomaly detection jobs and understanding results.
If you have a license that includes the machine learning features, you can create anomaly detection jobs and manage jobs and datafeeds from the Job Management pane:
The Anomaly Explorer and Single Metric Viewer display the results of your anomaly detection jobs. For example:
You can optionally add annotations by drag-selecting a period of time in the Single Metric Viewer and adding a description. For example, you can add an explanation for anomalies in that time period or provide notes about what is occurring in your operational environment at that time:
In some circumstances, annotations are also added automatically. For example, if the anomaly detection job detects that there is missing data, it annotates the affected time period. For more information, see Handling delayed data. The Job Management pane shows the full list of annotations for each job.
The Kibana machine learning features use pop-ups. You must configure your web browser so that it does not block pop-up windows or create an exception for your Kibana URL.
Data frame analyticsedit
The Elastic machine learning data frame analytics feature enables you to analyze your data using outlier detection and regression algorithms and generate new indices that contain the results alongside your source data.
If you have a license that includes the machine learning features, you can create outlier detection data frame analytics jobs and view their results on the Analytics page in Kibana. For example:
For more information about the data frame analytics feature, see Machine learning data frame analytics.