Anomaly detection, one of the Elastic machine learning features, has been a great way of signaling when there’s an issue in a dataset. Until now, there’s been no easy way to focus in on anomalies for a particular field. This is especially true when the field of interest is not one of the top N shown in the results view.
The release of Kibana 7.2 introduces a new query bar in the Machine Learning app in Kibana to make it easier for users to search the anomaly results for specific potential influencers.
Using the query bar with NGINX events
The data used in the following example contains anonymized data from elastic.co web access logs processed by Filebeat’s NGINX module configuration (14,479,391 events collected over 38 days).
In this example, we’ve created a job to detect unusual activity in the number of incoming requests — something that could be indicative of an attack.
A normal user is limited by how quickly they can manually navigate to website pages, whereas an attacker or bot may be able to flood a website with requests. Assuming the majority of clients are normal users, we can detect atypical behavior by looking at unusually high event rate for a client:
The results show several anomalous clients; 126.96.36.199 especially seems to show anomalous activity over a longer period of time.
In this case, we are interested in only viewing results for that specific IP. Using the query bar, we can narrow the results for 188.8.131.52.
With the query bar, we now have the ability to search through anomaly results to focus in on one or more selected influencers. The query bar suggests influencer field names and values for selected jobs as it receives input. Auto-suggest makes it easy to view possible fields and values we can search by; we’re no longer limited to seeing only the top ten shown in the results view.
The query bar accepts input in KQL syntax. Users can combine queries using AND or OR keywords and can filter by field value only.
Wildcard queries are supported. Searching for
source.address: 185*, for example, narrows the displayed results to all IP values beginning with
If the influencer of interest is listed in Top Influencers, clicking the plus icon next to it will add that field/value combination to the query bar and apply the search. Removing the search can be accomplished by clicking the minus icon. This can also be done from the results table.
At Elastic, we like to build search into everything we do. The query bar provides an easy way to search through anomalies for one or more particular influencers, allowing for easier result navigation. This is especially useful when a field is not one of the top N shown in the results view.