Tech Topics

Find influencers faster with the machine learning Anomaly Explorer query bar

Anomaly detection, one of the Elastic machine learning features, has been a great way of signaling when there’s an issue in a dataset. Until now, there’s been no easy way to focus in on anomalies for a particular field. This is especially true when the field of interest is not one of the top N shown in the results view.

The release of Kibana 7.2 introduces a new query bar in the Machine Learning app in Kibana to make it easier for users to search the anomaly results for specific potential influencers.

Using the query bar with NGINX events

The data used in the following example contains anonymized data from web access logs processed by Filebeat’s NGINX module configuration (14,479,391 events collected over 38 days).

In this example, we’ve created a job to detect unusual activity in the number of incoming requests — something that could be indicative of an attack.

A normal user is limited by how quickly they can manually navigate to website pages, whereas an attacker or bot may be able to flood a website with requests. Assuming the majority of clients are normal users, we can detect atypical behavior by looking at unusually high event rate for a client:

Display of IP addresses with high event rates

The results show several anomalous clients; especially seems to show anomalous activity over a longer period of time.

In this case, we are interested in only viewing results for that specific IP. Using the query bar, we can narrow the results for

Display narrowing results to specific IP address

With the query bar, we now have the ability to search through anomaly results to focus in on one or more selected influencers. The query bar suggests influencer field names and values for selected jobs as it receives input. Auto-suggest makes it easy to view possible fields and values we can search by; we’re no longer limited to seeing only the top ten shown in the results view.

The query bar accepts input in KQL syntax. Users can combine queries using AND or OR keywords and can filter by field value only.

Combining queries to show two different IP addresses

Wildcard queries are supported. Searching for source.address: 185*, for example, narrows the displayed results to all IP values beginning with 185.

Query including a wildcard

If the influencer of interest is listed in Top Influencers, clicking the plus icon next to it will add that field/value combination to the query bar and apply the search. Removing the search can be accomplished by clicking the minus icon. This can also be done from the results table.

Using Top Influencers to alter query

At Elastic, we like to build search into everything we do. The query bar provides an easy way to search through anomalies for one or more particular influencers, allowing for easier result navigation. This is especially useful when a field is not one of the top N shown in the results view.

Interested in trying it out? Nab a free 14-day trial of the Elasticsearch Service. If you have feedback or questions we invite you to join the conversation in the machine learning Discuss forum.