13 February 2018 Engineering

Scheduled Events and the Amorous Anomaly

By David Kyle

Editor's Note (September 7, 2018): This post refers to X-Pack. Starting with the 6.3 release, the X-Pack code is now open and fully integrated as features into the Elastic Stack.

Machine Learning in X-Pack learns continuously adapting its model to the changing input, redefining what is normal and what is anomalous. This is great as behaviour changes over time but what happens when there is a unique event that dramatically distorts the model? Consider ‘Black Friday’ or a public holiday; these are rare events where the behaviour is significantly different. These occasions are noisy, generating numerous anomalies and spurious alerts, they can pollute the model and it may take several days for ML to re-learn normal behaviour. We know about these events in advance so surely we can mitigate those effects. Scheduled Events are a new feature in Machine Learning 6.2 to address this problem.

A Very Special Event

As an example let’s consider a day close to our hearts, Valentine’s day 14th February a very important day for the smitten software engineer. Our besotted engineer makes a number of romantic gestures on a daily basis such as gifting flowers, saying ‘I love you’ or sending a 😍. Curiously, and somewhat conveniently, these gestures have been logged in Elasticsearch and make the perfect dataset for this demonstration.

Romantic Gestures

The image above shows the hourly count of romantic gestures over a 4 day period. No romance takes place overnight as our engineer soundly sleeps. During the day he is increasingly romantic, approaching peak romance in the early evening. Our engineer is a person of habit as the same pattern is repeated every day.

Now we can use Machine Learning to analyze these heart-warming endearments.

Valentine's Day Anomaly

The job is configured to analyse the count of romantic gestures. Only the cynical would be surprised by the anomalies found on Valentine’s Day.

Unfortunately alerting has been enabled on this job and is configured to send a slack notification. This is a particular nuisance and a distraction for the enamoured engineer on the most romantic day of the year. Secondly, there is a suspiciously high level of romantic activity on 17th February which was not picked because the amorous exertions of Valentine’s Day have inflated the model bounds. As a software engineer and incurable romantic (the Venn diagram of those 2 sets is a single circle), perhaps Scheduled Events can help!

Calendars

Scheduled Events are managed in Calendars, you will find the Calendar Management UI under the new Settings tab on the ML page in Kibana. Calendars have a list of events and jobs are assigned to calendars, jobs belonging to the calendar will pick up those scheduled events. You can also define a job group and add that group to the calendar

Romantic Days Calendar

Scheduled Events are one-off or exceptional occasions; do not be tempted to create a calendar for regularly occurring events such as a daily maintenance task as ML will learn these periodic patterns anyway.

Running the same job with a scheduled event reveals very different results.

Valentine's Day anomaly is silenced

The unusual ardour of Valentine’s Day has not created any anomalies nor has it affected the model bounds revealing the truly suspicious anomaly 3 days later on 17th February.

Summary

Scheduled events are known about ahead of time. If pre-configured, ML jobs will skip analyzing data. During the special event the model will not be altered and anomalies will not be calculated… and in conclusion, software engineers can be romantic.