With the release of the Elastic Stack 5.5, machine learning (ML) is now a supported GA version. Along with this, we have introduced an experimental feature which allows the end-user to estimate the bucket span for an anomaly detection job. This is available when using the single or multi metric job creation wizards, but not when using the advanced configuration wizard.
<img src="https://api.contentstack.io/v2/uploads/59695241fb2436ec743d99d0/download?uid=blt03376d801dda1b6d" data-sys-asset-uid="blt03376d801dda1b6d" alt="new-machine-learning-job-elasticsearch-cloudwatch.png">
The bucket span estimator gives you the minimum viable bucket span based on the characteristics of a pre-viewed subset of your data. It’s a useful starting point for your analysis. However, once comfortable with the concepts of anomaly detection, then we urge you to delve a little deeper and make sure you have the right bucket span for your use case and your needs.
Typically in our experience for machine data, bucket spans tend to be between 10 minutes and 1 hour, but it depends on the characteristics of the data.
To explain more about bucket spans, and how to configure this appropriately, let me answer a few commonly asked questions.
<h2>What is a bucket span?</h2>When analyzing data, we use the concept of a bucket to divide up a continuous stream of data into batches for processing.
For example, if you were monitoring the average response time of a system, using a bucket span of 1 hour means that at the end of each hour we would calculate the average (mean) value of the last hour’s worth of data and compute the anomalousness of that average value compared to previous hours.
Figure 1 shows a comparison of bucketed and non-bucketed data.
<img src="https://api.contentstack.io/v2/uploads/596952742b501f51435f137b/download?uid=bltcfe372fe49abe32f" data-sys-asset-uid="bltcfe372fe49abe32f" alt="machine -learning-elasticsearch-bucketed-vs-non-bucketed-data-comparison.png"> 
By analyzing the data in buckets, we can more easily see significant changes, random fluctuations are smoothed, and there is a discrete range to model.
As the time series data is gathered into buckets this also reduces the bandwidth and space required to process and store the data. This is important for online anomaly detection, especially in large scale environments.
<h2>Why is the bucket span important?</h2>In machine learning terms, the bucket span and function define the feature we model. Choosing a shorter bucket span allows anomalies to be detected more quickly but at the risk of being too sensitive to natural variations or noise in the input data. Choosing too long a bucket span can mean that interesting anomalies are smoothed out.
Figure 2 shows what the same data can look like when viewed at three different bucket spans. This chart plots the sum of a counter over time with coloured vertical lines denoting detected anomalies; light blue is a warning, dark blue a minor, orange a major and red a critical anomaly.
<img src="https://api.contentstack.io/v2/uploads/5969529499bd94c363c13900/download?uid=blt4c38edbaf77041eb" data-sys-asset-uid="blt4c38edbaf77041eb" alt="machine-learning-elasticsearch-bucketing-data-different-spans.png">
In the top most chart, a bucket span of 1 minute has been selected. This is as fine as the data rate, and the fluctuations in the data give rise to a very high background noise of anomalies. The periodic trend is not particularly visible.
In the bottom chart, a bucket span of 1 hour has been selected. We can clearly see that the trend becomes significantly more visible with respect to the noise as a larger bucketing interval is used. However, there are noticeably fewer anomalies.
Part of this reduction is due to the fact that anomalies are now only calculated hourly. However some anomalies have been hidden in the mass of data that exists in a larger bucket. Notice that the single important anomaly has a lower severity, and is only considered “major” rather than “critical”. For this data using the sum function, the use of a larger bucket has meant that the relative importance of the anomaly has been reduced in significance.
In this example for this dataset, choosing a bucket span of 10 minutes provides an optimal balance of granularity, time to alert and prominence of the anomaly.
<h2>How does the bucket span affect the time to alert?</h2>One might infer that (on average) an anomaly will be reported at least half a bucket span away from when it occurred. This is not true, but there are some caveats which depend on the job configuration and the characteristics of your data.
Assuming you are using a datafeed to pass data from Elasticsearch indices to ML for analysis, a search is performed every <code>frequency</code> for new data since the last received timestamp. The default value for <code>frequency</code> is adjusted depending on the bucket span.
<table>
<tbody>
<tr>
	<td>Bucket Span
	</td>
	<td>Default frequency
	</td>
</tr>
<tr>
	<td>Less than 2 minutes
	</td>
	<td>1 minute
	</td>
</tr>
<tr>
	<td>Less than 20 minutes
	</td>
	<td><code>bucket_span</code>
	</td>
</tr>
<tr>
	<td>Less than or equal to 12 hours
	</td>
	<td>10 minutes
	</td>
</tr>
<tr>
	<td>More than 12 hours
	</td>
	<td>1 hour
	</td>
</tr>
</tbody>
</table>Data is analyzed in time-order, so once a data point is received for the next bucket, end-of-bucket processing is triggered and the analysis moves forward in time. End-of-bucket processing applies the latest bucket summary statistics to the model and calculates the final results for this bucket (amongst other things).
As we receive interim data when using a datafeed, we also calculate interim results at the <code>frequency</code> interval. Interim results make adjustments for the fact that only partial data has been seen for the bucket. For functions such as min and max, interim results are very reliable. For functions such as sum, mean and count, then the reliability depends on the characteristics of the data. Adjustments for partial buckets with continuous data can be very accurate. However if your data time stamps are irregular and low volume, then it becomes difficult to assess how much of the bucket has been seen so confidence is less and interim anomalies may not be found (or in the case of <code>low_count</code> or <code>low_sum</code> functions may be inaccurate).
So by default for a 1-hour bucket span, interim results are calculated every 10 minute and alerts can occur at this granularity. This blog post describes how to implement <a href="https://www.elastic.co/blog/alerting-on-machine-learning-jobs-in-elasticsearch-v55">alerting on ML jobs</a>&nbsp;in more detail.
Interim results are short lived, and are only available for the latest partial bucket. Once end-of-bucket processing has occurred, they are updated as final results.
There is an overhead to calculating interim results, so setting the <code>frequency</code> value too low will negatively impact the performance of the system.
<h2>How does the bucket span estimator work?</h2>The bucket span estimator is available when creating a job using the single metric or multi metric job creation wizard.
It performs a series of tests on a subset of data, taken from the previous 250 hours from the end date in the Kibana time picker.
First, it checks if the data is polled. This is data that is stored at discrete time intervals, for example monitoring data that is reported every 5 mins would be polled data. If the data is polled, we estimate the bucket span such that at least one data point exists per bucket,
If the data is not considered to be polled, we then created multiple aggregated time series for a set of bucket spans (from 5 minutes to 3 hours) using the aggregation function that has been selected in the detector configuration.
If the job is multi metric, then there may be more than one detector and many entities. The aggregation described above is performed for every detector function and for a selection of 10 entities using a partitioned search.
It then performs a series of tests on each aggregated time series, starting with the smallest bucket span. The estimated bucket span is the smallest span for which all tests have passed.
<ul>
	<li>Empty bucket test - at least 20% of buckets must be populated</li>
	<li>Minimum doc count test - for functions sum and count, at least 5 documents are required per bucket</li>
	<li>Scale variation test - if a trend exists, the smoothing effect of bucketing does not hide this trend when compared to a longer reference span</li>
</ul>For multi metric jobs, the results are reduced by taking the median value of partitioned series per detector and then the median of the detector series to get a single value.
If the bucket span cannot be estimated, a warning message will be displayed and the bucket span should be manually entered. Consider the factors below in choosing a bucket span as a span shorter than 3 hours may still be appropriate.
<h2>What do I need to consider when choosing the bucket span?</h2>It is important to note that the estimated bucket span is only a rough calculation, based on the subset of data seen. It does not take into account other operational factors such as time to alert, the typical duration of anomalies, and the processing time. As such you may wish to choose your own span.
Selecting the optimal bucket span can be a compromise across multiple factors discussed below:
Data Characteristics - In order to get enough data in each bucket we have to take into account if the data is sparse or if the data is polled. Sparse data typically requires a longer bucket span or use of a sparse-aware function e.g. <code>non_null_sum</code> or <code>non_zero_count</code>. Polled data requires at least one data point per bucket.
Quality and Relevance - In general we want to maximise the anomaly detection rate and minimise the false positives. To do this we should match the bucket span to be similar to the typical duration of interesting anomalies. This requires knowledge of the system beforehand, which may not always be available.
Performance - The main computation is performed at the end of each bucket, known as end-of-bucket processing. With a longer bucket span we can achieve higher data throughput compared with a shorter bucket span. If end-of-bucket processing takes longer than the bucket span, then the job will not be able to keep up with real-time. In this instance, selecting a longer bucket span allows the processing time to catch up.
Timeliness - End-of-bucket processing will finalize results for the most recent bucket. If your data is not suited to interim results and the job is time critical, then you may wish to reduce the bucket span.
<h2>Why can’t I change the bucket span?</h2>Once the analysis is underway, we do not allow configuration changes to the bucket span. Changing the bucket span would fundamentally change what data is being modelled. Summary statistics that had been gathered for the life of the job up to this point, would no longer be applicable to future data. The analysis would need to start gathering these summary statistics from scratch, and thus the model would be reset.
As the model would be starting from zero again, then we feel it is preferably to do this as a new job. This job could be run on the same historical data to start with and then progress in real-time. The advantage of doing this would be that you could see the two job results in parallel. By comparing the results, you can then decide which bucket span provided the most informative and actionable results for your environment.

glass-time-watch-business-full.jpg

glass-time-watch-business-thumb.jpg

Explaining the Bucket Span in Machine Learning for Elasticsearch

Start free trial

Blog Footer CTA

Sign up for Elastic Cloud free trial

Machine Learning Team Lead

Sophie Chang

By submitting you acknowledge that you've read and agree to our <a href="/legal/elastic-cloud-account-terms" target="_self">Terms of Service</a>, and you agree to receive the <a href="/legal/privacy-statement#how-we-use-the-information" target="_self">selected communications</a> at the contact details you provided above. See <a href="/legal/privacy-statement/" target="_self">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.

Communication Preferences - ONLY for /communication-preferences

Durch das Absenden erklären Sie, dass Sie unsere <a href="/de/legal/elastic-cloud-account-terms" target="_self">Nutzungsbedingungen</a> gelesen haben und mit ihnen einverstanden sind und dass Sie Elastic erlauben, <a href="/de/legal/privacy-statement#how-we-use-the-information" target="_self">ausgewählte Nachrichten und Informationen</a> an die oben von Ihnen bereitgestellten Kontaktangaben zu senden. Weitere Informationen, darunter auch dazu, wie Sie Ihre Einwilligung zurückziehen können, finden Sie in der <a href="/de/legal/privacy-statement/" target="_self">Datenschutzerklärung von Elastic</a>.

By submitting you acknowledge that you've read and agree to our <a href="/legal/serverless-preview-terms" target="_blank">Terms of Service</a>, and that Elastic may <a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">contact you</a> about our related products and services, using the details you provide above. See <a href="/legal/privacy-statement/" target="_blank">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.

Serverless GDPR Text

Durch das Absenden erklären Sie, dass Sie unsere <a href="/de/legal/serverless-preview-terms" target="_self">Nutzungsbedingungen</a> gelesen haben und mit ihnen einverstanden sind und dass Sie Elastic erlauben, bezüglich unserer zugehörigen Produkte und Dienstleistungen und unter Nutzung der von Ihnen oben bereitgestellten Kontaktangaben <a href="/de/legal/privacy-statement#how-we-use-the-information" target="_self">mit Ihnen in Kontakt zu treten</a>. Weitere Informationen, darunter auch dazu, wie Sie Ihre Einwilligung zurückziehen können, finden Sie in der <a href="/de/legal/privacy-statement/" target="_self">Datenschutzerklärung von Elastic</a>.

By submitting you agree to <a href="/agreements/global/cloud-services-monthly" rel="nofollow">Elastic Cloud Terms of Service</a>. Your personal data will be processed in accordance with <a href="/legal/privacy-statement">Elastic's Privacy Statement</a>.

Cloud Trial GDPR Text

Beim Absenden akzeptieren Sie die <a href="/de/agreements/global/cloud-services-monthly" rel="nofollow">Nutzungsbedingungen von Elastic Cloud</a>. Die Verarbeitung Ihrer personenbezogenen Daten unterliegt der <a href="/de/legal/privacy-statement">Datenschutzerklärung von Elastic</a>.

By submitting you acknowledge that you've read and agree to our <a href="/legal/terms-of-use" target="_blank">Terms of Service</a>, and that Elastic may <a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">contact you</a> about our related products and services, using the details you provide above. See <a href="/legal/privacy-statement/" target="_blank">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.

Newsletter GDPR Text

Durch das Absenden erklären Sie, dass Sie unsere <a href="/de/legal/terms-of-use" target="_self">Nutzungsbedingungen</a> gelesen haben und mit ihnen einverstanden sind und dass Sie Elastic erlauben, bezüglich unserer zugehörigen Produkte und Dienstleistungen und unter Nutzung der von Ihnen oben bereitgestellten Kontaktangaben <a href="/de/legal/privacy-statement#how-we-use-the-information" target="_self">mit Ihnen in Kontakt zu treten</a>. Weitere Informationen, darunter auch dazu, wie Sie Ihre Einwilligung zurückziehen können, finden Sie in der <a href="/de/legal/privacy-statement/" target="_self">Datenschutzerklärung von Elastic</a>.

Explore similar demos

Overview

Speakers

Close

See more insights

The content on this page is not available in the selected language. As Elastic grows globally, we continue to support content in multiple languages.

The content on this page is not available in the selected language.

Der Inhalt dieser Seite ist in der ausgewählten Sprache nicht verfügbar. Wir bei Elastic arbeiten daran, die bereitgestellten Inhalte in verschiedenen Sprachen anzubieten. Bis dahin bitten wir Sie um etwas Geduld und hoffen auf Ihr Verständnis!

Author

Watch now (no PT)

Watch now

See all top stories

All (no PT translation)

Contact information

Press Release

Share on Reddit

Articles by

Share this story

Share by Email

Thank you for your interest!

Contact Info

Follow us on Youtube

Load More

Share on LinkedIn

Follow us on LinkedIn

Search Integrations

All Solutions

Video for

Follow us on Twitter

See when this webinar starts in my time zone

Register to Watch

Register now

See All Posts

Can't make it? Register and we'll send you the recording. You'll also receive an email with related content

Sie können nicht dabei sein? Registrieren Sie sich, damit wir Ihnen die Aufzeichnung zukommen lassen können. Außerdem erhalten Sie eine E‑Mail mit Begleitmaterial.

Small image for

On-demand webinar

Featured webinar

Register to Attend

Share on Facebook

Reset all

Upcoming webinar

View more posts

Print

Hosted by

Sign In to Attend

View next

Follow us on Facebook

Share

Highlights

Learn More

Read less

You'll also receive an email with related content

Wir schicken Ihnen zudem relevante Informationen, die von Interesse sein könnten.

Thank you for registering. We will send you a confirmation email soon.

Vielen Dank für Ihre Registrierung. Sie erhalten in Kürze eine Bestätigungs-E‑Mail von uns.

Author

Articles by Sophie Chang

Machine Learning Team Lead, Elastic

Explaining the Bucket Span in Machine Learning for Elasticsearch