Learn how to use Elastic Uptime to configure alerting and anomaly detection for sites, services, and APIs.

Being able to easily monitor the health of all your sites and services from multiple global locations is a powerful tool for site reliability. However, no one wants to sit and stare at a status dashboard all day. Naturally, teams want to be alerted when there is an issue. We can do that with <a href="https://www.elastic.co/what-is/kibana-alerting">alerting in Kibana</a>. And when coupled with Elastic machine learning, alerts can be automatically generated from anomalies that are automatically detected. That’s the power of <a href="https://www.elastic.co/observability">Elastic Observability</a>.
In this blog, we’re going to build on the recent <a href="https://www.elastic.co/blog/service-monitoring-and-availability-made-simple-with-elastic-uptime-and-heartbeat">service monitoring and availability made simple</a> post to configure alerting and anomaly detection for sites, services, and APIs. If you haven’t read that blog yet, start there first to learn all about Elastic Uptime and to get your test environment <a href="https://www.elastic.co/elasticsearch/service">deployed for free in Elastic Cloud</a>.
 <h2>The speed of Elastic&nbsp;</h2>To quote one of my favorite movies: Life moves pretty fast. If you don’t stop and look around once in a while, you could miss it. Nowhere is that more true than at Elastic. Since the first part of this blog was updated, <a href="https://www.elastic.co/blog/whats-new-in-elastic-7-9-0-free-workplace-search-engine-endpoint-security">Elastic 7.9 was released</a>. Among the many fantastic updates in that release were a few enhancements to Elastic Uptime, including the addition of alerting on anomaly detection, which we will discuss below. Another visual change you will see from the first post is the addition of availability percentages on a monitor’s detail page.
<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt91ecc9decddda04d/5f43d6c77581487312cf95d5/blog-uptime-alerting-anomaly-detection-availability-percentages.png" data-sys-asset-uid="blt91ecc9decddda04d" alt="Availability percentages on a monitor detail page" style="display: block; margin: auto;">
Availability information is now provided as an overall metric as well as on a per-location breakdown. The world map can still be accessed with the geo marker button in the upper right corner.
<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt5828d3c3464917e6/5f43d7405010420f8a6a9337/blog-uptime-alerting-anomaly-detection-world-map.png" data-sys-asset-uid="blt5828d3c3464917e6" alt="The world map can still be accessed with the geo marker button in the upper right corner" style="display: block; margin: auto;">
<h2>Status alert</h2>The first type of alert we are going to set up is a Monitor Status alert. This will automatically create alerts based on errors and outages.
From the overview page, select the Alerts dropdown, then select Create alert &gt; Monitor status alert. A flyout where you can create an alert will appear on the right side of the screen.
<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt14ee4db3cb046d76/5f43d7e15010420f8a6a9341/blog-uptime-alerting-anomaly-detection-alerts-dropdown2.png" data-sys-asset-uid="blt14ee4db3cb046d76" alt="Create an alert from the Alerts dropdown" style="display: block; margin: auto;">
Give your alert the name World Wide Status and give it a tag global so we can group it with other alerts we may create later. We want to check the status every five minutes looking for any monitor that shows <code>Down</code> more than 3 times in 10 minutes. When the condition is met, we want the alert triggered. And we only want to be notified every 30 minutes so we’re not constantly flooded with alert notifications.
<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blta589b6f8ad10ca6f/5f43d831ba7f910f91e002ab/blog-uptime-alerting-anomaly-detection-create-alert.png" data-sys-asset-uid="blta589b6f8ad10ca6f" alt="The Create Alert flyout" style="display: block; margin: auto;">
This alert covers all the monitors we have running; however, we can easily create alerts for one or more specific monitors and have different conditions for each.
The final step in creating an alert is selecting one or more action to take when the alert is triggered.
<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltafdfd5bc35eb9f58/5f43d86c8164860f7fcf0981/blog-uptime-alerting-anomaly-detection-actions.png" data-sys-asset-uid="bltafdfd5bc35eb9f58" alt="Selecting an action type" style="display: block; margin: auto;">
A lot of us use Slack as our primary channel of communication at work, so let’s create a Slack action. If you don’t have your desired connector created, you will see an option to “Create a connector.”
<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt2c827dbd6c5e311f/5f43d8d20aa31a7f956db6c8/blog-uptime-alerting-anomaly-detection-create-connector.png" data-sys-asset-uid="blt2c827dbd6c5e311f" alt="Setting up a Slack connector" style="display: block; margin: auto;">
The provided alert message is pretty good, but let’s customize it a little bit and add some Slack formatting. To make that easier, we can select the context dropdown to see a list of available variables we&nbsp;can use to enrich the content of our alert.&nbsp;
After hitting Save, you’ll see the confirmation!
<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt0d0e5ccfc2d7ce5a/5f43d9023b69050fbdf1d6bc/blog-uptime-alerting-anomaly-detection-save-confirmation.png" data-sys-asset-uid="blt0d0e5ccfc2d7ce5a" alt="Confirmation message" style="display: block; margin: auto;">
<h2>Let's break something!</h2>To test out this new alert we can wait for a legitimate site issue ... or we can break something! To be safe, you should only break something that won’t affect others. For this example, I’m going to disable my internal TCP port listener script.
<pre class="prettyprint">pkill -f echo_port_listener.py
</pre>And after a few seconds, we see my internal TCP monitor is reporting down as expected. Try it out for yourself!
<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt4d0cb469cd8f88b7/5f43d96dc982f501a32887dd/blog-uptime-alerting-anomaly-detection-overview.png" data-sys-asset-uid="blt4d0cb469cd8f88b7" alt="A dashboard showing that the internal TCP monitor is down" style="display: block; margin: auto;">
Let’s click the dropdown on the Overview page to get more information about the monitor.
<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt4d10db0c31b31856/5f43d9a0e3aef10fa0dabe88/blog-uptime-alerting-anomaly-detection-overview-dropdown.png" data-sys-asset-uid="blt4d10db0c31b31856" alt="The Overview page, showing more information about the monitor" style="display: block; margin: auto;">
You can see that the monitor running on my us-central-c host is <code>Down</code> but the australia-southwest1 and europe-west2 internal TCP listeners are still <code>Up</code>.&nbsp; You can also see the issue was <code>Connection Refused</code>.&nbsp;
Clicking on the monitor we can get additional information. On the world map, the US location is red while London and Australia are still gray (Up). Pings over time shows the counts of <code>Up</code> and <code>Down</code> pings, and in the History section we can expand the Down status and again see the connection refused error.
<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt1e06953b694243e7/5f43da287b460773e51d69ab/blog-uptime-alerting-anomaly-detection-world-map-history.png" data-sys-asset-uid="blt1e06953b694243e7" alt="Pings over time and History in Elastic Uptime" style="display: block; margin: auto;">
And after a few minutes, I’m notified in Slack that my connection is down.
<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt17427349091f9186/5f43da4f7581487312cf95ff/blog-uptime-alerting-anomaly-detection-slack-alert.png" data-sys-asset-uid="blt17427349091f9186" alt="A Slack notification showing that the connection is down" style="display: block; margin: auto;">
<h2>Certificate alerts</h2>You may have noticed from the Alerts dropdown there is a second alert type, a TLS alert.I discussed the importance of monitoring your site TLS certificates in <a href="https://www.elastic.co/blog/service-monitoring-and-availability-made-simple-with-elastic-uptime-and-heartbeat">the previous post</a>, so let’s set up an alert to notify us when any of the certs on sites we’re monitoring are close to expiring.
The alerting flyout panel is similar to the one we saw in the previous alert. In this case the middle section provides TLS-specific settings. Since we don’t need to check the expiration date of certs as frequently as a status alert, let’s check every 6 hours and notify every 12. The thresholds used in the alert for expiration within range and older than range are configured in the settings page, as they are also used for visual alerts. You can further modify the alert settings however you’d like
The default message looks good, so hit Save. The next time the alert runs, it will pick up one of the sites we’re monitoring that has a certificate expiring soon and will send an alert in Slack. This way we can avoid <a href="https://www.theverge.com/2020/2/3/21120248/microsoft-teams-down-outage-certificate-issue-status">potentially costly outages</a> to our site (and damage to our reputation).
<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltc02205176f5c09c7/5f43dabe7581487312cf9607/blog-uptime-alerting-anomaly-detection-slack-alert-cert.png" data-sys-asset-uid="bltc02205176f5c09c7" alt="A Slack notification showing that a certificate is expiring" style="display: block; margin: auto;">
<h2>Anomaly detection</h2>Knowing when your website or endpoint is down is extremely important. If your customer can’t load your webpage, they can’t purchase products or interact with the services you provide. But knowing when your site is slowing down can be just as important. Today, with everyone expecting websites to instantly load, having a slow webpage response time can just as easily drive a visitor to a competitor's website.&nbsp;
With <a href="https://www.elastic.co/what-is/elasticsearch-machine-learning">machine learning</a> integrated right into Elastic Uptime, you can enable anomaly detection with one click right from the Monitor duration panel of each monitor. Each individual monitor location is individually modelled and when a monitor runs for an unusual amount of time at a particular time an anomaly is recorded and highlighted right on the Monitor duration chart.
To enable anomaly detection, simply click on a monitor to go to the details page and in the Monitor duration panel select Enable anomaly detection. That’s it!
<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltfb361e4ce3a08d7d/5f43daf833bd560fb6d7d68c/blog-uptime-alerting-anomaly-detection-enable-anomaly-detection.png" data-sys-asset-uid="bltfb361e4ce3a08d7d" alt="Enabling anomaly detection" style="display: block; margin: auto;">
As mentioned above, one of the new features in the Elastic 7.9 release was the ability to create an alert when a monitor duration is detected. As part of automatically creating an anomaly detection job, an Alert Creation flyout will appear. This will allow you to configure which severity level to create an alert on. The flyout is the same as the alert we created above with the exception of the anomaly section. We will leave the default level of “Critical” to trigger an alert. We will use the same Slack connector, which is pre-populated with an anomaly-specific message.
<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltab9da2d0104fa36e/5f43de754e95ba73ebe4ee7f/blog-uptime-alerting-anomaly-detection-create-slack-alert.png" data-sys-asset-uid="bltab9da2d0104fa36e" alt="Uptime duration anomaly" style="display: block; margin: auto;">When an anomaly is detected, the duration is represented right on the Monitor duration chart along with the duration times. The colors represent the criticality of the anomaly (red == critical, yellow == minor). In addition, since we configured a Slack alert, when a critical anomaly is detected we will be alerted about it!
<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltbb6bd3fc4ae23c2b/5f43dc32ba7f910f91e002b9/blog-uptime-alerting-anomaly-detection-monitor-duration.png" data-sys-asset-uid="bltbb6bd3fc4ae23c2b" alt="Monitor duration chart" style="display: block; margin: auto;">
The anomaly detection job on the back end is the same as any other custom job you may create. The integration simply allows for ease of creation and viewing the anomalies along with the source data. You can view more information about the job and its history in the ML UI, which is easily accessed from the Anomaly detection dropdown in the Monitor duration screen. Here we can also create an alert (if we decided against it originally) or deactivate the job altogether.
Learn more about Elastic’s anomaly detection in our blog post on <a href="https://www.elastic.co/blog/interpretability-in-ml-identifying-anomalies-influencers-root-causes">identifying anomalies, influencers, and root causes</a>.
 <h2>Wrap up time</h2>Over these two blog posts we’ve configured Heartbeat to monitor multiple services from multiple global locations. We’ve walked through using Elastic Uptime to easily identify when there are disruptions to those services. We enabled anomaly detection on our monitors with one click, allowing us to easily see when a service is performing unusually slow. And, so we can go about our other work without having to keep an eye on all this, we enabled alerting right from within Uptime so we can be notified when action is needed.&nbsp;&nbsp;&nbsp;
If you’ve just been reading along, I encourage you to try Uptime out for yourself. The easiest way to get started is to sign up for a free trial of <a href="https://www.elastic.co/cloud/elasticsearch-service/signup">Elasticsearch Service</a> and <a href="https://www.elastic.co/downloads/beats/heartbeat">install Heartbeat on your laptop</a> — you’ll be observing in no time.&nbsp;
Now that you have Uptime covered, you might be interested in <a href="https://www.elastic.co/blog/identifying-and-monitoring-key-metrics-for-your-hosts-and-systems">identifying and monitoring key metrics for your hosts and systems</a>.

blog-banner-cloud-alerts.png

blog-thumb-cloud-alerts.png

Alerting and anomaly detection for uptime and reliability

Follow along as we configure Elastic Uptime to monitor your host, website, and API's up/down availability, response duration, status code, and TLS certificate validation.

AUTHOR

Articles by Jeff Vestal

Alerting and anomaly detection for uptime and reliability

Service monitoring and availability made simple with Elastic Uptime and Heartbeat

Sign up for Elastic Cloud free trial

Products & Solutions

Company

We're hiring

Resources

Solutions

Elastic (ELK) Stack

New

Featured topics

News

AUTHOR

Articles by Jeff Vestal

Alerting and anomaly detection for uptime and reliability

Service monitoring and availability made simple with Elastic Uptime and Heartbeat

Sign up for Elastic Cloud free trial

We're hiring