Elastic Synthetics Projects: A Git-friendly way to manage your synthetics monitors in Elastic Observability

Elastic has an entirely new Heartbeat/Synthetics workflow superior to the current workflow. If you’re a current user of the Elastic Uptime app, read on to learn about the improved workflow you can use today and should eventually migrate toward.

We’ve recently released a beta feature that provides a Git-friendly IaaC oriented workflow. You can now push Heartbeat monitors with the same ease with which you push code changes in Git or config changes in Terraform. The features discussed in this blog are all currently in beta, and we urge users trying these features out to upgrade to the latest stack version first. When these features become GA, this new workflow will be the preferred way of configuring monitors in the Elastic Stack. If you’re starting a new project, you may want to consider setting it up this way instead of via our more classic configuration.

Today, using Heartbeat is simple. You just need to write a little YAML and monitoring data shows up in Elasticsearch, visible in the Uptime UI. While the UI is indeed simple, there’s some hidden complexity there that we’ve improved with a new UI (the Synthetics app) and augmented with an even more automation friendly CLI workflow via our new Projects feature, which will be discussed below.

How do you manage your configs written in YAML? Many of our users will manage YAML in Git and use tooling such as Ansible, Helm, or similar to manage their infrastructure as code (IaaC). As with any other organization, Elastic also heavily utilizes IaaC in all parts of our operations. Hence it’s only natural we developed a capability to provide you with similar support for the current Heartbeat capability and the upcoming synthetics monitoring capabilities.

Projects: A new way to organize and distribute configs

Let’s dive right into what we’re calling “Synthetics Projects” and how they differ from traditional Heartbeat config files. To use this feature, you would start by creating a project in a Git repo containing your configs. At a high level, setting up a project requires performing the following tasks:

1. Run npx @elastic/synthetics init to create a project skeleton in a directory. See more details on the npmjs.com site.
2. Run git init and git push on the generated directory to version it as a Git repository.
3. Add your lightweight YAML files and browser javascript/typescript files to the journeys folder.
4. Test that it works by running npx @elastic/synthetics push command to sync your project to your Elastic Stack.
5. Configure a CI/CD pipeline to test pull requests to your Git repo and to execute npx @elastic/synthetics push on merges to the main branch. 

So, once configured, adding, removing, and editing monitors involves:

1. Editing a monitor’s config, either YAML for lightweight monitors, or Javascript/Typescript for browser based ones locally
2. Testing your local configs with npx @elastic/synthetics journeys
3. Creating a new PR to your main branch via a Git push
4. Waiting for your CI server to perform the same validation and waiting for someone else on your team to review your PR
5. Merging your result to the main branch
6. Waiting for your CI server to push the changes to your Elastic stack

We’ve depicted the flow of data in the diagram below:

This is, in fact, the way many of our users work today, with other software taking the place of npx @elastic/synthetics push as mentioned earlier. Indeed, in the future, we will most likely look into building a Terraform provider, though that isn’t something we’re actively working on now.

Just have a few monitors? Use the GUI!

The above approach is great for sophisticated users with larger numbers of configurations, but if you just want to monitor a few URLs, it’s overkill. If that sounds like you, consider the new Monitor Management UI in the Uptime app! It works in the exact same way, saving configs to your Elastic Stack, but with no need for Git, or a project, or all that other infrastructure. Simply, log in, fill out the form pictured below, and hit save. If you want to set up a private location, that is still done in the same way via Fleet.

What about my existing Fleet monitors?

A small subset of users have monitors configured today using the Synthetics Fleet integration. If that describes you, you’ll want to move onto either the GUI based approach or the Project based approach, as those methods supersede direct usage of the Fleet integration, which will eventually be restricted only to use via the above described methods. 

The Fleet approach is inferior in a few ways:

1. It can only configure monitors for a single location.
2. It creates a different UX for monitors configured on the service versus private locations.
3. It’s less fluid of an integration with the Uptime UI.

It’s rare for us to deprecate beta features, but in this case we had a clearly superior alternative. Maintaining both would have created a more confusing and unwieldy product. We don’t yet have an exact date for removing support for these monitors, but you can track this via this GitHub issue.