Many users may have a large number of TCP, ICMP, and HTTP monitors and need to migrate them to Synthetics. In this guide, we will explain how to perform this migration easily while ensuring that it will be future-proof or able to develop more advanced checks such as Browser monitors.

First, we must consider the number of monitors to migrate; if the number is small, the easiest way would be to do it manually through the Synthetics UI. However, in this guide we will assume that we have dozens or hundreds of monitors to migrate, and doing it manually in the Synthetics UI is not an option.

Private Location

Traditionally, uptime monitors required a Heartbeat to be deployed in your infrastructure, which indirectly allowed you to monitor endpoints or hosts on your private network. If this is still a requirement, you will need to either configure Private Location or allow Elastic’s global managed infrastructure to access your private endpoints (only on ECH & Serverless).

In this guide, we will use Private Locations, which will allow you to monitor both internal and external resources. More details can be found here: Monitor resources on private networks

Step 1: Set up Fleet Server and Elastic Agent

Private Locations are simply Elastic Agents enrolled in Fleet and managed through an agent policy. 

If you don't have a Fleet Server yet, start setting up a Fleet Server. This step is not necessary if you use ECH, as it comes by default.

Next, you will need to create an Agent Policy. Go to Observability → Monitors (Synthetics) → Settings (top right) → Private Location → + Create Location

Fill in the fields and create a new policy for this Private Location. It is important to know that Private Location should be set up against an agent policy that runs on a single Elastic Agent. 

Step 2: Deploy the Elastic Agent

Now we need to deploy the Elastic Agent that will be responsible for running all the monitors. We can use the same host we were using for Heartbeat. There is only one requirement: we must be able to run Docker containers, since to take advantage of all the features of Synthetics, we must use the elastic-agent-complete Docker Image.

1. Go to Fleet –> Enrollment tokens and note the enrollment token relevant to the policy you just created for the Private Location. Now go to Settings and note the default Fleet server host URL.

2. On the host, run the following commands. For more information on running Elastic Agent with Docker, refer to Run Elastic Agent in a container.

docker run \
  --env FLEET_ENROLL=1 \
  --env FLEET_URL={fleet_server_host_url} \
  --env FLEET_ENROLLMENT_TOKEN={enrollment_token} \
  --cap-add=NET_RAW \
  --cap-add=SETUID \
  --rm docker.elastic.co/elastic-agent/elastic-agent-complete:9.3.2

Synthetic Project

At this point, we already have the location from which our Synthetic monitors will run. Now we need to load our Uptime monitors as Synthetics.

As we mentioned earlier, there are two ways to do this: either manually through the Synthetics UI or through a Synthetics Project. In our case, since we have so many monitors to migrate and don't want to do it manually, we will use Synthetics Projects. 

The great thing about Synthetics Project is that it has some backward compatibility with the definition of monitors in heartbeat.yml and we will be leveraging it.

What's Synthetics project?

Synthetics project is the most powerful and flexible way to manage synthetic monitors in Elastic, based on the Infrastructure as Code principle and compatible with Git-Ops flows. Instead of configuring monitors from the interface, you define them as code: .yml files for lightweight monitors and JavaScript or TypeScript scripts for browser-type monitors (journeys).

This approach allows you to structure your monitors in a repository, version them with Git, validate them, and deploy them automatically using CI/CD flows, providing traceability, reviews, and consistent deployments.

Step 3: Initialize your Synthetics project

You will no longer need to connect to the hosts where you deployed the Elastic Agent, as the remaining steps can be performed locally as long as you have connectivity to Kibana!

Since Synthetics Projects is based on Node.js, make sure you have it installed. 

1. Install the package:

npm install -g @elastic/synthetics

1. Confirm your system is setup correctly:

npx @elastic/synthetics -h

1. Start by creating your first Synthetics project. Run the command below to create a new Synthetics project named synthetic-project-test in the current directory.

npx @elastic/synthetics init synthetic-project-test

1. Follow the prompt instructions to configure the default variables for your Synthetics project. Make sure to at least select your Private Location. Once that’s done, set the SYNTHETICS_API_KEY environment variable in your terminal, which allows the project to authenticate with Kibana.

2. To generate an API key go to Synthetics Kibana.

3. Click Settings.

4. Switch to the Project API Keys tab.

5. Click Generate Project API key.

More details for all the steps can be found here: Create monitors with a Synthetics project

Step 4: Add your heartbeat.yml files

Once the project is initialized, access the folder it has created and take a look at the project structure:

• journeys is where you’ll add .ts and .js files defining your browser monitors. It currently contains files defining sample monitors.

• lightweight is where we’ll add our heartbeat.yml files defining our lightweight monitors. It currently contains a file defining sample monitors.

Therefore, all we have to do is copy our heartbeat.yml files to this lightweight folder. Before copying heartbeat.yml, keep in mind that we don't need all the content, we are only interested in the heartbeat.monitors part. \ We recommend considering splitting the file into logical groups. Instead of maintaining a single large YAML file, you could create multiple smaller YAML files, with each file representing either a single check or a group of related checks. This approach may simplify management and improve compatibility with GitOps workflows.\ Each YAML file should look like this:

carles@synthetics-migration:synthetic-project-test/lightweight# cat heartbeat.yml

heartbeat.monitors:
- type: icmp
  schedule: '@every 10s'
  hosts: ["localhost"]
  id: my-icmp-service-synth
  name: My ICMP Service - Synthetic
- type: tcp
  schedule: '@every 10s'
  hosts: ["myremotehost:8123"]
  mode: any
  id: my-tcp-service-synth
  name: My TCP Service Synthetic
- type: http
  schedule: '@every 10s'
  urls: ["http://elastic.co"]
  id: my-http-service-synth
  name: My HTTP Service Synthetic

What we just did is define different ICMP, TCP, and HTTP checks as code.

Now we need to ask Synthetics project to create the monitors in Kibana based on what we have defined in our YAML files:

npx @elastic/synthetics push --auth $SYNTHETICS_API_KEY --url <kibana-url>

Unfortunately, we do not support a 1-to-1 mapping of the heartbeat schema to the lightweight schema, so you may encounter some errors during the execution of this command. One example is the definition of schedule. Heartbeat supports the use of crontab expressions, but Project requires the use of @every syntax.

If no syntax errors were found, the command output will show that the monitors have been successfully created in Kibana!

Then, go to Synthetics in Kibana. You should see your newly pushed monitors running. You can also go to the Management tab to see the monitors' configuration settings.