Installing Rally: Running your first Elasticsearch performance benchmark and air-gapped

Rally, also known as ES Rally, is the benchmarking tool that Elastic® relies on to identify Elasticsearch® performance improvements, regressions, and more. It is run nightly against a nightly build of Elasticsearch. You can also use it to benchmark your Elasticsearch cluster and identify any performance issues related to your setup. If you want to ingest 5TB of data daily, follow this blog post and the entire series to identify how you can ensure that your hardware is capable of this. If your workload is more on the search side, we will also cover that!

All of the installation steps are described in the official Rally documentation. You don’t need a Java installation when not benchmarking source build. This blog post series focuses on benchmarking an Elasticsearch running in our cloud.

Alternatively, you can use the official Docker image instead of installing it locally.

You need:

• Preferably the latest Git version, at least higher than 1.8

• Pbzip2

• Preferably the latest Python version, at least higher than 3.8

• Python’s pip

Once all these prerequisites are installed, you can move forward with or without virtual environments within Python. To keep it simpler throughout, we are not going down that road.

Install Rally using the following command: pip3 install esrally. Now it will install all the required packages, and you are nearly ready. In my case, I didn’t install Rally as root, and therefore, my installation path was not part of the $PATH.

If we are running any command like race then we will get an error that looks like this and is expected.

We are about to configure everything we need. What do we want to configure? Let us start with a diagram so it’s more straightforward to understand our setup. 

We have a target cluster that we want to benchmark and a stack monitoring cluster. In Rally, there are two ways to configure the metrics collection. That does not mean metrics around Elasticsearch, as you know from Stack monitoring. Its metrics are collected and generated during the benchmark. Rally tracks how long each task takes, the latencies, etc. This can be exported to Elasticsearch, allowing us to visualize it. 

Since we also have the Stack Monitoring data, we can do overlays and see the CPU usage of the cluster versus the latency of each call. We would discourage sending the metrics to the same cluster that you are benchmarking since you add an additional uncontrolled load on this cluster. Check out the official documentation. 

To alter the metrics exporter, we need to change the Rally configuration file located in the installed user's home directory. In my case, it’s /home/philippkahr/.rally/rally.ini. A blank configuration looks like this:

The important configuration section is the reporting part. Here we are going to put the Stack Monitoring cluster. We are going to create a new superuser called “rallymetrics.” We use this superuser since Rally can alter the permissions needed in any release.

For anyone using an OnPremise cluster and having self-signed certificates and errors because of that, there is a setting available: `datastore.ssl.verification_mode`. You can disable this by setting it to none or add your local certificates to the certificate store and point it toward this setting: `datastore.ssl.certificate_authorities`. Learn more about these settings.

I will break it down step by step.

esrally race
Just calls esrally and tells it to perform a race.

--user-tags={"benchmark_id": "1.1-http_logs-w20p1r1"}
Is used to identify any race uniquely, not every execution. I want the same race, with the same settings, to have the same tags. In this case, I just called it 1.1-http_logs-w20p1r1. Depending on what I do, I have used countless different namings. When you create a benchmarking playbook, you can use that number. It needs to be unique to compare different races against each other. Something like 1.1-http_logs-w20p1r1 is nice. You run task 1.1 from your benchmarking playbook, the track, and how many primaries and replicas 10 workers. You can specify them in a key-value pair similar to the client-options argument. Read more.

--track=http_logs
That is the track you are running. The track is called HTTP logs, which are the logs from the Web servers for the 1998 Football world cup. Some modifications to the original logs were done, all described on the GitHub for the track: http_logs . Here is a quick peek at the example document we are ingesting:

--kill-running-processes
This one is usually not needed. Sometimes when you cancel a track because you had an issue, esrally might not exit gracefully, and you have some esrally processes around. Rally will refuse to start until you kill all running processes. If this flag is specified, Rally will instead kill these processes itself.

--target-hosts=https://to-benchmark.elastic.co:9200
This is just where you set the target hosts. It can be one or more, depending on your setup. Always test the real data flow; if you have a load balancer in front of it, test it with the load balancer endpoint. Give your load balancer team a heads-up.

--pipeline=benchmark-only
Here you can tell Rally to spin up a cluster for you or just benchmark the cluster you pointed to. In most cases, benchmark-only is what you want to use.

--client-options="verify_certs:false,basic_auth_user:'rally',basic_auth_password:'rally123'"
The client options are always tricky to figure out without an example. It needs to be in the form of key:value and when more settings are needed, you separate them with a ,. Since this is a command line argument, don’t forget to escape special characters like !. The verify_certs is a nice little tool to tell Rally to ignore the certificates, which might be helpful for on-premise testing. In this case, I just write the user + password in cleartext in the command. You can also use an environment variable if you like. More about this can be found here: https://esrally.readthedocs.io/en/stable/command_line_reference.html#client-options.

--track-params='{"bulk_indexing_clients":20,"number_of_shards":1,"number_of_replicas":1}'
The last one is the meat and potatoes of any track. You can specify the track parameters like the client options above. The reason to pick JSON and get familiar with it is because more complex tracks, such as the Elastic Logs or Security track, have loads of options. Depending on the track, more or less parameters can be set. All possible parameters are described here: https://github.com/elastic/rally-tracks/tree/master/http_logs. Any default settings are explained. It needs to be in the form of a json. It can be a command line argument like this or a local parameters file.

In a table representation, we are using the following settings. A couple more settings must be described here, as they do not apply to us.

The original file is much longer than this excerpt, but this helps us understand what is happening and how to deal with it.

The first task in every track is the delete-index. We want to delete any existing index — otherwise, we cannot calculate correctly if all documents were indexed successfully, any segments, force merging durations, and so on are not representable values. The reasons for performing this as the first step and not at the end of the benchmark “as a cleanup” phase, if you will. Maybe you want to inspect the data in the cluster after the run, and you cannot do this if we delete it as a last step. Or when you cancel a running benchmark, this ensures that we always start fresh.

The second task is usually to create the index that is associated with this. We just do a PUT indexname.

The third task is very important as it is check-cluster-health, and it waits for the cluster to become green, signaling that all shards are assigned and the cluster is in a good healthy state. If you benchmark a single-node cluster, this cluster is never green, except when you make some changes to set all replicas to 0. 

This part: {{cluster_health | default('green')}} shows you that there is a parameter called cluster_health that you can override as we did with bulk_indexing_clients. Pass cluster_health: ‘yellow’ if you want to benchmark a yellow state cluster. I would not recommend it, though. This task will also be retried until the success criteria are reached.

Something interesting is coming up next, and that is an if condition. In this case, when we run with runtime_fields parameter, we run a different index-append task, more specifically: index-append-with-timestamp-pipeline. We have not used this parameter yet, therefore we land in the else portion and run the normal index-append. That is how those track files are built up and how they work. 

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.