Elasticsearch Testing & QA: Elasticsearch Continuous Integration

We're rolling out a three part series on our testing and QA processes for Elasticsearch. We'll be bringing you our second installment next Thursday, so stay tuned to this space!

Writing tests is a crucial part of developing new features for and fixing bugs in Elasticsearch. In order to make creating these tests as easy as possible, we use a couple of techniques to help developers. Many of these are available to our downstream users as well.

Elasticsearch CI

Elasticsearch regularly goes through quite a few test cycles, including after each new commit and triggered on a continuous basis.

After check in, the Elasticsearch code base goes through a set of automated tests (to a large extent inspired by what is run for Apache Lucene itself). On each commit, a smoke test verifies the system can be built at all, and whether basic functionality (connecting to the resulting search server, installing plugins, etc.) are possible.

Builds that pass this initial sanity check are examined further. The Java Unit and Integration test suites are executed, and afterwards backwards compatibility (where applicable) is verified. Finally, the REST tests are checked.

In addition to these checks, we run all Java level checks on a continuous basis on several hardware configurations and operating systems. All jobs are Jenkins managed. The key to reducing manual configuration overhead in our setting is an aggressive automation strategy. For the cloud in particular, it pays quickly to have dedicated scripts for initializing all instance types needed for testing. In our case this includes being able to spin up clusters of any topology to test various ingestion scenarios.

In the remaining two blog posts, we will first walk you through the various types of tests that we run to ensure Elasticsearch works as expected. Later in the series, we'll show how we use test randomization to increase test coverage not only at the Java level but also to optimize time and hardware resources needed to check our code base on a large set of deployment configurations including but not limited to testing against various JDK versions."