JVM essentials for Elasticsearch: Metrics, memory, and monitoring

Elasticsearch is a Java-based search and analytics engine and vector database built on Apache Lucene, and it is the core of Elastic’s Search AI Platform. To run Elasticsearch on its supported platforms, you need a Java virtual machine (JVM). JVM provides a platform-independent runtime environment, and you can run Elasticsearch in a virtual environment on top of the existing operating system. The JVM abstracts the underlying OS and hardware, so Java applications can run on any platform.

Understanding JVM’s memory management and object reaping via garbage collection (GC) is the key to troubleshooting issues like java.lang.OutOfMemoryError (Exit code 127). These issues occur when JVM runs out of memory or when there’s an Exit code 137, which typically indicates that the host’s out-of-memory (OOM) killer terminated the JVM process due to excessive memory usage. This blog will guide you on how to examine the memory usage patterns and troubleshoot these JVM issues.

I will also explain the role of JVM and how to correlate this with the help of existing Elasticsearch APIs, providing valuable insights into the state of the Elasticsearch JVM and helping you decide if you want to tune it further according to your own use case. 

As mentioned in Elasticsearch JVM documentation, the default values of the JVM options that are shipped with the product work efficiently in all of the supported production environments; these default values can handle most use cases for search and indexing operations. Changing any values or adding any custom values for the JVM options or settings is not recommended.

A Java virtual machine is an integral part of Java Runtime Environment (JRE), which comes bundled under a Java Development Kit (JDK).

JVM is similar to a computer program that helps run Java applications. It doesn't exist as a physical machine, but it acts like one by translating Java code into instructions that your supported operating system can understand. The JVM also takes care of important tasks like memory management, handling garbage collection, and keeping your applications secure.

Next, let’s focus on JVM memory management and garbage collection — a major point in Elasticsearch that we frequently check when troubleshooting any memory performance or OOM-related issues.

To understand memory management and garbage collection, first you need to understand the logical representation of the heap area (or heap memory) where all of the objects reside when a JVM starts.

Young generation area/region
When any Java application starts, it creates new objects and allocates these new objects to the area called heap memory. The young generation’s Eden space is the first place where the application allocates newly created objects. This area is reserved for the allocation of new objects. And when it becomes full, the eligible objects will be either reaped (garbage collected) or promoted to other regions.

When a minor garbage collection happens, the referenced objects will be moved to another subsection which is Survivor region S0. And the unreferenced objects will be deleted to free up the space in Eden. This region contains the objects that survived the garbage collection process, hence the name Survivor region. 

The same cycle repeats when the Eden space is full again. This time, the objects that survived both Eden and S0 will be moved to the S1 region, and this process continues. 

The objects that survived the garbage collections in Eden, S0, and S1 will be promoted to the old generation based on the age calculator.

Old generation area/region
The old generation (also known as the tenured generation) is a collection of long-lived objects that survived the multiple garbage collection events in the young generation (minor GCs). Objects that have survived a certain number of cycles in the survivor space are moved by the algorithm to the old generation, also known as object promotion.

The above saw tooth GC pattern from stack monitoring shows that when minor garbage collection is happening, the JVM heap memory is getting reduced because GC is removing the unwanted objects from the memory. After some cycles, objects accumulate and GC kicks in again to clear the objects. 

The G1GC has a pause-time target (-XX:MaxGCPauseMillis=200) that it tries to meet during the garbage collection. This means during the garbage collection, it ensures that JVM does not experience a pause more than this value, making it better than CMS or other collectors where this is not an option. Overall, it is recommended to keep the GC settings as default values, which will keep the GC pause within the limits for enhanced performance. 

This API will provide the details of the node configuration, including JVM input arguments, memory pools, and heap size. 

Additionally, if you want to check the metrics of the Elasticsearch JVM for a particular node’s uptime, heap used in all the memory pools, or statistics about the garbage collection, you can use the node statistics API. It will provide detailed metrics about each of the mentioned metrics.

If you are an advanced user and want to check real-time stats about a running JVM without using Elasticsearch APIs, then you can use the bundled java and run jstat to check the live statistics. To do this, first find the Java home directory or bin directory and the PID of the Elasticsearch process. From the bin directory, use the jstat tool:

Here, 2000 is the interval in milliseconds after which the stats will be fetched and printed on the CLI. This command is helpful to identify how frequently GC is happening in all the areas and how fast memory pools are getting filled by the objects (object creation and promotion rates). 

This is not an alternative to the APIs mentioned above, but it is more for developer-level debugging.

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.