Using Elastic Agent Performance Presets in 8.12


Big improvements to Elastic Agent performance in 8.12

The latest release, version 8.12, marks a significant shift in tuning for Elastic Agent and Beats. In this update, Elastic® introduces Performance Presets, aiming to simplify the tuning process for users and enhance the default settings for improved performance.

Default setting enhancements

The default tuning parameters used with Elastic Agent today have been in place since the beginning of Beats and were picked to ensure that data gets into Elasticsearch® as quickly as possible for analysis. Historically, tweaking performance of Elastic Agent and Beats involved deep knowledge, and benchmarking them for an optimal combination requires even more expertise.

With 8.12, we’ve made tuning Elastic Agent and Beats a thing of the past for the vast majority of customers. We’ve introduced new default settings for Elastic Agent and Beats that improve throughput by up to 50%, reduce memory usage by 10%, reduce concurrent connections to Elasticsearch by up to 80%, and reduce Disk I/O in Elasticsearch from Beats/Agent requests by up to 50%.

Introducing Performance Presets

If the new defaults don’t meet your needs, or perhaps you need more throughput or want to scale to a larger deployment of agents, the next step is not tuning parameters in a YAML file; instead, new Performance Presets take the best practices from our trusted customers, field teams, and performance test beds and allow you to pick the preset that best suits your needs, effectively offering a tailored configurations to meet your diverse performance requirements. Let's explore the four presets and the new Custom option and understand which one suits your needs best:

1. Balanced

From 8.12 onward, the Balanced preset is the default preset and optimizes Elastic Agent to achieve a reasonable level of throughput and resource utilization, making it the preferred choice for a wide array of Elastic Agent use cases.

For those of you who are familiar with the various settings available within Beats, here are the exact changes happening to the default values in 8.12. These new defaults increase the Events Per Second throughput (EPS) by 25%.

Please note: These settings may change in future versions as we continue to tune these for the best performance.

ConfigurationCurrent DefaultBalanced

2. Optimized for Throughput

The Optimized for Throughput preset focuses on achieving 4x higher data ingestion rates, making it ideal for scenarios where you need Agent to handle high volumes of events. It takes the best parts of the Balanced preset but more freely consumes CPU and memory and multiplexes events across multiple additional connections to Elasticsearch.

3. Optimized for Scale

If you’re deploying Elastic Agent to tens of thousands of systems, the Optimized for Scale preset fine-tunes Elastic Agent for gathering lower volumes of host data like logs or metrics but from tens or hundreds of thousands of devices. It takes the best parts of the Balanced preset, but trades a short delay in data ingestion for a total 30x reduction in open connections to Elasticsearch over the defaults. This makes a big difference if you’re using a reverse proxy or load balancer in front of Elasticsearch. This setting’s EPS performance is fairly similar to that of the Balanced setting.

4. Optimized for Latency

The Optimized for Latency preset reduces the time between data ingestion and processing, making it suitable for low throughput but real-time analytics and applications that demand low latency. This preset closely mirrors the pre-8.12 Elastic Agent default settings and is the perfect fallback for any issues encountered testing the newer presets.

In your Elastic environment, it’s important to note that these presets only impact Agents on version 8.12+. Older versions of Agent will continue to use their existing settings.

5. Custom

While presets are designed to simplify the tuning process for Elastic Agent, the Custom option allows the user to have more granular control over performance. You can still refer to the old guidance provided by Elastic, which is still relevant on Agent as the are now also configurable. The old guidance, available here, offers practical examples and insights on tuning Elastic Beats performance, including batch size, worker count, and more.

To provide further guidance, the table in our public documentation shows the effects on EPS throughput performance as a function of the performance of the Balanced preset, for various permutations of these tuning parameters, for users who require more granular control. These parameters can be set when the Custom preset is chosen.

In conclusion, Elastic's new Performance Presets provide a convenient way to fine-tune Elastic Agent for different performance requirements. Whether you need a balanced performance, high throughput, scalability, or low latency, these presets offer optimized configurations to meet your needs. Happy tuning!

What else is new in Elastic 8.12? Check out the 8.12 announcement post to learn more >>

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.