For new users, this is the fastest way to get high-quality vector search running. If you already use Elasticsearch, the new offering is vector search on the platform where your data already lives, with no new system to adopt. Elasticsearch Vector Database supports a range of scenarios, from grounding a large language model (LLM), to giving an AI agent retrieval and memory, to serving hundreds of billions of vectors. Spin up a new project and get started in minutes.

One engine, every vector use case

Elasticsearch Vector Database is built for anyone building applications using vectors:

• RAG: Retrieve the right context for your LLM with dense and sparse vector retrieval, or go with hybrid search combining both vector and lexical retrieval. The quality of your generation improves with the quality of your retrieval.

• AI agents: Give agents fast, filtered retrieval over documents and conversation memory, with the low latencies that multistep agent loops demand.

• Semantic search: Match on meaning, not keywords, with one field type and zero pipeline code.

• Recommendations and similarity: Find nearest neighbors across products, images, or whatever content you have, at scale.

Everything your vector workload needs, optimized out of the box

Building a vector-based application means wiring together several separate pieces: setting up and hosting embedding models, indexing your documents through them, storing the vectors efficiently, applying the embedding model to each query, matching against the vector store, and finally, retrieving the documents behind the matches. Elasticsearch Vector Database handles all of it for you, with no additional configuration or setup.

Vector indexing with vectordb_document index mode

vectordb_document index mode, a new index configuration purpose-built for vector-first workloads, is on by default, so you get the settings that experts would choose. Here's what it turns on:

• bfloat16 by default: Vectors are stored at half the size of float32 with negligible impact on recall, cutting your disk footprint roughly in half before quantization even enters the picture.

• Source vectors excluded: In Elasticsearch, your embeddings already live in the index structures used for search; keeping a second raw copy in _source just inflates storage and slows down fetching results. We exclude the duplicate so responses return faster and you store less.

• The right files preloaded into cache: The data structures that vector queries touch first are warmed into memory ahead of time, so your first (and your thousandth) query is lightning fast.

• Parallel merging: Merging consolidates segments into better-organized vector structures, which lifts both recall and latency, and running those merges multi-threaded means you get there faster.

Vector storage, compression, and auto-tuning

• Your vectors are compressed automatically. Better Binary Quantization (BBQ) shrinks vector memory footprints by up to 32x while preserving recall, and DiskBBQ reduces memory requirements further for large-scale workloads. 

• Opt in to auto-calibration, which tunes each segment's quantization to your data and retunes on every merge as data drifts. When tested across 18 datasets, queries per second (QPS) improved by an average of 16.7%, with recall gains in most of them.

Embeddings on managed GPU inference

• Generate embeddings with native Jina AI embedding and reranking models, or bring third-party models, all on managed GPUs via Elastic Inference Service (EIS) with no model servers to operate. Or self-host, if you prefer your own.

• The semantic_text field type automatically handles chunking and embedding, along with querying, the simplest path to semantic search in the market. 

Hybrid search and filtered vector search

• Hybrid search is built in, combining full-text and vector retrieval in a single query. Blend the results with reciprocal rank fusion (RRF) or any other blending mechanism you want. Vector search is usually the hardest part of hybrid search to configure well. With Elasticsearch Vector Database, you have it handled, and your whole hybrid stack gets better. 

• With filtered vector search, apply metadata filters as part of vector retrieval itself and not as an afterthought that wrecks recall.

Enterprise on day one

You also get role-based access control (RBAC), audit logging, and the compliance certifications that pure-play vector databases generally lack.

Affordable at scale and predictable

Elasticsearch Vector Database is built to stay affordable as you grow: BBQ and DiskBBQ compression that keeps storage linear and memory low means scaling to hundreds of billions of vectors doesn't blow up your bill. And what you do pay is built from numbers you already know: how much data you store and how much you index, along with how much search capacity you need. Estimate your document count and vector dimensions, plus your query load, and you can work out what you'll pay before you create the project. You can also understand your bill line by line at the end of the month. There are no opaque compute units and no surprise charges for background operations.

How to get started with Elasticsearch Vector Database

Create a serverless vector database project

Create a new serverless Vector Database project in Elastic Cloud. Point your data at the endpoint, and you're ready to index.

Create an index using semantic_text

Vector index mode handles the vector configuration. Using semantic_text means that embeddings and chunking setup are managed for you, as is index setup, on managed GPU inference, with no embedding pipeline to build.

Ingest documents

Index text, and the embeddings are generated for you.

Run a semantic search query

Query the same semantic field you just created:

And you get results back:

Semantic search is just the start. Run fully textual queries or combine both into hybrid queries. You can even craft your own vector queries for full control. Follow the semantic search quickstart in the docs for the full instructions.

What's next for vector search in Elasticsearch

We're already working on the next improvements:

• Better multi-tenant handling: If your data needs to stay separated per tenant, we'll give you a way to do it faster and with less code.

• Automatic index optimization: From "brand new index" to "fully optimized," with as little tinkering as possible.

• Continuous infrastructure improvements: Ongoing tuning of Vector Database's settings and infrastructure so you're always getting the best throughput and fastest responses.

Try Elasticsearch Vector Database on Elastic Cloud Serverless

Go from an empty project to a hybrid, filtered vector query in minutes, with production-grade defaults doing the tuning for you. Build fast, scalable AI apps, not infrastructure.

Start on Elastic Cloud Serverless, or dive into the full documentation and API reference. You can also access the new offering on AWS Marketplace, Google Cloud Marketplace and Microsoft Marketplace.

Why snapshots pin primary shards in stateful Elasticsearch

In traditional stateful Elasticsearch, snapshots lock primary shards to their active nodes, completely preventing relocation. That works fine when cluster topology is stable and nodes stay online between maintenance windows, but stateless Elasticsearch works differently. Index data lives in an external object store, with local disk as a cache, and the cluster scales automatically based on CPU, memory, and data size, both vertically (upsizing nodes) and horizontally (adding nodes).

During vertical scale-up, existing nodes must vacate all shards and shut down before new hardware takes over. Since Elasticsearch version 8.13, shard snapshots can pause during node shutdowns and resume after relocation, so a long-running snapshot doesn't block an infrastructure update.

Horizontal scale-out is a different story: No nodes shut down, so pause logic never triggers. New nodes sit idle while existing nodes finish their snapshots, and as soon as a snapshot is queued, the primary shard is pinned to its node, significantly delaying relocation.

Clusters typically scale out because they're already under heavy load. Blocking shard relocations at that moment limits the cluster's ability to reduce pressure, which shows up as degraded indexing throughput and higher latency. The resource imbalance can also trigger unexpected autoscaling behavior. And even when overall topology stays the same, shard locking disrupts hotspot mitigation and workload distribution. These failures used to surface as server-side warnings: "undesired allocation due to snapshot."

Stateful Elasticsearch

Stateless Elasticsearch

Snapshot reads from

Local shard data on the node holding the primary

The object store, using file locations recorded in the commit

Primary shard during snapshot

Pinned to its node until the snapshot completes

Free to relocate at any time

Effect on horizontal scale-out

New nodes wait for in-flight snapshots before taking shards

New nodes take shards immediately, regardless of snapshot state

How stateless snapshots read directly from the object store

A shard snapshot pins primary shards because it needs to read local shard data. In stateless Elasticsearch, that data already lives in the object store, so reading from local disk is unnecessary. Letting snapshots read directly from the object store removes the requirement to lock primary shards. They can relocate freely, and backup is decoupled from cluster balancing.

Stateless commits include location information for each data file in the object store, so snapshots can read and stream directly to the snapshot repository (a separate object store bucket). In the future, we plan to look at server-side ranged copies, which object stores support natively, to skip the local copy step entirely.

Tracking commits when shards relocate mid-snapshot

A snapshot is bound to a specific commit point that determines which files to back up, and those files must remain accessible for the full duration of the operation. In stateful clusters, this is simple: The snapshotting node and the data node are the same, so the commit is managed locally and held until completion.

In a stateless model, the snapshotting node and the data node can be entirely separate, or they can diverge if a shard relocates mid-snapshot. To handle this, we added a transport action that acquires commits on remote data nodes over the network. The data node tracks which commit belongs to which snapshot and releases it once cluster state signals completion.

There's a wrinkle during relocation. A stationary shard relies on its commit point to preserve files. A relocating shard must release its commit so its local store can close cleanly. To keep files accessible through that transition, a newly recovered primary temporarily preserves all existing data files in the object store until notified of snapshot completion via cluster state. This handles both graceful relocations and ungraceful recovery from node or engine failures.

No more allocation delays and improved cache stats

After stateless snapshots shipped, the server-side "undesired allocation due to snapshot" warnings stopped. The chart below shows the before and after, with the release marked by the red arrow. Hotspot mitigation became more responsive because shard relocations no longer had to wait for backup operations.

Cache use is also improved. Snapshots that bypass local shard data stop competing with indexing for cache space. After the release (also marked in the chart), we observed the following two positive changes in cache metrics:

1. The median cache population throughput, defined as bytes per second for filling the local disk cache from the object store, increased about 50%.

2. Cache misses, where data must be retrieved from the object store to fill local disk cache, have dropped more than 60%. 

What comes after stateless snapshots

Object-store-native architectures are increasingly the standard for cloud-native data systems, and stateless snapshots are a step toward fully exploiting that model across Elasticsearch operations. Backups read from the object store, and shards move freely. Neither process waits on the other. Removing the local shard dependency is a step toward further modularizing the stateless architecture.

How Elasticsearch Shard Balancing Works

Elasticsearch uses a weights-based algorithm to create a Desired Balance, an assignment of shards to data nodes. The Balancer determines the target allocation of shards across a cluster of nodes using four key metrics weighted in a linear algorithm. A total weight is calculated per node, and the shard balancer aims to equalize the total weights across cluster nodes. A final Desired Balance shard allocation is precomputed based on the latest cluster state information, and then the elected master node initiates incremental shard moves to reach the desired shard allocation.

The four metrics are:

• Write Load: the total write threadpool activity per node, using the sum of threadpool indexing activity per data-stream shard.

• Disk Usage: the total disk usage of shards per node, using the sum of disk space used per shard.

• Shard Count: the total number of shards assigned to a node.

• Index Balance (shard anti-affinity): per index, how many shards in the index are assigned to the node.

The total weight of a node is calculated using a linear algorithm that finds the deviation from the node-level cluster average for each individual metric, applies a different weight factor multiplier to each, and then takes the sum of all resultant values. The weight factor multipliers attempt to equalize the relative magnitude of each metric so that metrics with large values do not eclipse metrics with naturally small values. Write load tends to be a small value, related to thread usage, and thus gets multiplied by a relatively larger weight factor of 10; whereas disk usage in bytes is a very large number and therefore gets multiplied by a tiny weight factor of 2e-11.

The following are the cluster settings with default values, representing the different weight factors:

cluster.routing.allocation.balance.shard: 0.45

cluster.routing.allocation.balance.index: 0.55

cluster.routing.allocation.balance.disk_usage: 2e-11

cluster.routing.allocation.balance.write_load: 10.0

The linear algorithm looks something like this:

Shard movements are triggered to ensure that the difference in total node weight across cluster nodes remains below the cluster.routing.allocation.balance.threshold with a default value of 1: whenever the threshold is exceeded, shards are moved from the most heavily weighted nodes to the least heavily weighted nodes until the difference between the most heavily weighted and least heavily weighted node is at or below the threshold. Whenever cluster activity occurs that changes shard allocation (e.g., create/delete index, add/remove node, or the disk usage grows), the Balancer rechecks the weights across nodes and triggers shard rebalancing if the delta between the most and least heavily weighted nodes exceeds the configured threshold. The threshold-based approach attempts to balance the trade-off between keeping the cluster perfectly balanced and minimizing shard movements. Large Elasticsearch deployments that use nodes with greater resources typically benefit from raising the threshold setting: a larger weight delta between nodes reduces shard rebalancing.

Shard movement is also constrained by strict shard assignment rules that prohibit certain node assignments according to cluster and index level settings. Examples include: not assigning copies of the same shard to the same node or host; not allowing further assignment of shards to a node that does not have spare disk space; and excluding node(s) as host for a particular index. More on this below.

How a Balanced Cluster Looks (Based on Weights)

Using the linear algorithm and cluster setting defaults previously described, the following is an example of what the Balancer considers balanced. Notably, it can sometimes allow considerable deviation across nodes in any one particular metric. For simplicity, index balance is not included.

Weight Node1 = 0.45 (5 - 6) + 10 (0.3 - 0.33) + 2e-11 (1e+11 - 8e+10) =   - 0.35

Weight Node2 = 0.45 (7 - 6) + 10 (0.4 - 0.33) + 2e-11 (4e+10 - 8e+10) =   0.35

Weight Node3 = 0.45 (6 - 6) + 10 (0.3 - 0.33) + 2e-11 (1e+11 - 8e+10) =   0.10

Limitations of Weights-Based Shard Allocation

Elasticsearch Serverless deployments are managed by Elastic and could run into many edge cases that, without additional configuration, the weights-based shard allocation handles poorly. In self-managed Elasticsearch deployments, it is possible to work around many of these issues by configuring the cluster to suit the user’s workload. However, Elasticsearch Serverless is configured once and must work across all customer use cases. Issues experienced by some Elasticsearch customers (early adopters of Serverless among them) include:

• Continuous rebalancing background noise in active clusters. This could be because the threshold setting needs tuning or because the cluster is very busy.

• The Balancer’s behavior cannot be tuned in a predictable manner. Adjusting the Balancer settings (individual weight factors) can lead to unpredictable outcomes due to the linear algorithm. For example, decreasing the shard count weight factor relative to the other weight factors can lead to data node OOM events when shard count balancing is deprioritized and too many shards pile up on a single node.

• No explanation of why the Balancer is making shard moves. The linear algorithm is difficult to understand without relevant node metrics.

• The linear algorithm allows a high value in one metric to cancel out a low value in another metric. For example, a node can have a higher than average (across cluster nodes) write load, but counterbalance with a lower than average shard count (or vice versa), and the linear algorithm cancels out the spikes: no shards are moved to address the write load hotspot.

• Index-level hotspots can occur when a disproportionate number of index shards are assigned to the same node, rather than spreading out across nodes, despite the index balance weight in the linear algorithm. Index balance weight can, in some situations, be little compared to the other weight factors. It can also get skewed and counterbalanced by another non-average individual weight in the linear algorithm, as described in a previous bullet.

• No search load balancing.

• Regular indices do not have write load estimate support, leaving some write load hotspots unaddressed. Only data stream indices have write load estimates.

• Write load hotspots can be missed. Write load estimates are only refreshed at rollover time, which can be infrequent in some configurations, causing new load to be ignored for some time. The write load is also the average write load activity over a potentially large window of time between index rollover events, so temporary write load increases can disappear when averaged with inactive write periods.

The above issues persist in some Elasticsearch deployments and require monitoring and workload tuning to manage when they do occur. Shard allocation balancing in Elasticsearch Serverless aims to address these issues and avoid any manual intervention requirements using a new approach that is explained in subsequent sections of this article.

Elasticsearch Serverless Shard Allocation 

Elasticsearch Serverless considers node resources individually: shards are rebalanced away from a node when any resource usage on that node grows to threaten performance, and shard movements to a node are declined when the assignment could threaten that node’s performance.

The Elasticsearch single combined score per node is replaced in Elasticsearch Serverless with independent per-resource decisions:

Elasticsearch Weights-Based Balancing

Elasticsearch Serverless Resource-Aware Deciders

Decision Basis

Single weighted sum across four metrics

Each resource evaluated independently

Metric Interaction

A high value can offset a low one

No offsetting; each decider acts separately

Decision Types

YES / NO

YES / NO / NOT_PREFERRED

Rebalancing Trigger

Weight delta across nodes exceeds threshold

Individually configurable safe limits per resource

Explainability

Can only make an educated guess

Each move traces to a named decider

The Elasticsearch Balancer has three phases, in order of priority, for shard movement decisions. The first phase is to assign unassigned shards. Assignment of unassigned shards is the top priority for data availability reasons. The second phase is to move shards that can no longer remain where they are assigned due to cluster configuration changes. Internally, AllocationDecider implementations enforce cluster settings, like index-level shard allocation filtering, shard allocation awareness, disk usage thresholds, or moving shards off of a node before shutdown. The third phase rebalances shards when the cluster.routing.allocation.balance.threshold is exceeded, using the previously described weights algorithm.

The new Serverless balancing approach adds additional logic to the Balancer’s first and second phases, leveraging the existing AllocationDecider logic, and eliminates the third phase. Previously, each AllocationDecider had simple responses of YES and NO. Now, the decision type of NOT_PREFERRED has been added, along with several new AllocationDecider implementations. An AllocationDecider will return NOT_PREFERRED when it observes that performance might suffer from a shard’s assignment to a particular cluster node. The Serverless Balancer will prefer a node assignment for the shard where all AllocationDecider implementations reply YES.

A NOT_PREFERRED shard allocation may be left uncorrected if all other node assignments return NO or NOT_PREFERRED. Such responses mean that the shard cannot be assigned elsewhere without either violating a cluster/index rule or potentially degrading the performance of another cluster node. Serverless Autoscaling activates before all cluster nodes hotspot: even one unaddressable hotspot leads to a scale-up event. New AllocationDecider implementations have also been added for important finite resources, like available heap memory (further discussion below), using only the original YES and NO decisions: exceeding certain categories of resources can lead to node unavailability.

The individual weight metrics in the Balancer’s linear algorithm have been replaced by resource-aware AllocationDecider implementations, and new AllocationDecider implementations are being built for additional resources: Serverless Search Tier load-balancing improvements are currently in development. Each shard migration will have a clear purpose to address a potential resource usage bottleneck.

Internal stats have shown far fewer shard movements in general, without any noticeable accompanying node performance degradations – one workload showed a 50% reduction in shard movements with the same write throughput. Fewer shard movements has the benefit of: avoiding momentary read/write latencies from warming up local caches; and saving on cloud infrastructure costs moving data between servers.

Serverless IndexBalanceDecider: Avoid Colocation of Index Shards

The IndexBalanceDecider ensures index shard anti-affinity much more strictly than the original weights-based linear algorithm could achieve. Colocation of index shards in excess of the index’s average shards per available node is avoided, except in the case of a strict NO assignment (essentially non-existent right now in Serverless except for shutting down nodes and rolling upgrade incompatible version checks) or NOT_PREFERRED assignment due to temporary node hotspotting.

The IndexBalanceDecider is a very effective means of pre-balancing both write load and search load before user workloads begin to generate load statistics: each index begins life with its shards distributed across as many nodes as possible.

IndexBalanceDecider Results: Even Write Load Distribution Across Cluster Nodes

Write load across data nodes became much more evenly distributed after the IndexBalanceDecider was enabled in the Serverless Production environment. Projects fleet-wide generally show even ingest load (counted in saturated WRITE threadpool threads), combining the release of the IndexBalanceDecider and many other prior improvements:

A reproducible workload demonstrates a clear before and after view of the impact of the new IndexBalanceDecider when an ingest workload was run with and without it enabled:

The IndexBalanceDecider also serves in the Serverless Search Tier to distribute shards of the same index as much as allowed, similarly limited only by the tier’s node count and the number of shards in each index.

Serverless HeapUsageDecider: Assign Shards by Available Heap

The HeapUsageDecider limits shard count on a node based on available heap to hold in-memory shard metadata and run associated write/read operations, removing the dependency on shard count limits per node. The HeapUsageDecider returns a strict YES or NO decision, rather than using the new NOT_PREFERRED decision type, because a data node risks an OOM event if the estimated available heap memory is exceeded.

HeapUsageDecider Results: Reduced Data Node OOMs

Data node OOMs in the serverless index tier decreased significantly as the HeapUsageDecider rolled out to the Serverless production environment.

Index tier OOM errors still occur from time to time in the Serverless Index Tier, though at a much reduced rate, as miscellaneous runaway memory usage edge cases are surfaced. The remaining OOM errors are being progressively resolved as they are identified, through a combination of memory usage improvements in the code, adding component level limits, and updating the internal Elasticsearch Serverless memory model service to more completely account for memory usage.

The HeapUsageDecider is not yet turned on in the Serverless Search Tier, due to the need for additional and different metrics, but that work is in active development.

Serverless WriteLoadDecider: Prevent and Correct Write Load Hotspots

The WriteLoadDecider receives periodically refreshed (every 30 seconds by default) per shard and per node write load stats and uses the data to correct and avoid write load hotspots. The master node retrieves stats directly from each data node’s write threadpool: an Elasticsearch node tracks the total time that its WRITE threadpools is in use, and each individual Elasticsearch shard instance tracks how much time it spent using its node’s WRITE threadpool.

A write load hotspot is identified at the node level. The criteria for a hotspot is the presence of WRITE threadpool queue latency above a configured threshold and sufficiently high, and sustained, WRITE threadpool thread saturation. Once that situation is detected, the Balancer is signaled to select shards to move away from a hotspotting node, until fresh non-hotspotting write load stats are received from the node. The Balancer will do nothing if all nodes are hotspotting at once, expecting the Autoscaler to solve the problem by introducing more, or bigger, data nodes to the cluster.

The WriteLoadDecider uses a heuristic to choose shards to move away from a hotspotting node that aims to minimize ingest disruptions while still effectively reducing a node’s write load. A shard write load threshold is identified on a hotspotting node: the threshold is currently calculated as ½ the ingest load of the hottest shard on that node. Shards that can be moved are then prioritized in the following order:

threshold = ½ * maxWriteLoadShardOnNode

1. Shards with write load in the range [threshold, maxWriteLoadShardOnNode), the shard at or closest to threshold preferred.

2. Shards with write load in the range (threshold, 0], the shard closest to threshold preferred.

3. Shards with write load equal to maxWriteLoadShardOnNode.

4. Shards with zero write load.

The heuristic prefers to avoid disruption to the highest ingest shards and instead chooses middlingly loaded shards. Movement of the hottest shard will cause the most latency disruption; and movement of the coldest shards will be the least effective in resolving a hotspot.

The Balancer limits write load hotspot correction shard moves to one move per hotspotting node per stats refresh period, in order to see the effect of a move in real-time node-level write load, before attempting any further corrections. This was a simple initial design that proved effective. Furthermore, the Balancer will not move a shard whose write load alone is sufficient to meet the node-level hotspot criteria: this would just relocate a hotspot to another data node, not actually resolve the hotspot. The Serverless Autoscaler and Serverless Autosharding components are relied upon to resolve hotspots that reallocation of shards cannot.

The WriteLoadDecider returns NOT_PREFERRED when acceptance of a shard could cause a node to start experiencing WRITE threadpool queue latency and create a hotspot. A shard will still be relocated to a NOT_PREFERRED node, however, and risk some performance degradation, as a better option than, say, risking a data node OOM from keeping a shard on a data node where the HeapUsageDecider returns NO.

WriteLoadDecider Results: Hotspots are Quickly Corrected 

Hotspot stats showed general improvement as the WriteLoadDecider was rolled out to Serverless production, in particular the fleet-wide time to correct a hotspot decreased greatly:

Since these graphs were collected, additional work has been released incrementally to better prevent and correct hotspots, and improvements are still in progress.

Serverless Autoscaling, Autobalancing, and Autosharding

Elasticsearch Serverless relies on both new autobalancing logic and new autoscaling logic. The Serverless Balancer must sufficiently distribute shard resource usage across nodes in order to fully saturate the cluster’s resources. The Serverless Autoscaler will trigger a scale-up event when it receives a report that a certain percentage of the total cluster resources are in use and more resources are needed. The Autoscaler will not scale up the cluster if one node is hotspotting and another node has an excess of available resources because the resources are summed across nodes. Therefore, the Balancer must first do a good job on load distribution, and then the Autoscaler will activate as needed.

Autosharding based on write load is also in progress and coming soon to address shard hotspots. Elasticsearch Serverless projects have a default number of shards per index based on the project type. These defaults generally work, but do not account for all possible workloads. Hotspots can occur when an index has too few shards, as well as too many. Too few index shards leads to the Balancer being unable to further distribute an index’s write load across available data nodes, and then the Autoscaler will not see a problem because the cluster-level resources are not fully consumed. Conversely, indices cannot by default have too many shards, since that could degrade search performance for small indices and potentially strain cluster metadata operations if the total number of shards in a cluster grew too large.

Production Example: 708 TB Data Set, 37 Index Tier Nodes (not counting Search Tier), 4,100 Indices, 30,000 Shards

The following graphs cover a period when the Index Tier, in an Elasticsearch Serverless project, scales up from 10 to 37 indexing nodes and then back down to 10 after a write load spike dissipated.

Graph of the Ingest Load Per Index Node

This graph shows fairly even distribution of load, though a little less even temporarily during scale-up. There are nearly 250 fully saturated write threads at peak load. 

Graph of CPU Saturation Per Index Node

CPU usage remains within safe bounds. Usage is mostly below 60%, except for momentary outliers that reach into the 90% range as write load rises before nodes are added to the cluster.

Graph of WRITE Threadpool Queue Latency Per Node 

When a node’s WRITE threadpool is fully saturated, tasks are placed in the threadpool’s queue. Queuing can happen with few tasks, if active write tasks are long-running, or there may simply be a lot of tasks.

This graph’s time window is zoomed in further than the others. One node reaches 75 seconds of queue latency during the scale-up spike. There are 29 nodes when the queue latency spike occurs at 19h25m, before autoscaling calls for 37 nodes at 19h28m.

Graphs of Total Cluster Ingest per Second, in Documents and MBs

Ingest rate peaks at 190,000 documents / second and 54.40MB / second. The document ingest rate is respectable at 4000-5000 docs/sec per node. The MBs ingest rate, however, is very low in this case: this can happen when indexing operations involve heavy computation. Document ingestion rate can also vary depending on the size of the documents.

What’s Next for Elasticsearch Serverless Balancing

The team is currently working on shard balancing improvements for the Serverless Search Tier, focusing on creating metrics and AllocationDecider implementations for search performance. The team is excited to share these improvements soon!

Why OAuth is better than API keys for MCP server authentication

With OAuth, tokens are short-lived credentials that expire on their own, so a leaked token is a narrowing window rather than a standing grant. Every token traces back to an explicit consent: which user, which client, which time.

In contrast, an API key is a long-lived credential. It lives in a configuration file on the machine that runs the agent, working for whoever uses it until someone rotates or deletes it. That model is manageable for a CI pipeline you wrote and deployed for your team. It gets uncomfortable when the key is held by an AI agent that assembles its own requests, retrieves untrusted content that may contain injected instructions, and sometimes passes context to sub-agents.

The failure mode is familiar from every credential-leak postmortem: the key ends up somewhere it shouldn't (a log file, a prompt), and from that moment anyone who has the key can access everything its creator could. The audit trail doesn't help much: API key logs tell you that a key with a given name did something, but not who authorized the client that used it or when.

Attribute

OAuth 2.1

API Key

Credential lifetime

Short-lived, auto-refreshing

Long-lived until manually rotated

Audit trail

User, client and timestamp per connection

Key name only

Revocation

Per connection, immediate

Requires key rotation

Blast radius

Single client connection

Everything the key creator can access

How to set up OAuth 2.1 for the Elasticsearch MCP server

Setup is a one-time step per project:

1. In your Elastic Cloud Serverless project, open Agent Builder → Tools library → MCP clients → Create MCP client (OAuth). This gives you a client ID and the MCP server URL.

2. Add the server to your MCP host. In Cursor or Claude Desktop, the configuration file entry looks like this:

3. The first time the agent calls a tool, the host opens your browser on an Elastic Cloud consent screen. You sign in with your normal Elastic Cloud credentials and see what the agent is requesting access to.

4. Click Authorize. This creates an application connection between you, your machine, and your project, and Elastic Cloud issues the agent a short-lived access token.

From there, your agent has access to your project’s data. The mechanics are invisible. When the access token expires, the host refreshes it using a refresh token with a 30-day rolling expiry, so you are not re-authenticating every hour. If you stop trusting a connection, open the application connections page in Elastic Cloud and revoke the connection. Both tokens die immediately, and nothing else about your account changes.

The agent acts with the permissions of the user who consented. Calls to Agent Builder tools, such as ES|QL queries, Workflows, and Streams, are evaluated based on your role. If you want an agent with less access than your own, authorize the connection as a user with a tighter role, or keep using a scoped API key for that workload. 

How to manage and revoke MCP server connections in Elastic Cloud

Org owners can list every active application connection across an organization or a specific project: which MCP hosts were authorized, by whom, and when. Revocation is per connection, so cutting off one misbehaving agent does not disturb the authorizing user's account or any other integration. There is no shared credential to rotate and no blast radius beyond the one client.

This is the practical difference for teams. An OAuth app connection records who consented, to which client, and when. An API key log shows that agent-key-3 queried an index, but not who authorized that agent. 

What's next for OAuth authentication across the Elastic API

Agent Builder is the first Elastic surface behind OAuth, and the same authorization model will carry to the rest of the Elastic API surface, including Elasticsearch and Elastic Cloud management, as we work toward a single Elastic MCP endpoint.

To try it now, open Agent Builder in an Elastic Cloud Serverless project and create an OAuth client, or start with the documentation. If you don't have a project yet, you can start a free trial.

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

In our previous blog, we laid out the statistical model behind that calibration: a manifold model for how nearest-neighbor distances scale with index size, a Gaussian error model for quantization noise, and a closed-form way to combine the two into an expected recall@k for a given rerank depth. If you haven't read it, the one thing you need going in is this: given a candidate quantization encoding and a rerank depth, we can predict recall@k without building an index and benchmarking it, by fitting two small models to a sample of the corpus.

In this post we’ll go through how to score candidate configurations cheaply and how to leverage that to make merge-time decisions that are themselves cheap, correct, and consistent across real, constantly-merging indexes. This led to some pretty impressive improvements: we see almost 17% average improvement in QPS across a broad range of datasets all while increasing recall (in one case by a factor of 3). What’s more you get this immediately by adding one line to your index options, "auto_calibrate": true, and our plan is to make this our default once it has had the chance to bake a bit.

Why manual vector quantization tuning is unreliable

bbq_disk exposes several knobs: quantization bits for documents (1, 2, 4 or 7), a separate bit width for queries, an oversampling factor for reranking, and whether to precondition vectors before quantizing. None of these act independently, and their effect on recall depends on the data: a 4-bit/1-bit encoding might be plenty for one embedding model and clearly insufficient for another. A single index is also built out of many segments, merged over time, each with an eventually different vector distribution. Hand-tuning one configuration for an entire index is, at best, a compromise, which is the motivation for automatic calibration: let each segment have its own configuration, re-evaluated every time it is involved in a merge operation.

How Elasticsearch runs auto calibration at merge time

When a number of segments are merged and automatic calibration is enabled, Elasticsearch samples documents and queries from the vectors being merged and:

1. fits the manifold model over a sequence of nested samples of the merged corpus;

2. fits the error model, predicting the quantization error's standard deviation for each candidate (query bits, document bits, precondition) combination;

3. sweeps candidate configurations in ascending cost order; the candidate encodings are (1,1), (4,1), (4,2), (4,4) and (7,7) (query bits, document bits), each tried across oversampling factors of 1.25, 1.5, 1.75, 2.0, 2.5 and 3.0;

4. estimates recall@10 for each candidate using the model described in our first post, and stops at the first (cheapest) configuration predicted to hit the target of 90% recall@10.

The winning configuration (encoding, oversample factor, precondition flag) is stored directly in the segment's metadata, so it travels with the segment and is picked up automatically at query time unless a request explicitly overrides it.

Small segments skip this altogether: below 10,000 merged vectors, there isn't enough data to fit a reliable model, so Elasticsearch just uses the current DiskBBQ defaults (4-bit query / 1-bit document encoding, no preconditioning, 3x oversampling).

How the vector quantization cost model works

Following the principles described in our first post, we started by picking candidates with three nested loops that are essentially how you might imagine hand jamming a lookup table. Start with the quantization scheme as the outer loop, ordered cheapest to most expensive by document bits ((1,1) → (4,1) → (4,2) → (4,4) → (7,7)). Then we set the rerank depth within the middle loop, ordered shallow to deep (1.25× → 3.0×). Finally we set preconditioning within the inner loop (off → on).

That ordering has a cost model baked into it, it's just implicit rather than written down: exhaust every rerank depth at the current bit tier before ever trying more bits. Document bits were effectively the only resource priced as expensive; oversampling was treated as nearly free by comparison, since the sweep would always max out rerank depth on a cheap encoding before considering a pricier one.

The current implementation replaces that with an explicit, continuous cost function:

Query bits still don't factor into cost at all, only document bits (which drive index size) and rerank depth (which drives how many candidates get rescored per query). Preconditioning also stays outside the formula: Elasticsearch runs the whole cost-ordered sweep once with preconditioning off, and only if nothing meets the recall target does it re-run the sweep with preconditioning on, treating it as a fallback lever rather than something priced bit-for-bit against the other two.

With this cost model, rerank depth costs noticeably more per unit than a document bit, so the sweep will often prefer stepping up a bit tier over pushing oversampling deeper.

The main reason for this is that once you're running in a serverless deployment, compute and storage are billed and scaled independently, on very different clocks. An extra document bit is mostly a one-time, indexing-time cost; it makes the segment marginally bigger on object storage, which is cheap and doesn't need to be pre-provisioned against a spike in query traffic. It does carry a smaller recurring cost too, since quantized vectors sitting in page cache or loaded for scoring take proportionally more RAM per document as bit width grows, but that scales linearly and predictably with corpus size, and doesn't spike with query load. Rerank depth is the opposite: it's a recurring, per-query cost.

Every extra unit of oversample factor means fetching and rescoring that many more full-precision candidate vectors from disk, on every search request, for as long as the index is queried. That's compute and DRAM pressure on the search-serving tier, which has to autoscale in close to real time to match query concurrency. It sits on the hot path of the latency-and-cost budget in a way storage capacity, and the RAM footprint of the bits themselves, does not. Weighting rerank depth higher than document bits in the cost formula is what makes the sweep reflect that asymmetry.

Efficiently estimating vector quantization error

The cost model above works with the premise that the recall estimate behind it is trustworthy. The manifold and error models need to be accurate for the recall assessment to be trustworthy. While the manifold model of the k-th to N-th nearest neighbors distance is cheap to compute, the standard deviation of the quantization noise for a given candidate encoding is a bit more expensive in principle.

DiskBBQ uses fixed count clusters to accelerate nearest neighbor queries. Our quantization procedure takes advantage of this by only quantizing the vector residuals from the cluster centroids. This means as the data scales, the magnitude of vectors we quantize relative to the various components of the similarity calculation shrinks. As such, quantization accuracy increases. We need to account for this when converting our sample estimates to the segment as a whole.

Clustering the corpus at several sample sizes and fitting how the error scales with cluster size requires re-clustering a real sample of the corpus at several different sizes and fitting a regression model to see how the error shrinks as the effective cluster size grows. We also add a conservative +3σ margin on top of the fitted estimate to guard against noise in the fit itself. This is accurate and appropriately cautious; however, while benchmarking on common dense retrieval datasets, we found that performing several hierarchical k-means passes per candidate was expensive.

To speed things up, we tried approximating residuals with a synthetic isotropic-Gaussian formula. Instead of clustering increasing-size samples, this approach generated synthetic residuals from the manifold model's local density estimate. It was fast and fit for background merges, with the full repeated clustering approach reserved for force-merges only. However, it turned out to inflate error when embeddings (residuals) are anisotropic (some directions carry a lot more variance than others). As a result, the estimated error could grow significantly on strongly anisotropic data (e.g., Fashion-MNIST-style image embeddings).

So instead we looked for a still fast but more accurate way of calculating residuals. We opted for using a single clustering pass over a smaller sample (2,048 vectors). The clustering runs once per merge and is then warm-started for every candidate encoding evaluated afterward, instead of re-clustering from scratch each time. To get the error's dependence on corpus size, which the baseline learns by re-clustering at multiple sizes, this approach instead reuses the manifold model's invDim as a plug-in for that dependence, extrapolating from the single real measurement rather than fitting the size relationship separately.

We also trimmed the query sample used during calibration from 1,024 to 256 vectors, on the reasoning that a smaller sample is enough once the error is being measured from real data rather than synthesized (and validated by benchmarks). The net effect was comparable wall-clock cost to the synthetic residual formula it replaced, but grounded in real per-cluster residuals, accurate enough that force-merge and background merge could be unified onto one path.

As an example, we take five different benchmark datasets and calculate the quantization error standard deviation (SD) by directly measuring the gap between exact and quantized dot products on a sample of real (or, for the synthetic residual formula, fabricated) residuals, then extrapolating that measurement to the full corpus size. They differ only in how much sampling and regression goes into that extrapolation: the multi-sample scaling fit sweeps fifteen sample sizes and fits how error scales with cluster size, the single-pass real residual measurement takes one larger real residual sample and reuses the manifold's intrinsic dimension to estimate the size dependency, and the synthetic residual formula skips real residuals altogether and samples from a synthetic Gaussian from the manifold's expected rank distance. We treat the multi-sample scaling fit as ground truth in this comparison because it's the most sample rich of the three, not because it's a zero variance measurement of the "true" corpus-wide error (it has its own sampling noise too). The table below summarises the methods and findings.

Method

How it works

Speed

Accuracy

When used

Multi-sample scaling fit

Clusters at 15 sample sizes, fits regression

Slow

Gold standard

Ground truth baseline

Single-pass real residual

One clustering pass + manifold invDim plugin

Fast

Near gold standard

Background + force merge

Synthetic residual formula

Gaussian from manifold density estimate

Fast

Inflated on anisotropic data

Deprecated

In order to exchange methods, we only need to be confident that they agree. This question can be answered independently of the correctness of the actual estimates, which we verified in our previous post for the multi-sample scaling fit. The figures below report the predicted quantization SD and the predicted recall@10, which is influenced by how we estimate the error. We report the analytical recall the manifold model predicts as a function of the quantization parameters, given the estimated error distribution perturbing the true distance ordering. This way, we isolate the quantization error's effect on ranking from any separate recall loss the IVF index itself might introduce, which is a distinct error.

The single-pass real residual measurement's calculated error SD is closer to the multi-sample scaling fit (our gold standard), with respect to the synthetic Gaussian residuals. Consequently, the predicted recall is closer when using the single-pass + manifold plugin method. Indeed, we found the models to be essentially interchangeable regarding the indexing decisions they lead to. Critically, we lower the calibration overhead by an order of magnitude.

Auto calibration overhead on indexing performance

We compared the cost of auto calibration on indexing, when compared with ES defaults, over 18 public benchmarks. We noticed that more than 50% of the datasets report an auto calibration overhead below 2%. Three datasets report 16-27% overhead, while two datasets sit in the 31-35% overhead.

The merge overhead is larger for smaller datasets (Fashion-MNIST, FiQA) that get indexed in a few seconds; that is expected as the size of the vector samples being used for calibration is fixed and therefore more noticeable with tiny datasets. In fact, for larger datasets like DBPedia-Entity and HotpotQA (5M doc vectors) the overhead is sometimes not noticeable and within 11% in the worst case.

What quantization parameters does auto calibration choose?

Looking at the encoding auto-calibration landed on for each of the real datasets:

Query bits were 4 in every dataset. While query bits aren't priced into the cost formula, we still iterate through lower query bits first (e.g., at 1 bit doc vectors, we first evaluate recall for 1 bit query vectors, then for 4 bit query vectors); so it’s possible for some datasets to even choose symmetric 1-bit quantization. The center of mass is a 2-bit document encoding with somewhere between 1.5x and 1.75x oversampling; 4-bit only shows up for two genuinely harder datasets (Fashion-MNIST's image embeddings, GIST-1M), and 1-bit only for a handful of the text-embedding models that are most robust to quantization. In fact, our own models are among those that quantize best: we selected 1 bit documents for all three corpuses we tested with Jina v3.

Recall and QPS improvements from automatic calibration

Auto-calibration is a broad win across the eighteen datasets: QPS improves in 15 of 18 cases (often substantially, double digits on about ten, and over +50% on FiQA GTE, Fashion MNIST, and Glove-200), and recall improves in 15 of 18 cases too, including a dramatic +295.7% rescue on Fashion MNIST. Most datasets see gains on both metrics simultaneously, and even the more modest cases still land solidly positive, recall improvements are commonly in the high single digits to double digits, QPS gains follow a similar pattern. Where either metric does dip, the drops are small and contained: the three QPS regressions all stay under 1.5%, and the three recall regressions all stay under 2%.

How to enable auto-calibrated vector quantization in Elasticsearch

The feature is not enabled by default for now, and opt-in via auto_calibrate on bbq_disk index options:

With this set, you no longer need to guess at bits, oversampling, or preconditioning: each segment picks the cheapest configuration that's predicted to hit 90% recall@10 for its own vector distribution, and re-evaluates that choice every time it's merged.

What's next for automatic vector quantization in Elasticsearch

Our first post showed that recall could be predicted in closed form from a small sample. Turning that into something running inside a real merge path meant a second round of engineering decisions that the model itself doesn't answer: how to order a sweep over candidates so it's cheap in the common case, how to price oversampling against document bits given how each is actually paid for at query time, and how to estimate the error term itself cheaply without quietly wrecking its accuracy.

In the end, we have a feature that allows us to tailor indexing choices to the data characteristics, with less than 11% overhead to index time for large indices. This gives us the ability to accurately control recall while optimizing quantization and oversampling choices for query performance. We got an average increase of 16.7% in QPS when we enabled this feature compared to our previous default settings for DiskBBQ. All while reliably achieving our target recall. Taking away the configuration burden from the user actually allows us to make better choices; it is a win-win.

This is the beginning of a longer journey that we’re working on to bring automatic configuration based on a combination of better understanding of the operating environment and better understanding of the data characteristics. We look forward to sharing more of this work with you in the near future.

The party is getting crowded

You're hosting a pizza party. You've got a few friends helping you serve, each stationed at different spots around the room. You give each friend a pizza, and they start handing out slices to hungry guests as they arrive.

At first, things run smoothly. A few guests trickle in, your friends serve slices, everyone's happy. But then word spreads about your sourdough pizzas. The doorbell keeps ringing. Guests pour in. Soon, there's a crowd forming around one of your friends, the one holding the pepperoni pizza, which everyone seems to want.

Your friend with the pepperoni pizza is overwhelmed. Guests are waiting, getting impatient, and a large queue has formed. Meanwhile, your friend holding the margherita pizza is standing around with barely anyone asking for a slice.

What do you do?

You order a couple more pepperoni pizzas and hand them to other friends. Now three friends are holding pepperoni instead of one. The crowd spreads out, and suddenly you can serve three times as many guests at once.

A few things become clear as you host more parties:

• Not all pizzas are equally popular. Some are in high demand, others have fewer takers. You don't need extra "copies" of the unpopular ones. You need extras of the ones with queues.

• Order more pizzas before the queue gets too long. If you wait until your friend is completely overwhelmed and guests are leaving angry, you've waited too long. Better to get an extra pizza when you see a crowd forming.

• Don't throw away pizzas too quickly. Just because the crowd around the pepperoni thinned out for five minutes doesn't mean the rush is over. Maybe they're just refilling drinks, or even talking among themselves (is that still a thing?). Keep the extra pizzas ready. If the lull continues for a while, then you can put them away.

• You can only hand out as many pizzas as you have friends who are helping. If you've only got four friends helping, ten pizzas won’t change the outcome. Only four can be served at once. Match your pizza count to your available hands.

• When a friend leaves, take their pizza. If one of your friends needs to head out, grab their pizza immediately. You can't have pizzas sitting unattended. Hand it to someone else, or put it away.

From pizzas to replicas

Let's map this back to Elasticsearch.

In our analogy, pizzas are replicas (copies of your index shards), your friends helping serve are search nodes, hungry guests are search queries, and that popular pizza with a crowd around it is a hot index with high search load.

When search traffic increases on a particular index, we create additional replicas and distribute them across your search nodes. Any replica can serve any query for that index, just like any friend holding pepperoni can hand out pepperoni slices. More replicas means higher throughput: Three replicas can handle three times the queries per second of a single replica.

Measuring the hunger

Before we decide how many pizzas to order, we need to know how hungry the crowd is.

Elasticsearch tracks the search load for every shard. It's a metric that captures how much search activity a shard is handling. We aggregate this across all shards of an index to understand the total search demand.

What matters most is the relative search load: What proportion of your project's total search traffic is hitting each index? If one index is receiving 60% of all searches while another gets 5%, we know where to add capacity.

The math behind the pizzas

We calculate the optimal number of replicas following this formula:

Where:

• L = the index's relative search load (between 0 and 1).

• N = the number of desired search nodes in your project.

• S = the number of shards in the index.

• X = a threshold to avoid hot spots (default: 0.5).

An example: four search nodes, one index with two primary shards receiving 80% of search traffic:

This hot index gets four replicas distributed across the search nodes.

The threshold X (defaulting to 0.5) is important. We don't wait until a replica is completely overwhelmed; we scale up when it's at half capacity. Hand out the extra pizza when you see the crowd forming, not when guests are already leaving.

Scale up fast, scale down slow

When search load increases, we add replicas immediately. No reason to make users wait.

When search load drops, we wait a bit before taking any action. We need to see consistent low demand for about 30 minutes before reducing replicas. (This is to deal with spiky traffic where a quiet moment doesn't mean the party is over.)

This matters because adding a replica has a cost. The new replica copies data and warms its caches before serving queries efficiently. Removing replicas too eagerly means constantly paying this startup cost as traffic naturally fluctuates.

Respecting topology bounds

Replicas can never exceed the number of search nodes. Having more replicas than nodes provides no benefit (you can only serve as many pizzas as you have friends who are helping to serve slices).

When nodes are removed from your project, we reduce replicas immediately to match. No waiting for the cooldown, as you can't have unassigned replicas. The moment a friend leaves, we remove their pizza.

The bigger Serverless picture

Replicas for search load balancing works alongside other autoscaling systems:

• Search autoscaling adjusts the number of search nodes (how many friends are helping).

• Replicas for search load balancing distribute traffic by adjusting replica counts per index (how many pizzas of each kind we need).

• Data stream autosharding optimizes shard counts for writes (how to slice each pizza, covered in the previous post).

An important design principle: Replicas for load balancing don't directly trigger search autoscaling. Instead, by distributing search requests across more replicas, it enables increasing resource utilization across your search nodes. This higher utilization then triggers our existing autoscaling logic to add capacity if needed. Replicas for load balancing enables autoscaling to do its job, making sure your search nodes are actually being used, rather than having all traffic bottlenecked on a single replica while other nodes sit idle.

What this means for you

You don't need to predict which indices will be popular. You don't need to manually adjust replicas when traffic patterns change. You don't need to wake up at 3 a.m. because a surge overwhelmed your busiest index.

The system watches where queues are forming and orders more pizzas for those spots. Cold indices don't waste resources on unnecessary replicas. Hot indices get the capacity they need. Your budget goes where it matters.

Conclusion

In the autosharding post, we made sure your pizzas are sliced right. Now, with replicas for search load balancing, we make sure you have enough pizzas, in the right hands, when the hungry crowds arrive.

Try Elastic Cloud Serverless and let us handle the pizza logistics.

This paper isn't just an academic exercise. It's a foundational exploration of how the core of a search engine could be reimagined for a purely serverless world. We decouple storage from compute: Data lives in a cloud blob store with virtually infinite storage and scalability. That vision is the main driver behind our Elastic Cloud Serverless offering: seamless search over massive datasets, with the economics and operational simplicity of serverless.

The challenge: Rethinking stateful search for the cloud

For decades, search engines have been powerful, stateful systems. Deploying a production-grade cluster like Elasticsearch has meant:

• Provisioning servers and managing storage.

• Carefully tuning configurations for cost, performance, and reliability.

• Paying for idle capacity when workloads are spiky or unpredictable.

• Significant operational effort to scale up and down.

Modern cloud platforms have made some of this easier, but the fundamental tension remains:

Can we build a search engine that delivers the power and rich query capabilities of Elasticsearch with the economics and operational simplicity of a serverless architecture?

That question drove our research.

Our key contributions

The paper presents concrete innovations that make Elasticsearch Serverless possible:

• Object store as single source of truth: We offload index data, the transaction log (translog), and cluster state to a cloud object store. That eliminates replica shards for durability and makes the object store the sync point between indexing and search.

• "Thin" (stateless) shards: Shards recover and relocate quickly across nodes without copying large amounts of data. Disks are used only for caching, not for persistent storage.

• Batched compound commits (BCC): We wrap index commits in a custom format, cutting upload costs, while keeping the same read-after-write semantics as Elasticsearch.

• Batched translog uploads: Translog uploads are batched at the node level, cutting upload costs.

• Smart garbage collection: We track the usage of BCCs and translogs we’ve uploaded, and we delete them once they’re unused, to reduce storage footprint and retention costs.

• Autoscaling: We scale automatically with ingestion and search load so clients can call APIs without managing cluster size.

The bottom line: In our experiments, Elasticsearch Serverless achieves up to twice the indexing throughput of stateful Elasticsearch on comparable hardware and scales linearly with autoscaling to match ingestion load.

Visualizing the architecture

Figure 1 in the paper gives a clear side-by-side view: stateful Elasticsearch versus the new stateless architecture Elasticsearch Serverless.

Stateful Elasticsearch (top): Familiar data tiers: hot, warm, cold, frozen. Data lives on local disks; primaries and replicas are spread across nodes; colder tiers may use searchable snapshots on an object store.

Elasticsearch Serverless (bottom): Just two tiers: indexing and search. All durable data (Lucene commits, translogs, cluster state) lives in the object store. Indexing nodes write and upload; search nodes read from the object store and a shared cache, with no local persistence of index data.

The takeaway: a complete separation between the resources used for indexing and those used for querying.

A tale of two data flow paths

Figures 2 and 3 in the paper contrast how data flows in stateful versus Elasticsearch Serverless.

Stateful Elasticsearch (figure 2):

• Documents go to the primary shard's Lucene buffers and translog and then to replica shards.

• After refresh, the documents go to new searchable segments.

• After flush, they’re committed to disk.

• Thus, durability is given by the disk and the replicas.

Elasticsearch Serverless (figure 3):

• Documents go to Lucene and the translog on an indexing node.

• Before acknowledging the client, the translog is uploaded to the object store.

• After refresh, the documents go to new searchable segments and are committed to disk in the indexing nodes.

• After flush, they go into BCCs and are uploaded to the blob object store.

• Search nodes serve queries from the object store (and, for recent data not yet uploaded, directly from the indexing node).

• Thus, durability comes from the object store, not from disk or replicas.

Result: The indexing and search paths are fully decoupled.

Autoscaling

Section 7 of the paper describes the autoscaler. Because data lives in the object store, relocating shards doesn't mean copying full segment data; only metadata and, when needed, cache warming. So the cluster can scale up and down much faster than in stateful Elasticsearch.

How it works:

• The autoscaler is an external component that monitors metrics from Elasticsearch Serverless.

• Indexing tier: Scale-up is driven by memory usage and ingestion load (including queued work).

• Search tier: Scale-up is driven by memory, search load, and the user-configurable "search power" (how much of the dataset is cached locally).

• It polls every few seconds and adjusts each tier independently.

Outcome: automatic, workload-driven scaling so clients can focus on their applications instead of on capacity planning.

The experimental results

Section 8 of the paper presents our experimental evaluation.

Microbenchmarks show the impact of batching: fewer object store operations for both commits and translogs, with some trade-offs.

Autoscaling experiments: As we increase the number of indexing clients, throughput scales linearly while P50 and P99 latency stay stable. A real-world example shows bulk response times improving and stabilizing as the indexing tier scales up with demand.

Head-to-head comparison of stateful Elasticsearch versus Elasticsearch Serverless:

• Elasticsearch Serverless achieves roughly twice the indexing throughput of stateful Elasticsearch at the 50th percentile.

• The gain comes largely from using the object store for durability instead of replicating every operation to replica shards.

• Latency stays competitive.

The takeaway: The stateless design delivers both better peak performance and more efficient, automatic scaling.

Why this matters for the future of Elastic

The stateless architecture isn't just a technical achievement; it's the foundation for how we want search to work in the cloud.

• Pay-as-you-go: Customers can index and search over practically limitless data without provisioning clusters, tuning tiers, or managing replicas and snapshots.

• Automatic scaling: Each tier scales on its own automatically; no capacity planning required.

• Frequent, automated upgrades: Better security and time-to-value, without the operational cost of rolling upgrades over stateful data.

This work is a step toward making powerful search more accessible, cost-effective, and scalable for everyone.

Read the full paper, and join the conversation

We believe in the power of open research and collaboration to move technology forward. We encourage you to dive into the details. We provide a preprint of this paper for your information, which details in depth the architecture transformation.

Dive deeper: Explore related blog posts

While our paper offers a concise overview of the Elasticsearch Serverless architecture, the details and underlying innovations are explored more fully in a collection of in-depth blog posts written by our engineering team. These articles provide the background, nuance, and specific technical deep dives that make the stateless transformation possible.

We encourage you to delve into the following resources to gain a richer understanding of the components and concepts presented in the paper:

• Stateless — your new state of find with Elasticsearch (2022) and Serve more with Serverless (2023). Read the foundational posts introducing the concept of decoupling storage and compute.

• Stateless: Data safety in a stateless world (2024). Learn how data durability is achieved in the absence of local replicas.

• Autosharding of data streams in Elasticsearch Serverless (2024). Discover the logic behind automatic and dynamic data stream sharding.

• How we optimized refresh costs in Elasticsearch Serverless (2024). Understand the specific optimizations applied to reduce the cost of making data searchable.

• Introducing Serverless Thin Indexing Shards (2024). Explore the innovation of "thin" shards that enable rapid relocation and recovery.

• Search tier autoscaling in Elasticsearch Serverless (2024). Gain insight into the mechanisms driving the automatic scaling of search resources.

• Ingest autoscaling in Elasticsearch (2024). Learn how the ingestion tier scales automatically to meet fluctuating indexing load.

• Elastic Cloud Serverless pricing and packaging (2025). Learn how the pricing and packaging was initially structured for Elastic Cloud's Serverless offering.

• Elasticsearch vs. OpenSearch: Unraveling the performance gap (2023). Learn about the performance differences and key optimizations that distinguish Elasticsearch from OpenSearch, as observed in 2023.

Acknowledgments

We would like to thank all the co-authors of the paper: Iraklis Psaroudakis, Pooya Salehi, Jason Bryan, Francisco Fernández Castaño, Brendan Cully, Ankita Kumar, Henning Andersen, and Thomas Repantis. We would also like to thank the Elasticsearch Distributed Systems team for their contributions, and also the entire Elasticsearch engineering team.

1. Monitors indexing write load: Elasticsearch continuously tracks the write_load metric, representing the average number of write threads used for indexing. This metric informs sharding decisions.

2. Calculates optimal shard count: The system uses a formula that considers the current write_load, the minimum and maximum write threads available per node, to determine the ideal number of shards. This balances performance with overhead.

3. Triggers rollover based on write load: When the write_load necessitates a shard increase, a rollover operation is triggered. This creates a new index in the data stream with the calculated optimal number of shards. Regular rollover conditions (like shard size) also trigger rollovers, potentially leading to shard reduction if the write_load is lower.

4. Applies cooldown periods: To prevent rapid shard adjustments, cooldown periods are enforced. There's a 4.5-minute wait before increasing shards and a 3-day wait before reducing them.

5. Autoscaling integration: Autosharding works in conjunction with autoscaling. By dynamically adjusting shard counts, autosharding enables autoscaling to more effectively increase resources as needed, preventing the system from being constrained at low capacity during high indexing workloads. It also enables autoscaling to more effectively increase resources as needed, preventing the system from being constrained at low capacity during high indexing workloads.

Background

Traditionally, users change the sharding configuration of data streams in order to deal with various workloads and make the best use of the available resources. In Elastic Cloud Serverless we've introduced autosharding of data streams, enabling them to be managed and scaled automatically based on indexing load.

This post explores the mechanics of autosharding, its benefits, and its implications for users dealing with variable workloads. The autosharding philosophy is to increase the number of shards aggressively and reduce them very conservatively, such that an increase in shards is not followed prematurely by a reduction of shards due to a small period of reduced workload.

Autosharding of data streams in Serverless Elasticsearch

Imagine you have a large pizza that needs to be shared among your friends at a party. If you cut the pizza into only two slices for a group of six friends, each slice will need to serve multiple people. This will create a bottleneck, where one person hogs a whole slice while others wait, leading to a slow sharing process. Additionally, not everyone can enjoy the pizza at the same time; you can practically hear the sighs from the friends left waiting. If more friends show up unexpectedly, you’ll struggle to feed them with just two slices and find yourself scrambling to reshape those slices on the spot.

On the other hand, if you cut the pizza into 36 tiny slices for those same six friends, managing the sharing becomes tricky. Instead of enjoying the pizza, everyone spends more time figuring out how to grab their tiny portions. If the slices are too small, the pizza might even fall apart.

To ensure everyone enjoys the pizza efficiently, you’d aim to cut it into a number of slices that matches the number of friends. If you have six friends, cutting the pizza into 6 or 12 slices allows everyone to grab a slice without long waits. By finding the right balance in slicing your pizza, you’ll keep the party running smoothly and everyone happy.

You know it’s a good analogy when you immediately follow-up with the explanation; the pizza represents the data, the slices represent the index shards, and the friends are the Elasticsearch nodes in your cluster.

Traditionally, users of Elasticsearch had to anticipate their indexing throughput and manually configure the number of shards for each data stream. This approach relied heavily on predictive heuristics and required ongoing adjustments based on workload characteristics whilst also balancing data storage, search analytics, and application performance.

Businesses with seasonal traffic, like retail, often deal with spikes in data demands, while IoT applications can experience rapid load increases at specific times. Development and testing environments typically run only a few hours a week, making fixed shard configurations inefficient. New applications might struggle to estimate workload needs accurately, leading to potential over- or under-provisioning.

We've introduced autosharding of data streams in Elastic Cloud Serverless. Data streams in Serverless are managed and scaled automatically based on indexing load - automatically slicing your pizza as friends arrive to your party or finish eating.

The promise of autosharding

Autosharding addresses these challenges by automatically adjusting the number of shards in response to the current indexing load. This means that instead of users having to manually tweak configurations, Elasticsearch will dynamically manage shard counts for the data streams in your project based on real-time data traffic.

Elasticsearch keeps track of the indexing load for every index as part of a metric named write load, and exposes it for on-prem and ESS deployments as part of the index stats API under the indexing section.

The write_load represents the average number of write threads used while indexing documents.

For an index with one shard the maximum possible value of the write_load metric is the number of write threads available (e.g. all write threads are busy writing in the same shard).

For indices with multiple shards the maximum possible value for the write load is the number of write threads available in a node times the number of indexing nodes in the project. (e.g. all write threads on all the indexing nodes that host a shard for our index are busy writing in the shards belonging to our index, exclusively)

To get a sense of the values allowed for write_load let’s look at index logs with one shard running on one Elasticsearch machine with 2 allocated processors. The write thread pool will be sized to 2 threads. This means that if this Elasticsearch node is exclusively and constantly writing to the same index logs, the write_load we’ll report for index logs will be 2.0 (i.e. 2 write threads fully utilized for writing into index logs).

If logs has 2 primary shards and we’re now running on two Elasticsearch nodes, each with 2 allocated processors we’ll be able to get a maximum reported write_load of 4.0 if all write threads on both Elasticsearch nodes are exclusively writing into the logs index.

Serverless autoscaling

We just looked at how the write load capacity doubled when we increased the number of shards and Elasticsearch nodes. Elastic Cloud Serverless takes care automatically of both these operations using data stream autosharding and ingest autoscaling. Autoscaling refers to the process of dynamically adjusting resources - like memory, CPU, and disk - based on current demands.

In our serverless architecture, we start with a small 2GB memory server and use a step-function scaling approach to increase capacity efficiently. We scale up memory incrementally and then scale out by adding servers. This cycle continues, increasing memory per server incrementally up to 64GB while managing the number of servers.

Linking autoscaling and autosharding

The connection between autoscaling and autosharding is essential for optimizing performance. When calculating the optimal number of shards for a data stream, we consider the minimum and maximum number of available write threads per node in our scaling setup.

• For small projects, the system will move from 1 to 2 shards when the data stream uses more than half the capacity of a node (i.e., more than one indexing thread).

• For medium-sized projects, as the system scales across multiple nodes, it will not exceed 3 shards to avoid excessive overhead.

• Once we reach the largest node sizes, further sharding is enabled to accommodate larger workloads.

Autosharding also enables autoscaling to increase resources as needed, preventing the system from staying at low capacity during high indexing workloads, by enabling projects to reach higher ingestion load values.

Auto sharding formula

To determine the number of shards needed, we use the following formula:

This equation balances the need for increasing shards based on write_load while capping the number of shards to prevent oversharding. The division by 2 reflects the strategy of increasing shards only after exceeding half the capacity of a node. The min/max write threads represent the minimum and maximum number of write threads available in the autoscaling step function (i.e. the number of write threads available on the smallest 2GB step and the number of write threads available on the largest server)

Let’s visualize the output of the formula:

On the Y axis we have the number of shards. And on the X axis we have the write load. We start with 1 shard and we get to 3 shards when the write load is just over 3.0. We remain with 3 shards for quite some time until the write load is about 48.0.

This covers us for the time we scale up through the nodes but haven’t really got to 2 or more or the largest servers, at which point we unlock auto sharding to more than 3 shards, as many as needed to ingest data.

While adding shards can improve indexing performance, excessive sharding in an Elasticsearch cluster can have negative repercussions - imagine that pizza with 56 slices being shared by only 7 friends. Each shard carries overhead costs, including maintenance and resource allocation. Our algorithm accounts for and avoids the peril of excessive sharding until we get to the largest workloads where adding more than 3 shards makes a material difference to indexing performance and throughput.

Implementing autosharding with rollovers

The implementation of autosharding relies on the concept of rollover. A rollover operation creates a new index within the data stream, promoting it to the write index while designating the previous index as a regular backing index, which no longer accepts writes. This transition can occur based on specific conditions, such as exceeding a shard size of 50GB. We take care of configuring the optimal rollover conditions for data streams in Serverless.

In Serverless alongside the usual rollover conditions that relate to maintaining healthy indices and shards we introduce a new condition that evaluates whether the current write load necessitates an increase in shard count. If this condition is met, a rollover will be triggered and the new resulting data stream write index will be configured with the optimal number of shards.

For downscaling, the system will monitor the workload and will not trigger a rollover solely for reducing shards. Instead, it will wait until a regular rollover condition, like the primary shard size, triggers the rollover. The resulting write index will be configured with a lower number of shards.

Cooldown periods for shard adjustments

To ensure stability during shard adjustments, we implement cooldown periods:

• Increase shards cooldown: A minimum wait time of 4.5 minutes is enforced before increasing the number of shards since the last adjustment. The 4.5 minutes cooldown might seem peculiar but the interval has been chosen to make sure we can increase the number of shards every time data stream lifecycle checks if data streams should rollover (currently, every 5 minutes) but not more often than 5 minutes, covering for internal Elasticsearch cluster reconfiguration.

• Decrease shards cooldown: We maintain a 3-day minimum wait time before reducing shards to ensure that the decision is based on sustained workload patterns rather than temporary fluctuations.

Conclusion

The data streams autosharding feature in Serverless Elasticsearch represents significant progress in managing data streams effectively. By automatically adjusting shard counts based on real-time indexing loads, this feature simplifies operations and enhances scalability.

With the added benefits of autoscaling, users can expect a more efficient and responsive experience, whether they are handling small projects or large-scale applications. As data workloads continue to evolve, the adaptability provided by auto sharding ensures that Elasticsearch remains a robust solution for managing diverse indexing needs.

Try out our Serverless Elasticsearch offering to take advantage of data streams auto sharding and observe the indexing throughput scaling seamlessly as your data ingestion load increases.

Your pizzas will be optimally sliced as more friends arrive at your party, keen to try those sourdough craft pizzas you prepared for them.

Before we dive into the changes we made, it’s essential to first understand the interplay between Elasticsearch and Lucene.

Elasticsearch uses Lucene, a high-performance, open-source library written in Java, for full text indexing and search. When a document is indexed into Elasticsearch, it isn't immediately written to disk by Lucene. Instead, Lucene updates its internal in-memory data structures. Once enough data accumulates or a refresh is triggered, these documents are then written to disk, creating a new set of immutable files known as segments in Lucene terminology. The indexed documents are not available for search until the segments are written to disk. That’s the reason why refresh is such an important concept in Elasticsearch. You might be wondering how durability is ensured when documents are kept in memory until a refresh is triggered. This is achieved through the Translog, which stores durably every operation to guarantee data persistence and recovery in case of failure.

Now that we know what Lucene segments are and why refreshes are needed in Elasticsearch, we can explore how refresh behavior differs between stateful Elasticsearch and serverless Elasticsearch.

Refreshes in stateful Elasticsearch

In Elasticsearch, indices are divided into multiple shards, each consisting of a primary shard and potentially multiple replica shards. In stateful Elasticsearch, when a document is indexed, it is first routed to the primary shard, where Lucene processes and indexes it. After indexing on the primary shard, the document is then routed to the replica shards, where it is indexed by these copies.

As mentioned earlier, a refresh is needed to make these indexed documents searchable. In stateful Elasticsearch, a refresh writes the Lucene in-memory data structures to disk without performing an fsync. Refreshes are scheduled periodically, with each node executing them at different times. This process will create distinct Lucene segment files on each node, all containing the same set of documents.

Refreshes in serverless Elasticsearch

In contrast, serverless Elasticsearch employs a segment-based replication model. In this approach, one node per shard handles document indexing and generates Lucene segments. These segments are uploaded to the blob store once a refresh is initiated. Subsequently, search nodes are informed about these new Lucene segments, which they can read directly from the blob store.

The illustration above demonstrates how a refresh works in serverless Elasticsearch:

1. The indexing node, where all the documents were indexed, receives the refresh request and Lucene writes the in-memory data structures to disk, similar to how a stateful refresh operates.

2. The segment files are uploaded to the blob store as a single file (known as a stateless compound commit). In the illustration, S4 is uploaded.

3. Once the segment files are uploaded to the blob store, the indexing node sends a message to each search node, notifying them of the new segment files so they can perform searches on the newly indexed documents.

4. The search nodes fetch the necessary data from the blob store when executing searches.

This model offers the advantage of lightweight nodes, as data is stored in the blob store. This makes scaling or reallocating workloads between nodes more cost-effective compared to stateful Elasticsearch, where data must be transferred to the new node containing the new shard.

One aspect worth considering is the additional object store request costs associated with each refresh in serverless Elasticsearch. Every refresh operation created a new object in the object store, resulting in an object store PUT request that incurs associated costs. This led to a linear relationship between the number of indices and the number of object store PUT requests. With enough refreshes, object store costs could surpass the cost of the hardware itself. To address this, we initially implemented refresh throttling measures to manage costs effectively and mitigate potential issues over time. This blog post describes the next step in that effort, which allowed us to refresh faster and at a manageable cost.

Refresh cost optimizations in serverless Elasticsearch

As previously mentioned, the serverless Elasticsearch architecture provides numerous benefits. However, to manage refresh costs effectively, we made decisions that occasionally impacted user experience. One such decision was enforcing a default refresh interval of 15 seconds, meaning that in some cases, newly indexed data won't become searchable until 15 seconds have passed. Despite our efforts, scenarios arose where object store expenses became prohibitive, prompting us to reassess our approach. In this section, we will delve into how we successfully decoupled refresh operations from object store calls to address these challenges without compromising user experience.

After evaluating various solutions—from temporary storage of segments in distributed file systems like NFS to direct pushing of segments into search nodes—we settled on an approach relying on serving segment data from indexing nodes directly to search nodes.

Rather than letting refresh immediately upload new Lucene segments to the blob store, index nodes now accumulate segments from refreshes and upload them as a single blob later. This enables index nodes to serve reads from search nodes in a manner akin to a blob store, delaying segment uploads until sufficient data accumulates or a predetermined time interval elapses.

This strategy grants us complete control over the size of the blobs uploaded to the blob store, enabling us to determine when request costs become negligible in comparison to hardware costs.

Batched compound commits

We aimed to implement this enhancement incrementally and ensure backward compatibility with existing data stored in the blob store. Therefore, we opted to maintain the same file format for storing Lucene segments in the blob store. For context, Lucene segments comprise multiple files, each serving a distinct role. To streamline the upload process and minimize PUT requests, we introduced compound commits: single blobs containing all segment files consecutively, accompanied by a metadata header, including a directory of the files in the compound commit.

When retrieving a compound commit from the blob store, such as during shard relocation, our primary focus is typically on the compound commit header. This header is crucial as it contains the essential data needed to promptly populate internal data structures. With this in mind, we realized we could maintain the existing file format but streamline it so that each blob would sequentially append one compound commit after another. We denominated this new file format, batched compound commit.

Since each compound commit's size is stored in its header, retrieving the headers of all compound commits within a batched compound commit is straightforward; we can sequentially read each header by simply seeking the next entry. When handling blobs in the old format, they are treated as singleton batched compound commits. Another critical aspect of our file format is maintaining fixed offsets for each Lucene segment file once it's appended into a batched compound commit. This ensures consistency whether the file is served from the index node or the blob store. It also prevents the need to evict cached entries on search nodes when the batched compound commit is eventually uploaded to the blob store.

New refresh lifecycle

Index nodes will now accumulate Lucene segments from refreshes until enough data is gathered to upload them as a single blob. Let us explore how index and search nodes coordinate to determine where to access this data from.

As shown in the illustration above, the following steps occur during the optimized refresh process in serverless Elasticsearch:

1. The index node receives a refresh request, writes a new set of Lucene segments to its local disk, and adds these segments to the pending batched compound commit for eventual upload.

2. The index node notifies the search node about these new segments, providing details about the involved segments and their locations (blob store or index node).

3. When a search node needs a segment to fulfill a query, it decides whether to get it from the blob store or the index node and caches the data locally.

The image above illustrates the process of uploading data to the blob store in serverless Elasticsearch once enough segments have accumulated in the indexing nodes or after a specified amount of time has elapsed.

1. A refresh adds a new segment to the batched compound commit and the accumulated data reaches 16 MB, or a certain amount of time has passed since the last refresh, from this point onwards new segments are accumulated into a new batched compound commit.

2. The indexing node begins uploading the accumulated segments as a single blob to the object store.

3. The indexing node notifies the search node replicas of the latest segment uploaded to the object store, instructing them to fetch data from these segments from the blob store going forward.

4. If a search requires data that isn't cached locally, it will retrieve the necessary information from the blob store, while any previously fetched data from the indexing node remains valid even after the upload.

Considerations and tradeoffs

The approach chosen blurs the clear separation between storage and compute, requiring index nodes to handle storage requests until Lucene segments are eventually uploaded to the blob store. However, the overhead from these storage requests is minimal and we have not observed impact on indexing throughput.

We'll note that we keep translog entries until corresponding data has been uploaded to the blob store, hence the approach maintains existing data safety guarantees. Recovery times after a crash may be slightly longer, but we consider this an acceptable trade-off.

Conclusions

This blog post has explored our transition towards a more cloud-native approach, emphasizing its many benefits alongside the critical cost consideration. We traced our evolution from a model where each new Lucene segment generated a distinct object in the object store. This led to cost and user experience challenges in specific serverless workloads compared to stateful Elasticsearch. Batching object store uploads enabled us to minimize the number of object store requests and enhance the cost efficiency of our serverless offering.

Acknowledgments

We would like to acknowledge the contributions of Iraklis Psaroudakis, Tanguy Leroux, and Yang Wang. Their efforts were instrumental in the success of this project.

We rely on this mechanism to free users of our Elastic Cloud Serverless offering from having to make sizing decisions for the indexing tier. Ingest autoscaling requires continuously estimating the resources required to handle the incoming workload, and provisioning and de-provisioning these resources in a timely manner.

In this blog post we explore ingest autoscaling in Elasticsearch, covering the following:

• How ingest autoscaling works in Elasticsearch

• Which metrics we use to quantify the indexing workload the cluster experiences in order to estimate resources required to handle that workload

• How these metrics drive the autoscaling decisions.

Ingest autoscaling overview

Ingest autoscaling in Elasticsearch is driven by a set of metrics that is exposed by Elasticsearch itself. These metrics reflect the ingestion load and the memory requirement of the indexing tier. Elasticsearch provides an autoscaling metrics API that serves these metrics which allows an external component to monitor these metrics and make decisions whether the cluster size needs to change (see Figure 1).

In the Elastic Cloud Serverless service, there is an autoscaler component which is a Kubernetes Controller. The autoscaler polls the Elasticsearch autoscaling metrics API periodically and calculates the desired cluster size based on these metrics. If the desired cluster size is different from the current one, the autoscaler changes the cluster size to consolidate the available resources in the cluster towards the desired resources. This change is both in terms of the number of Elasticsearch nodes in the cluster and the CPU, memory and disk available to each node.

Figure 1: ingestion autoscaling overview

An important consideration for ingest autoscaling is that when the cluster receives a spike in the indexing load the autoscaling process can take some time until it effectively adapts the cluster size. While we try to keep this reaction time as low as possible, it cannot be instantaneous. Therefore, while the cluster is scaling up, the Elasticsearch cluster should be able to temporarily push back on the load it receives if the increased load is otherwise going to cause cluster instability issues. The increase in the indexing load can manifest itself in the cluster requiring more resources, i.e., CPU, memory or disk. Elasticsearch has protection mechanisms that allows nodes to push back on the indexing load if any of these resources becomes a bottleneck.

To handle indexing requests Elasticsearch uses dedicated thread pools sized based on the number of cores available to the node. If the increased indexing load results in CPU or other resources becoming a bottleneck, incoming indexing requests are queued. The maximum size of this queue is limited and any request arriving at the node when the queue is full will be rejected with a 429 HTTP code.

Elasticsearch also keeps track of the required memory to address ongoing indexing requests and rejects incoming requests (with a 429) if the indexing buffer grows beyond 10% of the available heap memory. This limits the memory used for indexing and ensures the node will not go out of memory.

The Elastic Cloud Serverless offering relies on the object store as the main storage for indexed data. The local disk on the nodes are used temporarily to hold indexed data. Periodically, Elasticsearch uploads the indexed data to the object store which allows freeing up the local disk space as we rely on the object store for durability of the indexed document. Nonetheless, under high indexing load, it is possible for the node to run out of disk space before the periodic upload task gets a chance to run and free up the local disk space. To handle these cases, Elasticsearch monitors the available local disk space and if necessary throttles the indexing activity while it attempts to free up space by enforcing an upload to the object store rather than waiting for the periodic upload to take place. Note that this throttling in turn results in queueing of the incoming indexing requests.

These protection mechanisms allow an Elasticsearch cluster to temporarily reject requests and provide the client with a response that indicates that the cluster is overloaded while the cluster tries to scale up. This push-back signal from Elasticsearch provides the client with a chance to react by reducing the load if possible or retrying the request which should eventually succeed if retried when the cluster is scaled up.

Metrics

The two metrics that are used for ingest autoscaling in Elasticsearch are ingestion load and memory.

Ingestion load

Ingestion load represents the number of threads that is needed to cope with the current indexing load. The autoscaling metrics API exposes a list of ingestion load values, one for each indexing node. Note that as the write thread pools (which handle indexing requests) are sized based on the number of CPU cores on the node, this essentially determines the total number of cores that is needed in the cluster to handle the indexing workload.

The ingestion load on each indexing node consists of two components:

• Thread pool utilization: the average number of threads in the write thread pool processing indexing requests during that sampling period.

• Queued ingestion load: the estimated number of threads needed to handle queued write requests.

The ingestion load of each indexing node is calculated as the sum of these two values for all the three write thread pools. The total ingestion load of the Elasticsearch cluster is the sum of the ingestion load of the individual nodes.

Figure 2: ingestion load components

The thread pool utilization is an exponentially weighted moving average (EWMA) of the number of busy threads in the thread pool, sampled every second. The EWMA of the sampled thread pool utilization values is configured such that the sampled values of the past 10 seconds have the most effect on the thread pool utilization component of the ingestion load and samples older than 60 seconds have very negligible impact.

To estimate the resources required to handle the queued indexing requests in the thread pool, we need to have an estimate for how long each queued task can take to execute. To achieve this, each thread pool also provides an EWMA of the request execution time. The request execution time for an indexing request is the (wall-clock) time taken for the request to finish once it is out of the queue and a worker thread starts executing it. As some queueing is acceptable and should be manageable by the thread pool, we try to estimate the resources needed to handle the excess queueing. We consider up to 30s worth of tasks in the queue manageable by the existing number of workers and account for an extra thread proportional to this value. For example, if the average task execution time is 200ms, we estimate that each thread is able to handle 150 indexing requests within 30s, and therefore account for one extra thread for each 150 queued items.

Note that since the indexing nodes rely on pushing indexed data into the object store periodically, we do not need to scale the indexing tier based on the total size of the indexed data. However, the disk IO requirements of the indexing workload needs to be considered for the autoscaling decisions. The ingestion load represents both CPU requirements of the indexing nodes as well as disk IO since both CPU and IO work is done by the write thread pool workers and we rely on the wall clock time to estimate the required time to handle the queued requests.

Each indexing node calculates its ingestion load and publishes this value to the master node periodically. The master node serves the per node ingestion load values via the autoscaling metrics API to the autoscaler.

Memory

The memory metrics exposed by the autoscaling metrics API are node memory and tier memory. The node memory represents the minimum memory requirement for each indexing node in the cluster. The tier memory metric represents the minimum total memory that should be available in the indexing tier. Note that these values only indicate the minimum to ensure that each node is able to handle the basic indexing workload and hold the cluster and indices metadata, while ensuring that the tier includes enough nodes to accommodate all index shards.

Node memory must have a minimum of 500MB to be able to handle indexing workloads, as well as a fixed amount of memory per each index. This ensures all nodes can hold metadata for the cluster, which includes metadata for every index. Tier memory is determined by accounting for the memory overhead of the field mappings of the indices and the amount of memory needed for each open shard allocated on a node in the cluster. Currently, the per-shard memory requirement uses a fixed estimate of 6MB. We plan to refine this value.

The estimate for the memory requirements for the mappings of each index is calculated by one of the data nodes that hosts a shard of the index. The calculated estimates are sent to the master node. Whenever there is a mapping change this estimate is updated and published to the master node again. The master node serves the node and total memory metrics based on these information via the autoscaling metrics API to the autoscaler.

Scaling the cluster

The autoscaler is responsible for monitoring the Elasticsearch cluster via the exposed metrics, calculating the desirable cluster size to adapt to the indexing workload, and updating the deployment accordingly. This is done by calculating the total required CPU and memory resources based on the ingestion load and memory metrics. The sum of all the ingestion load per node values determines the total number of CPU cores needed for the indexing tier.

The calculated CPU requirement and the provided minimum node and tier memory resources are mapped to a predetermined set of cluster sizes. Each cluster size determines the number of nodes and the CPU, memory and disk size of each node. All nodes within a certain cluster size have the same hardware specification. There is a fixed ratio between CPU, memory and disk, thus always scaling all 3 resources linearly. The existing cluster sizes for the indexing tier are based on node sizes starting from 4GB/2vCPU/100GB disk to 64GB/32vCPU/1600GB disk. Once the Elasticsearch cluster scales up to the largest node size (64GB memory), any further scale-up adds new 64GB nodes, allowing a cluster to scale up to 32 nodes of 64GB. Note that this is not a hard upper bound on the number of Elasticsearch nodes in the cluster and can be increased if necessary.

Every 5 seconds the autoscaler polls metrics from the master node, calculates the desirable cluster size and if it is different from the current cluster size, it updates the Elasticsearch Kubernetes Deployment accordingly. Note that the actual reconciliation of the deployment towards the desired cluster size and adding and removing the Elasticsearch nodes to achieve this is done by Kubernetes. In order to avoid very short-lived changes to the cluster size, we account for a 10% headroom when calculating the desired cluster size during a scale down and a scale down takes effect only if all desired cluster size calculations within the past 15 minute have indicated a scale-down.

Currently, the time that it takes for an increase in the metrics to lead to the first Elasticsearch node being added to the cluster and ready to process indexing load is under 1 minute.

Conclusion

In this blog post, we explained how ingest autoscaling works in Elasticsearch, the different components involved, and the metrics used to quantify the resources needed to handle the indexing workload. We believe that such an autoscaling mechanism is crucial to reduce the operational overhead of an Elasticsearch cluster for the users by automatically increasing the available resources in the cluster when necessary. Furthermore, it leads to cost reduction by scaling down the cluster when the available resources in the cluster are not required anymore.