GPU-accelerated vector indexing in Elasticsearch with NVIDIA cuVS: 138M vectors in under 10 minutes

Try out vector search for yourself using this self-paced hands-on learning for Search AI. You can start a free cloud trial or try Elastic on your local machine now.

Modern enterprise applications are ingesting terabyte- to petabyte-scale unstructured data to power semantic search, large language model–based (LLM-based) retrieval augmented generation (RAG), and recommender systems. At this scale, vector indexing on CPUs can take days or even weeks, slowing experimentation and making large index updates operationally expensive. This is especially challenging for workloads that require regular index rebuilds, such as those driven by frequent data updates or frequent embedding model updates. For example, several ecommerce teams refresh their product catalog nightly and train their own embedding models, each requiring an index rebuild.

CPU-based indexing can also degrade search performance when indexing and search happen simultaneously, especially in online systems, like ecommerce platforms and ad-serving pipelines. This happens because index builds consume significant CPU resources, leaving fewer cycles available for queries and driving up search latency and latency variance. Even after indexing completes, latency can remain high due to index fragmentation. Customers often run a forced merge to consolidate segments, but it can be slow on CPUs, further delaying recovery to the expected search latency by hours to days.

Elasticsearch has introduced GPU-accelerated vector indexing powered by NVIDIA cuVS. In this blog, we show how Elasticsearch achieves up to 7x faster vector indexing throughput by offloading hierarchical navigable small world (HNSW) index construction to GPUs, indexing 138 million vectors in under 10 minutes using eight NVIDIA RTX PRO 6000 GPUs.

Indexing over 100M Vectors in under 10 minutes

Figure 1 shows vector indexing throughput on 138 million 1,024-dimensional vectors from the MS MARCO dataset, representing roughly 4 TB of multimodal documents (Figure 1). The benchmark was run with Rally, Elasticsearch’s macro-benchmarking framework. The test ran on a server with 8 NVIDIA RTX Pro 6000 GPUs with a 2 socket AMD EPYC 9555. GPUs were turned on for the GPU test and turned off for the CPU test. The result was that Elasticsearch achieved 7x higher indexing throughput, lowering time to index 138M vectors from 1 hour on CPUs to under 10 minutes on GPUs.

Does GPU vector indexing change recall?

Index acceleration isn’t helpful if indexes built on GPU have lower search quality compared to CPUs. The throughput-versus-recall curves for GPU-built and CPU-built indexes overlap, showing that GPU indexing delivers the same recall profile as Elasticsearch’s CPU-based indexing path, with minimal accuracy or search-time performance penalty (Figure 2).

Search latency under concurrent indexing load

Offloading indexing to the GPU improved search latency on the CPU by 6x when running indexing and search simultaneously (Figure 3) because indexing on GPUs frees CPU resources for faster search latency. This allows a single Elasticsearch cluster to support real-time ingestion and low-latency retrieval at the same time, without forcing a tradeoff between freshness and search performance.

Force merge time for HNSW indexes on GPU versus CPU

Additionally, GPU-accelerated indexing in Elasticsearch reduced forced merge time from roughly four hours to five minutes, helping restore low-latency search in real-time (Figure 4). Note: Force merge was used to merge each shard into four segments.

Here are the full results for GPU versus CPU vector indexing in Elasticsearch:

What GPU vector indexing means for Elasticsearch users

Elasticsearch’s integration of NVIDIA cuVS brings GPU-accelerated vector indexing to production AI search with over 84% reduction in search latency to deliver faster ingest with a 640% increase in indexing throughput with a lower CPU overhead and limited infrastructure tradeoffs. This unlocks a new class of applications, such as faster RAG pipelines, semantic search, multimodal retrieval, and fraud detection at scale.

Get started with GPU vector indexing in Elasticsearch

Install Elasticsearch with GPU-accelerated vector indexing by following these instructions. Replicate the above benchmarks by running rally msmarco track with parameters in the GitHub issue. Learn more about GPU-accelerated vector indexing, search, and preprocessing by visiting NVIDIA cuVS. 

Acknowledgments

The authors would like to thank Chris Hegarty, Gilad Gal, and Mayya Sharipova from Elastic, as well as Nathan Stephens from NVIDIA, for their contributions to this article.