Elasticsearch supports a diverse range of use cases, including observability metrics and logs and complex geospatial analysis. However, as vector search becomes a core component of modern architectures, the need for specialized optimization has grown. Achieving peak performance for vector-heavy workloads often requires navigating a complex space of configuration knobs. To reduce this operational toil, we wanted to provide opinionated, high-performance defaults through a single setting that simplifies performance tuning for production environments. The vectordb_document index mode is designed specifically for optimal vector search workloads.

How to set up vectordb_document mode

Setup is a single setting. When you’re creating an index, define the following in the index settings:

vectordb_document mode is available on all Elasticsearch subscription tiers, including Basic.

Most indices used for vector search also support other operations, such as aggregations or geo-search. They also support hybrid search. Because vector search is typically the most computationally demanding part of these mixed workloads, we recommend using the vectordb_document index mode to prioritize performance for your most intensive operations. An index in vectordb_document mode remains highly capable, supporting almost all operations available in the default standard mode, while optimizing specifically for the resource-heavy demands of vector search.

The “_document” suffix represents our roadmap. We're also developing a “vectordb_columnar” mode as another way to optimize vector search, suited to different data and access patterns.

Elasticsearch vector search benchmarks across four datasets

To validate these defaults, we performed extensive benchmarking across various datasets and two index types: bbq_hnsw and bbq_disk. We ran all benchmarks on a single-node Elasticsearch instance on AWS using a c8gd.2xlarge instance (Graviton 4, ARM64, local NVMe SSD) with a pod limited to 8GB RAM (2GB heap) and 4 CPUs, using a single shard.

Datasets:

Dataset

Vectors

Dims

Use case

laion-img-emb-512-20M-cosine

20 million

512

Pure vector search (low dim)

msmarco-v2-10M-jina-v5-1024

10 million

1024

Pure vector search (med dim)

dbpedia-openai-1M-3072-angular

1 million

3072

Pure vector search (high dim)

arxiv-for-fanns-large

2.7 million

4096

Filtered search

bbq_hnsw: HNSW index performance with vectordb_document

Query throughput and recall

On all four datasets, vectordb_document produced a better QPS–recall curve, and the shape of the advantage is itself informative. On dbpedia-openai-1M, msmarco-v2-10M, and arxiv-for-fanns-large, the curves start close together at low recall (essentially identical on arXiv) and separate as recall rises, reaching roughly 1.4×, 2.2×, and 2× at the high-recall end. On laion-img-emb-512-20M, the curves are apart from the start and settle at about 2× from recall 0.80 upward.

The gap widens with recall because higher recall is bought with oversampling, and oversampling is exactly where vectordb_document saves. The two effects compound. bfloat16 storage halves the bytes read per rescored candidate. More importantly for hierarchical navigable small world (HNSW), the oversample factor is applied to the graph search itself;  each segment is searched for k × oversample candidates, so cost scales with oversample times segment count. Unthrottled parallel merging leaves fewer, larger graphs (14–21 segments versus 22–29), and the graph and quantized vector files are preloaded into the filesystem cache, so vectordb_document pays far less for each increment of oversample.

Comparing identical search settings, rather than equal recall, makes the effect explicit. With rescoring switched off, the two modes are within 1%–28% of each other; at oversample 5, vectordb_document is 2–4× faster; at oversample 10, up to 10×. Those high-oversample settings sit off the QPS–recall frontier, which is why the curves above top out nearer 2×, but they isolate where the saving comes from.

Oversample

DBpedia

LAION

MS MARCO

arXiv

off

1.23×

1.28×

1.11×

1.01×

5

4.07×

2.66×

2.66×

2.05×

10

10.5×

2.33×

2.46×

4.03×

Indexing speed and merge behavior 

In our benchmarks, vectordb_document increased upload time by roughly 16%. This is expected: merges now run unthrottled and parallelized across threads, so they compete with indexing for CPU while documents are still being ingested. Measured indexing time rose 9%–26% across the four datasets. HNSW graph construction is CPU-bound, so that contention is felt directly.

The same changes make the post-upload phase much cheaper. Disabling auto-throttling removes merge rate limiting entirely (in default mode, DBpedia spent 57% of its merge time throttled), and bfloat16 halves the raw vector data, cutting total bytes merged by 42%–49%. Post-ingest merging finished 75%–86% sooner on three of the four datasets, which brings total time to a search-ready index to only about 6% above the baseline. DBpedia was the exception in the other direction: Its merge tail dominated, so total time actually fell 25%.

bbq_disk: disk-based vector search performance with vectordb_document

bbq_disk query throughput and recall

For bbq_disk indices, vectordb_document mode's impact on search throughput varied by dataset. On the lower-dimensional datasets, QPS was essentially unchanged at equal recall: laion-img-emb-512-20M-cosine (512 dims) and msmarco-v2-10M-jina-v5-1024 (1024 dims) track each other closely across the recall range, with vectordb_document ahead in the lower recall end and a few percent behind at the high-recall end. On the higher-dimensional datasets, we saw a consistent gain of roughly 20% at equal recall: dbpedia-openai-1M-3072-angular (3072 dims) and arxiv-for-fanns-large (4096 dims).

Our interpretation is that this is mainly a rescoring effect. vectordb_document stores vectors as bfloat16, so each rescored candidate reads 2× dims bytes instead of 4× dims. The sweep supports this directly: The advantage grows with the query-time oversample factor, which is exactly what sets how many candidates get rescored. On DBpedia, the QPS ratio rises from 1.16× at oversample 3 to 4.3× at oversample 8, and on arXiv from 1.20× to 1.56×, while on laion and MS MARCO, it stays flat or drifts just below 1. Rescoring is simply a much larger share of the query on the smaller, higher-dimensional datasets; on the 10 million and 20 million ones, scanning 1-bit posting lists dominates.

Indexing speed and merge behavior 

Across the four datasets, vectordb_document cut total time to a fully merged, search-ready index by about 20%. Every dataset improved, from 6% on msmarco-v2-10M to 50% on dbpedia-openai-1M, where the post-upload merge phase alone fell from 322 seconds to 54 seconds. Upload time on its own is less clear-cut: On the day charted below, it finished 2%–15% sooner, depending on dataset, but on the earlier run set, it was marginally slower, so we read upload as unchanged to modestly faster and treat time-to-searchable as the real result.

The gain is almost entirely in merging. In default mode, Elasticsearch rate-limits how fast merges may write, and that limiter was binding hard: 75% of all merge time on DBpedia and 73% on laion was spent paused by it. vectordb_document disables the limiter, so paused time is zero and merge time drops 63% on DBpedia and 42% on laion. bfloat16 helps for the same reason: The limiter meters bytes written, so halving the raw vector data means less to write under the cap. The gradient across datasets follows the throttling rather than the byte count: arXiv, at 21% of merge time throttled, saw 30% less merge time, while MS MARCO, never throttled, got 55% fewer bytes but only 11% less merge time.

The two index types respond differently on ingest because merging costs something different in each. Merging bbq_hnsw segments means rebuilding HNSW graphs: CPU-bound work that competes directly with the equally CPU-bound graph construction on incoming documents, and on a 4-CPU pod that contention surfaces as a slower upload. bbq_disk merges are dominated by writing bytes rather than by CPU, so lifting the rate limiter spends I/O bandwidth that the local NVMe has to spare, and bfloat16 means that there are fewer bytes to write in the first place. Both index types reach a fully merged index far sooner; the difference is only whether the upload phase pays for it.

What vectordb_document sets under the hood

element_type (dense_vector)

• Value: bfloat16.

• Impact: Stores each dimension of the raw vector as bfloat16 instead of the default float32, halving the storage of raw vectors with negligible impact on recall.

• Benefit: Lower disk footprint (reduced total cost of ownership [TCO]), faster fetching of vectors for rescoring.

Dynamic float array mapping

• Value: Float arrays with 32 or more values are dynamically mapped as dense_vector. (In the default mode, this threshold is 128.)

• Impact: No need to explicitly declare the field as a dense vector field; the system identifies it automatically.

• Benefit: Configuration simplicity.

exclude_source_vectors

• Value: true.

• Impact: Vectors are stored once in the vector index and not duplicated in _source. They’re omitted from _source in responses, but they can still be retrieved on request.

Benefit: Lower disk footprint; faster queries, since large vectors are no longer shipped with every _source fetch.

index.store.preload

• Value: ["vex", "veq", "veb", "cenivf"].

• Impact: Preloads search-time vector data structures into the filesystem cache whenever new segments are opened.

• Benefit: Reduced query latency.

index.merge.intra_merge_parallelism_enabled

• Value: true.

• Impact: Use parallel threads for segment merging to achieve optimal segment sizes.

• Benefit: Faster convergence to fewer, larger segments leads to better recall, lower query latency.

index.merge.scheduler.auto_throttle

• Value: false.

• Impact: Allows merges to proceed at full speed.

• Benefit: Merges reach optimal segment sizes sooner, resulting in better recall, lower query latency.

While these defaults are optimized for the majority of use cases, most can be overridden individually to suit unique hardware constraints or extreme performance requirements, with one exception: exclude_source_vectors: true is locked in and cannot be changed on a vectordb_document index.

Summary

The vectordb_document index mode is a production-ready foundation for vector search: By adopting high-performance defaults, teams can focus on building features rather than on hand-tuning merge or storage settings or on adjusting preload settings. 

Across four datasets, the picture is consistently favorable, though the balance differs by index type. With bbq_hnsw, QPS recall improved on every dataset, from roughly on par at low recall to as much as double at the high-recall end, in exchange for a modest ingest cost of about 16% longer upload and 6% longer time to a search-ready index. With bbq_disk, the trade runs the other way: Total time to a search-ready index dropped by about 20%, while search improved by about 20% at equal recall on the higher-dimensional datasets (3072 and 4096 dims) and was essentially unchanged on the lower-dimensional ones. The split comes down to what merging costs in each: Rebuilding HNSW graphs is CPU work that competes with indexing; whereas bbq_disk merges are write-bound and simply run faster once they’re unthrottled.

In both cases, the new defaults move the system in the direction that most users want, with no per-index tuning required. And, for the settings that genuinely depend on the data itself, such as the degree of quantization, auto-calibration now derives them for you. (See How Elasticsearch auto-tunes vector quantization to hit your recall target.)

Try it out by creating an index with the vectordb_document index mode in Stateful Elasticsearch 9.5 or in Serverless.

Recap

First, let’s bring you up to speed. Elasticsearch has established itself as a powerful vector database, offering a rich set of features and strong performance for large-scale similarity search. With capabilities such as scalar quantization, Better Binary Quantization (BBQ), SIMD vector operations, and more disk-efficient algorithms like DiskBBQ, it already provides efficient and flexible options for managing vector workloads.

By integrating NVIDIA cuVS as a callable module for vector search tasks, we aim to deliver significant gains in vector indexing performance and efficiency to better support large-scale vector workloads.

The challenge

One of the toughest challenges in building a high-performance vector database is constructing the vector index - the HNSW graph. Index building quickly becomes dominated by millions or even billions of arithmetic operations as every vector is compared against many others. In addition, index lifecycle operations, such as compaction and merges, can further increase the overall compute overhead of indexing. As data volumes and associated vector embeddings grow exponentially, accelerated computing GPUs, built for massive parallelism and high-throughput math, are ideally positioned to handle these workloads.

Enter the Elasticsearch-GPU Plugin

NVIDIA cuVS is an open-source CUDA-X library for GPU-accelerated vector search and data clustering that enables fast index building and embedding retrieval for AI and recommendation workloads.

Elasticsearch uses cuVS through cuvs-java, an open-source library developed by the community and maintained by NVIDIA. The cuvs-java library is lightweight and builds on the cuVS C API using Panama Foreign Function to expose cuVS features in an idiomatic Java way, while remaining modern and performant.

The cuvs-java library is integrated into a new Elasticsearch plugin; therefore, vector indexing on the GPU can occur on the same Elasticsearch node and process, without the need to provision any external code or hardware. During index building, if the cuVS library is installed and a GPU is present and configured, Elasticsearch will use the GPU to accelerate the vector indexing process. The vectors are given to the GPU, which constructs a CAGRA graph. This graph is then converted to the HNSW format, making it immediately available for vector search on the CPU. The final format of the built graph is the same as what would be built on the CPU; this allows Elasticsearch to leverage GPUs for high-throughput vector indexing when the underlying hardware supports it, while freeing CPU power for other tasks (concurrent search, data processing, etc.).

Index build acceleration

As part of integrating GPU acceleration into Elasticsearch, several enhancements were made to cuvs-java, focusing on efficient data input/output and function invocation. A key enhancement is the use of cuVSMatrix to transparently model vectors, whether they reside on the Java heap, off-heap, or in GPU memory. This enables data to move efficiently between memory and the GPU, avoiding unnecessary copies of potentially billions of vectors.

Thanks to this underlying zero-copy abstraction, both transferring to GPU memory and retrieving the graph can occur directly. During indexing, vectors are first buffered in memory on the Java heap, then sent to the GPU to construct the CAGRA graph. The graph is subsequently retrieved from the GPU, converted into HNSW format, and persisted to disk.

At merge time, the vectors are already stored on disk, bypassing the Java heap entirely. Index files are memory-mapped, and data is transferred directly into GPU memory. The design also easily accommodates different bit-widths, such as float32 or int8, and naturally extends to other quantization schemes.

Drumroll…so, how does it perform?

Before we get into the numbers, a bit of context is helpful. Segment merging in Elasticsearch typically runs automatically in the background during indexing, which makes it difficult to benchmark in isolation. To obtain reproducible results, we used force-merge to explicitly trigger segment merging in a controlled experiment. Since force-merge performs the same underlying merge operations as background merging, its performance serves as a useful indicator of expected improvements, even though the exact gains may differ in real-world indexing workloads.

Now, let’s see the numbers.

Our initial benchmark results are very promising. We ran the benchmark on an AWS g6.4xlarge instance with locally attached NVMe storage. A single node of Elasticsearch was configured to use the default, optimal number of indexing threads (8 - one for each physical core), and to disable merge throttling (which is less applicable with fast NVMe disks).

For the dataset, we used 2.6 million vectors with 1,536 dimensions from the OpenAI Rally vector track, encoded as base64 strings, and indexed as float32 hnsw. In all scenarios, the constructed graphs achieve recall levels of up to 95%. Here’s what we found:

• Indexing Throughput: By moving graph construction to the GPU during in-memory buffer flushes, we increase throughput by ~12x.

• Force-merge: After indexing completes, the GPU continues to accelerate segment merging, speeding up the force-merge phase by ~7x.

• CPU usage: Offloading graph construction to the GPU significantly reduces both average and peak CPU utilization. The graphs below illustrate CPU usage during indexing and merging, highlighting how much lower it is when these operations run on the GPU. Lower CPU utilization during GPU indexing frees up CPU cycles that can be redirected to improve search performance.

• Recall: Accuracy remains effectively the same between CPU and GPU runs, with the GPU-built graph reaching marginally higher recall.

Comparing along another dimension: Price

The earlier comparison intentionally used identical hardware, with the only difference being whether the GPU was used during indexing. That setup is useful for isolating raw compute effects, but we can also look at the comparison from a cost perspective.

At roughly the same hourly price as the GPU-accelerated configuration, one can provision a CPU-only setup with approximately twice the comparable CPU and memory resources: 32 vCPUs (AMD EPYC) and 64 GB of RAM, allowing to double the number of indexing threads to 16.

To keep the comparison fair and consistent, we ran this CPU-only experiment on an AWS g6.8xlarge instance, with the GPU explicitly disabled. This allowed us to hold all other hardware characteristics constant while evaluating the cost–performance trade-off of GPU acceleration versus CPU-only indexing.

The more powerful CPU instance does show improved performance compared to the benchmarks in the above section, as you would expect. However, when we compare this more powerful CPU instance against the original GPU-accelerated results, the GPU still delivers substantial performance gains: ~5x improvement in indexing throughput, and ~6x in force merge, all while building graphs that achieve recall levels of up to 95%.

Conclusion

In end-to-end scenarios, GPU acceleration with NVIDIA cuVS delivers nearly a 12x improvement in indexing throughput and a 7x decrease in force-merge latency, with significantly lower CPU utilization. This shows that vector indexing and merge workloads benefit significantly from GPU acceleration. On a cost-adjusted comparison, GPU acceleration continues to yield substantial performance gains, with approximately 5x higher indexing throughput and 6x faster force-merge operations.

GPU-accelerated vector indexing is currently planned for Tech Preview in Elasticsearch 9.3, which is scheduled to be released early in 2026.

Stay tuned for more.

Our mission at Elastic is to make Apache Lucene the best vector database out there, and to continue to make Elasticsearch the best retrieval platform out there for search and RAG. Our investments into Lucene are key to ensure that every release of Elasticsearch brings increasing faster performance and scale.

Customers are already building the next generation of AI enabled search applications with Elastic’s vector database and vector search technology. Roboflow is used by over 500,000 engineers to create datasets, train models, and deploy computer vision models to production. Roboflow uses Elastic vector database to store and search billions of vector embeddings.

In this blog we summarize recent enhancements and optimisations that significantly improve vector search performance in Elasticsearch and Apache Lucene, over and above performance gains delivered with Lucene 9.9 and Elasticsearch 8.12.x.

The integration of vector search into Elasticsearch relies on Apache Lucene, the layer that orchestrates data storage and retrieval. Lucene's architecture organizes data into segments, immutable units that undergo periodic merging. This structure allows for efficient management of inverted indices, essential for text search. With vector search, Lucene extends its capabilities to handle multi-dimensional points, employing the hierarchical navigable small world (HNSW) algorithm to index vectors.

This approach facilitates scalability, enabling data sets to exceed available RAM size while maintaining performance. Additionally, Lucene's segment-based approach offers lock-free search operations, supporting incremental changes and ensuring visibility consistency across various data structures. The integration however comes with its own engineering challenges. Merging segments requires recomputing HNSW graphs, incurring index-time overhead. Searches must cover multiple segments, leading to possible latency overhead. Moreover, optimal performance requires scaling RAM as data grows, which may raise resource management concerns.

Lucene's integration into Elasticsearch comes with the benefit of robust vector search capabilities. This includes aggregations, document level security, geo-spatial queries, pre-filtering, to full compatibility with various Elasticsearch features. Imagine running vector searches using a geo bounding box, this is an example usecase enabled by Elasticsearch and Lucene.

Lucene's architecture lays a solid foundation for efficient and versatile vector search within Elasticsearch. Let’s explore optimization strategies and enhancements we have implemented to integrate vector search into Lucene, which delivers a high performance and comprehensive feature-set for developers.

Harnessing Lucene's architecture for multi-threaded search

Lucene's segmented architecture enables the implementation of multi-threaded search capabilities. Elasticsearch’s performance gains come from efficiently searching multiple segments simultaneously. Latency of individual searches is significantly reduced by using the processing power of all available CPU cores. While it may not directly improve overall throughput, this enhancement prioritizes minimizing response times, ensuring that users receive their search results as swiftly as possible.

Furthermore, this optimization is particularly beneficial for Hierarchical Navigable Small World (HNSW) searches, as each graph is independent of the others and can be searched in parallel, maximizing efficiency and speeding up retrieval times even further.

The advantage of having multiple independent segments extends to the architectural level, especially in serverless environments. In this new architecture, the indexing tier is responsible for creating new segments, each containing its own HSNW graph. The search tier can simply replicate these segments without incurring the CPU cost of indexation. This separation allows a significant portion of compute resources to be dedicated to searches, optimizing overall system performance and responsiveness.

Accelerating multi-graph vector search

In spite of gains achieved with parallelization, each segment's searches would remain independent, unaware of progress made by other segment searches. So our focus shifted towards optimizing the efficiency of concurrent searches across multiple segments.

The graph shows that the number queries per second increased from 104 queries/sec to 219 queries/sec.

Recognizing the potential for further speedups, we leveraged our insights from optimizing lexical search, to enable information exchange among segment searches allowing for better coordination and efficiency in vector search.

Our strategy for accelerating multi-graph vector search revolves around balancing exploration and exploitation within the proximity graph. By adjusting the size of the expanded match set, we control the trade-off between runtime and recall, crucial for achieving optimal search performance across multiple graphs.

In multi-graph search scenarios, the challenge lies in efficiently navigating individual graphs, while ensuring comprehensive exploration to avoid local minima. While searching multiple graphs independently yields higher recall, it incurs increased runtime due to redundant exploration efforts. To mitigate this, we devised a strategy to intelligently share state between searches, enabling informed traversal decisions based on global and local competitive thresholds.

This approach involves maintaining shared global and local queues of distances to closest vectors, dynamically adapting search parameters based on the competitiveness of each graph's local search. By synchronizing information exchange and adjusting search strategies accordingly, we achieve significant improvements in search latency while preserving recall rates comparable to single-graph searches.

The impact of these optimizations is evident in our benchmark results. In concurrent search and indexing scenarios, we notice up to 60% reduction in query latencies! Even for queries conducted outside of indexing operations, we observed notable speedups and a dramatic decrease in the number of vector operations required. These enhancements, integrated into Lucene 9.10 and subsequently Elasticsearch 8.13, mark significant strides towards enhancing vector database performance for search while maintaining excellent recall rates.

Harnessing Java's latest advancements for ludicrous speed

In the area of Java development, automatic vectorization has been a boon, optimizing scalar operations into SIMD (Single Instruction Multiple Data) instructions through the HotSpot C2 compiler. While this automatic optimization has been beneficial, it has its limitations, particularly in scenarios where explicit control over code shape yields superior performance. Enter Project Panama Vector API, a recent addition to the JDK offering an API for expressing computations reliably compiled to SIMD instructions at runtime.

Lucene's vector search implementation relies on fundamental operations like dot product, square, and cosine distance, both in floating point and binary variants. Traditionally, these operations were backed by scalar implementations, leaving performance enhancements to the JIT compiler. However, recent advancements introduce a paradigm shift, enabling developers to express these operations explicitly for optimal performance.

Consider the dot product operation, a fundamental vector computation. Traditionally implemented in Java with scalar arithmetic, recent innovations leverage the Panama Vector API to express dot product computations in a manner conducive to SIMD instructions. This revised implementation iterates over input arrays, multiplying and accumulating elements in batches, aligning with the underlying hardware capabilities.

By harnessing Panama Vector API, Java code now interfaces seamlessly with SIMD instructions, unlocking the potential for significant performance gains. The compiled code, when executed on compatible CPUs, leverages advanced vector instructions like AVX2 or AVX 512, resulting in accelerated computations. Disassembling the compiled code reveals optimized instructions tailored to the underlying hardware architecture.

Microbenchmarks comparing traditional Java implementations to those leveraging Panama Vector API illustrate dramatic performance improvements. Across various vector operations and dimension sizes, the optimized implementations outperform their predecessors by significant margins, offering a glimpse into the transformative power of SIMD instructions.

Micro-benchmark comparing dot product with the new Panama API (dotProductNew) and the scalar implementation (dotProductOld).

Beyond microbenchmarks, the real-world impact of these optimizations is quite exciting to think about. Vector search benchmarks, such as SO Vector, demonstrate notable enhancements in indexing throughput, merge times, and query latencies. Elasticsearch, embracing these advancements, incorporates the faster implementations by default, ensuring users reap the performance benefits seamlessly.

The graph shows indexing throughput increased from about 900 documents/sec to about 1300 documents/sec.

Despite the incubating status of Panama Vector API, its quality and potential benefits are undeniable. Lucene's pragmatic approach allows for selective adoption of non-final JDK APIs, balancing the promise of performance improvements with maintenance considerations. With Lucene and Elasticsearch, users can leverage these advancements effortlessly, with performance gains translating directly to real-world workloads.

The integration of Panama Vector API into Java development yields a new era of performance optimization, particularly in vector search scenarios. By embracing hardware-accelerated SIMD instructions, developers can unlock efficiency gains, visible both in microbenchmarks and macro-level benchmarks. As Java continues to evolve, leveraging its latest features promises to propel performance to new heights, enriching user experiences across diverse applications.

Maximizing memory efficiency with scalar quantization

Memory consumption has long been a concern for efficient vector database operations, particularly for searching large datasets. Lucene introduces a breakthrough optimization technique - scalar quantization - aimed at significantly reducing memory requirements without sacrificing search performance.

Consider a scenario where querying millions of float32 vectors of high dimensions demands substantial memory, leading to significant costs. By embracing byte quantization, Lucene slashes memory usage by approximately 75%, offering a viable solution to the memory-intensive nature of vector search operations.

For quantizing floats to bytes, Lucene implements Scalar quantization a lossy compression technique that transforms raw data into a compressed form, sacrificing some information for space efficiency. Lucene's implementation of scalar quantization achieves remarkable space savings with minimal impact on recall, making it an ideal solution for memory-constrained environments.

Lucene's architecture, consisting of nodes, shards, and segments, which facilitates efficient distribution and management of documents for search. Each segment stores raw vectors, quantized vectors, and metadata, ensuring optimized storage and retrieval mechanisms.

Lucene's vector quantization adapts dynamically over time, adjusting quantiles during segment merge operations to maintain optimal recall. By intelligently handling quantization updates and re-quantization when necessary, Lucene ensures consistent performance while accommodating changes in data distribution.

Example of merged quantiles where segments A and B have 1000 documents and C only has 100.

Experimental results demonstrate the efficacy of scalar quantization in reducing memory footprint while maintaining search performance. Despite minor differences in recall compared to raw vectors, Lucene's quantized vectors offer significant speed improvements and recall recovery with minimal additional vectors.

Recall@10 for quantized vectors vs raw vectors. The search performance of quantized vectors is significantly faster than raw, and recall is quickly recoverable by gathering just 5 more vectors; visible by quantized@15.

Lucene's scalar quantization presents a revolutionary approach to memory optimization in vector search operations. With no need for training or optimization steps, Lucene seamlessly integrates quantization into its indexing process, automatically adapting to changes in data distribution over time. As Lucene and Elasticsearch continue to evolve, widespread adoption of scalar quantization will revolutionize memory efficiency for vector database applications, paving the way for enhanced search performance at scale.

Achieving seamless compression with minimal impact on recall

To make compression even better, we aimed to reduce each dimension from 7 bits to just 4 bits. Our main goal was to compress data further while still keeping search results accurate. By making some improvements, we managed to compress data by a factor of 8 without making search results worse. Here's how we did it.

We focused on keeping search results accurate while making data smaller. By making sure we didn't lose important information during compression, we could still find things well even with less detailed data. To make sure we didn't lose any important information, we added a smart error correction system.

We checked our compression improvements by testing them with different types of data and real search situations. This helped us see how well our searches worked with different compression levels and what we might lose in accuracy by compressing more.

Comparison of int4 dot product values to the corresponding float values for a random sample of 100 documents and their 10 nearest neighbors.

These compression features were created to easily work with existing vector search systems. They help organizations and users save space without needing to change much in their setup. With this simple compression, organizations can expand their search systems without wasting resources.

In short, moving to 4 bits per dimension for scalar quantization was a big step in making compression more efficient. It lets users compress their original vectors by 8 times. By optimizing carefully, adding error correction, testing with real data, and offering scalable deployment, organizations could save a lot of storage space without making search results worse. This opens up new chances for efficient and scalable search applications.

Paving the way for binary quantization

The optimization to reduce each dimension to 4 bits not only delivers significant compression gains but also lays the groundwork for further advancements in compression efficiency. Specifically, future advancements like binary quantization into Lucene, a development that has the potential to revolutionize vector storage and retrieval.

In an ongoing effort to push the boundaries of compression in vector search, we are actively working on integrating binary quantization into Lucene using the same techniques and principles that underpin our existing optimization strategies. The goal is to achieve binary quantization of vector dimensions, thereby reducing the size of the vector representation by a factor of 32 compared to the original floating-point format.

Through our iterations and experiments, we want to deliver the full potential of vector search while maximizing resource utilization and scalability. Stay tuned for further updates on our progress towards integrating binary quantization into Lucene and Elasticsearch, and the transformative impact it will have on vector database storage and retrieval.

Multi-vector integration in Lucene and Elasticsearch

Several real world applications rely on text embedding models and large text inputs. Most embedding models have token limits, which necessitate chunking of longer text into passages. Therefore, instead of a single document, multiple passages and embeddings must be managed, potentially complicating metadata preservation.

Now instead of having a single piece of metadata indicating, for example the first chapter of the book “Little Women”, you have to index that information data for every sentence.

Lucene's "join" functionality, integral to Elasticsearch's nested field type, offers a solution. This feature enables multiple nested documents within a top-level document, allowing searches across nested documents and subsequent joins with their parent documents. So, how do we deliver support for vectors in nested fields with Elasticsearch?

The key lies in how Lucene joins back to parent documents when searching child vector passages. The parallel concept here is the debate around pre-filtering versus post-filtering in kNN methods, as the timing of joining significantly impacts result quality and quantity. To address this, recent enhancements to Lucene enable pre-joining against parent documents while searching the HNSW graph.

Practically, pre-joining ensures that when retrieving the k nearest neighbors of a query vector, the algorithm returns the k nearest documents instead of passages. This approach diversifies results without complicating the HNSW algorithm, requiring only a minimal additional memory overhead per stored vector.

Efficiency is improved by leveraging certain restrictions, such as disjoint sets of parent and child documents and the monotonicity of document IDs. These restrictions allow for optimizations using bit sets, providing rapid identification of parent document IDs.

Searching through a vast number of documents efficiently required investing in nested fields and joins in Lucene. This work helps storage and search for dense vectors that represent passages within long texts, making document searches in Lucene more effective. Overall, these advancements represent an exciting step forward in the area of vector database retrieval within Lucene.

Wrapping up (for now)

We're dedicated to making Elasticsearch and Lucene the best vector database with every release. Our goal is to make it easier for people to search for things. With some of the investments we discuss in this blog, there is significant progress, but we're not done!

To say that the gen AI ecosystem is rapidly evolving is an understatement. At Elastic, we want to give developers the most flexible and open tools to keep up with all the innovation—with features available across recent releases until 8.13 and serverless

kNN search in Elasticsearch is organized as a top level section of a search request. We have designed it this way so that:

• It can always return global k nearest neighbors regardless of a number of shards

• These global k results are combined with a results from other queries to form a hybrid search

• The global k results are passed to aggregations to form facets.

Here is a simplified diagram how kNN search is executed internally (some phases are omitted) :

Figure 1: The steps for the top level kNN search are:

1. A user submits a search request

2. The coordinator node sends a kNN search part of the request to data nodes in the DFS phase

3. Each data node runs kNN search and sends back the local top-k results to the coordinator

4. The coordinator merges all local results to form the global top k nearest neighbors.

5. The coordinator sends back the global k nearest neighbors to the data nodes with any additional queries provided

6. Each data node runs additional queries and sends back the local size results to the coordinator

7. The coordinator merges all local results and sends a response to the user

We first run kNN search in the DFS phase to obtain the global top k results. These global k results are then passed to other parts of the search request, such as other queries or aggregations. Even the execution looks complex, from a user’s perspective this model of running kNN search is simple, as the user can always be sure that kNN search returns the global k results.

Introducing kNN query in Elasticsearch

With time we realized there is also a need to represent kNN search as a query. Query is a core component of a search request in Elasticsearch, and representing kNN search as a query allows for flexibility to combine it with other queries to address more complex requests.

kNN query, unlike the top level kNN search, doesn’t have a k parameter. The number of results (nearest neighbors) returned is defined by the size parameter, as in other queries. Similar to kNN search, the num_candidates parameter defines how many candidates to consider on each shard while executing a kNN search.

kNN query is executed differently from the top level kNN search. Here is a simplified diagram that describes how a kNN query is executed internally (some phases are omitted):

Figure 2: The steps for query based kNN search are:

1. A user submits a search request

2. The coordinator sends to the data nodes a kNN search query with additional queries provided

3. Each data node runs the query and sends back the local size results to the coordinator node

4. The coordinator node merges all local results and sends a response to the user

We run kNN search on a shard to get num_candidates results; these results are passed to other queries and aggregations on a shard to get size results from the shard. As we don’t collect the global k nearest neighbors first, in this model the number of nearest neighbors collected and visible for other queries and aggregations depend on the number of shards.

kNN query API examples

Let’s look at API examples that demonstrate differences between the top level kNN search and kNN query.

We create an index of products and index some documents:

kNN query similar to the top level kNN search, has num_candidates and an internal filter parameter that acts as a pre-filter.

kNN query can get more diverse results than kNN search for collapsing and aggregations. For the kNN query below, on each shard we execute kNN search to obtain 10 nearest neighbors which are then passed to collapse to get 3 top results. Thus, we will get 3 diverse hits in a response.

The top level kNN search first gets the global top 3 results in the DFS phase, and then passes them to collapse in the query phase. We will get only 1 hit in a response, as all the global 3 nearest neighbors happened to be from the same brand.

Similarly for aggregations, a kNN query allows us to get 3 distinct buckets, while kNN search only allows 1.

Now, let’s look at other examples that show the flexibility of the kNN query. Specifically, how it can be flexibly combined with other queries.

kNN can be a part of a boolean query (with a caveat that all external query filters are applied as post-filters for kNN search). We can use a _name parameter for kNN query to enhance results with extra information that tells if the kNN query was a match and its score contribution.

kNN can also be a part of complex queries, such as a pinned query. This is useful when we want to display the top nearest results, but also want to promote a selected number of other results.

We can even make the kNN query a part of our function_score query. This is useful when we need to define custom scores for results returned by kNN query: ​

kNN query being a part of dis_max query is useful when we want to combine results from kNN search and other queries, so that a document’s score comes from the highest ranked clause with a tie breaking increment for any additional clause. ​

kNN search as a query has been introduced with the 8.12 release. Please try it out, and we would appreciate any feedback.