We didn't want to fire back with another round of numbers. Reproducing a benchmark is the easy part; understanding why it produces the numbers it does is the useful part, and it is the part both original posts skipped. So we stood up Qdrant's exact setup on our own cluster, loaded the same 21 million vectors, reproduced their result, and then traced every number back to its cause.

Here is what we found. The result comes down to a few setup choices: how many segments you build, whether the data is warm in memory, how you return the result ids, and which hardware you run on. Control those, and it cuts both ways: matched to Qdrant's setup the two engines are on par on query speed, and with fast default ingestion Elasticsearch is faster at search and faster to index. The big multipliers in both posts, our 7x and their reply, are artifacts of those choices, not a verdict on either engine. The two reasons Qdrant leaned on hardest, an io_uring disk scorer and Elasticsearch's memory use, turn out to be the two that mattered least: the disk is never read during the run, and the memory gap is a difference in labels, not in bytes.

We hold our own original post to the same standard. Both sides changed several things at once and reported the result rather than the reason. One rule for the rest of this post: every claim gets a number and a mechanism, or it doesn't ship.

Vector search benchmark setup: hardware, config and query set

Here is exactly what each side ran.

Elastic (original post)

Qdrant (response)

This post (apples-to-apples)

Nodes

3 × n4-standard-8 (7 vCPU / 26 GB), GCP

3 × m6g.large (2 vCPU / 8 GB), AWS

3 × m6g.large (2 vCPU / 8 GB)



Elasticsearch

bbq_disk 2-bit, replicas: 1

(cited original)

bbq_disk 2-bit, vectordb_document, bfloat16, replicas: 0



Qdrant

2-bit, async scorer off, RF=2

TurboQuant 4-bit, async scorer on, RF=1

TurboQuant 4-bit, async scorer on, RF=1



Copies per shard

2 = 2 (matched)

1

1 = 1 (matched)



Query set

10k fixed, recall@100

10k fixed, recall@100

10k fixed, recall@100

Three things are worth pinning down, because all three have been used as talking points.

1. Replica count was matched in both rounds: An Elasticsearch index with replicas: 1 keeps two copies of each shard, which is exactly what Qdrant's RF=2 does. In this round, both sides ran a single copy. It was never a thumb on the scale in either direction.

2. We changed a few things on our side on purpose, and we will own them: vectordb_document index mode, float32 to bfloat16 for stored full-precision vectors (3,072 down to 1,536 bytes each), and a merge policy tuned for vector data. Those choices matter later, so we are flagging them up front rather than burying them.

3. Memory is compared like for like. Qdrant's post sets Elasticsearch's JVM heap allocation against Qdrant's total memory use, but those are different accounting categories, since heap size is not total memory consumption. On these 8 GB nodes we give the JVM heap 2 GB, 25% of node RAM, deliberately: that follows Elastic's Vector Search Optimized profile, which leaves the other 75% for the OS page cache where the vectors actually live, rather than the general 50%-heap upper bound. The honest comparison is resident index plus runtime plus active page cache, and on an 8 GB node both engines live inside the same envelope.

Look at the middle column, though. Between our original run and their response, Qdrant changed almost everything at once: new hardware, 4-bit quantization instead of 2-bit, the async scorer on, a different replication factor. Then it credited the win to one of those changes. Drawing meaningful conclusions from many simultaneous changes is hard, and pinning down which change actually moved the number is the whole job of the sections that follow.

What does disk rescore cost in a vector search benchmark?

In this benchmark, the answer is almost nothing, because the disk is never read. Two back-of-the-envelope budgets show why, and the live measurements later in the post confirm it.

The design under test is built around one idea: keep a small quantized copy of each vector in RAM for approximate search, keep the full-precision originals on disk, and read a handful of those originals back to rescore the top candidates for accuracy.

Two budgets determine performance: disk I/O for rescore reads and RAM for what has to stay resident. Both are computable on the back of an envelope, so let us compute them before measuring anything.

The disk I/O budget for vector search rescoring

Rescoring reads the top 100 candidates per query. A 768-dimensional float32 vector is 768 × 4 = 3,072 bytes. Because those reads are not page-aligned, the real cost per vector is closer to an 8 KB page read. Each query needs roughly 800 KB and 100 random reads. Scale that up:

target QPS (cold)

rescored vectors/s

reads/s (IOPS) per node

MB/s per node

10

1,000

333

3

100

10,000

3,333

27

1,000

100,000

33,333

273

10,000

1,000,000

333,333

2,731

Two things fall out immediately: First, cold rescore is an IOPS problem, never a bandwidth one. Even at 10,000 QPS, you need only about 2.7 GB/s per node, but a third of a million IOPS per node. Second, the total working set for the benchmark is tiny: 100 candidates × 10,000 queries × 8 KB = 8 GB across the cluster, or 2.7 GB per node. That fits in the free page cache on an 8 GB node with room to spare. The first time the benchmark cycles through its 10,000 queries, the originals it touches are pulled into page cache, and every read after that is a memory copy.

Put a ceiling on it too. On gp3 at baseline (3,000 IOPS/node), if the reads did go to disk, the workload would top out around 90 QPS from IOPS alone; provisioned gp3 or local NVMe would push that to roughly 480 or 3,000. Hold that number. It is the ceiling that would matter in a benchmark that actually touched the disk.

The RAM budget for a disk-based vector index

What has to stay resident is the quantized copy plus the search structure. Everything else is page cache. At 4-bit quantization and an HNSW graph with m=16:

component

bytes/vector

per node @ 21M

4-bit quantized vectors (always_ram)

384

2.69 GB

HNSW graph, m=16 (2m links × 4 B)

136

0.95 GB

resident total

520

3.64 GB

raw float32 originals (on disk)

3,072

21.5 GB (on disk)

We measured the resident footprint on the running nodes at 3.4 to 3.9 GB, sitting right on the 3.64 GB estimate. The point to hold is that at 21 million vectors, the entire searchable index (quantized vectors plus graph) is about 3.6 GB per node and it fits in RAM, leaving a couple of gigabytes to spare. Nothing in this benchmark forces the disk-backed design to actually use the disk. We come back to that at the end, because it is the real story.

Does io_uring make vector search faster?

Qdrant's reply credits two levers it says we omitted: a two-stage prefetch-and-rescore retrieval pattern and an async, io_uring-based disk scorer. They are two faces of the same rescore step, and the same evidence answers both. Take the async scorer first, since it is the one Qdrant makes the centerpiece: io_uring to parallelize disk reads during rescoring. It is a good feature, and in this benchmark it had exactly one job: parallelize the disk reads during rescoring. It got to do none of them. We verified that in three independent ways.

First, the arithmetic. From the RAM budget, the quantized index and graph are pinned in RAM. From the I/O budget, the rescore working set is 2.7 GB per node and lives in page cache after the first pass. That leaves no disk reads on the hot path to accelerate.

Second, their own methodology guarantees it. Each operating point in the harness runs a full 10,000-query recall pass, followed by a timed throughput window that cycles through the same 10,000 queries again. By the time the stopwatch starts, every vector those queries will touch is already resident. The measurement is warm by construction.

Third, and this is the part we insisted on doing rather than arguing, we measured it. We were careful here because it is easy to accidentally test io_uring in the off state and not notice. Running Qdrant in Docker, we found that io_uring was not even initializing: the default seccomp profile blocks the io_uring syscalls, so it silently fell back to synchronous reads (failed to initialize io_uring instance: Operation not permitted). We fixed that, ran the container with seccomp unconfined, confirmed zero io_uring errors in the logs, and re-ran. This honestly means we have both states:

• Async scorer effectively off (synchronous fallback): 31.6 QPS at ef=50.
• Async scorer on (io_uring confirmed working): 35.8 QPS at ef=50.

A 13% move, and even that is within the run-to-run noise for a measurement whose ceiling is set by CPU, not disk. Put plainly, switching on the feature that the entire result was credited to changed about as much as running the benchmark a second time. To close the loop, during the actual throughput window we watched the block device on all three nodes with iostat: read throughput held at 0 MB/s and 0 IOPS, while CPU sat at 60-70% and climbed to 100% as we added concurrency. The bottleneck is the CPU doing quantized distance computations and graph traversal. It is not the disk, because the disk is asleep.

The two-stage prefetch-and-rescore pattern, the lever Qdrant lists first, is answered by the same run. Every Qdrant number here comes from Qdrant's own reproduction script, which uses their two-stage query throughout, so two-stage was on for the whole benchmark, including the io_uring comparison, and the disk still held at 0 IOPS. It is also not something we omitted: prefetch is the approximate search over the quantized vectors, which lives in RAM, and rescore reads the originals, which live in page cache here. That is the disk-rescore workload, and bbq_disk runs the identical shape. The one part of two-stage that is not about disk, how many candidates you rescore, is CPU rather than I/O; it is already included in these numbers and available to both engines, so it is not a hidden lever either.

There is a smaller detail worth noting. Qdrant's post does not say which disk they ran on. That would normally matter in a benchmark whose premise is disk access, but here it does not, for the same reason the async scorer does not: you cannot be bottlenecked on a device you never read from. The disk is left unspecified, and as it happens it is also beside the point.

Here's what I've seen, as an engineer who has sat through a lot of these conversations: io_uring gets treated as a universal fix far more often than it is the bottleneck. It is excellent when the working set overflows RAM, and the cold I/O budget above shows exactly that case, where parallelizing reads buys you throughput. This benchmark never enters that regime. Async disk reads are a real answer to a real question, and this benchmark just never asked it. Of everything that changed between the two runs, io_uring is the one that got the headline and moved the number the least.

Why does faster ingestion mean slower vector search queries?

This is where reproducing the number got interesting, and where the honest driver of the gap turned out to live.

Our first load produced 128 segments and roughly half the throughput we expected. Qdrant's published provenance, right there in their results JSON, records 67 segments. That one difference explained almost everything. When we merged our collection down to match their segment count, all three metrics moved together toward their numbers:

segments

recall@100

QPS

avg latency

128 (our first, parallel load)

0.9745

35.8

112 ms

66 (merged to match theirs)

0.9531

53.3

75 ms

67 (Qdrant published)

0.9596

67.2

59.5 ms

Look at what moves. Recall falls as segments drop, because with fewer segments each query examines fewer total candidates. That is the tell that segment count, not tuning, was inflating our recall. And throughput rises, because HNSW query cost is very sensitive to the number of segments: every query fans out across every segment's graph, so twice the segments is close to twice the per-query work, plus the contention of doing all that on two cores. The small remaining gap between our 53 and their 67 is warm-cache completeness in our shorter run, not the engine.

Here is the part worth stating carefully, because it is architectural and it cuts against us as much as for us. We use the same segmented design; this is not a Qdrant quirk. The question is how you get to few segments. You get there by ingesting single-threaded, so data concentrates into fewer, larger segments. Fewer segments means faster queries, but you pay for it with slow ingestion. Our parallel load was fast and produced many segments; their single-threaded load was slower and produced few. It is a real tradeoff, and it is the same one on both sides.

Where the two designs genuinely differ is how much that tradeoff hurts. Building an HNSW graph is expensive, and its query cost scales poorly with segment count. On these 2-vCPU nodes, constructing the graph at ef_construct=256 took about 1.6 hours pinned at 100% CPU. The IVF layout behind bbq_disk is cheap to build and far less sensitive to segment count, so we can ingest with many threads while keeping query latency low. We do not have to choose. That is not a benchmark trick. It is the IVF vs. HNSW tradeoff, stated honestly.

Why document retrieval, not vector search, explains the rest of the latency

Now turn the same lens on our own result, because the same accounting applies to our own numbers too.

At the search-light end of the sweep, a large share of the per-query time is not vector search at all. It is the fetch phase assembling the response. vectordb_document returns full source documents, which is exactly what you want for retrieval and hybrid search, where you actually need the document back. A pure vector benchmark only needs the top-N ids, and today _id lives in the same stored-fields column as _source, the text and the vector. So fetching an id pulls that entire compressed block through the decompressor for every hit. When we isolate the id from that column, Elasticsearch steps straight up into Qdrant's low-segment throughput range. That gap is document retrieval, not vector search.

The fix is structural, not a tuning flag: give _id its own doc-values field so returning an id never touches _source. That is what the vectordb_columnar mode we are building automatically does. We are calling it out here because good benchmarking means naming your own costs as clearly as anyone else's, and this one is a document-retrieval cost sitting inside a vector-search score.

Elasticsearch vs Qdrant vector search benchmark: our numbers, on the same box

Elasticsearch on the same three m6g.large nodes, vectordb_document with bfloat16, warm, shown both as it runs today and with the id isolated:

visit %

recall@100

QPS (default)

QPS (id-isolated)

1

0.894

44

89

2

0.939

39

73

3

0.956

35

56

5

0.970

31

40

The id-isolated column is measured by dropping the stored-fields fetch entirely, which is a near-upper-bound proxy for a doc-values _id (the doc-values read is cheap but not literally free). At around 0.96 recall, id-isolated Elasticsearch does about 56 QPS, sitting right alongside Qdrant's 53 to 67 on identical hardware. Same ballpark. The gaps in both posts came from setup and document retrieval, not from the vector engine: segment counts, the id fetch, the hardware. Their reply we can account for in full here; the larger 7x from our own first post we cannot yet, and we take that up below. When you control for those, two well-built systems doing the same work on the same box land close together, which is what you would expect.

So here is the plain claim: performance depends on the setup. Match Qdrant's slow, single-threaded ingestion, the one that gives them their 67-segment configuration, and the two engines are on par on the search itself: id-isolated Elasticsearch does about 56 QPS against their 53 to 67 at the same recall. But that low segment count is bought with slow ingestion. Let both engines ingest fast, which is the natural default, and Qdrant lands back at 128 segments and about 35 QPS, while Elasticsearch degrades less as the segment count climbs, because the IVF layout behind bbq_disk is less sensitive to segment count than HNSW. So with default, fast ingestion, Elasticsearch is faster at search too, and it reached a queryable index faster to begin with. The one place we still trail is returning the ids, which is document retrieval rather than search, and it is exactly what vectordb_columnar removes.

We deliberately did not run the 4 vCPU / 16 GB tier. It lifts both engines together and shows the identical trend, so it would add cost without adding insight.

Our original 7x Elasticsearch vs Qdrant benchmark number, explained

The rule in this post is that every claim gets a number and a mechanism. That rule was also applied to our original benchmark.

That post measured a 7x throughput advantage for Elasticsearch, and the number is real for that configuration. The setup was disclosed in full, including that Qdrant does better on NVMe. But the post attached a mechanism to the number: Qdrant was bottlenecked by random disk reads of the original vectors during rescoring, a problem made worse by network-attached storage. The arithmetic in this post undercuts that explanation. Those round-1 nodes had 26 GB of RAM, more headroom than the 8 GB nodes here where we watched the disk sit at 0 IOPS, so if the disk was idle here, it was almost certainly idle there too.

So what actually held round-1 Qdrant to 4.5 QPS at 0.97 recall? We cannot claim to know yet, but the original post's own configuration points the way: it ran Qdrant on 2-bit quantization with oversampling pinned at 1. Two-bit codes are coarse, and with oversampling fixed at 1 the only lever left to recover recall is ef. Reaching 0.97 recall on ef alone means a very large ef, and a very large ef makes each query expensive on its own, before disk enters the picture at all. That is still a hypothesis, but the mechanism the original post named, random disk reads during rescore, does not survive the same arithmetic we just applied to Qdrant.

Benchmarking, not benchmarketing

The numbers reproduce; the explanations don't. A benchmark that made "disk reads" its headline ran with the disk asleep, and we watched it sit at 0 IOPS for the whole window. The honest differences we could actually find were a segment count (an ingest-speed-versus-query-speed tradeoff where the IVF layout lets bbq_disk ingest fast and query fast) and a stored-field retrieval cost on our side (document retrieval, not vector search, and something vectordb_columnar removes). None of it is io_uring, and none of it is "Java is heavy."

The original and arguably bigger point got lost in the io_uring discussion: putting vectors on disk was never really about the rescore, it is about memory. The key was keeping the searchable index, the quantized vectors and the IVF or HNSW structure, compact and disk-resident enough to serve more vectors per gigabyte of RAM than a design that pins everything in memory. Qdrant's setup pins the 4-bit vectors with always_ram and keeps the HNSW graph in RAM, about 3.6 GB per node for 21 million vectors. bbq_disk keeps the quantized IVF on disk. At 21 million vectors on 8 GB nodes, everything fits in RAM either way, which is precisely why this benchmark cannot tell the two designs apart. It is measuring the case where the interesting variable has been held constant.

The interesting question is what happens when the searchable index stops fitting. Scale the corpus until the quantized vectors and the structure exceed page cache, and the two designs diverge: one keeps serving from disk, the other needs more RAM per node. That is the regime bbq_disk was built for, and it is the one neither post has measured. The rescore-under-disk-pressure case, many more distinct queries than fit in cache, is worth measuring too, but it is the second question, not the first.

We tried to get there, and our first attempt still fit in cache, because 21 million vectors on these nodes do not spill. So we are not done. We are building a benchmark with a corpus large enough that the searchable index no longer fits in RAM, and we will publish those numbers, with the reasons attached and checked the same way.

The ask here is to the reader, not to Qdrant: when a benchmark hands you a clean multiplier, chase every number back to a cause before you believe the story around it. Half the time it is a warm cache or a segment count.

The Qdrant team built a good engine, and we're not questioning that. The honest verdict is not about who is faster: matched to their setup, the two are on par, and on the fast default path, Elasticsearch is quicker for both indexing and searching. The numbers were fine. The reasons attached to them were the part worth checking, starting with our own. And the benchmark that would actually stress a disk-backed index, a working set that does not fit in RAM, is still to be written. We will bring the numbers.

This change reduces the overhead of parsing vectors in JSON by an order of magnitude, which translates to almost a 100% improvement on indexing throughput for DiskBBQ and around 20% improvement for hierarchical navigable small world (HNSW) workloads. In this blog, we’ll take a closer look at Base64-encoded strings and the improvements it brings to vector ingestion.

What’s the problem?

At Elastic, we’re always looking for ways to improve our vector search capabilities, whether that’s enhancing existing storage formats or introducing new ones. Recently, for example, we added a new disk-friendly storage format called DiskBBQ and enabled vector indexing with NVIDIA cuVS.

In both cases, we expected to see major gains in ingestion speed. However, once these changes were fully integrated into Elasticsearch, the improvements weren’t as large as we had hoped. A flamegraph of the ingestion process made the issue clear: JSON parsing had become one of the main bottlenecks.

Parsing JSON requires walking through every element in the arrays and converting numbers from text format into 32-bit floating-point values, which is very expensive.

Why Base64-encoded strings?

The most efficient way to parse vectors is directly from their binary representation, where each element uses a 32-bit floating-point value. However, JSON is a text-based format, and the way to include binary data in it is by using Base64-encoded strings. Base64 is just a binary-to-text encoding schema.

We can now send vectors encoded as Base64 strings:

Is it worth it? Our benchmarks suggest yes. When parsing 1,000 JSON documents, using Base64 encoded strings instead of float arrays resulted in performance improvements of more than an order of magnitude, at the cost of a small encode/decode trade-off (client-side Base64 encoding and a temporary byte array on the server for decoding) in exchange for eliminating expensive per-element numeric parsing.

Give me some ingestion numbers

We can see these improvements in practice when running the so_vector rally track with the different approaches. The actual gains depend on how fast indexing is for each storage format. For bbq_disk, indexing throughput increases by about 100%, while for bbq_hnsw, the improvement is closer to 20%, since indexing is inherently slower there.

Starting with Elasticsearch v9.2, vectors are excluded from _source by default and are stored internally as 32-bit floating-point values. This behavior also applies to Base64-encoded vectors, making the choice of indexing format completely transparent at search time.

Client support

Adding a new format for indexing vectors might require changes on ingestion pipelines. To help this effort, in v9.3, Elasticsearch official clients can transform vectors with 32-bit floating-point values into Base64-encoded strings and the other way around. You might need to check the client documentation for the specific implementation.

For example, here’s a snippet for implementing bulk loading using the Python client:

The only difference from a bulk ingest using floats is that the embedding is wrapped with the pack_dense_vector() auxiliary function.

Conclusion

By switching from JSON float arrays to Base64-encoded vectors, we remove one of the largest remaining bottlenecks in Elasticsearch’s vector ingestion pipeline: numeric parsing. The result is a simple change with outsized impact: up to 2× higher throughput for DiskBBQ workloads and meaningful gains even for slower indexing strategies, like HNSW.

Because vectors are already stored internally in a binary format and excluded from _source by default, this improvement is completely transparent at search time. With official client support landing in v9.3, adopting Base64 encoding requires only minimal changes to existing ingestion code, while delivering immediate performance benefits.

If you’re indexing large volumes of embeddings, especially in high-throughput or serverless environments, Base64-encoded vectors are now the fastest and most efficient way to get your data into Elasticsearch.Those interested in the implementation details can follow the related Elasticsearch issues and pull requests: #111281 and #135943.

Our mission at Elastic is to make Apache Lucene the best vector database out there, and to continue to make Elasticsearch the best retrieval platform out there for search and RAG. Our investments into Lucene are key to ensure that every release of Elasticsearch brings increasing faster performance and scale.

Customers are already building the next generation of AI enabled search applications with Elastic’s vector database and vector search technology. Roboflow is used by over 500,000 engineers to create datasets, train models, and deploy computer vision models to production. Roboflow uses Elastic vector database to store and search billions of vector embeddings.

In this blog we summarize recent enhancements and optimisations that significantly improve vector search performance in Elasticsearch and Apache Lucene, over and above performance gains delivered with Lucene 9.9 and Elasticsearch 8.12.x.

The integration of vector search into Elasticsearch relies on Apache Lucene, the layer that orchestrates data storage and retrieval. Lucene's architecture organizes data into segments, immutable units that undergo periodic merging. This structure allows for efficient management of inverted indices, essential for text search. With vector search, Lucene extends its capabilities to handle multi-dimensional points, employing the hierarchical navigable small world (HNSW) algorithm to index vectors.

This approach facilitates scalability, enabling data sets to exceed available RAM size while maintaining performance. Additionally, Lucene's segment-based approach offers lock-free search operations, supporting incremental changes and ensuring visibility consistency across various data structures. The integration however comes with its own engineering challenges. Merging segments requires recomputing HNSW graphs, incurring index-time overhead. Searches must cover multiple segments, leading to possible latency overhead. Moreover, optimal performance requires scaling RAM as data grows, which may raise resource management concerns.

Lucene's integration into Elasticsearch comes with the benefit of robust vector search capabilities. This includes aggregations, document level security, geo-spatial queries, pre-filtering, to full compatibility with various Elasticsearch features. Imagine running vector searches using a geo bounding box, this is an example usecase enabled by Elasticsearch and Lucene.

Lucene's architecture lays a solid foundation for efficient and versatile vector search within Elasticsearch. Let’s explore optimization strategies and enhancements we have implemented to integrate vector search into Lucene, which delivers a high performance and comprehensive feature-set for developers.

Harnessing Lucene's architecture for multi-threaded search

Lucene's segmented architecture enables the implementation of multi-threaded search capabilities. Elasticsearch’s performance gains come from efficiently searching multiple segments simultaneously. Latency of individual searches is significantly reduced by using the processing power of all available CPU cores. While it may not directly improve overall throughput, this enhancement prioritizes minimizing response times, ensuring that users receive their search results as swiftly as possible.

Furthermore, this optimization is particularly beneficial for Hierarchical Navigable Small World (HNSW) searches, as each graph is independent of the others and can be searched in parallel, maximizing efficiency and speeding up retrieval times even further.

The advantage of having multiple independent segments extends to the architectural level, especially in serverless environments. In this new architecture, the indexing tier is responsible for creating new segments, each containing its own HSNW graph. The search tier can simply replicate these segments without incurring the CPU cost of indexation. This separation allows a significant portion of compute resources to be dedicated to searches, optimizing overall system performance and responsiveness.

Accelerating multi-graph vector search

In spite of gains achieved with parallelization, each segment's searches would remain independent, unaware of progress made by other segment searches. So our focus shifted towards optimizing the efficiency of concurrent searches across multiple segments.

The graph shows that the number queries per second increased from 104 queries/sec to 219 queries/sec.

Recognizing the potential for further speedups, we leveraged our insights from optimizing lexical search, to enable information exchange among segment searches allowing for better coordination and efficiency in vector search.

Our strategy for accelerating multi-graph vector search revolves around balancing exploration and exploitation within the proximity graph. By adjusting the size of the expanded match set, we control the trade-off between runtime and recall, crucial for achieving optimal search performance across multiple graphs.

In multi-graph search scenarios, the challenge lies in efficiently navigating individual graphs, while ensuring comprehensive exploration to avoid local minima. While searching multiple graphs independently yields higher recall, it incurs increased runtime due to redundant exploration efforts. To mitigate this, we devised a strategy to intelligently share state between searches, enabling informed traversal decisions based on global and local competitive thresholds.

This approach involves maintaining shared global and local queues of distances to closest vectors, dynamically adapting search parameters based on the competitiveness of each graph's local search. By synchronizing information exchange and adjusting search strategies accordingly, we achieve significant improvements in search latency while preserving recall rates comparable to single-graph searches.

The impact of these optimizations is evident in our benchmark results. In concurrent search and indexing scenarios, we notice up to 60% reduction in query latencies! Even for queries conducted outside of indexing operations, we observed notable speedups and a dramatic decrease in the number of vector operations required. These enhancements, integrated into Lucene 9.10 and subsequently Elasticsearch 8.13, mark significant strides towards enhancing vector database performance for search while maintaining excellent recall rates.

Harnessing Java's latest advancements for ludicrous speed

In the area of Java development, automatic vectorization has been a boon, optimizing scalar operations into SIMD (Single Instruction Multiple Data) instructions through the HotSpot C2 compiler. While this automatic optimization has been beneficial, it has its limitations, particularly in scenarios where explicit control over code shape yields superior performance. Enter Project Panama Vector API, a recent addition to the JDK offering an API for expressing computations reliably compiled to SIMD instructions at runtime.

Lucene's vector search implementation relies on fundamental operations like dot product, square, and cosine distance, both in floating point and binary variants. Traditionally, these operations were backed by scalar implementations, leaving performance enhancements to the JIT compiler. However, recent advancements introduce a paradigm shift, enabling developers to express these operations explicitly for optimal performance.

Consider the dot product operation, a fundamental vector computation. Traditionally implemented in Java with scalar arithmetic, recent innovations leverage the Panama Vector API to express dot product computations in a manner conducive to SIMD instructions. This revised implementation iterates over input arrays, multiplying and accumulating elements in batches, aligning with the underlying hardware capabilities.

By harnessing Panama Vector API, Java code now interfaces seamlessly with SIMD instructions, unlocking the potential for significant performance gains. The compiled code, when executed on compatible CPUs, leverages advanced vector instructions like AVX2 or AVX 512, resulting in accelerated computations. Disassembling the compiled code reveals optimized instructions tailored to the underlying hardware architecture.

Microbenchmarks comparing traditional Java implementations to those leveraging Panama Vector API illustrate dramatic performance improvements. Across various vector operations and dimension sizes, the optimized implementations outperform their predecessors by significant margins, offering a glimpse into the transformative power of SIMD instructions.

Micro-benchmark comparing dot product with the new Panama API (dotProductNew) and the scalar implementation (dotProductOld).

Beyond microbenchmarks, the real-world impact of these optimizations is quite exciting to think about. Vector search benchmarks, such as SO Vector, demonstrate notable enhancements in indexing throughput, merge times, and query latencies. Elasticsearch, embracing these advancements, incorporates the faster implementations by default, ensuring users reap the performance benefits seamlessly.

The graph shows indexing throughput increased from about 900 documents/sec to about 1300 documents/sec.

Despite the incubating status of Panama Vector API, its quality and potential benefits are undeniable. Lucene's pragmatic approach allows for selective adoption of non-final JDK APIs, balancing the promise of performance improvements with maintenance considerations. With Lucene and Elasticsearch, users can leverage these advancements effortlessly, with performance gains translating directly to real-world workloads.

The integration of Panama Vector API into Java development yields a new era of performance optimization, particularly in vector search scenarios. By embracing hardware-accelerated SIMD instructions, developers can unlock efficiency gains, visible both in microbenchmarks and macro-level benchmarks. As Java continues to evolve, leveraging its latest features promises to propel performance to new heights, enriching user experiences across diverse applications.

Maximizing memory efficiency with scalar quantization

Memory consumption has long been a concern for efficient vector database operations, particularly for searching large datasets. Lucene introduces a breakthrough optimization technique - scalar quantization - aimed at significantly reducing memory requirements without sacrificing search performance.

Consider a scenario where querying millions of float32 vectors of high dimensions demands substantial memory, leading to significant costs. By embracing byte quantization, Lucene slashes memory usage by approximately 75%, offering a viable solution to the memory-intensive nature of vector search operations.

For quantizing floats to bytes, Lucene implements Scalar quantization a lossy compression technique that transforms raw data into a compressed form, sacrificing some information for space efficiency. Lucene's implementation of scalar quantization achieves remarkable space savings with minimal impact on recall, making it an ideal solution for memory-constrained environments.

Lucene's architecture, consisting of nodes, shards, and segments, which facilitates efficient distribution and management of documents for search. Each segment stores raw vectors, quantized vectors, and metadata, ensuring optimized storage and retrieval mechanisms.

Lucene's vector quantization adapts dynamically over time, adjusting quantiles during segment merge operations to maintain optimal recall. By intelligently handling quantization updates and re-quantization when necessary, Lucene ensures consistent performance while accommodating changes in data distribution.

Example of merged quantiles where segments A and B have 1000 documents and C only has 100.

Experimental results demonstrate the efficacy of scalar quantization in reducing memory footprint while maintaining search performance. Despite minor differences in recall compared to raw vectors, Lucene's quantized vectors offer significant speed improvements and recall recovery with minimal additional vectors.

Recall@10 for quantized vectors vs raw vectors. The search performance of quantized vectors is significantly faster than raw, and recall is quickly recoverable by gathering just 5 more vectors; visible by quantized@15.

Lucene's scalar quantization presents a revolutionary approach to memory optimization in vector search operations. With no need for training or optimization steps, Lucene seamlessly integrates quantization into its indexing process, automatically adapting to changes in data distribution over time. As Lucene and Elasticsearch continue to evolve, widespread adoption of scalar quantization will revolutionize memory efficiency for vector database applications, paving the way for enhanced search performance at scale.

Achieving seamless compression with minimal impact on recall

To make compression even better, we aimed to reduce each dimension from 7 bits to just 4 bits. Our main goal was to compress data further while still keeping search results accurate. By making some improvements, we managed to compress data by a factor of 8 without making search results worse. Here's how we did it.

We focused on keeping search results accurate while making data smaller. By making sure we didn't lose important information during compression, we could still find things well even with less detailed data. To make sure we didn't lose any important information, we added a smart error correction system.

We checked our compression improvements by testing them with different types of data and real search situations. This helped us see how well our searches worked with different compression levels and what we might lose in accuracy by compressing more.

Comparison of int4 dot product values to the corresponding float values for a random sample of 100 documents and their 10 nearest neighbors.

These compression features were created to easily work with existing vector search systems. They help organizations and users save space without needing to change much in their setup. With this simple compression, organizations can expand their search systems without wasting resources.

In short, moving to 4 bits per dimension for scalar quantization was a big step in making compression more efficient. It lets users compress their original vectors by 8 times. By optimizing carefully, adding error correction, testing with real data, and offering scalable deployment, organizations could save a lot of storage space without making search results worse. This opens up new chances for efficient and scalable search applications.

Paving the way for binary quantization

The optimization to reduce each dimension to 4 bits not only delivers significant compression gains but also lays the groundwork for further advancements in compression efficiency. Specifically, future advancements like binary quantization into Lucene, a development that has the potential to revolutionize vector storage and retrieval.

In an ongoing effort to push the boundaries of compression in vector search, we are actively working on integrating binary quantization into Lucene using the same techniques and principles that underpin our existing optimization strategies. The goal is to achieve binary quantization of vector dimensions, thereby reducing the size of the vector representation by a factor of 32 compared to the original floating-point format.

Through our iterations and experiments, we want to deliver the full potential of vector search while maximizing resource utilization and scalability. Stay tuned for further updates on our progress towards integrating binary quantization into Lucene and Elasticsearch, and the transformative impact it will have on vector database storage and retrieval.

Multi-vector integration in Lucene and Elasticsearch

Several real world applications rely on text embedding models and large text inputs. Most embedding models have token limits, which necessitate chunking of longer text into passages. Therefore, instead of a single document, multiple passages and embeddings must be managed, potentially complicating metadata preservation.

Now instead of having a single piece of metadata indicating, for example the first chapter of the book “Little Women”, you have to index that information data for every sentence.

Lucene's "join" functionality, integral to Elasticsearch's nested field type, offers a solution. This feature enables multiple nested documents within a top-level document, allowing searches across nested documents and subsequent joins with their parent documents. So, how do we deliver support for vectors in nested fields with Elasticsearch?

The key lies in how Lucene joins back to parent documents when searching child vector passages. The parallel concept here is the debate around pre-filtering versus post-filtering in kNN methods, as the timing of joining significantly impacts result quality and quantity. To address this, recent enhancements to Lucene enable pre-joining against parent documents while searching the HNSW graph.

Practically, pre-joining ensures that when retrieving the k nearest neighbors of a query vector, the algorithm returns the k nearest documents instead of passages. This approach diversifies results without complicating the HNSW algorithm, requiring only a minimal additional memory overhead per stored vector.

Efficiency is improved by leveraging certain restrictions, such as disjoint sets of parent and child documents and the monotonicity of document IDs. These restrictions allow for optimizations using bit sets, providing rapid identification of parent document IDs.

Searching through a vast number of documents efficiently required investing in nested fields and joins in Lucene. This work helps storage and search for dense vectors that represent passages within long texts, making document searches in Lucene more effective. Overall, these advancements represent an exciting step forward in the area of vector database retrieval within Lucene.

Wrapping up (for now)

We're dedicated to making Elasticsearch and Lucene the best vector database with every release. Our goal is to make it easier for people to search for things. With some of the investments we discuss in this blog, there is significant progress, but we're not done!

To say that the gen AI ecosystem is rapidly evolving is an understatement. At Elastic, we want to give developers the most flexible and open tools to keep up with all the innovation—with features available across recent releases until 8.13 and serverless