For new users, this is the fastest way to get high-quality vector search running. If you already use Elasticsearch, the new offering is vector search on the platform where your data already lives, with no new system to adopt. Elasticsearch Vector Database supports a range of scenarios, from grounding a large language model (LLM), to giving an AI agent retrieval and memory, to serving hundreds of billions of vectors. Spin up a new project and get started in minutes.

One engine, every vector use case

Elasticsearch Vector Database is built for anyone building applications using vectors:

• RAG: Retrieve the right context for your LLM with dense and sparse vector retrieval, or go with hybrid search combining both vector and lexical retrieval. The quality of your generation improves with the quality of your retrieval.

• AI agents: Give agents fast, filtered retrieval over documents and conversation memory, with the low latencies that multistep agent loops demand.

• Semantic search: Match on meaning, not keywords, with one field type and zero pipeline code.

• Recommendations and similarity: Find nearest neighbors across products, images, or whatever content you have, at scale.

Everything your vector workload needs, optimized out of the box

Building a vector-based application means wiring together several separate pieces: setting up and hosting embedding models, indexing your documents through them, storing the vectors efficiently, applying the embedding model to each query, matching against the vector store, and finally, retrieving the documents behind the matches. Elasticsearch Vector Database handles all of it for you, with no additional configuration or setup.

Vector indexing with vectordb_document index mode

vectordb_document index mode, a new index configuration purpose-built for vector-first workloads, is on by default, so you get the settings that experts would choose. Here's what it turns on:

• bfloat16 by default: Vectors are stored at half the size of float32 with negligible impact on recall, cutting your disk footprint roughly in half before quantization even enters the picture.

• Source vectors excluded: In Elasticsearch, your embeddings already live in the index structures used for search; keeping a second raw copy in _source just inflates storage and slows down fetching results. We exclude the duplicate so responses return faster and you store less.

• The right files preloaded into cache: The data structures that vector queries touch first are warmed into memory ahead of time, so your first (and your thousandth) query is lightning fast.

• Parallel merging: Merging consolidates segments into better-organized vector structures, which lifts both recall and latency, and running those merges multi-threaded means you get there faster.

Vector storage, compression, and auto-tuning

• Your vectors are compressed automatically. Better Binary Quantization (BBQ) shrinks vector memory footprints by up to 32x while preserving recall, and DiskBBQ reduces memory requirements further for large-scale workloads. 

• Opt in to auto-calibration, which tunes each segment's quantization to your data and retunes on every merge as data drifts. When tested across 18 datasets, queries per second (QPS) improved by an average of 16.7%, with recall gains in most of them.

Embeddings on managed GPU inference

• Generate embeddings with native Jina AI embedding and reranking models, or bring third-party models, all on managed GPUs via Elastic Inference Service (EIS) with no model servers to operate. Or self-host, if you prefer your own.

• The semantic_text field type automatically handles chunking and embedding, along with querying, the simplest path to semantic search in the market. 

Hybrid search and filtered vector search

• Hybrid search is built in, combining full-text and vector retrieval in a single query. Blend the results with reciprocal rank fusion (RRF) or any other blending mechanism you want. Vector search is usually the hardest part of hybrid search to configure well. With Elasticsearch Vector Database, you have it handled, and your whole hybrid stack gets better. 

• With filtered vector search, apply metadata filters as part of vector retrieval itself and not as an afterthought that wrecks recall.

Enterprise on day one

You also get role-based access control (RBAC), audit logging, and the compliance certifications that pure-play vector databases generally lack.

Affordable at scale and predictable

Elasticsearch Vector Database is built to stay affordable as you grow: BBQ and DiskBBQ compression that keeps storage linear and memory low means scaling to hundreds of billions of vectors doesn't blow up your bill. And what you do pay is built from numbers you already know: how much data you store and how much you index, along with how much search capacity you need. Estimate your document count and vector dimensions, plus your query load, and you can work out what you'll pay before you create the project. You can also understand your bill line by line at the end of the month. There are no opaque compute units and no surprise charges for background operations.

How to get started with Elasticsearch Vector Database

Create a serverless vector database project

Create a new serverless Vector Database project in Elastic Cloud. Point your data at the endpoint, and you're ready to index.

Create an index using semantic_text

Vector index mode handles the vector configuration. Using semantic_text means that embeddings and chunking setup are managed for you, as is index setup, on managed GPU inference, with no embedding pipeline to build.

Ingest documents

Index text, and the embeddings are generated for you.

Run a semantic search query

Query the same semantic field you just created:

And you get results back:

Semantic search is just the start. Run fully textual queries or combine both into hybrid queries. You can even craft your own vector queries for full control. Follow the semantic search quickstart in the docs for the full instructions.

What's next for vector search in Elasticsearch

We're already working on the next improvements:

• Better multi-tenant handling: If your data needs to stay separated per tenant, we'll give you a way to do it faster and with less code.

• Automatic index optimization: From "brand new index" to "fully optimized," with as little tinkering as possible.

• Continuous infrastructure improvements: Ongoing tuning of Vector Database's settings and infrastructure so you're always getting the best throughput and fastest responses.

Try Elasticsearch Vector Database on Elastic Cloud Serverless

Go from an empty project to a hybrid, filtered vector query in minutes, with production-grade defaults doing the tuning for you. Build fast, scalable AI apps, not infrastructure.

Start on Elastic Cloud Serverless, or dive into the full documentation and API reference. You can also access the new offering on AWS Marketplace, Google Cloud Marketplace and Microsoft Marketplace.

Elasticsearch supports a diverse range of use cases, including observability metrics and logs and complex geospatial analysis. However, as vector search becomes a core component of modern architectures, the need for specialized optimization has grown. Achieving peak performance for vector-heavy workloads often requires navigating a complex space of configuration knobs. To reduce this operational toil, we wanted to provide opinionated, high-performance defaults through a single setting that simplifies performance tuning for production environments. The vectordb_document index mode is designed specifically for optimal vector search workloads.

How to set up vectordb_document mode

Setup is a single setting. When you’re creating an index, define the following in the index settings:

vectordb_document mode is available on all Elasticsearch subscription tiers, including Basic.

Most indices used for vector search also support other operations, such as aggregations or geo-search. They also support hybrid search. Because vector search is typically the most computationally demanding part of these mixed workloads, we recommend using the vectordb_document index mode to prioritize performance for your most intensive operations. An index in vectordb_document mode remains highly capable, supporting almost all operations available in the default standard mode, while optimizing specifically for the resource-heavy demands of vector search.

The “_document” suffix represents our roadmap. We're also developing a “vectordb_columnar” mode as another way to optimize vector search, suited to different data and access patterns.

Elasticsearch vector search benchmarks across four datasets

To validate these defaults, we performed extensive benchmarking across various datasets and two index types: bbq_hnsw and bbq_disk. We ran all benchmarks on a single-node Elasticsearch instance on AWS using a c8gd.2xlarge instance (Graviton 4, ARM64, local NVMe SSD) with a pod limited to 8GB RAM (2GB heap) and 4 CPUs, using a single shard.

Datasets:

Dataset

Vectors

Dims

Use case

laion-img-emb-512-20M-cosine

20 million

512

Pure vector search (low dim)

msmarco-v2-10M-jina-v5-1024

10 million

1024

Pure vector search (med dim)

dbpedia-openai-1M-3072-angular

1 million

3072

Pure vector search (high dim)

arxiv-for-fanns-large

2.7 million

4096

Filtered search

bbq_hnsw: HNSW index performance with vectordb_document

Query throughput and recall

On all four datasets, vectordb_document produced a better QPS–recall curve, and the shape of the advantage is itself informative. On dbpedia-openai-1M, msmarco-v2-10M, and arxiv-for-fanns-large, the curves start close together at low recall (essentially identical on arXiv) and separate as recall rises, reaching roughly 1.4×, 2.2×, and 2× at the high-recall end. On laion-img-emb-512-20M, the curves are apart from the start and settle at about 2× from recall 0.80 upward.

The gap widens with recall because higher recall is bought with oversampling, and oversampling is exactly where vectordb_document saves. The two effects compound. bfloat16 storage halves the bytes read per rescored candidate. More importantly for hierarchical navigable small world (HNSW), the oversample factor is applied to the graph search itself;  each segment is searched for k × oversample candidates, so cost scales with oversample times segment count. Unthrottled parallel merging leaves fewer, larger graphs (14–21 segments versus 22–29), and the graph and quantized vector files are preloaded into the filesystem cache, so vectordb_document pays far less for each increment of oversample.

Comparing identical search settings, rather than equal recall, makes the effect explicit. With rescoring switched off, the two modes are within 1%–28% of each other; at oversample 5, vectordb_document is 2–4× faster; at oversample 10, up to 10×. Those high-oversample settings sit off the QPS–recall frontier, which is why the curves above top out nearer 2×, but they isolate where the saving comes from.

Oversample

DBpedia

LAION

MS MARCO

arXiv

off

1.23×

1.28×

1.11×

1.01×

5

4.07×

2.66×

2.66×

2.05×

10

10.5×

2.33×

2.46×

4.03×

Indexing speed and merge behavior 

In our benchmarks, vectordb_document increased upload time by roughly 16%. This is expected: merges now run unthrottled and parallelized across threads, so they compete with indexing for CPU while documents are still being ingested. Measured indexing time rose 9%–26% across the four datasets. HNSW graph construction is CPU-bound, so that contention is felt directly.

The same changes make the post-upload phase much cheaper. Disabling auto-throttling removes merge rate limiting entirely (in default mode, DBpedia spent 57% of its merge time throttled), and bfloat16 halves the raw vector data, cutting total bytes merged by 42%–49%. Post-ingest merging finished 75%–86% sooner on three of the four datasets, which brings total time to a search-ready index to only about 6% above the baseline. DBpedia was the exception in the other direction: Its merge tail dominated, so total time actually fell 25%.

bbq_disk: disk-based vector search performance with vectordb_document

bbq_disk query throughput and recall

For bbq_disk indices, vectordb_document mode's impact on search throughput varied by dataset. On the lower-dimensional datasets, QPS was essentially unchanged at equal recall: laion-img-emb-512-20M-cosine (512 dims) and msmarco-v2-10M-jina-v5-1024 (1024 dims) track each other closely across the recall range, with vectordb_document ahead in the lower recall end and a few percent behind at the high-recall end. On the higher-dimensional datasets, we saw a consistent gain of roughly 20% at equal recall: dbpedia-openai-1M-3072-angular (3072 dims) and arxiv-for-fanns-large (4096 dims).

Our interpretation is that this is mainly a rescoring effect. vectordb_document stores vectors as bfloat16, so each rescored candidate reads 2× dims bytes instead of 4× dims. The sweep supports this directly: The advantage grows with the query-time oversample factor, which is exactly what sets how many candidates get rescored. On DBpedia, the QPS ratio rises from 1.16× at oversample 3 to 4.3× at oversample 8, and on arXiv from 1.20× to 1.56×, while on laion and MS MARCO, it stays flat or drifts just below 1. Rescoring is simply a much larger share of the query on the smaller, higher-dimensional datasets; on the 10 million and 20 million ones, scanning 1-bit posting lists dominates.

Indexing speed and merge behavior 

Across the four datasets, vectordb_document cut total time to a fully merged, search-ready index by about 20%. Every dataset improved, from 6% on msmarco-v2-10M to 50% on dbpedia-openai-1M, where the post-upload merge phase alone fell from 322 seconds to 54 seconds. Upload time on its own is less clear-cut: On the day charted below, it finished 2%–15% sooner, depending on dataset, but on the earlier run set, it was marginally slower, so we read upload as unchanged to modestly faster and treat time-to-searchable as the real result.

The gain is almost entirely in merging. In default mode, Elasticsearch rate-limits how fast merges may write, and that limiter was binding hard: 75% of all merge time on DBpedia and 73% on laion was spent paused by it. vectordb_document disables the limiter, so paused time is zero and merge time drops 63% on DBpedia and 42% on laion. bfloat16 helps for the same reason: The limiter meters bytes written, so halving the raw vector data means less to write under the cap. The gradient across datasets follows the throttling rather than the byte count: arXiv, at 21% of merge time throttled, saw 30% less merge time, while MS MARCO, never throttled, got 55% fewer bytes but only 11% less merge time.

The two index types respond differently on ingest because merging costs something different in each. Merging bbq_hnsw segments means rebuilding HNSW graphs: CPU-bound work that competes directly with the equally CPU-bound graph construction on incoming documents, and on a 4-CPU pod that contention surfaces as a slower upload. bbq_disk merges are dominated by writing bytes rather than by CPU, so lifting the rate limiter spends I/O bandwidth that the local NVMe has to spare, and bfloat16 means that there are fewer bytes to write in the first place. Both index types reach a fully merged index far sooner; the difference is only whether the upload phase pays for it.

What vectordb_document sets under the hood

element_type (dense_vector)

• Value: bfloat16.

• Impact: Stores each dimension of the raw vector as bfloat16 instead of the default float32, halving the storage of raw vectors with negligible impact on recall.

• Benefit: Lower disk footprint (reduced total cost of ownership [TCO]), faster fetching of vectors for rescoring.

Dynamic float array mapping

• Value: Float arrays with 32 or more values are dynamically mapped as dense_vector. (In the default mode, this threshold is 128.)

• Impact: No need to explicitly declare the field as a dense vector field; the system identifies it automatically.

• Benefit: Configuration simplicity.

exclude_source_vectors

• Value: true.

• Impact: Vectors are stored once in the vector index and not duplicated in _source. They’re omitted from _source in responses, but they can still be retrieved on request.

Benefit: Lower disk footprint; faster queries, since large vectors are no longer shipped with every _source fetch.

index.store.preload

• Value: ["vex", "veq", "veb", "cenivf"].

• Impact: Preloads search-time vector data structures into the filesystem cache whenever new segments are opened.

• Benefit: Reduced query latency.

index.merge.intra_merge_parallelism_enabled

• Value: true.

• Impact: Use parallel threads for segment merging to achieve optimal segment sizes.

• Benefit: Faster convergence to fewer, larger segments leads to better recall, lower query latency.

index.merge.scheduler.auto_throttle

• Value: false.

• Impact: Allows merges to proceed at full speed.

• Benefit: Merges reach optimal segment sizes sooner, resulting in better recall, lower query latency.

While these defaults are optimized for the majority of use cases, most can be overridden individually to suit unique hardware constraints or extreme performance requirements, with one exception: exclude_source_vectors: true is locked in and cannot be changed on a vectordb_document index.

Summary

The vectordb_document index mode is a production-ready foundation for vector search: By adopting high-performance defaults, teams can focus on building features rather than on hand-tuning merge or storage settings or on adjusting preload settings. 

Across four datasets, the picture is consistently favorable, though the balance differs by index type. With bbq_hnsw, QPS recall improved on every dataset, from roughly on par at low recall to as much as double at the high-recall end, in exchange for a modest ingest cost of about 16% longer upload and 6% longer time to a search-ready index. With bbq_disk, the trade runs the other way: Total time to a search-ready index dropped by about 20%, while search improved by about 20% at equal recall on the higher-dimensional datasets (3072 and 4096 dims) and was essentially unchanged on the lower-dimensional ones. The split comes down to what merging costs in each: Rebuilding HNSW graphs is CPU work that competes with indexing; whereas bbq_disk merges are write-bound and simply run faster once they’re unthrottled.

In both cases, the new defaults move the system in the direction that most users want, with no per-index tuning required. And, for the settings that genuinely depend on the data itself, such as the degree of quantization, auto-calibration now derives them for you. (See How Elasticsearch auto-tunes vector quantization to hit your recall target.)

Try it out by creating an index with the vectordb_document index mode in Stateful Elasticsearch 9.5 or in Serverless.

How a search is executed in Elasticsearch

Let’s start with some Elasticsearch basics. When a search request lands on an Elasticsearch node, that node is called the coordinating nodefor the search. By default, any Elasticsearch node can act as a coordinating node. The coordinating node determines which shards need to be searched based on the index or indices specified by the search. The nodes that those shards live on are called data nodes. A coordinating node can also be a data node, as it may host shards that are relevant to the search.

Elasticsearch performs the search in two primary phases: the query phase (also known as the scatter phase) and the fetch phase(also known as the gather phase). The query phase is responsible for going to the data nodes and executing the query on each shard. Each shard responds with a set of document IDs (just the IDs, no data) and an associated score for each. These results are reduced. Next, the fetch phase goes back out to the data nodes to fetch the document _source (the data). 

What exactly is a reductionin Elasticsearch? A reduction turns per-shard results from the query phase into a single merged result for the client. Suppose a search asks for the top five hits in a three-shard index, according to some relevance score. Elasticsearch must then get the top five hits from each targeted shard. Why? Because it’s possible that one shard contains the global top five, or, more likely, that the top five docs are spread across shards. 

If three shards are being searched, the coordinating node will have 15 (docID, score) pairs after the query phase. These results are reduced: The documents with the top five scores are kept and the rest thrown away. Then the fetch phase reaches back out to the data nodes to get the documents’ _source data, which Elasticsearch then responds with.

shard1_results = [(id: 231, score: 0.871), (id: 445, score: 0.812), (id: 88, score: 0.754), (id: 312, score: 0.701), (id: 567, score: 0.643)]

shard2_results = [(id: 847, score: 0.921), (id: 76,  score: 0.843), (id: 125, score: 0.783), (id: 438, score: 0.729), (id: 590, score: 0.668)]

shard3_results = [(id: 512, score: 0.887), (id: 289, score: 0.798), (id: 74,  score: 0.741), (id: 631, score: 0.682), (id: 405, score: 0.619)]

// Take the top five scores from above (that’s the “reduction”) reduced_result = [(id: 847, score: 0.921), (id: 512, score: 0.887), (id: 231, score: 0.871), (id: 76, score: 0.843), (id: 445, score: 0.812)]

What is the batched query phase?

Without batching (shard fan-out)

To understand the batched query phase, we must first understand how the query phase worked without batching. The following diagram represents a three-node Elasticsearch cluster with 12 index shards. Suppose a client search request lands on Node 1. That makes Node 1 the coordinating node for the search.

The coordinating node first fans out to all shards, performing the client’s query on each shard. Each shard query is facilitated by a transport request, which is a protocol used for communication between Elasticsearch nodes. Notice that in this diagram, each shard gets its own transport request. Node 1 holds shards itself, but no network request is needed to query those shards.

After querying the shards, the coordinating node must reduce them, as described in the previous section. At this point, the coordinating node is holding 12 shard results. Once it reduces them all, it can proceed with the fetch phase and then respond back to the client.

With batching

So what does the batched query phase change? The batched query phase first takes effect before dispatching the shard query transport requests. Now the coordinating node batches the shards it needs to query for each data node and requests them all at once.

Notice that only one transport request was made to Node 2 and one to Node 3. This is the first change in the batched query phase. Each transport request induces some overhead, so this change alone buys us our first performance improvement, reducing latency and CPU cycles spent on overhead.

The second change in the batched query phase has to do with reducing shard results. Now partial reductions occur on the data nodes themselves. For instance, after querying shards 2, 5, 8, and 11, Node 2 then reduces those results. The coordinating node, upon receiving partially reduced results from Node 2 and Node 3, must then perform afinal reduction. This is the second major enhancement we get from the batched query phase: the spreading out of reduction work across the data nodes. This reduces memory pressure on the coordinating node, which we’ll see measured later in the benchmarking section.

The benefits of batching

Fan-out (no batching) is how search worked for a very long time. It has advantages: each shard is a separate request that returns and can be retried independently from all the others. With many shards involved, though, each shard request causes overhead due to many round trips going between the coordinating node and the data nodes.

Also, the coordinating node doesn’t have enough information to be able to determine the pace at which to send requests to each data node. It sends a maximum of five concurrent requests per data node by default, where five is a bit of a magic number which allows for some parallelism, while at the same time preventing a single query from taking over an entire data node. At the same time, data nodes also don't make distinctions between the different shard requests they receive, for instance based on what parent search request they belong to. In reality, if all shard requests involving a search request are presented in one batch to each data node, the data node can then look at its internal state and adapt its pace dynamically, removing the five concurrent shard requests artifical limit mentioned above.

The batched query phase results in several benefits, including:

• Increased efficiency, thanks to fewer round trips: less CPU spent on transport overhead and fewer bytes going through the transport layer.

• Spreading out the load of reductions: the coordinating node was previously the bottleneck for reductions, and now data nodes share the work.

• Better resource usage: we may be able to better max out the data node’s CPUs.

The average search against many shards can now be served much quicker, with lower latency and higher throughput.

Batched query phase benchmarks: Latency and memory usage

To measure the benefits of the batched query phase, we ran a couple of benchmarks. The first displays the benefit of reduced transport overhead. This benchmark was built off the “many-shards-quantitative” nightly benchmark. It runs on a three-node Elasticsearch cluster, running batches of searches targeting 1,000, 5,000, and 20,000 shards. The queries are match_all queries with size: 0. That means querying the shards themselves is effectively a no-op. This benchmark is meant to isolate the work of coordinating the search across the cluster.

We ran the same benchmark with and without the batched query phase (by toggling the cluster setting search.batched_query_phase). At 5,000 shards, the batched query phase makes searches twice as fast. By 20,000 shards, it’s 2.2x faster. These results are a ceiling for what a realistic workload can expect to benefit; any real query work at the shard level will dilute the overall result. However, the gains are real, and the coordination work of your queries will benefit as shown. Note that this first benchmark did not attempt to measure any gain brought by spreading reductions across data nodes, as opposed to performing them only on the coordinating node.

Next, we benchmarked the benefits of the batched query phase on large reductions. In our benchmark, we ran a large terms aggregation over a data set called http_logs (which can be found in our rally-tracks repo). This data set has 247 million documents, which we indexed in seven indices each with 100 shards, again in a three-node Elasticsearch cluster. We ran a single terms aggregation query for the clientip field with size: 1000 and shard_size: 50000. That means we’re asking for the top 1,000 terms, although each shard will return 50,000 buckets to be reduced to that 1,000. Requesting so many terms from each shard increases precision, but has a cost in terms of memory usage and reduction overhead, which helps highlight the gain provided by spreading incremental reductions across data nodes.

The memory used is measured by something called a circuit breaker, which accounts for memory usage in Elasticsearch. This field can be found in the /_nodes/stats response under nodes.<node_id>.breakers.request.estimated_size_in_bytes. It estimates the memory in use for processing in-flight search requests.

The memory benchmark shows how the work of reductions is now spread across the cluster. In blue is the memory used with search.batched_query_phase: false. We can see that the node elasticsearch-0 is the coordinating node, as it bears the entire reduction load itself. This effect is no longer the case with search.batched_query_phase: true in red. The node elasticsearch-1 is the coordinating node, but it uses far less memory. That’s because the data nodes elasticsearch-0 and elasticsearch-2 do reductions themselves, reducing the results before they’re returned to the coordinating node.

Summary

Batching shard queries per data node reduces transport overhead and allows each data node to perform partial results reduction locally. The result is lower latency for searches targeting many shards and better distribution of work across the cluster for reduction-heavy workloads. At the same time, the new model works well for all scenarios and opens the door for potential future enhancements which we’re excited to pursue, such as:

• Optimize out-of-the-box resource usage by introspecting data nodes' activity and adjusting the pace accordingly, which would replace the five concurrent shard requests per data node artifical limit.

• Assign priorities to search requests and let each data node process shard requests accordingly.

• Reduce the number of roundtrips further by folding the `can_match` phase into the query phase. Can match is a separate "batched" roundtrip used to shortcut the query against shards that can't possibly match based on index statistics. Now that the query phase is batched, both rounds can be executed in one go.

• Discontinue support for minimize roundtrips in cross-cluster searches in favour of batched query execution; in hindsight, minimize roundtrips achieves a similar goal but is specific to cross-cluster execution, while batched execution is applicable to every search. 

Why dense vectors create off-heap memory pressure that heap charts miss

Semantic search and kNN rely on dense_vector fields. Elasticsearch stores much of that data in off-heap memory. It’s related to how the Java Virtual Machine (JVM) operates, but it isn’t the same thing as heap usage.

In production, the heap versus off-heap split shows up in a familiar pattern:

• Heap looks fine for weeks, while the dense vector off-heap footprint quietly grows.

• Heap circuit breakers stay quiet or only spike late because the pressure sits outside the JVM.

• kNN search and bulk ingest slow down, queues build, and nothing on the dashboard points at vector RAM as the cause.

Heap limits protect Java allocations. They don’t tell you whether vector off-heap footprint still fits the RAM envelope that your deployment actually runs in. AutoOps already watches cluster health broadly; Vector memory pressure adds a focused read for vector-heavy nodes when memory and load signals line up.

How AutoOps measures vector RAM, heap, and headroom

AutoOps works from the same node stats metrics you already use for Stack Monitoring. For each node, it tracks three derived numbers:

Symbol

Meaning

Source (typical)

Chart (see below)

V

Vector off-heap footprint

indices.dense_vector.off_heap.total_size_bytes

First, green line

A

Available RAM in the product view

Delta between os.mem.total_in_bytes and os.mem.used_in_bytes

Second, green line

H

Headroom

A − V (headroom_bytes)

First, blue line

H > 0 means there’s a modeled runway: Vector use still fits comfortably in that accounting. H ≤ 0 means that you’re in a compression regime: Vector footprint (V) meets or exceeds the free RAM (A) picture that AutoOps can align in telemetry. On small tiers, that can be common under load. The insight emphasizes trends, growth in vector off-heap footprint, and corroborating stress, not a single negative snapshot.

AutoOps also tracks a compression regime flag (fraction of recent samples where H ≤ 0), so brief flickers don’t dominate the story (see third chart below):

How vector memory pressure detection works: Expansion and compression

Vector memory pressure is a single HIGH severity AutoOps event, which adapts to the compression regime:

• Expansion (H > 0): Emphasize shrinking headroom, hence a growing vector off-heap footprint.

• Compression (H ≤ 0): Emphasize ΔV, heap context, I/O, and latency. "Hours until H hits zero" isn’t the main narrative when headroom is already gone.

The detector requires three layers before it fires:

1. Memory carriers: Compression regime, shrinking headroom, or sustained growth in vector off-heap footprint.

2. Operational latch: Search or indexing latency versus rolling baselines, filesystem read stress (paired with latency or heap), indexing throttle, thread-pool queues or rejections, segment creep, or heap circuit breakers when paired with other stress, as circuit breakers alone don’t provide enough evidence to be escalated without corroborating stress.

3. Heap hot: Heap usage elevated versus a 24-hour rolling median on that node, so compression alone on a calm heap doesn’t fire the insight.

That pairing is intentional. Vector pressure without load might be capacity planning, and load without vector pressure might be a different root cause. Together, vector memory, operational stress, and heap heat surface the vector RAM story when the node is actually in trouble, not on every compressed mapping while the heap stays normal.

Validation: Memory pressure detection on a 4 GiB node under kNN load

We stress-tested vector memory pressure detection on 4 GiB Elastic Cloud Hosted deployments with throttled dense-vector ingest (~2,000 docs per minute) and steady kNN search (~8 queries per second). Across Hierarchical Navigable Small World (HNSW), Better Binary Quantization (BBQ) HNSW, and DiskBBQ mapping profiles over 24–48 hours:

• Vector off-heap footprint grew from near zero to about 6 GiB on the tightest runs (more than 4.7 million vectors indexed) in both HNSW test runs.

• Nodes spent most of each run in compression (H ≤ 0), which is expected when vector footprint exceeds total RAM in this model.

• Vector memory pressure stayed off while heap held near 50%, even with compression and pool stress building.

• On both HNSW and BBQ HNSW, the insight fired once heap climbed past ~75% with memory compression and thread-pool queue stress, roughly an hour before heap neared saturation. Node out of memory (OOM) and circuit breakers followed in the same window, as did slow search/indexing. As we can see on the dashboard below, performance drops drastically due to corroborating stress toward the end of the test run:

• On DiskBBQ, where compression was chronic but heap stayed normal, the insight didn’t fire,since storage rather than memory was the limiting factor. Disk and watermark signals are the right path to follow for that profile. As we can see on the screenshot below, all indicators stayed steady with constant performance throughout the test, even though we filled up the disk with more than 70 million vectors on the same instance type:

That timing is the point: Operators get a vector-first read tied to real RAM stress, with subsystem context, rather than an alert on every compressed index or only a red heap chart after the node is fighting on every front.

What to do when AutoOps raises vector memory pressure

Here’s the insight that AutoOps now raises when it detects vector memory pressure:

Recommendations in the product map to concrete actions:

1. Reduce vector footprint where quality allows: Fewer dimensions, quantized mappings, archive or split indices, reindex with a leaner mapping.

2. Tune kNN load: Lower num_candidates, reduce concurrent query rate, narrow filtered kNN where possible.

3. Consider DiskBBQ when HNSW in RAM is the bottleneck (evaluate recall/latency trade-offs for your use case). If you’re already on DiskBBQ and the heap memory is calm, treat disk and watermark insights as the primary signals. Note that DiskBBQ requires an Enterprise license.

4. Right-size RAM when vector off-heap footprint (V) trends up and headroom stays tight.

AutoOps links affected nodes and summarizes regime and stress in plain language. Treat it as “act now, rather than waiting for red on every chart.”

Where AutoOps vector memory pressure monitoring is available

Vector memory pressure is available wherever AutoOps runs against Elasticsearch 9.2+, including:

• Elastic Cloud Hosted (ECH).

• Elastic Cloud Serverless (coming soon).

• Self-managed via Cloud Connect.

AutoOps is included at all subscription levels for supported deployment types and doesn’t consume ECUs on ECH.

The palette takes shape: how multimodal search in Elasticsearch evolved from semantic_text

The semantic field is a convergence of several complementary features we've introduced over the past couple of years, bringing them together to create a cohesive multimodal search experience. Each solved an important piece of the semantic search puzzle on its own; together they enable native multimodal search.

The first brushstroke was semantic_text. Before it, running semantic search meant manually configuring mappings, wiring up ingest pipelines with an ML model, manually chunking content, and generating query-time embeddings yourself. The semantic_text field folds all of that away: it performs inference automatically at ingest time, chunks long documents for you, and simplifies the queries you write against it. Introduced in Elasticsearch 8.15 and released as GA in Elasticsearch 8.18, it has become the foundation for semantic search on the platform.

Next came the model to power multimodal search. jina-embeddings-v5-omni is our family of multimodal embedding models, capable of embedding text, images, video, audio, and PDFs into a shared vector space. Because those embeddings are semantically compatible across modalities, you can store diverse media in a single index and query across all of it at once, such as retrieving an image via a text description or matching audio against a written phrase, all without maintaining a separate pipeline for each content type. For more detailed information about how these embeddings are generated, see the model documentation.

We added the embedding query vector builder in Elasticsearch 9.4 to handle multimodal inputs at query time. Query vector builders are general-purpose tools you can use to convert input to a vector at query time as part of your request. For example, we have the text_embedding query vector builder for text-only models and input, and the lookup query vector builder for getting a vector from an existing document. The embedding query vector builder is a new type that works with multimodal models and accepts multimodal input, including text or base64-encoded binaries. This allows you to pose a query in whatever modality fits, and Elasticsearch generates the matching vector on the fly.

The final piece was multimodal ingest. The semantic_text field brought automatic embedding to text; the semantic field extends that same automatic experience to images, audio, video, and PDFs from ingest through query.

Painting the picture: creating an index with the semantic field

Let’s create an index with a semantic field. This is as simple as setting the field type to semantic and defining the inference endpoint you want to use:

In this example, we use the .jina-embeddings-v5-omni-small inference endpoint. This is our built-in jina-embeddings-v5-omni inference service, and it is available in all environments with access to the Elastic Inference Service (EIS). This includes:

• Serverless.

• Elastic Cloud Hosted (ECH).

• Self-managed with Cloud Connected Mode (CCM).

Indexing images, audio, video and PDFs

To index an image, provide an object with a type of image and a value containing the image as a base64-encoded data URL:

Arrays of objects are also accepted, allowing you to index multiple images in a single field value:

The semantic field also supports text values, just like semantic_text. You can provide such values standalone or intermix them with image values:

Text values are handled just like they are with semantic_text: long passages are chunked according to the chunking settings configured on either the inference service or field mapping. Multimodal values, such as images, are not chunked. Each multimodal value is represented as one chunk.

Other modalities are supported as well. Change the type value to match your content’s modality. Currently we support:

• image

• audio

• video

• pdf

For example, to index a video, the request would look like:

Image search and cross-modal retrieval with a text query

To find multimodal content using a text description, run a match query on the semantic field:

Just like with semantic_text, Elasticsearch automatically generates an embedding for the query text using the inference endpoint associated with the field. That query embedding is used to return semantically similar matches.

This query pattern enables easy text-to-image search. Just index an image and use a match query to retrieve it via text description! It also works for any other modality: index the multimodal input and search by description to retrieve it.

Querying with images, video, and other multimodal inputs

We can also search using a multimodal input by using the knn query with an embedding query vector builder. For example, we can search using an image:

The input object format is the same as when providing an image to index: set the type to image and value to a base64-encoded data URL.

Similar to when querying by text description, Elasticsearch automatically generates an embedding for the query image using the inference endpoint associated with the field. That query embedding is used to return semantically similar matches.

Just like with indexing, other modalities are supported, but are limited to those supported by your inference endpoint. For example, a search using a video clip would look like:

Extending the composition: highlighting, retrievers, and other semantic field features

The semantic field didn't start from a blank canvas. It's built on the same foundation as semantic_text, inheriting its behavior and its ergonomics, and extending them to multimodal content. In practice, that means nearly everything you already know about working with semantic_text carries over unchanged. If you've built with semantic_text before, the semantic field will feel immediately familiar.

Here’s a selection of the features that come along for the ride. See the documentation for a complete list.

Highlighting the best-matching chunks

If you index multiple values in a semantic field, you may want to know which value best matches the query. The semantic highlighter can be used to return the most relevant chunks as highlight fragments:

Setting order to score returns the fragments ranked by relevance, while number_of_fragments caps how many chunks come back. The response looks like:

Note how highlighted multimodal values are represented using their data URLs.

Controlling vector quantisation with index options

The semantic field stores its embeddings in an underlying vector field, and index_options lets you control how that vector field is indexed. For example, choosing a non-default quantization strategy:

Multi-field retrievers

The semantic field participates in the multi-field query format supported by the linear and rrf retrievers. Rather than hand-writing an inner retriever per field, you supply a single query and a list of fields, mixing lexical fields and semantic fields freely:

The retriever automatically separates lexical fields from semantic fields, queries each group, and normalizes the results so that each group contributes equally to the final ranking, preventing lexical matches from drowning out semantic ones.

Cross-cluster search

The semantic field supports cross-cluster search (CCS), enabling use of the field in large, multi-cluster deployments. Simply list the indices to query using the standard <cluster>:<index> format:

The fields queried across indices and clusters can use a mix of different inference endpoints that produce different query embeddings. The search request will automatically apply the proper query embedding to each individual field queried.

Off the easel, into the world: optimising multimodal embeddings for production

When you move multimodal search from experiment to production, the size of your multimodal inputs becomes a practical concern. Multimodal data is supplied as base64-encoded data URLs, and that data is stored in the index. Those strings can grow large in a hurry: a single high-resolution file can balloon into several megabytes of encoded text, which has several side effects:

• The index size on disk can increase significantly.

• Requests and responses containing multimodal data are larger, increasing transmission time and ingress/egress costs.

• Inference on larger multimodal inputs is slower.

The good news is that you don’t need that much fidelity. Multimodal embedding models reduce each input to a compact representation before generating a vector anyway, so a smaller, lower-fidelity version of a multimodal input (such as a downscaled image or a lower-bitrate audio clip) produces a very similar embedding, and similar search quality, to its full-size original. This also applies to PDF input. PDFs are generally processed visually by multimodal models, so the quality only needs to be good enough to perform operations like image embedding and OCR. Long PDFs should be broken up into chunks of smaller inputs, so the embeddings generated more accurately represent each chunk. Feeding the model small inputs keeps your documents lean, trims index and response sizes, and speeds up ingestion, all without meaningfully affecting relevance. 

Elasticsearch reinforces this practice with a guardrail: the indices.inference.max_binary_input_size cluster setting caps the size of each binary input, defaulting to 1 MB. Any individual value that exceeds the limit is rejected with a clear error, so oversized inputs surface as an actionable problem at index time rather than as silent bloat. This setting is adjustable in self-hosted and ECH through the cluster settings API. It is not adjustable in our serverless offering, where 1 MB is the hard limit for binary sizes.

When possible, it is also advised to use source filtering to exclude semantic fields from responses. For example:

This makes responses smaller, more performant, and easier to parse because multimodal data is not returned with each.

Try out the semantic field

The semantic field is available in Elasticsearch 9.5 and Serverless. Start a free trial and try it out today.

In our previous blog, we laid out the statistical model behind that calibration: a manifold model for how nearest-neighbor distances scale with index size, a Gaussian error model for quantization noise, and a closed-form way to combine the two into an expected recall@k for a given rerank depth. If you haven't read it, the one thing you need going in is this: given a candidate quantization encoding and a rerank depth, we can predict recall@k without building an index and benchmarking it, by fitting two small models to a sample of the corpus.

In this post we’ll go through how to score candidate configurations cheaply and how to leverage that to make merge-time decisions that are themselves cheap, correct, and consistent across real, constantly-merging indexes. This led to some pretty impressive improvements: we see almost 17% average improvement in QPS across a broad range of datasets all while increasing recall (in one case by a factor of 3). What’s more you get this immediately by adding one line to your index options, "auto_calibrate": true, and our plan is to make this our default once it has had the chance to bake a bit.

Why manual vector quantization tuning is unreliable

bbq_disk exposes several knobs: quantization bits for documents (1, 2, 4 or 7), a separate bit width for queries, an oversampling factor for reranking, and whether to precondition vectors before quantizing. None of these act independently, and their effect on recall depends on the data: a 4-bit/1-bit encoding might be plenty for one embedding model and clearly insufficient for another. A single index is also built out of many segments, merged over time, each with an eventually different vector distribution. Hand-tuning one configuration for an entire index is, at best, a compromise, which is the motivation for automatic calibration: let each segment have its own configuration, re-evaluated every time it is involved in a merge operation.

How Elasticsearch runs auto calibration at merge time

When a number of segments are merged and automatic calibration is enabled, Elasticsearch samples documents and queries from the vectors being merged and:

1. fits the manifold model over a sequence of nested samples of the merged corpus;

2. fits the error model, predicting the quantization error's standard deviation for each candidate (query bits, document bits, precondition) combination;

3. sweeps candidate configurations in ascending cost order; the candidate encodings are (1,1), (4,1), (4,2), (4,4) and (7,7) (query bits, document bits), each tried across oversampling factors of 1.25, 1.5, 1.75, 2.0, 2.5 and 3.0;

4. estimates recall@10 for each candidate using the model described in our first post, and stops at the first (cheapest) configuration predicted to hit the target of 90% recall@10.

The winning configuration (encoding, oversample factor, precondition flag) is stored directly in the segment's metadata, so it travels with the segment and is picked up automatically at query time unless a request explicitly overrides it.

Small segments skip this altogether: below 10,000 merged vectors, there isn't enough data to fit a reliable model, so Elasticsearch just uses the current DiskBBQ defaults (4-bit query / 1-bit document encoding, no preconditioning, 3x oversampling).

How the vector quantization cost model works

Following the principles described in our first post, we started by picking candidates with three nested loops that are essentially how you might imagine hand jamming a lookup table. Start with the quantization scheme as the outer loop, ordered cheapest to most expensive by document bits ((1,1) → (4,1) → (4,2) → (4,4) → (7,7)). Then we set the rerank depth within the middle loop, ordered shallow to deep (1.25× → 3.0×). Finally we set preconditioning within the inner loop (off → on).

That ordering has a cost model baked into it, it's just implicit rather than written down: exhaust every rerank depth at the current bit tier before ever trying more bits. Document bits were effectively the only resource priced as expensive; oversampling was treated as nearly free by comparison, since the sweep would always max out rerank depth on a cheap encoding before considering a pricier one.

The current implementation replaces that with an explicit, continuous cost function:

Query bits still don't factor into cost at all, only document bits (which drive index size) and rerank depth (which drives how many candidates get rescored per query). Preconditioning also stays outside the formula: Elasticsearch runs the whole cost-ordered sweep once with preconditioning off, and only if nothing meets the recall target does it re-run the sweep with preconditioning on, treating it as a fallback lever rather than something priced bit-for-bit against the other two.

With this cost model, rerank depth costs noticeably more per unit than a document bit, so the sweep will often prefer stepping up a bit tier over pushing oversampling deeper.

The main reason for this is that once you're running in a serverless deployment, compute and storage are billed and scaled independently, on very different clocks. An extra document bit is mostly a one-time, indexing-time cost; it makes the segment marginally bigger on object storage, which is cheap and doesn't need to be pre-provisioned against a spike in query traffic. It does carry a smaller recurring cost too, since quantized vectors sitting in page cache or loaded for scoring take proportionally more RAM per document as bit width grows, but that scales linearly and predictably with corpus size, and doesn't spike with query load. Rerank depth is the opposite: it's a recurring, per-query cost.

Every extra unit of oversample factor means fetching and rescoring that many more full-precision candidate vectors from disk, on every search request, for as long as the index is queried. That's compute and DRAM pressure on the search-serving tier, which has to autoscale in close to real time to match query concurrency. It sits on the hot path of the latency-and-cost budget in a way storage capacity, and the RAM footprint of the bits themselves, does not. Weighting rerank depth higher than document bits in the cost formula is what makes the sweep reflect that asymmetry.

Efficiently estimating vector quantization error

The cost model above works with the premise that the recall estimate behind it is trustworthy. The manifold and error models need to be accurate for the recall assessment to be trustworthy. While the manifold model of the k-th to N-th nearest neighbors distance is cheap to compute, the standard deviation of the quantization noise for a given candidate encoding is a bit more expensive in principle.

DiskBBQ uses fixed count clusters to accelerate nearest neighbor queries. Our quantization procedure takes advantage of this by only quantizing the vector residuals from the cluster centroids. This means as the data scales, the magnitude of vectors we quantize relative to the various components of the similarity calculation shrinks. As such, quantization accuracy increases. We need to account for this when converting our sample estimates to the segment as a whole.

Clustering the corpus at several sample sizes and fitting how the error scales with cluster size requires re-clustering a real sample of the corpus at several different sizes and fitting a regression model to see how the error shrinks as the effective cluster size grows. We also add a conservative +3σ margin on top of the fitted estimate to guard against noise in the fit itself. This is accurate and appropriately cautious; however, while benchmarking on common dense retrieval datasets, we found that performing several hierarchical k-means passes per candidate was expensive.

To speed things up, we tried approximating residuals with a synthetic isotropic-Gaussian formula. Instead of clustering increasing-size samples, this approach generated synthetic residuals from the manifold model's local density estimate. It was fast and fit for background merges, with the full repeated clustering approach reserved for force-merges only. However, it turned out to inflate error when embeddings (residuals) are anisotropic (some directions carry a lot more variance than others). As a result, the estimated error could grow significantly on strongly anisotropic data (e.g., Fashion-MNIST-style image embeddings).

So instead we looked for a still fast but more accurate way of calculating residuals. We opted for using a single clustering pass over a smaller sample (2,048 vectors). The clustering runs once per merge and is then warm-started for every candidate encoding evaluated afterward, instead of re-clustering from scratch each time. To get the error's dependence on corpus size, which the baseline learns by re-clustering at multiple sizes, this approach instead reuses the manifold model's invDim as a plug-in for that dependence, extrapolating from the single real measurement rather than fitting the size relationship separately.

We also trimmed the query sample used during calibration from 1,024 to 256 vectors, on the reasoning that a smaller sample is enough once the error is being measured from real data rather than synthesized (and validated by benchmarks). The net effect was comparable wall-clock cost to the synthetic residual formula it replaced, but grounded in real per-cluster residuals, accurate enough that force-merge and background merge could be unified onto one path.

As an example, we take five different benchmark datasets and calculate the quantization error standard deviation (SD) by directly measuring the gap between exact and quantized dot products on a sample of real (or, for the synthetic residual formula, fabricated) residuals, then extrapolating that measurement to the full corpus size. They differ only in how much sampling and regression goes into that extrapolation: the multi-sample scaling fit sweeps fifteen sample sizes and fits how error scales with cluster size, the single-pass real residual measurement takes one larger real residual sample and reuses the manifold's intrinsic dimension to estimate the size dependency, and the synthetic residual formula skips real residuals altogether and samples from a synthetic Gaussian from the manifold's expected rank distance. We treat the multi-sample scaling fit as ground truth in this comparison because it's the most sample rich of the three, not because it's a zero variance measurement of the "true" corpus-wide error (it has its own sampling noise too). The table below summarises the methods and findings.

Method

How it works

Speed

Accuracy

When used

Multi-sample scaling fit

Clusters at 15 sample sizes, fits regression

Slow

Gold standard

Ground truth baseline

Single-pass real residual

One clustering pass + manifold invDim plugin

Fast

Near gold standard

Background + force merge

Synthetic residual formula

Gaussian from manifold density estimate

Fast

Inflated on anisotropic data

Deprecated

In order to exchange methods, we only need to be confident that they agree. This question can be answered independently of the correctness of the actual estimates, which we verified in our previous post for the multi-sample scaling fit. The figures below report the predicted quantization SD and the predicted recall@10, which is influenced by how we estimate the error. We report the analytical recall the manifold model predicts as a function of the quantization parameters, given the estimated error distribution perturbing the true distance ordering. This way, we isolate the quantization error's effect on ranking from any separate recall loss the IVF index itself might introduce, which is a distinct error.

The single-pass real residual measurement's calculated error SD is closer to the multi-sample scaling fit (our gold standard), with respect to the synthetic Gaussian residuals. Consequently, the predicted recall is closer when using the single-pass + manifold plugin method. Indeed, we found the models to be essentially interchangeable regarding the indexing decisions they lead to. Critically, we lower the calibration overhead by an order of magnitude.

Auto calibration overhead on indexing performance

We compared the cost of auto calibration on indexing, when compared with ES defaults, over 18 public benchmarks. We noticed that more than 50% of the datasets report an auto calibration overhead below 2%. Three datasets report 16-27% overhead, while two datasets sit in the 31-35% overhead.

The merge overhead is larger for smaller datasets (Fashion-MNIST, FiQA) that get indexed in a few seconds; that is expected as the size of the vector samples being used for calibration is fixed and therefore more noticeable with tiny datasets. In fact, for larger datasets like DBPedia-Entity and HotpotQA (5M doc vectors) the overhead is sometimes not noticeable and within 11% in the worst case.

What quantization parameters does auto calibration choose?

Looking at the encoding auto-calibration landed on for each of the real datasets:

Query bits were 4 in every dataset. While query bits aren't priced into the cost formula, we still iterate through lower query bits first (e.g., at 1 bit doc vectors, we first evaluate recall for 1 bit query vectors, then for 4 bit query vectors); so it’s possible for some datasets to even choose symmetric 1-bit quantization. The center of mass is a 2-bit document encoding with somewhere between 1.5x and 1.75x oversampling; 4-bit only shows up for two genuinely harder datasets (Fashion-MNIST's image embeddings, GIST-1M), and 1-bit only for a handful of the text-embedding models that are most robust to quantization. In fact, our own models are among those that quantize best: we selected 1 bit documents for all three corpuses we tested with Jina v3.

Recall and QPS improvements from automatic calibration

Auto-calibration is a broad win across the eighteen datasets: QPS improves in 15 of 18 cases (often substantially, double digits on about ten, and over +50% on FiQA GTE, Fashion MNIST, and Glove-200), and recall improves in 15 of 18 cases too, including a dramatic +295.7% rescue on Fashion MNIST. Most datasets see gains on both metrics simultaneously, and even the more modest cases still land solidly positive, recall improvements are commonly in the high single digits to double digits, QPS gains follow a similar pattern. Where either metric does dip, the drops are small and contained: the three QPS regressions all stay under 1.5%, and the three recall regressions all stay under 2%.

How to enable auto-calibrated vector quantization in Elasticsearch

The feature is not enabled by default for now, and opt-in via auto_calibrate on bbq_disk index options:

With this set, you no longer need to guess at bits, oversampling, or preconditioning: each segment picks the cheapest configuration that's predicted to hit 90% recall@10 for its own vector distribution, and re-evaluates that choice every time it's merged.

What's next for automatic vector quantization in Elasticsearch

Our first post showed that recall could be predicted in closed form from a small sample. Turning that into something running inside a real merge path meant a second round of engineering decisions that the model itself doesn't answer: how to order a sweep over candidates so it's cheap in the common case, how to price oversampling against document bits given how each is actually paid for at query time, and how to estimate the error term itself cheaply without quietly wrecking its accuracy.

In the end, we have a feature that allows us to tailor indexing choices to the data characteristics, with less than 11% overhead to index time for large indices. This gives us the ability to accurately control recall while optimizing quantization and oversampling choices for query performance. We got an average increase of 16.7% in QPS when we enabled this feature compared to our previous default settings for DiskBBQ. All while reliably achieving our target recall. Taking away the configuration burden from the user actually allows us to make better choices; it is a win-win.

This is the beginning of a longer journey that we’re working on to bring automatic configuration based on a combination of better understanding of the operating environment and better understanding of the data characteristics. We look forward to sharing more of this work with you in the near future.

Elasticsearch's inference API now connects directly to NVIDIA-hosted models. You get text embedding, completion, chat completion, and reranking, plus access to NVIDIA's catalog of NVIDIA Inference Microservices–optimized (NIM-optimized) retrieval and generative models, without writing any custom integration code.

In practice, that's vector search and retrieval augmented generation (RAG) applications built on NVIDIA-hosted embeddings. It's also multi-turn conversations through the chat completion API and reranking with NVIDIA's cross-encoder models to push relevance past keyword matching. All four task types run natively through the inference API, with support for both streaming and non-streaming responses. How do I get an NVIDIA API key?

NVIDIA offers a broad catalog of models designed for a wide range of use cases, all of which can be explored on the NVIDIA Build model catalog. Throughout this article, we provide specific examples of high-performance models optimized for each inference task type. After identifying the model that best aligns with your application requirements, choose the deployment approach that fits your infrastructure and operational needs. This could mean running it on-premises for greater control or using a serverless option for faster experimentation and simplified scaling.

To get started quickly, you’ll first need access to NVIDIA’s model catalog and APIs. Create an account or log in at https://www.build.nvidia.com/ to explore available models, evaluate their capabilities, and compare which ones best fit your use case before proceeding toward full-scale deployment. This site provides a web-based interface for testing models, which is useful during evaluation and experimentation. For production-level requirements, you can use NVIDIA NIM to deploy endpoints on your own infrastructure.

To access NVIDIA models, you need to generate an API key. This key will serve as the authorization mechanism when making calls to NVIDIA's endpoints. You can create, access, and manage your API keys at API keys. To create a new key, click the Generate API Key link in the top right, and then specify a name and expiration period for the key. After generating the API key, select the appropriate model for your task and set up the corresponding Elasticsearch inference endpoint.

Setting up Elasticsearch inference endpoints

Once you have set up your NVIDIA account and obtained the necessary API keys, you can create an Elasticsearch inference endpoint.

Endpoint setup can be done directly in Kibana using the console, which allows you to input the required steps into Elasticsearch even without using an API. The following sections provide examples and details on how to create and use endpoints for text embeddings, completion, chat completion, and reranking.For more examples and detailed information, please consult the Elasticsearch API reference documentation.

Creating and using a text embeddings inference endpoint

To create a text embedding inference endpoint, you first select an appropriate model that can perform embedding operations. NVIDIA lists its models in the NVIDIA Build model catalog. You can select the Text-to-Embedding or Retrieval Augmented Generation use case on the left to filter the appropriate models. You can also find NVIDIA’s text embedding models in the NVIDIA documentation. NVIDIA’s retrieval APIs include text embedding and reranking models. When choosing a model, make sure it explicitly supports text embedding inference. Text embedding models typically include an API description, such as "Creates an embedding vector from the input text."

A good example of an embedding model is the nvidia/nemotron-3-embed-1b model. You can access the deployment page for this model, which allows you to deploy this model on-premises.

Once you have selected a suitable model, open its API reference page, where you’ll find the parameters required to create an Elasticsearch inference endpoint.

Two parameters are relevant:

• model_id(required): Specifies the NVIDIA model to be used for embedding inference. This parameter is named model on the NVIDIA side.

• url(optional): The endpoint URL used to send requests to the NVIDIA model (either deployed on-premises or in a serverless environment). It must be accessible from your Elasticsearch instance.

For most text embedding models, the URL is static and NVIDIA differentiates models solely via the model parameter. If the url parameter isn’t provided during endpoint creation, the default text embedding task specific value https://integrate.api.nvidia.com/v1/embeddings will be used.

To generate text embeddings, set up an endpoint configured with the required NVIDIA model values in the service settings map:

Upon sending this request, you should receive a successful 200 OK response. This response confirms that the endpoint is functioning correctly and the settings are specified accurately, and it will detail your newly created Elasticsearch endpoint for the text embedding task type.

You can now use the newly created endpoint to generate embeddings. The request for this operation will be similar to the example shown below:

The text embeddings will be returned, accompanied by a successful HTTP 200 OK status.

This integration allows users to use the NVIDIA models directly within Elasticsearch, making advanced search and RAG applications easier to build. These production-ready models offer a reliable and robust foundation for enterprise-scale deployments.

Creating and using a completion inference endpoint

To create a completion inference endpoint, you first select an appropriate model.

NVIDIA lists its models in the NVIDIA Build model catalog. You can search for the model there, but you can also find NVIDIA’s completion models in the left-hand navigation of this large language model (LLM) API documentation.Each entry in the list links to a general description of the model. From there, you can navigate to a nested link that opens the API reference specific to the selected model. A good example of a completion model is the nvidia/nemotron-3-super-120b-a12b model. You can access the deployment page for this model, which allows you to deploy this model on-premises.

Once you have selected a suitable model, open its API reference page, where you’ll find the parameters required to successfully create an Elasticsearch inference endpoint.

Two parameters are relevant:

• model_id(required): Specifies the NVIDIA model to be used for completion inference. This parameter is named model on the NVIDIA side.

• url(optional): The endpoint URL used to send requests to the NVIDIA model (either deployed on-premises or in a serverless environment). It must be accessible from your Elasticsearch instance.

For most completion models, the URL is static, and NVIDIA differentiates between models using only the model parameter. If the url parameter isn’t specified during endpoint creation, the default value of https://integrate.api.nvidia.com/v1/chat/completions will be used.

To use a generative model for the Elasticsearch inference completion task, you configure an endpoint that supports completion operations. The service settings map must include the required configuration for the selected NVIDIA model.

Upon success, you’ll receive a 200 OK response. This response provides the details of your new Elasticsearch endpoint, which is configured for completion tasks.

The created endpoint allows you to generate both streaming and non-streaming completions. These refer to how the endpoint delivers its output. Non-streaming completions wait until the entire response is generated before sending it back in a single block, resulting in a single, slower response time. In contrast, streaming completions send the generated text back in small, continuous chunks as they’re produced, which allows you to start reading the response immediately. This continuous delivery creates the perception of faster interaction and is essential for real-time conversational interfaces.

Generating non-streaming completions

To generate non-streaming completions, you call the newly created endpoint with a request similar to the following:

You'll receive a successful 200 OK response, with the completion result:

Generating streaming completions

To use the streaming functionality for the completion task type, you need to send the identical request used for non-streaming completions, but with _stream included in the URL path:

This command will initiate a continuous flow of events, delivering a sequence of outputs similar to the example provided below:

This capability empowers users to easily integrate NVIDIA generative models directly into their Elastic applications, supporting both single-response and engaging streaming experiences for dynamic content generation.

Creating and using a chat completion inference endpoint

To enable more dynamic and flexible interactions than those supported by the standard completion inference endpoint, you configure a chat completion inference endpoint, specifically designed to handle chat-based completion tasks.

To identify the parameters required to construct the service settings map, refer to the completion inference endpoint section of this blog post. The same configuration principles apply to the chat completion inference endpoint.

The service settings map must include the required configuration settings for the selected NVIDIA model.

Upon success, you’ll receive a 200 OK response, which includes the details of your new Elasticsearch endpoint specifically for the chat completion task type.

You can now use the new endpoint to stream generated completions. Your request should resemble the following example:

The chat completion results will be delivered to you as a continuous stream of events, formatted as follows:

The chat completion capability, distinct from the simpler completion API, allows users to build stateful, multi-turn conversational AI applications directly within the Elastic Stack, using the full flexibility of NVIDIA models for dynamic user interactions following Elasticsearch inference chat completion API.

Creating and using a rerank inference endpoint

Reranking is the process of reordering the results from an initial search query to improve their relevance to your intent. Reranking is a second-stage relevance step that reorders the results returned by an initial retriever. In many cases, it uses a different model than the retriever itself, typically a cross-encoder model, which evaluates the query and each candidate document together to produce a more accurate relevance score. The output is a list of results ranked based on their relevancy, thereby drastically improving the quality and contextual accuracy of the search results.

To create a rerank inference endpoint, you first select an appropriate model that can perform reranking operations. NVIDIA lists its models in the NVIDIA Build model catalog. You can use the reranking label to select the appropriate models. You can also find NVIDIA’s reranking models in the left-hand navigation of this retrieval APIs documentation. NVIDIA includes rerankingand text embedding models in the Retrieval APIs section in its API documentation. When selecting a model, ensure that it explicitly supports rerank inference requests. Rerank models typically include an API description, such as “Ranks passages by their relation to a query.” This wording indicates that the model supports the rerank task type.

A good example of a reranking model is the nvidia/llama-nemotron-rerank-vl-1b-v2. You can access the deployment page for this model, which allows you to deploy this model on-premises.

Once you have selected a suitable model, open its API reference page, where you’ll find the parameters required to create an Elasticsearch inference endpoint. Two parameters are relevant:

• model_id(required): Specifies the NVIDIA model to be used for rerank inference.

• url(optional): The endpoint URL used by the inference endpoint to send requests to the NVIDIA service.

For most models, the URL is static and NVIDIA differentiates between models using only the model parameter. If the url parameter isn’t specified during endpoint creation, the default value

https://ai.api.nvidia.com/v1/retrieval/nvidia/reranking will be used. The nvidia/llama-nemotron-rerank-vl-1b-v2 model requires a custom URL to be specified, and it will be used in the example below.

To perform reranking tasks, you configure an inference endpoint that executes reranking operations. The service settings map must include the required configuration for the selected NVIDIA model.

The successful creation of your new Elasticsearch endpoint for the rerank task type will be confirmed by a 200 OK response, which will also provide the specific details of the endpoint.

You can then start using the new endpoint to perform a ranking operation with a request like the one shown below:

A successful HTTP 200 OK status will be returned, and the ranked entries will be included in the response. Since models are not deterministic, the results you receive may vary and may be ordered differently across calls, as the same outcome isn’t guaranteed each time.

Integrating the rerank capability with Elasticsearch and NVIDIA elevates search applications to deliver the most accurate, contextually relevant results. By using the NVIDIA reranking models within the search infrastructure of Elasticsearch, the system moves beyond simple keyword matching. This capability prioritizes the most relevant documents after the initial search, drastically improving the user experience and the utility of the data.

NVIDIA and Elasticsearch: What's next

The integration of Elasticsearch's inference API with NVIDIA marks a considerable step forward for users. By providing a standardized, simpler path to access NVIDIA's high-performance, optimized models, this integration significantly expands Elastic's capabilities. Users can now work with these models for key AI tasks, including generating text embeddings for vector search, generating and streaming content with completion models, building stateful conversational AI applications with chat completion, and drastically improving search result accuracy through reranking. This simplification streamlines the development of sophisticated AI-powered applications, from advanced RAG systems to dynamic conversational interfaces, making powerful AI more accessible for Elastic users.

Ready to get started?

• Explore the Elasticsearch API reference documentation to dive deeper into setup.

• Browse the NVIDIA model catalog to see all available models.

• Check out the NVIDIA API Documentation hub for integration guides and API references.

• Start your journey by creating an NVIDIA account and API key to begin integrating the models today.

Frequently asked questions

How do I connect Elasticsearch to NVIDIA hosted models?

Create an NVIDIA API key at build.nvidia.com, and then create an Elasticsearch inference endpoint using the nvidia service with your API key and a model_id. Elasticsearch's inference API supports four NVIDIA task types: text embedding, completion, chat completion, and reranking.

What's the default endpoint URL for NVIDIA text embedding models in Elasticsearch?

If no url is specified when creating the endpoint, Elasticsearch defaults to https://integrate.api.nvidia.com/v1/embeddings for text embedding tasks. Completion and chat completion tasks default to https://integrate.api.nvidia.com/v1/chat/completions instead.

Can I use NVIDIA NIM models deployed on my own infrastructure with Elasticsearch?

Yes. NVIDIA NIM supports on-premises deployment, and Elasticsearch's inference endpoint accepts a custom url parameter pointing to your self-hosted NIM endpoint instead of NVIDIA's serverless API.

How do I stream chat completion responses from NVIDIA models in Elasticsearch?

Append _stream to the chat completion endpoint's URL path (POST _inference/chat_completion/{id}/_stream). Elasticsearch returns results as a continuous event stream instead of a single blocking response, ending with a [DONE] event.

What's the difference between the completion and rerank task types in Elasticsearch's NVIDIA integration?

Completion and chat completion generate new text from a prompt. Reranking takes an existing list of documents and a query and then reorders them by relevance score using a cross-encoder model; it doesn't generate text, it rescores what you already retrieved.

How do NVIDIA's reranking models improve Elasticsearch search results?

NVIDIA's reranking models evaluate the query and each candidate document together, producing a relevance score used to reorder results beyond keyword matching. Elasticsearch's rerank endpoint returns each document's index and relevance score, so the highest-scoring passages surface first.

Do I need a paid NVIDIA account to use hosted models with Elasticsearch?

You need an NVIDIA account and an API key generated at build.nvidia.com; NVIDIA's build platform offers both free evaluation access and paid production tiers, depending on usage. Elasticsearch itself doesn't add separate licensing for the NVIDIA service beyond your existing NVIDIA account terms.

A vector store that achieves good performance without tuning is more valuable than one that requires expert tuning. In fact, our contention is a data store that can be coaxed to exceptional performance by an expert who spends a week hand-tuning it is less useful than one that beats it consistently out of the box. In other words, easily achieving good performance is a first class property, not a nice to have. We can see this clearly in our telemetry. The great majority of users will never tune the internals of vector search at all, and why should they: it is just an enabler for what they're trying to build.

This is the imperative behind features like auto-calibration. The system as a whole should look at your data and your quality target and choose good parameters for you. Indeed we think this is a win-win, since it has far more nuanced information available to it to make these choices than we expose.

To make "good performance" precise, it helps to name the three attributes that characterize any vector search system, because they trade off against one another and you can't talk about one without fixing the others:

1. Performance: throughput (QPS), latency, and so on.

2. Hardware cost: a fair comparison always holds cost fixed. It's trivial to buy your way to more QPS or better recall by throwing hardware at the problem; the interesting question is what you achieve per dollar.

3. Search quality: recall, nDCG, and related measures of whether you're returning the right results.

The three form a frontier. Push one and, at fixed budget, you pay in another. Any honest comparison of approaches pins two down and measures the third. What we describe in this post is the mechanism we're introducing to pick quantization parameters for a fixed recall budget. It is a step on a longer journey towards a vector store that configures itself well across the board.

Why recall is the right quality metric for vector search

Search quality is tricky, because the "right" results depend on relevance labels you usually don't have at index time. So we lean on recall as a safe proxy. The argument is simple: recall measures how well the approximate index reproduces the results of exact search over the same embeddings. If recall is high, you have not degraded search quality relative to what the underlying model can do; you can be confident you’ve faithfully preserved the baseline. You might still wish for a better embedding model, we've got you covered, but that's a separate concern from the index not damaging what the model already gives you.

This is why controlling recall matters so much, and why you should be wary of any system that doesn't reliably control it. If a vendor can't control recall, they can silently degrade your search experience, achieving impressive QPS numbers while quietly returning worse results, and you'd have no way to know without a labeled evaluation set. The method in this post is about maximizing performance while keeping a firm, predictable grip on quality.

Why vector quantization parameters must be chosen at index time

What makes the problem genuinely hard is that vectors are quantized as they are indexed, so the parameters that govern quality (how many bits, how deep to rerank, whether to precondition) have to be evaluated before we've seen the data laid out in its final form. We can't index everything, measure recall, and iterate; by then the quantization is baked in.

So we need to estimate what we'll need from a small sample, cheaply and in advance. Fortunately the Elasticsearch gives us natural moments to do this: segment merges are exactly such an opportunity. When segments are combined we have to rewrite the data anyway and can assess the data and (re)choose parameters. And as we'll see, models fit to small random samples give excellent estimates of the quantities we actually need to control. They’re typically good enough to set parameters once, with a small margin, and trust them as the index grows.

How vector quantization affects nearest-neighbor recall

With that motivation in place, let's start to dig into the details.

Vector quantization is a critical component for making approximate nearest-neighbor (ANN) search affordable at scale; it's an area we've innovated in the past. Instead of storing and comparing full-precision embeddings, we store a lossy, compressed representation and search over that. The catch is the one above: lossy representations move distances around, so the "nearest" neighbors under quantized distances are not always the true nearest neighbors and recall suffers.

The standard fix is to over-retrieve and rerank. We use the cheap quantized distances to pull back the top candidates, then recompute exact distances for those and keep the best . As long as the true top- are present somewhere in the retrieved top-, reranking recovers them exactly.

Reranking isn’t free, we have to fetch high precision vectors from disk. However, we can precisely characterize the performance of reranking based on hardware characteristics alone. This reframes the whole problem. The question is no longer "how much does quantization distort distances?" in the abstract, but something which relates back to the attributes we care about:

This post derives a model that answers exactly that. The core of it is a single, surprisingly clean idea: if we can characterize the distribution of distances to the -th nearest neighbor, and we have a model of the quantization error distribution, then we can compute expected recall after reranking in closed form (up to a one-dimensional integral). Everything else – bit counts, rerank depth, whether to precondition – becomes a search over a model we can fit cheaply from a small sample, instead of an expensive empirical sweep over full indices built with those parameters.

We build it up to this in three stages: the geometry of nearest-neighbor distances, the scaling law that falls out of it, and then the recall model that ties quantization error to recall given a reranking budget. Be warned, the following gets a little bit involved, but to give you intuition about what is happening see the video below.

Quantization error vs. the nearest-neighbor distance gap

Fix a query and rank the database vectors by their true distance to it: , so is the distance to the -th nearest neighbor. Reranking the top succeeds for the true -th neighbor whenever it is not pushed past rank by quantization noise.

Two competing quantities govern this:

• The quantization error that is essentially fixed for a given scheme and dataset: it depends on the embedding dimension, the vector distribution, and the number of bits, but not on how big the index is.

• The criticality gap , which is the distance between the -th and the -th nearest neighbor. This is the margin we have to absorb error. Crucially, it shrinks as the index grows: pack more vectors into the same region and neighbors crowd together.

There’s a detail here we’ll gloss over for the sake of presentation: for IVF style indices, we’re quantizing the residual from a cluster’s centroid. This does in fact couple the quantization error to the index size, but we can handle it much the same way we handle the distance to the -th nearest neighbor.

For reranking to recover the recall lost to quantization, we need the error to only rarely exceed the gap. If we can write down the distribution of and the distribution of the error, we can make that statement quantitative. The first job is to estimate the distribution of nearest-neighbor distances.

Deriving the nearest-neighbor distance distribution

Real embeddings don't fill their ambient space; they concentrate on a lower-dimensional manifold. Near a query, though, we can make a mild local assumption: in a small neighborhood around the query, the data density is roughly uniform. Here is the intrinsic dimension of the manifold; it is unknown and generally far smaller than the embedding dimension. How to estimate it is the subject of Section 4.

Let be the vectors falling in , modeled as i.i.d. uniform on , and define the distance from to its nearest neighbor:

To get the distribution of we use the standard order-statistics trick: rather than ask where the minimum is, ask for the probability it exceeds some radius . The event is exactly the event that every point lands outside the -ball centered on the query .

A single point lands inside with probability equal to the ratio of the ball's volume to the region's volume

where is the volume of the unit -ball. (We assume is large enough that the relevant is small, so the ball doesn't spill outside and boundary effects are negligible.) Because the points positions are assumed to be independent, the survival function is

What we're really interested in is how R behaves on average. To compute this, we use the identity that the expectation of a non-negative random variable is the integral of its survival function, . Evaluating this with (2) gives the headline result:

(The exact integral carries an extra factor; it's an constant that we can fold into a fitted coefficient later, so we drop it here.)

Glacial scaling: why neighbor distances barely change as your index grows

It is interesting to consider what this formula tells us about how distances change with dataset size: . The exponent is , and in high intrinsic dimensions that is a very small number. This is a property the method leans on, so it's worth plugging in some numbers:

• If then doubling multiplies by , so distances drop by ~30%.

• If then doubling multiplies by , so distances drop by a little over 1%.

In high dimensions, neighbor distances barely move even if you add a lot of data; call it glacial scaling. It's the reason we can choose quantization parameters once from a tiny sample, with a small safety margin, and trust them to remain valid even after the index grows substantially before the next re-quantization.

Expected distance to the k-th neighbor and the criticality gap

We actually care about the whole sequence of order statistics , , not just the minimum. There's a simple way to get them.

Map each radius to the cumulative volume it encloses by defining

By (1), each is exactly the probability of landing within radius , so the are uniform on . The order statistics of uniforms are textbook: the -th smallest of uniforms follows a Beta distribution,

Inverting the volume map, , gives the scaling of the -th neighbor distance:

That's all we need for the expected gap:

The last form is the intuitive one: the gap between the -th and -th neighbors is the distance to the -th neighbor, scaled by . Widening the rerank depth relative to opens the gap; higher intrinsic dimension closes it (the exponent pushes toward 1).

Why the expected gap is sufficient to predict recall

Working with an expectation is only legitimate if the gap doesn't fluctuate wildly around it. It doesn't because concentration of measure saves us. Applying the delta method to and using from the Beta distribution, a little algebra gives

So the coefficient of variation is about . For any reasonable intrinsic dimension this is negligible, which justifies modeling only the expected distances. (If you're worried about the delta method approximation, you can check the results numerically: the delta-method variance and the resulting coefficient of variation match the exact expressions to several significant figures.)

Extending the model to cosine similarity and inner product search

The derivation is for the Euclidean metric, but the other common metrics reduce to it:

• For cosine similarity, the equidistant surface is the intersection of a sphere around the query with the unit sphere. This is called a hyperspherical cap, whose volume scales as for small . Therefore, the analysis carries over unchanged up to constants, with the dimension reduced by one.

• For MIPS (maximum inner product), some extra care is needed, because nearest neighbors aren't confined to a compact region. A distant vector can still win on inner product if its norm is large enough, so the gap is really governed by the tail of the norm distribution. However, there is a clean fix, which is to use the Neyshabur–Srebro transformation. This lifts vectors onto a unit hypersphere in dimensions. After this operation, it's just the cosine case.

Fitting intrinsic dimension and scale from a small sample

Equation (3) has a known functional form but two unknown parameters: the intrinsic dimension and the scale . Both are easy to fit, and it's more convenient to fit them from raw neighbor distances than from gaps directly.

Sample several subsets of database vectors of sizes and a set of query vectors . For each query and each subset, measure , the distance to the -th nearest neighbor of within . Taking logs of the scaling law linearises it:

Specifically, this is linear in and , so ordinary least squares recovers and . Varying the subset size is what makes it possible to estimate : it's precisely the rate at which distances shrink with data volume. With the fitted parameters, the whole-index expected gap is

Figure 1 shows how well this fits in practice (and it’s remarkably good): predicted versus actual average distance to the -th neighbor, across a range of datasets and metrics, have between 0.996 and 0.999.

Modeling vector quantization error as Gaussian

With the nearest-neighbor distance model established, the second component is the quantization error distribution. For every metric we use, the quantized distance estimate differs from the true distance by an error that is a sum of many independent per-dimension contributions. By the Central Limit Theorem that sum tends to Gaussian, so we model the error as normal with a variance we estimate empirically:

where is the quantized distance estimate using -bit vectors and is the total number of (query, neighbor) pairs in our sample set. In other words: sample, quantize, measure the squared distance errors, average.

Figure 2 shows the empirical basis for the Gaussian assumption: measured quantization error densities against best-fit Gaussians across a variety of datasets. The fit is good, which is what lets the rest of the model stay analytic.

We could stop here and take a minimax view: threshold the probability that the -th and -th neighbors swap, using the expected gap (4) against the error scale . But that controls a worst-case event, and what we actually want to control is average recall. The outcome would be overly conservative quantization parameters and we'd pay some performance. The next section estimates expected recall properly.

Predicting expected recall after reranking

Combining the distance model and the error model gives a closed-form estimate of expected recall after reranking. Model the noisy distance of the -th true neighbor as a Gaussian centered on its true distance:

The -th neighbor survives reranking, i.e., lands in the retrieved top , if fewer than other vectors have a smaller noisy distance. Condition on and count the competitors closer than :

Then the probability of recalling neighbor integrates over where its own noisy distance lands:

The terms of are independent Bernoullis but not identically distributed, since every neighbor sits at a different true distance , so each has its own probability of intruding on the top- set:

with the standard normal CDF. This makes a Poisson-binomial variable. Since we sum many of them (because ), the Lyapunov CLT applies and we approximate

with the standard Poisson-binomial moments

The survival probability then has a clean closed form:

This is where the two halves of the post so far finally meet. We don't need to know the individual because the manifold scaling law from Section 3 supplies them: . So the moments become explicit sums over ranks, which we truncate at a safe cutoff (say , since distant neighbors contribute negligibly):

Finally, average recall@ given rerank depth sums the per-neighbor recall over the top :

Here is the standard normal density. Each integral is smooth and one-dimensional, so Gauss–Legendre quadrature evaluates it in microseconds. The entire recall prediction for a set of candidate parameters costs a handful of quadrature evaluations, not index build and benchmark run.

Figure 3 validates the end-to-end model: predicted average recall against measured recall across many parameter settings and multiple datasets has .

How the recall model selects vector quantization parameters

With a fast recall predictor available, parameter selection becomes a cheap ordered search. Given a target recall and a rerank budget (typically expressed as a multiple of ), we can find the minimum document and query bit counts, and other knobs, that clear the target. There are a few things to note that are practically important:

1. Glacial scaling gives us some safety because moves so slowly with for even moderate intrinsic dimension. A small margin in the calculation means the chosen parameters stay valid if a lot of vectors are added before parameters are restimated.

2. Small is the worst case if is a fixed multiple of . The gap \mathbb{E}[R_{(k)}]( is smallest for small so if a parameter choice satisfies the recall target at then it will for larger will too.

3. We can treat quantization as a black box because the error model only needs the empirical error variance. This means we can test any configuration, including preconditioning, the same way and we can simply order candidate parameter tuples by increasing index and query cost, and stop at the first choice that hits the target recall. For tuples of (query bits, doc bits, rerank depth, precondition) a sensible search sequence increases query precision first, then document precision , , , , , , , , and each combined (via an outer product ) with rerank depths like and precondition , exiting as soon as the target is met.

Results: auto-selected quantization parameters and recall across datasets

In this section, we discuss the results of the initial experiments on the end-to-end behavior. We’ve made some further refinements as part of the work to fully integrate with Elasticsearch that we discuss in our other post.

The table below shows auto-selected parameters targeting recall 0.97, measured with brute-force search, so the number reflects loss due to quantization alone (64 query clusters, targeting document clusters of size 384, which matches the settings of DiskBBQ).

Dataset

Query bits

Doc bits

Precondition

Depth

Recall

FiQA E5 small

4

2

false

30

0.97

FiQA arctic

2

2

false

30

0.95

FiQA GTE

2

1

true

30

0.98

MNIST

3

1

true

30

0.99

Fashion MNIST

3

1

true

30

0.99

Quora E5 small

2

2

false

30

0.99

Quora arctic

2

1

false

30

0.97

Quora GTE

1

1

false

30

0.98

Dbpedia E5 small

4

2

false

30

0.99

Dbpedia arctic

2

1

false

30

0.94

Dbpedia GTE

2

1

false

30

0.96

Wiki Cohere

2

2

false

30

0.99

Hotpot E5 small

4

2

false

30

0.97

Hotpot GTE

2

1

false

30

0.96

Glove 100

4

2

false

30

0.87

Glove 200

4

2

false

30

0.89

SIFT128

4

4

false

20

0.99

There are a few things worth highlighting:

• The recall is very sensitive to rerank depth. This is why we nearly always end up choosing the maximum depth available: a step up in rerank depth from 20 to 30 is typically what pushes us to hit the recall target for fewer bits and we prefer fewer bits. In the real system, we tuned this behavior based on a more representative reranking cost.

• Glove underperforms partly we approximate the query distribution with random samples from the corpus, but Glove is also less well characterized by the model than the other datasets. A plausible explanation is that the approximately uniform local density assumption from Section 2 is less reliable for Glove embeddings, which would show up as higher recall variance between queries. However, Glove embeddings are not representative of the actual vectors we need to store.

• The FiQA GTE preconditioning choice is a knife-edge case: preconditioning produced only a tiny expected recall improvement, but the prediction sat right at the recall cutoff and allows us to drop the query from 3 to 2 bits. If we'd rather only keep preconditioning where its benefit is clear-cut, we can enforce a minimum uplift threshold. This sort of fine-tuning of the decision logic leaves all the heavy lifting to estimate recall unaffected.

Key takeaways: auto-tuning vector quantization from first principles

We presented a method to pick optimal quantization parameters to achieve a target recall. It rests on two models that compose cleanly:

1. A geometric model of neighbor distances that follows from a local uniform density assumption. We use this to derive the nearest-neighbor distance, the glacial scaling law of the expected distance, and the expected distance profile . We show that fitting and by a simple log-linear regression to average distances in small random samples from the corpus gives an extremely accurate predictive model.

2. A Gaussian quantization error model that is justified by the CLT. Its only parameter is an empirical variance we estimate by comparing quantized and raw vector similarities for a sample of the corpus.

Finally, we show that it is possible to feed the estimated distance model into a Poisson-binomial count of neighbors that intrude on the top- set. Applying the Lyapunov CLT the expected recall@ after reranking to depth falls out as a one-dimensional integral we evaluate by quadrature.

The outcome is an accurate () predictive model of recall as a function of the quantization parameters. Choosing quantization parameters then becomes an ordered search with a predictive model telling us if we’ve hit the recall constraint. And nicely one that also comes with a built-in argument (glacial scaling) for why the chosen parameters remain safe even when estimated from a relatively small fraction of the data.

We’ve built this entire mechanism into Elasticsearch using segment merges as an opportunity to reassess our quantization choices. Aside from the peace of mind this brings (that you’ll achieve good recall whatever vectors you throw at it), it also allows us to chose near optimal parameters from a performance perspective. This closes the loop on our original objective: near optimal performance out of the box, at least as far as quantization goes. We’re pretty excited about the advantages that model based tuning can bring to vector search and look forward to sharing other work we have in this direction in the near future.

Combining image and text into one embedding beats either alone, and the gap isn't small. In our tests, averaged image and text embeddings put the correct product in the top spot up to 1.5 times as often as image embeddings alone. We benchmarked 5,000 real apparel and footwear products in English and German across two Jina embedding models, jina-clip-v2 and jina-embeddings-v5-omni-small, to see which model and which indexing strategy actually wins for ecommerce search. The older, narrower Contrastive Language–Image Pre-training–style (CLIP-style) model beat the newer, more general one, and that wasn't what we expected. This post walks through the data, the method, and what we'd recommend doing with it.

jina-clip-v2 vs. jina-embeddings-v5-omni-small: What's different

Multimodal embedding models work by generating representative semantic vectors for inputs of different kinds in a single high-dimensional space.

We used two models that do this:

jina-clip-v2

jina-embeddings-v5-omni-small

Architecture

Dual encoder (separate text + image towers)

Single shared backbone + frozen encoders

Parameters

~865M total

~1.74B total

Embedding dimensions

1024

1024

Max input

512×512 images, 8k tokens text

32k tokens

Language coverage

Broad multilingual

~100 languages

Modality handling

Text ↔ image alignment (purpose-built)

Text, image, audio, video via projectors

jina-clip-v2 is a CLIP-style dual encoder: a text tower (Jina XLM-RoBERTa, 561M parameters) and a separate image tower (EVA02-L14, 304M parameters), about 865M parameters in total. The two towers are independently trained but fine-tuned to output to a common semantic space. It produces 1024-dimensional embeddings, handles images up to 512×512 and up to 8k tokens of text, and has broad multilingual support. It has been engineered specifically to support text-to-image, image-to-text, and text-to-text matching.

jina-embeddings-v5-omni-small has a broader scope. It extends the jina-embeddings-v5-text model to support images, audio, and video by attaching frozen vision and audio encoders to the frozen text backbone. The encoders connect through cross-modal projectors, small trained layers that map each encoder's output into the text model's embedding space. These projectors are the only part of the model to receive additional training. The resulting model produces 1024-dimensional embeddings, supports a 32k-token input context, and covers roughly 100 languages. It encodes queries and documents asymmetrically: a query with the retrieval.query task, a document with retrieval.passage.

There’s an important functional difference between the two models: jina-clip-v2 is really two separate models trained to work together, but jina-embeddings-v5-omni-small uses a single shared backbone that produces embeddings for all its supported media types. Every modality maps into one shared vector space. In principle, it can handle text, images, audio, or video, or combine materials of different media types into one input, yielding one embedding that encompasses all the data. However, there are two important caveats when using jina-embeddings-v5-omni-small: Combining image and text into one input is a documented weak spot for the model, and the Jina API only allows users to embed one modality per request. Theoretically, it can create a joint image and text vector, but in practice, you can’t with the API (and shouldn’t anyway).

The ecommerce product dataset we used

For this article, we downloaded the Fashion Product Images dataset from Kaggle. It contains roughly 44,000 catalog entries for products from a real fashion retailer, each with a high-resolution photo and structured metadata. We only used the Apparel and Footwear categories (about 30,600 products) and sampled 5,000 from them with a fixed random seed.

For each product, the dataset contains three records:

• Each product is pictured in a 1800×2400 JPEG against a clean background.

• Metadata with the labels gender, masterCategory, subCategory, articleType, baseColour, season, year, usage, and productDisplayName.

• A collection of additional informationwith labels like Neckline, Pattern, Sleeve Length, Fit, and Fabric, and a free-text description in English.

For example, item #13885 is labelled "Scullers Men Check Black Shirts," with an accompanying image (see below) and a description that reads "Black and white checked shirt, made of 100% cotton, full length buttoned placket, long sleeves with buttoned cuffs."

How we generated the search queries

We generated our test queries without an AI language model, using rules and substitution lists.

For each product, we started from its color and article type and added one descriptive modifier, drawn at random from the product’s available metadata: neckline, pattern, sleeve length, length, surface styling, fit, fabric, season, or usage. We then reworded the query by substituting words from a fixed synonym table. This avoids making queries that reuse the catalog's own words. For example, ″t-shirt″ becomes ″tee″, ″regular fit″ becomes ″classic cut″, ″printed″ becomes ″with graphic design″, ″sleeveless″ becomes ″no sleeves″. The longest phrase with a synonym was replaced first, so we swapped ″sports shoes″ rather than ″shoes″. Because the pipeline is rule-based and seeded, query production is reproducible and all variation is accounted for. German queries were generated the same way, using a German term table, and then a native speaker corrected them for natural retail phrasing. Some examples:

The German queries used their own term table, mapping the same catalog attributes to natural German retail phrasing (later checked by a native speaker). Some examples:

The six embedding configurations we tested

We tested retrieval in six configurations, using the same text queries in each test condition and the same 5,000 product indexed dataset. For both jina-clip-v2 and jina-embeddings-v5-omni-small, we tested three different ways of generating embeddings for indexing:

1. Image-only. We generated embeddings from the images alone without any other data.

2. Text-only. We generated embeddings for the free text descriptions alone.

3. Averaged image and text. For each product, we generated embeddings for the image and text description separately and then averaged the two vectors into one.

We used the Jina API to generate document and query embeddings for product images and free text descriptions, as shown in the code below. All images were resized to fit into a 512x512px square before processing.

We combined images and texts by embedding them separately, averaging the two vectors, and then normalizing the result so we can speed up calculating cosines. This works for multimodal models because both embeddings share the same semantic space. The sum of the two vectors is a new vector with the semantic features of both.

This is very easy to do using the numpy package in Python. We used the code below:

For this article, we did exact retrieval, calculating the cosine between queries and all 5,000 stored product embeddings. In Elasticsearch, we would use a shortcut to approximate the same result. From the ranked results, we calculate Recall@1, Recall@5, Recall@10, Mean Reciprocal Rank (MRR), and Normalized Discounted Cumulative Gain at position 10 (nDCG@10).

Each query has exactly one correct answer, so Recall@K is the share of queries whose product lands in the top K. MRR scores the results by how close the correct answer is to the top. nDCG@10 is a standard metric that penalizes putting the best answer lower on the results list.

Product search benchmark results

The table below is the German cross-lingual run. We evaluated German queries to find products with English descriptions:

Query benchmark results for jina-clip-v2 and jina-embeddings-v5-omni-small using German-language text queries and the images and English descriptions of garments.

The averaged image/text embeddings score the best, both using jina-clip-v2 and jina-embeddings-v5-omni-small. Surprisingly, averaged vectors from jina-clip-v2 lead the table. It ranks the correct product first about 1.5 times as often as the image-only setup and significantly more often than the text-only setup. Furthermore, it beats every jina-embeddings-v5-omni-small condition. Results for the same tests using English-language queries:

Query benchmark results for jina-clip-v2 and jina-embeddings-v5-omni-small using English-language text queries and the images and English descriptions of garments.

The English run, on the same 5,000 products, tells the same story. The main difference is that the jina-embeddings-v5-omni-small scores are significantly closer to those of jina-clip-v2, although still lower.

What the benchmark scores actually mean

The relatively low scores in the German and English benchmark tables above are to be expected. This is a real-world dataset full of near duplicates. A search for a "black tee with classic cut" has to sort through dozens of basic black T-shirts, and even on the best ecommerce sites, you would expect a result like this. The important thing to understand is the difference in scores between the different conditions, not their absolute values. Our key finding is that combining text and image embeddings yields better performance than either one alone, highlighting how multimodal AI-driven search can use different information sources to produce better performance than non-multimodal strategies.

When a picture is worth a thousand words, and when it isn’t

The gain from adding images to text embeddings isn’t evenly distributed. We did a deep dive to see if there was a pattern to the results and discovered a few things:

• Images add a lot for visually distinctive items and attributes. For example, on footwear, image-only retrieval is on par with text-only: 0.029 versus 0.028 for Recall@1. Shoe styles have distinctive shapes, so the picture does the work (image-only 0.029 versus text 0.028). The same holds for visible attributes more broadly (color, pattern, sleeve length), where fusing image and text gives the biggest lift over text-only (0.085 versus 0.073).

• Images help the least with things the model can’t see. For example, if we query for fabric types, adding images to embeddings adds next to nothing. Humans and AI models alike struggle to see that something is or isn’t made of linen or polyester or some other fabric type. That information is only in the text description and metadata.

As a rule of thumb, we find that searches for clothing lean more heavily on accurate text descriptions, while footwear leans more on the semantics of images. But in both cases, merging the two embeddings either improves results or doesn’t make them worse. This highlights how use-case–specific considerations drive optimal search strategies.

Cross-language queries gain a lot from multimodal embeddings

The gap in performance between text-only and combined image and text embeddings using jina-clip-v2 is much larger for German queries (0.074 versus 0.065 Recall@1) than for English ones (0.076 versus 0.075). This implies that English queries can take advantage of being in the same language as the product descriptions. Whether that’s due to overlaps in the words or that the model is simply more competent with single-language semantics than cross-language doesn’t matter. But adding images to the text embeddings compensates almost completely for the model’s shortcomings in cross-language retrieval.

This gap is even larger for jina-embeddings-v5-omni-small. In any kind of cross-language or multilingual context, multimodal embeddings seem to significantly improve retrieval performance.

Can AI-generated product descriptions replace human ones?

AI-generated descriptions scored worse than human-written ones in our tests. We tried replacing human-authored product descriptions with ones written by jina-vlm based on the image. For this test, we used a 1,000-product random subset. The results were much worse than with the original human text. This was what we expected: The automatically generated description was less accurate and less oriented toward the salient features of the product than the human authored one.

So it turns out that not everyone’s job can be replaced by AI. People who write blurbs for catalogs ought to be safe for now.

How should you index your ecommerce data?

Our tests aren’t totally scientifically rigorous, but they do offer some insights into the issues you might face if you have similar data. We offer the following as provisional conclusions:

• If you have aligned texts and images (and most catalogs do), combine them in your embeddings. In every case, using a multimodal embedding model like the ones Jina AI by Elastic provides and then averaging the image and text embeddings significantly outperformed all other options. The combination adds no computing costs at inference time but does create additional costs at embedding time. For each product, you’ll need to generate two embeddings and combine them, roughly doubling the cost.

• Use the right model. You need to identify a model that supports all the modalities and languages you plan to use. All the inputs have to be embedded in the same semantic vector space or none of this will work. It won’t do to get two single-modality models or multiple language-specific text models, average their outputs, and hope for the best. Jina AI by Elastic currently supports texts in up to 100 languages, including computer code and technical terminology images of all kinds, such as scans and infographics, as well as audio and video data. You can change your mind about your models later, but only if you’re willing to reindex all your data.

• Whatever you pick, test it on your own data. The only way to know what the best model is for you, your data, and your use case is to try them out. We were very surprised that our older CLIP-style model outperformed our latest on this dataset, but it was trained almost specifically for this use case. Your data and use case could easily show the opposite. This result is from one catalog, with one style of photography and one kind of query. The ranking between two models can flip with a different domain, image style, or query mix. Run the same sort of benchmark on a sample of your own products before you commit. It’s the only way to know which model really fits your case.

• Use generated descriptions to fill gaps, not to replace good text. AI isn’t a replacement for good work done by conscientious people. Replacing human-made descriptions with machine-made ones made results worse. AI should only replace humans when it has to, like when data is missing or needs to be augmented and it’s impractical to have humans fill in the gaps. Yes, we have tools that work in those situations, but they aren’t necessarily good substitutes. They’re OK substitutes, sometimes.

Average your embeddings. Semantic embeddings are very robust, and averaging them is a relatively cheap solution that doesn’t affect inference-time costs at all. This is a real boon over methods that index each modality separately and require multiple queries to satisfy a single request. But they do require compatible multimodal models.

Limitations of this product search benchmark

A few things to keep in mind before generalizing too much from this experiment:

This article doesn’t perfectly match real-world use cases. Human users make messier queries and have more ambiguous matching criteria. The queries we used were generated specifically for this data. A test with actual customer-made queries from system logs would be a better one.

Embedding averaging isn’t the same as a true joint embedding. Embedding models rely on the different parts of their input to interact in order to extract a semantic representation of the whole. The approach used here is a bit of a hack, one that relies on the robust nature of semantic embedding spaces to get the job done. We expect future models from Jina AI to produce better embeddings by supporting more than one input modality at a time.

This is one dataset in one domain with distinctive features. Fashion photography is very foreground-focused and the descriptions are attribute-rich. Other kinds of materials, even for ecommerce, may look very different. It’s important to test as much as possible with your own data or something very similar.

How to get started with multimodal product search embeddings

jina-clip-v2 and jina-embeddings-v5-omni-small are available through the Jina API, Elastic Inference Service (EIS), and Hugging Face. The omni models are free to download under a CC-BY-NC-4.0 license and free for noncommercial use, with commercial licensing through Elastic. If you use Elasticsearch, EIS exposes both models through the semantic field type, with non-text media in Base64 encoding.

The takeaway from this article is intended to be practical and actionable: For product search, a picture and its description aren’t the same signal. Both add information, and you don’t have to pick one. Average your multimodal embeddings, and benchmark the results with your own data to get a good picture of the kinds of results you can expect.

We didn't want to fire back with another round of numbers. Reproducing a benchmark is the easy part; understanding why it produces the numbers it does is the useful part, and it is the part both original posts skipped. So we stood up Qdrant's exact setup on our own cluster, loaded the same 21 million vectors, reproduced their result, and then traced every number back to its cause.

Here is what we found. The result comes down to a few setup choices: how many segments you build, whether the data is warm in memory, how you return the result ids, and which hardware you run on. Control those, and it cuts both ways: matched to Qdrant's setup the two engines are on par on query speed, and with fast default ingestion Elasticsearch is faster at search and faster to index. The big multipliers in both posts, our 7x and their reply, are artifacts of those choices, not a verdict on either engine. The two reasons Qdrant leaned on hardest, an io_uring disk scorer and Elasticsearch's memory use, turn out to be the two that mattered least: the disk is never read during the run, and the memory gap is a difference in labels, not in bytes.

We hold our own original post to the same standard. Both sides changed several things at once and reported the result rather than the reason. One rule for the rest of this post: every claim gets a number and a mechanism, or it doesn't ship.

Vector search benchmark setup: hardware, config and query set

Here is exactly what each side ran.

Elastic (original post)

Qdrant (response)

This post (apples-to-apples)

Nodes

3 × n4-standard-8 (7 vCPU / 26 GB), GCP

3 × m6g.large (2 vCPU / 8 GB), AWS

3 × m6g.large (2 vCPU / 8 GB)



Elasticsearch

bbq_disk 2-bit, replicas: 1

(cited original)

bbq_disk 2-bit, vectordb_document, bfloat16, replicas: 0



Qdrant

2-bit, async scorer off, RF=2

TurboQuant 4-bit, async scorer on, RF=1

TurboQuant 4-bit, async scorer on, RF=1



Copies per shard

2 = 2 (matched)

1

1 = 1 (matched)



Query set

10k fixed, recall@100

10k fixed, recall@100

10k fixed, recall@100

Three things are worth pinning down, because all three have been used as talking points.

1. Replica count was matched in both rounds: An Elasticsearch index with replicas: 1 keeps two copies of each shard, which is exactly what Qdrant's RF=2 does. In this round, both sides ran a single copy. It was never a thumb on the scale in either direction.

2. We changed a few things on our side on purpose, and we will own them: vectordb_document index mode, float32 to bfloat16 for stored full-precision vectors (3,072 down to 1,536 bytes each), and a merge policy tuned for vector data. Those choices matter later, so we are flagging them up front rather than burying them.

3. Memory is compared like for like. Qdrant's post sets Elasticsearch's JVM heap allocation against Qdrant's total memory use, but those are different accounting categories, since heap size is not total memory consumption. On these 8 GB nodes we give the JVM heap 2 GB, 25% of node RAM, deliberately: that follows Elastic's Vector Search Optimized profile, which leaves the other 75% for the OS page cache where the vectors actually live, rather than the general 50%-heap upper bound. The honest comparison is resident index plus runtime plus active page cache, and on an 8 GB node both engines live inside the same envelope.

Look at the middle column, though. Between our original run and their response, Qdrant changed almost everything at once: new hardware, 4-bit quantization instead of 2-bit, the async scorer on, a different replication factor. Then it credited the win to one of those changes. Drawing meaningful conclusions from many simultaneous changes is hard, and pinning down which change actually moved the number is the whole job of the sections that follow.

What does disk rescore cost in a vector search benchmark?

In this benchmark, the answer is almost nothing, because the disk is never read. Two back-of-the-envelope budgets show why, and the live measurements later in the post confirm it.

The design under test is built around one idea: keep a small quantized copy of each vector in RAM for approximate search, keep the full-precision originals on disk, and read a handful of those originals back to rescore the top candidates for accuracy.

Two budgets determine performance: disk I/O for rescore reads and RAM for what has to stay resident. Both are computable on the back of an envelope, so let us compute them before measuring anything.

The disk I/O budget for vector search rescoring

Rescoring reads the top 100 candidates per query. A 768-dimensional float32 vector is 768 × 4 = 3,072 bytes. Because those reads are not page-aligned, the real cost per vector is closer to an 8 KB page read. Each query needs roughly 800 KB and 100 random reads. Scale that up:

target QPS (cold)

rescored vectors/s

reads/s (IOPS) per node

MB/s per node

10

1,000

333

3

100

10,000

3,333

27

1,000

100,000

33,333

273

10,000

1,000,000

333,333

2,731

Two things fall out immediately: First, cold rescore is an IOPS problem, never a bandwidth one. Even at 10,000 QPS, you need only about 2.7 GB/s per node, but a third of a million IOPS per node. Second, the total working set for the benchmark is tiny: 100 candidates × 10,000 queries × 8 KB = 8 GB across the cluster, or 2.7 GB per node. That fits in the free page cache on an 8 GB node with room to spare. The first time the benchmark cycles through its 10,000 queries, the originals it touches are pulled into page cache, and every read after that is a memory copy.

Put a ceiling on it too. On gp3 at baseline (3,000 IOPS/node), if the reads did go to disk, the workload would top out around 90 QPS from IOPS alone; provisioned gp3 or local NVMe would push that to roughly 480 or 3,000. Hold that number. It is the ceiling that would matter in a benchmark that actually touched the disk.

The RAM budget for a disk-based vector index

What has to stay resident is the quantized copy plus the search structure. Everything else is page cache. At 4-bit quantization and an HNSW graph with m=16:

component

bytes/vector

per node @ 21M

4-bit quantized vectors (always_ram)

384

2.69 GB

HNSW graph, m=16 (2m links × 4 B)

136

0.95 GB

resident total

520

3.64 GB

raw float32 originals (on disk)

3,072

21.5 GB (on disk)

We measured the resident footprint on the running nodes at 3.4 to 3.9 GB, sitting right on the 3.64 GB estimate. The point to hold is that at 21 million vectors, the entire searchable index (quantized vectors plus graph) is about 3.6 GB per node and it fits in RAM, leaving a couple of gigabytes to spare. Nothing in this benchmark forces the disk-backed design to actually use the disk. We come back to that at the end, because it is the real story.

Does io_uring make vector search faster?

Qdrant's reply credits two levers it says we omitted: a two-stage prefetch-and-rescore retrieval pattern and an async, io_uring-based disk scorer. They are two faces of the same rescore step, and the same evidence answers both. Take the async scorer first, since it is the one Qdrant makes the centerpiece: io_uring to parallelize disk reads during rescoring. It is a good feature, and in this benchmark it had exactly one job: parallelize the disk reads during rescoring. It got to do none of them. We verified that in three independent ways.

First, the arithmetic. From the RAM budget, the quantized index and graph are pinned in RAM. From the I/O budget, the rescore working set is 2.7 GB per node and lives in page cache after the first pass. That leaves no disk reads on the hot path to accelerate.

Second, their own methodology guarantees it. Each operating point in the harness runs a full 10,000-query recall pass, followed by a timed throughput window that cycles through the same 10,000 queries again. By the time the stopwatch starts, every vector those queries will touch is already resident. The measurement is warm by construction.

Third, and this is the part we insisted on doing rather than arguing, we measured it. We were careful here because it is easy to accidentally test io_uring in the off state and not notice. Running Qdrant in Docker, we found that io_uring was not even initializing: the default seccomp profile blocks the io_uring syscalls, so it silently fell back to synchronous reads (failed to initialize io_uring instance: Operation not permitted). We fixed that, ran the container with seccomp unconfined, confirmed zero io_uring errors in the logs, and re-ran. This honestly means we have both states:

• Async scorer effectively off (synchronous fallback): 31.6 QPS at ef=50.
• Async scorer on (io_uring confirmed working): 35.8 QPS at ef=50.

A 13% move, and even that is within the run-to-run noise for a measurement whose ceiling is set by CPU, not disk. Put plainly, switching on the feature that the entire result was credited to changed about as much as running the benchmark a second time. To close the loop, during the actual throughput window we watched the block device on all three nodes with iostat: read throughput held at 0 MB/s and 0 IOPS, while CPU sat at 60-70% and climbed to 100% as we added concurrency. The bottleneck is the CPU doing quantized distance computations and graph traversal. It is not the disk, because the disk is asleep.

The two-stage prefetch-and-rescore pattern, the lever Qdrant lists first, is answered by the same run. Every Qdrant number here comes from Qdrant's own reproduction script, which uses their two-stage query throughout, so two-stage was on for the whole benchmark, including the io_uring comparison, and the disk still held at 0 IOPS. It is also not something we omitted: prefetch is the approximate search over the quantized vectors, which lives in RAM, and rescore reads the originals, which live in page cache here. That is the disk-rescore workload, and bbq_disk runs the identical shape. The one part of two-stage that is not about disk, how many candidates you rescore, is CPU rather than I/O; it is already included in these numbers and available to both engines, so it is not a hidden lever either.

There is a smaller detail worth noting. Qdrant's post does not say which disk they ran on. That would normally matter in a benchmark whose premise is disk access, but here it does not, for the same reason the async scorer does not: you cannot be bottlenecked on a device you never read from. The disk is left unspecified, and as it happens it is also beside the point.

Here's what I've seen, as an engineer who has sat through a lot of these conversations: io_uring gets treated as a universal fix far more often than it is the bottleneck. It is excellent when the working set overflows RAM, and the cold I/O budget above shows exactly that case, where parallelizing reads buys you throughput. This benchmark never enters that regime. Async disk reads are a real answer to a real question, and this benchmark just never asked it. Of everything that changed between the two runs, io_uring is the one that got the headline and moved the number the least.

Why does faster ingestion mean slower vector search queries?

This is where reproducing the number got interesting, and where the honest driver of the gap turned out to live.

Our first load produced 128 segments and roughly half the throughput we expected. Qdrant's published provenance, right there in their results JSON, records 67 segments. That one difference explained almost everything. When we merged our collection down to match their segment count, all three metrics moved together toward their numbers:

segments

recall@100

QPS

avg latency

128 (our first, parallel load)

0.9745

35.8

112 ms

66 (merged to match theirs)

0.9531

53.3

75 ms

67 (Qdrant published)

0.9596

67.2

59.5 ms

Look at what moves. Recall falls as segments drop, because with fewer segments each query examines fewer total candidates. That is the tell that segment count, not tuning, was inflating our recall. And throughput rises, because HNSW query cost is very sensitive to the number of segments: every query fans out across every segment's graph, so twice the segments is close to twice the per-query work, plus the contention of doing all that on two cores. The small remaining gap between our 53 and their 67 is warm-cache completeness in our shorter run, not the engine.

Here is the part worth stating carefully, because it is architectural and it cuts against us as much as for us. We use the same segmented design; this is not a Qdrant quirk. The question is how you get to few segments. You get there by ingesting single-threaded, so data concentrates into fewer, larger segments. Fewer segments means faster queries, but you pay for it with slow ingestion. Our parallel load was fast and produced many segments; their single-threaded load was slower and produced few. It is a real tradeoff, and it is the same one on both sides.

Where the two designs genuinely differ is how much that tradeoff hurts. Building an HNSW graph is expensive, and its query cost scales poorly with segment count. On these 2-vCPU nodes, constructing the graph at ef_construct=256 took about 1.6 hours pinned at 100% CPU. The IVF layout behind bbq_disk is cheap to build and far less sensitive to segment count, so we can ingest with many threads while keeping query latency low. We do not have to choose. That is not a benchmark trick. It is the IVF vs. HNSW tradeoff, stated honestly.

Why document retrieval, not vector search, explains the rest of the latency

Now turn the same lens on our own result, because the same accounting applies to our own numbers too.

At the search-light end of the sweep, a large share of the per-query time is not vector search at all. It is the fetch phase assembling the response. vectordb_document returns full source documents, which is exactly what you want for retrieval and hybrid search, where you actually need the document back. A pure vector benchmark only needs the top-N ids, and today _id lives in the same stored-fields column as _source, the text and the vector. So fetching an id pulls that entire compressed block through the decompressor for every hit. When we isolate the id from that column, Elasticsearch steps straight up into Qdrant's low-segment throughput range. That gap is document retrieval, not vector search.

The fix is structural, not a tuning flag: give _id its own doc-values field so returning an id never touches _source. That is what the vectordb_columnar mode we are building automatically does. We are calling it out here because good benchmarking means naming your own costs as clearly as anyone else's, and this one is a document-retrieval cost sitting inside a vector-search score.

Elasticsearch vs Qdrant vector search benchmark: our numbers, on the same box

Elasticsearch on the same three m6g.large nodes, vectordb_document with bfloat16, warm, shown both as it runs today and with the id isolated:

visit %

recall@100

QPS (default)

QPS (id-isolated)

1

0.894

44

89

2

0.939

39

73

3

0.956

35

56

5

0.970

31

40

The id-isolated column is measured by dropping the stored-fields fetch entirely, which is a near-upper-bound proxy for a doc-values _id (the doc-values read is cheap but not literally free). At around 0.96 recall, id-isolated Elasticsearch does about 56 QPS, sitting right alongside Qdrant's 53 to 67 on identical hardware. Same ballpark. The gaps in both posts came from setup and document retrieval, not from the vector engine: segment counts, the id fetch, the hardware. Their reply we can account for in full here; the larger 7x from our own first post we cannot yet, and we take that up below. When you control for those, two well-built systems doing the same work on the same box land close together, which is what you would expect.

So here is the plain claim: performance depends on the setup. Match Qdrant's slow, single-threaded ingestion, the one that gives them their 67-segment configuration, and the two engines are on par on the search itself: id-isolated Elasticsearch does about 56 QPS against their 53 to 67 at the same recall. But that low segment count is bought with slow ingestion. Let both engines ingest fast, which is the natural default, and Qdrant lands back at 128 segments and about 35 QPS, while Elasticsearch degrades less as the segment count climbs, because the IVF layout behind bbq_disk is less sensitive to segment count than HNSW. So with default, fast ingestion, Elasticsearch is faster at search too, and it reached a queryable index faster to begin with. The one place we still trail is returning the ids, which is document retrieval rather than search, and it is exactly what vectordb_columnar removes.

We deliberately did not run the 4 vCPU / 16 GB tier. It lifts both engines together and shows the identical trend, so it would add cost without adding insight.

Our original 7x Elasticsearch vs Qdrant benchmark number, explained

The rule in this post is that every claim gets a number and a mechanism. That rule was also applied to our original benchmark.

That post measured a 7x throughput advantage for Elasticsearch, and the number is real for that configuration. The setup was disclosed in full, including that Qdrant does better on NVMe. But the post attached a mechanism to the number: Qdrant was bottlenecked by random disk reads of the original vectors during rescoring, a problem made worse by network-attached storage. The arithmetic in this post undercuts that explanation. Those round-1 nodes had 26 GB of RAM, more headroom than the 8 GB nodes here where we watched the disk sit at 0 IOPS, so if the disk was idle here, it was almost certainly idle there too.

So what actually held round-1 Qdrant to 4.5 QPS at 0.97 recall? We cannot claim to know yet, but the original post's own configuration points the way: it ran Qdrant on 2-bit quantization with oversampling pinned at 1. Two-bit codes are coarse, and with oversampling fixed at 1 the only lever left to recover recall is ef. Reaching 0.97 recall on ef alone means a very large ef, and a very large ef makes each query expensive on its own, before disk enters the picture at all. That is still a hypothesis, but the mechanism the original post named, random disk reads during rescore, does not survive the same arithmetic we just applied to Qdrant.

Benchmarking, not benchmarketing

The numbers reproduce; the explanations don't. A benchmark that made "disk reads" its headline ran with the disk asleep, and we watched it sit at 0 IOPS for the whole window. The honest differences we could actually find were a segment count (an ingest-speed-versus-query-speed tradeoff where the IVF layout lets bbq_disk ingest fast and query fast) and a stored-field retrieval cost on our side (document retrieval, not vector search, and something vectordb_columnar removes). None of it is io_uring, and none of it is "Java is heavy."

The original and arguably bigger point got lost in the io_uring discussion: putting vectors on disk was never really about the rescore, it is about memory. The key was keeping the searchable index, the quantized vectors and the IVF or HNSW structure, compact and disk-resident enough to serve more vectors per gigabyte of RAM than a design that pins everything in memory. Qdrant's setup pins the 4-bit vectors with always_ram and keeps the HNSW graph in RAM, about 3.6 GB per node for 21 million vectors. bbq_disk keeps the quantized IVF on disk. At 21 million vectors on 8 GB nodes, everything fits in RAM either way, which is precisely why this benchmark cannot tell the two designs apart. It is measuring the case where the interesting variable has been held constant.

The interesting question is what happens when the searchable index stops fitting. Scale the corpus until the quantized vectors and the structure exceed page cache, and the two designs diverge: one keeps serving from disk, the other needs more RAM per node. That is the regime bbq_disk was built for, and it is the one neither post has measured. The rescore-under-disk-pressure case, many more distinct queries than fit in cache, is worth measuring too, but it is the second question, not the first.

We tried to get there, and our first attempt still fit in cache, because 21 million vectors on these nodes do not spill. So we are not done. We are building a benchmark with a corpus large enough that the searchable index no longer fits in RAM, and we will publish those numbers, with the reasons attached and checked the same way.

The ask here is to the reader, not to Qdrant: when a benchmark hands you a clean multiplier, chase every number back to a cause before you believe the story around it. Half the time it is a warm cache or a segment count.

The Qdrant team built a good engine, and we're not questioning that. The honest verdict is not about who is faster: matched to their setup, the two are on par, and on the fast default path, Elasticsearch is quicker for both indexing and searching. The numbers were fine. The reasons attached to them were the part worth checking, starting with our own. And the benchmark that would actually stress a disk-backed index, a working set that does not fit in RAM, is still to be written. We will bring the numbers.

Prerequisites

• Elasticsearch 9.x with jina-embeddings-v5-text-small inference endpoint available.

• Python 3.10+,

• Elasticsearch API key,

What is quantization?

An embedding is a list of numbers. By default, each number is a float32, which uses 4 bytes. Quantization stores each number with fewer bits, trading precision for space.

Like a JPEG, a quantized vector is a smaller, lower-fidelity copy of the original that still gets the job done.

Name

Bytes / dim

1024-d vector

Compression

`Float` (Baseline)

4

4096 B

1x

`int8`

1

1024 B

4x

`int4`

0.5

512 B

8x

`bbq`

~0.14

142 B

~29x

What is BBQ?

Better Binary Quantization (BBQ) is Elasticsearch's 1-bit quantization mode for dense vectors. Each dimension of the vector is stored as a single bit, plus a few corrective bytes per vector. Then, a rescoring step is applied at query time. This keeps the final retrieval quality close to a full precision search.

For the math behind each level, see Scalar quantization 101, Optimized Scalar Quantization, and the BBQ deep dive.

How does BBQ preserve search accuracy?

Plain 1-bit quantization leads to too high a search quality degradation on its own. BBQ maintains high retrieval quality through three mechanisms:

1. Asymmetric precision: Stored vectors use 1 bit per dimension.

2. Corrective factors: A few floats per vector record the rounding error and correct distances at scoring time.

3. Oversample and rescore: BBQ scans candidates with the bits and then reranks the top ones with higher precision. Fetching the top 10 means scanning about 30 candidates.

The result is the vectors that are roughly 32x smaller, with retrieval quality close to full precision. In the next section of the article, we’ll measure the memory savings and the recall on a real corpus.

How Jina embeddings v5 works

Jina embeddings v5 is a multilingual embedding model with quantization-aware training, which makes it a natural fit for BBQ in Elasticsearch: The 1024-dimensional vectors from jina-embeddings-v5-text-small sit above the dimensional floor where binary quantization stays accurate, and the model is trained so that 1-bit quantization loses little quality. Its main features are:

• One model for many tasks: v5 uses small Low-Rank Adaptation (LoRA) adapters on top of a single base model, one for each task: retrieval, text-matching, clustering, and classification. Elasticsearch picks the right adapter automatically at index and query time.

• Matryoshka dimensions: v5 is trained so you can truncate the vector (1024, 512 to 256) and minimize search quality reduction. This is another way to shrink vectors, independent of quantization.

• Quantization-aware training: v5 is trained to work with BBQ, so its 1-bit vectors lose little accuracy.

We use jina-embeddings-v5-text-small. This model is available through Elastic Inference Service (EIS) and outputs 1024 dimensions with 32k token context and is multilingual across 93 languages. That puts it above the 384-dimension threshold, below which Elasticsearch no longer defaults to bbq_hnsw.

Full model details are in the Jina v5 article on Search Labs.

Setting up the BBQ vs. float32 comparison

We’ll create two indices: Both share mappings, and what changes is the index_options.type parameter, which tells Elasticsearch how to store the dense vector field (as raw float32 HNSW or as 1-bit BBQ):

Index

`index_options`

Loaded into memory

`vectors-float32`

`hnsw`

Raw `float32` with no quantization (baseline)

`vectors-bbq`

`bbq_hnsw`

1-bit BBQ quantization + corrective factors

We then embed the corpus once with Jina v5, index those same vectors into both, and compare them on disk usage, memory footprint, and recall. You can follow along with the full supporting blog content notebook.

Connect to Elasticsearch

Create the two indices

Note: In production, you can use semantic_text to let Elasticsearch manage the mapping and inference endpoint automatically.

Point at the Jina v5 inference endpoint

We call the model jina-embeddings-v5-text-small directly (no need to create an inference endpoint) to turn text into vectors.

As result of the test, we got:

Load a multilingual news dataset

We stream real news articles from hotchpotch/multilingual_cc_news, a parquet mirror of CC-News. We take about 1,000 articles from five languages (around 3,000 docs total), plus a small held-out set of headlines to use as search queries. Using multiple languages also lets Jina v5 show its multilingual strength.

Generate the embeddings and bulk index

We embed the corpus a single time and feed those exact vectors into both indices.

We force-merge to a single segment so the storage numbers are stable and comparable.

Results: Disk versus memory

The disk usage API reports how many bytes each index spends on vectors (knn_vectors).

Result:

On disk, the two indices are about the same size. A quantized index still keeps the raw float32 vectors (needed for rescoring and requantization during merges) and adds the 1-bit vectors on top, so BBQ ends up slightly larger on disk.

The real savings is in memory. The HNSW scan only needs the 1-bit vectors in RAM, while the raw floats are read from disk to rescore the top candidates. We size that footprint using the documented kNN memory formulas: float uses num_vectors × dims × 4 and bbq uses num_vectors × (dims/8 + 14).

BBQ's extra bytes on disk should match the 1-bit payload we computed for memory. Here, that’s 13.25 - 12.80 = 0.45 MB versus the computed 0.44 MB. They line up.

Results: Recall

To check whether the quantized index returns results similar to the float baseline, we use recall:

recall@k = | BBQ top-k ∩ float32 top-k | / k, averaged over all queries.

We vary the oversampling factor (num_candidates / k) that’s the number of candidates BBQ scans with 1-bit vectors before reranking the top ones against the original floats to find the lowest value that still matches float32.

As a result, we have:

BBQ starts at 0.994 recall@10 at 1x oversampling, holds there up to 3x, and then settles at 0.989 at higher factors, meaning it returns at least 98.9% of the same top-10 documents as float32 across all oversampling values. For more on how recall varies across datasets under quantization, see Fast vs. accurate: Measuring the recall of quantized vector search.

BBQ quantization results summary

The same vectors, two storage formats, and one experiment:

• Disk: Roughly the same (12.80 MB versus 13.25 MB). BBQ keeps the raw floats around for rescoring and merging.

• Memory: 29x smaller (12.71 MB versus 0.44 MB). This is the number that decides whether your cluster fits the corpus.

• Recall@10: 0.994 at 1x oversampling. Quantization-aware training pays off.

When to enable BBQ: If your dimension count is above the 384-dim floor, if your vectors are the dominant memory cost, and if you can afford a few extra candidates to rescore. For Jina v5 specifically, the model is trained for it, so the recall hit on most corpora is small.

Further reading on BBQ and vector quantization

• Run the full notebook from this article in the supporting blog content repo.

• For the math behind BBQ, see Better Binary Quantization in Lucene and Elasticsearch.

• For more on Jina v5's architecture, see Jina embeddings v5 on Search Labs.

• For a broader walkthrough on adopting BBQ, see How to implement BBQ into your use case.

• For the original research behind BBQ, see the paper RaBitQ: Quantizing High-Dimensional Vectors with a Theoretical Error Bound for Approximate Nearest Neighbor Search.

Vector search is a critical foundation for large language model (LLM) applications, retrieval augmented generation (RAG), and other AI workloads. In this benchmark, Elasticsearch achieved up to 7x higher throughput than Qdrant at comparable recall on the same storage topology. Elasticsearch as a vector database offers strong vector search performance even when network-attached persistent storage remains on the query path.

The difference reflects how the two systems interact with disk. Elasticsearch DiskBBQ is designed to keep vector search efficient when persistent storage remains on the query path, using a compact quantized representation and limiting costly access to full precision vectors during search. In this setup, Qdrant relies on a graph-based search path with rescoring against original vectors stored on disk. On network-attached persistent storage, that random access cost becomes much more significant, which is why the performance gap widens as recall increases. This benchmark therefore focuses specifically on network-attached persistent storage, a common deployment model in managed cloud and Kubernetes environments.

The key pattern in the latency curve is not only the size of the gap but also its shape. Elasticsearch latency remains comparatively flat as recall increases, suggesting that higher recall doesn’t require a dramatic increase in expensive storage activity. Qdrant’s latency rises sharply as hnsw_ef increases, which is consistent with broader candidate exploration leading to more rescoring work against original vectors on disk.

Full results table

The table below shows the full parameter sweep for both Elasticsearch and Qdrant. Because the two engines expose different tuning controls for vector search, the results are reported using each engine’s full parameter key rather than attempting a one-to-one mapping between settings.

A few notes on the metrics:

• ParamKey: The complete parameter setting used for a given run.

• Recall: Recall@100 against a ground-truth top-100 result set for the benchmark queries. Values range from 0 to 1, and higher is better.

• Latency_Avg: The average end-to-end latency per query measured from the benchmarking client across the full run, in milliseconds. Lower is better.

• Latency_P95: The 95th percentile query latency, in milliseconds, showing the upper range of typical slow queries. Lower is better.

• Throughput: The average number of queries processed per second across the full run. Higher is better.

Engine

ParamKey

Recall

Latency_Avg

Latency_P95

Throughput

qdrant

hnsw_ef=50, oversampling=1, size=100

0.8694

315.7849

503.4754

12.629

elasticsearch

k=100, oversample=1, size=100, visit_percentage=1

0.8789

135.0802

218.494

29.343

elasticsearch

k=100, oversample=1, size=100, visit_percentage=1.5

0.9123

127.8286

195.2318

31.1107

qdrant

hnsw_ef=100, oversampling=1, size=100

0.9287

895.9933

1213.0448

4.4493

elasticsearch

k=100, oversample=1, size=100, visit_percentage=2

0.9317

124.846

183.6314

31.8225

elasticsearch

k=100, oversample=1, size=100, visit_percentage=2.5

0.9444

123.517

180.4831

32.1883

qdrant

hnsw_ef=150, oversampling=1, size=100

0.9518

884.7236

1195.2603

4.5066

elasticsearch

k=100, oversample=1, size=100, visit_percentage=3

0.9532

123.276

183.8379

32.2364

elasticsearch

k=100, oversample=1, size=100, visit_percentage=3.5

0.9599

122.5559

184.2858

32.4469

qdrant

hnsw_ef=200, oversampling=1, size=100

0.964

883.2114

1188.6597

4.5143

elasticsearch

k=100, oversample=1, size=100, visit_percentage=4

0.965

122.7946

184.9058

32.3635

elasticsearch

k=100, oversample=1, size=100, visit_percentage=4.5

0.9689

122.7062

182.9559

32.3976

elasticsearch

k=100, oversample=1, size=100, visit_percentage=5

0.9722

122.5761

187.3536

32.4221

qdrant

hnsw_ef=256, oversampling=1, size=100

0.9722

881.9643

1185.4948

4.5192

elasticsearch

k=100, oversample=1, size=100, visit_percentage=5.5

0.9747

122.5609

184.5128

32.4176

Each row pairs the closest measured Elasticsearch and Qdrant configurations in the sweep by achieved recall.

Matched comparisons at similar recall

To make the comparison fair, speedup is calculated only between configurations that achieve similar recall. This avoids comparing settings that trade off accuracy very differently.

Recall band

Elasticsearch recall

Elasticsearch Latency_Avg

Elasticsearch throughput

Qdrant recall

Qdrant Latency_Avg

Qdrant throughput

Throughput speedup

~0.87

0.8789

135.0802

29.343

0.8694

315.7849

12.629

2.32x

~0.93

0.9317

124.846

31.8225

0.9287

895.9933

4.4493

7.15x

~0.95

0.9532

123.276

32.2364

0.9518

884.7236

4.5066

7.15x

~0.96

0.9599

122.5559

32.4469

0.964

883.2114

4.5143

7.19x

~0.97

0.9722

122.5761

32.4221

0.9722

881.9643

4.5192

7.17x

This matched-recall view is the clearest expression of the underlying systems difference. At similar recall levels, Elasticsearch delivers both lower latency and much higher throughput, and the gap widens as recall rises. The recall-throughput pattern matters because higher recall in this benchmark requires broader search. DiskBBQ absorbs that increase with relatively little additional cost, while Qdrant’s graph plus rescoring path becomes much more constrained by random access to original vectors on persistent storage.

Benchmark methodology

Jingra, the benchmarking tool used for these tests, was originally written in Python and has since been rebuilt as a Java project. For these tests, Jingra runs in a Kubernetes pod within the same cluster as the engine being measured. This helps reduce external network variability and keeps the test environment consistent across runs. For each run, Jingra executed the query set at a fixed client concurrency, recorded end-to-end client-side latency and throughput, and computed recall against a precomputed ground-truth top-100 set.

This benchmark was intentionally run on network-attached persistent storage rather than local NVMe. For the published results, the storage used the baseline performance allocation for a 200 GiB GCP Hyperdisk Balanced volume, with no explicit IOPS or throughput provisioning. We chose this topology on purpose because it’s a relevant cloud deployment model and because it keeps storage efficiency materially on the query path.

Qdrant often performs better on local NVMe, so deployments using local NVMe should expect different results than the ones shown here. This benchmark specifically tests network-attached persistent storage because that’s a common managed-cloud deployment model and because it makes storage-path efficiency visible in end-to-end query performance.

Because Elasticsearch and Qdrant expose different query parameters for controlling vector search behavior, there’s no clean one-to-one mapping between their tuning settings. Instead of comparing equivalent parameter values directly, we use recall as the primary point of comparison. The matched comparisons below therefore pair configurations that achieve similar recall, rather than configurations with superficially similar parameter values.

Recall cannot be known in advance for a given parameter setting, so we sweep across a range of search configurations for each engine and then compare results at similar recall levels. In the published results, oversampling was fixed at 1 for both engines so that recall was primarily tuned via search breadth rather than rescoring expansion.

How does Elasticsearch configure vector search?

• query_vector: The input vector used for similarity search. Elasticsearch compares this vector against the stored vectors in the field.

• k: The number of nearest neighbors to retrieve.

• visit_percentage: Controls how much of the DiskBBQ, Elasticsearch’s disk optimized vector index, is explored during the approximate search phase. Higher values usually improve recall but increase latency.

• oversample: Controls how many extra candidate vectors are passed into rescoring relative to k. Higher values can improve recall, but usually at additional cost.

• size: The number of hits returned in the final response.

• _source: false: Disables returning the document _source field, reducing response size and avoiding extra retrieval overhead during benchmarking.

Example

Params

We keep k = size = 100 so the search request is aligned with the benchmark target: returning the top 100 results. To improve recall, we tune visit_percentage rather than inflating the final result count, while keeping oversample = 1 fixed across runs.

How does Qdrant configure vector search?

• query_vector / vector: The input vector used for similarity search. Qdrant compares this vector against the stored vectors in the collection.

• size / limit: The number of nearest neighbor results returned in the response.

• with_payload: false: Disables returning payload fields, reducing response size and avoiding additional retrieval overhead during benchmarking.

• with_vector: false: Disables returning stored vectors in the response, again reducing response size and keeping the benchmark focused on search performance.

• hnsw_ef: Controls the number of candidates explored during HNSW search. Higher values usually improve recall but increase latency. Like visit_percentage in Elasticsearch, it affects search breadth, but the two controls are engine-specific and not directly equivalent.

• quantization.rescore: true: Enables rescoring of the candidate set using the original vectors after quantized search.

• oversampling: Controls how many extra candidates are considered during rescoring relative to the final result count. Higher values can improve recall, but usually at additional cost.

Example

Params

We keep size = 100 so that each request is aligned with the evaluation target, in this case top 100 retrieval. Recall is then tuned by sweeping hnsw_ef, which controls how many candidates are explored during search. Higher hnsw_ef values generally improve recall but also increase latency and reduce throughput. We keep oversampling = 1 fixed across runs so that the main tuning variable is the search breadth rather than the rescoring expansion.

Cluster setup and DiskBBQ configuration

We ran the benchmark on GCP using three n4-standard-8 nodes, with each pod allocated 7 vCPUs and 26 GB of RAM, and using 200 GiB GCP Hyperdisk Balanced volumes at baseline performance allocation. The corpus contains 21 million vectors, (see dataset section below for more details and download links), which account for about 60.1 GiB of raw float vector data. With 2-bit quantization, the vector payload drops to roughly 3.8 to 4.0 GB. However, the full index footprint is much larger once graph and other index structures are included. That means the workload remains meaningfully sensitive to network-attached storage performance, especially because exact vector values still need to be read from disk during rescoring.

We chose this node size intentionally to keep the benchmark in a regime where network-attached persistent storage remains on the query path rather than allowing the full working set to remain comfortably memory-resident. Each system was therefore configured using the best-performing setup we identified for this workload within the tuning scope described in this post. In Elasticsearch, this meant bbq_disk. In Qdrant, the original vectors were stored on disk, while the 2-bit quantized representation used for approximate search was kept in RAM with always_ram: true. Because the two systems expose different search strategies and tuning controls, we compare them at matched recall rather than trying to map parameters one to one.

Elasticsearch was configured to use DiskBBQ, its disk-optimized approach for approximate nearest neighbor vector search, with 2-bit quantization. DiskBBQ uses aggressive quantization to keep the searchable index compact and then rescores with the original vectors to preserve accuracy. This helps maintain strong recall while keeping disk-based search efficient.

bbq_disk is an Elasticsearch Enterprise feature. We used it here because the goal of this benchmark was to compare the strongest disk-oriented vector search configuration available in each engine for this workload, rather than licensing tiers or default features.

We didn’t include bbq_hnsw in this comparison because the benchmark was specifically designed to evaluate disk-oriented vector search under a disk-sensitive workload.

This storage topology matters because Qdrant’s rescore step reads the original float32 vectors from disk with random access on each query. On local NVMe, those reads are much faster, and Qdrant correspondingly performs better. On network-attached persistent storage, the results are consistent with that random-read rescore path becoming a more important bottleneck. Qdrant latency rises sharply as hnsw_ef increases, while Elasticsearch remains comparatively flat across the same recall progression.

We chose 2-bit quantization because Qdrant couldn’t reach the target recall range with 1-bit binary quantization. Since the two systems expose different disk-oriented vector search strategies, we tuned each one to the strongest configuration available within its current feature set.

Both systems were configured with three shards distributed across the three nodes and with two total copies of each shard in the cluster. In Elasticsearch, number_of_shards: 3 and number_of_replicas: 1 means one primary plus one replica, for two total copies. In Qdrant, shard_number: 3 and replication_factor: 2 also means two total copies, since Qdrant’s replication factor refers to the total number of copies rather than the number of additional replicas. So although the field names differ, the effective replication level was the same in both systems.

Setting

Elasticsearch

Qdrant

Shards

number_of_shards: 3

shard_number: 3

Copies

number_of_replicas: 1 (1 primary + 1 replica = 2 total)

replication_factor: 2 (2 total)

Elasticsearch mapping

Qdrant mapping

Dataset

For this benchmark, we used the kenhktsui/wiki_dpr_e5 dataset from Hugging Face, a large-scale Wikipedia passage retrieval dataset designed for dense vector search. The corpus contains 21 million embedded passages, each represented as a 768-dimensional float32 vector, or 3,072 bytes per vector. That corresponds to about 60.1 GiB of raw vector data, before accounting for additional fields and file format overhead in the source dataset. The downloadable data.parquet file is larger at 85.2 GB for that reason.

We chose this dataset because it reflects a common production pattern in LLM, RAG, and retrieval systems: searching a large corpus of semantically embedded text while balancing recall, latency, and throughput. At 21 million vectors and roughly 60 GiB of raw vector data, it’s large enough to make disk-based vector search a relevant operating mode to evaluate.

Both engines used 2-bit quantization, reducing each vector from 3,072 bytes to 192 bytes, a 16x reduction that brings the quantized vector corpus to around 4 GB. In Qdrant, that quantized representation was kept in RAM for search, while the original vectors remained on disk. Even so, the workload remained meaningfully sensitive to network-attached storage performance because rescoring still required access to the original vectors on disk.

You can download the dataset and query files from the links below:

• data.parquet

• queries.parquet

Jingra and recreating the benchmark

For this benchmark, we used Jingra v0.2.3 with the configurations described es-9.4-vs-qd-1.18-vector-search. Jingra handled data loading, query execution, parameter sweeps, and metric collection for both Elasticsearch and Qdrant, making the benchmark repeatable and easier to compare.

To reproduce the experiment, you need the published dataset, query set, engine configurations, and comparable cluster hardware. With those in place, Jingra can rerun the benchmark and generate similar recall, latency, and throughput measurements shown in this post.

Conclusion

At comparable recall levels, Elasticsearch DiskBBQ consistently delivered faster vector search than Qdrant in this benchmark, with higher throughput and lower latency across the recall range we tested. These results are especially notable because the comparison was made on network-attached persistent storage, where efficient storage-aware vector search becomes critical. Elasticsearch as a vector database allows organizations to achieve high recall with lower latency and higher throughput on slower persistent storage.

Just as importantly, this benchmark highlights the value of comparing engines at matched recall rather than by nominal parameter settings. Elasticsearch and Qdrant expose different controls, so the fairest comparison isn’t parameter to parameter but outcome to outcome. Across the recall range tested here, Elasticsearch maintained a clear advantage in both latency and throughput.

If you want to reproduce the experiment yourself, we’re publishing the dataset and query set used in this benchmark so others can validate the results and build on them.

Further reading:

• Introducing a new vector storage format: DiskBBQ

• Elasticsearch BBQ vs TurboQuant

Vector Search on Elasticsearch Serverless

Elasticsearch Serverless is built on Stateless Elasticsearch, a fully decoupled compute and storage architecture where index data lives in remote object storage and search nodes maintain only a local cache. For vector search to be fast on this architecture, the scoring engine needs to work directly with the local cache, not copy it to the heap first.

Elasticsearch simdvec is the engine behind every vector distance computation in Elasticsearch. It provides hand-tuned AVX-512 and NEON kernels, bulk scoring with explicit prefetching, and off-heap memory access that keeps data flowing from storage straight to CPU registers. On stateful Elasticsearch, simdvec has always had a direct fuel line: Memory-mapped files feed native pointers straight into SIMD intrinsics. On serverless, the data was sitting right there in the blob cache's memory-mapped regions, in exactly the right form, but there was no path connecting it to the scoring engine.

We've now built that path. simdvec runs on Serverless with the same off-heap, native SIMD scoring as stateful. And because serverless gives us control over the entire storage layer, this is just the beginning.

Premium fuel only: why simdvec requires off-heap memory for vector scoring

simdvec's speed comes from working directly with off-heap memory. It takes a native pointer to memory-mapped data and passes it straight to C++ SIMD intrinsics. No intermediate copies, no heap allocations. The data flows from storage straight to CPU registers. This matters more than it sounds: simdvec's kernels process vectors faster than the data can be copied, so any copy in the path becomes the bottleneck, not the scoring itself.

On stateful Elasticsearch, this just works. Lucene memory-maps index files from local disk, and the scorer extracts a native pointer directly from the mapped region. This is the path that delivers the benchmark numbers we've published, and it's what we wanted to bring to serverless. To see how, we first need to understand how serverless stores and accesses data.

The serverless blob cache: how Elasticsearch stores vector data

In the stateless architecture, the primary copy of all index data lives in remote object storage, such as S3. Each search node maintains a local cache (called the blob cache) that keeps recently and frequently accessed portions of the index data on local SSD. The frozen tier on stateful Elasticsearch uses the same architecture: Searchable snapshots are backed by a similar blob cache that memory-maps regions from remote storage onto local disk. When a search hits cached data, it's served from fast local storage. When it misses, the blob cache fetches the data from the remote store and caches it for future queries.

The blob cache is organized into fixed-size memory-mapped regions, 16MB by default. It manages its own lifecycle: tracking which regions are in use, applying a least-frequently-used eviction policy when the cache is full, and reference counting to ensure regions aren't evicted while being read. The regions are still memory-mapped through the OS, but the blob cache controls which regions exist, which are populated, and when they're reclaimed. On stateful, those decisions are left entirely to the OS.

Crucially, because each region is memory-mapped, the blob cache already holds vector data in exactly the form simdvec needs. But before we built the connection, there was no way to get at it. Every vector comparison was copied into a heap array and handed to a slower scorer. No direct memory pointers, no SIMD, and garbage collection pressure on every call.

Unified scoring: one SIMD path for all storage tiers

We introduced a new abstraction that lets the scorer safely borrow direct memory from whatever storage layer is underneath, just long enough to run the SIMD computation. If the data is available as direct memory, simdvec's native kernels run. If not (data not yet cached or spanning a region boundary), the scorer falls back to a heap copy. In practice, the fallback is rare.

This gave us a single scoring entry point across all tiers:

1. Stateful (local disk): The scorer extracts a native pointer from the OS memory map.

2. Blob cache (serverless, frozen tier): The scorer borrows a direct memory slice from a cache region.

3. Fallback: The scorer copies bytes to the heap. Rare in practice.

The scorer doesn't know which tier it's running on, and it doesn't need to. It also means we no longer maintain separate scoring implementations; previously, there was a fast native path for stateful and a slower path for everything else. Now every improvement to simdvec benefits all tiers automatically, including its most powerful capability: bulk scoring.

Bulk vector scoring across blob cache regions

A single query may score thousands of candidate vectors. simdvec's bulk scoring processes these in batches with multi-accumulator inner loops, query amortization, and cache-line prefetching, up to 4x faster than single-vector alternatives when data exceeds CPU cache.

Search over an Inverted file (IVF) index is where bulk scoring has the most impact. The query selects a set of candidate posting lists and sweeps through the quantized vectors, scoring them in large batches against the query vector. On stateful, those vectors live in one contiguous memory-mapped file, so bulk scoring resolves them with straightforward pointer arithmetic and scores a batch in a single native call.

On serverless, a sweep through a posting list may cross blob cache region boundaries. We extended the direct memory abstraction with a bulk access method that resolves multiple vector offsets to their respective cache regions in a single call. If all vectors in the batch are cached and none cross a region boundary, the scorer gets a direct memory slice and passes the whole batch to simdvec's native bulk kernel with the same prefetching and pipelining as stateful. When a vector does cross a boundary, the system falls back to per-vector scoring: still zero-copy, just without the batching benefit. With 16MB regions and 1024-byte vectors, that happens roughly once every 16,000 vectors.

simdvec's bulk scoring architecture, the key differentiator highlighted in the simdvec benchmarks, now operates on serverless with the same characteristics that make it fast on stateful. So how does it perform in practice?

simdvec on Elasticsearch Serverless: vector search lap times

We benchmarked with an 18 million vector MSMARCO dataset at 1024 dimensions, using IVF with Better Binary Quantization (BBQ) 1-bit quantization. All results are on a warm blob cache with the full dataset resident in local cache regions, so we're measuring the scoring path rather than remote fetch latency.

Throughput. Under concurrent load, search throughput nearly doubled, jumping from 398 to 739 ops/s. Single-client gains were 23-39%, but the real difference shows up under concurrency: The improvement was 2-3x larger because eliminating heap copies removes the GC pressure and allocation contention that previously throttled concurrent scoring.

Tail latency. The direct memory path transformed tail latency under load:

• p99.9 dropped from 237 ms to 30 ms (87% reduction).

• p99.99 dropped from 9.1 seconds to 55 ms (99.4% reduction).

p100 dropped from 11.4 seconds to under 100 ms.

The worst-case outliers that previously took seconds now complete in tens of milliseconds. The heap-copy-induced queueing that caused latency spikes is gone.

Recall is identical. The same vectors are scored, producing the same results. And we're just getting started.

Beyond parity: what Elasticsearch Serverless can do for vector search that stateful can't

Reaching parity with stateful was the goal. But the more interesting realization is what the stateless architecture lets us do that stateful can’t.

On stateful, the OS controls memory-mapped file behavior: which pages stay resident, when to evict, how aggressively to read ahead. The application can offer hints, but they apply to entire file mappings, and the kernel may ignore them. Worse, search and indexing happen concurrently on the same node, so a hint that benefits one access pattern can hurt another. In practice, to balance different needs, you have to be conservative.

On serverless, two things are fundamentally different. The blob cache manages its own memory-mapped regions with full application-level control. And serverless separates indexing and search onto dedicated tiers: Search nodes never merge, indexing nodes never serve queries. No conflicting access patterns means we can be aggressive with memory advice. Here’s what we’re working on:

• Per-region memory advice. The blob cache knows what type of data each region holds. It can issue random-access hints for rescoring regions, where raw float32 vectors are read in unpredictable order and the kernel’s default readahead would waste memory on pages that will never be used. It can apply sequential readahead for scans through quantized vectors. On the indexing tier, merges read data sequentially, so aggressive readahead brings pages in before they're needed, with no risk of harming concurrent random reads that simply aren't happening on that node.

• Cache-aware prefetching. simdvec already prefetches at the CPU cache-line level. On serverless, we can coordinate this with the blob cache's knowledge of region residency, prefetching at multiple levels: remote store to cache, OS pages to RAM, and cache lines to CPU. The blob cache can tell the scorer which regions are resident before scoring begins, avoiding work on data that would trigger a remote fetch.

• Workload-aware eviction. The blob cache can prioritize retaining data that vector search depends on: IVF centroid indexes that are checked on every query or quantized vectors that are scored in bulk, over data that's accessed infrequently. The OS page cache evicts based on generic heuristics with no understanding of what the data represents. On serverless, eviction policy can be tuned to the workload.

The blob cache gives us a level of control over the memory hierarchy that the OS page cache simply can’t. This is why we see serverless as the most promising platform for the next generation of vector search performance work. Not just matching stateful, but surpassing it. And vectors are just the beginning.

Vector search on Elasticsearch Serverless: what we shipped and what's next

simdvec now runs everywhere Elasticsearch runs (stateful, serverless, and frozen tier) with the same native SIMD scoring, the same bulk scoring, and the same off-heap efficiency. The abstraction we built is general-purpose and already wired through every layer in the storage chain, so the same approach could benefit term lookups, aggregations, sorting, and stored field retrieval in the future.

Elasticsearch Serverless is where we're investing most heavily in vector search performance. Every improvement to simdvec, every optimization to the blob cache, and every new storage-level improvement lands here first. If you're choosing where to run your vector workloads, serverless is the platform that keeps getting faster. You can get started with a free Elastic Cloud trial.

How DiskBBQ uses two centroid tiers for asymmetric quantization

DiskBBQ now uses two centroid tiers (fine-grained document centroids and coarser query centroids) so queries are quantized once per parent centroid instead of once per document centroid.

The old mental model is "one centroid does everything for a posting list." The new model splits responsibilities:

• Document centroids (fine-grained): Still used for posting-list structure and document centering.

• Query centroids (coarser): A parent centroid reused across multiple document centroids.

So instead of quantizing the query independently for every document centroid we visit, we quantize per parent centroid and reuse that work across all of its children. Since we were already using two-tier clustering logic as the index size grew, it was a natural fit. We can reuse the work we already do during querying.

These images are a simple representation of our goal: Quantizing per centroid gives us overhead per centroid. Let’s get rid of it!

The goal is to significantly reduce the number of times we actually need to quantize a given query.

The math behind asymmetric BBQ in Elasticsearch

To center the data prior to computing quantized query and document vectors, and , we rewrite the dot product as and expand. We can perform exactly the same operation but using different centroids for the query vector and document vector . Specifically,

As for standard Better Binary Quantization (BBQ), we quantize and in order to estimate the per (document, query) pair component of the dot product. The quantities and are scalars so just two extra additions per dot product we compute. For , we compute naturally when finding the nearest centroid. For , this can be stored with the quantized document vectors, which are just 4 bytes overhead. Below, we’ll discuss how to manage the other term on the fly.

Asymmetric BBQ in DiskBBQ

We cluster the document centroids (using k-means, for example) into clusters, for and the query and document centroid count, respectively. This means there’s a many-to-one mapping from document centroids to query centroids. We’ll denote the document centroids by their index and define this mapping to the query centroids as →.

Since there’s a unique query centroid for each document centroid, we only need to cache one value for per quantized document vector, that is, for each document vector in posting list , we need to cache with the quantized document vector.

When we come to compute the dot products between a query and the document vectors in a cluster, we look up the quantized query vector corresponding to and we compute once and use it to process the whole posting list. The quantization process is significantly more expensive than computing the dot product, so this is a big net win.

The term is estimated using the usual BBQ machinery, that is, these vectors will be quantized and the dot product value estimated from the quantized vectors. Then we can use (1) to compute the final dot product estimate. Notice that this means we only need to quantize the query at most times. Furthermore, we typically visit many centroids from the same parent centroid in a search because they’re close to one another.

Euclidean distance corrections for asymmetric quantization

For Euclidean, we can write and treat the term exactly as above. In fact, there’s a slightly nicer form. Substituting, we have that:

We can rewrite this as follows:

The corrective terms are the norm of query vector minus the document centroid , the norm of the document vector minus the query centroid , and the norm of the difference of query and document centroids. As before, can be stored as a single float with each document.

What changed in DiskBBQ indexing and scoring

At indexing/merge time, centroids can be clustered into parent groups when centroid count is large enough. Posting metadata moved from "centroid ordinal + centroid score" to a shape that explicitly carries query-centroid ordinal and document-centroid score. That decoupling is what lets scoring read documents and query centering from different places. For Euclidean, let’s break it down further by our mathematics above:

<- This is the distance from a “query vector ” to “document centroid ”. We already gather this when we find the nearest centroids during querying. No new work.

<- This is the distance from “document vector ” to “query centroid ”. However, recalling our original quantization work, this can simply replace a previously stored float value. No new storage is required.

<- This is just the distance between query centroid and document centroid . This is just a single extra floating point value per postings list.

The practical change for dot product spaces is even simpler; the only correction value change is being stored instead of .

These changes don’t introduce new computation costs and marginally reduce storage costs because we no longer quantize queries with document centroids. Those raw centroids don’t need to be present with the posting lists.

One cost we did add is a small cache of quantized query values. This is to account for clustering edge cases. For example, it's possible that query is very close to query centroid but not quite as close as . That said, the actual nearest three document centroids could have a relative order: . So, to prevent the query from being quantized twice, we keep a limited cache of the most recent quantized values for a given query.

Here’s a visualization of the situation described above. In the typical iteration scenario, we don’t want to risk unnecessarily quantizing the query against the same query centroid multiple times.

DiskBBQ asymmetric quantization: performance results

The flame graphs below show a before and after comparison. Before, about 20% of the time was spent quantizing queries when we visited each cluster. After our adjustment, it dropped to about 4%.

Of course, the bulk of the cost is still just scoring the vectors in each cluster. But every little bit helps.

Here’s a better view of the full end-to-end performance and recall. The data set was 1 million DBpedia docs encoded with the GTE-Base model. Here, “sec” indicates the number of clusters per secondary (parent) cluster. Note that symmetric quantization is still impacted by the secondary cluster size as it also impacts the two-tier clustering indexing we do already.

However, the impact on our current index structure is still dominated by centroid scoring and scoring vectors in the cluster. Asymmetric quantization removes a frustratingly expensive part of our scoring overhead, but the impact isn’t dramatic given our current structure.

What's next for DiskBBQ quantization

This simple piece of mathematics decouples our query quantization from our document quantization, giving us better storage efficiency and faster queries. This is in Elasticsearch Serverless now and will be in Elastic Stack version 9.4.0.

This now means that query quantization time isn’t a direct concern for future decisions. We can make larger index changes without worrying about the consistent overhead of quantization directly with document centroids.

This was a nerdy one. I hope you survived all the math (and that I copied it all down correctly). It’s always fun to be able to tackle complex problems with simple mathematics, and the results are actually positive in real use cases and data.

What is recall?

Recall measures on a scale from 0 to 1 how many of the documents that your users actually want appear somewhere in your search results. If a query should surface three products and your search returns only two of them in the top 10, recall@10 = 0.67 for that query. It’s a set-based metric: It doesn’t care about the position of the relevant documents within those k results. A relevant document in position 10 counts the same as one in position 1. Having a high recall means that you’re not losing relevant results.

The diagram shows two sets: all relevant documents (left) and what BM25 actually retrieved (top 10, right). Only the intersection counts toward recall, prod_1 and prod_2 were found, while prod_3, prod_4, and prod_6 were missed entirely. Result: Recall@10 = 2/5 = 0.40.

Prerequisites

Let's get down to business to better understand how recall works. This demonstration uses Python. You can follow along with it on the companion notebook (notebook.ipynb), where every code block is a cell ready to run.

The code provided uses the following:

• Elasticsearch 9.3+

• Python 3.10+

• A .env file with your Elasticsearch credentials

The dataset

We’ll use a product catalog of 1,000 products, spanning categories such as footwear, electronics, tools, and more.

Each document has four fields:

Field

Type

`title`

text

`description`

text

`brand`

keyword

`category`

keyword

The dataset is loaded from dataset.csv.

The power and limits of lexical search

BM25 is the default ranking algorithm in Elasticsearch and most search engines. It scores documents by how often your query terms appear in them, adjusted for document length and the frequency of those terms across the entire index. You get analyzers on top: lowercase normalization, stemming, and stopword removal. A query for "running shoes" will match "Running Shoes" and likely "run" as well.

This works well for a large class of queries:

• "running shoes" immediately matches products with those exact tokens in the title.

• "bluetooth speaker" surfaces portable audio products because the tokens appear verbatim.

The results are deterministic and explainable: A document ranks highly because the query terms appear in it. Debugging relevance is straightforward.

Where it breaks

Now let’s try these queries against the same catalog:

• "skincare routine": The word "routine" doesn’t appear in any product title. BM25 can partially match on "skincare," but face serums, body oils, and moisturizers are described using terms like "vitamin C," "retinol," or "brightening," none of which overlap with the query. Products that form a complete skincare routine are scattered across the index with no shared token to anchor them.

• "pet travel accessories": This is a use-case grouping, not a product category. A dog sling carrier, a pet car seat, and a travel crate are all relevant, but their descriptions talk about portability, safety, and comfort rather than "travel accessories." BM25 matches "pet" broadly but has no signal to distinguish travel-specific products from the rest of the pet catalog.

This is a recall problem. The relevant documents exist in your index. BM25 just cannot find them because the user's words and the document's words do not match closely enough.

Adding synonyms helps for known cases. But you cannot enumerate every way a user might express an intent. That is where vectors come in.

Why you should measure recall

Before fixing a problem, you need to quantify it.

Recall@k measures how many of the documents that your users actually want appear somewhere in your search results. Formally:

Precision@k measures the top k results and how many are actually relevant:

High precision means that the results you do return are good. In ecommerce, missing a relevant product (low recall) is often worse than showing a slightly imperfect result (lower precision), because a hidden product is a lost sale.

Elasticsearch's rank_eval API lets you measure both systematically. You provide a list of queries, each with a set of rated documents, and Elasticsearch computes the metrics for you across all queries.

Setting up the evaluation

The rank_eval API needs a ratings dataset: a mapping of queries to the documents that are relevant for each one, along with a relevance grade (0 = not relevant, 1 = relevant, 2 = highly relevant).

In the notebook, this is the judgments list:

The mix is intentional: q1 is a query that BM25 handles well (exact tokens in product titles), while q2, q3, and q4 are intent-based queries where the user's intent is expressed as a concept rather than specific product keywords.

Measuring BM25 baseline recall

First, set up the Elasticsearch client and index the raw text data:

Now build the rank_eval request for BM25. Each request in the list combines a query with its ratings:

Result:

0.43 means that across all four queries, BM25 finds only 43% of the documents it should find. The shortfall is concentrated in the intent-based queries: "skincare routine" misses face serums and body oils because "routine" never appears in product titles, and "pet travel accessories" retrieves off-topic pet products while missing carriers and crates described in terms of portability and safety rather than "travel accessories."

This is our baseline. Now we have a number to beat.

Adding vector search with Jina embeddings

Vector search encodes documents and queries as high-dimensional vectors, a type of vector made up of hundreds or thousands of numerical values, each encoding a specific feature of the data it represents. Documents with similar meaning end up close together in vector space, even if they share no words. "Gym equipment" and "dumbbell set" will be nearby because the concepts are related. I chose Elasticsearch as my vector database because it supports hybrid search, giving me both semantic understanding and keyword precision out of the box.

EIS includes out-of-the-box support for embedding models through its inference API.

Step 1: Using Jina embeddings v5 as an inference endpoint

If your cluster has GPU resources (available in Elastic Cloud and Elasticsearch 9.3+), the embeddings are generated on GPU, which is significantly faster than CPU inference and removes the performance trade-off that historically made vectors expensive at scale.

Why Jina embeddings specifically? jina-embeddings-v5-text is a multilingual model (119+ languages) with a 32,000-token context window and support for task-specific Low-Rank Adaptation (LoRA) adapters. It works well for short product descriptions out of the box. Read more about jina-embeddings-v5-text model here.

Step 2: Create the index with a semantic field

The semantic_text field type is the key here. It’s a higher-level abstraction over dense_vector: You point it at an inference endpoint, and Elasticsearch takes care of generating embeddings automatically.

The copy_to property on title and description means content from both fields flows into semantic_field for embedding, so a single vector captures the full product representation.

Step 3: Index the products

At index time, Elasticsearch calls the inference endpoint for each document and stores the resulting embedding in semantic_field. No extra code on your side.

Hybrid search: Combining BM25 and vectors with RRF

Adding vectors improves recall, but using vectors alone risks losing precision on exact-match queries; "running shoes" should still rank verbatim matches first. Hybrid search retains the lexical component specifically to preserve that precision.

Hybrid search with Reciprocal Rank Fusion (RRF) keeps the best of both:

• BM25 handles exact and near-exact queries with high precision.

• Semantic search handles intent-based and multilingual queries with high recall.

• RRF combines the two ranked lists into a single ranking.

The RRF formula assigns each document a score based on its rank in each result list:

A document that ranks highly in both lists gets a higher combined score. The rank_constant controls how much weight lower-ranked documents receive.

Result:

Hybrid improves substantially over BM25 (0.43) and preserves precision for exact-match queries like "running shoes."

Results: Before and after

Here’s the full comparison across all three approaches:

Result:

Method

Recall@10

BM25 (Lexical)

0.43

Hybrid (BM25 + Vectors)

0.75

Breaking it down by query:

Conclusion

Throughout this post, we saw that BM25 lexical search is reliable when users type exact queries, but it loses recall when they search by intent rather than keywords. Using rank_eval, we established a reproducible baseline to measure that gap with real numbers. From there, we added a semantic_text field powered by Jina embeddings and ran the evaluation again. The result: Hybrid search improved recall from 0.43 to 0.75 while preserving precision on exact-match queries, though the actual margin will depend on your query mix.

The pattern scales beyond this example: Collect judgments from your users' actual queries, run rank_eval as a baseline, add semantic_text, and measure again. You'll know exactly what improved and by how much.

Next steps

• Dive deeper into recall and vector search: Recall and vector search quantization by Jeff Vestal

• Add reranking for even better precision on the top results

• Explore Elasticsearch hybrid search documentation

• Read more about the rank_eval API

Your first query

Start by defining a plain old CLR object (POCO) that maps to your Elasticsearch index. Property names are resolved to ES|QL column names through standard System.Text.Json attributes, like [JsonPropertyName], or through a configured JsonNamingPolicy. The same source serialization rules that apply across the rest of the client apply here as well.

With the type in place, a query looks like this:

The provider translates this into the following ES|QL:

A few details to note:

• Property name resolution: p.Price becomes price_usd because of the [JsonPropertyName] attribute, and p.Brand becomes brand following the default camelCase naming policy.

• Parameter capturing: The C# variables minPrice and brand are captured as named parameters (?minPrice, ?brand). They’re sent separately from the query string in the JSON payload, which prevents injection and enables server-side query plan caching.

• Streaming: QueryAsync<T> returns IAsyncEnumerable<T>. Rows are materialized one at a time as they arrive from Elasticsearch.

You can also inspect the generated query and its parameters without executing it:

How does this work? A quick LINQ refresher

The mechanism that makes LINQ providers possible is the distinction between IEnumerable<T> and IQueryable<T>.

When you call .Where(p => p.Price > 100) on an IEnumerable<T>, the lambda compiles to a Func<Product, bool>, a regular delegate that the runtime executes in-process. This is LINQ-to-Objects.

When you call the same method on an IQueryable<T>, the C# compiler wraps the lambda in an Expression<Func<Product, bool>> instead. This is a data structure that represents the structure of the code rather than its executable form. The expression tree can be inspected, analyzed, and translated into another language at runtime.

The IQueryProvider interface is the extension point. Any provider can implement CreateQuery<T> and Execute<T> to translate these expression trees into a target language. Entity Framework uses this to emit SQL. The LINQ to ES|QL provider uses it to emit ES|QL.

The expression tree for the query above looks like this:

Expression tree for the example query.

The tree is nested inside out: Take wraps OrderByDescending, which wraps Where, which wraps From, which wraps the root EsqlQueryable<Product> constant. The Where predicate is itself a subtree of BinaryExpression nodes for the &&, >=, and == operators, with MemberExpression leaves for property accesses and closure captures for the minPrice and brand variables. This is the data structure that the provider walks to produce the final ES|QL.

Under the hood: The translation pipeline

The path from a LINQ expression to query results follows a six-stage pipeline:

Translation pipeline overview.

1. Expression tree capture

When you chain .Where(), .OrderBy(), .Take() and other operators on an IQueryable<T>, the standard LINQ infrastructure builds an expression tree. EsqlQueryable<T> implements IQueryable<T> and delegates to EsqlQueryProvider.

2. Translation

When the query is executed (by enumerating, calling ToList(), or using await foreach), the EsqlExpressionVisitor walks the expression tree inside out. It dispatches each LINQ method call to a specialized visitor:

Visitor

Translates

Into

WhereClauseVisitor

.Where(predicate)

WHERE condition

SelectProjectionVisitor

.Select(selector)

EVAL + KEEP + RENAME

GroupByVisitor

.GroupBy().Select()

STATS ... BY

OrderByVisitor

.OrderBy() / .ThenBy()

SORT field [ASC\|DESC]

EsqlFunctionTranslator

EsqlFunctions.*, Math.*, string methods

80+ ES|QL functions

During translation, C# variables referenced in expressions are captured as named parameters.

3. Query model

The visitors don’t produce strings directly. Instead, they produce QueryCommand objects, an immutable intermediate representation. A FromCommand, a WhereCommand, a SortCommand, and a LimitCommand, each representing one ES|QL processing command. These are collected into an EsqlQuery model.

Query model and command pattern.

This intermediate model is decoupled from both the expression tree and the output format. It can be inspected, intercepted (via IEsqlQueryInterceptor), or modified before formatting.

4. Formatting

EsqlFormatter visits each QueryCommand in order and produces the final ES|QL string. Each command becomes one line, separated by the pipe (|) operator that ES|QL uses to chain processing commands. Identifiers containing special characters are automatically escaped with backticks.

5. Execution

The formatted ES|QL string and captured parameters are sent to Elasticsearch’s /_query endpoint as a JSON payload. The IEsqlQueryExecutor interface abstracts the transport layer, which is where the layered package architecture comes into play.

6. Materialization

EsqlResponseReader streams the JSON response without buffering the entire result set into memory. A ColumnLayout tree, precomputed once per query, maps flat ES|QL column names (like address.street, address.city) to nested POCO properties. Each row is assembled into a T instance and yielded one at a time via IEnumerable<T> or IAsyncEnumerable<T>.

The layered architecture

The LINQ to ES|QL functionality is split across three packages:

Package architecture. Elastic.Esql is the pure translation engine. It has zero HTTP dependencies and contains the expression visitors, query model, formatter, and response reader. You can use it stand alone to build and inspect ES|QL queries without an Elasticsearch connection, which is useful for testing, query logging, or building your own execution layer.

Elastic.Clients.Esql is a lightweight stand-alone ES|QL client. It adds HTTP execution on top of Elastic.Esql via Elastic.Transport. If your application only needs ES|QL and none of the other Elasticsearch APIs, this is the minimal dependency option.

Elastic.Clients.Elasticsearch is the full Elasticsearch .NET client. It also builds on Elastic.Esql and exposes the LINQ provider through the client.Esql namespace. This is the recommended entry point for most applications.

Both execution-layer packages provide their own implementation of IEsqlQueryExecutor, the strategy interface that bridges translation and transport.

All three packages are compatible with Native AOT when used with a source-generated JsonSerializerContext. For the full client, see the Native AOT documentation.

Beyond the basics

The example above covered filtering, sorting, and pagination. The provider supports a broader set of operations.

Aggregations

GroupBy, combined with aggregate functions in Select, translates to ES|QL STATS ... BY:

Projections

Select, with anonymous types generates EVAL, KEEP, and RENAME commands:

Rich function library

Over 80 ES|QL functions are available through the EsqlFunctions class, covering date/time, string, math, IP, pattern matching, and scoring. Standard Math.* and string.* methods are also translated:

LOOKUP JOIN

Cross-index lookups translate to ES|QL LOOKUP JOIN:

Raw ES|QL escape hatch

For ES|QL features not yet covered by the LINQ provider, you can append raw fragments:

Server-side async queries

For long-running queries, submit them for background processing on the server:

Server-side async queries are especially useful for long-running analytical queries / large dataset processing that might exceed typical timeout thresholds, or in timeout-sensitive environments with load balancers, API gateways, or proxies that enforce strict HTTP timeouts. Async queries avoid connection drops by decoupling submission from result retrieval.

Getting started

LINQ to ES|QL is available starting from:

• Elastic.Clients.Elasticsearch v9.3.4 (9.x branch)

• Elastic.Clients.Elasticsearch v8.19.18 (8.x branch)

Install from NuGet:

dotnet add package Elastic.Clients.Elasticsearch

The entry points are on client.Esql:

Method

Returns

Use case

Query<T>(...)

IEnumerable<T>

Synchronous execution

QueryAsync<T>(...)

IAsyncEnumerable<T>

Async streaming

CreateQuery<T>()

IEsqlQueryable<T>

Advanced composition and inspection

SubmitAsyncQueryAsync<T>(...)

EsqlAsyncQuery<T>

Long-running server-side queries

For the full feature reference, including query options, multifield access, nested objects, and multivalue field handling, see the LINQ to ES|QL documentation.

Conclusion

LINQ to ES|QL brings the full expressiveness of C# LINQ to Elasticsearch's ES|QL query language, letting you write strongly typed, composable queries without handcrafting query strings. With automatic parameter capturing, streaming materialization, and a layered package architecture that scales from stand-alone translation to the full Elasticsearch client, it fits naturally into .NET applications of any size. Install the latest client, point your LINQ expressions at an index, and let the provider handle the rest.

To solve this, search engines like Elasticsearch use two main optimization strategies:

1. Approximate search (hierarchical navigable small world [HNSW]): Instead of scanning every document, we build a navigation graph to jump quickly to the likely neighborhood of the answer.

2. Quantization: We compress the vectors (for example, from 32-bit floats to 8-bit integers or even 1-bit binary values) to reduce memory usage and speed up calculations.

But optimization often comes with a tax: accuracy.

The fear is valid: "If I compress my data and take shortcuts during the search, will I miss the best results?" "Does this optimization degrade the relevance of my search engine?"

To prove that Elastic’s quantization doesn’t degrade results, we built a repeatable test harness using the DBPedia-14 dataset to calculate exactly how much accuracy (specifically, recall) we trade for speed when using default optimizations in Elasticsearch.

tldr: It’s likely much less than you think. Check out the notebook here, and try it yourself

The definitions (for the non-experts)

Before we look at the code, let’s level-set on some terms.

• Relevance versus recall: Relevance is subjective (did I find good stuff?). Recall is mathematical. If there are 10 documents in the database that are the perfect mathematical matches for your query, and the search engine finds nine of them, your recall is 90% (or 0.9).

• Exact search (flat): Sometimes called the "brute force" method. The search engine scans every single document in an index and calculates the distance.

  • Pros: 100% perfect recall.

  • Cons: Computationally expensive and slow at scale.

• Approximate search (HNSW): The "shortcut" method. The search engine builds an HNSW graph. It traverses the graph to find the nearest neighbors.

  • Pros: Extremely fast and scalable.

  • Cons: You might miss a neighbor if the graph traversal stops too early.

The experiment: Exact versus approximate

To test recall, we used the DBPedia-14 dataset, a large dataset of titles and abstracts across 14 ontology classes, commonly used for training and evaluating text categorization models. Specifically, we’ll focus on the "Film" category. We wanted to compare the optimized production settings against a mathematically perfect ground truth.

For this experiment, we are using the jina-embeddings-v5-text-small model, a state-of-the-art multilingual model that leads industry benchmarks for text representation. We chose this model because it defines the current standard for high-performance embeddings. By combining Jina v5’s elite accuracy with Elasticsearch’s native quantization, we can demonstrate a search architecture that is both computationally efficient and uncompromising on retrieval quality.

We set up an index with dual mapping. We ingested the same text into two different fields simultaneously:

1. content.rawwith type: flat. This forces Elasticsearch to perform a brute-force scan of the full Float32 vectors. This returns exact match results and will be used for our baseline.

2. contentwith type semantic_text. With defaults using HNSW + Better Binary Quantization (BBQ). This is the standard, optimized production setting for approximate match.

The Recall@10 test

For our metric, we used Recall@10.

We picked 50 random movies and ran the same query against both fields.

• If the exact (flat) search says the top 10 neighbors are IDs [1, 2, 3... 10].

• And the approximate (HNSW) search returns IDs [1, 2, 3... 9, 99].

• We found nine out of the top 10 correctly. The score is 0.9.

Here’s the mapping we used:

The results: The "flat line" of success

We ran a scale test, reloading the full dataset and testing against index sizes of 1,000 to 40,000 documents.

Here’s what happened to the recall score:

Documents

Recall@10 score

1,000

1.000 (100%)

5,000

0.998 (100%)

10,000

0.992 (99.4%)

20,000

0.999 (99.0%)

40,000

0.992 (98.8%)

The results were incredibly stable. Even as we scaled up, the approximate search matched the brute-force exact search >99% of the time.

Why did it work so well?

You might expect that compressing vectors to binary values would hurt accuracy more than this. The reason it doesn't lies in how Elasticsearch handles the retrieval.

Most embedding models today output Float32 vectors, which are large. To make search efficient, Elasticsearch uses quantization for high-dimensional vectors. Specifically, since 9.2, it uses BBQ by default.

BBQ uses a rescoring mechanism:

1. Traversal: The search engine uses the compressed (quantized) vectors to traverse the HNSW graph quickly. Because the vectors are small, it can efficiently over-sample, gathering a larger list of candidates (for example, the top 100 roughly similar docs) without a performance penalty.

2. Rescore: Once it has those candidates, it retrieves the full-precision values for just those few documents to calculate the final, precise ranking.

This gives you the best of both worlds, the speed of quantization for the heavy lifting, and the precision of floats for the final sort.

Can we do better?

It’s worth noting that the results we’re seeing here are using default settings and a random sampling of data. Think of this as a high-performance starting point. While Jina v5 is a beast, these recall scores aren't a "one size fits all" guarantee for every dataset. Every data collection has its own quirks, and while you can definitely tune things further to squeeze out even more performance, you should always benchmark against your own specific data to see where your ceiling is.

Conclusion

This is a very small-scale test. But the point of the exercise is not to measure the embedding model or BBQ specifically, it’s to demonstrate how you can easily measure the recall of your dataset with minimal setup.

If you want to run this test on your own data, you can check out the notebook here and try it yourself.

In HNSW, search proceeds by iteratively expanding candidate nodes in the graph, maintaining a bounded set of nearest neighbors discovered so far. Each expansion has a cost (vector operations, random seeks to disk, and more), and the marginal benefit of that cost tends to decrease as the search progresses.

One way to optimize HNSW graph traversal is to stop searching when the marginal likelihood of finding new true neighbors doesn’t increase. For this reason, in Elasticsearch 9.2 we introduced a new early termination mechanism. This stops the search process when visiting graph nodes doesn’t provide enough new nearest neighbors, consecutively, for a fixed number of times.

This article guides you through how we improved over the mentioned early termination mechanism in HNSW to make it better suited for different datasets and data distributions.

Early termination in HNSW

In HNSW, search proceeds by iteratively expanding candidate nodes in the proximity graph, maintaining a bounded set of nearest neighbors discovered so far, until it either has visited the whole graph or meets some early stop criteria.

Early termination is therefore not necessarily always an optimization, it’s part of the search algorithm itself. The moment we decide to stop determines the balance between efficiency and recall. In Elasticsearch, there are already a number of ways a query on HNSW can early terminate:

• A fixed maximum number of nodes is visited.

• A fixed timeout is reached.

While simple and predictable, these rules are largely agnostic to what the search is actually doing. Also they’re used mostly to make sure that the query finishes in reasonable time for the end user.

In a previous blogpost, we introduced the concept of redundancy in HNSW. In short, redundant computations occur when HNSW continues to evaluate new candidate nodes that don’t result in finding more nearest neighbors.

Patience: Measuring progress instead of effort

The notion of patience reframes early termination around progress rather than effort.

Instead of asking:

“How many steps have we taken?”

The new question becomes:

“What is the amount of computation we accept to waste, until we lose hope?”

During HNSW search, early exploration typically produces peak improvements to the top-k candidate set. During first steps of the HNSW graph exploration, the set of neighbors is continuously updated as the algorithm keeps discovering nearer and nearer neighbors to the query vector. Over time, these improvements become rarer as the search converges. Patience-based termination monitors this pattern and terminates the search once improvements have ceased for a sustained period.

In practice, while visiting the HNSW graph we also compute the queue saturation ratio as we hop through candidate nodes. This measures the percentage of nearest neighbors that were left unchanged while visiting the most recent graph node (or the inverse of the number of new neighbors introduced during the last iteration). When such a ratio becomes too big for too many consecutive iterations, we stop visiting the graph.

Conceptually, patience treats HNSW search as a diminishing returns process. When returns flatten out, continuing to explore the graph yields little benefit.

This framing is powerful because it ties termination directly to observable outcomes rather than to arbitrary fixed limits.

The benefit of using this smart early termination technique is that HNSW graph explorations tend to visit a smaller number of graph nodes while retaining an almost perfect relative recall.

To visualize this, we can plot the amount of recall per visited node that we got with the patience based early termination (labeled as et=static), when compared to the default HNSW behavior (labeled as et=no) on a couple of datasets, FinancialQA and Quora, and models, JinaV3 and E5-small.

Static thresholds and HNSW dynamics

In practice, in Elasticsearch this is implemented using static thresholds. One threshold refers to the saturation threshold: that is, the ratio of saturation that we consider suboptimal. The other threshold refers to the number of consecutive graph nodes that we allow to be visited while still having a suboptimal queue saturation: that is, the patience threshold.

When we introduced this early termination strategy in Elasticsearch 9.2, we decided to opt for conservative defaults, so as to let the recall as much as possible, while still gaining in terms of latency and memory consumption. For this reason, we set the saturation threshold to be 100% and the patience threshold to be set as a (bounded) 30% of the num_candidates in the KNN query.

In many scenarios, these settings resulted to work nicely; however, two queries requesting the same number of neighbors might have radically different convergence behaviors. Some queries encounter dense local neighborhoods and saturate quickly; others must traverse long, sparse paths before finding competitive candidates. The latter resulted to be the most difficult to handle effectively.

As a result, we sometimes noticed:

• Over-exploration for easy queries.

• Premature termination for hard queries.

Therefore, we figured that fixed threshold values encode global assumptions about convergence, whereas we could make HNSW better adapt to different dynamics.

Making HNSW early termination adaptive

Adaptive early termination approaches this problem from a different angle. Instead of enforcing predefined stopping thresholds, the algorithm infers when to stop from the search dynamics themselves.

So instead of comparing the queue saturation ratio between two consecutive candidates, we decided to introduce both an instant smoothed discovery rate (how many new neighbors were introduced for a query q, in the last visit i) together with rolling mean and standard deviation of such a discovery rate during the graph visit (using Welford’s algorithm). These statistics about the discovery rate are calculated per query, so that this information can be used to decide different degrees of patience for each query.

The previously static thresholds become adaptive to the discovery rate statistics: The saturation threshold becomes the rolling mean plus the standard deviation; whereas we make the patience adapt and scale inversely with the standard deviation.

The early exit rules remain the same; the saturation happens when the instant discovery rate is lower than the adaptive saturation threshold. The graph visit stops if the saturation persists for a number of consecutive candidate visits that’s larger than the adaptive patience.

This way, we obtain a behavior that doesn’t depend on the num_candidates parameter in the KNN query (which might be always set or left as the default, regardless of early exit) and that better adapts to each query and vector distribution dynamically.

The recall per visited node on FinancialQA and Quora with the adaptive strategy (labeled as et=adaptive) reports a higher recall per visited node, when compared to the static strategy (et=static) and the default HNSW behavior (et=no).

Adaptive early termination is turned on by default in Elasticsearch 9.3 for HNSW dense vector fields (and it can eventually be turned off via the same index level setting).

Our benchmarks on a 20M document corpus show that Elasticsearch delivers up to 8x higher throughput than OpenSearch for filtered vector search, while also achieving higher Recall@100 across the configurations we tested. Context engineering depends on more than fast vector retrieval. Teams also need strong relevance controls, like hybrid search and filtering, operational simplicity, and predictable performance, as workflows iterate. But because agents often run retrieve, reason, retrieve loops many times per request, retrieval latency becomes a multiplier, so improvements here translate directly into better end-to-end responsiveness and lower cost.

For context engineering, retrieval isn’t a one-time step. Agents and applications repeatedly run loops, such as retrieve → reason → retrieve, to refine queries, verify facts, assemble grounded context, and complete tasks. This pattern is common in agentic workflows and iterative retrieval augmented generation (RAG). Because retrieval may be invoked many times per user request, it adds delay to the response and/or increases infrastructure costs.

Why is vector search performance critical?

Imagine a shopping assistant answering the question, “I need a carry-on backpack under $60 that fits a 15-inch laptop, is water resistant, and can arrive by Friday.”

In production, the assistant rarely issues one vector query and stops. It runs a retrieval loop to build the right context, and each step is typically constrained by filters, like availability, region, shipping promise, brand rules, and policy eligibility.

Step 1: Interpret intent and translate to constraints.

The agent turns the request into structured filters and a semantic query, such as:

• Filters: In stock, deliverable to the user’s postcode, delivery by Friday, price under $60, valid listing

• Vector query: “Carry-on backpack 15-inch laptop water resistant”

Step 2: Retrieve candidates, and then refine.

It often repeats retrieval with variations to avoid missing good matches:

• “travel backpack carry on laptop sleeve”

• “water resistant commuter backpack 15 inch”

• “lightweight cabin backpack”

Each query uses the same eligibility filters, because retrieving irrelevant or unavailable items is wasted context.

Step 3: Expand to confirm details and reduce risk.

The agent then retrieves again to verify key attributes that affect the final answer:

• Material and water resistance wording

• Dimensions and laptop compartment fit

• Return policy or warranty constraints

• Alternate options if inventory is low

This is multistep context engineering: Retrieve, reason, retrieve, assemble.

Why latency and recall matter for context engineering

These interactions can involve dozens of filtered retrieval calls per user session. That makes per-call latency a direct multiplier on end-to-end response time, and low recall forces extra retries or causes the agent to miss eligible items, degrading answer quality.

Takeaway: In context-engineered systems, filtered approximate nearest neighbors (ANN) isn’t a single lookup. It’s a repeated operation under constraints, so vector search performance shows up immediately in latency, throughput, and cost, even when the large language model (LLM) is the most visible component.

Benchmarking

Results

In Graph 2, each dot represents one test configuration. The best results appear toward the top left, meaning higher recall with lower latency. Elasticsearch’s results are consistently closer to the top left than OpenSearch’s, indicating better speed and accuracy under the same workload settings.

Some key insights

• s_n_r_value: Shorthand for size_numCandidates_rescoreOversample (k and numCandidates set equal to numCandidates in these tests), for example, 100_500_1 means size=100, numCandidates=500 and k=500, rescore oversample=1

• Recall: Measured Recall@100 for that configuration

• Avg latency (ms): Average end-to-end latency per query

• Throughput: Queries per second

• Recall %: Relative recall lift of Elasticsearch versus OpenSearch (Elasticsearch minus OpenSearch) / OpenSearch

• Latency Xs: OpenSearch average latency divided by Elasticsearch average latency

• Throughput Xs: Elasticsearch throughput divided by OpenSearch throughput

Engine

`s_n_r_value`

Recall

Avg Latency (ms)

Throughput

Recall %

Latency Xs

Throughput Xs

Elasticsearch

100_250_1

0.7704

25

534.75

9.70%

2.28

1.91

OpenSearch

100_250_1

0.7023

57.08

279.58

Elasticsearch

100_500_1

0.8577

25.42

524.14

7.20%

2.4

2

OpenSearch

100_500_1

0.8001

60.9

262.12

Elasticsearch

100_750_1

0.8947

29.67

528.09

5.72%

2.25

2.21

OpenSearch

100_750_1

0.8463

66.76

239.11

Elasticsearch

100_1000_1

0.9156

29.65

534.5

4.66%

2.46

2.44

OpenSearch

100_1000_1

0.8748

72.88

219.01

Elasticsearch

100_1500_1

0.9386

31.84

497.3

3.38%

2.71

2.68

OpenSearch

100_1500_1

0.9079

86.16

185.4

Elasticsearch

100_2000_1

0.9507

34.69

457.2

2.57%

2.98

2.96

OpenSearch

100_2000_1

0.9269

103.36

154.55

Elasticsearch

100_2500_1

0.9582

37.9

418.43

1.99%

3.28

3.26

OpenSearch

100_2500_1

0.9395

124.29

128.53

Elasticsearch

100_3000_1

0.9636

41.86

379.4

1.62%

3.46

3.44

OpenSearch

100_3000_1

0.9482

144.67

110.34

Elasticsearch

100_4000_1

0.9705

50.28

316.21

1.06%

3.87

3.85

OpenSearch

100_4000_1

0.9603

194.36

82.22

Elasticsearch

100_5000_1

0.9749

58.77

270.91

0.73%

4.43

4.41

OpenSearch

100_5000_1

0.9678

260.33

61.38

Elasticsearch

100_6000_1

0.9781

66.75

238.59

0.52%

4.91

4.89

OpenSearch

100_6000_1

0.973

327.44

48.81

Elasticsearch

100_7000_1

0.9804

74.64

213.49

0.38%

5.28

5.27

OpenSearch

100_7000_1

0.9767

394.24

40.53

Elasticsearch

100_8000_1

0.9823

82.28

193.59

0.27%

6.86

6.83

OpenSearch

100_8000_1

0.9797

564.14

28.33

Elasticsearch

100_9000_1

0.9837

90.08

176.96

0.16%

7.63

7.61

OpenSearch

100_9000_1

0.9821

687.25

23.25

Elasticsearch

100_10000_1

0.9848

97.64

163.31

0.08%

8.38

8.36

OpenSearch

100_10000_1

0.984

818.64

19.53

For example, at 100_9000_1, OpenSearch averages 687 milliseconds per retrieval versus 90 milliseconds on Elasticsearch, and in a 10-step retrieval loop that’s about 10 x (687 - 90) = six seconds of additional waiting time.

See the full results.

Methodology

Using Python to send the queries and track the response timing and other statistics, we sent the following queries to the engines. Bear in mind that the performance of any vector search engine depends on how you tune its core parameters: how many candidates to consider, how aggressively to rescore, and how much context to return. These settings directly affect both recall (the likelihood of finding the right answer) and latency (how fast you get results).

In our benchmarks, we used the same candidate, rescore, and result-size settings you’d typically tune in an agentic retrieval loop, and we measured how Elasticsearch performs under that workload. We then ran OpenSearch with the same settings as a reference.

OpenSearch

• "size": <RESULT_SIZE>: Number of hits returned to the client. In this benchmark, result size is 100 to compute Recall@100.

• "k": <NUMBER_OF_CANDIDATES>: The number of nearest neighbor candidates.

• "ef_search": <NUMBER_OF_CANDIDATES>: The number of vectors to examine.

• "oversample_factor": <OVERSAMPLE>: How many candidate vectors are retrieved before rescoring.

Elasticsearch

• "size": <RESULT_SIZE>: Number of hits returned to the client. In this benchmark, result size is 100 to compute Recall@100.

• "k": <NUMBER_OF_CANDIDATES>: Number of nearest neighbors to return from each shard.

• "num_candidates": <NUMBER_OF_CANDIDATES>: Number of nearest neighbor candidates to consider per shard while doing knn search.

• "oversample": <OVERSAMPLE>: How many candidate vectors are retrieved before rescoring.

Example

Knn query, (100_500_1), would be as follows:

OpenSearch

Elasticsearch

The full configuration, alongside Terraform scripts, Kubernetes manifests and the benchmarking code is available in this repository in the folder es-9.3-vs-os-3.5-vector-search.

Cluster setup

We ran our tests on six e2-standard-16 cloud servers, each with 16 vCPUs and 64 GB RAM. On each server, we allocated 15 vCPUs and 56 GB RAM to each Kubernetes pod running the search engine node, with 28 GB reserved for the JVM heap.

The clusters ran Elasticsearch 9.3.0 and OpenSearch 3.5.0 (Lucene 10.3.2). Because both systems use the same Lucene version in this benchmark, the throughput and latency differences we observe cannot be attributed to Lucene alone and instead reflect differences in how each engine integrates and executes filtered k-nearest neighbor (kNN) retrieval and rescoring. We used a single index with three primary shards and one replica (so 6 shards total, 1 per node).

We also used a separate server in the same region to run the benchmark client and collect timing statistics.

The dataset

For this benchmark, we used a large-scale ecommerce-style catalog embedding dataset with 20 million documents, designed to reflect real-world filtered vector retrieval at scale.

Each document represents a catalog item and includes:

• A 128-dimensional dense vector embedding used for approximate kNN retrieval.

• Structured metadata fields used for filtering (for example, item validity and availability plus other catalog constraints) enabling the common production pattern of retrieving the nearest neighbors but only within an eligible subset.

We chose this dataset because it captures the core performance challenge we see in agentic and RAG-style systems in production: Vector similarity alone is not enough, retrieval is frequently constrained by filters, and the system must maintain high recall while keeping latency low under those constraints. Compared to smaller QA-style datasets, a 20M document corpus also better reflects the scale and candidate pressure that filtered ANN systems face in practice.

Conclusion

In modern AI architectures, especially those built around context engineering, vector search speed isn’t a minor implementation detail. It’s a multiplier. When agents and workflows iterate through retrieve → reason → retrieve, retrieval performance directly shapes end-to-end latency, throughput, and the quality of the context fed into the model.

In our benchmarks, Elasticsearch consistently delivered higher recall at lower latency than OpenSearch in scenarios where correctness depends on retrieving the right document, not just a similar vector. On a controlled dataset, the difference is clear, and in production those gains accumulate across large volumes of retrieval calls, improving responsiveness, increasing capacity headroom, and reducing infrastructure costs.

Further reading

1. What is context engineering?

2. The evolution of hybrid search and context engineering

3. The impact of relevance in context engineering for AI agents

Storage formats in dense_vector fields

Prior to 9.3, dense_vector fields support vectors of single bits, 1-byte integers, and 4-byte floats. We store the original vectors on top of any quantization and/or hierarchical navigable small world (HNSW) graph used for indexing, and the original vectors make up the vast majority of the required disk space of the vector indices. If your vectors are floating point, then the only option versions of Elasticsearch prior to 9.3 provide is to store 4 bytes per vector value: That’s 4kB for a single 1024-dimensional vector.

There are other floating-point sizes available, of course: IEEE-754 specifies floating-point sizes of many different lengths, including the 4-byte float32 and 8-byte float64 used by Java float and double types. It also specifies a float16 format, which only uses 2 bytes per value. However, this only has a maximum value of 65,504, compared to the 3.4x1038 of 4-byte float32 values, and the conversion between the two involves several arithmetic operations.

As an alternative, many machine learning (ML) applications now use bfloat16, which is a modification of IEEE-754 float32 to only use 2 bytes. It does this by discarding the lowest 2 bytes of the fractional part of the value, leaving the sign and exponent unchanged. This effectively reduces the precision of the floating-point value without a corresponding reduction in range. The conversion from float32 to bfloat16 is a simple bitwise truncation on the float32 value, with a bit of jiggling to account for rounding.

bfloat16 in Elasticsearch 9.3

Elasticsearch 9.3 now supports storing vector element types as bfloat16. In memory, it will still process every vector value as a 4-byte float32, as Java does not have built-in support for bfloat16. As it writes vector data to disk, it will simply truncate and round each float32 value to a 2-byte bfloat16, and zero-expand each bfloat16 value back to float32 on reading the value into memory.

This effectively halves your vector index sizes, as it uses 2 bytes per value rather than 4 bytes. There may be a small performance cost during reading and writing data as Elasticsearch performs the necessary conversions, but this is often counterbalanced by a significant reduction in the I/O required, as the OS now has to read half as much data. And, for most datasets, there is a minimal effect on search recall.

As an example, this is the difference in sizes for bfloat16 on our dense_vector dataset:

So, if your input vectors are already at bfloat16 precision, then happy days! Elasticsearch accepts raw bfloat16 vectors as float values, and as Base64-encoded vectors. The vectors are persisted to disk with the same precision as your original source data, immediately halving your data storage requirements.

If your input vectors are at 4-byte precision, then you can also use bfloat16 format to halve your index data sizes. Elasticsearch will truncate and round each value to 2-byte precision, throwing away the least significant bits of the fraction. This means that the vector values you get back from Elasticsearch won’t be exactly the same as what you originally indexed, so don’t use bfloat16 if you need to maintain the full 4-byte precision of float32.

Starting in Elasticsearch 9.3, and on Elasticsearch Serverless, you can specify element_type: bfloat16 with all dense_vector index types on any newly created indices. If you wish to use bfloat16 with existing indices, you can reindex into an index with element_type: bfloat16 and Elasticsearch will automatically convert your existing float vectors to bfloat16.

This change reduces the overhead of parsing vectors in JSON by an order of magnitude, which translates to almost a 100% improvement on indexing throughput for DiskBBQ and around 20% improvement for hierarchical navigable small world (HNSW) workloads. In this blog, we’ll take a closer look at Base64-encoded strings and the improvements it brings to vector ingestion.

What’s the problem?

At Elastic, we’re always looking for ways to improve our vector search capabilities, whether that’s enhancing existing storage formats or introducing new ones. Recently, for example, we added a new disk-friendly storage format called DiskBBQ and enabled vector indexing with NVIDIA cuVS.

In both cases, we expected to see major gains in ingestion speed. However, once these changes were fully integrated into Elasticsearch, the improvements weren’t as large as we had hoped. A flamegraph of the ingestion process made the issue clear: JSON parsing had become one of the main bottlenecks.

Parsing JSON requires walking through every element in the arrays and converting numbers from text format into 32-bit floating-point values, which is very expensive.

Why Base64-encoded strings?

The most efficient way to parse vectors is directly from their binary representation, where each element uses a 32-bit floating-point value. However, JSON is a text-based format, and the way to include binary data in it is by using Base64-encoded strings. Base64 is just a binary-to-text encoding schema.

We can now send vectors encoded as Base64 strings:

Is it worth it? Our benchmarks suggest yes. When parsing 1,000 JSON documents, using Base64 encoded strings instead of float arrays resulted in performance improvements of more than an order of magnitude, at the cost of a small encode/decode trade-off (client-side Base64 encoding and a temporary byte array on the server for decoding) in exchange for eliminating expensive per-element numeric parsing.

Give me some ingestion numbers

We can see these improvements in practice when running the so_vector rally track with the different approaches. The actual gains depend on how fast indexing is for each storage format. For bbq_disk, indexing throughput increases by about 100%, while for bbq_hnsw, the improvement is closer to 20%, since indexing is inherently slower there.

Starting with Elasticsearch v9.2, vectors are excluded from _source by default and are stored internally as 32-bit floating-point values. This behavior also applies to Base64-encoded vectors, making the choice of indexing format completely transparent at search time.

Client support

Adding a new format for indexing vectors might require changes on ingestion pipelines. To help this effort, in v9.3, Elasticsearch official clients can transform vectors with 32-bit floating-point values into Base64-encoded strings and the other way around. You might need to check the client documentation for the specific implementation.

For example, here’s a snippet for implementing bulk loading using the Python client:

The only difference from a bulk ingest using floats is that the embedding is wrapped with the pack_dense_vector() auxiliary function.

Conclusion

By switching from JSON float arrays to Base64-encoded vectors, we remove one of the largest remaining bottlenecks in Elasticsearch’s vector ingestion pipeline: numeric parsing. The result is a simple change with outsized impact: up to 2× higher throughput for DiskBBQ workloads and meaningful gains even for slower indexing strategies, like HNSW.

Because vectors are already stored internally in a binary format and excluded from _source by default, this improvement is completely transparent at search time. With official client support landing in v9.3, adopting Base64 encoding requires only minimal changes to existing ingestion code, while delivering immediate performance benefits.

If you’re indexing large volumes of embeddings, especially in high-throughput or serverless environments, Base64-encoded vectors are now the fastest and most efficient way to get your data into Elasticsearch.Those interested in the implementation details can follow the related Elasticsearch issues and pull requests: #111281 and #135943.

Along with support for IBM watsonx™ Slate embedding models, Elasticsearch vector database powers watsonx Assistant for Conversational Search—now with semantic reranking for even better answer quality.

Reranking refines LLM responses by prioritizing the most relevant documents using advanced scoring methods, ensuring accurate responses in multi-stage retrieval, and making it broadly applicable even for datasets that you don’t want to reindex or remap.

IBM watsonx offers high-quality reranker models that accurately score and prioritize passages based on query relevance, helping refine search results for better precision. These models enhance tasks like semantic search and document comparison, making them essential for delivering highly relevant answers in AI-driven retrieval systems.

In this blog, we’ll explore how to use IBM watsonx™ reranking when building search experiences in the Elasticsearch vector database to reorder search results by meaning, giving you sharper, more context-aware answers without altering your existing index.

How reranking can create powerful search experiences

Semantic reranking is crucial because users expect the best answers at the top, and GenAI models require accurate results to avoid generating incorrect information. Semantic reranking provides consistent scoring, ensuring the most relevant documents are used by AI models and enabling effective cutoff points to prevent hallucinations.

Prerequisites & Creation of an Inference Endpoint

Create an Elasticsearch Serverless Project.

Elasticsearch Cloud Serverless offers fast query execution and seamless integration with the open Inference API, making it ideal for deploying reranking without infrastructure overhead.

Generate an API key in IBM Cloud

• Go to IBM watsonx.ai Cloud and log in using your credentials. You will land on the welcome page.

• Go to the API keys page.

• Create an API key.

Steps in Elasticsearch

Using DevTools in Kibana, create an inference endpoint using the watsonxai service for reranking. This example uses the MS Marco MiniLM L-12 v2 model which is supported by IBM, for ensuring high relevance in passage retrieval.

You will receive the following response on the successful creation of the inference endpoint:

Let us now create an index.

Next, insert data into the created index.

Next, let’s search using a text_similarity_reranker retriever, which enhances search results by reranking documents based on semantic similarity to a specified inference text, using an ML model.

The retriever helps you configure both the retrieval and reranking of search results in a single API call.

Next, let’s verify the returned result.

The passages are now reordered to show the passages with the highest scores first. In this example, lexical retrieval initially selected The Avengers (_id: 3) and Star Wars (_id: 2) based on word matches—“lightning” in one and “feeling” in the other. This approach considers surface-level overlaps and keywords.

IBM watsonx.ai rerank then re-evaluated the results based on context, ranking The Avengers higher because “lightning” directly aligned with the query "feeling lightning." This demonstrates that by prioritizing meaning over simple keyword matches, reranking ensures more relevant search results.

Try semantic reranking with watsonx and Elasticsearch today

With the integration of IBM watsonx™ rerank models, the Elasticsearch Open Inference API continues to empower developers with enhanced capabilities for building powerful and flexible AI-powered search experiences. Explore more supported encoder foundation models available with watsonx.ai.

Additionally, use IBM watsonx Assistant’s new Conversational Search feature and IBM watsonx Discovery today. Visit IBM watsonx Discovery to learn more about this new capability using Elasticsearch. You can follow these steps for setup and integration with IBM watsonx Assistants.

As part of our mission to leave no efficiencies behind, we are exploring acceleration opportunities with these curious computer chips, which you may have heard of—NVIDIA GPUs! (Seriously, have you not?).

When obsessing over performance, we have several problem spaces to explore—how to index exponentially more data, how to retrieve insights from it, and how to do it when your ML models are involved. You should be able to eke out every last benefit available when you have GPUs.

In this post, we dive into our collaboration with the NVIDIA vector search team as we explore GPU-accelerated vector search in Elasticsearch. This work paves the way for use cases where developers could use a mix of GPUs and CPUs for real-world Elasticsearch-powered apps. Exciting times!

Elasticsearch GPUs

We are excited to share that the Elasticsearch engineering team is helping build the open-source cuVS Java API experience for developers, which exposes bindings for vector search algorithms. This work leverages our previous experience with Panama FFI. Elasticsearch and Apache Lucene use the NVIDIA cuVS API to build the graph during indexing. Okay, we are jumping ahead; let’s rewind a bit.

NVIDIA cuVS, an open-source C++ library, is at the heart of this collaboration. It aims to bring GPU acceleration to vector search by providing higher throughput, lower latency, and faster index build times. But Elasticsearch and Apache Lucene are written in Java; how will this work?

Enter lucene-cuvs and the Elastic-NVIDIA-SearchScale collaboration to bring it into the Lucene ecosystem to explore GPU-accelerated vector search in Elasticsearch. In the recent NVIDIA cuVS 25.02 release, we added a Java API for cuVS. The new API is experimental and will continue to evolve, but it’s currently available for use. The question may arise: aren’t Java to native function calls slow? Not anymore! We’re using the new Panama FFI (Foreign Function Interface) for the bindings, which has minimal overhead for Java to native downcalls.

We’ve been using Panama FFI in Elasticsearch and Lucene for a while now. It’s awesome! But... there is always a “but”, isn’t there? FFI has availability challenges across Java versions. We overcame this by compiling the cuVS API to Java 21 and encapsulating the implementation within a multi-release jar targeting Java 22. This allows the use of cuVS Java directly in Lucene and Elasticsearch.

Ok, now that we have the cuVS Java API, what else would we need?

A tale of two algorithms for CPU

Elasticsearch supports the HNSW algorithm for scalable approximate KNN search. However, to get the most out of the GPU, we use a different algorithm, CAGRA [CUDA ANN GRAph], which has been specifically designed for the high levels of parallelism offered by the GPU.

Before we get into how we look to add support for CAGRA, let’s look at how Elasticsearch and Lucene access index data through a “codec format”. This consists of

1. the on-disk representation,

2. the interfaces for reading and writing data,

3. and the machinery for dealing with Lucene’s segment-based architecture.

We are implementing a new KNN (k-nearest neighbors) vector format that internally uses the cuVS Java API to index and search on the GPU. From here, we “plumb” this codec type through Elasticsearch’s mappings to a field type in the index. As a result, your existing KNN queries continue to work regardless of whether the backing index is using a CAGRA or HNSW graph. Of course, this glosses over many details, which we plan to cover in a future blog. The following is the high-level architecture for a GPU-accelerated Elasticsearch.

This new codec format defaults to CAGRA. However, it also supports converting a CAGRA graph to an HNSW graph for search on the CPU.

Indexing and searching on the GPU: Making some “core” decisions

With the stateless architecture for Elasticsearch Serverless, which separates indexing and search, there is now a clear delineation of responsibilities. We pick the best hardware profile to fulfill each of these independent responsibilities.

We anticipate users to consider two main deployment strategies:

1. Index and search on the GPU: During indexing, build a CAGRA graph and use it during search - ideal when extremely low latency search is required.

2. Index on GPU and search on CPU: During indexing, build a CAGRA graph and convert it to an HNSW graph. The HNSW graph is stored in the index, which can later be used on the CPU for searching.

This flexibility provides different deployment models, offering tradeoffs between cost and performance. For example, an indexing service could use GPU to efficiently build and merge graphs in a timely manner while using a lower-powered CPU for searching.

So here is the plan for GPU-accelerated vector search in Elasticsearch

We are looking forward to bringing performance gains and flexibility with deployment strategies to users, offering various knobs to balance cost and performance. Here is the NVIDIA GTC 2025 session where this work was presented in detail.

We’d like to thank the engineering teams at NVIDIA and SearchScale for their fantastic collaboration. In an upcoming blog, we will explore the implementation details and performance analysis in greater depth. Hold on to your curiosity hats 🎩!

This ease of use does come with some tradeoffs: we simplified semantic search with semantic_text by making judgments on default behavior that fit the majority of use cases. Unfortunately, this means that some customizations available for traditional vector search queries aren’t present in the semantic query. We didn’t want to add all of these options directly to the semantic query, as that would undermine the simplicity that we strive for. Instead, we expanded the queries that support the semantic_text field, leaving it up to you to choose the best query that meets your needs.

Let’s walk through these changes, starting with creating a simple index with a semantic_text field:

We made match happen in semantic search!

First and most importantly, the match query will now work with semantic_text fields!

This means that you can change your old semantic query:

Into a simple match query:

We can see the benefits of semantic search here because we’re searching for “song lyrics about love”, none of which appears in the indexed document. This is because of ELSER’s text expansion.

But wait, it gets better!

If you have multiple indices, and the same field name is semantic_text in one field and perhaps text in the other field, you can still run match queries against these fields. Let’s create another index, with the same field names, but different types (text instead of semantic_text). Here’s a simple example to illustrate:

Here, searching for “crazy” brings up both the lexical match that has “crazy” in the title, and the semantic lyric “lose my mind.”

There are some caveats to keep in mind when using the match functionality with semantic_text:

• The underlying semantic_text field has a limitation where you can’t use multiple inference IDs on the same field. This limitation extends to match—meaning that if you have two semantic_text fields with the same name, they need to have the same inference ID or you’ll get an error. You can work around this by creating different names and querying them in a boolean query or a compound retriever.

• Depending on what model you use, the scores between lexical (text) matches and semantic matches will likely be very different. In order to get the best ranking of results, we recommend using second stage rerankers such as semantic reranking or RRF.

Semantic search using the match query is also available in ES|QL! Here’s the same example as above, but using ES|QL:

Expert-level semantic search with knn and sparse_vector

Match is great, but sometimes you want to specify more vector search options than the semantic query supports. Remember, the tradeoff of making the semantic query as simple as it is involved making some decisions on default behavior.

This means that if you want to take advantage of some of the more advanced vector search features, perhaps num_candidates or filter from the knn query or token pruning in the sparse_vector query, you won’t be able to do so using the semantic query.

In the past, we provided some workarounds to this, but they were convoluted and required knowing the inner workings and architecture of the semantic_text field and constructing a corresponding nested query. If you’re doing that workaround now, it will still work—however, we now support query DSL using knn or sparse_vector queries on semantic_text fields.

All about that dense (vector), no trouble

Here’s an example script that populates a text_embedding model and queries a semantic_text field using the knn query:

The knn query can be modified with extra options to enable more advanced queries against the semantic_text field. Here, we perform the same query but add a pre-filter against the semantic_text field:

Keepin’ it sparse (vector), keepin’ it real

Similarly, sparse embedding models can be queried more specifically using semantic_text fields as well. Here’s an example script that adds a few more documents and uses the sparse_vector query:

The sparse_vector query can be modified with extra options, to enable more advanced queries against the semantic_text field. Here, we perform the same query but add token pruning against a semantic_text field:

This example significantly decreases the token frequency ratio required to pruning, which helps us show differences with such a small dataset, though they’re probably more aggressive than you’d want to see in production (remember, token pruning is about pruning irrelevant tokens to improve performance, not drastically change recall or relevance). You can see in this example that the Avril Lavigne song is no longer returned, and the scores have changed due to the pruned tokens. (Note that this is an illustrative example, and we still recommend a rescore adding pruned tokens back into scoring for most use cases).

You’ll note that with all of these queries if you’re only querying a semantic_text field, you no longer need to specify the inference ID in knn’s query_vector_builder or in the sparse_vector query. This is because it will be inferred from the semantic_text field. You can specify it if you want to override with a different (compatible!) inference ID for some reason or if you’re searching combined indices that have both semantic_text and sparse_vector or dense_vector fields though.

Try it out yourself

We’re keeping the original semantic query simple, but expanding our semantic search capabilities to power more use cases and seamlessly integrate semantic search with existing workflows. These power-ups are native to Elasticsearch and are already available in Serverless. They’ll be available in stack-hosted Elasticsearch starting with version 8.18.

Try it out today!

In this blog, we’ll demonstrate how to use DocParse and DocPrep to prepare and load a dataset of complex PDFs into Elasticsearch for a RAG application. We will use ~75 PDF reports from the National Transportation Safety Board (NTSB) about aircraft incidents. An example document from the collection is here.

What is Aryn DocParse and DocPrep

Aryn DocParse segments and labels documents, extracts tables, and images, and does OCR – turning 30+ document types into structured JSON. It runs the open-source Aryn Partitioner and its open-source deep learning DETR AI model trained on 80k+ enterprise documents. This leads to up to 6x more accurate data chunking and 2x improved recall on hybrid search or RAG compared to off-the-shelf systems.

Aryn DocPrep is a tool for creating document ETL pipelines to prepare and load this data into vector databases and hybrid search indexes like Elasticsearch. The first step in a pipeline is using DocParse to process each document. DocPrep creates Python code using Sycamore, an open-source, scalable, LLM-powered document ETL library. Though DocPrep can easily create ETL pipelines using Sycamore code, you will likely need to customize the pipeline using additional Sycamore data transforms, chunking/merging, extraction, and cleaning functions.

As can be seen, these documents are complex, containing tables, images, section headings, and complicated layouts. Let’s begin!

Building high-quality RAG apps with effective data preparation

Launch an Elasticsearch vector database container

We’ll install Elasticsearch locally using a Docker container for the demo RAG application. Follow these instructions to deploy it.

Prepare data for RAG using Aryn DocPrep and DocParse

Aryn DocPrep is a tool for creating document ETL pipelines that prepare and load data into vector databases and hybrid search indexes like Elasticsearch. The first step in a pipeline is using DocParse to process each document.

We will use Aryn DocParse in Aryn Cloud to generate our initial ETL pipeline code. You can sign up for free to use Aryn Cloud and go to the DocPrep UI in the Aryn Cloud console.

You can also write an ETL pipeline and run a version of the Aryn Partitioner (used in DocParse) locally. Visit the Sycamore documentation to learn more.

Create ETL pipeline with Aryn DocPrep

DocPrep creates Python code using Sycamore, an open-source, scalable, LLM-powered document ETL library. While DocPrep can easily create ETL pipelines using Sycamore code, you may need to customize the pipeline further with additional Sycamore data transforms, extraction, and cleaning functions.

DocPrep simplifies the creation of a base ETL pipeline to prepare unstructured data for RAG and semantic search.

First, we provide the document type (PDF) and the source location of our PDFs in Amazon S3 (s3://aryn-public/ntsb/):

Next, we will select MiniLM for our embedding model to create our vector embeddings locally. DocPrep uses DocParse for document segmentation, extraction, and other processing, but we don’t need to change the default configuration.

Finally, we select Elasticsearch as our target database and add the Host URL and Index name. Note that the URL is set to “localhost” because we are running Elasticsearch locally. We will also run DocPrep/Sycamore ETL pipeline locally so it can easily load the cluster.

Now, click “Generate pipeline” to create the ETL pipeline. Click “Download notebook” on the next page to download the code as a Jupyter notebook file.

Install Jupyter and Sycamore

We will run the ETL pipeline locally in a Jupyter notebook and use the Sycamore document ETL library. As a first step, install Jupyter and Sycamore with the Elasticsearch connector and local inference libraries to create vector embeddings.

Run Pipeline

Run Jupyter and open the notebook with the ETL pipeline downloaded in the earlier step.

If you haven’t set your Aryn Cloud API key as an environmental variable called ARYN_API_KEY, you can set it directly in the notebook.

In the second-to-last cell, update the Elasticsearch loading configuration. Replace the es_client_args from setting an Elasticsearch password to the Elasticsearch basic auth config from your container:

If the password isn’t set as an environment variable, you can add it directly here.

Now, run the cells in the notebook. Each of the ~75 PDFs is sent to DocParse for processing, and this step in the pipeline will take a few minutes. One of the cells will output three pages of a document with bounding boxes to show how DocParse segments the data.

The final cell runs a read query to verify if the data has been loaded correctly. Now, you can use the prepared data in the Elasticsearch index with your RAG application.

Add additional data enrichment and transforms

The code generated in DocPrep is great for a basic ETL pipeline, however, you may want to extract metadata and perform data cleaning. The pipeline code is fully customizable, and you can use additional transformations in Sycamore or arbitrary Python code.

Here is an example notebook with additional data transforms, metadata extraction, and data cleaning steps. You can use this metadata in your RAG applications to filter your results.

Conclusion

This blog used Aryn DocParse, DocPrep, and Sycamore to parse, extract, enrich, clean, embed, and load data into vector and keyword indexes in the Elasticsearch vector database. We used DocPrep to create an initial ETL pipeline and then used a notebook with additional Sycamore code to demonstrate additional data enrichment and cleaning.

How your documents are parsed, enriched, and processed significantly impacts the quality of your RAG queries. Use the examples in this blog post to quickly and easily build your own RAG systems with Aryn and Elasticsearch and iterate on the processing and retrieval strategies as you build your GenAI application.

Below are some resources for your next steps:

• Sample notebook with Aryn DataPrep and Elasticsearch

• start-local with Elasticsearch vector database

• Get started with Aryn Cloud DocPrep

• Sycamore documentation

• Elasticsearch vector database ecosystem integrations

In this article, we’ll build a RAG application in Golang, using Ollama as the LLM server and Elasticsearch as the vector database.

Steps for building RAG application with Ollama & Go

1. Install Ollama

2. Ingesting data

3. RAG App in Go

Install Ollama

What is Ollama?

Ollama is a framework that allows you to download and access models locally with a CLI. With simple commands we can download, chat, and set up a server with the model we want to consume from our app.

Download the Ollama installer here:

https://ollama.com/

The library with the available models is here:

https://ollama.com/library

Once you have installed Ollama, we can test that everything works by running one of the available models. Let’s install llama3.2 with 3B parameters. The library includes the necessary commands to download and run the model:

We’ll run the command for the 3B version:

The first time, it will download the model and then open a chat in the terminal:

Now we can type /exit to exit and use the server set up in this location: http://localhost:11434. Let's test the endpoints to make sure everything is working as expected.

Ollama offers two answer modes: generate to provide a single answer and chat to have conversations with the model:

Generate

We use generate when we expect a single answer to a single question and nothing else.

Chat

We use chat when we expect to continue the conversation after the first question, and we want the LLM to remember the previous interactions.

By default, the answer is generated as stream: true but we’ll use stream: false so that the answer is generated in just one message and it’s easier to read. stream: true is useful in UI applications as tokens are sent as they are generated, as opposed to blocking until the entire response is complete.

Let’s move on to the data.

Ingesting data

Let's index some medical documents in Elasticsearch as text and vectors. We'll use these to test the quality of the answers from a medical-oriented model like openbiollm compared to a general one.

Before we start, make sure that we have created the inference endpoint to use ELSER as our embeddings model:

Now, let’s continue creating the index using the semantic_text field type that will allow us to control the chunking size, as well as vector configuration. This allows our index to support full text, semantic, and hybrid searches.

Now, let’s index the documents:

Done! Now that we have the model and data ready, we can put everything together with our Go app.

RAG App in Go

For our Go application, we could make calls to the Ollama server directly but I’ve decided to use parakeet instead. Parakeet is a library to create GenAI applications based on Go text. It provides Go interfaces to abstract the HTTP communication, besides giving helpers for embeddings, chunking, and memory, among others, so it makes it very easy to create an application.

We’ll begin by creating our work folder and setting up dependencies:

Now, create a main.go file with the minimum you need to test that everything is configured correctly:

main.go

Run it:

In the terminal, you should see an answer similar to this:

Elastic, a company known for its innovative and user-friendly software solutions, has disrupted the traditional IT industry by empowering businesses to create, deploy, and manage applications quickly and reliably.

This answer is based on the LLM's training data, which is not something that we can provide or control and present some disadvantages:

• Information may be wrong

• Information may be outdated

• There is no way to get citations of the source

Now, let’s create a file called elasticsearch/elasticsearch.go to connect with Elasticsearch using the Go’s official client and be able to use the information in our documents to generate grounded answers based on our data.

The EsClient function initializes the Elasticsearch client using the Cloud Credentials provided, and SemanticRetriever performs a semantic query to build the context the LLM needs to answer a question.

To find your Cloud ID and API Key, go to this link.

Let’s go back to our main.go file and update with the above features to call Elasticsearch and run a semantic query: This builds the LLM context:

main.go

As you can see, we send the user’s question together with all of the documents related to it. This is how we get an answer based on documents in Elasticsearch.

We can test by running the code:

You should see something like:

• JAK inhibitors target intracellular signaling pathways, specifically the Janus kinase (JAK) pathway, which is involved in inflammation and immune response. They have a rapid onset of action and are administered orally.
• Monoclonal antibodies target specific proteins involved in the inflammatory process, such as tumor necrosis factor-alpha (TNF-α), interleukin-6 (IL-6), and interleukin-17 (IL-17).

• "JAK Inhibitors vs. Monoclonal Antibodies in Rheumatoid Arthritis Treatment" (document title)

Parakeet will handle the Ollama interaction for us, including the token stream! From now on, we can test different models quite easily without making changes to the code.

Besides the models in the main library, we also have access to the ones uploaded by community members.

To use one, we just have to make sure to download into Ollama first. For example, let’s test openbiollm:

ollama run taozhiyuai/openbiollm-llama-3:8b_q8_0

Once installed, we can use it with our Go code:

Let’s run it again with the same question. Do you notice any differences?

It seems that the openbiollm model provided more details with the technical lingo, but it didn’t follow instructions about referencing the documents provided in the context and giving brief answers. In comparison, Llama3.2 followed instructions better.

You can find the full working example here

Conclusion

Ollama provides a very straightforward and simple way to download and test different open models, from the better-known to those fine-tuned by community members. Pairing it with Parakeet and the official Elasticsearch Go client, makes it very easy to create a RAG application. In addition, by using the semantic_text field type, you can create a semantic-query-ready index that uses ELSER–the Elastic Sparse embeddings model–without any additional configurations, thus simplifying the chunking, indexing, and vector querying process, too.

The out-of-the-box Semantic Kernel Elasticsearch Vector Store Connector supports the Semantic Kernel vector store abstractions which make it very easy for developers to plugin Elasticsearch as a vector store while building AI agents.

Elasticsearch has a strong foundation in the open-source community and recently adopted the AGPL license. Combined with the open-source Microsoft Semantic Kernel, these tools offer a powerful, enterprise-ready solution. You can get started locally by spinning up Elasticsearch in a few minutes by running this command curl -fsSL https://elastic.co/start-local | sh (refer start-local for details) and move to cloud-hosted or self-hosted versions while productionizing your AI agents.

In this blog we look at how to use Semantic Kernel Elasticsearch Vector Store Connector when using Semantic Kernel. A Python version of the connector will be made available in the future.

High-level scenario: Building a RAG app with Semantic Kernel & Elasticsearch

In the following section we go through an example. At a high-level we are building a RAG (Retrieval Augmented Generation) application which takes a user's question as input and returns an answer. We will use Azure OpenAI (local LLM can be used as well) as the LLM, Elasticsearch as the vector store and Semantic Kernel (.net) as the framework to tie all components together.

If you are not familiar with RAG architectures, you can have a quick introduction with this article: https://www.elastic.co/search-labs/blog/retrieval-augmented-generation-rag.

The answer is generated by the LLM which is fed with context, relevant to the question, retrieved from Elasticsearch vectorstore. The response also includes the source that was used as the context by the LLM.

RAG example

In this specific example, we build an application that allows users to ask questions about hotels stored in an internal hotel database. The user could e.g. search for a specific hotel, based on different criteria, or ask for a list of hotels.

For the example database, we generated a list of hotels containing 100 entries. The sample size is intentionally small to allow you to try out the connector demo as easily as possible. In a real-world application, the Elasticsearch connector would show its advantages over other options, such as the `InMemory` vector store implementation, especially when working with extremely large amounts of data.

The complete demo application can be found in the Elasticsearch vector store connector repository.

Let’s start with adding the required NuGet packages and using directives to our project:

We can now create our data model and provide it with Semantic Kernel specific attributes to define the storage model schema and some hints for the text search:

The Storage Model Schema attributes (`VectorStore*`) are most relevant for the actual use of the Elasticsearch Vector Store Connector, namely:

• VectorStoreRecordKey to mark a property on a record class as the key under which the record is stored in a vector store.

• VectorStoreRecordData to mark a property on a record class as 'data'.

• VectorStoreRecordVector to mark a property on a record class as a vector.

All of these attributes accept various optional parameters that can be used to further customize the storage model. In the case of VectorStoreRecordKey , for example, it is possible to specify a different distance function or a different index type.

The text search attributes (TextSearch*) will be important in the last step of this example. We will come back to them later.

In the next step, we initialize the Semantic Kernel engine and obtain references to the core services. In a real world application, dependency injection should be used instead of directly accessing the service collection. The same thing applies to the hardcoded configuration and secrets, which should be read using a configuration provider instead:

The vectorStoreCollection service can now be used to create the collection and to ingest a few demo records:

This shows how Semantic Kernel reduces the use of a vector store with all its complexity to a few simple method calls.

Under the hood, a new index is created in Elasticsearch and all the necessary property mappings are created. Our data set is then mapped completely transparently into the storage model and finally stored in the index. Below is how the mappings look in Elasticsearch.

The embeddings.GenerateEmbeddingsAsync() calls transparently called the configured Azure AI Embeddings Generation service.

Even more magic can be observed in the last step of this demo.

With just a single call to InvokePromptAsync, all of the following operations are performed when the user asks a question about the data:

1. An embedding for the user's question is generated

2. The vector store is searched for relevant entries

3. The results of the query are inserted into a prompt template

4. The actual query in the form of the final prompt is sent to the AI chat completion service

Remember the TextSearch* attributes, we previously defined on our data model? These attributes enable us to use corresponding placeholders in our prompt template which are automatically populated with the information from our entries in the vector store.

The final response to our question "Please show me all hotels that have a rooftop bar." is as follows:

The answer correctly refers to the following entry in our hotels.csv

This example shows very well how the use of Microsoft Semantic Kernel achieves a significant reduction in complexity through its well thought abstractions, as well as enabling a very high level of flexibility. By changing a single line of code, for example, the vector store or the AI services used can be replaced without having to refactor any other part of the code.

At the same time, the framework provides an enormous set of high-level functionality, such as the `InvokePrompt` function, or the template or search plugin system.

The complete demo application can be found in the Elasticsearch vector store connector repository.

What else is possible with Elasticsearch

• Elasticsearch new semantic_text mapping: Simplifying semantic search

• Semantic reranking in Elasticsearch with retrievers

• Advanced RAG techniques part 1: Data processing

• Advanced RAG techniques part 2: Querying and testing

• Building RAG with Llama 3 open-source and Elastic

• A tutorial on building local agent using LangGraph, LLaMA3 and Elasticsearch vector store from scratch

Elasticsearch & Semantic Kernel: What's next?

• We showed how the Elasticsearch vector store can be easily plugged into Semantic Kernel while building GenAI applications in .NET. Stay tuned for a Python integration next.

• As Semantic Kernel builds abstractions for advanced search features like hybrid search, the Elasticsearch connect will enable .NET developers to easily implement them while using Semantic Kernel.

The code that accompanies this article implements a database of famous quotes. It includes a back end written in Python with the FastAPI web framework, and a front end written in TypeScript and React. Regarding vector search, this application demonstrates how to:

• run a local Elasticsearch service using Docker,

• bulk-ingest a large number of documents efficiently,

• generate vector embeddings for documents as they are ingested,

• leverage the power of a GPU to accelerate the generation of vector embeddings through parallelization,

• run vector search queries using the approximate kNN algorithm,

• aggregate results from vector search,

• compare vector search results against those resulting from a standard match (BM25) query.

Below you can see a screenshot of the application. In this article you will find a detailed explanation of how the ingest and search features work. You then have the option to install and run the code on your own computer to experiment and learn!

What is the Elasticsearch-DSL client for Python?

Sometimes called the "high-level" Python client, Elasticsearch-DSL offers idiomatic (or "Pythonic") access to your Elasticsearch database, in contrast with the official (or "low-level") Python client, which provides direct access to the complete range of Elasticsearch features and endpoints.

When using Elasticsearch-DSL, the structure (or "mappings") of Elasticsearch indices are defined as classes, with a syntax that is similar to that of Python dataclasses. The documents stored in these indices are represented by instances of these classes. All the transformations that are necessary to map between Python objects and Elasticsearch documents are automatically and transparently carried out, resulting in application code that is simple and idiomatic.

To add Elasticsearch-DSL to your Python project, you can install it with pip:

If your project is asynchronous, then there are additional dependencies that need to be installed, so in that case use the following command instead:

Index definition

As stated above, with Elasticsearch-DSL the structure of an Elasticsearch index is defined as a Python class. The example application featured in this article uses a dataset of famous quotes that have the following fields:

• quote: the text of the quote, as a string

• author: the name of the author, as a string

• tags: a list of tag names that apply to the quote, each a string

As part of this application we are going to add one additional field, the vector embedding that we will use to search for quotes:

• embedding: a list of floating point numbers representing a vector embedding for the quote

Let's write an initial document class to describe our famous quotes index:

The AsyncDocument class that is used as a base class for our QuoteDoc class implements all the functionality to connect the class to an Elasticsearch index. The choice of an asynchronous document base class was made because this examples uses the FastAPI web framework, which is also asynchronous. For projects that do not use asynchronous Python, the Document base class must be used when declaring document classes.

The name attribute given in the Index inner class defines the name of the Elasticsearch index that will be used with documents of this class.

If you have used Python dataclasses before, you likely find the way fields are defined very familiar, with each field being given a Python type hint. These Python types are mapped to the closest Elasticsearch type, so for example, in the case of str, the corresponding field in the Elasticsearch index will be given the type text, the standard type that is used for text that needs to be indexed for full-text search, while float is mapped to the equally named float on the Elasticsearch side.

While it can be useful to leave the quote field as is so that we can use it for both vector and full-text searches, the author and tags fields do not really need all the extra work associated with full-text search. The best Elasticsearch type for these fields is keyword, which just stores the text, without doing any indexing. Likewise, the embedding field is not just a simple list of floating point numbers, we are going to use it for vector search, which is a behavior associated with the dense_vector type in Elasticsearch.

To assign a type override to a field, we add an assignment with the mapped_field() function, as shown in the improved version of the QuoteDoc class that follows:

As you can see in this updated version, the elasticsearch_dsl package includes classes such as Keyword and DenseVector to represent all the native Elasticsearch field types.

Did you notice the init=False argument given in this new definition of the embedding field? If you are familiar with Python dataclasses you may recognize init as one of the options available in the dataclasses field() function, used to indicate that the given attribute should be omitted from the constructor for instances of the class. The behavior is the same here, which means that when creating an instance of QuoteDoc, this argument should not be given.

How will the vector embeddings be generated if they will not be passed down to the document constructor? Elasticsearch-DSL always calls the clean() method in all documents before serializing them and sending them to Elasticsearch. This method is a convenience entry point where the application can add any custom field processing logic. For example, fields that are optional or auto-generated can be added in this method. Here is the final version of the QuoteDoc document class, including the logic that generates the embeddings:

For this example we are going to use embeddings from a SentenceTransformers model. These embeddings are easy to generate locally and being open source and free they are convenient to use when experimenting. The all-MiniLM-L6-v2 model is a great general purpose embedding model for English text. There are many other models that are also compatible with the SentenceTransformers framework, so feel free to use a different one if you prefer.

The clean() method can be used for more advanced use cases as well. For example, it is common when working with large bodies of text to split the text into smaller chunks, and then generate embeddings for each chunk. Elasticsearch accommodates this use case through nested objects. If you want to see an advanced example that implements this type of solution, check out the vectors example in the Elasticsearch-DSL repository.

Document ingestion

With the structure of the index in place, we can now create the index. This is done with the init() class method:

In many cases it is useful to delete a previously existing index to make sure an ingest process begins from a clean starting point. This can be done using the _index class attribute, which provides access to the Elasticsearch index, along with its exists() and delete() methods:

The example dataset used by the example application is a collection of almost 37,000 famous quotes. It comes as a CSV file with the quote, author and tags columns. The tags are given as a comma-separated string. The dataset is available for download from the example GitHub repository.

To ingest the data contained in this dataset, Python's csv module can be used:

The csv.DictReader class creates a CSV file importer that returns a dictionary for each row in the data file. For each row, we create a QuoteDoc instance and pass the quote, author and tags in the constructor. For the tags, the string that is read from the CSV file has to be split into a list, which is how it will be stored in the Elasticsearch index.

To write a document to the index, the save() method is invoked. This method will call the document's clean() method, which in turn will generate the vector embedding for the quote.

Starting an Elasticsearch instance

Before the above ingest script can be executed, you need to have access to a running instance of Elasticsearch. By far the easiest (and also 100% free) way to do this is with a Docker container.

To start a single-node Elasticsearch service on your computer first make sure you have Docker running, and then execute the following command:

To make sure you are running the latest and greatest version, open the release notes page to find out what is the current version, then replace the version number in the last line of the above command.

The -v option in the command above sets up a mapping between a directory named data in your local system and the data directory in the Elasticsearch container. All the data files used by Elasticsearch will be saved in this directory, so that in case you need to restart your container you do not lose any data. If you prefer to not store the data files in your computer, then you can remove the -v line and the data will be stored ephemerally in the container.

Note that deploying Elasticsearch using this method is only adequate for local experimentation. If you intend to deploy Elasticsearch on a production server, consider using our Elasticsearch on Docker Compose or Elasticsearch on Kubernetes guides.

Connecting to Elasticsearch

The ingestion script needs to know how to connect to Elasticsearch. If you are running a Docker container as demonstrated in the previous section, add the following line between the imports and the definition of the QuoteDoc class:

To complete the script, the ingest_quotes() function should be called. Add the following snippet at the bottom of your source file:

The asyncio.run() function will launch the asynchronous application. If your application is not asynchronous, then you would just call the ingest function directly.

For your convenience, below you can find the complete code for the script up to this point. You can save this file as search.py. You can find an example of this file here.

Create a virtual environment for your project using the tool of your choice, and then install the dependencies on it:

Make sure you have the quotes.csv file in the current directory, and then start the ingest by running the script:

The script does not print anything, so it will run for a while adding the quotes from the CSV file into your Elasticsearch index. The file has about 37,000 quotes, so expect the process to run for several minutes.

Luckily you do not need to wait that long. If you start the script and no error appears, that is confirmation that everything is working. You can press Ctrl-C to stop it and continue reading to learn about ingest performance.

Performance tuning part 1: bulk processing

If your dataset is small, then the above ingest solution will work just fine, and it has the benefit that it is simple to code and easy to understand.

For larger ingest jobs, however, it is necessary to sacrifice code clarity and pay attention to performance, so let's see what optimizations can be done in this application.

First of all, to evaluate performance we need to be able to measure the performance of the existing solution. Below is the updated ingest_quotes() function, which now calls ingest_progress() every 100 ingested documents to show how many documents have been ingested, along with an average document per second.

This version of the ingest is nicer than the previous one because it prints regular status updates. If you let the script run for a while you may see an output similar to the one below:

The data file has close to 37,000 quotes, so now you can have a good idea of how long the ingest will take. Assuming the average of 97 ingested documents per second holds throughout the entire ingest job, it should take less than 7 minutes to ingest the entire dataset. You can press Ctrl-C to stop this ingest process, there is no need to let it run to completion yet.

Elasticsearch offers a very flexible bulk ingest feature, which is made available in the Elasticsearch-DSL package's bulk() method. Instead of saving each document, the entire import loop can be moved into a generator function which is given to the bulk() method as an argument:

Here the get_next_quote() inner generator function yields QuoteDoc instances. The QuoteDoc.bulk() method will run the generator and issue batch updates to Elasticsearch. With this change, you can expect to see a small speed improvement:

For another small improvement, the JSON serializer used by the Elasticsearch client can be changed to the orjson library, which performs better than Python's own:

This should lead to another small performance improvement:

Performance tuning part 2: GPU accelerated embeddings

You have seen in the previous section that we have obtained some modest performance improvements by processing ingest requests in bulk. But while ingestion requests are now being grouped, the embeddings continue to be generated one by one in the clean() method of the QuoteDoc class.

Is there a way to optimize embedding generation? The SentenceTransformers model uses PyTorch, which in turn uses a GPU if one is available. But the embeddings are generated individually, which does not lead to an optimal utilization of the GPU hardware. GPUs are very good at parallelization, so we can reorganize the ingest function to generate embeddings in batches. And once again the price we pay for this comes in increased code complexity.

So we are going to stop using the clean() method to generate document embeddings, and instead we are going to accumulate the QuoteDoc instances in a list, and once we reach a good number we'll generate embeddings for all of them in a single operation.

Let's start by writing a helper function that generates embeddings for a list of QuoteDoc instances:

Note how now the model.encode() method is given a list of quotes to embed instead of a single one. When the input argument is a list, the model generates an embedding for each list element. The method accepts an optional batch_size argument (not used in the example above) that defaults to 32 that can be used to control the size of each batch of samples that are sent to the model for computation. Depending on the GPU hardware you may find that different values of this argument help tune performance to the best possible. Once the embeddings are generated, they are assigned to each quote using a for-loop.

Now the ingest function can be refactored to accumulate quotes and use the helper function to generate embeddings:

In this version of ingest_quotes(), each QuoteDoc instance is added to the quotes list, and when 512 elements have accumulated the embed_quotes() function added above is used to generate the embeddings more efficiently. Once the objects have their embeddings, they are yielded, so that the bulk() method from Elasticsearch-DSL can add them to the index as before.

What is the significance of the 512 number? There isn't any. We know that the model uses a batch size of 32, so it makes sense to accumulate at least that many documents. Starting from 32, you can try if larger powers of 2 provide better performance. With the hardware available to me, I've found 512 to give the best performance.

Here is an example run using batched embeddings:

And now the ingestion process runs much faster, with the entire dataset ingested in about 1 minutes and 16 seconds.

If you decide to try to optimize your ingest, you are encouraged to try different options and see what works best with your hardware.

Querying the index

If you are following along, by now you have an Elasticsearch index called quotes that is populated with about 37K famous quotes, each with a searchable vector embedding. Now it is time to learn how to query this index.

When using Elasticsearch-DSL, the document classes return a search object from their search() method:

The search object has a large number of methods that map to the query options in the Elasticsearch query DSL.

The simplest query that can be issued is the match all query, which returns all the elements. With the class-based approach used by Elasticsearch-DSL, this is how to run the query:

This would obviously print a listing of the entire list of quotes stored in the index, up to 10,000, which is the maximum number of results Elasticsearch returns by default.

In many cases it is useful to request a subset of the results. The search object uses Python style slicing for this. Here is how to request the first 25 results only:

Here is how to request the second page of results, at 25 results per page:

Elasticsearch offers approximate and exact vector search queries, also called k-nearest neighbor (kNN) queries. To run a vector search query with the approximate k-nearest neighbor algorithm, the Knn query should be used:

The Knn query class accepts the field that stores the embeddings and a search vector as arguments. In the above snippet the variable q has the search text entered by the user.

If instead you prefer to run a regular full-text search, the Match query class is used:

Filters

One of the most important benefits of using Elasticsearch as a vector database is that it is a robust database system, and all the options you can expect to have from a database nicely integrates with your vector search queries.

A great example of this is filters. The famous quotes database stores a list of tags for each quote, so it is only natural to have the option to restrict a query to quotes that have a specific tag.

Given a list of tag filters stored in a tags variable, the following snippet configures a search object to only return results that include the given tags using a "terms" filter:

Aggregations

Another example of a useful database function that is fully integrated with vector search is aggregations. Given a query, Elasticsearch can aggregate the tags and provide the counts of quotes per tag.

The next snippet shows how to add a Terms aggregation to an existing query, which will return the 100 most referenced tags in the results:

Recall that the tags field was declared with the Keyword() type, which means that the tags will be stored as is on the index, without any processing. This is required by the Terms aggregation, which will count the occurrences of each tag in the results.

A complete query example

You have seen a few isolated query examples. In this section you can see how they can all be integrated into a function that performs a query in the example application.

The search_quotes() function shown below accepts a query string q, a list of filters tags and a use_knn flag to choose between kNN or full-text search query. It also accepts start and size pagination arguments.

The function decides which of the three queries you've seen above to issue depending on the input arguments. If q is empty, then it selects a "match all" query, and in any other case it selects a kNN or match query depending on the use_knn flag, which the user can control from a checkbox in the application's user interface.

The function returns three results as a tuple:

• a list of QuoteDoc instances that are the search results,

• the tag aggregations as a list of tuples, each with tag name and document count,

• the total number of results, which is useful to show in paginated queries

Here is the complete code of this function:

To be able to access both the search results and the aggregation results, we now issue the request explicitly through the execute() method and store the response is stored in r. The hits attribute of the response object contains the actual search results, and the aggs attribute provides access to the aggregations. The format in which the aggregation results is provided is described in the terms aggregation documentation.

Conclusion

The complete quotes example is available in a GitHub repository that you can install and run on your computer. Follow the instructions on the README.md file to set it up.

You are welcome to use this example to experiment with vector embeddings and Elasticsearch!

Welcome to Part 2 of our article on Advanced RAG Techniques! In part 1 of this series, we set up, discussed, and implemented the data processing components of the advanced RAG pipeline:

In this part, we're going to proceed with querying and testing out our implementation. Let's get right to it!

Table of contents

• Searching and retrieving, generating answers

  • Enriching queries with synonyms

  • HyDE (Hypothetical Document Embedding)

  • Hybrid search

• Experiments

  • Summary of results

  • Test 1: Who audits Elastic?

    • AdvancedRAG

    • SimpleRAG

  • Test 2: total revenue 2023

    • AdvancedRAG

    • SimpleRAG

  • Test 3: What product does growth primarily depend on? How much?

    • AdvancedRAG

    • SimpleRAG

  • Test 4: Describe employee benefit plan

    • AdvancedRAG

    • SimpleRAG

  • Test 5: Which companies did Elastic acquire?

    • AdvancedRAG

    • SimpleRAG

• Conclusion

• Appendix

  • Prompts

    • RAG question answering prompt

    • Elastic query generator prompt

    • Potential questions generator prompt

    • HyDE generator prompt

  • Sample hybrid search query

Searching and retrieving, generating answers

Let's ask our first query, ideally some piece of information found primarily in the annual report. How about:

Now, let's apply a few of our techniques to enhance the query.

Enriching queries with synonyms

Firstly, let's enhance the diversity of the query wording, and turn it into a form that can be easily processed into an Elasticsearch query. We'll enlist the aid of GPT-4o to convert the query into a list of OR clauses. Let's write this prompt:

When applied to our query, GPT-4o generates synonyms of the base query and related vocabulary.

In the ESQueryMaker class, I've defined a function to split the query:

Its role is to take this string of OR clauses and split them into a list of terms, allowing us do a multi-match on our key document fields:

Finally ending up with this query:

This covers many more bases than the original query, hopefully reducing the risk of missing a search result because we forgot a synonym. But we can do more.

Back to top

HyDE (Hypothetical Document Embedding)

Let's enlist GPT-4o again, this time to implement HyDE.

The basic premise of HyDE is to generate a hypothetical document - The kind of document that would likely contain the answer to the original query. The factuality or accuracy of the document is not a concern. With that in mind, let's write the following prompt:

Since vector search typically operates on cosine vector similarity, the premise of HyDE is that we can achieve better results by matching documents to documents instead of queries to documents.

What we care about is structure, flow, and terminology. Not so much factuality. GPT-4o outputs a HyDE document like this:

It looks pretty believable, like the ideal candidate for the kinds of documents we'd like to index. We're going to embed this and use it for hybrid search.

Back to top

Hybrid search

This is the core of our search logic. Our lexical search component will be the generated OR clause strings. Our dense vector component will be embedded HyDE Document (aka the search vector). We use KNN to efficiently identify several candidate documents closest to our search vector. We call our lexical search component Scoring with TF-IDF and BM25 by default. Finally, the lexical and dense vector scores will be combined using the 30/70 ratio recommended by Wang et al.

Finally, we can piece together a RAG function. Our RAG, from query to answer, will follow this flow:

1. Convert Query to OR Clauses.

2. Generate HyDE document and embed it.

3. Pass both as inputs to Hybrid Search.

4. Retrieve top-n results, reverse them so that the most relevant score is the "most recent" in the LLM's contextual memory (Reverse Packing) Reverse Packing Example: Query: "Elasticsearch query optimization techniques" Retrieved documents (ordered by relevance): Reversed order for LLM context: By reversing the order, the most relevant information (1) appears last in the context, potentially receiving more attention from the LLM during answer generation.

   1. "Use bool queries to combine multiple search criteria efficiently."

   2. "Implement caching strategies to improve query response times."

   3. "Optimize index mappings for faster search performance."

   4. "Optimize index mappings for faster search performance."

   5. "Implement caching strategies to improve query response times."

   6. "Use bool queries to combine multiple search criteria efficiently."

5. Pass the context to the LLM for generation.

Let's run our query and get back our answer:

Nice. That's correct.

Back to top

Experiments

There's an important question to answer now. What did we get out of investing so much effort and additional complexity into these implementations?

Let's do a little comparison. The RAG pipeline we've implemented versus baseline hybrid search, without any of the enhancements we've made. We'll run a small series of tests and see if we notice any substantial differences. We'll refer to the RAG we have just implemented as AdvancedRAG, and the basic pipeline as SimpleRAG.

Summary of results

This table summarizes the results of five tests of both RAG pipelines. I judged the relative superiority of each method based on answer detail and quality, but this is a totally subjective judgement. The actual answers are reproduced below this table for your consideration. With that said, let's take a look at how they did!

SimpleRAG was unable to answer questions 1 & 5. AdvancedRAG also went into far greater detail on questions 2, 3, and 4. Based on the increased detail, I judged the quality of AdvancedRAG's answers better.

Test

Question

AdvancedRAG Performance

SimpleRAG Performance

AdvancedRAG Latency

SimpleRAG Latency

Winner

1

Who audits Elastic?

Correctly identified PwC as the auditor.

Failed to identify the auditor.

11.6s

4.4s

AdvancedRAG

2

What was the total revenue in 2023?

Provided the correct revenue figure. Included additional context with revenue from previous years.

Provided the correct revenue figure.

13.3s

2.8s

AdvancedRAG

3

What product does growth primarily depend on? How much?

Correctly identified Elastic Cloud as the key driver. Included overall revenue context & greater detail.

Correctly identified Elastic Cloud as the key driver.

14.1s

12.8s

AdvancedRAG

4

Describe employee benefit plan

Gave a comprehensive description of retirement plans, health programs, and other benefits. Included specific contribution amounts for different years.

Provided a good overview of benefits, including compensation, retirement plans, work environment, and the Elastic Cares program.

26.6s

11.6s

AdvancedRAG

5

Which companies did Elastic acquire?

Correctly listed recent acquisitions mentioned in the report (CmdWatch, Build Security, Optimyze). Provided some acquisition dates and purchase prices.

Failed to retrieve relevant information from the provided context.

11.9s

2.7s

AdvancedRAG

Test 1: Who audits Elastic?

AdvancedRAG

SimpleRAG

Summary: SimpleRAG did not identify PWC as the auditor

Okay that's actually quite surprising. That looks like a search failure on SimpleRAG's part. No documents related to auditing were retrieved. Let's dial down the difficulty a little with the next test.

Test 2: total revenue 2023

AdvancedRAG

SimpleRAG

Summary: Both RAGs got the right answer: $1,068,989,000 total revenue in 2023

Both of them were right here. It does seem like AdvancedRAG may have acquired a broader range of documents? Certainly the answer is more detailed and incorporates information from previous years. That is to be expected given the enhancements we made, but it's far too early to call.

Let's raise the difficulty.

Test 3: What product does growth primarily depend on? How much?

AdvancedRAG

SimpleRAG

Summary: Both RAGs correctly identified Elastic Cloud as the key growth driver. However, AdvancedRAG includes more detail, factoring in subscription revenues and customer growth, and explicitly mentions other Elastic offerings.

Test 4: Describe employee benefit plan

AdvancedRAG

SimpleRAG

Summary: AdvancedRAG goes into much greater depth and detail, mentioning the 401K plan for US-based employees, as well as defining contribution plans outside of the US. It also mentions Health and Well-Being plans but misses the Elastic Cares program, which SimpleRAG mentions.

Test 5: Which companies did Elastic acquire?

AdvancedRAG

SimpleRAG

Summary: SimpleRAG does not retrieve any relevant info about acquisitions, leading to a failed answer. AdvancedRAG correctly lists CmdWatch, Build Security, and Optimyze, which were the key acquisitions listed in the report.

Back to top

Conclusion

Based on our tests, our advanced techniques appear to increase the range and depth of the information presented, potentially enhancing quality of RAG answers.

Additionally, there may be improvements in reliability, as ambiguously worded questions such as Which companies did Elastic acquire? and Who audits Elastic were correctly answered by AdvancedRAG but not by SimpleRAG.

However, it is worth keeping in perspective that in 3 out of 5 cases, the basic RAG pipeline, incorporating Hybrid Search but no other techniques, managed to produce answers that captured most of the key information.

We should note that due to the incorporation of LLMs at the data preparation and query phases, the latency of AdvancedRAG is generally between 2-5x larger that of SimpleRAG. This is a significant cost which may make AdvancedRAG suitable only for situations where answer quality is prioritized over latency.

The significant latency costs can be alleviated using a smaller and cheaper LLM like Claude Haiku or GPT-4o-mini at the data preparation stage. Save the advanced models for answer generation.

This aligns with the findings of Wang et al. As their results show, any improvements made are relatively incremental. In short, simple baseline RAG gets you most of the way to a decent end-product, while being cheaper and faster to boot. For me, it's an interesting conclusion. For use cases where speed and efficiency are key, SimpleRAG is the sensible choice. For use cases where every last drop of performance needs squeezing out, the techniques incorporated into AdvancedRAG may offer a way forward.

Back to top

Appendix

Prompts

RAG question answering prompt

Prompt for getting the LLM to generate answers based on query and context.

Elastic query generator prompt

Prompt for enriching queries with synonyms and converting them into the OR format.

Potential questions generator prompt

Prompt for generating potential questions, enriching document metadata.

HyDE generator prompt

Prompt for generating hypothetical documents using HyDE

Sample hybrid search query

The recent paper Searching for Best Practices in Retrieval-Augmented Generation empirically assesses the efficacy of various RAG enhancing techniques, with the goal of converging on a set of best-practices for RAG.

We'll implement a few of these proposed best-practices, namely the ones which aim to improve the quality of search (Sentence Chunking, HyDE, Reverse Packing).

For brevity, we will omit those techniques focused on improving efficiency (Query Classification and Summarization).

We will also implement a few techniques that were not covered, but which I personally find useful and interesting (Metadata Inclusion, Composite Multi-Field Embeddings, Query Enrichment).

Finally, we'll run a short test to see if the quality of our search results and generated answers has improved versus the baseline. Let's get to it!

RAG overview

RAG aims to enhance LLMs by retrieving information from external knowledge bases to enrich generated answers. By providing domain-specific information, LLMs can be quickly adapted for use cases outside the scope of their training data; significantly cheaper than fine-tuning, and easier to keep up-to-date.

Measures to improve the quality of RAG typically focus on two tracks:

1. Enhancing the quality and clarity of the knowledge base.

2. Improving the coverage and specificity of search queries.

These two measures will achieve the goal of improving the odds that the LLM has access to relevant facts and information, and is thus less likely to hallucinate or draw upon its own knowledge - which may be outdated or irrelevant.

The diversity of methods is difficult to clarify in just a few sentences. Let's go straight to implementation to make things clearer.

Table of contents

• Overview

  • Table of contents

• Set-up

• Ingesting, processing, and embedding documents

  • Data ingestion

  • Sentence-level, token-wise chunking

  • Metadata inclusion and generation

    • Keyphrases extracted by TextRank

    • Potential questions generated by GPT-4o

    • Entities extracted by Spacy

  • Composite multi-field embeddings

    • Indexing to Elastic

• Cat break

• Appendix

  • Definitions

Set-up

All code may be found in the Searchlabs repo.

First things first. You will need the following:

1. An Elastic Cloud Deployment

2. An LLM API - We are using a GPT-4o deployment on Azure OpenAI in this notebook

3. Python Version 3.12.4 or later

We will be running all the code from the main.ipynb notebook.

Go ahead and git clone the repo, navigate to supporting-blog-content/advanced-rag-techniques, then run the following commands:

Once that's done, create a .env file and fill out the following fields (Referenced in .env.example). Credits to my co-author, Claude-3.5, for the helpful comments.

Next, we'll choose the document to ingest, and place it in the documents folder. For this article, we'll be using the Elastic N.V. Annual Report 2023. It's a pretty challenging and dense document, perfect for stress testing our RAG techniques.

Now we're all set, let's go to ingestion. Open main.ipynb and execute the first two cells to import all packages and intialize all services.

Back to top

Ingesting, processing, and embedding documents

Data ingestion

• Personal note: I am stunned by LlamaIndex's convenience. In the olden days before LLMs and LlamaIndex, ingesting documents of various formats was a painful process of collecting esoteric packages from all over. Now it's reduced to a single function call. Wild.

The SimpleDirectoryReader will load every document in the directory_path. For .pdf files, it returns a list of document objects, which I convert to Python dictionaries because I find them easier to work with.

Each dictionary contains the key content in the text field. It also contains useful metadata such as page number, filename, file size, and type.

Back to top

Sentence-level, token-wise chunking

The first thing to do is reduce our documents to chunks of a standard length (to ensure consistency and manageability). Embedding models have unique token limits (maximum input size they can process). Tokens are the basic units of text that models process. To prevent information loss (truncation or omission of content), we should provide text that does not exceed those limits (by splitting longer texts into smaller segments).

Chunking has a significant impact on performance. Ideally, each chunk would represent a self-contained piece of information, capturing contextual information about a single topic. Chunking methods include word-level chunking, where documents are split by word count, and semantic chunking which uses an LLM to identify logical breakpoints.

Word-level chunking is cheap, fast, and easy, but runs a risk of splitting sentences and thus breaking context. Semantic chunking gets slow and expensive, especially if you're dealing with documents like the 116-page Elastic Annual Report.

Let's choose a middleground approach. Sentence level chunking is still simple, but can preserve context more effectively than word-level chunking while being significantly cheaper and faster. Additionally, we'll implement a sliding window to capture some of the surrounding context, and alleviate the impact of splitting paragraphs.

The Chunker class takes in the embedding model's tokenizer to encode and decode text. We'll now build chunks of 512 tokens each, with an overlap of 20 tokens. To do this, we'll split the text into sentences, tokenize those sentences, and then add the tokenized sentences to our current chunk until we cannot add more without breaching our token limit.

Finally, decode the sentences back to the original text for embedding, storing it in a field called original_text. Chunks are stored in a field called chunk. To reduce noise (aka useless documents), we will discard any documents smaller than 50 tokens in length.

Let's run it over our documents:

And get back chunks of text that look like this:

Back to top

Metadata inclusion and generation

We've chunked our documents. Now it's time to enrich the data. I want to generate or extract additional metadata. This additional metadata can be used to influence and enhance search performance.

We'll define a DocumentEnricher class, whose role is to take in a list of documents (Python dictionaries), and a list of processor functions. These functions will run over the documents' original_text column, and store their outputs in new fields.

First, we extract keyphrases using TextRank. TextRank is a graph-based algorithm that extracts key phrases and sentences from text by ranking their importance based on the relationships between words.

Next, we'll generate potential_questions using GPT-4o.

Finally, we'll extract entities using Spacy.

Since the code for each of these is quite lengthy and involved, I will refrain from reproducing it here. If you are interested, the files are marked in the code samples below.

Let's run the data enrichment:

And take a look at the results:

Keyphrases extracted by TextRank

These keyphrases are a stand-in for the chunk's core topics. If a query has to do with cybersecurity, this chunk's score will be boosted.

Potential questions generated by GPT-4o

These potential questions may directly match with user queries, offering a boost in score. We prompt GPT-4o to generate questions which can be answered using the information found in the current chunk.

Entities extracted by Spacy

These entities serve a similar purpose to the keyphrases, but capture organizations' and individuals' names, which keyphrase extraction may miss.

Back to top

Composite multi-field embeddings

Now that we have enriched our documents with additional metadata, we can leverage this information to create more robust and context-aware embeddings.

Let's review our current point in the process. We've got four fields of interest in each document.

Each field represents a different perspective on the document's context, potentially highlighting a key area for the LLM to focus on.

The plan is to embed each of these fields, and then create a weighted sum of the embeddings, known as a Composite Embedding.

With luck, this Composite Embedding will allow the system to become more context aware, in addition to introducing another tunable hyperparameter from controlling the search behavior.

First, let's embed each field and update each document in place, using our locally defined embedding model imported at the beginning of the main.ipynb notebook.

Each embedding function returns the embedding's field, which is just the original input field with an _embedding postfix.

Let's now define the weightings of our composite embedding:

The weightings allow you to assign priorities to each component, based on your usecase and the quality of your data. Intuitively, the size of these weightings is dependent on the semantic value of each component. Since the chunk text itself is by far the richest, I assign a weighting of 70%. Since the entities are the smallest, being just a list of org or person names, I assign it a weighting of 5%. The precise setting for these values has to be determined empirically, on a use-case by use-case basis.

Finally, let's write a function to apply the weightings, and create our composite embedding. We'll delete all the component embeddings as well to save space.

With this, we've completed our document processing. We now have a list of document objects which look like this:

Indexing to Elastic

Let's bulk upload our documents to Elastic Search. For this purpose, I long-ago defined a set of Elastic Helper functions in elastic_helpers.py. It is a very lengthy piece of code so let's sticking to looking at the function calls.

es_bulk_indexer.bulk_upload_documents works with any list of dictionary objects, taking advantage of Elasticsearch's convenient dynamic mappings.

Head on over to Kibana and verify that all documents have been indexed. There should be 224 of them. Not bad for such a large document!

Back to top

Cat break

Let's take a break, article's a little heavy, I know. Check out my cat:

Adorable. The hat went missing and I half suspect she stole and hid it somewhere :(

Congrats on making it this far :)

Join me in Part 2 for testing and evaluation of our RAG pipeline!

Appendix

Definitions

1. Sentence Chunking

• A preprocessing technique used in RAG systems to divide text into smaller, meaningful units.

• Process:

  1. Input: Large block of text (e.g., document, paragraph)

  2. Output: Smaller text segments (typically sentences or small groups of sentences)

• Purpose:

  • Creates granular, context-specific text segments

  • Allows for more precise indexing and retrieval

  • Improves the relevance of retrieved information in RAG systems

• Characteristics:

  • Segments are semantically meaningful

  • Can be independently indexed and retrieved

  • Often preserves some context to ensure standalone comprehensibility

• Benefits:

  • Enhances retrieval precision

  • Enables more focused augmentation in RAG pipelines

2. HyDE (Hypothetical Document Embedding)

• A technique that uses an LLM to generate a hypothetical document for query expansion in RAG systems.

• Process:

  1. Input query to an LLM

  2. LLM generates a hypothetical document answering the query

  3. Embed the generated document

  4. Use the embedding for vector search

• Key difference:

  • Traditional RAG: Matches query to documents

  • HyDE: Matches documents to documents

• Purpose:

  • Improve retrieval performance, especially for complex or ambiguous queries

  • Capture richer semantic context than a short query

• Benefits:

  • Leverages LLM's knowledge to expand queries

  • Can potentially improve relevance of retrieved documents

• Challenges:

  • Requires additional LLM inference, increasing latency and cost

  • Performance depends on quality of generated hypothetical document

3. Reverse Packing

• A technique used in RAG systems to reorder search results before passing them to the LLM.

• Process:

  1. Search engine (e.g., Elasticsearch) returns documents in descending order of relevance.

  2. The order is reversed, placing the most relevant document last.

• Purpose:

  • Exploits the recency bias of LLMs, which tend to focus more on the latest information in their context.

  • Ensures the most relevant information is "freshest" in the LLM's context window.

• Example: Original order: [Most Relevant, Second Most, Third Most, ...] Reversed order: [..., Third Most, Second Most, Most Relevant]

4. Query Classification

• A technique to optimize RAG system efficiency by determining whether a query requires RAG or can be answered directly by the LLM.

• Process:

  1. Develop a custom dataset specific to the LLM in use

  2. Train a specialized classification model

  3. Use the model to categorize incoming queries

• Purpose:

  • Improve system efficiency by avoiding unnecessary RAG processing

  • Direct queries to the most appropriate response mechanism

• Requirements:

  • LLM-specific dataset and model

  • Ongoing refinement to maintain accuracy

• Benefits:

  • Reduces computational overhead for simple queries

  • Potentially improves response time for non-RAG queries

5. Summarization

• A technique to condense retrieved documents in RAG systems.

• Process:

  1. Retrieve relevant documents

  2. Generate concise summaries of each document

  3. Use summaries instead of full documents in the RAG pipeline

• Purpose:

  • Improve RAG performance by focusing on essential information

  • Reduce noise and interference from less relevant content

• Benefits:

  • Potentially improves relevance of LLM responses

  • Allows for inclusion of more documents within context limits

• Challenges:

  • Risk of losing important details in summarization

  • Additional computational overhead for summary generation

6. Metadata Inclusion

• A technique to enrich documents with additional contextual information.

• Types of metadata:

  • Keyphrases

  • Titles

  • Dates

  • Authorship details

  • Blurbs

• Purpose:

  • Increase contextual information available to the RAG system

  • Provide LLMs with clearer understanding of document content and relevance

• Benefits:

  • Potentially improves retrieval accuracy

  • Enhances LLM's ability to assess document usefulness

• Implementation:

  • Can be done during document preprocessing

  • May require additional data extraction or generation steps

7. Composite Multi-Field Embeddings

• An advanced embedding technique for RAG systems that creates separate embeddings for different document components.

• Process:

  1. Identify relevant fields (e.g., title, keyphrases, blurb, main content)

  2. Generate separate embeddings for each field

  3. Combine or store these embeddings for use in retrieval

• Difference from standard approach:

  • Traditional: Single embedding for entire document

  • Composite: Multiple embeddings for different document aspects

• Purpose:

  • Create more nuanced and context-aware document representations

  • Capture information from a wider variety of sources within a document

• Benefits:

  • Potentially improves performance on ambiguous or multi-faceted queries

  • Allows for more flexible weighting of different document aspects in retrieval

• Challenges:

  • Increased complexity in embedding storage and retrieval processes

  • May require more sophisticated matching algorithms

8. Query Enrichment

• A technique to expand the original query with related terms to improve search coverage.

• Process:

  1. Analyze the original query

  2. Generate synonyms and semantically related phrases

  3. Augment the query with these additional terms

• Purpose:

  • Increase the range of potential matches in the document corpus

  • Improve retrieval performance for queries with specific or technical language

• Benefits:

  • Potentially retrieves relevant documents that don't exactly match the original query terms

  • Can help overcome vocabulary mismatch between queries and documents

• Challenges:

  • Risk of query drift if not carefully implemented

  • May increase computational overhead in the retrieval process

Back to top

Imagine a multinational company with support tickets and solutions in different languages and wants to take advantage of that knowledge across divisions. Currently, knowledge is limited to the language the agent speaks. Let's fix that!

In this article, I’m going to show you how to test Mixtral’s language capabilities, by creating a multilingual RAG system.

You can follow the notebook to reproduce this article's example here

Steps

1. Creating embeddings endpoint

2. Creating mappings

3. Indexing data

4. Asking questions

Creating embeddings endpoint

Our support tickets for this example will come in English, Spanish, and German. The Mistral embeddings model is not multilingual, but we can generate multilingual embeddings using the e5 model, so we can index text on different languages and manage it as a single source, giving us a much richer context.

To create e5 multilingual embeddings you can use Kibana:

Or the _inference API:

Creating Mappings

For the mappings we will use semantic_text mapping type, which is one of my favorite features. It handles the process of chunking the data, generating embeddings, and querying embeddings for you!

We call the text field super_body because with a single mapping type it will handle chunks and embeddings.

Indexing data

We will index a couple of support tickets with problems and solutions in two languages, and then ask a question about problems within many documents in a third.

The following documents will be added to the index:

1. English Support Ticket: Calendar Sync Issue

Support Ticket #EN1234 Subject: Calendar sync not working with Google Calendar

Description: I'm having trouble syncing my project deadlines with Google Calendar. Whenever I try to sync, I get an error message saying "Unable to connect to external calendar service."

Resolution: The issue was resolved by following these steps:

1. Go to Settings > Integrations

1. Disconnect the Google Calendar integration

1. Clear browser cache and cookies

1. Reconnect the Google Calendar integration

1. Authorize the app again in Google's security settings

The sync should now work correctly. If problems persist, ensure that third-party cookies are enabled in your browser settings.

2. German Support Ticket: File Upload Problem

Support-Ticket #DE5678 Betreff: Datei-Upload funktioniert nicht

Beschreibung: Ich kann keine Dateien mehr in meine Projekte hochladen. Jedes Mal, wenn ich es versuche, bleibt der Ladebalken bei 99% stehen und dann erscheint eine Fehlermeldung.

Lösung: Das Problem wurde durch folgende Schritte gelöst:

1. Überprüfen Sie die Dateigröße. Die maximale Uploadgröße beträgt 100 MB.

1. Deaktivieren Sie vorübergehend den Virenschutz oder die Firewall.

1. Versuchen Sie, die Datei im Inkognito-Modus hochzuladen.

1. Wenn das nicht funktioniert, leeren Sie den Browser-Cache und die Cookies.

1. Als letzten Ausweg, versuchen Sie einen anderen Browser zu verwenden.

In den meisten Fällen lag das Problem an zu großen Dateien oder an Interferenzen durch Sicherheitssoftware. Nach Anwendung dieser Schritte sollte der Upload funktionieren.

3. Marketing Campaign Ideas (noise)

Q3 Marketing Campaign Ideas

1. Social media contest: "Share Your Productivity Hack"

   • Users share tips using our software, best entry wins a premium subscription.

1. Webinar series: "Mastering Project Management"

   • Invite industry experts to share insights using our tool.

1. Email campaign: "Unlock Hidden Features"

   • Series of emails highlighting lesser-known but powerful features.

1. Partner with a productivity podcast for sponsored content.

1. Create a "Project Management Memes" social media account for lighter, shareable content.

4. Mitarbeiter des Monats (noise)

Mitarbeiter des Monats: Juli 2023

Wir freuen uns, bekannt zu geben, dass Sarah Schmidt zur Mitarbeiterin des Monats Juli gewählt wurde!

Sarah hat außergewöhnliche Leistungen in folgenden Bereichen gezeigt:

• Kundenbetreuung: Sarah hat durchschnittlich 95% positive Bewertungen erhalten.

• Teamarbeit: Sie hat maßgeblich zur Verbesserung unseres internen Wissensmanagementsystems beigetragen.

• Innovation: Sarah hat eine neue Methode zur Priorisierung von Support-Tickets vorgeschlagen, die unsere Reaktionszeiten um 20% verbessert hat.

Bitte gratulieren Sie Sarah zu dieser wohlverdienten Anerkennung!

This is how a document will look like inside Elasticsearch:

Asking questions

Now, we are going to ask a question in Spanish:

The expectation is retrieving documents #1 and #2, then sending them to the LLM as additional context, and finally, getting an answer in Spanish.

Retrieving documents

To retrieve the relevant documents, we can use this nice and short query that will run a search on the embeddings, and return the support tickets most relevant to the question.

Notes about the parameters set: size: 2 Because we know we want the top 2 documents. excludes For clarity in the response. Documents are short so each one will be one chunk long.

Answering the question

Now we can call the Mistral completion API using the Python library to answer the question.

The answer is in perfect Spanish and on point!

Show answer

Hola, me alegra que te hayas comunicado con nosotros. Parece que hay dos problemas distintos.

En cuanto a la sincronización del calendario, puedes seguir estos pasos para resolver el problema:

1. Ve a Configuración > Integraciones

1. Desconecta la integración del Calendario de Google

1. Borra la caché y las cookies del navegador

1. Vuelve a conectar la integración del Calendario de Google

1. Autoriza de nuevo la aplicación en la configuración de seguridad de Google

Si sigues teniendo problemas, asegúrate de que las cookies de terceros están habilitadas en la configuración de tu navegador.

En cuanto al problema de subir un archivo, hay varias cosas que puedes probar:

1. Comprueba el tamaño del archivo. El tamaño máximo de carga es de 100 MB.

1. Desactiva temporalmente el antivirus o el cortafuegos.

1. Intenta cargar el archivo en modo incógnito.

1. Si eso no funciona, borra la caché y las cookies del navegador.

1. Como último recurso, prueba a usar un navegador diferente.

En la mayoría de los casos, el problema se debe a archivos demasiado grandes o a interferencias causadas por software de seguridad. Al seguir estos pasos, deberías poder cargar el archivo correctamente.

¡Espero que esto te ayude a resolver tus problemas! Si tienes alguna otra pregunta, no dudes en preguntar.

Conclusion

Mixtral 8x22B is a powerful model that enables us to leverage data sources in different languages, being able to answer, understand, and translate in many languages. This ability– together with multilingual embeddings– allows you to have multilingual support both in the data retrieval and the answer generation stages, removing language barriers entirely.

If you are interested on reproducing the examples of this article, you can find the Python Notebook with the requests here

Background

Vector search is evolving, and as our needs for vector search evolve so does the need for a consistent and forward thinking vector search API.

When Elastic first launched semantic search, we leveraged existing rank_features fields using the text_expansion query. We then reintroduced the sparse_vector field type for semantic search use cases.

As we think about what sparse vector search is going forward, we’ve introduced a new sparse vector query. As of Elasticsearch 8.15.0, both the text_expansion query and weighted_tokens query have been deprecated in favor of the new sparse vector query.

The sparse vector query supports two modes of querying: using an inference ID and using precomputed query vectors. Both modes of querying require data to be indexed in a sparse_vector mapped field.

These token-weight pairs are then used in a query against a sparse vector. At query time, query vectors are calculated using the same inference model that was used to create the tokens.

Let’s look at an example: let’s say we’ve indexed a document detailing when Orion is most visible in the night sky:

Now, assume we’re looking for constellations that are visible in the northern hemisphere, and we run this query through the same learned sparse encoder model. The output might look similar to this:

At query time, these vectors are ORed together, and scoring is effectively a dot product calculation between the stored dimensions and the query dimensions, which would score this example at 10.84:

Sparse vector queries with inference

Sparse vector queries using inference work in a very similar way to the previous text expansion query, instead of sending in a trained model, we create an inference endpoint associated with the model we want to use.

Here’s an example of how to create an inference endpoint for ELSER:

You should use an inference endpoint to index your sparse vector data, and use the same endpoint as input to your sparse_vector query. For example:

Sparse vector queries with precomputed query vectors

You may have precomputed vectors that don’t require inference at query time. These can be sent into the sparse_vector query instead of using inference. Here is an example:

Query optimization with token pruning

Like text expansion search, the sparse vector query is subject to performance penalties from huge boolean queries. Therefore the same token pruning strategies available for text expansion strategies are available in the sparse vector query. You can see the impact of token pruning in our nightly MS Marco Passage Ranking benchmarks.

In order to enable pruning with the default pruning configuration (which has been tuned for ELSER V2), simply add prune: true to your request:

Alternately, you can adjust the pruning configuration by sending it directly in with the request:

Because token pruning will incur a recall penalty, we recommend adding the pruned tokens back in a rescore:

What's next?

While the text_expansion query is GA’d and will be supported throughout Elasticsearch 8.x, we recommend updating to the sparse_vector query as soon as possible in order to ensure you’re using the most up to date features as we continually improve the vector search experience in Elasticsearch.

If you are using the weighted_tokens query, this was never GA’d and will be replaced by the sparse_vector query very soon.

The sparse_vector query will be available starting with 8.15.0 and is already available in Serverless - try it out today!

Vector search is revolutionizing the way we conduct similarity searches, particularly in fields like AI and machine learning. With the increasing adoption of vector embedding models, the ability to efficiently search through millions of high-dimension vectors becomes critical.

When it comes to powering vector databases, Elastic and OpenSearch have taken notably different approaches. Elastic has invested heavily in optimizing Apache Lucene together with Elasticsearch to elevate them as the top-tier choice for vector search applications. In contrast, OpenSearch has broadened its focus, integrating other vector search implementations and exploring beyond Lucene's scope. Our focus on Lucene is strategic, enabling us to provide highly integrated support in our version of Elasticsearch, resulting in an enhanced feature set where each component complements and amplifies the capabilities of the other.

This blog presents a detailed comparison between Elasticsearch 8.14 and OpenSearch 2.14 accounting for different configurations and vector engines. In this performance analysis, Elasticsearch proved to be the superior platform for vector search operations, and upcoming features will widen the differences even more significantly. When pitted against OpenSearch, it excelled in every benchmark track — offering 2x to 12x faster performance on average. This was across scenarios using varying vector amounts and dimensions including so_vector (2M vectors, 768D), openai_vector (2.5M vectors, 1536D), and dense_vector (10M vectors, 96D), all available in this repository alongside the Terraform scripts for provisioning all the required infrastructure on Google Cloud and Kubernetes manifests for running the tests.

The results detailed in this blog complement the results from a previously published and third-party validated study that shows Elasticsearch is 40%–140% faster than OpenSearch for the most common search analytics operations: Text Querying, Sort, Range, Date Histogram and Terms filtering. Now we can add another differentiator: Vector Search.

Up to 12x faster out-of-the-box

Our focused benchmarks across the four vector data sets involved both Approximate KNN and Exact KNN searches, considering different sizes, dimensions and configurations, totaling 40.189.820 uncached search requests. The results: Elasticsearch is up to 12x faster than OpenSearch for vector search and therefore requires fewer computational resources.

Figure 1: Grouped tasks for ANN and Exact KNN across different combinations in Elasticsearch and OpenSearch.

The groups like knn-10-100 means KNN search with and . In HNSW vector search, determines the number of nearest neighbors to retrieve for a query vector. It specifies how many similar vectors to find as a result. sets the number of candidate vectors to retrieve at each segment. More candidates can enhance accuracy but require greater computational resources.

We also tested with different quantization techniques and leveraged engine-specific optimizations, the detailed results for each track, task and vector engine are available below.

Exact KNN and Approximate KNN

When dealing with varying data sets and use cases, the right approach for vector search will differ. In this blog all tasks stated as knn-* like knn-10-100 use Approximate KNN and script-score-* refer to Exact KNN, but what is the difference between them, and why are they important?

In essence, if you're handling more substantial data sets, the preferred method is the Approximate K-Nearest Neighbor (ANN) due to its superior scalability. For more modest data sets that may require a filtration process, Exact KNN method is ideal.

Exact KNN uses a brute-force method, calculating the distance between one vector and every other vector in the data set. It then ranks these distances to find the nearest neighbors. While this method ensures an exact match, it suffers from scalability challenges for large, high-dimensional data sets. However, there are many cases in which Exact KNN is needed:

• Rescoring: In scenarios involving lexical or semantic searches followed by vector-based rescoring, Exact KNN is essential. For example, in a product search engine, initial search results can be filtered based on textual queries (e.g., keywords, categories), and then vectors associated with the filtered items are used for a more accurate similarity assessment.

• Personalization: When dealing with a large number of users, each represented by a relatively small number (like 1 million) of distinct vectors, sorting the index by user-specific metadata (e.g., user_id) and brute-force scoring with vectors becomes efficient. This approach allows for personalized recommendations or content delivery based on precise vector comparisons tailored to individual user preferences.

Exact KNN therefore ensures that the final ranking and recommendations based on vector similarity are precise and tailored to user preferences.

Approximate KNN (or ANN) on the other hand employs methods to make data searching faster and more efficient than Exact KNN, especially in large, high-dimensional data sets. Instead of a brute-force approach, which measures the exact nearest distance between a query and all points leading to computation and scaling challenges, ANN uses certain techniques to efficiently restructure the indexes and dimensions of searchable vectors in the data set. While this may cause a slight inaccuracy, it significantly boosts the speed of the search process, making it an effective alternative for dealing with large data sets.

In this blog all tasks stated as knn-* like knn-10-100 use Approximate KNN and script-score-* refer to Exact KNN.

Testing methodology

While Elasticsearch and OpenSearch are similar in terms of API for BM25 search operations, since the latter is a fork of the former, it is not the case for Vector Search, which was introduced after the fork. OpenSearch took a different approach than Elasticsearch when it comes to algorithms, by introducing two other engines — nmslib and faiss — apart from lucene, each with their specific configurations and limitations (e.g., nmslib in OpenSearch does not allow for filters, an essential feature for many use cases).

All three engines use the Hierarchical Navigable Small World (HNSW) algorithm, which is efficient for approximate nearest neighbor search, and especially powerful when dealing with high-dimensional data. It's important to note that faiss also supports a second algorithm, ivf, but since it requires pre-training on the data set, we are going to focus solely on HNSW. The core idea of HNSW is to organize the data into multiple layers of connected graphs, with each layer representing a different granularity of the data set. The search begins at the top layer with the coarsest view and progresses down to finer and finer layers until reaching the base level.

Both search engines were tested under identical conditions in a controlled environment to ensure fair testing grounds. The method applied is similar to this previously published performance comparison, with dedicated node pools for Elasticsearch, OpenSearch, and Rally. The terraform script is available (alongside all sources) to provision a Kubernetes cluster with:

• 1 Node pool for Elasticsearch with 3 e2-standard-32 machines (128GB RAM and 32 CPUs)

• 1 Node pool for OpenSearch with 3 e2-standard-32 machines (128GB RAM and 32 CPUs)

• 1 Node pool for Rally with 2 t2a-standard-16 machines (64GB RAM and 16 CPUs)

Each "track" (or test) ran for 10 times for each configuration, which included different engines, different configurations and different vector types. The tracks have tasks that repeat between 1000 and 10000 times, depending on the track. If one of the tasks in a track failed for instance due to a network timeout, then all tasks were discarded, so all results represent tracks that started and finished without problems. All test results are statistically validated, ensuring that improvements aren’t coincidental.

Detailed findings

Why compare using the 99th percentile and not the average latency? Consider a hypothetical example of average house prices in a certain neighborhood. The average price may indicate an expensive area, but on closer inspection, it may turn out that most homes are valued much lower, with only a few luxury properties inflating the average figure. This illustrates how the average price can fail to accurately represent the full spectrum of house values in the area. This is akin to examining response times, where the average can conceal critical issues.

Tasks

• Approximate KNN with k:10 n:50

• Approximate KNN with k:10 n:100

• Approximate KNN with k:100 n:1000

• Approximate KNN with k:10 n:50 and keyword filters

• Approximate KNN with k:10 n:100 and keyword filters

• Approximate KNN with k:100 n:1000 and keyword filters

• Approximate KNN with k:10 n:100 in conjunction with indexing

• Exact KNN (script score)

Vector engines

• lucene in Elasticsearch and OpenSearch, both on version 9.10

• faiss in OpenSearch

• nmslib in OpenSearch

Vector types

• hnsw in Elasticsearch and OpenSearch

• int8_hnsw in Elasticsearch (HNSW with automatic 8 bit quantization: link)

• sq_fp16 hnsw in OpenSearch (HNSW with automatic 16 bit quantization: link)

Out-of-the-box and Concurrent Segment Search

As you probably know, Lucene is a highly performant text search engine library written in Java that serves as the backbone for many search platforms like Elasticsearch, OpenSearch, and Solr. At its core, Lucene organizes data into segments, which are essentially self-contained indices that allow Lucene to execute searches more efficiently. So when you issue a search to any Lucene-based search engine, your search will end up being executed in those segments, either sequentially or in parallel.

OpenSearch introduced concurrent segment search as an optional flag, and does not use it by default, you must enable it using a special index setting index.search.concurrent_segment_search.enabled as detailed here, with some limitations.

Elasticsearch on the other hand searches on segments concurrently out-of-the-box, therefore the comparisons we make in this blog will take into consideration, on top of the different vector engines and vector types, also the different configurations:

• Elasticsearch ootb: Elasticsearch out-of-the-box, with concurrent segment search;

• OpenSearch ootb: without concurrent segment search enabled;

• OpenSearch css: with concurrent segment search enabled

Now, let’s dive into some detailed results for each vector data set tested:

2.5 million vectors, 1536 dimensions (openai_vector)

Starting with the simplest track, but also the largest in terms of dimensions, openai_vector - which uses the NQ data set enriched with embeddings generated using OpenAI's text-embedding-ada-002 model. It is the simplest since it tests only Approximate KNN and has only 5 tasks. It tests in standalone (without indexing) as well as alongside indexing, and using a single client and 8 simultaneous clients.

Tasks

• standalone-search-knn-10-100-multiple-clients: searching on 2.5 million vectors with 8 clients simultaneously, k: 10 and n:100

• standalone-search-knn-100-1000-multiple-clients: searching on 2.5 million vectors with 8 clients simultaneously, k: 100 and n:1000

• standalone-search-knn-10-100-single-client: searching on 2.5 million vectors with a single client, k: 10 and n:100

• standalone-search-knn-100-1000-single-client: searching on 2.5 million vectors with a single client, k: 100 and n:1000

• parallel-documents-indexing-search-knn-10-100: searching on 2.5 million vectors while also indexing additional 100000 documents, k:10 and n:100

The averaged p99 performance is outlined below:

Here we observed that Elasticsearch is between 3x-8x faster than OpenSearch when performing vector search alongside indexing (i.e. read+write) with :10 and :100 and 2x-3x faster without indexing for the same k and n. For :100 and :1000 (standalone-search-knn-100-1000-single-client and standalone-search-knn-100-1000-multiple-clients Elasticsearch is 2x to 7x faster than OpenSearch, on average.

The detailed results show the exact cases and vector engines compared:

Recall

knn-recall-10-100

knn-recall-100-1000

Elasticsearch-8.14.0@lucene-hnsw

0.969485

0.995138

Elasticsearch-8.14.0@lucene-int8_hnsw

0.781445

0.784817

OpenSearch-2.14.0@lucene-hnsw

0.96519

0.995422

OpenSearch-2.14.0@faiss

0.984154

0.98049

OpenSearch-2.14.0@faiss-sq_fp16

0.980012

0.97721

OpenSearch-2.14.0@nmslib

0.982532

0.99832

10 million vectors, 96 dimensions (dense_vector)

In dense_vector with 10M vectors and 96 dimensions. It is based on the Yandex DEEP1B image data set. The data set is created from the first 10 million vectors of the "sample data" file called learn.350M.fbin. The search operations use vectors from the "query data" file query.public.10K.fbin.

Both Elasticsearch and OpenSearch perform very well on this data set, especially after a force merge which is usually done on read-only indices and it’s similar to defragmenting the index to have a single "table" to search on.

Tasks

Each task warms up for 100 requests and then 1000 requests are measured

• knn-search-10-100: searching on 10 million vectors, k: 10 and n:100

• knn-search-100-1000: searching on 10 million vectors, k: 100 and n:1000

• knn-search-10-100-force-merge: searching on 10 million vectors after a force merge, k: 10 and n:100

• knn-search-100-1000-force-merge: searching on 10 million vectors after a force merge, k: 100 and n:1000

• knn-search-100-1000-concurrent-with-indexing: searching on 10 million vectors while also updating 5% of the data set, k: 100 and n:1000

• script-score-query: Exact KNN search of 2000 specific vectors.

Both Elasticsearch and OpenSearch performed well for Approximate KNN. When the index is merged (i.e. has just a single segment) in knn-search-100-1000-force-merge and knn-search-10-100-force-merge, OpenSearch performs better than the others when using nmslib and faiss, even though they are all around 15ms and all very close.

However, when the index has multiple segments (a typical situation where an index receives updates to its documents) in knn-search-10-100 and knn-search-100-1000, Elasticsearch keeps the latency in about ~7ms and ~16ms, while all other OpenSearch engines are slower.

Also when the index is being searched and written to at the same time (knn-search-100-1000-concurrent-with-indexing), Elasticsearch maintains the latency below 15ms (at 13.8ms), being almost 4x faster than OpenSearch out-of-the-box (49.3ms) and still faster when concurrent segment search is enabled (17.9ms), but too close to be significative.

As for Exact KNN, the difference is much larger: Elasticsearch is 6x faster than OpenSearch (~260ms vs ~1600ms).

Recall

knn-recall-10-100

knn-recall-100-1000

Elasticsearch-8.14.0@lucene-hnsw

0.969843

0.996577

Elasticsearch-8.14.0@lucene-int8_hnsw

0.775458

0.840254

OpenSearch-2.14.0@lucene-hnsw

0.971333

0.996747

OpenSearch-2.14.0@faiss

0.9704

0.914755

OpenSearch-2.14.0@faiss-sq_fp16

0.968025

0.913862

OpenSearch-2.14.0@nmslib

0.9674

0.910303

2 million vectors, 768 dimensions (so_vector)

This track, so_vector, is derived from a dump of StackOverflow posts downloaded on April, 21st 2022. It only contains question documents — all documents representing answers have been removed. Each question title was encoded into a vector using the sentence transformer model multi-qa-mpnet-base-cos-v1. This data set contains the first 2 million questions.

Unlike the previous track, each document here contains other fields besides vectors to support testing features like Approximate KNN with filtering and hybrid search. nmslib for OpenSearch is notably absent in this test since it does not support filters.

Tasks

Each task warms up for 100 requests and then 100 requests are measured. Note the tasks were grouped for sake of simplicity, since the test contains 16 search types * 2 different k values * 3 different n values.

• knn-10-50: searching on 2 million vectors without filters, k:10 and n:50

• knn-10-50-filtered: searching on 2 million vectors with filters, k:10 and n:50

• knn-10-50-after-force-merge: searching on 2 million vectors with filters and after a force merge, k:10 and n:50

• knn-10-100: searching on 2 million vectors without filters, k:10 and n:100

• knn-10-100-filtered: searching on 2 million vectors with filters, k:10 and n:100

• knn-10-100-after-force-merge: searching on 2 million vectors with filters and after a force merge, k:10 and n:100

• knn-100-1000: searching on 2 million vectors without filters, k:100 and n:1000

• knn-100-1000-filtered: searching on 2 million vectors with filters, k:100 and n:1000

• knn-100-1000-after-force-merge: searching on 2 million vectors with filters and after a force merge, k:100 and n:1000

• exact-knn: Exact KNN search with and without filters.

Elasticsearch is consistently faster than OpenSearch out-of-the-box on this test, only in two cases OpenSearch is faster, and not by much (knn-10-100 and knn-100-1000). Tasks involving knn-10-50, knn-10-100 and knn-100-1000 in combination with filters show a difference of up to 7x (112ms vs 803ms).

The performance of both solutions seems to even out after a "force merge", understandably, as evidenced by knn-10-50-after-force-merge, knn-10-100-after-force-merge and knn-100-1000-after-force-merge. On those tasks faiss is faster.

The performance for Exact KNN once again is very different, Elasticsearch being 13 times faster than OpenSearch this time (~385ms vs ~5262ms).

Recall

knn-recall-10-100

knn-recall-100-1000

knn-recall-10-50

Elasticsearch-8.14.0@lucene-hnsw

1

1

1

Elasticsearch-8.14.0@lucene-int8_hnsw

1

0.986667

1

OpenSearch-2.14.0@lucene-hnsw

1

1

1

OpenSearch-2.14.0@faiss

1

1

1

OpenSearch-2.14.0@faiss-sq_fp16

1

1

1

OpenSearch-2.14.0@nmslib

0.9674

0.910303

0.976394

Elasticsearch and Lucene as clear victors

At Elastic, we are relentlessly innovating Apache Lucene and Elasticsearch to ensure we are able to provide the premier vector database for search and retrieval use cases, including RAG (Retrieval Augmented Generation). Our recent advancements have dramatically increased performance, making vector search faster and more space efficient than before, building upon the gains from Lucene 9.10. This blog presented a study that shows when comparing up-to-date versions Elasticsearch is up to 12 times faster than OpenSearch.

It's worth noting both products use the same version of Lucene (Elasticsearch 8.14 Release Notes and OpenSearch 2.14 Release Notes).

The pace of innovation at Elastic will deliver even more not only for our on-premises and Elastic Cloud customers but those using our stateless platform. Features like support for scalar quantization to int4 will be offered with rigorous testing to ensure customers can utilize these techniques without a significant drop in recall, similar to our testing for int8.

Vector search efficiency is becoming a non-negotiable feature in modern search engines due to the proliferation of AI and machine learning applications. For organizations looking for a powerful search engine capable of keeping up with the demands of high-volume, high-complexity vector data, Elasticsearch is the definitive answer.

Whether expanding an established platform or initiating new projects, integrating Elasticsearch for vector search needs is a strategic move that will yield tangible, long-term benefits. With its proven performance advantage, Elasticsearch is poised to underpin the next wave of innovations in search.

Do you want to start using semantic search for your data, but focus on your model and results instead of on the technical details? We’ve introduced the semantic_text field type that will take care of the details and infrastructure that you need.

Semantic search is a sophisticated technique designed to enhance the relevance of search results by utilizing machine learning models. Unlike traditional keyword-based search, semantic search focuses on understanding the meaning of words and the context in which they are used. This is achieved through the application of machine learning models that provide a deeper semantic understanding of the text.

These models generate vector embeddings, which are numeric representations capturing the text meaning. These embeddings are stored alongside your document data, enabling vector search techniques that take into account the word meaning and context instead of pure lexical matches.

How to perform semantic search

To perform semantic search, you need to go through the following steps:

• Choose an inference model to create embeddings, both for indexing documents and performing queries.

• Create your index mapping to store the inference results, so they can be efficiently searched afterwards.

• Setting up indexing so inference results are calculated for new documents added to your index.

• Automatically handle long text documents, so search can be accurate and cover the entire document.

• Querying your data to retrieve results.

Configuring semantic search from the ground up can be complex. It requires setting up mappings, ingestion pipelines, and queries tailored to your chosen inference model. Each step offers opportunities for fine-tuning and optimization, but also demands careful configuration to ensure all components work together seamlessly.

While this offers a great degree of control, it makes using semantic search a detailed and deliberate process, requiring you to configure separate pieces that are all related to each other and to the inference model.

semantic_text simplifies this process by focusing on what matters: the inference model. Once you have selected the inference model, semantic_text will make it easy to start using semantic search by providing sensible defaults, so you can focus on your search and not on how to index, generate, or query your embeddings.

Let's take a look at each of these steps, and how semantic_text simplifies this setup.

Choosing an inference model

The inference model will generate embeddings for your documents and queries. Different models have different tradeoffs in terms of:

• Accuracy and relevance of the results

• Scalability and performance

• Language and multilingual support

• Cost

Elasticsearch supports both internal and external inference services:

• Internal services are deployed in the Elasticsearch cluster. You can use already included models like ELSER and E5, or import external models into the cluster using eland.

• External services are deployed by model providers. Elasticsearch supports the following:

  • Cohere

  • Hugging Face

  • Mistral

  • OpenAI

  • Azure AI Studio

  • Azure OpenAI

  • Google AI Studio

Once you have chosen the inference mode, create an inference endpoint for it. The inference endpoint identifier will be the only configuration detail that you will need to set up semantic_text.

Creating your index mapping

Elasticsearch will need to index the embeddings generated by the model so they can be efficiently queried later.

Before semantic_text, you needed to understand about the two main field types used for storing embeddings information:

• sparse_vector: It indexes sparse vector embeddings, like the ones generated by ELSER. Each embedding consists of pairs of tokens and weights. There is a small number of tokens generated per embedding.

• dense_vector: It indexes vectors of numbers, which contains the embedding information. A model produces vectors of a fixed size, called the vector dimension.

The field type to use is conditioned by the model you have chosen. If using dense vectors, you will need to configure the field to include the dimension count, the similarity function used to calculate vectors proximity, and storage customizations like quantization or the specific data type used for each element.

Now, if you're using semantic_text, you define a semantic_text field mapping by just specifying the inference endpoint identifier for your model:

That's it. No need for you to define other mapping options, or to understand which field type you need to use.

Setting up indexing

Once your index is ready to store the embeddings, it's time to generate them.

Before semantic_text, to generate embeddings automatically on document ingestion you needed to set up an ingestion pipeline.

Ingestion pipelines are used to automatically enrich or transform documents when ingested into an index, or when explicitly specified as part of the ingestion process.

You need to use the inference processor to generate embeddings for your fields. The processor needs to be configured using:

• The text fields from which to generate the embeddings

• The output fields where the generated embeddings will be added

• Specific inference configuration for text embeddings or sparse embeddings, depending on the model type

With semantic_text, you simply add documents to your index. semantic_text fields will automatically calculate the embeddings using the specified inference endpoint.

This means there's no need to create an inference pipeline to generate the embeddings. Using bulk, index, or update APIs will do that for you automatically:

Inference requests in semantic_text fields are also batched. If you have 10 documents in a bulk API request, and each document contains 2 semantic_text fields, then that request will perform a single inference request with 20 texts to your inference service in one go, instead of making 10 separate inference requests of 2 texts each.

Automatically handling long text passages

Part of the challenge of selecting a model is the number of tokens that the model can generate embeddings for. Models have a limited number of tokens they can process. This is referred to as the model’s context window.

If the text you need to work with is longer than the model’s context window, you may truncate the text and use just part of it to generate embeddings. This is not ideal as you'll lose information; the resulting embeddings will not capture the full context of the input text.

Even if you have a long context window, having a long text means a lot of content will be reduced to a single embedding, making it an inaccurate representation.

Also, returning a long text will be difficult for the users to understand, as they will have to scan the text to check it's what they are looking for. Using smaller snippets would be preferable instead.

Another option is to use chunking to divide long texts into smaller fragments. These smaller chunks are added to each document to provide a better representation of the complete text. You can then use a nested query to search over all the individual fragments and retrieve the documents that contain the best-scoring chunks.

Before semantic_text, chunking was not done out of the box - the inference processor did not support chunking. If you needed to use chunking, you needed to do it before ingesting your documents or use the script processor to perform the chunking in Elasticsearch.

Using semantic_text means that chunking will be done on your behalf when indexing. Long documents will be split into 250-word sections with a 100-word overlap so that each section shares 100 words with the previous section. This overlap ensures continuity and prevents vital contextual information in the input text from being lost by a hard break.

If the model and inference service support batching the chunked inputs are automatically batched together into as few requests as possible, each optimally sized for the Inference Service. The resulting chunks will be stored in a nested object structure so you can check the text contained in each chunk.

Querying your data

Now that the documents and their embeddings are indexed in Elasticsearch, it's time to do some queries!

Before semantic_text, you needed to use a different query depending on the type of embeddings the model generates (dense or sparse). A sparse vector query is needed to query sparse_vector field types, and either a knn search or a knn query can be used to search dense_vector field types.

The query process can be further customized for performance and relevance. For example, sparse vector queries can define token pruning to avoid considering irrelevant tokens. Knn queries can specify the number of candidates to consider and the top k results to be returned from each shard.

You don't need to deal with those details when using semantic_text. You use a single query type to search your documents:

Just include the field and the query text. There’s no need to decide between sparse vector and knn queries, semantic text does this for you.

Compare this with using a specific knn search with all its configuration parameters:

Under the hood: How semantic_text works

To understand how semantic_text works, you can create a semantic_text index and check what happens when you ingest a document. When the first document is ingested, the inference endpoint calculates the embeddings. When indexed, you will notice changes in the index mapping:

Now there is additional information about the model settings. Text embedding models will also include information like the number of dimensions or the similarity function for the model.

You can check the document already includes the embedding results:

The field does not just contain the input text, but also a structure storing the original text, the model settings, and information for each chunk the input text has been divided into.

This structure consists of an object with two elements:

• text: Contains the original input text

• inference: Inference information added by the inference endpoint, that consists of:

  • inference_id of the inference endpoint

  • model_settings that contain model properties

  • chunks: Nested object that contains an element for each chunk that has been created from the input text. Each chunk contains:

    • The text for the chunk

    • The calculated embeddings for the chunk text

Customizing semantic_text

semantic_text simplifies semantic search by making default decisions about indexing and querying your data:

• uses sparse_vector or dense_vector field types depending on the inference model type

• Automatically defines the number of dimensions and similarity according to the inference results

• Uses int8_hnsw index type for dense vector field types to leverage scalar quantization.

• Uses query defaults. No token pruning is applied for sparse_vector queries, nor custom k and num_candidates are set for knn queries.

Those are sensible defaults and allow you to quickly and easily start working with semantic search. Over time, you may want to customize your queries and data types to optimize search relevance, index and query performance, and index storage.

Query customization

There are no customization options - yet - for semantic queries. If you want to customize queries against semantic_text fields, you can perform advanced semantic_text search using explicit knn and sparse vector queries.

We're planning to add retrievers support for semantic_text, and adding configuration options to the semantic_text field so they won't be needed at query time. Stay tuned!

Data type customization

If you need deeper customization for the data indexing, you can use the sparse_vector or dense_vector field types. These field types give you full control over how embeddings are generated, indexed, and queried.

You need to create an ingest pipeline with an inference processor to generate the embeddings. This tutorial walks you through the process.

What's next with semantic_text?

We're just getting started with semantic_text! There are quite a few enhancements that we will keep working on, including:

• Better inference error handling

• Customize the chunking strategy

• Hiding embeddings in _source by default, to avoid cluttering the search responses

• Inner hits support, to retrieve the relevant chunks of information for a query

• Filtering and retrievers support

• Kibana support

Try it out!

semantic_textis available on Elasticsearch Serverless now! It will be available soon on Elasticsearch 8.15 version for Elastic Cloud and on Elasticsearch downloads.

If you already have an Elasticsearch serverless cluster, you can see a complete example for testing semantic search using semantic_text in this tutorial, or try it with this notebook.

We'd love to hear about your experience with semantic_text! Let us know what you think in the forums, or open an issue in the GitHub repository. Let's make semantic search easier together!

This blog builds on that notebook and explores fetch surrounding chunking, an emerging pattern in RAG that uses intelligent chunking and Elasticsearch vector database to optimize LLM responses. The approach balances data input to enhance the accuracy and relevance of LLM-generated answers through semantic hybrid search.

The motivation: A refined approach to RAG data chunking

The primary motivation behind building this notebook was to demonstrate a refined approach to RAG by addressing the challenge of data chunking. Traditional methods often fall short in dynamically adjusting the data size fed to LLMs, either overwhelming the model with too much context or starving it with too little. This notebook aims to strike the right balance, providing just enough information for the LLM to generate precise and contextually relevant responses. However, it must be noted that there is no one-size-fits-all solution.

This method works especially well with books and similar texts where content flows within longer sections or chapters. However, it may require adaptation for texts structured into shorter, distinct sections, such as research papers or articles, where each segment might cover a different topic. In such cases, additional strategies may be necessary to effectively chunk and retrieve related content.

The methodology: Intelligent RAG data chunking

Fetch surrounding chunks

The core idea is to partition the source text into manageable chunks, ensuring each chunk contains just the right amount of information. For this demonstration, I used text from "Harry Potter and the Sorcerer's Stone." The text was partitioned into chapters, and each chapter was further divided into smaller chunks. These chunks, along with their dense and sparse (ELSER) vector representations, were indexed in the Elasticsearch vector database.

Assigning numbers to chunks

Each chunk within a chapter was assigned a sequential integer, allowing us to identify its position. When a matching chunk is found, the chapter number and chunk number are used to retrieve surrounding chunks, providing additional context for the LLM.

Vector database in Elasticsearch

These chunks and their vector representations were ingested into an Elasticsearch Cloud instance. Elasticsearch's robust vector search capabilities make it ideal for hosting these chunks, allowing for efficient retrieval of the most relevant chunks based on the semantic content or text match of a user's query.

AI search

To retrieve the relevant chunks, I employed a hybrid search strategy using dense vector comparisons, sparse vector comparisons, and text search in parallel. This multi-faceted approach ensures that the search results are both semantically rich and contextually accurate. A query is issued to find the matched chunk, which returns the chunk number and chapter. Surrounding chunks for that chapter are then fetched based on the matched chunk.

The RAG pattern

When a query is made, the search flow performs the following steps:

1. Query analysis: The user's query is translated into dense and sparse vectors to retrieve the most relevant chunks from the Elasticsearch index.

2. Chunk retrieval: Using the AI search strategy, the system retrieves the top relevant chunks.

3. Contextual expansion: Adjacent chunks (n-1 and n+1) are also retrieved to provide a more comprehensive context. If the chunk is the last in the chapter, it fetches n-1 and n-2; if it's the first, it fetches n+1 and n+2.

4. LLM response: These intelligently selected chunks are then fed into the LLM, ensuring it receives the optimal amount of information to generate a precise and contextually relevant response.

Why intelligent RAG data chunking matters

This approach addresses a critical aspect of RAG by optimizing the input data fed to LLMs. By leveraging intelligent chunking and hybrid semantic search, this method enhances the accuracy and relevance of the responses generated by LLMs. It showcases a pattern that can be widely applied in various applications within the RAG space, from customer support to content generation and beyond.

Conclusion

This notebook underscores the importance of intelligent data chunking in the RAG framework and demonstrates how Elasticsearch vector database can be leveraged to achieve optimal results. By ensuring the LLM receives just the right amount of information, this methodology paves the way for more accurate and contextually rich responses, enhancing the overall effectiveness of RAG systems.

Vector search has emerged as a game-changer in the current generative AI/ML world. It allows us to find similar items based on their semantic meaning rather just exact keyword matches.

Elasticsearch's k-Nearest Neighbors (kNN) algorithm is a foundational ML technique for classification and regression tasks. It found a significant place within Elasticsearch's ecosystem with the introduction of vector search capabilities. Introduced in Elasticsearch 8.5, kNN based vector search allows users to perform high-speed similarity searches on dense vector fields.

Users can find documents in the index "closest" to a given vector by leveraging the kNN algorithm using an underlying specified distance metric such as Euclidean or Cosine similarity. This feature marked a pivotal advancement as it is particularly useful in applications requiring semantic search, recommendations and other use cases such as anomaly detection.

The introduction of dense vector fields and k-nearest neighbor (kNN) search functionality in Elasticsearch has opened new horizons for implementing sophisticated search capabilities that go beyond traditional text search.

This article delves into strategies for selecting the optimal values for k and num_candidates parameters, illustrated with practical examples using Kibana.

kNN search query

Elasticsearch provides a kNN search option for nearest-neighbors - something like the following:

As the snippet shows, the knn query fetches the relevant results for the query in question (having a movie title as "Good Ugly") using vector search. The search is conducted in a multi-dimensional space, producing the closest vectors to the given query vector.

From the above query, notice two attributes: num_candidates which is the initial pool of candidates to consider and k, the number of nearest neighbors.

kNN critical parameters - k and num_candidates

To leverage the kNN feature effectively, one requires a nuanced understanding of the two critical parameters: k - the number of global nearest neighbors to retrieve, and num_candidates - the number of candidate neighbors considered for each shard during the search.

Choosing the optimal values for the k and num_candidates involves balancing precision, recall, and performance. These parameters play a crucial role to efficiently handle high-dimensional vector spaces commonly found in machine learning applications.

The optimal value for k largely depends on the specific use case. For example, if you're building a recommendation system, a smaller k (e.g., 10-20) might be sufficient to provide relevant recommendations. In contrast, for a use case where you'd want clustering or outlier detection capabilities, you might need a larger k.

Note that the higher k value can significantly increase both computation and memory usage, especially with large datasets. It's important to test different values of k to find a balance between result relevance and system resource usage.

K: Unveiling the closest neighbors

We have an option of choosing the k value as per our requirements. Sometimes, setting up a lower k value receives more or less exactly what you want with the exception that a few results might not make it to the final output. However, setting up a higher k value might broaden your search results in numbers, with a caveat that you may receive diversified results at times.

Imagine you're searching for a new book in the vast library of recommendations. k, also known as the number of nearest neighbors, determines how many books you'll be presented with. Think of it as the inner circle of your search results. Let's see how setting the lower and higher k values affects the number of books that the query returns.

Setting lower K

The lower K setting prioritizes extreme precision - meaning we will receive a handful of books that are the most similar to our query vector. This ensures a high degree of relevance to our specific interests. This might be ideal if you're searching for a book with a very specific theme or writing style.

Setting higher K

With a larger K value, we will be fetching a broader exploration result set. Note that the results might not be as tightly focused on your exact query. However, you'll encounter a wider range of potentially interesting books. This approach can be valuable for diversifying your reading list and discovering unexpected gems, perhaps.

The num_candidates attribute: Behind the curtain

While k determines the final number of books you see, num_candidates plays a crucial role under the hood. It essentially defines the search space per shard – the initial pool of books in a shard from which the most relevant K neighbors are identified. When we issue the query, we are expected to hint Elasticsearch to run the query amongst top "x" number of candidates on each shard.

For example, say our books index contains 5000 books evenly distributed amongst five primary shards (i.e., ~1000 books per shard). When we are performing a search, obviously choosing all 1000 documents for each shard is neither a viable nor a correct option. Instead, we will be pick up to say 25 documents (which is our num_candidates) from the 1000 documents. That amounts to 125 documents as our total search space (5 shards times 25 documents each).

We will let the kNN query know to choose the 25 documents from each shard and this number is the num_candidates parameter. When the kNN search is executed, the "coordinator" node sends the request query to all of the involved shards. The num_candidates documents from each shard will constitute the search space and the top k documents will be fetched from that space. Say, if k is 3, the top 3 documents out of the 25 candidate documents will be selected in each shard and returned to the coordinator node. That is, the coordinator node will receive 15 documents in total from all the involved nodes. These top 15 documents are then ranked to fetch the global top 3 (k==3) documents.

The process is depicted in the following figure:

Here's what num_candidates means for your search:

Setting the lower num_candidates

This approach might restrict the search space, potentially missing some relevant books that fall outside the initial exploration set. Think of it as surveying a smaller portion of the library's shelves.

Setting the higher num_candidates

A higher num_candidates value increases the likelihood of finding the true nearest neighbors within our chosen K. It expands the search space - that is - more number of candidates are considered - and hence leads to a slight increase in search time. So, a higher value generally increases accuracy (as the chance of missing relevant vectors decreases) but at the cost of performance.

Balancing precision & performance for kNN parameters

The optimal values for k and num_candidates depend on a few factors and specific needs. If we prioritize extreme precision with a smaller set of highly relevant results, a lower k with a moderate num_candidates might be ideal. Conversely, if exploration and discovering unexpected books are your goals, a higher K with a larger num_candidates could be more suitable.

While there is no hard-and-fast rule to define the "lower" or "higher" number for the num_candidates, you need to decide this number based on your dataset, computing power and the expected precision.

Experimentation to optimize kNN parameters

By experimenting with different K and num_candidates combinations and monitoring search results and performance, you can fine-tune your searches to achieve the perfect balance between precision, exploration, and speed. Remember, there's no one-size-fits-all solution – the best approach depends on your unique goals and data characteristics.

Practical example: Using kNN for movie recommendations

Let's consider an example of movies to create a manual "simple" framework for understanding the effect of k and num_candidates attributes while searching for movies.

Manual framework

Let's understand how we can develop a home grown framework for tweaking the k and num_of_candidates attributes for a kNN search.

The mechanics of the framework is as follows:

• Create a movies index with a couple of dense_vector fields in the mapping to hold our vectorised data.

• Create an embedding pipeline so each and every movie's title and synopsis fields will be embedded with a multilingual-e5-small model to store vectors.

• Perform the indexing operation,which goes through the above embedding pipeline. The respective fields will be vectorised

• Create a search query using kNN feature

• Tweak the k and num_candidates options as you'd want

Let's dig in.

Creating an inference pipeline

We will need to index data via Kibana - far from ideal - but it will do for this manual framework understanding. However, every movie that gets indexed must have the title and synopsis field vectorised to enable semantic search on our data. We can do this by elegantly creating a inference pipeline processor and attaching it to our batch indexing operation.

Let's create an inference pipeline:

The inference pipeline movie_embedding_pipeline, as shown above, creates vector fields text embedding for title and synopsis fields. It uses the inbuilt multilingual-e5-small model to create the text embeddings.

Creating index mappings

We will need to create a mapping with couple of properties as dense_vector fields. The following code snippet does the job:

Once the above command gets executed, we have a new movies index with the appropriate dense vector fields, including title_vector.predicted_value and synopsis_vector.predicted_value fields that hold respective vectors.

Next step is to ingest the data.

Indexing movies

We can use _bulk operation to index a set of movies - I'm reusing a dataset that I had created for my Elasticsearch in Action 2nd edition book - which is available here:

For completeness, a snippet of the ingestion using the _bulk operation is provided here:

Make sure you replace the script with the full dataset.

As we primed our movies indexed with vector embeddings, it's time to start our experiments on fine tuning k and num_candidates attributes.

kNN search

As we have vector data in our movies index, we will be using approximate k-nearest neighbor (kNN) search. For example, to recommend movies similar that has father-son sentiment ("Father and son" as search query), we'll use a kNN search to find the nearest neighbors:

In the given example, the query leverages the top-level kNN search option parameter that directly focuses on finding documents closest to a given query vector. One key difference between this search with knn query at the top level as opposed to query at the top level is that in the former case, the query vector will be generated on-the-fly by a machine learning model.

The part in bold is not technically correct. On-the-fly vector generation is only achieved by using query_vector_builder instead of query_vector where you pass in the vector (computed outside of ES) but both the top-level knn search option and the knn search query provide this capability.

The script fetches the relevant results based on our search query (which is built using the query_vector_builder block). We are using a random k and num_candidates values set to 5 and 10 respectively.

kNN query attributes

The above query has a set of attributes that would make up the kNN query. The following information about these attributes will help you understand the query better:

The field attribute specifies the field in the index that contains the vector representations of our documents. In this case, title_vector.predicted_value is the field storing the document vectors.

The query_vector_builder attribute is where the example significantly diverges from simpler kNN queries. Instead of providing a static query vector, this configuration dynamically generates a query vector using a text embedding model. The model transforms a piece of text ("Father and son" in the example) into a vector that represents its semantic meaning.

The text_embedding indicates that a text embedding model will be used to generate the query vector.

The model_id is the identifier for the pre-trained machine learning model to use, It is the .multilingual-e5-small model in this example.

The model_text attribute is the text input that will be converted into a vector by the specified model. Here, it's the words "Father and son", which the model will interpret semantically to find similar movie titles.

The k is the number of nearest neighbors to retrieve - that is, it determines how many of the most similar documents to return based on the query vector.

The num_candidates attribute is the broader set of candidate documents per shard as potential matches to ensure the final results are as accurate as possible.

kNN results

Executing the kNN basic search script should get us top 5 results - for brevity, I'm providing just the list of the movies.

As you can expect, Godfather (both parts) are part of the father-and-son bonding while Pulp Fiction shouldn't have been part of the results (though the query is asking about "bonding" - Pulp Fiction is all about the bonding between few people).

Now that we have a basic framework setup, we can tweak the parameters appropriate and deduce the approximate settings. Before we tweak the settings, let's understand the optimal setting of k attribute.

Choosing optimal K value

Choosing the optimal value of k in k-Nearest Neighbors (kNN) algorithms is crucial for attaining the best possible performance on our dataset with minimal errors. However, there isn't a one-size-fits-all answer, as the best k value can depend on a few factors such as specifics of our data and what we are trying to predict.

To choose an optimal k value, one must create a custom framework with several strategies and considerations.

• k = 1: Try running the search query with k=1 as a first step. Make sure you change the input query for each run. The query should give you unreliable results as changing the input query will return incorrect results over time. This leads to a ML pattern called "overfitting" where the model becomes overly reliant on the specific data points in the immediate neighborhood. Model, thus, struggles to generalize to unseen examples.

• k = 5: Run the search query with k=5 and check the predictions. The stability of the search query should ideally improved and you should be getting adequate reliable predictions.

You can either incrementally increase the value of k - may be increase in the steps of 5 or x - until you find that sweet spot where you'd find the results for the input queries are pretty much spot on with less number of errors.

You can go to extreme values of k too, for example, pick a higher value of k=50, as discussed below:

• k = 50: Increase the k value to 50 and check the search results. The errored results most likely outshine the actual/expected predictions. This is when you know that you are hitting the hard boundary of the k value. Larger k values leads to a ML feature called "underfitting" - a underfitting in KNN happens when the model is too simplistic and fails to capture the underlying patterns in the data.

Choosing the optimal num_candidates value

The num_candidates parameter plays a crucial role in finding the optimal balance between search accuracy and performance. Unlike k, which directly influences the number of search results returned, num_candidates determines the size of the initial candidate set from which the final k nearest neighbors are selected. As discussed earlier, the num_candidates parameter defines how many nearest neighbors will be selected on each shard.

Adjusting this parameter is essential for ensuring that the search process is both efficient and yields high-quality results.

• num_candidates = Small Value (e.g., 10): Start with a low value ("low-value-exploration") for num_candidates as a preliminary step. The aim is to establish a baseline for performance at this stage. As the candidate bunch is just a handful of candidates, the search will be fast but might miss relevant results - which leads to poor accuracy. This scenario helps us to understand the minimum threshold where the search quality is noticeably compromised.

• num_candidates = Moderate Value (e.g., 25?): Increase the num_candidates to a moderate value ("moderate-value-exploration") and observe the changes in search quality and execution time. A moderate number of candidates is likely to improve the accuracy of the results by considering a wider pool of potential neighbors. As the number of candidates increased, there's going to be cost of resources, be mindful of that. So, keep monitoring the performance metrics closely. However, as the search accuracy increases, perhaps the increase in computational cost could be justifiable.

• num_candidates = Step Increase: Continue to incrementally increase num_candidates (incremental-increase-exploration), possibly in steps of 20 or 50 (depending on the size of your dataset). Evaluate whether the additional candidates contribute to a meaningful improvement in search accuracy with each of the increments. There will be a a point of diminishing returns where increasing num_candidates further yields little to no improvement in result quality. At the same time you may have noticed, this will strain our resources and significantly impacts performance.

• num_candidates = High Value (say, 1000, 5000): Experiment with a high value for num_candidates to understand the upper bounds of the impact of choosing the higher settings. There's a possibility of your search accuracy stabilizing or degrading slightly due to the inclusion of less relevant candidates. This may lead to dilute the precision of the final k results. Do note that, as we've been talking about it, the high values of num_candidates will always increase the computational load - thus longer query times and potential resource constraints.

Finding the optimal balance

We now know how to adjust the k and num_candidates attributes and how our experiments to different settings would change the outcome of search accuracy.

The goal is to find a sweet spot where the search results are consistently accurate with lower performance overhead from processing a large candidate set is manageable.

Of course, the optimal value will vary depending on the specifics of our data, the dimensionality of the vectors, and other performance requirements.

Wrap up

The optimal K value lies in finding the sweet spot by experiment and trials. You want to use enough neighbors (K being lower side) to capture the essential patterns but not so many (k being on the higher side) that the model becomes overly influenced by noise or irrelevant details. You also want to tweak the candidates so that the search results are accurate at a given k value.

This new functionality further solidifies our commitment to not only working with Microsoft and the Azure platform, but also toward our commitment to offering our customers more flexibility with their AI solutions.

Ongoing Investment in AI at Elastic

This is the latest in a series of additional features and integrations on AI enablement for Elasticsearch following on from:

• Elasticsearch open inference API adds Azure AI Studio support

• Elasticsearch open inference API adds support for Azure OpenAI chat completions

• Elasticsearch open inference API adds support for OpenAI chat completions

• Elasticsearch open inference API adds support for Cohere Embeddings

• Introducing Elasticsearch vector database to Azure OpenAI Service On Your Data (preview)

The new inference embeddings service provider for Azure OpenAI is already available in our stateless offering on Elastic Cloud, and will be soon available to everyone in an upcoming Elastic release.

Using Azure OpenAI Embeddings with the Elasticsearch Inference API

Deploying an Azure OpenAI Embeddings Model

To get started, you will need a Microsoft Azure Subscription as well as access to Azure OpenAI service. Once you have registered and have access, you will need to create a resource in your Azure Portal, and then deploy an embedding model to Azure OpenAI Studio. To do this, if you do not already have an Azure OpenAI resource in your Azure Portal, create a new one from the “Azure OpenAI” type which can be found in the Azure Marketplace, and take note of your resource name as you will need this later. When you create your resource, the region you choose may impact what models you have access to. See the Azure OpenAI deployment model availability table for additional details.

Once you have your resource, you will also need one of your API keys which can be found in the “Keys and Endpoint” information from the Azure Portal's left side navigation:

Now, to deploy your Azure OpenAI Embedding model, go into your Azure OpenAI Studio's console and create your deployment using an OpenAI Embeddings model such as text-embedding-ada-002. Once your deployment is created, you should see the deployment overview. Also take note of the deployment name, in the example below it is “example-embeddings-model”.

Using your deployed Azure OpenAI embeddings model with the Elasticsearch Inference API

With an Azure OpenAI embeddings model deployed, we can now configure your Elasticsearch deployment's _inference API and create a pipeline to index embeddings vectors in your documents. Please refer to the Elastic Search Labs GitHub repository for more in-depth guides and interactive notebooks.

To perform these tasks, you can use the Kibana Dev Console, or any REST console of your choice.

First, configure your inference endpoint using the create inference model endpoint - we'll call this “example_model”:

For your inference endpoint, you will need your API key, your resource name, and the deployment id that you created above. For the “api_version”, you will want to use an available API version from the Azure OpenAI embeddings documentation - we suggest always using the latest version which is “2024-02-01” as of this writing. You can also optionally add a username in the task setting's “user” field which should be a unique identifier representing your end-user to help Azure OpenAI to monitor and detect abuse. If you do not want to do this, omit the entire “task_settings” object.

After running this command you should receive a 200 OK status indicating that the model is properly set up.

Using the perform inference endpoint, we can see an example of your inference endpoint at work:

The output from the above command should provide the embeddings vector for the input text:

Now that we know our inference endpoint works, we can create a pipeline that uses it:

This will create an ingestion pipeline named “azureopenai_embeddings” that will read the contents of the “name” field upon ingestion and apply the embeddings inference from our model to the “name_embedding” output field. You can then use this ingestion pipeline when documents are ingested (e.g. via the _bulk ingest endpoint), or when reindexing an index that is already populated.

This is currently available through the open inference API in our stateless offering on Elastic Cloud. It'll also be soon available to everyone in an upcoming versioned Elasticsearch release, with additional semantic text capabilites that will make this step even simpler to integrate into your existing workflows.

For an additional use case, you can walk through the semantic search with inference tutorial for how to perform ingestion and semantic search on a larger scale with Azure OpenAI and other services such as reranking or chat completions.

Plenty more on the horizon

This new extensibility is only one of many new features we are bringing to the AI table from Elastic. Bookmark Elastic Search Labs now to stay up to date! Ready to build RAG into your apps? Want to try different LLMs with a vector database? Check out our sample notebooks for LangChain, Cohere and more on Github, and join the Elasticsearch Engineer training starting soon!

Developers can use the capabilities of the world's most downloaded vector database to store and utilize embeddings generated from OpenAI models from Azure AI studio or access the wide array of chat completion model deployments for quick access to conversational models like mistral-small.

Just recently we've added support for Azure OpenAI text embeddings and completion, and now we've added support for utilizing Azure AI Studio. Microsoft Azure developers have complete access to Azure OpenAI & Microsoft Azure AI Studio service capabilities and can bring their Elasticsearch data to revolutionize conversational search.

Let's walk you through just how easily you can use these capabilities with Elasticsearch.

Deploying a model in Azure AI Studio

To get started, you'll need a Microsoft Azure subscription as well as access to Azure AI Studio. Once you are set up, you'll need to deploy either a text embedding model or a chat completion model from the Azure AI Studio model catalog. Once your model is deployed, on the deployment overview page take note of the target URL and your deployment's API key - you'll need these later to create your inference endpoint in Elasticsearch.

Furthermore, when you deploy your model, Azure offers two different types of deployment options - a “pay as you go” model (where you pay by the token), and a “realtime” deployment which is a dedicated VM that is billed by the hour. Not all models will have both deployment types available, so be sure to take note as well as which deployment type is used.

Creating an Inference API Endpoint in Elasticsearch

Once your model is deployed, we can now create an endpoint for your inference task in Elasticsearch. For the examples below we are using the Cohere Command R model to perform chat completion.

In Elasticsearch, create your endpoint by providing the service as “azureaistudio”, and the service settings including your API key and target from your deployed model. You'll also need to provide the model provider, as well as the endpoint type from before (either “token” or “realtime”). In our example, we've deployed a Cohere model with a token type endpoint.

When you send Elasticsearch the command, it should return back the created model to confirm that it was successful. Note that the API key will never be returned and is stored in Elasticsearch's secure settings.

Adding a model for using text embeddings is just as easy. For reference, if we had deployed the Cohere-embed-v3-english model, we can create our inference model in Elasticsearch with the “text_embeddings” task type by providing the appropriate API key and target URL from that deployment's overview page:

Let's perform some inference

That's all there is to setting up your model. Now that that's out of the way, we can use the model. First, let's test the model out by asking it to provide some text given a simple prompt. To do this, we'll call the _inference API with our input text:

And we should see Elasticsearch provide a response. Behind the scenes, Elasticsearch is calling out to Azure AI Studio with the input text and processes the results from the inference. In this case, we received the response:

We've tried to make it easy for the end user to not have to deal with all the technical details behind the scenes, but we can also control our inference a bit more by providing additional parameters to control the processing such as sampling temperature and requesting the maximum number of tokens to be generated:

That was easy. What else can we do?

This becomes even more powerful when we are able to use our new model in other ways such as adding additional text to a document when it's used in an Elasticsearch ingestion pipeline. For example, the following pipeline definition will use our model and anytime a document using this pipeline is ingested, any text in the field “question_field” will be sent through the inference API and the response will be written to the “completed_text_answer” field in the document. This allows large batches of documents to be augmented.

Limitless possibilities

By harnessing the power of Azure AI Studio deployed models in your Elasticsearch inference pipelines, you can enhance your search experience's natural language processing and predictive analytics capabilities.

In upcoming versions of Elasticsearch, users can take advantage of new field mapping types that simplify the process even further where designing an ingest pipeline would no longer be necessary. Also, as alluded to in our accelerated roadmap for semantic search the future will provide dramatically simplified support for inference tasks with Elasticsearch retrievers at query time.

These capabilities are available through the open inference API in our stateless offering on Elastic Cloud. It'll also be soon available to everyone in an upcoming versioned Elasticsearch release.

Elastic Cloud on Kubernetes (ECK) is a certified offering on Red Hat OpenShift. Elastic is an IBM partner, and IBM Watsonx Assistant and Watsonx Discovery use Elastic vector search for question-answering and retrieval augmentation use cases.

With this collaboration, Elasticsearch users can benefit from Red Hat OpenShift AI, a flexible, scalable MLOps platform for building, training, testing, and serving models for AI-enabled applications.

Elasticsearch vector database for generative AI and RAG apps

Elasticsearch Relevance Engine (ESRE) is a comprehensive suite of developer tools for building generative AI and RAG applications. ESRE incorporates a vector database that stores embeddings for text, image, and video data. ESRE’s native hybrid search can effectively combine results containing text, vectors, and geospatial data, with filtering, aggregations, and document-level security.

With ESRE, developers can implement vector search and semantic search, including k-nearest neighbors (kNN) and approximate nearest neighbor (ANN) search, along with support for both built-in and third-party natural language processing (NLP) models. ESRE also seamlessly integrates with key third-party ecosystem products from providers such as Cohere, LangChain, and LlamaIndex. Elasticsearch can be self-managed or deployed with Elastic Cloud.

As part of this collaboration, users are now able to leverage ESRE capabilities by downloading Elasticsearch directly from the Red Hat Ecosystem Catalog.

What is Red Hat OpenShift AI for generative AI apps

Red Hat OpenShift AI is a hybrid MLOps platform that brings IT, data science, and app dev teams together. Designed to simplify Generative AI application development and deployment, it provides a comprehensive infrastructure stack tailored for distributed workloads. This includes training, optimizing, fine-tuning, and deploying foundational and predictive AI models. Collaborating with model builders helps provide access to a variety of pre-built models. Developers and data scientists can work together on the same platform, greatly enhancing collaboration. The platform facilitates end-to-end AI lifecycle management—from model development and training to deployment, serving, and continuous monitoring.

• Model development: Conduct exploratory data science in JupyterLab with access to core AI / ML libraries and frameworks, including TensorFlow and PyTorch using our notebook images or your own.

• Model serving & monitoring: Deploy models across on-premise or any cloud, either in a fully managed or self-managed Red Hat OpenShift footprint and centrally monitor their performance.

• Lifecycle Management: Create repeatable data science pipelines for model training and validation and integrate them with DevOps pipelines for the delivery of models across your enterprise.

• Increased capabilities and collaboration: Create projects and share them across teams. Combine Red Hat components, open-source software, and ISV-certified software.

Get started with Red Hat and Elasticsearch

To get started, just follow the installation instructions provided in the Red Hat Ecosystem Catalog, and start building your next generative AI application with RAG!

Visit Elasticsearch Labs for articles and sample notebooks on vector search, RAG, and more.

Our mission at Elastic is to make Apache Lucene the best vector database out there, and to continue to make Elasticsearch the best retrieval platform out there for search and RAG. Our investments into Lucene are key to ensure that every release of Elasticsearch brings increasing faster performance and scale.

Customers are already building the next generation of AI enabled search applications with Elastic’s vector database and vector search technology. Roboflow is used by over 500,000 engineers to create datasets, train models, and deploy computer vision models to production. Roboflow uses Elastic vector database to store and search billions of vector embeddings.

In this blog we summarize recent enhancements and optimisations that significantly improve vector search performance in Elasticsearch and Apache Lucene, over and above performance gains delivered with Lucene 9.9 and Elasticsearch 8.12.x.

The integration of vector search into Elasticsearch relies on Apache Lucene, the layer that orchestrates data storage and retrieval. Lucene's architecture organizes data into segments, immutable units that undergo periodic merging. This structure allows for efficient management of inverted indices, essential for text search. With vector search, Lucene extends its capabilities to handle multi-dimensional points, employing the hierarchical navigable small world (HNSW) algorithm to index vectors.

This approach facilitates scalability, enabling data sets to exceed available RAM size while maintaining performance. Additionally, Lucene's segment-based approach offers lock-free search operations, supporting incremental changes and ensuring visibility consistency across various data structures. The integration however comes with its own engineering challenges. Merging segments requires recomputing HNSW graphs, incurring index-time overhead. Searches must cover multiple segments, leading to possible latency overhead. Moreover, optimal performance requires scaling RAM as data grows, which may raise resource management concerns.

Lucene's integration into Elasticsearch comes with the benefit of robust vector search capabilities. This includes aggregations, document level security, geo-spatial queries, pre-filtering, to full compatibility with various Elasticsearch features. Imagine running vector searches using a geo bounding box, this is an example usecase enabled by Elasticsearch and Lucene.

Lucene's architecture lays a solid foundation for efficient and versatile vector search within Elasticsearch. Let’s explore optimization strategies and enhancements we have implemented to integrate vector search into Lucene, which delivers a high performance and comprehensive feature-set for developers.

Harnessing Lucene's architecture for multi-threaded search

Lucene's segmented architecture enables the implementation of multi-threaded search capabilities. Elasticsearch’s performance gains come from efficiently searching multiple segments simultaneously. Latency of individual searches is significantly reduced by using the processing power of all available CPU cores. While it may not directly improve overall throughput, this enhancement prioritizes minimizing response times, ensuring that users receive their search results as swiftly as possible.

Furthermore, this optimization is particularly beneficial for Hierarchical Navigable Small World (HNSW) searches, as each graph is independent of the others and can be searched in parallel, maximizing efficiency and speeding up retrieval times even further.

The advantage of having multiple independent segments extends to the architectural level, especially in serverless environments. In this new architecture, the indexing tier is responsible for creating new segments, each containing its own HSNW graph. The search tier can simply replicate these segments without incurring the CPU cost of indexation. This separation allows a significant portion of compute resources to be dedicated to searches, optimizing overall system performance and responsiveness.

Accelerating multi-graph vector search

In spite of gains achieved with parallelization, each segment's searches would remain independent, unaware of progress made by other segment searches. So our focus shifted towards optimizing the efficiency of concurrent searches across multiple segments.

The graph shows that the number queries per second increased from 104 queries/sec to 219 queries/sec.

Recognizing the potential for further speedups, we leveraged our insights from optimizing lexical search, to enable information exchange among segment searches allowing for better coordination and efficiency in vector search.

Our strategy for accelerating multi-graph vector search revolves around balancing exploration and exploitation within the proximity graph. By adjusting the size of the expanded match set, we control the trade-off between runtime and recall, crucial for achieving optimal search performance across multiple graphs.

In multi-graph search scenarios, the challenge lies in efficiently navigating individual graphs, while ensuring comprehensive exploration to avoid local minima. While searching multiple graphs independently yields higher recall, it incurs increased runtime due to redundant exploration efforts. To mitigate this, we devised a strategy to intelligently share state between searches, enabling informed traversal decisions based on global and local competitive thresholds.

This approach involves maintaining shared global and local queues of distances to closest vectors, dynamically adapting search parameters based on the competitiveness of each graph's local search. By synchronizing information exchange and adjusting search strategies accordingly, we achieve significant improvements in search latency while preserving recall rates comparable to single-graph searches.

The impact of these optimizations is evident in our benchmark results. In concurrent search and indexing scenarios, we notice up to 60% reduction in query latencies! Even for queries conducted outside of indexing operations, we observed notable speedups and a dramatic decrease in the number of vector operations required. These enhancements, integrated into Lucene 9.10 and subsequently Elasticsearch 8.13, mark significant strides towards enhancing vector database performance for search while maintaining excellent recall rates.

Harnessing Java's latest advancements for ludicrous speed

In the area of Java development, automatic vectorization has been a boon, optimizing scalar operations into SIMD (Single Instruction Multiple Data) instructions through the HotSpot C2 compiler. While this automatic optimization has been beneficial, it has its limitations, particularly in scenarios where explicit control over code shape yields superior performance. Enter Project Panama Vector API, a recent addition to the JDK offering an API for expressing computations reliably compiled to SIMD instructions at runtime.

Lucene's vector search implementation relies on fundamental operations like dot product, square, and cosine distance, both in floating point and binary variants. Traditionally, these operations were backed by scalar implementations, leaving performance enhancements to the JIT compiler. However, recent advancements introduce a paradigm shift, enabling developers to express these operations explicitly for optimal performance.

Consider the dot product operation, a fundamental vector computation. Traditionally implemented in Java with scalar arithmetic, recent innovations leverage the Panama Vector API to express dot product computations in a manner conducive to SIMD instructions. This revised implementation iterates over input arrays, multiplying and accumulating elements in batches, aligning with the underlying hardware capabilities.

By harnessing Panama Vector API, Java code now interfaces seamlessly with SIMD instructions, unlocking the potential for significant performance gains. The compiled code, when executed on compatible CPUs, leverages advanced vector instructions like AVX2 or AVX 512, resulting in accelerated computations. Disassembling the compiled code reveals optimized instructions tailored to the underlying hardware architecture.

Microbenchmarks comparing traditional Java implementations to those leveraging Panama Vector API illustrate dramatic performance improvements. Across various vector operations and dimension sizes, the optimized implementations outperform their predecessors by significant margins, offering a glimpse into the transformative power of SIMD instructions.

Micro-benchmark comparing dot product with the new Panama API (dotProductNew) and the scalar implementation (dotProductOld).

Beyond microbenchmarks, the real-world impact of these optimizations is quite exciting to think about. Vector search benchmarks, such as SO Vector, demonstrate notable enhancements in indexing throughput, merge times, and query latencies. Elasticsearch, embracing these advancements, incorporates the faster implementations by default, ensuring users reap the performance benefits seamlessly.

The graph shows indexing throughput increased from about 900 documents/sec to about 1300 documents/sec.

Despite the incubating status of Panama Vector API, its quality and potential benefits are undeniable. Lucene's pragmatic approach allows for selective adoption of non-final JDK APIs, balancing the promise of performance improvements with maintenance considerations. With Lucene and Elasticsearch, users can leverage these advancements effortlessly, with performance gains translating directly to real-world workloads.

The integration of Panama Vector API into Java development yields a new era of performance optimization, particularly in vector search scenarios. By embracing hardware-accelerated SIMD instructions, developers can unlock efficiency gains, visible both in microbenchmarks and macro-level benchmarks. As Java continues to evolve, leveraging its latest features promises to propel performance to new heights, enriching user experiences across diverse applications.

Maximizing memory efficiency with scalar quantization

Memory consumption has long been a concern for efficient vector database operations, particularly for searching large datasets. Lucene introduces a breakthrough optimization technique - scalar quantization - aimed at significantly reducing memory requirements without sacrificing search performance.

Consider a scenario where querying millions of float32 vectors of high dimensions demands substantial memory, leading to significant costs. By embracing byte quantization, Lucene slashes memory usage by approximately 75%, offering a viable solution to the memory-intensive nature of vector search operations.

For quantizing floats to bytes, Lucene implements Scalar quantization a lossy compression technique that transforms raw data into a compressed form, sacrificing some information for space efficiency. Lucene's implementation of scalar quantization achieves remarkable space savings with minimal impact on recall, making it an ideal solution for memory-constrained environments.

Lucene's architecture, consisting of nodes, shards, and segments, which facilitates efficient distribution and management of documents for search. Each segment stores raw vectors, quantized vectors, and metadata, ensuring optimized storage and retrieval mechanisms.

Lucene's vector quantization adapts dynamically over time, adjusting quantiles during segment merge operations to maintain optimal recall. By intelligently handling quantization updates and re-quantization when necessary, Lucene ensures consistent performance while accommodating changes in data distribution.

Example of merged quantiles where segments A and B have 1000 documents and C only has 100.

Experimental results demonstrate the efficacy of scalar quantization in reducing memory footprint while maintaining search performance. Despite minor differences in recall compared to raw vectors, Lucene's quantized vectors offer significant speed improvements and recall recovery with minimal additional vectors.

Recall@10 for quantized vectors vs raw vectors. The search performance of quantized vectors is significantly faster than raw, and recall is quickly recoverable by gathering just 5 more vectors; visible by quantized@15.

Lucene's scalar quantization presents a revolutionary approach to memory optimization in vector search operations. With no need for training or optimization steps, Lucene seamlessly integrates quantization into its indexing process, automatically adapting to changes in data distribution over time. As Lucene and Elasticsearch continue to evolve, widespread adoption of scalar quantization will revolutionize memory efficiency for vector database applications, paving the way for enhanced search performance at scale.

Achieving seamless compression with minimal impact on recall

To make compression even better, we aimed to reduce each dimension from 7 bits to just 4 bits. Our main goal was to compress data further while still keeping search results accurate. By making some improvements, we managed to compress data by a factor of 8 without making search results worse. Here's how we did it.

We focused on keeping search results accurate while making data smaller. By making sure we didn't lose important information during compression, we could still find things well even with less detailed data. To make sure we didn't lose any important information, we added a smart error correction system.

We checked our compression improvements by testing them with different types of data and real search situations. This helped us see how well our searches worked with different compression levels and what we might lose in accuracy by compressing more.

Comparison of int4 dot product values to the corresponding float values for a random sample of 100 documents and their 10 nearest neighbors.

These compression features were created to easily work with existing vector search systems. They help organizations and users save space without needing to change much in their setup. With this simple compression, organizations can expand their search systems without wasting resources.

In short, moving to 4 bits per dimension for scalar quantization was a big step in making compression more efficient. It lets users compress their original vectors by 8 times. By optimizing carefully, adding error correction, testing with real data, and offering scalable deployment, organizations could save a lot of storage space without making search results worse. This opens up new chances for efficient and scalable search applications.

Paving the way for binary quantization

The optimization to reduce each dimension to 4 bits not only delivers significant compression gains but also lays the groundwork for further advancements in compression efficiency. Specifically, future advancements like binary quantization into Lucene, a development that has the potential to revolutionize vector storage and retrieval.

In an ongoing effort to push the boundaries of compression in vector search, we are actively working on integrating binary quantization into Lucene using the same techniques and principles that underpin our existing optimization strategies. The goal is to achieve binary quantization of vector dimensions, thereby reducing the size of the vector representation by a factor of 32 compared to the original floating-point format.

Through our iterations and experiments, we want to deliver the full potential of vector search while maximizing resource utilization and scalability. Stay tuned for further updates on our progress towards integrating binary quantization into Lucene and Elasticsearch, and the transformative impact it will have on vector database storage and retrieval.

Multi-vector integration in Lucene and Elasticsearch

Several real world applications rely on text embedding models and large text inputs. Most embedding models have token limits, which necessitate chunking of longer text into passages. Therefore, instead of a single document, multiple passages and embeddings must be managed, potentially complicating metadata preservation.

Now instead of having a single piece of metadata indicating, for example the first chapter of the book “Little Women”, you have to index that information data for every sentence.

Lucene's "join" functionality, integral to Elasticsearch's nested field type, offers a solution. This feature enables multiple nested documents within a top-level document, allowing searches across nested documents and subsequent joins with their parent documents. So, how do we deliver support for vectors in nested fields with Elasticsearch?

The key lies in how Lucene joins back to parent documents when searching child vector passages. The parallel concept here is the debate around pre-filtering versus post-filtering in kNN methods, as the timing of joining significantly impacts result quality and quantity. To address this, recent enhancements to Lucene enable pre-joining against parent documents while searching the HNSW graph.

Practically, pre-joining ensures that when retrieving the k nearest neighbors of a query vector, the algorithm returns the k nearest documents instead of passages. This approach diversifies results without complicating the HNSW algorithm, requiring only a minimal additional memory overhead per stored vector.

Efficiency is improved by leveraging certain restrictions, such as disjoint sets of parent and child documents and the monotonicity of document IDs. These restrictions allow for optimizations using bit sets, providing rapid identification of parent document IDs.

Searching through a vast number of documents efficiently required investing in nested fields and joins in Lucene. This work helps storage and search for dense vectors that represent passages within long texts, making document searches in Lucene more effective. Overall, these advancements represent an exciting step forward in the area of vector database retrieval within Lucene.

Wrapping up (for now)

We're dedicated to making Elasticsearch and Lucene the best vector database with every release. Our goal is to make it easier for people to search for things. With some of the investments we discuss in this blog, there is significant progress, but we're not done!

To say that the gen AI ecosystem is rapidly evolving is an understatement. At Elastic, we want to give developers the most flexible and open tools to keep up with all the innovation—with features available across recent releases until 8.13 and serverless

GCP Vector Search optimized instances: what you need to know

Elastic Cloud users benefit from having Elastic managed infrastructure across all major cloud providers (GCP, AWS and Azure) along with wide region support for GCP users. For more specific details on the instance configuration for this hardware profile, refer to our documentation for instance type: gcp.es.datahot.n2d.64x8x11

Vector Search, HNSW, and memory

Elasticsearch uses the Hierarchical Navigable Small World graph (HNSW) data structure to implement its Approximate Nearest Neighbor search (ANN). Because of its layered approach, HNSW's hierarchical aspect offers excellent query latency. To be most performant, HNSW requires the vectors to be cached in the node's memory. This caching is done automatically and uses the available RAM not taken up by the Elasticsearch JVM. Because of this, memory optimizations are important steps for scalability.

Consult our vector search tuning guide to determine the right setup for your vector search embeddings and whether you have adequate memory for your deployment.

With this in mind, the Vector Search optimized hardware profile is configured with a smaller than standard Elasticsearch JVM heap setting. This provides more RAM for caching vectors on a node, allowing users to provision fewer nodes for their vector search use cases.

If you’re using compression techniques like scalar quantization, the memory requirement is lowered by a factor of 4. To store quantized embeddings (available in versions Elasticsearch 8.12 and later) simply ensure that you’re storing in the correct element_type: byte. To utilize our automatic quantization of float vectors update your embeddings to use index type: int8_hnsw like in the following mapping example.

In upcoming versions, Elasticsearch will provide this as the default mapping, removing the need for users to adjust their mapping.

Combining this optimized hardware profile with Elasticsearch’s automatic quantization are two examples where Elastic is focused on vector search to be cost-effective while still being extremely performant.

Getting Started with Elastic Cloud vector search optimized profile for GCP

Start a free trial on Elastic Cloud and simply select the new Vector Search optimized profile to get started.

Migrating existing Elastic Cloud deployments

Migrating to this new Vector Search optimized hardware profile is a few clicks away. Simply navigate to your Elastic Cloud management UI, click to manage the specific deployment, and edit the hardware profile. In this example, we are migrating from a ‘Storage optimized’ profile to the new ‘Vector Search’ optimized profile. When choosing to do so, while there is a reduction to available storage and vCPU, what is gained is the ability to store more vectors per memory with vector search.

Migrating to a new hardware profile uses the grow and shrink approach for deployment changes. This approach adds new instances, migrates data from old instances to the new ones, and then shrinks the deployment by removing the old instances. This approach allows for high availability during configuration changes even for single availability zones.

The following image shows a typical architecture for a deployment running in Elastic Cloud, where vector search will be the primary use case.

This example deployment uses our new Vector Search optimized hardware profile, now available in GCP. This setup includes:

• Two data nodes in our hot tier with our vector search profile

• One Kibana node

• One Machine Learning node

• One integration server

• One master tiebreaker

By deploying these two “full-sized” data nodes with the Vector Search optimized hardware profile and while taking advantage of Elastic’s automatic dense vector scalar quantization, you can index roughly 60 million vectors, including one replica (with 768 dimensions).

Conclusion

Vector search is a powerful tool when building modern search applications, be it for semantic document retrieval on its own or integrating with an LLM service provider in a RAG setup. Elasticsearch provides a full-featured vector database natively integrated with a full-featured search platform. Along with improving vector search feature set and usability, Elastic continues to improve scalability. The vector search node type is the latest example, allowing users to scale their search application.

Elastic is committed to providing scalable, price effective infrastructure to support enterprise grade search experiences. Customers can depend on us for reliable and easy to maintain infrastructure and cost levers like vector compression, so you benefit from the lowest possible total cost of ownership for building search experiences powered by AI.

So briefly, what is reranking? Rerankers take the ‘top n’ search results from existing vector search and keyword search systems, and provide a semantic boost to those results. With good reranking in place, you have better ‘top n’ results without requiring you to change your model or your data indexes – ultimately providing better search results you can send to large language models (LLMs) as context.

Recently, we collaborated with the Cohere team to make it easy for Elasticsearch developers to use Cohere’s embeddings (available in Elasticsearch 8.13 and Serverless!). It is a natural evolution to include Cohere’s incredible reranking capabilities to unlock all of the tools necessary for true refinement of results past the first-stage of retrieval.

Cohere’s Rerank 3 model can be added to any existing Elasticsearch retrieval flow without requiring any significant code changes. Given Elastic’s vector database and hybrid search capabilities, users can also bring embeddings from any 3rd party model to Elastic, to use with Rerank 3.

Elastic’s approach to hybrid search

When looking to implement RAG (Retrieval Augmented Generation), the strategy for retrieval and reranking is a key optimization for customers to ground LLMs and achieve accurate results. Customers have trusted Elastic for years with their private data, and are able to leverage several first-stage retrieval algorithms (e.g. for BM25/keyword, dense, and sparse vector retrieval). More importantly, most real-world search use cases benefit from hybrid search which we have supported since Elasticsearch 8.9.

For mid-stage reranking, we also offer native support for Learning To Rank and query rescore. In this walkthrough, we will focus on Cohere’s last stage reranking capabilities, and will cover Elastic’s mid stage reranking capabilities in a subsequent blog post!

Cohere’s approach to reranking

Cohere has seen phenomenal results with their new Rerank model. In the testing, Cohere is reporting that reranking models in particular benefit from long context. Chunking for model token limits is a necessary constraint when preparing your document for dense vector retrieval. But with Cohere’s approach for reranking, a considerable benefit to reranking can be seen based on context contained in the full document, rather than a specific chunk within the document. Rerank has a 4k token limit to enable the input of more context to unlock the full relevance benefits of incorporating this model into your Elasticsearch based search system.

(i) General retrieval based on BEIR benchmark; accuracy measured as nDCG@10

(ii) Code retrieval based on 6 common code benchmarks; accuracy measured as nDCG@10

(iii) Long context retrieval based on 7 common benchmarks; accuracy measured as nDCG@10

(iv) Semi-structured (JSON) retrieval based on 4 common benchmarks; accuracy measured as nDCG@10

If you’re interested in how to chunk with LangChain and LlamaIndex, we provide chat application reference code, integrations and more in Search Labs and our open source repository. Alternatively, you can leverage Elastic’s passage retrieval capabilities and chunk with ingest pipelines.

Building a RAG implementation with Elasticsearch and Cohere

Now that you have a general understanding of how these capabilities can be leveraged, let’s jump into an example on building a RAG implementation with Elasticsearch and Cohere.

You'll need a Cohere account and some working knowledge of the Cohere Rerank endpoint. If you’re intending to use Cohere’s newest generative model Command R+ familiarize yourself with the Chat endpoint.

In Kibana, you'll have access to a console for you to input these next steps in Elasticsearch even without an IDE set up. If you prefer to use a language client - you can revisit these steps in the provided guide.

Elasticsearch vector database

In an earlier announcement, we had some steps to get you started with the Elasticsearch vector database. You can review the steps to cover ingesting a sample books catalog, and generate embeddings using Cohere’s Embed capabilities by reading the announcement. Alternatively, if you prefer we also provide a tutorial and Jupyter notebook to get you started on this process.

Cohere reranking

The following section assumes that you’ve ingested data and have issued your first search. This will give you a baseline as to how the search results are ranked with your first dense vector retrieval.

The previous announcement concluded with a query issued against the sample books catalog, and, and generated the following results in response to the query string “Snow”. These results are returned in descending order of relevance.

You’ll next want to configure an inference endpoint for Cohere Rerank by specifying the Rerank 3 model and API key.

Once this inference endpoint is specified, you’ll now be able to rerank your results by passing in the original query used for retrieval, “Snow” along with the documents we just retrieved with the kNN search. Remember, you can repeat this with any hybrid search query as well!

To demonstrate this while still using the dev console, we’ll do a little cleanup on the JSON response above.

Take the hits from the JSON response and form the following JSON for the input, and then POST to the cohere_rerank endpoint we just configured.

And there you have it, your results have been reranked using Cohere's Rerank 3 model.

The books corpus that we used to illustrate these capabilities does not contain large passages, and is a relatively simple example. When instrumenting this for your own search experience, we recommend that you follow Cohere’s approach to populate your input with the context from the full documents returned from the first retrieved result set, not just a retrieved chunk within the documents.

Elasticsearch’s accelerated roadmap to semantic reranking and retrievers

In upcoming versions of Elasticsearch we will continue to build seamless support for mid and final stage rerankers. Our end goal is to enable developers to have the ability to use semantic reranking to improve the results from any search whether it is BM25, dense or sparse vector retrieval, or a combination with hybrid retrieval. To provide this experience, we are building a concept called retrievers into the query DSL. Retrievers will provide an intuitive way to execute semantic reranking, and will also enable direct execution of what you’ve configured in the open inference API in the Elasticsearch stack without relying on you to execute this in your application logic.

When incorporating the use of retrievers in the earlier dense vector example, this is how different the reranking experience can be:

(i) Elastic’s roadmap: The indexing step is simplified with the addition of Elastic’s future capabilities to automatically chunk indexed data

(ii) Elastic’s roadmap: The kNN retriever specifies the model (in this case Cohere’s Rerank 3) that was configured as an inference endpoint

(iii) Cohere’s roadmap: The step between sending the resulting data to Cohere’s Command R+ will benefit from a planned feature named extractive snippets which will enable the user to return a relevant chunk of the reranked document to the Command R+ model

This was our original kNN dense vector search executed on the books corpus to return the first set of results for “Snow”.

As explained in this blog, there are a few steps to retrieve the documents and pass on the correct response to the inference endpoint. At the time of this publication, this logic should be handled in your application code.

In the future, retrievers can be configured to use the Cohere rerank inference endpoint directly within a single API call.

In this case, the kNN query is exactly the same as my original, but the cleansing of the response before input to the rerank endpoint will no longer be a necessary step. A retriever will know that a kNN query has been executed and seamlessly rerank using the Cohere rerank inference endpoint specified in the configuration. This same principle can be applied to any search, BM25, dense, sparse and hybrid.

Retrievers as an enabler of great semantic reranking is on our active and near term roadmap.

Cohere’s generative model capabilities

Now you’re ready with a semantically reranked set of documents that can be used to ground the responses for the large language model of your choice! We recommend Cohere’s newest generative model Command R+. When building the full RAG pipeline, in your application code you can easily issue a command to Cohere’s Chat API with the user query and the reranked documents.

An example of how this might be achieved in your Python application code can be seen below:

This integration with Cohere is offered in Serverless and soon will be available to try in a versioned Elasticsearch release either on Elastic Cloud or on your laptop or self-managed environment. We recommend you use our Elastic Python client v0.2.0 against your Serverless project to get started!

Happy reranking!

Search has evolved from simple text queries yielding straightforward results to a complex system accommodating various formats like text, images, videos, and questions.

Search not too long ago comprised of a text query and relevant results. Today's search results are enhanced with generative AI, machine learning, and interactive chat features, offering a richer, more dynamic, and contextually relevant user experience. Additionally, voice search and speech avatars have transformed traditional search, offering a more interactive and convenient user experience.

The desire for dialogue in search

In a realm where dialogue underpins every interaction, whether with fellow humans or bots, shouldn't our search experiences reflect this fundamental aspect? Envision the vast array of document corpora residing within an enterprise. Naturally, this environment sparks curiosity and a multitude of questions, leading to subsequent inquiries. This innate human trait drives us to seek answers, delve deeper following initial responses, and continuously explore. Yet, traditional question-and-answer mechanisms fall short, as they often disregard the context of preceding exchanges, leading to a disjointed and laborious process that feels unnatural and prompts users to disengage prematurely.

Beyond question and answer search

Consider the act of using a television to search for content, such as seeking action movies featuring Nicolas Cage. While most current systems adeptly provide relevant results, the inquiry rarely ends there. Subsequent questions, such as inquiring about the runtime or release dates of these movies, are a natural progression in our quest for information. However, standard search applications are not designed to facilitate a continuous dialogue; they are structured around isolated question-and-answer formats, which limits the depth of interaction and exploration.

Avatar assisted voice search experience

This is where the concept of an avatar-assisted search experience comes into play, especially in scenarios where users, myself included, prefer direct answers without the need to sift through information. Occasionally, we desire the convenience of having answers delivered to us, bypassing the effort of reading through content. The development of an avatar to generate responses could further modernize this interaction, providing a more engaging, efficient, and natural user experience.

Live demo: creating an avatar assisted voice search experience

This demo showcases a seamless integration of speech-to-text, Elasticsearch's semantic search capabilities, Azure OpenAI's RAG, and a synthesized avatar for responses.

Integration details

Speech to search

The advanced search experience begins with user voice interactions, which are converted into text by Azure Speech to Text, forming the basis of the search query. This query is then processed through Elasticsearch, using the ELSER, to retrieve relevant documents, such as TV guides listing “action movies featuring Nicolas Cage.” This ensures precision and relevance in the search results.

RAG & cache

In the enhanced search framework, merely fetching documents isn't enough. Azure OpenAI's GPT-4 refines raw data into understandable responses, ensuring smooth conversation flow. Additionally, Elasticsearch boosts efficiency as a GenAI caching layer, recycling answers for related queries, thus conserving resources. For example, if there's a cached response for "action movies featuring Nicolas Cage," the caching API will swiftly use this for similar questions like “Nicolas Cage high-intensity movies,” accelerating the search experience.

Avatar response generation

The experience is further enriched with an avatar response feature, powered by Azure Synthesizer, adding a visual and auditory dimension that surpasses traditional text-based interfaces. This creates a more engaging and interactive user experience, integrating various advanced technologies to deliver a dynamic, intuitive, and compelling search experience.

Summary

The shift from traditional Google searches to platforms like ChatGPT for answering queries illustrates a broader trend: our preference for dialogue over static information retrieval. This predilection underscores the importance for enterprises to adopt a more intuitive and conversational approach in their search functionalities. By embracing this paradigm, businesses can better align with the natural human inclination towards dialogue, thereby enhancing the overall search and discovery process within their data ecosystems.

Demo assets

Still curious, here is the link to the source code.

There is a distinction between inspiration and paraphrasing. It is possible to read a content, get inspired, and then explore the idea with your own words, even if you come to a similar conclusion.

While plagiarism has been a topic of discussion for a long time, the accelerated production and publication of content have kept it relevant and posed an ongoing challenge.

This challenge isn't limited to books, academic research, or judicial documents, where plagiarism checks are frequently conducted. It can also extend to newspapers and even social media.

With the abundance of information and easy access to publishing, how can plagiarism be effectively checked on a scalable level?

Universities, government entities, and companies employ diverse tools, but while a straightforward lexical search can effectively detect direct plagiarism, the primary challenge lies in identifying paraphrased content.

Plagiarism detection with Generative AI

A new challenge emerges with Generative AI. Is content generated by AI considered plagiarism when copied?

The OpenAI terms of use, for example, specify that OpenAI will not claim copyright over content generated by the API for users. In this case, individuals using their Generative AI can use the generated content as they prefer without citation.

However, the acceptance of using Generative AI to improve efficiency remains a topic of discussion.

In an effort to contribute to plagiarism detection, OpenAI developed a detection model but later acknowledged that its accuracy is not sufficiently high.

"We believe this is not high enough accuracy for standalone detection and needs to be paired with metadata-based approaches, human judgment, and public education to be more effective."

The challenge persists; however, with the availability of more tools, there are now increased options for detecting plagiarism, even in cases of paraphrased and AI content.

Detecting plagiarism with Elasticsearch

Recognizing this, in this blog we are exploring one more use case with Natural Language Processing (NLP) models and Vector Search, plagiarism detection, beyond metadata searches.

This is demonstrated with Python examples, where we utilize a dataset from SentenceTransformers containing NLP-related articles. We check the abstracts for plagiarism by performing 'semantic textual similarity' considering 'abstract' embeddings generated with a text embedding model previously imported into Elasticsearch. Additionally, to identify AI-generated content — AI plagiarism, an NLP model developed by OpenAI was also imported into Elasticsearch.

The following image illustrates the data flow:

During the ingest pipeline with an inference processor, the 'abstract' paragraph is mapped to a 768-dimensional vector, the 'abstract_vector.predicted_value'.

Mapping:

The similarity between vector representations is measured using a vector similarity metric, defined using the 'similarity' parameter.

Cosine is the default similarity metric, computed as '(1 + cosine(query, vector)) / 2'. Unless you need to preserve the original vectors and cannot normalize them in advance, the most efficient way to perform cosine similarity is to normalize all vectors to unit length. This helps avoid performing extra vector length computations during the search, instead use 'dot_product'.

In this same pipeline, another inference processor containing the text classification model detects whether the content is 'Real' probably written by humans, or 'Fake' probably written by AI, adding the 'openai-detector.predicted_value' to each document.

Ingest Pipeline:

At query time, the same text embedding model is also employed to generate the vector representation of the query 'model_text' in a 'query_vector_builder' object.

A k-nearest neighbor (kNN) search finds the k nearest vector to the query vector measured by the similarity metric.

The _score of each document is derived from the similarity, ensuring that a larger score corresponds to a higher ranking. This means that the document is more similar semantically. As a result, we are printing three possibilities: if score > 0.9, we are considering 'high similarity'; if < 0.7, 'low similarity’, otherwise, 'moderate similarity’. You have the flexibility to set different threshold values to determine what level of _score qualifies as plagiarism or not, based on your use case.

Additionally, text classification is performed to also check for AI-generated elements in the text query.

Query:

Output:

In this example, after utilizing one of the 'abstract' values from our dataset as the text query 'model_text', plagiarism was identified. The similarity score is 1.0, indicating a high level of similarity — direct plagiarism. The vectorized query and document were not recognized as AI-generated content, which was expected.

Query:

Output:

By updating the text query 'model_text' with an AI-generated text that conveys the same message while minimizing the repetition of similar words, the detected similarity was still high, but the score was 0.9302529 instead of 1.0 — paraphrase plagiarism. It was also expected that this query, which was generated by AI, would be detected.

Lastly, considering the text query 'model_text' as a text about Elasticsearch, which is not an abstract of one of these documents, the detected similarity was 0.68991005, indicating low similarity according to the considered threshold values.

Query:

Output:

Although plagiarism was accurately identified in the text query generated by AI, as well as in cases of paraphrasing and direct copied content, navigating the landscape of plagiarism detection involves acknowledging various aspects.

In the context of AI-generated content detection, we explored a model that makes a valuable contribution. However, it is crucial to recognize the inherent limitations in standalone detection, necessitating the incorporation of other methods to boost the accuracy.

The variability introduced by the choice of text embedding models is another consideration. Different models, trained with distinct datasets, result in varying levels of similarity, highlighting the importance of the text embeddings generated.

Lastly, in these examples, we used the document's abstract. However, plagiarism detection often involves large documents, making it essential to address the challenge of text length. It is common for the text to exceed a model's token limit, requiring segmentation into chunks before building embeddings. A practical approach to handling this involves utilizing nested structures with dense_vector.

Conclusion

In this blog, we discussed the challenges of detecting plagiarism, particularly in paraphrased and AI-generated content, and how semantic textual similarity and text classification can be used for this purpose.

By combining these methods, we provided an example of plagiarism detection where we successfully identified AI-generated content, direct and paraphrased plagiarism.

The primary goal was to establish a filtering system that simplifies detection but human assessment remains essential for validation.

If you are interested in learning more about semantic textual similarity and NLP, we encourage you to also check out these links:

• What is semantic search?

• What is natural language processing (NLP)?

• Lexical and Semantic Search with Elasticsearch

• Chunking Large Documents via Ingest pipelines plus nested vectors equals easy passage search

kNN search in Elasticsearch is organized as a top level section of a search request. We have designed it this way so that:

• It can always return global k nearest neighbors regardless of a number of shards

• These global k results are combined with a results from other queries to form a hybrid search

• The global k results are passed to aggregations to form facets.

Here is a simplified diagram how kNN search is executed internally (some phases are omitted) :

Figure 1: The steps for the top level kNN search are:

1. A user submits a search request

2. The coordinator node sends a kNN search part of the request to data nodes in the DFS phase

3. Each data node runs kNN search and sends back the local top-k results to the coordinator

4. The coordinator merges all local results to form the global top k nearest neighbors.

5. The coordinator sends back the global k nearest neighbors to the data nodes with any additional queries provided

6. Each data node runs additional queries and sends back the local size results to the coordinator

7. The coordinator merges all local results and sends a response to the user

We first run kNN search in the DFS phase to obtain the global top k results. These global k results are then passed to other parts of the search request, such as other queries or aggregations. Even the execution looks complex, from a user’s perspective this model of running kNN search is simple, as the user can always be sure that kNN search returns the global k results.

Introducing kNN query in Elasticsearch

With time we realized there is also a need to represent kNN search as a query. Query is a core component of a search request in Elasticsearch, and representing kNN search as a query allows for flexibility to combine it with other queries to address more complex requests.

kNN query, unlike the top level kNN search, doesn’t have a k parameter. The number of results (nearest neighbors) returned is defined by the size parameter, as in other queries. Similar to kNN search, the num_candidates parameter defines how many candidates to consider on each shard while executing a kNN search.

kNN query is executed differently from the top level kNN search. Here is a simplified diagram that describes how a kNN query is executed internally (some phases are omitted):

Figure 2: The steps for query based kNN search are:

1. A user submits a search request

2. The coordinator sends to the data nodes a kNN search query with additional queries provided

3. Each data node runs the query and sends back the local size results to the coordinator node

4. The coordinator node merges all local results and sends a response to the user

We run kNN search on a shard to get num_candidates results; these results are passed to other queries and aggregations on a shard to get size results from the shard. As we don’t collect the global k nearest neighbors first, in this model the number of nearest neighbors collected and visible for other queries and aggregations depend on the number of shards.

kNN query API examples

Let’s look at API examples that demonstrate differences between the top level kNN search and kNN query.

We create an index of products and index some documents:

kNN query similar to the top level kNN search, has num_candidates and an internal filter parameter that acts as a pre-filter.

kNN query can get more diverse results than kNN search for collapsing and aggregations. For the kNN query below, on each shard we execute kNN search to obtain 10 nearest neighbors which are then passed to collapse to get 3 top results. Thus, we will get 3 diverse hits in a response.

The top level kNN search first gets the global top 3 results in the DFS phase, and then passes them to collapse in the query phase. We will get only 1 hit in a response, as all the global 3 nearest neighbors happened to be from the same brand.

Similarly for aggregations, a kNN query allows us to get 3 distinct buckets, while kNN search only allows 1.

Now, let’s look at other examples that show the flexibility of the kNN query. Specifically, how it can be flexibly combined with other queries.

kNN can be a part of a boolean query (with a caveat that all external query filters are applied as post-filters for kNN search). We can use a _name parameter for kNN query to enhance results with extra information that tells if the kNN query was a match and its score contribution.

kNN can also be a part of complex queries, such as a pinned query. This is useful when we want to display the top nearest results, but also want to promote a selected number of other results.

We can even make the kNN query a part of our function_score query. This is useful when we need to define custom scores for results returned by kNN query: ​

kNN query being a part of dis_max query is useful when we want to combine results from kNN search and other queries, so that a document’s score comes from the highest ranked clause with a tie breaking increment for any additional clause. ​

kNN search as a query has been introduced with the 8.12 release. Please try it out, and we would appreciate any feedback.

As technology advances, chatbots are used to handle more complex tasks — and quickly — while still providing a personalized experience for users. Natural language processing (NLP) enables chatbots to process the user's language, identifies the intent behind their message, and extracts relevant information from it. For example, Named Entity Recognition extracts key information in a text by classifying them into a set of categories. Sentiment Analysis identifies the emotional tone, and Question Answering the “answer” to a query. The goal of NLP is to enable algorithms to process human language and perform tasks that historically only humans were capable of, such as finding relevant passages among large amounts of text, summarizing text, and generating new, original content.

These advanced NLP capabilities are built upon a technology known as vector search. Elastic has native support for vector search, performing exact and approximate k-nearest neighbor (kNN) search, and for NLP, enabling the use of custom or third-party models directly in Elasticsearch.

In this blog post, we will explore how vector search and NLP work to enhance chatbot capabilities and demonstrate how Elasticsearch facilitates the process. Let's begin with a brief overview of vector search.

Vector search

Although humans can comprehend the meaning and context of written language, machines cannot do the same. This is where vectors come in. By converting text into vector representations (numerical representations of the meaning of the text), machines can overcome this limitation. Compared to a traditional search, instead of relying on keywords and lexical search based on frequencies, vectors enable the process of text data using operations defined for numerical values.

This allows vector search to locate data that shares similar concepts or contexts by using distances in the "embedding space" to represent similarity given a query vector. When the data is similar, the corresponding vectors will be alike.

Vector search is not only utilized in NLP applications, but it’s also used in various other domains where unstructured data is involved, including image and video processing.

In a chatbot flow, there can be several approaches to users' queries, and as a result, there are different ways to improve information retrieval for a better user experience. Since each alternative has its own set of advantages and possible disadvantages, it is essential to take into account the available data and resources, as well as the training time (when applicable) and expected accuracy. In the following section, we will cover these aspects for question-answering NLP models.

Question-answering

A question-answering (QA) model is a type of NLP model that is designed to answer questions asked in natural language. When users have questions that require inferring answers from multiple resources, without a pre-existing target answer available in the documents, generative QA models can be useful. However, these models can be computationally expensive and require large amounts of data for domain related training, which may make them less practical in some situations, even though this method can be particularly valuable to handle out-of-domain questions.

On the other hand, when users have questions on a specific topic, and the actual answer is present in the document, extractive QA models can be used. These models directly extract the answer from the source document, providing transparent and verifiable results, making them a more practical option for businesses or organizations that want to provide a simple and efficient way of answering questions.

The example below demonstrates the use of a pre-trained extractive QA model, available on Hugging Face and deployed into Elasticsearch, to extract answers from a given context:

Deploy trained models.

Add a model to an inference ingest pipeline.

There are various ways to handle user queries and retrieve information, and using multiple language models and data sources can be an effective alternative when dealing with unstructured data. To illustrate this, we have an example of the data processing of a chatbot employed to respond to queries with answers considering data extracted from selected documents.

Chatbot data processing: NLP and vector search

As shown above, the data processing for our chatbot can be divided into three parts:

• Vector processing: This part converts documents into vector representations.

• User input processing: This part extracts relevant information from the user query and performs semantic search and hybrid retrieval.

• Optimization: This part includes monitoring and is crucial for ensuring the chatbot's reliability, optimal performance, and great user experience.

Vector processing

For the processing part, the first step is to determine component parts of each document to then convert each element to a vector representation; these representations can be created for a wide range of data formats.

There are various methods that can be used to compute embeddings, including pre-trained models and libraries.

It's important to note that the effectiveness of search and retrieval on these representations depends on the existing data and the quality and relevance of the method used.

As the vectors are computed, they are stored in Elasticsearch with a dense_vector field type.

Chatbot user input processing

For the user part, after receiving a question, it's useful to extract all possible information from it before proceeding. This helps to understand the user's intention, and in this case, we are using a Named Entity Recognition model (NER) to assist with that. NER is the process of identifying and classifying named entities into predefined entity categories.

Although not a necessary step, by using structured data or the above or another NLP model result to categorize the user's query, we can restrict the kNN search using a filter. This helps to improve performance and accuracy by reducing the amount of data that needs to be processed.

Semantic search and hybrid retrieval

Since the prompt originates from user queries and the chatbot needs to process human language with its variability and ambiguity, semantic search is a great fit. In Elasticsearch, you can perform semantic search in a single step by passing the query string and the ID of the embedding model into a query_vector_builder object. This will vectorize the query and perform kNN search to retrieve top k matches that are closest in meaning to the query:

End-to-end example: How to deploy a text embedding model and use it for semantic search. Elasticsearch uses the Lucene implementation of the Okapi BM25, a sparse model , to rank text queries for relevance, while dense models are used for semantic search. To combine the strengths of both , vector matches and matches obtained from the text query, you can perform a hybrid retrieval :

Combining both sparse and dense models often yields the best results

Sparse models generally perform better on short queries and specific terminologies, while dense models leverage context and associations. If you want to learn more about how these methods compare and complement each other, here we benchmark BM25 against two dense models that have been specifically trained for retrieval.

The most relevant result can usually be the first answer given to the user, the_score is a number used to determine the relevance of the returned document.

Chatbot optimization

To help improve the user experience, performance, and reliability of your chatbot, in addition to applying hybrid scoring, you can incorporate the following approaches: Sentiment Analysis: To provide awareness of user comments and reactions as the dialog unfolds, you can incorporate a sentiment analysis model:

GPT's capabilities : As an alternative to enhance the overall experience, you can combine Elasticsearch's search relevance with OpenAI's GPT question-answering capabilities, utilizing the Chat Completion API to return to the user model-generated responses considering these top k documents as a context. Prompt: "answer this question <user_question> using only this document <top_search_result>"

Observability: Ensuring the performance of any chatbot is crucial, and monitoring is an essential component in achieving this. In addition to logs that capture chatbot interactions, it's important to track response time, latency, and other relevant chatbot metrics. By doing so, you can identify patterns, trends, and even detect anomalies.Elastic Observability tools enable you to collect and analyze this information.

Summary

This blog post covers what NLP and vector search are and delves into an example of a chatbot employed to respond to user queries by considering data extracted from the vector representation of documents.

As demonstrated, using NLP and vector search, chatbots are capable of performing complex tasks that go beyond structured, targeted data. This includes making recommendations and answering specific product or business-related queries using multiple data sources and formats as context, while also providing a personalized user experience.

Use cases range from providing customer service by assisting customers with their inquiries to helping developers with their queries, by providing step-by-step guidance, suggesting recommendations, or even automating tasks. Depending on the goal and existing data, other models and methods can also be utilized to achieve even better results and improve the overall user experience.

Here are some links on the topic that may be useful:

1. How to deploy natural language processing (NLP): Getting started

2. Overview of image similarity search in Elasticsearch

3. ChatGPT and Elasticsearch: OpenAI meets private data

4. Monitor OpenAI API and GPT models with OpenTelemetry and Elastic

5. 5 reasons IT leaders need vector search to improve search experiences

By incorporating NLP and native vector search in Elasticsearch, you can take advantage of its speed, scalability, and search capabilities to create highly efficient and effective chatbots capable of handling large amounts of data, whether structured or unstructured.

Ready to get started? Begin a free trial of Elastic Cloud.

In this blog post, we may have used or we may refer to third party generative AI tools, which are owned and operated by their respective owners. Elastic does not have any control over the third party tools and we have no responsibility or liability for their content, operation or use, nor for any loss or damage that may arise from your use of such tools. Please exercise caution when using AI tools with personal, sensitive or confidential information. Any data you submit may be used for AI training or other purposes. There is no guarantee that information you provide will be kept secure or confidential. You should familiarize yourself with the privacy practices and terms of use of any generative AI tools prior to use.

Elastic, Elasticsearch and associated marks are trademarks, logos or registered trademarks of Elasticsearch N.V. in the United States and other countries. All other company and product names are trademarks, logos or registered trademarks of their respective owners.