The idea came from a video editor here at Elastic. She was losing whole afternoons scrubbing through B-roll folders to find one specific visual.

Searching a footage library by description

Here's what the app does:

1. Select a folder of footage.

2. Search for clips with a visual query (drone shot over a coastline).

3. Search for similar clips using stored vectors.

4. Find similar chunks within the same clip.

5. Reveal the clip in your file browser, ready to drop on a timeline.

How the AI video search pipeline works

The pipeline is as follows:

1. A watcher polls the linked library folder (clips/ by default) every four seconds for new footage. 

2. PySceneDetect detects the scene boundaries in each clip. 

3. FFmpeg cuts the clip at those boundaries with a lossless stream copy and then transcodes each scene chunk into a small proxy. 

4. The proxy goes to jina-embeddings-v5-omni-small through the Jina API, which samples 32 frames and returns a 1,024-dimension vector. 

5. That vector is ingested into Elasticsearch in a dense_vector field with a hierarchical navigable small world (HNSW) index (an approximate nearest neighbor graph). At query time, the editor's text runs through the same model and we do a k-nearest neighbor (kNN) search against the stored chunk vectors.

The same model embeds both text and video, which is what makes Jina’s omni model so versatile. Video chunks and text queries are embedded into the same vector space, so a nearest-neighbor search effectively means find the footage that looks like what I described in text.

What you need to build AI video search

• Elasticsearch (Serverless or 8.x+ with dense vector support)

• A Jina API key

• PySceneDetect

• FFmpeg

• Python 3.9+

Downloading sample footage to search

If you don’t have any footage handy to search through, the repo includes two download scripts: one for short stock clips from the Pexels API, across categories like nature, urban, animals, and more. The other uses yt-dlp to grab longer documentary-style videos directly from YouTube.

We’re aiming to have a good mix between short and long videos to demonstrate how important the chunking strategy becomes. We’ll discuss that in the next part.

How to chunk video for embeddings

jina-embeddings-v5-omni-small samples 32 evenly spaced frames from whatever video you send it. For a 10-second clip, 32 frames is dense coverage; almost every moment gets captured. For a 10-minute documentary, those same 32 frames are spread so thin that entire shots can be missed. Here’s a diagram illustrating how the Jina model samples each video clip and the limitations with longer clips:

So before we embed anything, we chunk. The question is where to cut. Here's a diagram comparing three strategies:

1. Fixed-length chunking splits every N seconds. It’s simple and predictable, but the cuts can land mid-shot. You could end up with a chunk that’s half drone shot and half talking head, which can make your search results unusable. This isn’t the right fit for this use case.
2. Transcript-based chunking first runs speech to text over the clip to generate a transcript and then applies a text chunking technique to it, splitting at topic boundaries and mapping those back to timestamps in the video. This strategy is great for podcasts, talks, and educational content but not for B-roll since it usually has no dialogue.
3. Scene-based chunking splits at visual changes, like shot changes, transitions, and cuts. Each chunk is one specific visual, which is exactly what a video editor would search for. This is the best one for our use case.

The three strategies side by side:

Strategy

How it cuts

Best for

Weakness

Fixed-length

Every N seconds

Uniform content, predictable cost

Cuts land mid-shot, producing mixed chunks

Transcript-based

At topic boundaries in speech-to-text output

Podcasts, talks, educational video

Fails on B-roll with no dialogue

Scene-based

At visual cuts and transitions

B-roll and footage libraries

Depends on reliable shot detection

Detecting scene boundaries with PySceneDetect

To implement it, we use PySceneDetect to find the cut points:

AdaptiveDetector compares each frame change against a rolling average, keeping camera pans and handheld motion from registering as cuts. The minimum scene length defaults to 1.5 seconds, which, at an assumed 24 frames per second (FPS), works out to 36 frames, so rapid cuts can't produce sub-second slivers. If no boundaries are detected at all, the whole clip becomes a single chunk. Each detected scene is then cut into its own chunk file with a lossless FFmpeg stream copy (-c copy), so nothing is re-encoded before the proxy step.

Now those 32 sampled frames cover a few seconds and a specific visual scene instead of skimming across clips.

Why send a 640px proxy instead of the original file?

We don't send the original, full-sized files to the embedding API. A 4K ProRes chunk can run hundreds of megabytes, and the model can't use that resolution anyway. Its vision encoder produces roughly one token per 28x28-pixel block, and the model's default config caps each frame at 1,280 vision tokens, which is about 1 megapixel. Anything bigger gets downscaled to fit the budget before encoding, so a 4K frame arrives at the model with roughly an eighth of its pixels. Uploading full-resolution footage just wastes bandwidth and time on resolution that’s never used. For a deeper breakdown of how the model turns frames into patch tokens, see our architecture deep dive on jina-embeddings-v5-omni.

Instead, FFmpeg transcodes each scene chunk into a lightweight proxy: 640 px wide, audio stripped, aggressive compression.

Let’s go over two parts of the function that are easy to miss. The scale filter checks orientation gt(iw,ih), so landscape clips are capped at 640 px wide and portrait clips at 640 px tall instead of getting squashed. And max_seconds puts a limit on each proxy at three seconds. This works because each chunk is one visual, so the first few seconds are already a good representative and 32 frames sampled across three seconds is already dense coverage. The function returns {"video": <base64>}, which is the input shape that the Jina API expects.

The quality loss doesn’t matter much, up to a point. Embeddings care about what’s in the frame, not whether the footage is high enough quality for final delivery. But if you compress too aggressively and visual details start disappearing, the model may no longer be able to reliably tell what’s in each frame. The goal is to shrink the file without losing the visual information that matters. Proxies come out at a few hundred kilobytes instead of hundreds of megabytes, so ingesting a folder of footage takes minutes instead of hours.

Each proxy then goes to the Jina API as a base64 string, and the API returns the 1,024-dimension vector for that chunk:

The task parameter is important here. Chunks are indexed with task="retrieval.passage", and at search time the query text is embedded with task="retrieval.query". The model produces asymmetric embeddings tuned for retrieval, one side for documents and one for queries. 

We also pin dimensions to 1024 and set normalized to true. Normalization isn't required for cosine similarity, since cosine measures the angle between vectors and length never affects the score. What it does is make every vector unit-length, and for unit vectors the dot product equals the cosine, so the engine can skip the magnitude math and compare vectors with a plain dot product.

With Elasticsearch, you can take advantage of normalized vectors by mapping the field with similarity: "dot_product" instead of cosine and skip the normalization overhead at query time. The repo keeps cosine for safety, since it works even if a non-normalized vector ever slips in. 

In the repo, this request lives in a small client class (backend/lib/embed_jina.py) that also retries with exponential backoff on rate limits and transient server errors. After getting the vectors back, we can ingest these into Elasticsearch.

Ingesting video embeddings into Elasticsearch

First, we define an explicit mapping. Elasticsearch can dynamically infer field types at index time, meaning the first document that lands can determine how a field is mapped. Here, we want to be deliberate: IDs should be keywords; in-video timestamps, like start_sec and end_sec, should be floats; and, most importantly, the embedding field needs to be configured as a 1,024-dimensional dense_vector with cosine similarity and an HNSW index.

Here’s the mapping for the index:

A few things worth noting:

• dims is 1024 to match the model output.

• similarity is cosine because the Jina embeddings are trained for cosine distance and, as we mentioned earlier, dot_product would score identically on normalized vectors and slightly faster, but we keep cosine for safety.

• index: True with HNSW builds an approximate nearest neighbor graph at index time, so queries don't brute-force every vector.

• start_sec/end_sec are what let us jump the editor straight to the right moment in the clip instead of just the right file.

• The remaining fields are plain metadata: path is stored but not searchable ("index": False) so the app can play the chunk file back, duration and strategy describe how the chunk was made, and tags/transcript leave room for keyword and transcript search later.

That means one document per scene chunk, not per clip. A 10-minute documentary might become 80 documents. The point is that each one is independently findable. Note that none of the video itself goes into Elasticsearch. A document is just the vector plus a few fields of metadata, including a path pointing at the chunk file on disk. The footage stays where it is, and Elasticsearch acts purely as the index that tells us which file, and which seconds of it, match the query. 

This app runs against a local folder, so path is a filesystem path. If you were building this as a hosted service, the clips would live in object storage, like Amazon S3 or Google Cloud Storage instead, and that field would hold a pointer to the object in storage, but the architecture stays the same.

Running a vector search over video chunks

At query time, the editor's text goes through the same model. Warm sunset light becomes a 1,024-dimension vector in the same space as the video chunks, and we ask Elasticsearch for its nearest neighbors:

k is how many neighbors come back; num_candidates is how many each shard considers before ranking. Higher candidates means better recall at slightly higher latency. We also exclude the embedding field from the response because 1,000 floats per hit is a lot of payload for a value that the UI never reads. We fetch more than we display (50 for a nine-card grid) because of what happens next.

Find similar clips works the same way, except the query vector is a stored chunk embedding instead of embedded text. No second model call is needed; the vector's already in the index.

Deduplicating results from the same clip

One problem I ran into was that there were too many chunks from the same clip filling the entire results grid. Search for mountains and a single 10-minute nature documentary can match with a dozen chunks, pushing every other clip out of the top results. This is technically correct but practically useless to an editor who wants options.

The fix is to keep the best-scoring chunk from each clip as the representative for each result card, and let users expand a card to see the other matching chunks from the same clip:

This is why we over-fetch at query time. Pull 50 hits, collapse to one card per clip, and then show nine. exclude_id covers the find similar case, where the seed chunk would otherwise come back as its own top hit, and _chunk_payload simply trims each hit down to the fields that the UI needs. The more_matches count becomes a "+4 more from this clip" badge in the UI. Expanding that badge doesn't call the embedding API again; the app caches recent query vectors and reruns the kNN search filtered to that clip with a term filter on clip_id.

Conclusion: What scene chunking costs you at scale

That's the full setup; it starts with watching a folder, cutting at scene boundaries, embedding proxies, indexing vectors, and finally, searching with plain language. The editor types what visuals they need, and the footage comes to them with timestamps and a way to quickly get ahold of the actual clip.

Every vector in this index is 1,024 float32 values, about 4 KB each. It’s not much, even for a demo. Our 70-clip folder produces a few hundred chunks, a couple of megabytes of vectors. But scene chunking multiplies fast. At roughly three seconds per scene, one hour of footage is already ~1,200 vectors, so a modest 1,000-hour archive is over a million vectors and several gigabytes of floats, and a serious footage library in the hundreds of millions of vectors is a terabyte-level index. Since HNSW wants those vectors in memory to search fast, it can get expensive quickly. In Part 2 of this series, we'll look at quantization versus dimensionality reduction, two very different ways to shrink the index, and what each one costs you in recall.

While AI indices will be included in future Stack releases, today we recommend using Serverless.

How it works: AI Index, Kibana Workflows, and the query-ki skill

Building context in this walkthrough has three moving parts:

1. An AI Index, where KIs live. It's a regular Elasticsearch index or data stream with a specific naming convention triggering component templates to configure the right mappings automatically.

2. Kibana Workflows, which read from your data sources, run an LLM to structure content into KIs, and write those KIs into the AI Index.

3. A query-ki skill, a skill that queries KIs directly from the AI Index using ES|QL, and that a chat agent can call as a tool.

Prerequisites

This tutorial assumes you have:

1. An Elasticsearch Serverless project. You can sign up for a trial if you don't have one.

2. An API key to access your Elasticsearch project. 

Create sample indices for agent routing

First, we’ll need some sources. Sources can be data that already exists in your Elasticsearch indices, or external data accessed via connectors or ES|QL data sources. 

For this blog, we’ll create some indices with example data. We’ll start with an example using three datasets: BEIR/fiqa (financial), beir-nfcorpus (biomedical/nutrition), and beir-scifact (scientific fact-checking). Each index is populated with its own _meta.description. 

Here are the mappings we define for these indices:

Then, using the above convenience scripts, load a handful of documents into each index with the _bulk API.

Now imagine an agent with a question and the indices we’ve just created. The agent has no idea which one is relevant at the start. Without pre-computed context, it either performs exploratory lookups (mappings, test searches) to figure out which source to use, or searches all three and hopes the merged results contain something useful. Either approach costs tokens, and if you multiply that inefficiency across every query an agent makes, it adds up.

Create your AI Index

Before generating any KIs, you need an index to store them. We call this an AI Index.

The naming convention is what triggers automatic configuration. Any index whose name starts with ai-index-idx- is a regular index; ai-index-ds- is a data stream. You’ll want to choose data streams for observability use cases, time series data, and when recency is important. Conversely, standard indices are a good choice for static data that will exist for a long while, where recency is not as much of a concern, and may need to occasionally be updated on demand. This naming convention is required for AI indices. 

When Elasticsearch sees the ai-index- prefixes, it automatically applies component templates that configure the right mappings and settings.

Creating an AI Index is a single call:

To see exactly what the component templates applied, inspect the mappings:

The response shows the fields every AI Index gets out of the box:

title, description, and content are each a text field with a .semantic sub-field of type semantic_text, supporting hybrid retrieval.

Data stream indices (ai-index-ds-*) additionally carry a default 90-day data retention policy. This blog uses a standard index (ai-index-idx-*).

Index Metadata as a Knowledge Indicator

The target use case for this example is how the query-index-metadata-ki skill can route an agent to the correct Elasticsearch index, even when index or field names are vague. This reduces mistakes from choosing the wrong index or formulating queries based on incomplete schema exploration.

Since we're creating KIs for our own indices, we can give the LLM a head start: annotate index mappings with human-written _meta.description content. The workflow generates better KIs with more context to work from.

To address this, we'll manually create a Kibana Workflow that profiles each index and writes routing KIs into the AI Index. The workflow chains four steps:

Step

Type

What it does

get_mapping

elasticsearch.request

Read the mapping, including _meta.description and per-field descriptions.

sample_docs

elasticsearch.search

Pull a few real documents so the profile reflects actual value shapes.

profile_index

ai.agent

Generate a structured index profile as structured output.

sink_index_ki

elasticsearch.bulk

Write the profile into the AI Index as a KI.

Paste the following YAML into the Workflows editor:

Let's walk through what this workflow does. We loop over three specified indices with a foreach loop. For each:

1. get_mapping fetches the Elasticsearch index mappings, including any _meta.description annotations we added earlier.

2. sample_docs pulls 3 real documents. Concrete examples give the LLM much better signal than schema alone.

3. profile_index calls ai.agent with the index name, mappings, and sample documents. The LLM returns structured output describing the index's purpose, key fields, and an example ES|QL query showing how to use it.

4. sink_index_ki writes the result into the AI Index as a KI of type index_metadata_entry, keyed on the index name so re-runs are idempotent.

A few things to point out: 

• This workflow hard-codes a specific set of indices. In practice, you could derive the list from an index pattern or a dynamic source. 

• The foreach loop also runs iterations sequentially, which is fine for this guide but slow in production because each iteration involves an LLM call. For scale, use workflow.executeAsync or native parallel support. The cheat sheet has tips on both.

• In the profile_index step, the agent prompt is the special sauce. This is what shapes the accuracy and usefulness of the KIs. 

• Using ai.prompt can improve workflow efficiency (and cost) if you don’t need to load other tools. 

• Cost can be controlled in multiple ways. Richer prompts and structured output often result in higher token utilization, and of course the model you choose significantly impacts total costs. The Elastic Inference Service (EIS) can be a great playground to test different models against the profile_index’s ai.agent step to compare how different models stack up against each other when generating KIs. 

Query your AI Index to verify Knowledge Indicators

Once the beir-index-profile-ki workflow runs, query the AI Index directly in the Discover tab using the following ES|QL query to confirm what got written:

This will result in the following output: 

Build a portable skill to retrieve AI agent context

Retrieval is a critical component in an AI index. A KI is a document in the AI Index, and finding one is a single ES|QL query. We package that query as a small, portable skill so any agent can call it, regardless of the harness it runs in.

We write the skill as a SKILL.md: a YAML header with a name and description, followed by markdown instructions. This is the same Agent Skills format that many harnesses, including Claude Code, LangChain's Deep Agents, and others, load directly. 

The harness reads the header content up front, and only pulls in the full instructions when a question matches the description. The one thing the skill asks of the harness is a way to run ES|QL against Elasticsearch.

Here is a sample query-index-metadata-ki skill:

Let’s break down what the skill is doing: 

• We’re defining index-metadata-entry as a KI type/use case.

• We’re performing a hybrid ES|QL search on our AI indices, filtering by the appropriate type using RRF as the default method to fuse results.

• The KI results will directly ground the agent’s answer when determining what indices are relevant to the query.

Because the skill is just instructions plus a query, it travels wherever your agent does. You can point the same file at a Kibana Workflow agent, Claude Code, LangChain Deep Agents, or any other harness without changing a line of it.

Connect your AI Index to an agent harness 

We want to demonstrate how you can use AI indices to query your data with any harness. For these examples, we’ll use LangChain Deep Agents and an OpenAI-compatible key, but any other agent harness can be easily substituted in, including Elastic Agent Builder.

First, let’s create a baseline to see how an agent will perform without using KIs:

Here’s a modified example that could run the same agent, but now with the ability to search AI indices to return KIs: 

This agent will always query the KI indices to get the answer.

How much do Knowledge Indicators reduce agent token usage?

Since we’re using agents, the results of these scripts are non-deterministic. However, when I ran these results against the query Is there scientific evidence that vitamin D supplementation prevents cancer?, both agents led to the same conclusion, but they took different paths to get there: 

Baseline (No AI Index)

With AI Index

Total tool calls

12

8

read_file calls

0

2

get_mapping calls

3

0

esql_query calls

9

6 

Total indices queried

2 (bounced between beir-scifact and beir-nfcorpus)

1 (beir-nfcorpus)

Tokens consumed

167,763

92,711

Latency

39.58s

36.15s

Answer

Grounded, correct

Grounded, correct

The KI answers were both grounded and correct, but an interesting datapoint is the fact that the overall tool usage and token utilization was smaller when using KIs (latency was roughly equivalent). Here’s how both paths went, side by side: 

Run the full AI Index pipeline in Serverless

This walkthrough offered a deep dive into the build-it-yourself version of AI indices and KIs. In production, you wouldn't hand-write these workflows; a setup agent would generate them, and a feedback loop would refine KIs from the agent's own traces. But the primitives are exactly what you just used: extract KIs with a workflow, store them in an AI Index, and retrieve them with a skill.

Managing context is key to a relevant and efficient agentic search system, and AI indices are a way to manage this context with the full power of the Elastic stack. Try it out in Serverless and let us know what you think in our Discuss forums or the #stack-kibana channel in our Community Slack! 

Vector search is a critical foundation for large language model (LLM) applications, retrieval augmented generation (RAG), and other AI workloads. In this benchmark, Elasticsearch achieved up to 7x higher throughput than Qdrant at comparable recall on the same storage topology. Elasticsearch as a vector database offers strong vector search performance even when network-attached persistent storage remains on the query path.

The difference reflects how the two systems interact with disk. Elasticsearch DiskBBQ is designed to keep vector search efficient when persistent storage remains on the query path, using a compact quantized representation and limiting costly access to full precision vectors during search. In this setup, Qdrant relies on a graph-based search path with rescoring against original vectors stored on disk. On network-attached persistent storage, that random access cost becomes much more significant, which is why the performance gap widens as recall increases. This benchmark therefore focuses specifically on network-attached persistent storage, a common deployment model in managed cloud and Kubernetes environments.

The key pattern in the latency curve is not only the size of the gap but also its shape. Elasticsearch latency remains comparatively flat as recall increases, suggesting that higher recall doesn’t require a dramatic increase in expensive storage activity. Qdrant’s latency rises sharply as hnsw_ef increases, which is consistent with broader candidate exploration leading to more rescoring work against original vectors on disk.

Full results table

The table below shows the full parameter sweep for both Elasticsearch and Qdrant. Because the two engines expose different tuning controls for vector search, the results are reported using each engine’s full parameter key rather than attempting a one-to-one mapping between settings.

A few notes on the metrics:

• ParamKey: The complete parameter setting used for a given run.

• Recall: Recall@100 against a ground-truth top-100 result set for the benchmark queries. Values range from 0 to 1, and higher is better.

• Latency_Avg: The average end-to-end latency per query measured from the benchmarking client across the full run, in milliseconds. Lower is better.

• Latency_P95: The 95th percentile query latency, in milliseconds, showing the upper range of typical slow queries. Lower is better.

• Throughput: The average number of queries processed per second across the full run. Higher is better.

Engine

ParamKey

Recall

Latency_Avg

Latency_P95

Throughput

qdrant

hnsw_ef=50, oversampling=1, size=100

0.8694

315.7849

503.4754

12.629

elasticsearch

k=100, oversample=1, size=100, visit_percentage=1

0.8789

135.0802

218.494

29.343

elasticsearch

k=100, oversample=1, size=100, visit_percentage=1.5

0.9123

127.8286

195.2318

31.1107

qdrant

hnsw_ef=100, oversampling=1, size=100

0.9287

895.9933

1213.0448

4.4493

elasticsearch

k=100, oversample=1, size=100, visit_percentage=2

0.9317

124.846

183.6314

31.8225

elasticsearch

k=100, oversample=1, size=100, visit_percentage=2.5

0.9444

123.517

180.4831

32.1883

qdrant

hnsw_ef=150, oversampling=1, size=100

0.9518

884.7236

1195.2603

4.5066

elasticsearch

k=100, oversample=1, size=100, visit_percentage=3

0.9532

123.276

183.8379

32.2364

elasticsearch

k=100, oversample=1, size=100, visit_percentage=3.5

0.9599

122.5559

184.2858

32.4469

qdrant

hnsw_ef=200, oversampling=1, size=100

0.964

883.2114

1188.6597

4.5143

elasticsearch

k=100, oversample=1, size=100, visit_percentage=4

0.965

122.7946

184.9058

32.3635

elasticsearch

k=100, oversample=1, size=100, visit_percentage=4.5

0.9689

122.7062

182.9559

32.3976

elasticsearch

k=100, oversample=1, size=100, visit_percentage=5

0.9722

122.5761

187.3536

32.4221

qdrant

hnsw_ef=256, oversampling=1, size=100

0.9722

881.9643

1185.4948

4.5192

elasticsearch

k=100, oversample=1, size=100, visit_percentage=5.5

0.9747

122.5609

184.5128

32.4176

Each row pairs the closest measured Elasticsearch and Qdrant configurations in the sweep by achieved recall.

Matched comparisons at similar recall

To make the comparison fair, speedup is calculated only between configurations that achieve similar recall. This avoids comparing settings that trade off accuracy very differently.

Recall band

Elasticsearch recall

Elasticsearch Latency_Avg

Elasticsearch throughput

Qdrant recall

Qdrant Latency_Avg

Qdrant throughput

Throughput speedup

~0.87

0.8789

135.0802

29.343

0.8694

315.7849

12.629

2.32x

~0.93

0.9317

124.846

31.8225

0.9287

895.9933

4.4493

7.15x

~0.95

0.9532

123.276

32.2364

0.9518

884.7236

4.5066

7.15x

~0.96

0.9599

122.5559

32.4469

0.964

883.2114

4.5143

7.19x

~0.97

0.9722

122.5761

32.4221

0.9722

881.9643

4.5192

7.17x

This matched-recall view is the clearest expression of the underlying systems difference. At similar recall levels, Elasticsearch delivers both lower latency and much higher throughput, and the gap widens as recall rises. The recall-throughput pattern matters because higher recall in this benchmark requires broader search. DiskBBQ absorbs that increase with relatively little additional cost, while Qdrant’s graph plus rescoring path becomes much more constrained by random access to original vectors on persistent storage.

Benchmark methodology

Jingra, the benchmarking tool used for these tests, was originally written in Python and has since been rebuilt as a Java project. For these tests, Jingra runs in a Kubernetes pod within the same cluster as the engine being measured. This helps reduce external network variability and keeps the test environment consistent across runs. For each run, Jingra executed the query set at a fixed client concurrency, recorded end-to-end client-side latency and throughput, and computed recall against a precomputed ground-truth top-100 set.

This benchmark was intentionally run on network-attached persistent storage rather than local NVMe. For the published results, the storage used the baseline performance allocation for a 200 GiB GCP Hyperdisk Balanced volume, with no explicit IOPS or throughput provisioning. We chose this topology on purpose because it’s a relevant cloud deployment model and because it keeps storage efficiency materially on the query path.

Qdrant often performs better on local NVMe, so deployments using local NVMe should expect different results than the ones shown here. This benchmark specifically tests network-attached persistent storage because that’s a common managed-cloud deployment model and because it makes storage-path efficiency visible in end-to-end query performance.

Because Elasticsearch and Qdrant expose different query parameters for controlling vector search behavior, there’s no clean one-to-one mapping between their tuning settings. Instead of comparing equivalent parameter values directly, we use recall as the primary point of comparison. The matched comparisons below therefore pair configurations that achieve similar recall, rather than configurations with superficially similar parameter values.

Recall cannot be known in advance for a given parameter setting, so we sweep across a range of search configurations for each engine and then compare results at similar recall levels. In the published results, oversampling was fixed at 1 for both engines so that recall was primarily tuned via search breadth rather than rescoring expansion.

How does Elasticsearch configure vector search?

• query_vector: The input vector used for similarity search. Elasticsearch compares this vector against the stored vectors in the field.

• k: The number of nearest neighbors to retrieve.

• visit_percentage: Controls how much of the DiskBBQ, Elasticsearch’s disk optimized vector index, is explored during the approximate search phase. Higher values usually improve recall but increase latency.

• oversample: Controls how many extra candidate vectors are passed into rescoring relative to k. Higher values can improve recall, but usually at additional cost.

• size: The number of hits returned in the final response.

• _source: false: Disables returning the document _source field, reducing response size and avoiding extra retrieval overhead during benchmarking.

Example

Params

We keep k = size = 100 so the search request is aligned with the benchmark target: returning the top 100 results. To improve recall, we tune visit_percentage rather than inflating the final result count, while keeping oversample = 1 fixed across runs.

How does Qdrant configure vector search?

• query_vector / vector: The input vector used for similarity search. Qdrant compares this vector against the stored vectors in the collection.

• size / limit: The number of nearest neighbor results returned in the response.

• with_payload: false: Disables returning payload fields, reducing response size and avoiding additional retrieval overhead during benchmarking.

• with_vector: false: Disables returning stored vectors in the response, again reducing response size and keeping the benchmark focused on search performance.

• hnsw_ef: Controls the number of candidates explored during HNSW search. Higher values usually improve recall but increase latency. Like visit_percentage in Elasticsearch, it affects search breadth, but the two controls are engine-specific and not directly equivalent.

• quantization.rescore: true: Enables rescoring of the candidate set using the original vectors after quantized search.

• oversampling: Controls how many extra candidates are considered during rescoring relative to the final result count. Higher values can improve recall, but usually at additional cost.

Example

Params

We keep size = 100 so that each request is aligned with the evaluation target, in this case top 100 retrieval. Recall is then tuned by sweeping hnsw_ef, which controls how many candidates are explored during search. Higher hnsw_ef values generally improve recall but also increase latency and reduce throughput. We keep oversampling = 1 fixed across runs so that the main tuning variable is the search breadth rather than the rescoring expansion.

Cluster setup and DiskBBQ configuration

We ran the benchmark on GCP using three n4-standard-8 nodes, with each pod allocated 7 vCPUs and 26 GB of RAM, and using 200 GiB GCP Hyperdisk Balanced volumes at baseline performance allocation. The corpus contains 21 million vectors, (see dataset section below for more details and download links), which account for about 60.1 GiB of raw float vector data. With 2-bit quantization, the vector payload drops to roughly 3.8 to 4.0 GB. However, the full index footprint is much larger once graph and other index structures are included. That means the workload remains meaningfully sensitive to network-attached storage performance, especially because exact vector values still need to be read from disk during rescoring.

We chose this node size intentionally to keep the benchmark in a regime where network-attached persistent storage remains on the query path rather than allowing the full working set to remain comfortably memory-resident. Each system was therefore configured using the best-performing setup we identified for this workload within the tuning scope described in this post. In Elasticsearch, this meant bbq_disk. In Qdrant, the original vectors were stored on disk, while the 2-bit quantized representation used for approximate search was kept in RAM with always_ram: true. Because the two systems expose different search strategies and tuning controls, we compare them at matched recall rather than trying to map parameters one to one.

Elasticsearch was configured to use DiskBBQ, its disk-optimized approach for approximate nearest neighbor vector search, with 2-bit quantization. DiskBBQ uses aggressive quantization to keep the searchable index compact and then rescores with the original vectors to preserve accuracy. This helps maintain strong recall while keeping disk-based search efficient.

bbq_disk is an Elasticsearch Enterprise feature. We used it here because the goal of this benchmark was to compare the strongest disk-oriented vector search configuration available in each engine for this workload, rather than licensing tiers or default features.

We didn’t include bbq_hnsw in this comparison because the benchmark was specifically designed to evaluate disk-oriented vector search under a disk-sensitive workload.

This storage topology matters because Qdrant’s rescore step reads the original float32 vectors from disk with random access on each query. On local NVMe, those reads are much faster, and Qdrant correspondingly performs better. On network-attached persistent storage, the results are consistent with that random-read rescore path becoming a more important bottleneck. Qdrant latency rises sharply as hnsw_ef increases, while Elasticsearch remains comparatively flat across the same recall progression.

We chose 2-bit quantization because Qdrant couldn’t reach the target recall range with 1-bit binary quantization. Since the two systems expose different disk-oriented vector search strategies, we tuned each one to the strongest configuration available within its current feature set.

Both systems were configured with three shards distributed across the three nodes and with two total copies of each shard in the cluster. In Elasticsearch, number_of_shards: 3 and number_of_replicas: 1 means one primary plus one replica, for two total copies. In Qdrant, shard_number: 3 and replication_factor: 2 also means two total copies, since Qdrant’s replication factor refers to the total number of copies rather than the number of additional replicas. So although the field names differ, the effective replication level was the same in both systems.

Setting

Elasticsearch

Qdrant

Shards

number_of_shards: 3

shard_number: 3

Copies

number_of_replicas: 1 (1 primary + 1 replica = 2 total)

replication_factor: 2 (2 total)

Elasticsearch mapping

Qdrant mapping

Dataset

For this benchmark, we used the kenhktsui/wiki_dpr_e5 dataset from Hugging Face, a large-scale Wikipedia passage retrieval dataset designed for dense vector search. The corpus contains 21 million embedded passages, each represented as a 768-dimensional float32 vector, or 3,072 bytes per vector. That corresponds to about 60.1 GiB of raw vector data, before accounting for additional fields and file format overhead in the source dataset. The downloadable data.parquet file is larger at 85.2 GB for that reason.

We chose this dataset because it reflects a common production pattern in LLM, RAG, and retrieval systems: searching a large corpus of semantically embedded text while balancing recall, latency, and throughput. At 21 million vectors and roughly 60 GiB of raw vector data, it’s large enough to make disk-based vector search a relevant operating mode to evaluate.

Both engines used 2-bit quantization, reducing each vector from 3,072 bytes to 192 bytes, a 16x reduction that brings the quantized vector corpus to around 4 GB. In Qdrant, that quantized representation was kept in RAM for search, while the original vectors remained on disk. Even so, the workload remained meaningfully sensitive to network-attached storage performance because rescoring still required access to the original vectors on disk.

You can download the dataset and query files from the links below:

• data.parquet

• queries.parquet

Jingra and recreating the benchmark

For this benchmark, we used Jingra v0.2.3 with the configurations described es-9.4-vs-qd-1.18-vector-search. Jingra handled data loading, query execution, parameter sweeps, and metric collection for both Elasticsearch and Qdrant, making the benchmark repeatable and easier to compare.

To reproduce the experiment, you need the published dataset, query set, engine configurations, and comparable cluster hardware. With those in place, Jingra can rerun the benchmark and generate similar recall, latency, and throughput measurements shown in this post.

Conclusion

At comparable recall levels, Elasticsearch DiskBBQ consistently delivered faster vector search than Qdrant in this benchmark, with higher throughput and lower latency across the recall range we tested. These results are especially notable because the comparison was made on network-attached persistent storage, where efficient storage-aware vector search becomes critical. Elasticsearch as a vector database allows organizations to achieve high recall with lower latency and higher throughput on slower persistent storage.

Just as importantly, this benchmark highlights the value of comparing engines at matched recall rather than by nominal parameter settings. Elasticsearch and Qdrant expose different controls, so the fairest comparison isn’t parameter to parameter but outcome to outcome. Across the recall range tested here, Elasticsearch maintained a clear advantage in both latency and throughput.

If you want to reproduce the experiment yourself, we’re publishing the dataset and query set used in this benchmark so others can validate the results and build on them.

Further reading:

• Introducing a new vector storage format: DiskBBQ

• Elasticsearch BBQ vs TurboQuant

Prerequisites

• Elastic Cloud deployment running Elasticsearch 9.3+ or start a free trial

• Workflows enabled (Advanced Settings)

• Python 3.9+

• A Mistral API key

How LLM query routing works in this system

We're going to build a two-stage system: a router that decides how to answer, and an answering model that produces the response.

The router looks at the query and the metadata of the top search hits, like scores, categories, and complexity labels. From that, it picks one of two strategies: Answer directly from the top FAQ article, or synthesize across multiple articles with citations. That decision can be made from structured signals alone, so a small, fast model handles it.

The answering step varies. A single-article answer is bounded work that a small model does well and returns quickly. A multisource synthesis with citations benefits from a more capable model, and the extra time is worth it. Matching each query to the model that fits keeps simple answers fast and complex answers good.

Why use a small model for routing instead of the large model?

Because the router runs on every query, including the simple ones. A slow router makes every answer slow, even the ones a small model could have produced in a fraction of the time.

The key design choice is that the routing step only sees metadata, not full documents. A query like "my OTG isn't heating evenly" only needs to know that the top hits are in the "Product Troubleshooting - Appliances" category with issue_complexity: medium, not the full conversation transcripts. This keeps the classification prompt tiny (a few hundred tokens) and cheap. The full article content is only loaded in the response step once.

Set up AI connectors

We use two AI connectors for the workflow:

Connector

Model

Type

Role

Mistral Small

mistral-small-latest

Custom (OpenAI-compatible)

Classify query complexity from metadata, answer simple FAQ-style questions

Anthropic Claude Sonnet 4.6

Claude Sonnet

Elastic Managed LLM

Synthesize complex answers from multiple articles, with citations

Both connectors are billed per million tokens, with the smaller model costing significantly less. Routing simple queries to it saves money on top of the latency win. To learn more about Elastic Managed LLM, see this documentation.

The Claude Sonnet connector is already available as an Elastic Managed large language model (LLM). We only need to create a custom connector for Mistral using the .gen-ai connector type, which supports any OpenAI-compatible API. You can also create it through the Kibana UI.

All the setup code in this article is available in the companion notebook. You can run each section there as you follow along.

The connector ID is auto-generated by Kibana. We let the platform handle this instead of trying to set it manually.

Once created, the connector appears in the Connectors UI:

Load and index the dataset

We use the e-commerce-customer-support-qa dataset from Hugging Face. It contains 1,000 real customer support interactions from an ecommerce platform (BrownBox) with customer questions, agent solutions, issue categories, complexity levels, and customer sentiment.

The index mapping uses semantic_text with the .jina-embeddings-v5-text-small model from Elastic Inference Service. This field handles semantic search end-to-end: embedding generation, chunking, and querying. We use copy_to to aggregate the conversation and QA summary into a single searchable field:

Defining the query routing workflow in Elastic Workflows YAML

The routing workflow has four steps: semantic search, metadata-only classification, conditional branching, and a model-appropriate response step.

We use Elastic Workflows to encapsulate this routing logic. Workflows let us:

1. Expose the triaging as a tool in Elastic Agent Builder, so a conversational agent can call it.

2. Trigger it directly via manual execution, schedules, or alerts.

This flexibility means the same logic serves both programmatic and conversational interfaces without duplicating code.

Workflows are defined in YAML and configured directly in the Workflow UI (Elasticsearch > Workflows > Create a New Workflow). Each step can query Elasticsearch, call Kibana APIs, or prompt an LLM.

Here’s the complete workflow definition:

The workflow has four key parts:

1. search_es uses elasticsearch.search with a semantic query to find the five most relevant articles.

2. classify_querysends the customer query plus only metadata from the search results to Mistral Small. The prompt includes scores, categories, complexity labels, and product categories. This keeps the classification step cheap, preventing the use of large amounts of tokens.

3. route_by_complexity uses an if step to branch based on the classifier's output.

4. The response step depends on the route. For simple queries, Mistral Small gets the top FAQ article and rephrases it. For complex queries, Claude Sonnet gets all five articles and synthesizes a detailed response with citations. This is the only step where full document content is loaded.

Using the workflow as a tool in Agent Builder

Beyond the default triggers (manual, schedule, alerts), workflows can also be exposed as tools in Agent Builder. This adds a conversational layer where users interact through a chat interface, and the agent decides when to call the workflow.

We use the Agent Builder APIs to create the tool and the agent. After creating the workflow in the Kibana UI, copy its ID and use it to register the workflow as a tool:

Then create an agent that uses the tool:

The agent is now available in the Agent Builder UI in Kibana. You can also create the agent and its tools directly through the Agent Builder UI.

Once created, the agent appears in the Agent Builder UI with the workflow tool assigned:

Testing simple vs. complex query routing

Simple query

The workflow searches the knowledge base, finds a direct match in the FAQ articles about order tracking, classifies it as simple, and routes to Mistral Small. The response is concise and drawn directly from the matched article: instructions for using the "My Orders" section or the tracking number from the confirmation email.

Complex query

This query involves three distinct issues (damaged product, duplicate charge, address update) across different support categories. The workflow classifies it as complex and routes to Claude Sonnet, which synthesizes information from multiple knowledge base articles, addresses each issue separately, cites the relevant articles, and provides clear resolution steps for each.

Conclusion

Routing LLM queries by complexity in Elasticsearch reduces latency and cost for simple queries without sacrificing quality on complex ones. The small model answers FAQ-style queries in a fraction of the time the larger model would take, and the larger model is reserved for the queries that actually benefit from its capabilities. Cost savings come along for the ride: Simple queries routed to the smaller model are cheaper, too.

The pattern that makes this work is searching the knowledge base before routing. Without that context, the router is guessing based on surface-level cues. With it, the structure of the search results, like scores, categories, and complexity labels, tells the router whether the answer lives in a single article or needs synthesis across several. That's the actual signal for how to handle the query.

Elastic Workflows makes this possible without writing orchestration code. The entire routing logic lives in YAML inside Kibana, using native steps for search, LLM prompts, and conditional branching. Combined with Agent Builder, the same workflow serves programmatic triggers and conversational interfaces.

Next steps

• Try the notebook with the complete implementation.

• Add LLM monitoring with OpenRouter to track cost per routing tier.

• Explore Elastic Workflows for other automation patterns.

• Learn more about Agent Builder and how to expose workflows as conversational tools.

• Read about building AI agentic workflows with Elastic Agent Builder.

As we saw in the previous post, the consistency provided by function calling is not just a nice optimization; it’s essential. Once we removed structural errors from the evaluation loop, results on standard scenarios (such as those in the tier 4 dataset) improved dramatically.

Yet there’s an obvious question left to answer:

Does this approach still work when things get genuinely messy?

Real-world entity resolution rarely fails because of simple cases. It fails when names cross languages, cultures, writing systems, time periods, and organizational boundaries. It fails when people are referenced by titles instead of names, when companies change names, when transliterations aren’t consistent, and when context (not spelling) is the only thing tying a mention to a real-world entity.

So, for the final post in this series, we put the system through what we called the ultimate challenge.

What makes this the ultimate challenge?

In earlier evaluations, we tested the system using increasingly complex datasets. By the time we reached tier 4, discussed in the previous post, we were already dealing with a mix of nicknames, titles, multilingual names, and semantic references. Those tests showed that the architecture itself was sound, but that reliability issues, especially malformed JSON, were suppressing recall.

With function calling in place, we finally had a stable foundation. That gave us the opportunity to ask a more interesting question:

Can one unified pipeline handle many different kinds of entity resolution problems at once?

The ultimate challenge dataset was designed to push precisely on that dimension.

Instead of focusing on a single difficulty (like nicknames or transliteration), this dataset combines 50+ distinct challenge types, including:

• Cultural naming conventions.

• Title-based references.

• Business relationships and historical name changes.

• Multilingual and cross-script mentions.

• Compound challenges that mix several of the above.

Crucially, this isn’t about optimizing for any one narrow use case. It’s about testing whether the design pattern holds up when the rules change from entity to entity.

The dataset at a glance

The ultimate challenge dataset consists of:

• 50 entities, spanning people, organizations, and institutions.

• ~60 articles, with varying structure and linguistic complexity.

• 51 distinct challenge categories, grouped broadly into:

  • Cultural naming conventions.

  • Titles and professional context.

  • Business and organizational relationships.

  • Multilingual and transliteration challenges.

  • Combined and edge‑case scenarios.

Earlier in the series, we saw that using generative AI (GenAI) to create datasets can be a mixed blessing. Without it, assembling sufficiently large and diverse test data would be extremely difficult. But left unchecked, the model has a tendency to make things too easy.

On an early generation pass, for example, we discovered that the model had included phrases like “the Russian president” as explicit aliases for Vladimir Putin. That might seem reasonable today, but it defeats the purpose of testing contextual resolution. What happens if the article is discussing Russia in the 1990s? The system should infer the correct entity from context, not rely on a hard-coded alias.

For that reason, this dataset was deliberately designed so that shortcuts don’t work. Aliases are not explicitly listed when the system is expected to infer meaning. Descriptive phrases are not prelinked to entities. Correct matches often depend on article-level context, not just local text.

Important note: Although we demonstrate the system’s capabilities across diverse scenarios, this is still an educational prototype. Production systems handling real-world sanctioned-entity monitoring would require additional validation, compliance checks, audit trails, and specialized handling for sensitive use cases.

Why these scenarios are hard

Back in the first post in this series, we introduced a simple but ambiguous example: “The new Swift update is here!” The challenge is that “Swift” can resolve to multiple real-world entities, depending on context. That example captures a broader truth: Natural language is inherently ambiguous.

Entity resolution, therefore, is not just a string-matching problem. Humans routinely rely on shared knowledge, cultural norms, and situational context to resolve references, and we rarely even notice we’re doing it.

Consider a few common cases:

• A title like “the president” is meaningless without geopolitical and temporal context.

• A company name may refer to a parent, a subsidiary, or a former brand depending on when the article was written.

• A person’s name may appear in different orders, scripts, or transliterations, depending on language and culture.

• The same phrase can legitimately refer to different entities in different contexts, and the system must be able to reject matches just as confidently as it accepts them.

There is no single rule set that handles all of this cleanly. That’s why this prototype separates concerns so aggressively:

• Elasticsearch narrows the candidate space efficiently and transparently.

• The LLM is used only where judgment is required and is forced to explain itself.

• Retrieval and reasoning remain distinct steps.

This separation becomes even more important as the diversity of challenge types increases.

How the system handles diversity without special cases

One of the most interesting outcomes of this evaluation is what didn’t change:

• We did not add special logic for Japanese names.

• We did not add custom rules for Arabic patronymics.

• We did not add hard-coded mappings for historical company names.

Instead, the system relied on the same core ingredients introduced earlier in the series:

• Context-enriched entities indexed for semantic search.

• Hybrid retrieval (exact, alias, and semantic) in Elasticsearch.

• A small, well-defined set of candidate matches.

• LLM judgment constrained by function calling and minimal schemas.

This suggests that the system’s flexibility comes from representation and architecture, not from an ever-growing collection of rules.

When the system succeeds, it’s because the right candidates are retrieved and the LLM has enough context to explain why a reference does (or does not) map to a specific entity.

Results: How did it perform?

On the ultimate challenge dataset, the system produced the following overall results:

• Precision: ~91%

• Recall: ~86%

• F1 Score: ~89%

• LLM acceptance rate: ~72%

Performance across challenge types

Breaking down results by challenge type reveals strengths and limitations:

Strongest performance (100% F1 score) was observed in areas such as:

• Cross-script matching (Cyrillic, Korean, Chinese business entities).

• Hebrew scenarios (patronymics, professional titles, religious titles, transliteration).

• Business hierarchies (aerospace, diversified manufacturing, multidivision corporations).

• Professional titles (academic, military, political, religious).

• Combined Japanese scenarios involving multiple writing systems.

Strong performance (80–99% F1 score) included:

• International political figures (98%).

• Historical name changes (90%).

• Complex business hierarchies (89%).

• Japanese company names (93%).

• Cross-script transliteration (86%).

• Arabic patronymics (86%).

More challenging areas included:

• Advanced transliteration (Chinese, Korean): 0% F1.

• Certain Japanese scenarios (honorifics, name order, writing system variation): ~67% F1.

• Some Arabic scenarios (company names, institutional references): ~40% F1.

What’s important here is why the system struggled in these cases. The failures were not due to the overall approach breaking down, but to limitations in specific components, most notably the dense vector model used for semantic search in certain multilingual scenarios.

Because retrieval and judgment are cleanly separated, improving performance does not require rewriting the system. Swapping in a more capable multilingual embedding model, enriching entity context, or refining retrieval strategies would improve results across these categories without changing the core architecture.

From an architectural standpoint, that’s the real success metric.

What this tells us about the design

Looking back across the series, a few patterns stand out:

• Preparation matters more than clever matching. Enriching entities with context up front dramatically reduces ambiguity later.

• LLMs are most valuable as judges, not retrievers. Asking them to explain why a match makes sense is far more powerful than asking them to search.

• Reliability enables accuracy. Function calling didn’t just clean up JSON; it unlocked recall that was already latent in the retrieval step.

• Generalization beats specialization. A small number of well-chosen abstractions handled dozens of challenge types without custom logic.

This is why the prototype is intentionally Elasticsearch-native and intentionally conservative in how it uses LLMs. The goal isn’t to replace search; it’s to make search explainable in situations where meaning matters.

Final thoughts

The ultimate challenge wasn’t about chasing perfect metrics; it was about answering a more fundamental question:

Can a transparent, search-first, LLM-assisted architecture handle real-world entity ambiguity without collapsing into rules or black boxes?

For this educational prototype, the answer is yes, with clear caveats around production hardening, compliance, monitoring, and data quality. If you’re building systems that need to justify why an entity match was made, this pattern is worth serious consideration. I hope this series has shown that entity resolution doesn’t have to be mysterious. With the right separation of concerns, it becomes something you can reason about, measure, and improve.

This work also suggests a broader architectural pattern. What emerges is a slight but important evolution of classic retrieval augmented generation (RAG). Instead of allowing retrieval to feed generation directly, we introduce an explicit evaluation step. The LLM is first used to judge and sanity-check retrieved candidates, and only those approved results are allowed to augment generation. You can think of this as Generation-Augmented Retrieval-Augmented Generation with Evaluation, or GARAGE, because who doesn’t love a good acronym.

What other use cases could benefit from this pattern? Systems that require trust, transparency, and defensible reasoning are natural candidates. Future work in this area should prove as compelling as the results we’ve seen here, and I’m excited to see where the community takes it next.

Next steps: Try it yourself

Want to see the ultimate challenge in action? Check out the Ultimate Challenge notebook for a complete walkthrough, with real implementations, detailed explanations, and hands-on examples.

The complete entity resolution pipeline demonstrates the core concepts and architecture needed for production use. You can use it as a foundation to build systems that monitor news articles, track entity mentions, and answer questions about which entities appear in which articles, all while retaining transparency and explainability.

The root of this problem was our somewhat naive choice to use prompt-based JSON generation in which the LLM generates JSON responses in text format. If we asked the LLM to judge more than a couple of matches at a time, the generated JSON was often ill-formed. To mitigate this, we were forced to reduce the processing batch size, which simply won't scale in a production system.

So the prompt-based JSON generation helped validate our approach to entity resolution, but we need a more systematic and reliable method. OpenAI function calling provides a better path by guaranteeing structure and type safety while reducing errors and costs. We chose OpenAI's functions for the educational prototype, but other LLM providers typically provide similar functionality (for example, Claude tools).

Note: While we discuss production challenges here, this is still an educational prototype demonstrating optimization techniques. Real production systems would need additional considerations, like monitoring, alerting, fallback strategies, and comprehensive error handling.

Key concepts: Function calling, schema design, and cost benefits

What is function calling? Function calling is OpenAI's structured output API. With it, we can define schemas for LLM responses, so we always know exactly what we're going to get. By enforcing the JSON format rather than trying to define it in the LLM prompt, we should be able to eliminate parsing errors.

Why is it better than prompt-based JSON? LLMs generate nondeterministic output. One hopes that they'll at least generate content that contains the correct response, but the presentation of that response is unpredictable. With a chatbot, this is often not a problem, but our prototype is trying to programmatically process the output. Computer programs demand consistency, so when the LLM generates what we expect, everything is fine, but as soon as it goes off script, so to speak, the code errors out. We could try to account for the different possibilities, but it would be very difficult to catch everything. We could try to enforce more consistent behavior by adding something like "Always return parsable JSON". We tried this exact technique in the prototype's prompt, but we've seen that prompt-based JSON still goes off the rails pretty quickly, particularly if we try to process a batch of matches.

Function calling makes the LLM generation controllable and predictable, exactly what we need for entity resolution. To aid in the definition of the functions, we’ll also follow minimal schema design principles.

What are minimal schema design principles? Minimal schema design means defining only the fields you need, using simple types, and avoiding nested structures when possible. This reduces token usage (smaller schemas mean fewer tokens), improves reliability (simpler schemas are easier for the LLM to follow), and lowers costs (fewer tokens mean lower API costs).

What are the cost and reliability benefits? Since fewer errors means match processing is much more likely to succeed, even with large batch sizes, we don't have to retry judging matches. The elimination of retries reduces costs by reducing token usage, but using minimal schemas also keeps our token count down. This all leads to a less expensive and more reliable approach that’s much more suitable to use in production.

We need to check one more thing, though. While matches may be getting processed without error, are the errorless results actually correct? How does this new approach compare to the promising results we saw with the prompt-based approach?

Real-world results: Side-by-side comparison

As we did in the previous blog, we ran the function calling approach against the tier 4 dataset, which consists of 206 expected matches across 69 articles. The results demonstrate a dramatic improvement:

Metric

Prompt-based

Function calling

Improvement

Error rate

30.2%

0.0%

100% elimination

Precision

83.8%

90.3%

+6.5pp

Recall

62.6%

90.8%

+28.2pp

F1 score

71.7%

90.6%

+18.9pp

Acceptance rate

44.8%

60.2%

+15.4pp

True positives

129

187

+45.0%

False negatives

77

19

-75.3%

Error elimination: The key differentiator

The most striking difference is the complete elimination of JSON parsing errors. This resulted in a modest precision improvement and a far more dramatic recall improvement. The precision metric captures how often the matches the system accepts were expected in the golden document. So the prototype was decent at judging matches correctly in the prompt-based approach, but function calling does that even better.

Conversely, recall tells us how many of the expected matches were found. When a batch of matches comes back with malformed JSON, the system loses all of those matches. It's likely that Elasticsearch sends many of these matches for judgment, but we lose those matches if judgment fails. The significant recall improvement shows that this hypothesis is correct. Elasticsearch identifies the potential matches and function calling verifies which of those matches are correct.

Note: It’s expected that Elasticsearch will find some incorrect matches because we look at the top two or three results from hybrid search. Most of the time, hybrid search returns the correct match as the top result, but having the LLM judge the top few hits ensures that we see how the LLM handles incorrect matches. If we move from the educational prototype to a production system, we’ll likely tune the Elasticsearch queries more carefully so that we only send promising matches to the LLM, further optimizing our LLM costs.

What's next: The ultimate challenge

Now that we've optimized our LLM integration with function calling, we have a complete entity resolution pipeline with improved reliability and cost efficiency. However, can it handle the ultimate challenge? In the next post, we'll explore how the system handles diverse entity resolution scenarios across 50 different challenge types, including cultural naming conventions, business relationships, titles, and multilingual variations.

Try it yourself

Want to see function calling optimization in action? Check out the Function Calling Optimization notebook for a complete walkthrough with real implementations, detailed explanations, and hands-on examples. The notebook shows you exactly how to use function calling for structured output, compare it with prompt-based JSON, and analyze cost and reliability benefits.

Remember: This is an educational prototype designed to teach optimization concepts. When building production systems, consider additional factors, like multi-provider support, advanced caching strategies, monitoring and alerting, comprehensive error handling, and compliance requirements that aren't covered in this learning-focused prototype.

But if you work with languages like Hebrew, Arabic, German, or Polish, you know that standard rule-based analyzers often fail. They either under-analyze (missing relevant matches) or overanalyze (returning garbage results).

For years, we’ve had to rely on complex dictionaries and fragile regex rules. Today, we can do better. By replacing rule-based logic with neural models for text analysis (small, efficient language models that understand context), we can drastically improve search quality.

Here’s how to solve the morphology challenge by using the Elasticsearch inference API and a custom model service.

The problem: Why rules fail

Most standard analyzers are context-free. They look at one word at a time and apply a static set of rules.

• Algorithmic analyzers (like Snowball) strip suffixes based on patterns.

• Dictionary analyzers (like Hunspell) look up words in a list.

This approach breaks down when the structure of a word (its root and affixes) changes based on the sentence it lives in.

1. The semitic ambiguity (roots versus prefixes)

Semitic languages, like Hebrew and Arabic, are built on root systems and often attach prepositions (such as, in, to, or from) directly to the word. This creates ambiguous tokens that rule-based systems cannot solve.

• Word: בצל (B-Tz-L).

• Context A: “The soup tastes better with onion (batzal).”

• Context B: “We sat in the shadow (ba-tzel) of the tree.”

In Context A, בצל is a noun (onion). In Context B, it’s a preposition ב (in) attached to the noun צל (shadow).

A standard analyzer is forced to guess. If it aggressively strips the ב prefix, it turns "onion" into "shadow." If it’s conservative and leaves it alone, a user searching for "shadow" (tzel) will fail to find documents containing "in the shadow" (batzel). Neural models solve this by reading the sentence to determine whether the ב is part of the root or a separate preposition.

2. The compound problem (German, Dutch, and more)

Languages like German, Dutch, Swedish, and Finnish concatenate nouns without spaces to form new concepts. This results in a theoretically infinite vocabulary. To search effectively, you must split (decompound) these words.

• Word: Wachstube.

• Split A: Wach (guard) + Stube (room) = guardroom.

• Split B: Wachs (wax) + Tube (tube) = wax tube.

A dictionary-based decompounder acts blindly. If both “Wach” and “Wachs” are in its dictionary, it might pick the wrong split, polluting your index with irrelevant tokens.

To see this problem in English: A naive algorithm might split “carpet” into “car” + “pet.” Without understanding meaning, rules fail.

The solution: “Neural analyzers” (neural models for text analysis)

We don’t need to abandon the inverted index. We just need to feed it better tokens.

Instead of a regex rule, we use a neural model (like BERT or T5) to perform the analysis. Because these models are trained on massive datasets, they understand context. They look at the surrounding words to decide whether בצל means "onion" or "in shadow" or if Wachstube belongs in a military or cosmetic context.

Architecture: The inference sidecar

We can integrate these Python-based models directly into the Elasticsearch ingestion pipeline using the inference API.

The pattern:

1. External model service: A simple Python service (for example, FastAPI) hosts the model.

2. Elasticsearch inference API: Defines this service as a custom model within Elasticsearch.

3. Ingest pipeline: Sends text to the inference processor, which calls your Python service.

4. Index mapping: Create a whitespace target field for the analyzed text.

5. Indexing: The service returns the cleaned text, which Elasticsearch stores in the target field.

6. Search: Queries are analyzed via the inference API before matching.

Implementation guide

Let’s build this for Hebrew (using DictaBERT) and German (using CompoundPiece).

To follow along, you’ll need:

• Python 3.10+.

• Elasticsearch 8.9.x+.

Install the Python dependencies:

Step 1: External model service

To connect Elasticsearch to our neural model, we need a simple API service that:

1. Receives text from the Elasticsearch inference API.

2. Passes it through the neural model.

3. Returns analyzed text in a format Elasticsearch understands.

This service interfaces Elasticsearch with the neural model. At ingest time, the Elasticsearch pipeline calls this API to analyze and store document fields; at search time, the application calls it to process the user's query. You can deploy this on any infrastructure, including EC2, Lambda, or SageMaker.

The code below loads both models at startup and exposes /analyze/hebrew and /analyze/german endpoints:

Save the code above to a file (for example, analyzer_service.py), and run:

Wait for “Models loaded successfully!” (takes ~30–60 seconds for models to download on first run).

Test locally:

Expected output:

Step 2: Configure Elasticsearch inference API

We’ll use the custom inference endpoint. This allows us to define exactly how Elasticsearch talks to our Python endpoint.

Note: Use response.json_parser to extract the content from our normalized JSON structure. You do not need to stick with the OpenAI output format. We’re using it here for consistency reasons, since we’re using the completion task type, which is text to text.

Exposing your local service

For testing, we’ll use ngrok to expose the local Python service to the internet. This allows any Elasticsearch deployment (self-managed, Elastic Cloud, or Elastic Cloud Serverless) to reach your service.

Install and run ngrok:

Expose your local service:

ngrok will display a forwarding URL like:

Forwarding https://abc123.ngrok.io -> http://localhost:8000

Copy the HTTPS URL. You’ll use this in the Elasticsearch configuration.

Configure the inference endpoint

Replace https://abc123.ngrok.io with your actual ngrok URL.

Note: ngrok is used here for fast testing and development. The free tier has request limits, and URLs change on restart. For production, deploy your service to a persistent infrastructure.

For production (with API Gateway)

In production, deploy your Python service to a secure, persistent endpoint (such as AWS API Gateway + Lambda, EC2, ECS, or any cloud provider). Use secret_parameters to securely store API keys:

Step 3: Ingest pipeline

Create a pipeline that passes the raw text field to our model and stores the result in a new field.

Step 4: Index mapping

This is the most critical step. The output from our neural model is already analyzed. We do not want a standard analyzer to mess it up again. We use the whitespace analyzer to simply tokenize the text we received.

Step 5: Indexing

Option A: Single document.

Option B: Reindex existing data.

If you have existing data in another index, reindex it through the pipeline:

Option C: Set pipeline as default for index.

Make all future documents automatically use the pipeline:

Then index normally (no ?pipeline= needed):

Step 6: Search

Search using a neural analyzer in Elasticsearch is a two-step process, so analyze the query first using the inference API, and then search with the result:

A. Analyze the query.

B. Search with the result.

In production, wrap these two calls in your application code for a seamless experience.

Available models

The architecture above works for any language. You simply swap the Python model and adjust the post-processing of the output. Here are verified models for common complex languages:

• Hebrew: Context-aware lemmatization. Handles prefix ambiguity (ב, ה, ל, and more) dicta-il/dictabert-lex.

• German: Generative decompounding. Supports 56 languages, including Dutch, Swedish, Finnish, and Turkish. benjamin/compoundpiece.

• Arabic: BERT-based disambiguation and lemmatization for Modern Standard Arabic. CAMeL Tools.

• Polish: Case-sensitive lemmatization for Polish inflections. amu-cai/polemma-large.

Conclusion

You don’t need to choose between the precision of lexical search and the intelligence of AI. By moving the “smart” part of the process into the analysis phase using the inference API, you fix the root cause of poor search relevance in complex languages.

The tools are here. The models are open-source. The pipelines are configurable. It’s time to teach our search engines to read.

Code

All code snippets from this article are available at https://github.com/noamschwartz/neural-text-analyzer.

References:

• https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-inference-put-custom

• https://www.elastic.co/docs/manage-data/ingest/transform-enrich/ingest-pipelines

• https://ngrok.com

• https://huggingface.co/dicta-il/dictabert-lex

• https://huggingface.co/benjamin/compoundpiece

• https://arxiv.org/pdf/2305.14214

Our goal was to transition to an automated, adaptive approach capable of handling both log parsing (field extraction) and log partitioning (source identification). We hypothesized that Large Language Models (LLMs), with their inherent understanding of code syntax and semantic patterns, could automate these tasks with minimal human intervention.

We are happy to announce that this feature is already available in Streams!

Dataset description

We chose a Loghubcollection of logs for PoC purposes. For our investigation, we selected representative samples from the following key areas:

• Distributed systems: We used the HDFS (Hadoop Distributed File System) and Spark datasets. These contain a mix of info, debug, and error messages typical of big data platforms.

• Server & web applications: Logs from Apache web servers and OpenSSH provided a valuable source of access, error, and security-relevant events. These are critical for monitoring web traffic and detecting potential threats.

• Operating systems: We included logs from Linux and Windows. These datasets represent the common, semi-structured system-level events that operations teams encounter daily.

• Mobile systems: To ensure our model could handle logs from mobile environments, we included the Android dataset. These logs are often verbose and capture a wide range of application and system-level activities on mobile devices.

• Supercomputers: To test performance on high-performance computing (HPC) environments, we incorporated the BGL (Blue Gene/L) dataset, which features highly structured logs with specific domain terminology.

A key advantage of the Loghub collection is that the logs are largely unsanitized and unlabeled, mirroring a noisy live production environment with microservice architecture.

Log examples:

In addition, we created a Kubernetes cluster with a typical web application + database set up to mine extra logs in the most common domain.

Example of common log fields: timestamp, log level (INFO, WARN, ERROR), source, message.

Few-shot log parsing with an LLM

Our first set of experiments focused on a fundamental question: Can an LLM reliably identify key fields and generate consistent parsing rules to extract them?

We asked a model to analyse raw log samples and generate log parsing rules in regular expression (regex) and Grok formats. Our results showed that this approach has a lot of potential, but also significant implementation challenges.

High confidence & context awareness

Initial results were promising. The LLM demonstrated a strong ability to generate parsing rules that matched the provided few-shot examples with high confidence. Besides simple pattern matching, the model showed a capacity for log understanding —it could correctly identify and name the log source (e.g., health tracking app, Nginx web app, Mongo database).

The "Goldilocks" dilemma of input samples

Our experiments quickly surfaced a significant lack of robustness because of extreme sensitivity to the input sample. The model's performance fluctuates wildly based on the specific log examples included in the prompt. We observed a log similarity problem where the log sample needs to include just diverse enough logs:

• Too homogeneous (overfitting): If the input logs are too similar, the LLM tends to overspecify. It treats variable data—such as specific Java class names in a stack trace—as static parts of the template. This results in brittle rules that cover a tiny ratio of logs and extract unusable fields.

• Too heterogeneous (confusion): Conversely, if the sample contains significant formatting variance—or worse, "trash logs" like progress bars, memory tables, or ASCII art—the model struggles to find a common denominator. It often resorts to generating complex, broken regexes or lazily over-generalizing the entire line into a single message blob field.

The context window constraint

We also encountered a context window bottleneck. When input logs were long, heterogeneous, or rich in extractable fields, the model's output often deteriorated, becoming "messy" or too long to fit into the output context window. Naturally, chunking helps in this case. By splitting logs using character-based and entity-based delimiters, we could help the model focus on extracting the main fields without being overwhelmed by noise.

The consistency & standardization gap

Even when the model successfully generated rules, we noted slight inconsistencies:

• Service naming variations: The model proposes different names for the same entity (e.g., labeling the source as "Spark," "Apache Spark," and "Spark Log Analytics" in different runs).

• Field naming variations: Field names lacked standardization (e.g., id vs. service.id vs. device.id). We normalized names using a standardized Elastic field naming.

• Resolution variance: The resolution of the field extraction varied depending on how similar the input logs were to one another.

Log format fingerprint

To address the challenge of log similarity, we introduce a high-performance heuristic: log format fingerprint (LFF).

Instead of feeding raw, noisy logs directly into an LLM, we first apply a deterministic transformation to reveal the underlying structure of each message. This pre-processing step abstracts away variable data, generating a simplified "fingerprint" that allows us to group related logs.

The mapping logic is simple to ensure speed and consistency:

1. Digit abstraction: Any sequence of digits (0-9) is replaced by a single ‘0’.

2. Text abstraction: Any sequence of alphabetical characters with whitespace is replaced by a single ‘a’.

3. Whitespace normalization: All sequences of whitespace (spaces, tabs, newlines) are collapsed into a single space.

4. Symbol preservation: Punctuation and special characters (e.g., :, [, ], /) are preserved, as they are often the strongest indicators of log structure.

We introduce the log mapping approach. The basic mapping patterns include the following:

• Digits 0-9 of any length -> to ‘0.’

• Text (alphabetical characters with spaces) of any length -> to ‘a’.

• White spaces, tabs, and new lines -> to a single space.

Let's look at an example of how this mapping allows us to transform the logs.

As a result, we obtain the following log masks:

Notice the fingerprints of the first two logs. Despite different timestamps, source classes, and message content, their prefixes (0/0/0 0:0:0 a a.a:) are identical. This structural alignment allows us to automatically bucket these logs into the same cluster.

The third log, however, produces a completely divergent fingerprint (0-0-0...). This allows us to algorithmically separate it from the first group before we ever invoke an LLM.

Bonus part: Instant implementation with ES|QL

It’s as easy as passing this query in Discover.

Query breakdown:

FROM loghub: Targets our index containing the raw log data.

EVAL pattern = …: The core mapping logic. We chain REPLACE functions to perform the abstraction (e.g., digits to '0', text to 'a', etc.) and save the result in a “pattern” field.

STATS [column1 =] expression1, … BY SUBSTRING(pattern, 0, 15):

This is a clustering step. We group logs that share the first 15 characters of their pattern and create aggregated fields such as total log count per group, list of log datasources, pattern prefix, 3 log examples

SORT total_count DESC | LIMIT 100 : Surfaces the top 100 most frequent log patterns

The query results on LogHub are displayed below:

As demonstrated in the visualization, this “LLM-free” approach partitions logs with high accuracy. It successfully clustered 10 out of 16 data sources (based on LogHub labels) completely (>90%) and achieved majority clustering in 13 out of 16 sources (>60%) —all without requiring additional cleaning, preprocessing, or fine-tuning.

Log format fingerprint offers a pragmatic, high-impact alternative and addition to sophisticated ML solutions like log pattern analysis. It provides immediate insights into log relationships and effectively manages large log clusters.

• Versatility as a primitive

Thanks to ES|QL implementation, LFF serves both as a standalone tool for fast data diagnostics/visualisations, and as a building block in log analysis pipelines for high-volume use cases.

• Flexibility

LFF is easy to customize and extend to capture specific patterns, i.e. hexadecimal numbers and IP addresses.

• Deterministic stability

Unlike ML-based clustering algorithms, LFF logic is straightforward and deterministic. New incoming logs do not retroactively affect existing log clusters.

• Performance and mMemory

It requires minimal memory, no training or GPU making it ideal for real-time high-throughput environments.

Combining log format fingerprint with an LLM

To validate the proposed hybrid architecture, each experiment contained a random 20% subset of the logs from each data source. This constraint simulates a real-world production environment where logs are processed in batches rather than as a monolithic historical dump.

The objective was to demonstrate that LFF acts as an effective compression layer. We aimed to prove that high-coverage parsing rules could be generated from small, curated samples and successfully generalized to the entire dataset.

Execution pipeline

We implemented a multi-stage pipeline that filters, clusters, and applies stratified sampling to the data before it reaches the LLM.

1. Two-stage hierarchical clustering

• Subclasses (exact match): Logs are aggregated by identical fingerprints. Every log in one subclass shares the exact same format structure.

• Outlier cleaning. We discard any subclasses that represent less than 5% of the total log volume. This ensures the LLM focuses on the dominant signal and won’t be sidetracked by noise or malformed logs.

• Metaclasses (prefix match): Remaining subclasses are grouped into Metaclasses by the first N characters of the format fingerprint match. This grouping strategy effectively splits lexically similar formats under a single umbrella.We chose N=5 for Log parsing and N=15 for Log partitioning when data sources are unknown.

2. Stratified sampling. Once the hierarchical tree is built, we construct the log sample for the LLM. The strategic goal is to maximize variance coverage while minimizing token usage.

• We select representative logs from each valid subclass within the broader metaclass.

• To manage an edge case of too numerous subclasses, we apply random down-sampling to fit the target window size.

3. Rule generation Finally, we prompt the LLM to generate a regex parsing rule that fits all logs in the provided sample for each Metaclass. For our PoC, we used the GPT-4o mini model.

Experimental results & observations

We achieved 94% parsing accuracy and 91% partitioning accuracy on the Loghub dataset.

The confusion matrix above illustrates log partitioning results. The vertical axis represents the actual data sources, and the horizontal axis represents the predicted data sources. The heatmap intensity corresponds to log volume, with lighter tiles indicating a higher count. The diagonal alignment demonstrates the model's high fidelity in source attribution, with minimal scattering.

Our performance benchmarks insights:

• Optimal baseline: a context window of 30–40 log samples per category proved to be the "sweet spot," consistently producing robust parsing with both Regex and Grok patterns.

• Input minimisation: we pushed the input size to 10 logs per category for Regex patterns and observed only 2% drop in parsing performance, confirming that diversity-based sampling is more critical than raw volume.

This article will show you how to build an AI agent for HR using GPT-OSS and Elastic Agent Builder. The agent can answer your questions without sending data to OpenAI, Anthropic, or any external service.

We’ll use LM Studio to serve GPT-OSS locally and connect it to Elastic Agent Builder.

By the end of this article, you’ll have a custom AI agent that can answer natural language questions about your employee data while maintaining full control over your information and model.

Prerequisites

For this article, you need:

• Elastic Cloud hosted 9.2, serverless or local deployment

• Machine with 32GB RAM recommended (minimum 16GB for GPT-OSS 20B)

• LM Studio installed

• Docker Desktop Installed

Why use GPT-OSS?

With a local LLM you have the control to deploy it in your own infrastructure and fine-tune it to fit your own needs. All this while maintaining control over the data that you share with the model, and of course, you don’t have to pay a license fee to an external provider.

OpenAI released GPT-OSS on August 5, 2025, as part of their commitment to the open model ecosystem.

The 20B parameter model offers:

• Tool use capabilities

• Efficient inference

• OpenAI SDK compatible

• Compatible with agentic workflows

Benchmark comparison:

Solution architecture

The architecture runs entirely on your local machine. Elastic (running in Docker) communicates directly with your local LLM through LM Studio, and the Elastic Agent Builder uses this connection to create custom AI agents that can query your employee data.

For more details, refer to this documentation.

Building an AI agent for HR: Steps

We’ll divide the implementation into 5 steps:

1. Configure LM studio with a local model

2. Deploy Local Elastic with Docker

3. Create the OpenAI connector in Elastic

4. Upload employee data to Elasticsearch

5. Build and test your AI Agent

Step 1: Configure LM Studio with GPT-OSS 20B

LM Studio is a user-friendly application that allows you to run large language models locally on your computer. It provides an OpenAI-compatible API server, making it easy to integrate with tools like Elastic without a complex setup process. For more details, refer to the LM Studio Docs.

First, download and install LM Studio from the official website. Once installed, open the application.

In the LM Studio interface:

1. Go to the search tab and search for “GPT-OSS”

2. Select the openai/gpt-oss-20b from OpenAI

3. Click download

The size of this model should be approximately 12.10GB. The download may take a few minutes, depending on your internet connection.

Once the model is downloaded:

1. Go to the local server tab

2. Select the openai/gpt-oss-20b

3. Use the default port 1234

4. On the right panel, go to Load and set the Context Length to 40K or higher

5. Click start server

You should see this if the server is running.

Step 2: Deploy Local Elastic with Docker

Now we’ll set up Elasticsearch and Kibana locally using Docker. Elastic provides a convenient script that handles the entire setup process. For more details refer to the official documentation.

Run the start-local script

Execute the following command in your terminal:

This script will:

• Download and configure Elasticsearch and Kibana

• Start both services using Docker Compose

• Automatically activate a 30-day Platinum trial license

Expected output

Just wait for the following message and save the password and API key shown; you’ll need them to access Kibana:

Access Kibana

Open your browser and navigate to:

Log in using the credentials obtained in the terminal output.

Enable Agent Builder

Once logged in to Kibana, navigate to Management > AI > Agent Builder and activate the Agent Builder.

Step 3: Create the OpenAI connector in Elastic

Now we’ll configure Elastic to use your local LLM.

Access Connectors

1. In Kibana

2. Go to Project Settings > Management

3. Under Alerts and Insights, select Connectors

4. Click Create Connector

Configure the connector

Select OpenAI from the list of connectors. LM Studio uses the OpenAI SDK, making it compatible.

Fill in the fields with these values:

• Connector name: LM Studio - GPT-OSS 20B

• Select an OpenAI provider: Other (OpenAI Compatible Service)

• URL: http://host.docker.internal:1234/v1/chat/completions

• Default model: openai/gpt-oss-20b

• API Key: testkey-123 (any text works, because LM Studio Server doesn't require authentication.)

To finish the configuration, click Save & test.

Important: Toggle ON the “Enable native function calling”; this is required for the Agent Builder to work properly. If you don’t enable this, you’ll get a No tool calls found in the response error.

Test the connection

Elastic should automatically test the connection. If everything is configured correctly, you’ll see a success message like this:

Response:

Step 4: Upload employee data to Elasticsearch

Now we’ll upload the HR employee dataset to demonstrate how the agent works with sensitive data. I generated a fictional dataset with this structure.

Dataset structure

Create the index with mappings

First, create the index with proper mappings. Note that we’re using semantic_text fields for some key fields; this enables semantic search capabilities for our index.

Index with Bulk API

Copy and paste the dataset into your Dev Tools in Kibana and execute it:

Verify the data

Run a query to verify:

Step 5: Build and test your AI agent

With everything configured, it’s time to build a custom AI agent using Elastic Agent Builder. For more details refer to the Elastic documentation.

Add the connector

Before we can create our new agent, we have to set our Agent builder to use our custom connector called LM Studio - GPT-OSS 20B because the default one is the Elastic Managed LLM. For that, we need to go to Project Setting > Management > GenAI Settings; now we select the one we created and click Save.

Access Agent Builder

1. Go to Agents

2. Click on Create a new agent

Configure the agent

To create a new agent, the required fields are the Agent ID, Display Name, and Display Instructions.

But there are more customization options, like the Custom Instructions that guide how your agent is going to behave and interact with your tools, similar to a system prompt, but for our custom agent. Labels help organize your agents, avatar color, and avatar symbol.

The ones that I chose for our agent based on the dataset are: Agent ID: hr_assistant

Custom instructions:

Labels: Human Resources and GPT-OSS

Display name: HR Analytics Assistant

Display description:

With all the data in there, we can click on Save our new agent.

Test the agent

Now you can ask natural language questions about your employee data, and GPT-OSS 20B will understand the intent and generate an appropriate response.

Prompt:

Answer:

The Agent process was:

1. Understand your question using the GPT-OSS connector

2. Generate the appropriate Elasticsearch query (using the built-in tools or custom ES|QL)

3. Retrieve matching employee records

4. Present results in natural language with proper formatting

Unlike traditional lexical search, the agent powered by GPT-OSS understands intent and context, making it easier to find information without knowing exact field names or query syntax. For more details on the agent's thinking process, refer to this article.

Conclusion

In this article, we built a custom AI agent using Elastic’s Agent Builder to connect to the OpenAI GPT-OSS model running locally. By deploying both Elastic and the LLM on your local machine, this architecture allows you to leverage generative AI capabilities while maintaining full control over your data, all without sending information to external services.

We used GPT-OSS 20B as an experiment, but the officially recommended models for Elastic Agent Builder are referenced here. If you need more advanced reasoning capabilities, there's also the 120B parameter variant that performs better for complex scenarios, though it requires a higher-spec machine to run locally. For more details, refer to the official OpenAI documentation.

At its core, this solution combines the power of Elasticsearch's search capabilities with Amazon's serverless AI infrastructure. Here's how it works:

• Your natural language questions are processed through the MCP server that is deployed on Amazon Bedrock AgentCore Runtime

• The MCP server translates these questions into precise Elasticsearch queries

• Results are returned in human-readable format, making your data instantly accessible

• All of this happens in a secure, scalable environment that's production-ready, deployed on Amazon Bedrock AgentCore Runtime

In this blog post, we'll explore how to:

• Deploy Elastic's MCP server on Amazon Bedrock AgentCore Runtime

• Transform local MCP prototypes into production-ready solutions

• How to Build Conversational Interfaces on top of Elasticsearch

• Implement secure, scalable AI agent architectures

Background

Model Context Protocol (MCP)

MCP is an open protocol that revolutionizes how businesses interact with their data through AI. Unlike traditional Retrieval-Augmented Generation (RAG) systems that simply retrieve documents, MCP enables AI agents to dynamically construct and execute complex tasks in real-time, mirroring the flexibility of human problem-solving

In practice, this means a business analyst can ask a series of increasingly specific questions about market trends, and the MCP-powered system will intelligently select and combine the appropriate data sources and analytical tools to provide comprehensive answers, while still maintaining context, allowing for follow-up questions without repetition.

For instance, when analyzing a product launch, the AI might integrate data from sales reports, customer feedback, and social media sentiment, orchestrating multiple tools simultaneously to provide a holistic view, thus enabling businesses to uncover deeper insights and make informed decisions, all through natural language interactions with their data systems.

Agents

Agents are AI-powered software applications that can think, plan, and act to achieve specific goals with minimal human supervision. They use foundation models (advanced AI models) to understand and complete complex tasks.

There are two types of AI agents.

Knowledge AI agents

These agents focus on enterprise knowledge. They gather context from company data — documents, logs, dashboards, communications, and customer records — and use that information to complete business tasks.

Example: A knowledge AI agent that can search across contracts, policies, and past tickets to help a customer support representative instantly resolve an issue.

General AI agents

These agents go further. They can understand goals and autonomously execute tasks on behalf of a user in broader, cross-domain workflows.

Example: A general AI agent that books travel, manages schedules, and negotiates with other systems to complete a user’s request end-to-end.

Elastic’s role in agentic AI

For knowledge AI agents: Elastic enables secure access to enterprise data, retrieves relevant context, and grounds responses in facts.

For general AI agents: Elastic serves as the knowledge store and context engine, providing trusted information so agents can perform more complex, goal-driven tasks.

In short, Elastic isn’t just storing data; it’s making enterprise knowledge usable, actionable, and AI-ready. This is the foundation for building intelligent agents that both understand and act.

AWS partnership and MCP server

Elastic has earned the AWS Generative AI Competency status. This recognition is awarded to AWS partners who deliver cutting-edge generative AI solutions that drive measurable gains in business efficiency, creativity, and productivity

Elastic also integrates with the Model Context Protocol (MCP), providing a seamless way for AI agents and applications to interact with Elasticsearch data through natural language conversations.

With the MCP server, you can connect to Elasticsearch directly from any MCP client — such as Claude Desktop, MCP Inspector, or an agentic application. The Elasticsearch MCP server is free to use (though infrastructure and Elasticsearch cluster costs may apply).

And with Amazon Bedrock models (such as Anthropic’s Claude) supporting MCP clients, organizations can now deploy intelligent, data-aware agents more easily and powerfully than ever before.

Amazon Bedrock AgentCore

Amazon Bedrock AgentCore is an enterprise-grade orchestration platform designed for scalable AI agent deployment and management. 

• The platform provides serverless runtime environments with session isolation capabilities, enabling concurrent agent operations across multiple frameworks.

• It implements memory management systems for both session-state and persistent storage, facilitating context-aware model interactions and learning capabilities.

• The architecture includes observability features with granular logging, metrics collection, and advanced debugging capabilities for agent trajectory analysis.

• The platform's robust identity and access management layer enables secure service-to-service authentication and fine-grained authorization controls for AWS and third-party service integrations.

It features a protocol-agnostic gateway for API transformations and tool discovery, supporting MCP-compliant interfaces. The infrastructure also includes containerized browser instances for web automation workflows and isolated compute environments for secure code execution. This end-to-end solution eliminates the need for building custom infrastructure components while maintaining enterprise security and compliance standards.

Solution overview

High-level architecture

The architecture consists of four main components:

1. Python client: Handles user interactions and AWS authentication

2. Amazon Bedrock AgentCore Runtime: Provides serverless hosting and session management

3. Elastic MCP server: Processes MCP protocol requests and queries Elasticsearch

4. Elasticsearch cluster: Stores and indexes the searchable data

Step-by-step workflow walkthrough

1. User authenticates using an authentication mechanism such as OAuth.

2. User access secured Client application running in a Customer AWS account using authenticated credentials.

3. The client application invokes a Supervisor Agent that further invokes and orchestrates other Agents.

4. All the agents are deployed on Amazon AgentCore Runtime and their tools are made available for the Agents, including Elastic’s MCP server and its tools.

5. Foundation Models are available for the agentic AI application through Amazon Bedrock.

6. Elastic Cloud is deployed on AWS and its endpoints are accessed by the Elastic MCP Server. Elastic MCP server automatically crafts the required queries, runs the queries against the Elastic data and fetches the response back to the Supervisor Agent.

7. Supervisor Agent responds back to the end user via the Client Application.

Implementation guide

Please refer to this GitHub repo to get a hands-on experience of how this solution can be implemented. Pay close attention to the prerequisites before getting started.

Step 1: Deploy Elastic MCP server to ECR

The automated deployment script handles the entire container build and upload process:

Here is what the script does when you execute:

1. Downloads the official Elastic MCP server repository

2. Builds Docker container using Dockerfile-8000

3. Creates ECR repository with image scanning enabled

4. Uploads container image to ECR with proper tagging

Step 2: Create AgentCore Runtime host

Navigate to the AWS Console and configure your AgentCore Runtime:

1. Access AgentCore: Go to Amazon Bedrock AgentCore > Build and Deploy > Agent Runtime > Host Agent

2. Basic configuration: Click on “Host Agent” and give it a meaningful name if you prefer. Point to the Container Image you have uploaded to Amazon ECR.

3. Service role: Choose "Create and use a new service role"

4. Protocol settings: Choose MCP, and for the Inbound Identity, select Use IAM username

5. Environment variables: Finally, configure your Elasticsearch endpoints and pass them as environment variables to your Docker container.

6. After creating the host agent, go ahead and copy the Agent Runtime ARN from the "View invocation code" section. Here is an example:

Step 3: Configure Python client

Install dependencies:

Go ahead and initialize a virtual environment and install the Python libraries.

Update Agent ARN in my_mcp_client_remote.py:

Next, update the Python MCP client with the agent ARN you obtained in the previous steps.

Here are the key client components in this Python file.

AWS authentication class:

MCP request formation / payload:

Step 4: Run the client

Execute the Python client to test the integration. This Python program implements an asynchronous client for interacting with Amazon Bedrock AgentCore, specifically designed to query event information. The code utilizes AWS SigV4 authentication and consists of two main functions: test_mcp_endpoint() and chat_with_agentcore(). The first function demonstrates basic API interaction by listing available tools and performing a search query, while the second function implements a more sophisticated search functionality specifically for events in Paris.

The program uses the httpx library for async HTTP requests and handles Server-Sent Events (SSE) responses, parsing and displaying event details including names, venues, dates, and descriptions. The authentication is managed through a custom AWSAuth class that handles AWS SigV4 signing of requests. The code includes comprehensive error handling and formatted output display, making it suitable for both testing and production use cases.

Use case demonstrations

Use case 1: Data discovery

Scenario: Finding events in a specific city using natural language.

Query: "Events in Paris"

MCP request: Here is the payload you supply to the Amazon Bedrock Agentcore Runtime.

Response: And here is the response you get after Elastic’s MCP server runs a query in Elastic Search AI Platform and returns the result.

Use case 2: Elastic MCP tool discovery

Scenario: Discovering the available MCP tools that Elastic’s MCP server offers.

MCP request:

This returns a list of available tools that the MCP server provides, enabling dynamic tool discovery.

Use case 3: Complex search queries

Scenario: Advanced filtering with multiple criteria. You can run more advanced Elastic Search Query Language based queries, like one shown below.

Query: Events with specific price ranges, dates, and categories.

MCP request:

How it works (technical deep dive)

MCP implementation

The Model Context Protocol uses JSON-RPC 2.0 format for all communications:

Key protocol features:

• Stateless operation: Each request is independent with session isolation

• Tool discovery: Dynamic discovery of available capabilities

• Structured responses: Consistent response format across all tools

• Error handling: Standardized error reporting and recovery

AWS authentication flow:

Session management

AgentCore automatically adds Mcp-Session-Id headers for session isolation:

• Each client session gets a unique identifier

• Stateless servers can maintain conversation context

• Automatic cleanup of inactive sessions

Response processing pipeline

1. Server-Sent Events (SSE): Responses come as data: {...} in JSON format

2. JSON parsing: Extract JSON from SSE wrapper

3. Content extraction: Parse the MCP result structure

4. Data formatting: Convert Elasticsearch results to user-friendly format

Cleanup

After you have played around with this setup, if you would like to clean up the environment, please follow the steps outlined below.

Delete AgentCore Runtime:

1. Navigate to Amazon Bedrock AgentCore in AWS Console

2. Select your agent runtime

3. Click "Delete" and confirm

Remove ECR repository:

Clean local environment:

Conclusion

By deploying Elastic's MCP server on Amazon Bedrock AgentCore Runtime, we've created a powerful, scalable, production-ready solution for natural language interaction with Elasticsearch data. This implementation opens up new possibilities for data exploration and analysis, making complex queries accessible through simple conversations.

Key takeaways include:

• Seamless integration: MCP protocol enables natural language querying of complex data

• Production scalability: AgentCore provides enterprise-grade hosting with minimal configuration

• Developer productivity: Transform local prototypes to production with minimal code changes

• Security first: Built-in AWS security and authentication mechanisms

Potential applications can be in any of the following areas of implementation:

• Customer support: Natural language querying of support ticket databases

• Business intelligence: Conversational analytics for business metrics

• Content discovery: Intelligent search across document repositories

• IoT data analysis: Natural language queries for sensor and telemetry data

Additional resources:

• Amazon Bedrock integration documentation

• Elastic MCP server documentation

Let’s build something fun to show it off: a Chuck Norris fact generator. We'll combine movie descriptions with a GPT model to generate facts so legendary even Rambo would be impressed.

What you'll need

• Access to an LLM (like OpenAI’s GPT-4o in our example below)

• A dataset of movie descriptions

You can download a sample dataset from Kaggle and upload it to your Elasticsearch cluster using the Data Visualizer in Kibana or the _bulk API.

Setting up the inference endpoint

Before you can run the COMPLETION command, you need to create an inference endpoint for the model you want to use via the _inference API.

Here’s how to set up GPT-4o with OpenAI:

Once this is in place, you can reference my-gpt-4o-endpoint directly in your query.

The query

Here’s the magic in action. This single ES|QL query handles the entire workflow: it finds a movie based on your input, constructs a prompt from its description, and then calls the LLM to generate a legendary Chuck Norris fact. Below is the full ES|QL query that powers our Chuck Norris fact generator. It takes in a movie query, retrieves the most relevant description, turns it into a prompt, and sends it off to the LLM—all in a single, piped query.

Here’s what comes back:

Yes, the model really said that. 💪🐐🚁

Dissecting the query

Let’s dissect the query and break down what’s happening, step by step.

Step 1: Retrieve relevant movie data

We begin by searching for the most relevant movie for the user query. We use the MATCH function to search both the title and overview fields for the text provided by the query parameter, keeping only the first result, sorted by relevance using the metadata _score field:

This narrows down our dataset to the best match, giving us the movie's title and description, which will become the context for the LLM.

Step 2: Build the prompt from the context

Now we create the input prompt for the LLM by concatenating a static instruction provided as a query parameter, denoted by ?instruction, with the movie’s overview:

This creates a new prompt column combining the provided instruction with the overview field from the returned document, which for our request looks a bit like this:

You can easily swap in different instructions to change the tone or style of what the LLM generates by tweaking the instruction parameter. And because the prompt is just another ES|QL expression, you can compose it with any string-generating function—whether it’s simple concatenation, conditional logic, or even formatting based on your document content.

Step 3: Generate text using the LLM

Finally, we pass the prompt to the inference endpoint connected to our model using our new COMPLETION command, and select which fields to return:

The result? A Chuck Norris fact, rooted in your movie data without any extra tooling required.

This example also demonstrates the full power of ES|QL's piped structure. Each step flows naturally into the next, letting you express a full retrieval augmented generation (RAG) pipeline in a single, declarative query. It’s clean, composable, and stays entirely inside Elasticsearch.

What’s next?

While the COMPLETION command is still a tech preview, this new feature unlocks a whole new world of possibilities—from summarization and content generation to enrichment and storytelling. Try it yourself! Point it at your favorite movie, tweak the prompt, or go wild and generate haikus from SQL errors. The power is yours.

Let us know what you build! 💬

Grounding is the process by which you connect specific data sources to a model to “ground” it to truthful content instead of only relying on the patterns learned during the model’s training, thus giving more reliable and accurate answers.

Grounding helps reduce model hallucinations, generate responses based on your data sources, and allows you to examine the answers by providing citations for them.

Retrieval Augmented Generation(RAG) is a grounding technique where you use search algorithms to retrieve relevant information from external sources, then you use that information as context for the LLM, and finally the model uses the augmented context together with their original training data to generate an answer.

Flow diagram of how RAG works:

RAG allows you to easily scale by updating or expanding the external data sources the model has access to. It is also a cost-effective alternative to fine-tuning LLMs since you can just add data without extensive customization.

And since RAG can access and utilize up-to-date information, it is ideal for use cases when the latest information is key.

Hallucination example

For this example, we’ll use DeepSeek and ask, “Who is the author who won the Chilean National Literature Prize in 1932?” This is a tricky question since the prize was created in 1942. Let’s see how the model answers:

As you can see, since the AI did not have all the information, it hallucinated and provided a made-up answer. Though the information is real in the sense that both the author and work exist, the other parts of the answer are wrong.

Now, let’s see how the model does when we ground it using RAG. For this, we will upload the Wikipedia page about the Chilean National Prize for Literature:

Now, let’s ask the same question and check the answer:

As you can see, with RAG we got the right answer. It says there was no prize in 1932 and asks for clarification from the user.

Using RAG in Playground

By using Elasticsearch, you can easily scale with only having your cluster capacity as a limit. You can use different data sources and connectors to get access to the data you need. Additionally, you have total ownership of your data since it stays in your infrastructure and is not uploaded to a 3rd party service; if you run a local LLM, your data won’t even leave your network. Finally, you have control over search by designing the queries and how to filter data based on access control (RBAC).

We will use Playground, our low-code platform that allows you to quickly and simply create a RAG application using your Elasticsearch content.

Here’s a step-by-step guide on how to upload your PDFs or other documents into Playground. You can also read more about it here and try Playground here.

Upload the PDF

We’ll index into Kibana the same PDF file we provided to DeepSeek. If you followed the instructions in the article above and created the semantic_text field, you’ll be creating a vector database with its corresponding embeddings, ready to be used.

Ask the question

Ask the following question:

“Who is the author who won the Chilean National Literature Prize in 1932?”

Playground sends this query to Elasticsearch, which in turn, runs a semantic search and localizes the fragments with information that is relevant to the question. Then, these fragments are included as context in the prompt sent to the LLM to ground the answer to the information source we provided.

Finally, Playground generated an answer saying that there was no prize in 1932 and provides citations for the relevant fragments as evidence.

Playground also offers two very useful features to understand the RAG system underlying components:

Query

You can see the query Elasticsearch is running to retrieve the relevant documents, and you can enable/disable fields based on your needs.

View code

If you can deploy your RAG application, Playground got you covered. Under the View Code tab, you can see the code used under the hood to create the entire RAG workflow. You can choose between two Python alternatives: Elasticsearch Client with OpenAI, or a Langchain based implementation.

If you want to customize the experience and deploy the code elsewhere, you can use this snippet as a starting point.

Conclusion

Grounding is a process that connects LLMs to external data sources so they can go beyond their training to provide more accurate and trustworthy answers. Retrieval Augmented Generation (RAG) is a grounding method that is scalable, cost-effective, and ensures access to up-to-date information.

Tools like Playground simplify RAG implementation by enabling large-scale indexing, customized searches, and responses with citations, which allow you to easily verify an answer and make sure you’re getting accurate and trustworthy results.

If you want to read more in-depth articles about RAG features, you can start with this one to get a more technical definition of RAG. You can also check Rag vs. fine-tuning: When RAG is the best decision, How to leverage document security using RAG and RAG systems in production.

In this blog, we’ll demonstrate how to use DocParse and DocPrep to prepare and load a dataset of complex PDFs into Elasticsearch for a RAG application. We will use ~75 PDF reports from the National Transportation Safety Board (NTSB) about aircraft incidents. An example document from the collection is here.

What is Aryn DocParse and DocPrep

Aryn DocParse segments and labels documents, extracts tables, and images, and does OCR – turning 30+ document types into structured JSON. It runs the open-source Aryn Partitioner and its open-source deep learning DETR AI model trained on 80k+ enterprise documents. This leads to up to 6x more accurate data chunking and 2x improved recall on hybrid search or RAG compared to off-the-shelf systems.

Aryn DocPrep is a tool for creating document ETL pipelines to prepare and load this data into vector databases and hybrid search indexes like Elasticsearch. The first step in a pipeline is using DocParse to process each document. DocPrep creates Python code using Sycamore, an open-source, scalable, LLM-powered document ETL library. Though DocPrep can easily create ETL pipelines using Sycamore code, you will likely need to customize the pipeline using additional Sycamore data transforms, chunking/merging, extraction, and cleaning functions.

As can be seen, these documents are complex, containing tables, images, section headings, and complicated layouts. Let’s begin!

Building high-quality RAG apps with effective data preparation

Launch an Elasticsearch vector database container

We’ll install Elasticsearch locally using a Docker container for the demo RAG application. Follow these instructions to deploy it.

Prepare data for RAG using Aryn DocPrep and DocParse

Aryn DocPrep is a tool for creating document ETL pipelines that prepare and load data into vector databases and hybrid search indexes like Elasticsearch. The first step in a pipeline is using DocParse to process each document.

We will use Aryn DocParse in Aryn Cloud to generate our initial ETL pipeline code. You can sign up for free to use Aryn Cloud and go to the DocPrep UI in the Aryn Cloud console.

You can also write an ETL pipeline and run a version of the Aryn Partitioner (used in DocParse) locally. Visit the Sycamore documentation to learn more.

Create ETL pipeline with Aryn DocPrep

DocPrep creates Python code using Sycamore, an open-source, scalable, LLM-powered document ETL library. While DocPrep can easily create ETL pipelines using Sycamore code, you may need to customize the pipeline further with additional Sycamore data transforms, extraction, and cleaning functions.

DocPrep simplifies the creation of a base ETL pipeline to prepare unstructured data for RAG and semantic search.

First, we provide the document type (PDF) and the source location of our PDFs in Amazon S3 (s3://aryn-public/ntsb/):

Next, we will select MiniLM for our embedding model to create our vector embeddings locally. DocPrep uses DocParse for document segmentation, extraction, and other processing, but we don’t need to change the default configuration.

Finally, we select Elasticsearch as our target database and add the Host URL and Index name. Note that the URL is set to “localhost” because we are running Elasticsearch locally. We will also run DocPrep/Sycamore ETL pipeline locally so it can easily load the cluster.

Now, click “Generate pipeline” to create the ETL pipeline. Click “Download notebook” on the next page to download the code as a Jupyter notebook file.

Install Jupyter and Sycamore

We will run the ETL pipeline locally in a Jupyter notebook and use the Sycamore document ETL library. As a first step, install Jupyter and Sycamore with the Elasticsearch connector and local inference libraries to create vector embeddings.

Run Pipeline

Run Jupyter and open the notebook with the ETL pipeline downloaded in the earlier step.

If you haven’t set your Aryn Cloud API key as an environmental variable called ARYN_API_KEY, you can set it directly in the notebook.

In the second-to-last cell, update the Elasticsearch loading configuration. Replace the es_client_args from setting an Elasticsearch password to the Elasticsearch basic auth config from your container:

If the password isn’t set as an environment variable, you can add it directly here.

Now, run the cells in the notebook. Each of the ~75 PDFs is sent to DocParse for processing, and this step in the pipeline will take a few minutes. One of the cells will output three pages of a document with bounding boxes to show how DocParse segments the data.

The final cell runs a read query to verify if the data has been loaded correctly. Now, you can use the prepared data in the Elasticsearch index with your RAG application.

Add additional data enrichment and transforms

The code generated in DocPrep is great for a basic ETL pipeline, however, you may want to extract metadata and perform data cleaning. The pipeline code is fully customizable, and you can use additional transformations in Sycamore or arbitrary Python code.

Here is an example notebook with additional data transforms, metadata extraction, and data cleaning steps. You can use this metadata in your RAG applications to filter your results.

Conclusion

This blog used Aryn DocParse, DocPrep, and Sycamore to parse, extract, enrich, clean, embed, and load data into vector and keyword indexes in the Elasticsearch vector database. We used DocPrep to create an initial ETL pipeline and then used a notebook with additional Sycamore code to demonstrate additional data enrichment and cleaning.

How your documents are parsed, enriched, and processed significantly impacts the quality of your RAG queries. Use the examples in this blog post to quickly and easily build your own RAG systems with Aryn and Elasticsearch and iterate on the processing and retrieval strategies as you build your GenAI application.

Below are some resources for your next steps:

• Sample notebook with Aryn DataPrep and Elasticsearch

• start-local with Elasticsearch vector database

• Get started with Aryn Cloud DocPrep

• Sycamore documentation

• Elasticsearch vector database ecosystem integrations

The out-of-the-box Semantic Kernel Elasticsearch Vector Store Connector supports the Semantic Kernel vector store abstractions which make it very easy for developers to plugin Elasticsearch as a vector store while building AI agents.

Elasticsearch has a strong foundation in the open-source community and recently adopted the AGPL license. Combined with the open-source Microsoft Semantic Kernel, these tools offer a powerful, enterprise-ready solution. You can get started locally by spinning up Elasticsearch in a few minutes by running this command curl -fsSL https://elastic.co/start-local | sh (refer start-local for details) and move to cloud-hosted or self-hosted versions while productionizing your AI agents.

In this blog we look at how to use Semantic Kernel Elasticsearch Vector Store Connector when using Semantic Kernel. A Python version of the connector will be made available in the future.

High-level scenario: Building a RAG app with Semantic Kernel & Elasticsearch

In the following section we go through an example. At a high-level we are building a RAG (Retrieval Augmented Generation) application which takes a user's question as input and returns an answer. We will use Azure OpenAI (local LLM can be used as well) as the LLM, Elasticsearch as the vector store and Semantic Kernel (.net) as the framework to tie all components together.

If you are not familiar with RAG architectures, you can have a quick introduction with this article: https://www.elastic.co/search-labs/blog/retrieval-augmented-generation-rag.

The answer is generated by the LLM which is fed with context, relevant to the question, retrieved from Elasticsearch vectorstore. The response also includes the source that was used as the context by the LLM.

RAG example

In this specific example, we build an application that allows users to ask questions about hotels stored in an internal hotel database. The user could e.g. search for a specific hotel, based on different criteria, or ask for a list of hotels.

For the example database, we generated a list of hotels containing 100 entries. The sample size is intentionally small to allow you to try out the connector demo as easily as possible. In a real-world application, the Elasticsearch connector would show its advantages over other options, such as the `InMemory` vector store implementation, especially when working with extremely large amounts of data.

The complete demo application can be found in the Elasticsearch vector store connector repository.

Let’s start with adding the required NuGet packages and using directives to our project:

We can now create our data model and provide it with Semantic Kernel specific attributes to define the storage model schema and some hints for the text search:

The Storage Model Schema attributes (`VectorStore*`) are most relevant for the actual use of the Elasticsearch Vector Store Connector, namely:

• VectorStoreRecordKey to mark a property on a record class as the key under which the record is stored in a vector store.

• VectorStoreRecordData to mark a property on a record class as 'data'.

• VectorStoreRecordVector to mark a property on a record class as a vector.

All of these attributes accept various optional parameters that can be used to further customize the storage model. In the case of VectorStoreRecordKey , for example, it is possible to specify a different distance function or a different index type.

The text search attributes (TextSearch*) will be important in the last step of this example. We will come back to them later.

In the next step, we initialize the Semantic Kernel engine and obtain references to the core services. In a real world application, dependency injection should be used instead of directly accessing the service collection. The same thing applies to the hardcoded configuration and secrets, which should be read using a configuration provider instead:

The vectorStoreCollection service can now be used to create the collection and to ingest a few demo records:

This shows how Semantic Kernel reduces the use of a vector store with all its complexity to a few simple method calls.

Under the hood, a new index is created in Elasticsearch and all the necessary property mappings are created. Our data set is then mapped completely transparently into the storage model and finally stored in the index. Below is how the mappings look in Elasticsearch.

The embeddings.GenerateEmbeddingsAsync() calls transparently called the configured Azure AI Embeddings Generation service.

Even more magic can be observed in the last step of this demo.

With just a single call to InvokePromptAsync, all of the following operations are performed when the user asks a question about the data:

1. An embedding for the user's question is generated

2. The vector store is searched for relevant entries

3. The results of the query are inserted into a prompt template

4. The actual query in the form of the final prompt is sent to the AI chat completion service

Remember the TextSearch* attributes, we previously defined on our data model? These attributes enable us to use corresponding placeholders in our prompt template which are automatically populated with the information from our entries in the vector store.

The final response to our question "Please show me all hotels that have a rooftop bar." is as follows:

The answer correctly refers to the following entry in our hotels.csv

This example shows very well how the use of Microsoft Semantic Kernel achieves a significant reduction in complexity through its well thought abstractions, as well as enabling a very high level of flexibility. By changing a single line of code, for example, the vector store or the AI services used can be replaced without having to refactor any other part of the code.

At the same time, the framework provides an enormous set of high-level functionality, such as the `InvokePrompt` function, or the template or search plugin system.

The complete demo application can be found in the Elasticsearch vector store connector repository.

What else is possible with Elasticsearch

• Elasticsearch new semantic_text mapping: Simplifying semantic search

• Semantic reranking in Elasticsearch with retrievers

• Advanced RAG techniques part 1: Data processing

• Advanced RAG techniques part 2: Querying and testing

• Building RAG with Llama 3 open-source and Elastic

• A tutorial on building local agent using LangGraph, LLaMA3 and Elasticsearch vector store from scratch

Elasticsearch & Semantic Kernel: What's next?

• We showed how the Elasticsearch vector store can be easily plugged into Semantic Kernel while building GenAI applications in .NET. Stay tuned for a Python integration next.

• As Semantic Kernel builds abstractions for advanced search features like hybrid search, the Elasticsearch connect will enable .NET developers to easily implement them while using Semantic Kernel.

Using Anthropic’s messages to answer questions

In this blog, we’ll use the Claude Messages API to answer questions during ingestion to have answers ready ahead of searching. Before we start interacting with Elasticsearch, make sure you have an Anthropic API key by creating an evaluation account first and generating a key. We’ll use Kibana's Console to execute these next steps in Elasticsearch without setting up an IDE.

First, we configure an inference endpoint, which will interact with Anthropic’s messages API:

We’ll get back a response similar to the following with status code 200 OK on successful inference endpoint creation:

We can now call the configured endpoint to perform completion on any text input. Let’s ask the model for a short description of GenAI:

We should get a response back with a status code 200 OK providing a short description of GenAI:

Now we can set up a catalog of questions which we want to be answered during ingestion. We’ll use the Elasticsearch Bulk API to index these questions about Elastic products:

A response similar to the one below should be returned upon successful indexing:

We’ll now create our question and answering ingest pipeline using the script, inference, and remove processors:

The pipeline prefixes the question field with the text: “Please answer the following question: “ in a temporary field called prompt. The content of the temporary prompt field is sent to the Anthropic service via the inference API. Using an ingest pipeline provides extensive flexibility as you can set the pre-prompt to fit your needs. This approach can be used to summarize documents as well.

Next, we’ll send our documents containing the questions through the question and answering pipeline by calling the reindex API.

We should get back a response similar to the following:

In a production setup, you’ll likely use another ingestion mechanism to ingest your documents in an automated manner. Check out our Adding data to Elasticsearch guide to learn more about the various options offered by Elastic to ingest data into Elasticsearch. We’re also committed to showcasing ingest mechanisms and providing guidance on bringing data into Elasticsearch using 3rd party tools. For example, take a look at Ingest Data from Snowflake to Elasticsearch using Meltano: A developer’s journey to see how to use Meltano for ingesting data.

We can now search for our pre-generated answers using the Search API:

The response will contain the pre-generated answers:

Pre-generating answers for frequently asked questions is particularly effective in reducing operational costs. By minimizing the need for on-the-fly response generation, you can significantly cut down on the amount of computational resources required. Additionally, this method ensures that every user receives the same precise information. Consistency is critical, especially in fields requiring high reliability and accuracy such as medical, legal, or technical support.

In this blog, you will learn how to:

• Create an Elasticsearch Serverless project

• Create an Inference Endpoint to generate embeddings with ELSER

• Use a Semantic Text field for auto-chunking and calling the Inference Endpoint

• Use the Open Crawler to crawl blogs

• Connect to an LLM using Elastic’s Playground to test prompts and context settings for a RAG chat application.

If you want to jump right into the code, you can view the accompanying Jupyter Notebook here.

ChatGPT and Elasticsearch (April 2023)

A lot has changed since I wrote the initial ChatGPT and Elasticsearch: OpenAI meets private data. Most people were just playing around with ChatGPT, if they had tried it at all. And every booth at every tech conference didn’t feature the letters “AI” (whether it is a useful fit or not).

Updates in Elasticsearch (August 2024)

Since then, Elastic has embraced being a full featured vector database and is putting a lot of engineering effort into making it the best vector database option for anyone building a search application. So as not to spend several pages talking about all the enhancements to Elasticsearch, here is a non-exhaustive list in no particular order:

• ELSER - The Elastic Learned Sparse Encoder

• Elastic Serverless Service was built and is in public beta

• Elasticsearch open Inference API

  • Embeddings

  • Chat completion

  • Semantic rerankers

• Semantic_text type - Simplify semantic search

  • Automatic chunking

• Playground - Visually experiment with RAG application building in Elasticsearch

• Retrievers

• Open web crawler

With all that change and more, the original blog needs a rewrite. So let’s get started.

Updated flow: ChatGPT, Elasticsearch & RAG

The plan for this updated flow will be:

1. Setup

   1. Create a new Elasticsearch serverless search project

   2. Create an embedding inference API using ELSER

   3. Configure an index template with a semantic_text field

   4. Create a new LLM connector

   5. Configure a chat completion inference service using our LLM connector

2. Ingest and Test

   1. Crawl the Elastic Labs sites (Search, Observability, Security) with the Elastic Open Web Crawler.

   2. Use Playground to test prompts using our indexed Labs content

3. Configure and deploy our App

   1. Export the generated code from Playground to an application using FastAPI as the backend and React as the front end.

   2. Run it locally

   3. Optionally deploy our chatbot to Google Cloud Run

Setup

Elasticsearch Serverless Project

We will be using an Elastic serverless project for our chatbot. Serverless removes much of the complexity of running an Elasticsearch cluster and lets you focus on actually using and gaining value from your data. Read more about the architecture of Serverless here.

If you don’t have an Elastic Cloud account, you can create a free two-week trial at elastic.co (Serverless pricing available here). If you already have one, you can simply log in.

Once logged in, you will need to create a cloud API key.

NOTE: In the steps below, I will show the relevant parts of Python code. For the sake of brevity, I’m not going to show complete code that will import required libraries, wait for steps to complete, catch errors, etc.

For more robust code you can run, please see the accompanying Jypyter notebook!

Create Serverless Project

We will use our newly created API key to perform the next setup steps.

First off, create a new Elasticsearch project.

• url - This is the standard Serverless endpoint for Elastic Cloud

• project_data - Your Elasticsearch Serverless project settings

  • name - Name we want for the project

  • region_id - Region to deploy

  • optimized_for - Configuration type - We are using vector which isn’t strictly required for the ELSER model but can be suitable if you select a dense vector model such as e5.

Create Elasticsearch Python client

One nice thing about creating a programmatic project is that you will get back the connection information and credentials you need to interact with it!

ELSER Embedding API

Once the project is created, which usually takes less than a few minutes, we can prepare it to handle our labs’ data.

The first step is to configure the inference API for embedding. We will be using the Elastic Learned Sparse Encoder (ELSER).

• Command to create the inference endpoint

• Specify this endpoint will be for generating sparse embeddings

• model_config - Settings we want to use for deploying our semantic reranking model

  • service - Use the pre-defined elser inference service

  • service_settings.num_allocations - Deploy the model with 8 allocations

  • service_settings.num_threads - Deploy with one thread per allocation

• inference_id - The name you want to give to you inference endpoint

• task_type- Specifies this endpoint will be for generating sparse embeddings

This single command will trigger Elasticsearch to perform a couple of tasks:

1. It will download the ELSER model.

2. It will deploy (start) the ELSER model with eight allocations and one thread per allocation.

3. It will create an inference API we use in our field mapping in the next step.

Index Mapping

With our ELSER API created, we will create our index template.

• index_patterns - The pattern of indices we want this template to apply to.

• body - The main content of a web page the crawler collects will be written to

  • type - It is a text field

  • copy_to - We need to copy that text to our semantic text field for semantic processing

• semantic_body is our semantic text field

  • This field will automatically handle chunking of long text and generating embeddings which we will later use for semantic search

  • inference_id specifies the name of the inference endpoint we created above, allowing us to generate embeddings from our ELSER model

• headings - Heading tags from the html

• id - crawl id for this document

• meta_description - value of the description meta tag from the html

• title is the title of the web page the content is from

Other fields will be indexed but auto-mapped. The ones we are focused on pre-defining in the template will not need to be both keyword and text type, which is defined automatically otherwise.

Most importantly, for this guide, we must define our semantic_text field and set a source field to copy from with copy_to. In this case, we are interested in performing semantic search on the body of the text, which the crawler indexes into the body.

Crawl All the Labs!

We can now install and configure the crawler to crawl the Elastic * Labs. We will loosely follow the excellent guide from the Open Crawler released for tech-preview Search Labs blog.

The steps below will use docker and run on a MacBook Pro. To run this with a different setup, consult the Open Crawler Github readme.

Clone the repo

Open the command line tool of your choice. I’ll be using Iterm2. Clone the crawler repo to your machine.

Build the crawler container

Run the following command to build and run the crawler.

Configure the crawler

Create a new YAML in your favorite editor (vim):

We want to crawl all the documents on the three labs’ sites, but since blogs and tutorials on those sites tend to link out to other parts of elastic.co, we need to set a couple of runs to restrict the scope. We will allow crawling the three paths for our site and then deny anything else.

Paste the following in the file and save

Copy the configuration into the Docker container:

Validate the domain

Ensure the config file has no issues by running:

Start the crawler

When you first run the crawler, processing all the articles on the three lab sites may take several minutes.

Confirm articles have been indexed

We will confirm two ways.

First, we will look at a sample document to ensure that ELSER embeddings have been generated. We just want to look at any doc so we can search without any arguments:

Ensure you get results and then check that the field body contains text and semantic_body.inference.chunks.0.embeddings contains tokens.

We can check we are gathering data from each of the three sites with a terms aggregation:

You should see results that start with one of our three site paths.

To the Playground!

With our data ingested, chunked, and inference, we can start working on the backend application code that will interact with the LLM for our RAG app.

LLM Connection

We need to configure a connection for Playground to make API calls to an LLM. As of this writing, Playground supports chat completion connections to OpenAI, AWS Bedrock, and Google Gemini. More connections are planned, so check the docs for the latest list.

When you first enter the Playground UI, click on “Connect to an LLM”

Since I used OpenAI for the original blog, we’ll stick with that. The great thing about the Playground is that you can switch connections to a different service, and the Playground code will generate code specifically to that service’s API specification. You only need to select which one you want to use today.

In this step, you must fill out the fields depending on which LLM you wish to use. As mentioned above, since Playground will abstract away the API differences, you can use whichever supported LLM service works for you, and the rest of the steps in this guide will work the same.

If you don’t have an Azure OpenAI account or OpenAI API account, you can get one here (OpenAI now requires a $5 minimum to fund the API account).

Once you have completed that, hit “Save,” and you will get confirmation that the connector has been added. After that, you just need to select the indices we will use in our app. You can select multiple, but since all our crawler data is going into elastic-labs, you can choose that one.

Click “Add data sources” and you can start using Playground!

Select the “restaurant_reviews” index created earlier.

Playing in the Playground

After adding your data source you will be in the Playground UI.

To keep getting started as simple as possible, we will stick with all the default settings other than the prompt. However, for more details on Playground components and how to use them, check out the Playground: Experiment with RAG applications with Elasticsearch in minutes blog and the Playground documentation.

Experimenting with different settings to fit your particular data and application needs is an important part of setting up a RAG-backed application.

The defaults we will be using are:

• Querying the semantic_body chunks

• Using the three nearest semantic chunks as context to pass to the LLM

Creating a more detailed prompt

The default prompt in Playground is simply a placeholder. Prompt engineering continues to develop as LLMs become more capable. Exploring the ever-changing world of prompt engineering is a blog, but there are a few basic concepts to remember when creating a system prompt:

• Be detailed when describing the app or service the LLM response is part of. This includes what data will be provided and who will consume the responses.

• Provide example questions and responses. This technique, called few-shot-prompting, helps the LLM structure its responses.

• Clearly state how the LLM should behave.

• Specify the Desired Output Format.

• Test and Iterate on Prompts.

With this in mind, we can create a more detailed system prompt:

Feel free to to test out different prompts and context settings to see what results you feel are best for your particular data. For more examples on advanced techiques, check out the Prompt section on the two part blog Advanced RAG Techniques. Again, see the Playground blog post for more details on the various settings you can tweak.

Export the Code

Behind the scenes, Playground generates all the backend chat code we need to perform semantic search, parse the relevant contextual fields, and make a chat completion call to the LLM. No coding work from us required!

In the upper right corner click on the “View Code” button to expand the code flyout

You will see the generated python code with all the settings your configured as well as the the functions to make a semantic call to Elasticsearch, parse the results, built the complete prompt, make the call to the LLM, and parse those results.

Click the copy icon to copy the code.

You can now incorporate the code into your own chat application!

Wrapup

A lot has changed since the first iteration of this blog over a year ago, and we covered a lot in this blog. You started from a cloud API key, created an Elasticsearch Serverless project, generated a cloud API key, configured the Open Web Crawler, crawled three Elastic Lab sites, chunked the long text, generated embeddings, tested out the optimal chat settings for a RAG application, and exported the code!

Where’s the UI, Vestal?

Be on the lookout for part two where we will integrate the playground code into a python backend with a React frontend. We will also look at deploying the full chat application.

For a complete set of code for everything above, see the accompanying Jypyter notebook

The code that accompanies this article implements a database of famous quotes. It includes a back end written in Python with the FastAPI web framework, and a front end written in TypeScript and React. Regarding vector search, this application demonstrates how to:

• run a local Elasticsearch service using Docker,

• bulk-ingest a large number of documents efficiently,

• generate vector embeddings for documents as they are ingested,

• leverage the power of a GPU to accelerate the generation of vector embeddings through parallelization,

• run vector search queries using the approximate kNN algorithm,

• aggregate results from vector search,

• compare vector search results against those resulting from a standard match (BM25) query.

Below you can see a screenshot of the application. In this article you will find a detailed explanation of how the ingest and search features work. You then have the option to install and run the code on your own computer to experiment and learn!

What is the Elasticsearch-DSL client for Python?

Sometimes called the "high-level" Python client, Elasticsearch-DSL offers idiomatic (or "Pythonic") access to your Elasticsearch database, in contrast with the official (or "low-level") Python client, which provides direct access to the complete range of Elasticsearch features and endpoints.

When using Elasticsearch-DSL, the structure (or "mappings") of Elasticsearch indices are defined as classes, with a syntax that is similar to that of Python dataclasses. The documents stored in these indices are represented by instances of these classes. All the transformations that are necessary to map between Python objects and Elasticsearch documents are automatically and transparently carried out, resulting in application code that is simple and idiomatic.

To add Elasticsearch-DSL to your Python project, you can install it with pip:

If your project is asynchronous, then there are additional dependencies that need to be installed, so in that case use the following command instead:

Index definition

As stated above, with Elasticsearch-DSL the structure of an Elasticsearch index is defined as a Python class. The example application featured in this article uses a dataset of famous quotes that have the following fields:

• quote: the text of the quote, as a string

• author: the name of the author, as a string

• tags: a list of tag names that apply to the quote, each a string

As part of this application we are going to add one additional field, the vector embedding that we will use to search for quotes:

• embedding: a list of floating point numbers representing a vector embedding for the quote

Let's write an initial document class to describe our famous quotes index:

The AsyncDocument class that is used as a base class for our QuoteDoc class implements all the functionality to connect the class to an Elasticsearch index. The choice of an asynchronous document base class was made because this examples uses the FastAPI web framework, which is also asynchronous. For projects that do not use asynchronous Python, the Document base class must be used when declaring document classes.

The name attribute given in the Index inner class defines the name of the Elasticsearch index that will be used with documents of this class.

If you have used Python dataclasses before, you likely find the way fields are defined very familiar, with each field being given a Python type hint. These Python types are mapped to the closest Elasticsearch type, so for example, in the case of str, the corresponding field in the Elasticsearch index will be given the type text, the standard type that is used for text that needs to be indexed for full-text search, while float is mapped to the equally named float on the Elasticsearch side.

While it can be useful to leave the quote field as is so that we can use it for both vector and full-text searches, the author and tags fields do not really need all the extra work associated with full-text search. The best Elasticsearch type for these fields is keyword, which just stores the text, without doing any indexing. Likewise, the embedding field is not just a simple list of floating point numbers, we are going to use it for vector search, which is a behavior associated with the dense_vector type in Elasticsearch.

To assign a type override to a field, we add an assignment with the mapped_field() function, as shown in the improved version of the QuoteDoc class that follows:

As you can see in this updated version, the elasticsearch_dsl package includes classes such as Keyword and DenseVector to represent all the native Elasticsearch field types.

Did you notice the init=False argument given in this new definition of the embedding field? If you are familiar with Python dataclasses you may recognize init as one of the options available in the dataclasses field() function, used to indicate that the given attribute should be omitted from the constructor for instances of the class. The behavior is the same here, which means that when creating an instance of QuoteDoc, this argument should not be given.

How will the vector embeddings be generated if they will not be passed down to the document constructor? Elasticsearch-DSL always calls the clean() method in all documents before serializing them and sending them to Elasticsearch. This method is a convenience entry point where the application can add any custom field processing logic. For example, fields that are optional or auto-generated can be added in this method. Here is the final version of the QuoteDoc document class, including the logic that generates the embeddings:

For this example we are going to use embeddings from a SentenceTransformers model. These embeddings are easy to generate locally and being open source and free they are convenient to use when experimenting. The all-MiniLM-L6-v2 model is a great general purpose embedding model for English text. There are many other models that are also compatible with the SentenceTransformers framework, so feel free to use a different one if you prefer.

The clean() method can be used for more advanced use cases as well. For example, it is common when working with large bodies of text to split the text into smaller chunks, and then generate embeddings for each chunk. Elasticsearch accommodates this use case through nested objects. If you want to see an advanced example that implements this type of solution, check out the vectors example in the Elasticsearch-DSL repository.

Document ingestion

With the structure of the index in place, we can now create the index. This is done with the init() class method:

In many cases it is useful to delete a previously existing index to make sure an ingest process begins from a clean starting point. This can be done using the _index class attribute, which provides access to the Elasticsearch index, along with its exists() and delete() methods:

The example dataset used by the example application is a collection of almost 37,000 famous quotes. It comes as a CSV file with the quote, author and tags columns. The tags are given as a comma-separated string. The dataset is available for download from the example GitHub repository.

To ingest the data contained in this dataset, Python's csv module can be used:

The csv.DictReader class creates a CSV file importer that returns a dictionary for each row in the data file. For each row, we create a QuoteDoc instance and pass the quote, author and tags in the constructor. For the tags, the string that is read from the CSV file has to be split into a list, which is how it will be stored in the Elasticsearch index.

To write a document to the index, the save() method is invoked. This method will call the document's clean() method, which in turn will generate the vector embedding for the quote.

Starting an Elasticsearch instance

Before the above ingest script can be executed, you need to have access to a running instance of Elasticsearch. By far the easiest (and also 100% free) way to do this is with a Docker container.

To start a single-node Elasticsearch service on your computer first make sure you have Docker running, and then execute the following command:

To make sure you are running the latest and greatest version, open the release notes page to find out what is the current version, then replace the version number in the last line of the above command.

The -v option in the command above sets up a mapping between a directory named data in your local system and the data directory in the Elasticsearch container. All the data files used by Elasticsearch will be saved in this directory, so that in case you need to restart your container you do not lose any data. If you prefer to not store the data files in your computer, then you can remove the -v line and the data will be stored ephemerally in the container.

Note that deploying Elasticsearch using this method is only adequate for local experimentation. If you intend to deploy Elasticsearch on a production server, consider using our Elasticsearch on Docker Compose or Elasticsearch on Kubernetes guides.

Connecting to Elasticsearch

The ingestion script needs to know how to connect to Elasticsearch. If you are running a Docker container as demonstrated in the previous section, add the following line between the imports and the definition of the QuoteDoc class:

To complete the script, the ingest_quotes() function should be called. Add the following snippet at the bottom of your source file:

The asyncio.run() function will launch the asynchronous application. If your application is not asynchronous, then you would just call the ingest function directly.

For your convenience, below you can find the complete code for the script up to this point. You can save this file as search.py. You can find an example of this file here.

Create a virtual environment for your project using the tool of your choice, and then install the dependencies on it:

Make sure you have the quotes.csv file in the current directory, and then start the ingest by running the script:

The script does not print anything, so it will run for a while adding the quotes from the CSV file into your Elasticsearch index. The file has about 37,000 quotes, so expect the process to run for several minutes.

Luckily you do not need to wait that long. If you start the script and no error appears, that is confirmation that everything is working. You can press Ctrl-C to stop it and continue reading to learn about ingest performance.

Performance tuning part 1: bulk processing

If your dataset is small, then the above ingest solution will work just fine, and it has the benefit that it is simple to code and easy to understand.

For larger ingest jobs, however, it is necessary to sacrifice code clarity and pay attention to performance, so let's see what optimizations can be done in this application.

First of all, to evaluate performance we need to be able to measure the performance of the existing solution. Below is the updated ingest_quotes() function, which now calls ingest_progress() every 100 ingested documents to show how many documents have been ingested, along with an average document per second.

This version of the ingest is nicer than the previous one because it prints regular status updates. If you let the script run for a while you may see an output similar to the one below:

The data file has close to 37,000 quotes, so now you can have a good idea of how long the ingest will take. Assuming the average of 97 ingested documents per second holds throughout the entire ingest job, it should take less than 7 minutes to ingest the entire dataset. You can press Ctrl-C to stop this ingest process, there is no need to let it run to completion yet.

Elasticsearch offers a very flexible bulk ingest feature, which is made available in the Elasticsearch-DSL package's bulk() method. Instead of saving each document, the entire import loop can be moved into a generator function which is given to the bulk() method as an argument:

Here the get_next_quote() inner generator function yields QuoteDoc instances. The QuoteDoc.bulk() method will run the generator and issue batch updates to Elasticsearch. With this change, you can expect to see a small speed improvement:

For another small improvement, the JSON serializer used by the Elasticsearch client can be changed to the orjson library, which performs better than Python's own:

This should lead to another small performance improvement:

Performance tuning part 2: GPU accelerated embeddings

You have seen in the previous section that we have obtained some modest performance improvements by processing ingest requests in bulk. But while ingestion requests are now being grouped, the embeddings continue to be generated one by one in the clean() method of the QuoteDoc class.

Is there a way to optimize embedding generation? The SentenceTransformers model uses PyTorch, which in turn uses a GPU if one is available. But the embeddings are generated individually, which does not lead to an optimal utilization of the GPU hardware. GPUs are very good at parallelization, so we can reorganize the ingest function to generate embeddings in batches. And once again the price we pay for this comes in increased code complexity.

So we are going to stop using the clean() method to generate document embeddings, and instead we are going to accumulate the QuoteDoc instances in a list, and once we reach a good number we'll generate embeddings for all of them in a single operation.

Let's start by writing a helper function that generates embeddings for a list of QuoteDoc instances:

Note how now the model.encode() method is given a list of quotes to embed instead of a single one. When the input argument is a list, the model generates an embedding for each list element. The method accepts an optional batch_size argument (not used in the example above) that defaults to 32 that can be used to control the size of each batch of samples that are sent to the model for computation. Depending on the GPU hardware you may find that different values of this argument help tune performance to the best possible. Once the embeddings are generated, they are assigned to each quote using a for-loop.

Now the ingest function can be refactored to accumulate quotes and use the helper function to generate embeddings:

In this version of ingest_quotes(), each QuoteDoc instance is added to the quotes list, and when 512 elements have accumulated the embed_quotes() function added above is used to generate the embeddings more efficiently. Once the objects have their embeddings, they are yielded, so that the bulk() method from Elasticsearch-DSL can add them to the index as before.

What is the significance of the 512 number? There isn't any. We know that the model uses a batch size of 32, so it makes sense to accumulate at least that many documents. Starting from 32, you can try if larger powers of 2 provide better performance. With the hardware available to me, I've found 512 to give the best performance.

Here is an example run using batched embeddings:

And now the ingestion process runs much faster, with the entire dataset ingested in about 1 minutes and 16 seconds.

If you decide to try to optimize your ingest, you are encouraged to try different options and see what works best with your hardware.

Querying the index

If you are following along, by now you have an Elasticsearch index called quotes that is populated with about 37K famous quotes, each with a searchable vector embedding. Now it is time to learn how to query this index.

When using Elasticsearch-DSL, the document classes return a search object from their search() method:

The search object has a large number of methods that map to the query options in the Elasticsearch query DSL.

The simplest query that can be issued is the match all query, which returns all the elements. With the class-based approach used by Elasticsearch-DSL, this is how to run the query:

This would obviously print a listing of the entire list of quotes stored in the index, up to 10,000, which is the maximum number of results Elasticsearch returns by default.

In many cases it is useful to request a subset of the results. The search object uses Python style slicing for this. Here is how to request the first 25 results only:

Here is how to request the second page of results, at 25 results per page:

Elasticsearch offers approximate and exact vector search queries, also called k-nearest neighbor (kNN) queries. To run a vector search query with the approximate k-nearest neighbor algorithm, the Knn query should be used:

The Knn query class accepts the field that stores the embeddings and a search vector as arguments. In the above snippet the variable q has the search text entered by the user.

If instead you prefer to run a regular full-text search, the Match query class is used:

Filters

One of the most important benefits of using Elasticsearch as a vector database is that it is a robust database system, and all the options you can expect to have from a database nicely integrates with your vector search queries.

A great example of this is filters. The famous quotes database stores a list of tags for each quote, so it is only natural to have the option to restrict a query to quotes that have a specific tag.

Given a list of tag filters stored in a tags variable, the following snippet configures a search object to only return results that include the given tags using a "terms" filter:

Aggregations

Another example of a useful database function that is fully integrated with vector search is aggregations. Given a query, Elasticsearch can aggregate the tags and provide the counts of quotes per tag.

The next snippet shows how to add a Terms aggregation to an existing query, which will return the 100 most referenced tags in the results:

Recall that the tags field was declared with the Keyword() type, which means that the tags will be stored as is on the index, without any processing. This is required by the Terms aggregation, which will count the occurrences of each tag in the results.

A complete query example

You have seen a few isolated query examples. In this section you can see how they can all be integrated into a function that performs a query in the example application.

The search_quotes() function shown below accepts a query string q, a list of filters tags and a use_knn flag to choose between kNN or full-text search query. It also accepts start and size pagination arguments.

The function decides which of the three queries you've seen above to issue depending on the input arguments. If q is empty, then it selects a "match all" query, and in any other case it selects a kNN or match query depending on the use_knn flag, which the user can control from a checkbox in the application's user interface.

The function returns three results as a tuple:

• a list of QuoteDoc instances that are the search results,

• the tag aggregations as a list of tuples, each with tag name and document count,

• the total number of results, which is useful to show in paginated queries

Here is the complete code of this function:

To be able to access both the search results and the aggregation results, we now issue the request explicitly through the execute() method and store the response is stored in r. The hits attribute of the response object contains the actual search results, and the aggs attribute provides access to the aggregations. The format in which the aggregation results is provided is described in the terms aggregation documentation.

Conclusion

The complete quotes example is available in a GitHub repository that you can install and run on your computer. Follow the instructions on the README.md file to set it up.

You are welcome to use this example to experiment with vector embeddings and Elasticsearch!

Welcome to Part 2 of our article on Advanced RAG Techniques! In part 1 of this series, we set up, discussed, and implemented the data processing components of the advanced RAG pipeline:

In this part, we're going to proceed with querying and testing out our implementation. Let's get right to it!

Table of contents

• Searching and retrieving, generating answers

  • Enriching queries with synonyms

  • HyDE (Hypothetical Document Embedding)

  • Hybrid search

• Experiments

  • Summary of results

  • Test 1: Who audits Elastic?

    • AdvancedRAG

    • SimpleRAG

  • Test 2: total revenue 2023

    • AdvancedRAG

    • SimpleRAG

  • Test 3: What product does growth primarily depend on? How much?

    • AdvancedRAG

    • SimpleRAG

  • Test 4: Describe employee benefit plan

    • AdvancedRAG

    • SimpleRAG

  • Test 5: Which companies did Elastic acquire?

    • AdvancedRAG

    • SimpleRAG

• Conclusion

• Appendix

  • Prompts

    • RAG question answering prompt

    • Elastic query generator prompt

    • Potential questions generator prompt

    • HyDE generator prompt

  • Sample hybrid search query

Searching and retrieving, generating answers

Let's ask our first query, ideally some piece of information found primarily in the annual report. How about:

Now, let's apply a few of our techniques to enhance the query.

Enriching queries with synonyms

Firstly, let's enhance the diversity of the query wording, and turn it into a form that can be easily processed into an Elasticsearch query. We'll enlist the aid of GPT-4o to convert the query into a list of OR clauses. Let's write this prompt:

When applied to our query, GPT-4o generates synonyms of the base query and related vocabulary.

In the ESQueryMaker class, I've defined a function to split the query:

Its role is to take this string of OR clauses and split them into a list of terms, allowing us do a multi-match on our key document fields:

Finally ending up with this query:

This covers many more bases than the original query, hopefully reducing the risk of missing a search result because we forgot a synonym. But we can do more.

Back to top

HyDE (Hypothetical Document Embedding)

Let's enlist GPT-4o again, this time to implement HyDE.

The basic premise of HyDE is to generate a hypothetical document - The kind of document that would likely contain the answer to the original query. The factuality or accuracy of the document is not a concern. With that in mind, let's write the following prompt:

Since vector search typically operates on cosine vector similarity, the premise of HyDE is that we can achieve better results by matching documents to documents instead of queries to documents.

What we care about is structure, flow, and terminology. Not so much factuality. GPT-4o outputs a HyDE document like this:

It looks pretty believable, like the ideal candidate for the kinds of documents we'd like to index. We're going to embed this and use it for hybrid search.

Back to top

Hybrid search

This is the core of our search logic. Our lexical search component will be the generated OR clause strings. Our dense vector component will be embedded HyDE Document (aka the search vector). We use KNN to efficiently identify several candidate documents closest to our search vector. We call our lexical search component Scoring with TF-IDF and BM25 by default. Finally, the lexical and dense vector scores will be combined using the 30/70 ratio recommended by Wang et al.

Finally, we can piece together a RAG function. Our RAG, from query to answer, will follow this flow:

1. Convert Query to OR Clauses.

2. Generate HyDE document and embed it.

3. Pass both as inputs to Hybrid Search.

4. Retrieve top-n results, reverse them so that the most relevant score is the "most recent" in the LLM's contextual memory (Reverse Packing) Reverse Packing Example: Query: "Elasticsearch query optimization techniques" Retrieved documents (ordered by relevance): Reversed order for LLM context: By reversing the order, the most relevant information (1) appears last in the context, potentially receiving more attention from the LLM during answer generation.

   1. "Use bool queries to combine multiple search criteria efficiently."

   2. "Implement caching strategies to improve query response times."

   3. "Optimize index mappings for faster search performance."

   4. "Optimize index mappings for faster search performance."

   5. "Implement caching strategies to improve query response times."

   6. "Use bool queries to combine multiple search criteria efficiently."

5. Pass the context to the LLM for generation.

Let's run our query and get back our answer:

Nice. That's correct.

Back to top

Experiments

There's an important question to answer now. What did we get out of investing so much effort and additional complexity into these implementations?

Let's do a little comparison. The RAG pipeline we've implemented versus baseline hybrid search, without any of the enhancements we've made. We'll run a small series of tests and see if we notice any substantial differences. We'll refer to the RAG we have just implemented as AdvancedRAG, and the basic pipeline as SimpleRAG.

Summary of results

This table summarizes the results of five tests of both RAG pipelines. I judged the relative superiority of each method based on answer detail and quality, but this is a totally subjective judgement. The actual answers are reproduced below this table for your consideration. With that said, let's take a look at how they did!

SimpleRAG was unable to answer questions 1 & 5. AdvancedRAG also went into far greater detail on questions 2, 3, and 4. Based on the increased detail, I judged the quality of AdvancedRAG's answers better.

Test

Question

AdvancedRAG Performance

SimpleRAG Performance

AdvancedRAG Latency

SimpleRAG Latency

Winner

1

Who audits Elastic?

Correctly identified PwC as the auditor.

Failed to identify the auditor.

11.6s

4.4s

AdvancedRAG

2

What was the total revenue in 2023?

Provided the correct revenue figure. Included additional context with revenue from previous years.

Provided the correct revenue figure.

13.3s

2.8s

AdvancedRAG

3

What product does growth primarily depend on? How much?

Correctly identified Elastic Cloud as the key driver. Included overall revenue context & greater detail.

Correctly identified Elastic Cloud as the key driver.

14.1s

12.8s

AdvancedRAG

4

Describe employee benefit plan

Gave a comprehensive description of retirement plans, health programs, and other benefits. Included specific contribution amounts for different years.

Provided a good overview of benefits, including compensation, retirement plans, work environment, and the Elastic Cares program.

26.6s

11.6s

AdvancedRAG

5

Which companies did Elastic acquire?

Correctly listed recent acquisitions mentioned in the report (CmdWatch, Build Security, Optimyze). Provided some acquisition dates and purchase prices.

Failed to retrieve relevant information from the provided context.

11.9s

2.7s

AdvancedRAG

Test 1: Who audits Elastic?

AdvancedRAG

SimpleRAG

Summary: SimpleRAG did not identify PWC as the auditor

Okay that's actually quite surprising. That looks like a search failure on SimpleRAG's part. No documents related to auditing were retrieved. Let's dial down the difficulty a little with the next test.

Test 2: total revenue 2023

AdvancedRAG

SimpleRAG

Summary: Both RAGs got the right answer: $1,068,989,000 total revenue in 2023

Both of them were right here. It does seem like AdvancedRAG may have acquired a broader range of documents? Certainly the answer is more detailed and incorporates information from previous years. That is to be expected given the enhancements we made, but it's far too early to call.

Let's raise the difficulty.

Test 3: What product does growth primarily depend on? How much?

AdvancedRAG

SimpleRAG

Summary: Both RAGs correctly identified Elastic Cloud as the key growth driver. However, AdvancedRAG includes more detail, factoring in subscription revenues and customer growth, and explicitly mentions other Elastic offerings.

Test 4: Describe employee benefit plan

AdvancedRAG

SimpleRAG

Summary: AdvancedRAG goes into much greater depth and detail, mentioning the 401K plan for US-based employees, as well as defining contribution plans outside of the US. It also mentions Health and Well-Being plans but misses the Elastic Cares program, which SimpleRAG mentions.

Test 5: Which companies did Elastic acquire?

AdvancedRAG

SimpleRAG

Summary: SimpleRAG does not retrieve any relevant info about acquisitions, leading to a failed answer. AdvancedRAG correctly lists CmdWatch, Build Security, and Optimyze, which were the key acquisitions listed in the report.

Back to top

Conclusion

Based on our tests, our advanced techniques appear to increase the range and depth of the information presented, potentially enhancing quality of RAG answers.

Additionally, there may be improvements in reliability, as ambiguously worded questions such as Which companies did Elastic acquire? and Who audits Elastic were correctly answered by AdvancedRAG but not by SimpleRAG.

However, it is worth keeping in perspective that in 3 out of 5 cases, the basic RAG pipeline, incorporating Hybrid Search but no other techniques, managed to produce answers that captured most of the key information.

We should note that due to the incorporation of LLMs at the data preparation and query phases, the latency of AdvancedRAG is generally between 2-5x larger that of SimpleRAG. This is a significant cost which may make AdvancedRAG suitable only for situations where answer quality is prioritized over latency.

The significant latency costs can be alleviated using a smaller and cheaper LLM like Claude Haiku or GPT-4o-mini at the data preparation stage. Save the advanced models for answer generation.

This aligns with the findings of Wang et al. As their results show, any improvements made are relatively incremental. In short, simple baseline RAG gets you most of the way to a decent end-product, while being cheaper and faster to boot. For me, it's an interesting conclusion. For use cases where speed and efficiency are key, SimpleRAG is the sensible choice. For use cases where every last drop of performance needs squeezing out, the techniques incorporated into AdvancedRAG may offer a way forward.

Back to top

Appendix

Prompts

RAG question answering prompt

Prompt for getting the LLM to generate answers based on query and context.

Elastic query generator prompt

Prompt for enriching queries with synonyms and converting them into the OR format.

Potential questions generator prompt

Prompt for generating potential questions, enriching document metadata.

HyDE generator prompt

Prompt for generating hypothetical documents using HyDE

Sample hybrid search query

The recent paper Searching for Best Practices in Retrieval-Augmented Generation empirically assesses the efficacy of various RAG enhancing techniques, with the goal of converging on a set of best-practices for RAG.

We'll implement a few of these proposed best-practices, namely the ones which aim to improve the quality of search (Sentence Chunking, HyDE, Reverse Packing).

For brevity, we will omit those techniques focused on improving efficiency (Query Classification and Summarization).

We will also implement a few techniques that were not covered, but which I personally find useful and interesting (Metadata Inclusion, Composite Multi-Field Embeddings, Query Enrichment).

Finally, we'll run a short test to see if the quality of our search results and generated answers has improved versus the baseline. Let's get to it!

RAG overview

RAG aims to enhance LLMs by retrieving information from external knowledge bases to enrich generated answers. By providing domain-specific information, LLMs can be quickly adapted for use cases outside the scope of their training data; significantly cheaper than fine-tuning, and easier to keep up-to-date.

Measures to improve the quality of RAG typically focus on two tracks:

1. Enhancing the quality and clarity of the knowledge base.

2. Improving the coverage and specificity of search queries.

These two measures will achieve the goal of improving the odds that the LLM has access to relevant facts and information, and is thus less likely to hallucinate or draw upon its own knowledge - which may be outdated or irrelevant.

The diversity of methods is difficult to clarify in just a few sentences. Let's go straight to implementation to make things clearer.

Table of contents

• Overview

  • Table of contents

• Set-up

• Ingesting, processing, and embedding documents

  • Data ingestion

  • Sentence-level, token-wise chunking

  • Metadata inclusion and generation

    • Keyphrases extracted by TextRank

    • Potential questions generated by GPT-4o

    • Entities extracted by Spacy

  • Composite multi-field embeddings

    • Indexing to Elastic

• Cat break

• Appendix

  • Definitions

Set-up

All code may be found in the Searchlabs repo.

First things first. You will need the following:

1. An Elastic Cloud Deployment

2. An LLM API - We are using a GPT-4o deployment on Azure OpenAI in this notebook

3. Python Version 3.12.4 or later

We will be running all the code from the main.ipynb notebook.

Go ahead and git clone the repo, navigate to supporting-blog-content/advanced-rag-techniques, then run the following commands:

Once that's done, create a .env file and fill out the following fields (Referenced in .env.example). Credits to my co-author, Claude-3.5, for the helpful comments.

Next, we'll choose the document to ingest, and place it in the documents folder. For this article, we'll be using the Elastic N.V. Annual Report 2023. It's a pretty challenging and dense document, perfect for stress testing our RAG techniques.

Now we're all set, let's go to ingestion. Open main.ipynb and execute the first two cells to import all packages and intialize all services.

Back to top

Ingesting, processing, and embedding documents

Data ingestion

• Personal note: I am stunned by LlamaIndex's convenience. In the olden days before LLMs and LlamaIndex, ingesting documents of various formats was a painful process of collecting esoteric packages from all over. Now it's reduced to a single function call. Wild.

The SimpleDirectoryReader will load every document in the directory_path. For .pdf files, it returns a list of document objects, which I convert to Python dictionaries because I find them easier to work with.

Each dictionary contains the key content in the text field. It also contains useful metadata such as page number, filename, file size, and type.

Back to top

Sentence-level, token-wise chunking

The first thing to do is reduce our documents to chunks of a standard length (to ensure consistency and manageability). Embedding models have unique token limits (maximum input size they can process). Tokens are the basic units of text that models process. To prevent information loss (truncation or omission of content), we should provide text that does not exceed those limits (by splitting longer texts into smaller segments).

Chunking has a significant impact on performance. Ideally, each chunk would represent a self-contained piece of information, capturing contextual information about a single topic. Chunking methods include word-level chunking, where documents are split by word count, and semantic chunking which uses an LLM to identify logical breakpoints.

Word-level chunking is cheap, fast, and easy, but runs a risk of splitting sentences and thus breaking context. Semantic chunking gets slow and expensive, especially if you're dealing with documents like the 116-page Elastic Annual Report.

Let's choose a middleground approach. Sentence level chunking is still simple, but can preserve context more effectively than word-level chunking while being significantly cheaper and faster. Additionally, we'll implement a sliding window to capture some of the surrounding context, and alleviate the impact of splitting paragraphs.

The Chunker class takes in the embedding model's tokenizer to encode and decode text. We'll now build chunks of 512 tokens each, with an overlap of 20 tokens. To do this, we'll split the text into sentences, tokenize those sentences, and then add the tokenized sentences to our current chunk until we cannot add more without breaching our token limit.

Finally, decode the sentences back to the original text for embedding, storing it in a field called original_text. Chunks are stored in a field called chunk. To reduce noise (aka useless documents), we will discard any documents smaller than 50 tokens in length.

Let's run it over our documents:

And get back chunks of text that look like this:

Back to top

Metadata inclusion and generation

We've chunked our documents. Now it's time to enrich the data. I want to generate or extract additional metadata. This additional metadata can be used to influence and enhance search performance.

We'll define a DocumentEnricher class, whose role is to take in a list of documents (Python dictionaries), and a list of processor functions. These functions will run over the documents' original_text column, and store their outputs in new fields.

First, we extract keyphrases using TextRank. TextRank is a graph-based algorithm that extracts key phrases and sentences from text by ranking their importance based on the relationships between words.

Next, we'll generate potential_questions using GPT-4o.

Finally, we'll extract entities using Spacy.

Since the code for each of these is quite lengthy and involved, I will refrain from reproducing it here. If you are interested, the files are marked in the code samples below.

Let's run the data enrichment:

And take a look at the results:

Keyphrases extracted by TextRank

These keyphrases are a stand-in for the chunk's core topics. If a query has to do with cybersecurity, this chunk's score will be boosted.

Potential questions generated by GPT-4o

These potential questions may directly match with user queries, offering a boost in score. We prompt GPT-4o to generate questions which can be answered using the information found in the current chunk.

Entities extracted by Spacy

These entities serve a similar purpose to the keyphrases, but capture organizations' and individuals' names, which keyphrase extraction may miss.

Back to top

Composite multi-field embeddings

Now that we have enriched our documents with additional metadata, we can leverage this information to create more robust and context-aware embeddings.

Let's review our current point in the process. We've got four fields of interest in each document.

Each field represents a different perspective on the document's context, potentially highlighting a key area for the LLM to focus on.

The plan is to embed each of these fields, and then create a weighted sum of the embeddings, known as a Composite Embedding.

With luck, this Composite Embedding will allow the system to become more context aware, in addition to introducing another tunable hyperparameter from controlling the search behavior.

First, let's embed each field and update each document in place, using our locally defined embedding model imported at the beginning of the main.ipynb notebook.

Each embedding function returns the embedding's field, which is just the original input field with an _embedding postfix.

Let's now define the weightings of our composite embedding:

The weightings allow you to assign priorities to each component, based on your usecase and the quality of your data. Intuitively, the size of these weightings is dependent on the semantic value of each component. Since the chunk text itself is by far the richest, I assign a weighting of 70%. Since the entities are the smallest, being just a list of org or person names, I assign it a weighting of 5%. The precise setting for these values has to be determined empirically, on a use-case by use-case basis.

Finally, let's write a function to apply the weightings, and create our composite embedding. We'll delete all the component embeddings as well to save space.

With this, we've completed our document processing. We now have a list of document objects which look like this:

Indexing to Elastic

Let's bulk upload our documents to Elastic Search. For this purpose, I long-ago defined a set of Elastic Helper functions in elastic_helpers.py. It is a very lengthy piece of code so let's sticking to looking at the function calls.

es_bulk_indexer.bulk_upload_documents works with any list of dictionary objects, taking advantage of Elasticsearch's convenient dynamic mappings.

Head on over to Kibana and verify that all documents have been indexed. There should be 224 of them. Not bad for such a large document!

Back to top

Cat break

Let's take a break, article's a little heavy, I know. Check out my cat:

Adorable. The hat went missing and I half suspect she stole and hid it somewhere :(

Congrats on making it this far :)

Join me in Part 2 for testing and evaluation of our RAG pipeline!

Appendix

Definitions

1. Sentence Chunking

• A preprocessing technique used in RAG systems to divide text into smaller, meaningful units.

• Process:

  1. Input: Large block of text (e.g., document, paragraph)

  2. Output: Smaller text segments (typically sentences or small groups of sentences)

• Purpose:

  • Creates granular, context-specific text segments

  • Allows for more precise indexing and retrieval

  • Improves the relevance of retrieved information in RAG systems

• Characteristics:

  • Segments are semantically meaningful

  • Can be independently indexed and retrieved

  • Often preserves some context to ensure standalone comprehensibility

• Benefits:

  • Enhances retrieval precision

  • Enables more focused augmentation in RAG pipelines

2. HyDE (Hypothetical Document Embedding)

• A technique that uses an LLM to generate a hypothetical document for query expansion in RAG systems.

• Process:

  1. Input query to an LLM

  2. LLM generates a hypothetical document answering the query

  3. Embed the generated document

  4. Use the embedding for vector search

• Key difference:

  • Traditional RAG: Matches query to documents

  • HyDE: Matches documents to documents

• Purpose:

  • Improve retrieval performance, especially for complex or ambiguous queries

  • Capture richer semantic context than a short query

• Benefits:

  • Leverages LLM's knowledge to expand queries

  • Can potentially improve relevance of retrieved documents

• Challenges:

  • Requires additional LLM inference, increasing latency and cost

  • Performance depends on quality of generated hypothetical document

3. Reverse Packing

• A technique used in RAG systems to reorder search results before passing them to the LLM.

• Process:

  1. Search engine (e.g., Elasticsearch) returns documents in descending order of relevance.

  2. The order is reversed, placing the most relevant document last.

• Purpose:

  • Exploits the recency bias of LLMs, which tend to focus more on the latest information in their context.

  • Ensures the most relevant information is "freshest" in the LLM's context window.

• Example: Original order: [Most Relevant, Second Most, Third Most, ...] Reversed order: [..., Third Most, Second Most, Most Relevant]

4. Query Classification

• A technique to optimize RAG system efficiency by determining whether a query requires RAG or can be answered directly by the LLM.

• Process:

  1. Develop a custom dataset specific to the LLM in use

  2. Train a specialized classification model

  3. Use the model to categorize incoming queries

• Purpose:

  • Improve system efficiency by avoiding unnecessary RAG processing

  • Direct queries to the most appropriate response mechanism

• Requirements:

  • LLM-specific dataset and model

  • Ongoing refinement to maintain accuracy

• Benefits:

  • Reduces computational overhead for simple queries

  • Potentially improves response time for non-RAG queries

5. Summarization

• A technique to condense retrieved documents in RAG systems.

• Process:

  1. Retrieve relevant documents

  2. Generate concise summaries of each document

  3. Use summaries instead of full documents in the RAG pipeline

• Purpose:

  • Improve RAG performance by focusing on essential information

  • Reduce noise and interference from less relevant content

• Benefits:

  • Potentially improves relevance of LLM responses

  • Allows for inclusion of more documents within context limits

• Challenges:

  • Risk of losing important details in summarization

  • Additional computational overhead for summary generation

6. Metadata Inclusion

• A technique to enrich documents with additional contextual information.

• Types of metadata:

  • Keyphrases

  • Titles

  • Dates

  • Authorship details

  • Blurbs

• Purpose:

  • Increase contextual information available to the RAG system

  • Provide LLMs with clearer understanding of document content and relevance

• Benefits:

  • Potentially improves retrieval accuracy

  • Enhances LLM's ability to assess document usefulness

• Implementation:

  • Can be done during document preprocessing

  • May require additional data extraction or generation steps

7. Composite Multi-Field Embeddings

• An advanced embedding technique for RAG systems that creates separate embeddings for different document components.

• Process:

  1. Identify relevant fields (e.g., title, keyphrases, blurb, main content)

  2. Generate separate embeddings for each field

  3. Combine or store these embeddings for use in retrieval

• Difference from standard approach:

  • Traditional: Single embedding for entire document

  • Composite: Multiple embeddings for different document aspects

• Purpose:

  • Create more nuanced and context-aware document representations

  • Capture information from a wider variety of sources within a document

• Benefits:

  • Potentially improves performance on ambiguous or multi-faceted queries

  • Allows for more flexible weighting of different document aspects in retrieval

• Challenges:

  • Increased complexity in embedding storage and retrieval processes

  • May require more sophisticated matching algorithms

8. Query Enrichment

• A technique to expand the original query with related terms to improve search coverage.

• Process:

  1. Analyze the original query

  2. Generate synonyms and semantically related phrases

  3. Augment the query with these additional terms

• Purpose:

  • Increase the range of potential matches in the document corpus

  • Improve retrieval performance for queries with specific or technical language

• Benefits:

  • Potentially retrieves relevant documents that don't exactly match the original query terms

  • Can help overcome vocabulary mismatch between queries and documents

• Challenges:

  • Risk of query drift if not carefully implemented

  • May increase computational overhead in the retrieval process

Back to top

Imagine a multinational company with support tickets and solutions in different languages and wants to take advantage of that knowledge across divisions. Currently, knowledge is limited to the language the agent speaks. Let's fix that!

In this article, I’m going to show you how to test Mixtral’s language capabilities, by creating a multilingual RAG system.

You can follow the notebook to reproduce this article's example here

Steps

1. Creating embeddings endpoint

2. Creating mappings

3. Indexing data

4. Asking questions

Creating embeddings endpoint

Our support tickets for this example will come in English, Spanish, and German. The Mistral embeddings model is not multilingual, but we can generate multilingual embeddings using the e5 model, so we can index text on different languages and manage it as a single source, giving us a much richer context.

To create e5 multilingual embeddings you can use Kibana:

Or the _inference API:

Creating Mappings

For the mappings we will use semantic_text mapping type, which is one of my favorite features. It handles the process of chunking the data, generating embeddings, and querying embeddings for you!

We call the text field super_body because with a single mapping type it will handle chunks and embeddings.

Indexing data

We will index a couple of support tickets with problems and solutions in two languages, and then ask a question about problems within many documents in a third.

The following documents will be added to the index:

1. English Support Ticket: Calendar Sync Issue

Support Ticket #EN1234 Subject: Calendar sync not working with Google Calendar

Description: I'm having trouble syncing my project deadlines with Google Calendar. Whenever I try to sync, I get an error message saying "Unable to connect to external calendar service."

Resolution: The issue was resolved by following these steps:

1. Go to Settings > Integrations

1. Disconnect the Google Calendar integration

1. Clear browser cache and cookies

1. Reconnect the Google Calendar integration

1. Authorize the app again in Google's security settings

The sync should now work correctly. If problems persist, ensure that third-party cookies are enabled in your browser settings.

2. German Support Ticket: File Upload Problem

Support-Ticket #DE5678 Betreff: Datei-Upload funktioniert nicht

Beschreibung: Ich kann keine Dateien mehr in meine Projekte hochladen. Jedes Mal, wenn ich es versuche, bleibt der Ladebalken bei 99% stehen und dann erscheint eine Fehlermeldung.

Lösung: Das Problem wurde durch folgende Schritte gelöst:

1. Überprüfen Sie die Dateigröße. Die maximale Uploadgröße beträgt 100 MB.

1. Deaktivieren Sie vorübergehend den Virenschutz oder die Firewall.

1. Versuchen Sie, die Datei im Inkognito-Modus hochzuladen.

1. Wenn das nicht funktioniert, leeren Sie den Browser-Cache und die Cookies.

1. Als letzten Ausweg, versuchen Sie einen anderen Browser zu verwenden.

In den meisten Fällen lag das Problem an zu großen Dateien oder an Interferenzen durch Sicherheitssoftware. Nach Anwendung dieser Schritte sollte der Upload funktionieren.

3. Marketing Campaign Ideas (noise)

Q3 Marketing Campaign Ideas

1. Social media contest: "Share Your Productivity Hack"

   • Users share tips using our software, best entry wins a premium subscription.

1. Webinar series: "Mastering Project Management"

   • Invite industry experts to share insights using our tool.

1. Email campaign: "Unlock Hidden Features"

   • Series of emails highlighting lesser-known but powerful features.

1. Partner with a productivity podcast for sponsored content.

1. Create a "Project Management Memes" social media account for lighter, shareable content.

4. Mitarbeiter des Monats (noise)

Mitarbeiter des Monats: Juli 2023

Wir freuen uns, bekannt zu geben, dass Sarah Schmidt zur Mitarbeiterin des Monats Juli gewählt wurde!

Sarah hat außergewöhnliche Leistungen in folgenden Bereichen gezeigt:

• Kundenbetreuung: Sarah hat durchschnittlich 95% positive Bewertungen erhalten.

• Teamarbeit: Sie hat maßgeblich zur Verbesserung unseres internen Wissensmanagementsystems beigetragen.

• Innovation: Sarah hat eine neue Methode zur Priorisierung von Support-Tickets vorgeschlagen, die unsere Reaktionszeiten um 20% verbessert hat.

Bitte gratulieren Sie Sarah zu dieser wohlverdienten Anerkennung!

This is how a document will look like inside Elasticsearch:

Asking questions

Now, we are going to ask a question in Spanish:

The expectation is retrieving documents #1 and #2, then sending them to the LLM as additional context, and finally, getting an answer in Spanish.

Retrieving documents

To retrieve the relevant documents, we can use this nice and short query that will run a search on the embeddings, and return the support tickets most relevant to the question.

Notes about the parameters set: size: 2 Because we know we want the top 2 documents. excludes For clarity in the response. Documents are short so each one will be one chunk long.

Answering the question

Now we can call the Mistral completion API using the Python library to answer the question.

The answer is in perfect Spanish and on point!

Show answer

Hola, me alegra que te hayas comunicado con nosotros. Parece que hay dos problemas distintos.

En cuanto a la sincronización del calendario, puedes seguir estos pasos para resolver el problema:

1. Ve a Configuración > Integraciones

1. Desconecta la integración del Calendario de Google

1. Borra la caché y las cookies del navegador

1. Vuelve a conectar la integración del Calendario de Google

1. Autoriza de nuevo la aplicación en la configuración de seguridad de Google

Si sigues teniendo problemas, asegúrate de que las cookies de terceros están habilitadas en la configuración de tu navegador.

En cuanto al problema de subir un archivo, hay varias cosas que puedes probar:

1. Comprueba el tamaño del archivo. El tamaño máximo de carga es de 100 MB.

1. Desactiva temporalmente el antivirus o el cortafuegos.

1. Intenta cargar el archivo en modo incógnito.

1. Si eso no funciona, borra la caché y las cookies del navegador.

1. Como último recurso, prueba a usar un navegador diferente.

En la mayoría de los casos, el problema se debe a archivos demasiado grandes o a interferencias causadas por software de seguridad. Al seguir estos pasos, deberías poder cargar el archivo correctamente.

¡Espero que esto te ayude a resolver tus problemas! Si tienes alguna otra pregunta, no dudes en preguntar.

Conclusion

Mixtral 8x22B is a powerful model that enables us to leverage data sources in different languages, being able to answer, understand, and translate in many languages. This ability– together with multilingual embeddings– allows you to have multilingual support both in the data retrieval and the answer generation stages, removing language barriers entirely.

If you are interested on reproducing the examples of this article, you can find the Python Notebook with the requests here

This blog will walk through implementing RAG using two approaches.

1. Elastic, Llamaindex, Llama 3 (8B) version running locally using Ollama.

2. Elastic, Langchain, ELSER v2, Llama 3 (8B) version running locally using Ollama.

The notebooks are available at this GitHub location.

Before we get started, let's take a quick dive into Llama 3.

Llama 3 overview

Llama 3 is an open source large language model recently launched by Meta. This is a successor to Llama 2 and based on published metrics, is a significant improvement. It has good evaluation metrics, when compared to some of the recently published models such as Gemma 7B Instruct, Mistral 7B Instruct, etc. The model has two variants, which are the 8 billion and 70 billion parameter. An interesting thing to note is that at the time of writing this blog, Meta was still in the process of training 400B+ variant of Llama 3.

Meta Llama 3 Instruct Model Performance. (from https://ai.meta.com/blog/meta-llama-3/)

The above figure shows data on Llama3 performance across different datasets as compared to other models. In order to be optimized for performance for real world scenarios, Llama3 was also evaluated on a high quality human evaluation set.

Aggregated results of Human Evaluations across multiple categories and prompts (from https://ai.meta.com/blog/meta-llama-3/)

How to build RAG with Llama 3 open-source and Elastic

Dataset

For the dataset, we will use a fictional organization policy document in json format, available at this location.

Configure Ollama and Llama3

As we are using the Llama 3 8B parameter size model, we will be running that using Ollama. Follow the steps below to install Ollama.

1. Browse to the URL https://ollama.com/download to download the Ollama installer based on your platform.

Note: The Windows version is in preview at the moment.

1. Follow the instructions to install and run Ollama for your OS.

2. Once installed, follow the commands below to download the Llama3 model.

This should take some time depending upon your network bandwidth. Once the run completes, you should end with the interface below.

To test Llama3, run the following command from a new terminal or enter the text at the prompt itself.

At the prompt, the output looks like below.

We now have Llama3 running locally using Ollama.

Elasticsearch setup

We will use Elastic cloud setup for this. Please follow the instructions here. Once successfully deployed, note the API Key and the Cloud ID, we will require them as part of our setup.

Application setup

There are two notebooks, one for RAG implemented using Llamaindex and Llama3, the other one with Langchain, ELSER v2 and Llama3. In the first notebook, we use Llama3 as a local LLM as well as provide embeddings. For the second notebook, we use ELSER v2 for the embeddings and Llama3 as the local LLM.

Method 1: Elastic, Llamaindex, Llama 3 (8B) version running locally using Ollama.

Step 1 : Install required dependencies

The above section installs the required llamaindex packages.

Step 2: Import required dependencies

We start with importing the required packages and classes for the app.

We start with providing a prompt to the user to capture the Cloud ID and API Key values.

If you are not familiar with obtaining the Cloud ID and API Key, please follow the links in the code snippet above to guide you with the process.

Step 3: document processing

We start with downloading the json document and building out Document objects with the payload.

We now define the Elasticsearch vector store (ElasticsearchStore), the embedding created using Llama3 and a pipeline to help process the payload constructed above and ingest into Elasticsearch.

The ingestion pipeline allows us to compose pipelines using different components, one of which allows us to generate embeddings using Llama3.

ElasticsearchStore is defined with the name of the index to be created, the vector field and the content field. And this index is created when we run the pipeline.

The index mapping created is as below:

The pipeline is executed using the step below. Once this pipeline run completes, the index workplace_index is now available for querying. Do note that the vector field content_vector is created as a dense vector with dimension 4096. The dimension size comes from the size of the embeddings generated from Llama3.

Step 4: LLM configuration

We now setup Llamaindex to use the Llama3 as the LLM. This as we covered before is done with the help of Ollama.

Step 5: Semantic search

We now configure Elasticsearch as the vector store for the Llamaindex query engine. The query engine is then used to answer your questions with contextually relevant data from Elasticsearch.

The response I received with Llama3 as the LLM and Elasticsearch as the Vector database is below.

This concludes the RAG setup based on using Llama3 as a local LLM and to generate embeddings.

Let's now move to the second method, which uses Llama3 as a local LLM, but we use Elastic’s ELSER v2 to generate embeddings and for semantic search.

Method 2: Elastic, Langchain, ELSER v2, Llama 3 (8B) version running locally using Ollama.

Step 1: Install required dependencies

The above section installs the required langchain packages.

Step 2: Import required dependencies

We start with importing the required packages and classes for the app. This step is similar to Step 2 in Method 1 above.

Next, provide a prompt to the user to capture the Cloud ID and API Key values.

Step 3: Document processing

Next, we move to downloading the json document and building the payload.

This step differs from the Method 1 approach, from how we use the LlamaIndex provided pipeline to process the document. Here we use the RecursiveCharacterTextSplitter to generate the chunks.

We now define the Elasticsearch vector store ElasticsearchStore.

The vector store is defined with the index to be created and the model to be used for embedding and retrieval. You can retrieve the model_id by navigating to Trained Models under Machine Learning.

This also results in the creation of an ingest pipeline in Elastic, which generates and stores the embeddings as the documents are ingested into Elastic.

We now add the documents processed above.

Step 4: LLM configuration

We set up the LLM to be used with the following. This is again different from method 1, where we used Llama3 for embeddings too.

Step 5: Semantic search

The necessary building blocks are all in place now. We tie them up together to perform semantic search using ELSER v2 and Llama3 as the LLM. Essentially, Elasticsearch ELSER v2 provides the contextually relevant response to the users question using its semantic search capabilities. The user's question is then enriched with the response from ELSER and structured using a template. This is then processed with Llama3 to generate relevant responses.

The response with Llama3 as the LLM and ELSER v2 for semantic search is as below:

This concludes the RAG setup based on using Llama3 as a local LLM and ELSER v2 for semantic search.

Conclusion

In this blog we looked at two approaches to RAG with Llama3 and Elastic. We explored Llama3 as an LLM and to generate embeddings. Next we used Llama3 as the local LLM and ELSER for embeddings and semantic search. We utilized two different frameworks, LlamaIndex and Langchain. You could implement the two methods using either of these frameworks. The notebooks were tested with the Llama3 8B parameter version. Both the notebooks are available at this GitHub location.

This blog builds on that notebook and explores fetch surrounding chunking, an emerging pattern in RAG that uses intelligent chunking and Elasticsearch vector database to optimize LLM responses. The approach balances data input to enhance the accuracy and relevance of LLM-generated answers through semantic hybrid search.

The motivation: A refined approach to RAG data chunking

The primary motivation behind building this notebook was to demonstrate a refined approach to RAG by addressing the challenge of data chunking. Traditional methods often fall short in dynamically adjusting the data size fed to LLMs, either overwhelming the model with too much context or starving it with too little. This notebook aims to strike the right balance, providing just enough information for the LLM to generate precise and contextually relevant responses. However, it must be noted that there is no one-size-fits-all solution.

This method works especially well with books and similar texts where content flows within longer sections or chapters. However, it may require adaptation for texts structured into shorter, distinct sections, such as research papers or articles, where each segment might cover a different topic. In such cases, additional strategies may be necessary to effectively chunk and retrieve related content.

The methodology: Intelligent RAG data chunking

Fetch surrounding chunks

The core idea is to partition the source text into manageable chunks, ensuring each chunk contains just the right amount of information. For this demonstration, I used text from "Harry Potter and the Sorcerer's Stone." The text was partitioned into chapters, and each chapter was further divided into smaller chunks. These chunks, along with their dense and sparse (ELSER) vector representations, were indexed in the Elasticsearch vector database.

Assigning numbers to chunks

Each chunk within a chapter was assigned a sequential integer, allowing us to identify its position. When a matching chunk is found, the chapter number and chunk number are used to retrieve surrounding chunks, providing additional context for the LLM.

Vector database in Elasticsearch

These chunks and their vector representations were ingested into an Elasticsearch Cloud instance. Elasticsearch's robust vector search capabilities make it ideal for hosting these chunks, allowing for efficient retrieval of the most relevant chunks based on the semantic content or text match of a user's query.

AI search

To retrieve the relevant chunks, I employed a hybrid search strategy using dense vector comparisons, sparse vector comparisons, and text search in parallel. This multi-faceted approach ensures that the search results are both semantically rich and contextually accurate. A query is issued to find the matched chunk, which returns the chunk number and chapter. Surrounding chunks for that chapter are then fetched based on the matched chunk.

The RAG pattern

When a query is made, the search flow performs the following steps:

1. Query analysis: The user's query is translated into dense and sparse vectors to retrieve the most relevant chunks from the Elasticsearch index.

2. Chunk retrieval: Using the AI search strategy, the system retrieves the top relevant chunks.

3. Contextual expansion: Adjacent chunks (n-1 and n+1) are also retrieved to provide a more comprehensive context. If the chunk is the last in the chapter, it fetches n-1 and n-2; if it's the first, it fetches n+1 and n+2.

4. LLM response: These intelligently selected chunks are then fed into the LLM, ensuring it receives the optimal amount of information to generate a precise and contextually relevant response.

Why intelligent RAG data chunking matters

This approach addresses a critical aspect of RAG by optimizing the input data fed to LLMs. By leveraging intelligent chunking and hybrid semantic search, this method enhances the accuracy and relevance of the responses generated by LLMs. It showcases a pattern that can be widely applied in various applications within the RAG space, from customer support to content generation and beyond.

Conclusion

This notebook underscores the importance of intelligent data chunking in the RAG framework and demonstrates how Elasticsearch vector database can be leveraged to achieve optimal results. By ensuring the LLM receives just the right amount of information, this methodology paves the way for more accurate and contextually rich responses, enhancing the overall effectiveness of RAG systems.

This new functionality further solidifies our commitment to not only working with Microsoft and the Azure platform, but also toward our commitment to offering our customers more flexibility with their AI solutions.

Ongoing Investment in AI at Elastic

This is the latest in a series of additional features and integrations on AI enablement for Elasticsearch following on from:

• Elasticsearch open inference API adds Azure AI Studio support

• Elasticsearch open inference API adds support for Azure OpenAI chat completions

• Elasticsearch open inference API adds support for OpenAI chat completions

• Elasticsearch open inference API adds support for Cohere Embeddings

• Introducing Elasticsearch vector database to Azure OpenAI Service On Your Data (preview)

The new inference embeddings service provider for Azure OpenAI is already available in our stateless offering on Elastic Cloud, and will be soon available to everyone in an upcoming Elastic release.

Using Azure OpenAI Embeddings with the Elasticsearch Inference API

Deploying an Azure OpenAI Embeddings Model

To get started, you will need a Microsoft Azure Subscription as well as access to Azure OpenAI service. Once you have registered and have access, you will need to create a resource in your Azure Portal, and then deploy an embedding model to Azure OpenAI Studio. To do this, if you do not already have an Azure OpenAI resource in your Azure Portal, create a new one from the “Azure OpenAI” type which can be found in the Azure Marketplace, and take note of your resource name as you will need this later. When you create your resource, the region you choose may impact what models you have access to. See the Azure OpenAI deployment model availability table for additional details.

Once you have your resource, you will also need one of your API keys which can be found in the “Keys and Endpoint” information from the Azure Portal's left side navigation:

Now, to deploy your Azure OpenAI Embedding model, go into your Azure OpenAI Studio's console and create your deployment using an OpenAI Embeddings model such as text-embedding-ada-002. Once your deployment is created, you should see the deployment overview. Also take note of the deployment name, in the example below it is “example-embeddings-model”.

Using your deployed Azure OpenAI embeddings model with the Elasticsearch Inference API

With an Azure OpenAI embeddings model deployed, we can now configure your Elasticsearch deployment's _inference API and create a pipeline to index embeddings vectors in your documents. Please refer to the Elastic Search Labs GitHub repository for more in-depth guides and interactive notebooks.

To perform these tasks, you can use the Kibana Dev Console, or any REST console of your choice.

First, configure your inference endpoint using the create inference model endpoint - we'll call this “example_model”:

For your inference endpoint, you will need your API key, your resource name, and the deployment id that you created above. For the “api_version”, you will want to use an available API version from the Azure OpenAI embeddings documentation - we suggest always using the latest version which is “2024-02-01” as of this writing. You can also optionally add a username in the task setting's “user” field which should be a unique identifier representing your end-user to help Azure OpenAI to monitor and detect abuse. If you do not want to do this, omit the entire “task_settings” object.

After running this command you should receive a 200 OK status indicating that the model is properly set up.

Using the perform inference endpoint, we can see an example of your inference endpoint at work:

The output from the above command should provide the embeddings vector for the input text:

Now that we know our inference endpoint works, we can create a pipeline that uses it:

This will create an ingestion pipeline named “azureopenai_embeddings” that will read the contents of the “name” field upon ingestion and apply the embeddings inference from our model to the “name_embedding” output field. You can then use this ingestion pipeline when documents are ingested (e.g. via the _bulk ingest endpoint), or when reindexing an index that is already populated.

This is currently available through the open inference API in our stateless offering on Elastic Cloud. It'll also be soon available to everyone in an upcoming versioned Elasticsearch release, with additional semantic text capabilites that will make this step even simpler to integrate into your existing workflows.

For an additional use case, you can walk through the semantic search with inference tutorial for how to perform ingestion and semantic search on a larger scale with Azure OpenAI and other services such as reranking or chat completions.

Plenty more on the horizon

This new extensibility is only one of many new features we are bringing to the AI table from Elastic. Bookmark Elastic Search Labs now to stay up to date! Ready to build RAG into your apps? Want to try different LLMs with a vector database? Check out our sample notebooks for LangChain, Cohere and more on Github, and join the Elasticsearch Engineer training starting soon!

This blog quickly goes over the catalog of supported providers in the open inference API and explains how to use Azure’s OpenAI chat completions to answer questions through an example.

The inference API is growing…fast!

We’re heavily extending the catalog of supported providers in the open inference API. Check out some of our latest blog posts on Elastic Search labs to learn more about recent integrations around embeddings, completions and reranking:

• Elasticsearch open inference API adds support for Azure Open AI Studio

• Elasticsearch open inference API adds support for Azure Open AI embeddings

• Elasticsearch open inference API adds support for OpenAI chat completions

• Elasticsearch open Inference API adds support for Cohere’s Rerank 3 model

• Elasticsearch open inference API adds support for Cohere Embeddings

• ...more to come!

Azure OpenAI chat completions support is available through the open inference API in our stateless offering on Elastic Cloud. It’ll also be soon available to everyone in an upcoming versioned Elasticsearch release. This also complements the capability to use the Elasticsearch vector database in the Azure OpenAI service.

Using Azure’s OpenAI chat completions to answer questions

In my last blog post about OpenAI chat completions we’ve learned how to summarize text using OpenAI’s chat completions. In this guide we’ll use Azure OpenAI chat completions to answer questions during ingestion to have answers ready ahead of searching. Make sure you have your Azure OpenAI api key, deployment id and resource name ready by creating a free Azure account first and setting up a model suited for chat completions. You can follow Azure's OpenAI Service GPT quickstart guide to get a model up and running. In the following example we’ve used `gpt-4` with the version `2024-02-01`. You can read more about supported models and versions here.

In Kibana, you'll have access to a console for you to input these next steps in Elasticsearch without even needing to set up an IDE.

First, we configure a model, which will perform completions:

You’ll get back a response similar to the following with status code `200 OK` on successful inference creation:

You can now call the configured model to perform completion on any text input. Let’s ask the model what’s inference in the context of GenAI:

You should get back a response with status code `200 OK` explaining what inference is:

Now we can set up a small catalog of questions, which we want to be answered during ingestion. We’ll use the Bulk API to index three questions about products of Elastic:

You’ll get back a response with status `200 OK` back similar to the following upon successful indexing:

We’ll create now our question and answering ingest pipeline using the script-, inference- and remove-processor:

This pipeline prefixes the content with the instruction “Please answer the following question: “ in a temporary field named `prompt`. The content of this temporary `prompt` field will be sent to Azure’s OpenAI Service through the inference API to perform a completion. Using an ingest pipeline allows for immense flexibility as you can change the pre-prompt to anything you would like. This allows you to summarize documents for example, too. Check out Elasticsearch open inference API adds support for OpenAI chat completions to learn about how to build a summarisation ingest pipeline!

We now send our documents containing questions through the question and answering pipeline by calling the reindex API.

You'll get back a response with status 200 OK similar to the following:

In a real world setup you’ll probably use another ingestion mechanism to ingest your documents in an automated way. Check out our Adding data to Elasticsearch guide to learn more about the various options offered by Elastic to ingest data into Elasticsearch. We’re also committed to showcase ingest mechanisms and provide guidance on how to bring data into Elasticsearch using 3rd party tools. Take a look at Ingest Data from Snowflake to Elasticsearch using Meltano: A developer’s journey for example on how to use Meltano for ingesting data.

You're now able to search for your pre-generated answers using the Search API:

In the response you'll get back your pre-generated answers:

Pre-generating answers for frequently asked questions is particularly effective in reducing operational costs. By minimizing the need for on-the-fly response generation, you can significantly cut down on the amount of computational resources required like token usage. Additionally, this method ensures that every user receives the same, precise information. Consistency is crucial, especially in fields requiring high reliability and accuracy such as medical, legal, or technical support.

More to come!

We’re already working on adding support for more task types using Cohere, Google Vertex AI and many more. Furthermore we’re actively developing an intuitive UI in Kibana for managing Inference endpoints. Lots of exciting stuff to come! Bookmark Elastic Search Labs now to keep with Elastic’s innovations in the GenAI space!

Developers can use the capabilities of the world's most downloaded vector database to store and utilize embeddings generated from OpenAI models from Azure AI studio or access the wide array of chat completion model deployments for quick access to conversational models like mistral-small.

Just recently we've added support for Azure OpenAI text embeddings and completion, and now we've added support for utilizing Azure AI Studio. Microsoft Azure developers have complete access to Azure OpenAI & Microsoft Azure AI Studio service capabilities and can bring their Elasticsearch data to revolutionize conversational search.

Let's walk you through just how easily you can use these capabilities with Elasticsearch.

Deploying a model in Azure AI Studio

To get started, you'll need a Microsoft Azure subscription as well as access to Azure AI Studio. Once you are set up, you'll need to deploy either a text embedding model or a chat completion model from the Azure AI Studio model catalog. Once your model is deployed, on the deployment overview page take note of the target URL and your deployment's API key - you'll need these later to create your inference endpoint in Elasticsearch.

Furthermore, when you deploy your model, Azure offers two different types of deployment options - a “pay as you go” model (where you pay by the token), and a “realtime” deployment which is a dedicated VM that is billed by the hour. Not all models will have both deployment types available, so be sure to take note as well as which deployment type is used.

Creating an Inference API Endpoint in Elasticsearch

Once your model is deployed, we can now create an endpoint for your inference task in Elasticsearch. For the examples below we are using the Cohere Command R model to perform chat completion.

In Elasticsearch, create your endpoint by providing the service as “azureaistudio”, and the service settings including your API key and target from your deployed model. You'll also need to provide the model provider, as well as the endpoint type from before (either “token” or “realtime”). In our example, we've deployed a Cohere model with a token type endpoint.

When you send Elasticsearch the command, it should return back the created model to confirm that it was successful. Note that the API key will never be returned and is stored in Elasticsearch's secure settings.

Adding a model for using text embeddings is just as easy. For reference, if we had deployed the Cohere-embed-v3-english model, we can create our inference model in Elasticsearch with the “text_embeddings” task type by providing the appropriate API key and target URL from that deployment's overview page:

Let's perform some inference

That's all there is to setting up your model. Now that that's out of the way, we can use the model. First, let's test the model out by asking it to provide some text given a simple prompt. To do this, we'll call the _inference API with our input text:

And we should see Elasticsearch provide a response. Behind the scenes, Elasticsearch is calling out to Azure AI Studio with the input text and processes the results from the inference. In this case, we received the response:

We've tried to make it easy for the end user to not have to deal with all the technical details behind the scenes, but we can also control our inference a bit more by providing additional parameters to control the processing such as sampling temperature and requesting the maximum number of tokens to be generated:

That was easy. What else can we do?

This becomes even more powerful when we are able to use our new model in other ways such as adding additional text to a document when it's used in an Elasticsearch ingestion pipeline. For example, the following pipeline definition will use our model and anytime a document using this pipeline is ingested, any text in the field “question_field” will be sent through the inference API and the response will be written to the “completed_text_answer” field in the document. This allows large batches of documents to be augmented.

Limitless possibilities

By harnessing the power of Azure AI Studio deployed models in your Elasticsearch inference pipelines, you can enhance your search experience's natural language processing and predictive analytics capabilities.

In upcoming versions of Elasticsearch, users can take advantage of new field mapping types that simplify the process even further where designing an ingest pipeline would no longer be necessary. Also, as alluded to in our accelerated roadmap for semantic search the future will provide dramatically simplified support for inference tasks with Elasticsearch retrievers at query time.

These capabilities are available through the open inference API in our stateless offering on Elastic Cloud. It'll also be soon available to everyone in an upcoming versioned Elasticsearch release.

Elastic Cloud on Kubernetes (ECK) is a certified offering on Red Hat OpenShift. Elastic is an IBM partner, and IBM Watsonx Assistant and Watsonx Discovery use Elastic vector search for question-answering and retrieval augmentation use cases.

With this collaboration, Elasticsearch users can benefit from Red Hat OpenShift AI, a flexible, scalable MLOps platform for building, training, testing, and serving models for AI-enabled applications.

Elasticsearch vector database for generative AI and RAG apps

Elasticsearch Relevance Engine (ESRE) is a comprehensive suite of developer tools for building generative AI and RAG applications. ESRE incorporates a vector database that stores embeddings for text, image, and video data. ESRE’s native hybrid search can effectively combine results containing text, vectors, and geospatial data, with filtering, aggregations, and document-level security.

With ESRE, developers can implement vector search and semantic search, including k-nearest neighbors (kNN) and approximate nearest neighbor (ANN) search, along with support for both built-in and third-party natural language processing (NLP) models. ESRE also seamlessly integrates with key third-party ecosystem products from providers such as Cohere, LangChain, and LlamaIndex. Elasticsearch can be self-managed or deployed with Elastic Cloud.

As part of this collaboration, users are now able to leverage ESRE capabilities by downloading Elasticsearch directly from the Red Hat Ecosystem Catalog.

What is Red Hat OpenShift AI for generative AI apps

Red Hat OpenShift AI is a hybrid MLOps platform that brings IT, data science, and app dev teams together. Designed to simplify Generative AI application development and deployment, it provides a comprehensive infrastructure stack tailored for distributed workloads. This includes training, optimizing, fine-tuning, and deploying foundational and predictive AI models. Collaborating with model builders helps provide access to a variety of pre-built models. Developers and data scientists can work together on the same platform, greatly enhancing collaboration. The platform facilitates end-to-end AI lifecycle management—from model development and training to deployment, serving, and continuous monitoring.

• Model development: Conduct exploratory data science in JupyterLab with access to core AI / ML libraries and frameworks, including TensorFlow and PyTorch using our notebook images or your own.

• Model serving & monitoring: Deploy models across on-premise or any cloud, either in a fully managed or self-managed Red Hat OpenShift footprint and centrally monitor their performance.

• Lifecycle Management: Create repeatable data science pipelines for model training and validation and integrate them with DevOps pipelines for the delivery of models across your enterprise.

• Increased capabilities and collaboration: Create projects and share them across teams. Combine Red Hat components, open-source software, and ISV-certified software.

Get started with Red Hat and Elasticsearch

To get started, just follow the installation instructions provided in the Red Hat Ecosystem Catalog, and start building your next generative AI application with RAG!

Visit Elasticsearch Labs for articles and sample notebooks on vector search, RAG, and more.

Our mission at Elastic is to make Apache Lucene the best vector database out there, and to continue to make Elasticsearch the best retrieval platform out there for search and RAG. Our investments into Lucene are key to ensure that every release of Elasticsearch brings increasing faster performance and scale.

Customers are already building the next generation of AI enabled search applications with Elastic’s vector database and vector search technology. Roboflow is used by over 500,000 engineers to create datasets, train models, and deploy computer vision models to production. Roboflow uses Elastic vector database to store and search billions of vector embeddings.

In this blog we summarize recent enhancements and optimisations that significantly improve vector search performance in Elasticsearch and Apache Lucene, over and above performance gains delivered with Lucene 9.9 and Elasticsearch 8.12.x.

The integration of vector search into Elasticsearch relies on Apache Lucene, the layer that orchestrates data storage and retrieval. Lucene's architecture organizes data into segments, immutable units that undergo periodic merging. This structure allows for efficient management of inverted indices, essential for text search. With vector search, Lucene extends its capabilities to handle multi-dimensional points, employing the hierarchical navigable small world (HNSW) algorithm to index vectors.

This approach facilitates scalability, enabling data sets to exceed available RAM size while maintaining performance. Additionally, Lucene's segment-based approach offers lock-free search operations, supporting incremental changes and ensuring visibility consistency across various data structures. The integration however comes with its own engineering challenges. Merging segments requires recomputing HNSW graphs, incurring index-time overhead. Searches must cover multiple segments, leading to possible latency overhead. Moreover, optimal performance requires scaling RAM as data grows, which may raise resource management concerns.

Lucene's integration into Elasticsearch comes with the benefit of robust vector search capabilities. This includes aggregations, document level security, geo-spatial queries, pre-filtering, to full compatibility with various Elasticsearch features. Imagine running vector searches using a geo bounding box, this is an example usecase enabled by Elasticsearch and Lucene.

Lucene's architecture lays a solid foundation for efficient and versatile vector search within Elasticsearch. Let’s explore optimization strategies and enhancements we have implemented to integrate vector search into Lucene, which delivers a high performance and comprehensive feature-set for developers.

Harnessing Lucene's architecture for multi-threaded search

Lucene's segmented architecture enables the implementation of multi-threaded search capabilities. Elasticsearch’s performance gains come from efficiently searching multiple segments simultaneously. Latency of individual searches is significantly reduced by using the processing power of all available CPU cores. While it may not directly improve overall throughput, this enhancement prioritizes minimizing response times, ensuring that users receive their search results as swiftly as possible.

Furthermore, this optimization is particularly beneficial for Hierarchical Navigable Small World (HNSW) searches, as each graph is independent of the others and can be searched in parallel, maximizing efficiency and speeding up retrieval times even further.

The advantage of having multiple independent segments extends to the architectural level, especially in serverless environments. In this new architecture, the indexing tier is responsible for creating new segments, each containing its own HSNW graph. The search tier can simply replicate these segments without incurring the CPU cost of indexation. This separation allows a significant portion of compute resources to be dedicated to searches, optimizing overall system performance and responsiveness.

Accelerating multi-graph vector search

In spite of gains achieved with parallelization, each segment's searches would remain independent, unaware of progress made by other segment searches. So our focus shifted towards optimizing the efficiency of concurrent searches across multiple segments.

The graph shows that the number queries per second increased from 104 queries/sec to 219 queries/sec.

Recognizing the potential for further speedups, we leveraged our insights from optimizing lexical search, to enable information exchange among segment searches allowing for better coordination and efficiency in vector search.

Our strategy for accelerating multi-graph vector search revolves around balancing exploration and exploitation within the proximity graph. By adjusting the size of the expanded match set, we control the trade-off between runtime and recall, crucial for achieving optimal search performance across multiple graphs.

In multi-graph search scenarios, the challenge lies in efficiently navigating individual graphs, while ensuring comprehensive exploration to avoid local minima. While searching multiple graphs independently yields higher recall, it incurs increased runtime due to redundant exploration efforts. To mitigate this, we devised a strategy to intelligently share state between searches, enabling informed traversal decisions based on global and local competitive thresholds.

This approach involves maintaining shared global and local queues of distances to closest vectors, dynamically adapting search parameters based on the competitiveness of each graph's local search. By synchronizing information exchange and adjusting search strategies accordingly, we achieve significant improvements in search latency while preserving recall rates comparable to single-graph searches.

The impact of these optimizations is evident in our benchmark results. In concurrent search and indexing scenarios, we notice up to 60% reduction in query latencies! Even for queries conducted outside of indexing operations, we observed notable speedups and a dramatic decrease in the number of vector operations required. These enhancements, integrated into Lucene 9.10 and subsequently Elasticsearch 8.13, mark significant strides towards enhancing vector database performance for search while maintaining excellent recall rates.

Harnessing Java's latest advancements for ludicrous speed

In the area of Java development, automatic vectorization has been a boon, optimizing scalar operations into SIMD (Single Instruction Multiple Data) instructions through the HotSpot C2 compiler. While this automatic optimization has been beneficial, it has its limitations, particularly in scenarios where explicit control over code shape yields superior performance. Enter Project Panama Vector API, a recent addition to the JDK offering an API for expressing computations reliably compiled to SIMD instructions at runtime.

Lucene's vector search implementation relies on fundamental operations like dot product, square, and cosine distance, both in floating point and binary variants. Traditionally, these operations were backed by scalar implementations, leaving performance enhancements to the JIT compiler. However, recent advancements introduce a paradigm shift, enabling developers to express these operations explicitly for optimal performance.

Consider the dot product operation, a fundamental vector computation. Traditionally implemented in Java with scalar arithmetic, recent innovations leverage the Panama Vector API to express dot product computations in a manner conducive to SIMD instructions. This revised implementation iterates over input arrays, multiplying and accumulating elements in batches, aligning with the underlying hardware capabilities.

By harnessing Panama Vector API, Java code now interfaces seamlessly with SIMD instructions, unlocking the potential for significant performance gains. The compiled code, when executed on compatible CPUs, leverages advanced vector instructions like AVX2 or AVX 512, resulting in accelerated computations. Disassembling the compiled code reveals optimized instructions tailored to the underlying hardware architecture.

Microbenchmarks comparing traditional Java implementations to those leveraging Panama Vector API illustrate dramatic performance improvements. Across various vector operations and dimension sizes, the optimized implementations outperform their predecessors by significant margins, offering a glimpse into the transformative power of SIMD instructions.

Micro-benchmark comparing dot product with the new Panama API (dotProductNew) and the scalar implementation (dotProductOld).

Beyond microbenchmarks, the real-world impact of these optimizations is quite exciting to think about. Vector search benchmarks, such as SO Vector, demonstrate notable enhancements in indexing throughput, merge times, and query latencies. Elasticsearch, embracing these advancements, incorporates the faster implementations by default, ensuring users reap the performance benefits seamlessly.

The graph shows indexing throughput increased from about 900 documents/sec to about 1300 documents/sec.

Despite the incubating status of Panama Vector API, its quality and potential benefits are undeniable. Lucene's pragmatic approach allows for selective adoption of non-final JDK APIs, balancing the promise of performance improvements with maintenance considerations. With Lucene and Elasticsearch, users can leverage these advancements effortlessly, with performance gains translating directly to real-world workloads.

The integration of Panama Vector API into Java development yields a new era of performance optimization, particularly in vector search scenarios. By embracing hardware-accelerated SIMD instructions, developers can unlock efficiency gains, visible both in microbenchmarks and macro-level benchmarks. As Java continues to evolve, leveraging its latest features promises to propel performance to new heights, enriching user experiences across diverse applications.

Maximizing memory efficiency with scalar quantization

Memory consumption has long been a concern for efficient vector database operations, particularly for searching large datasets. Lucene introduces a breakthrough optimization technique - scalar quantization - aimed at significantly reducing memory requirements without sacrificing search performance.

Consider a scenario where querying millions of float32 vectors of high dimensions demands substantial memory, leading to significant costs. By embracing byte quantization, Lucene slashes memory usage by approximately 75%, offering a viable solution to the memory-intensive nature of vector search operations.

For quantizing floats to bytes, Lucene implements Scalar quantization a lossy compression technique that transforms raw data into a compressed form, sacrificing some information for space efficiency. Lucene's implementation of scalar quantization achieves remarkable space savings with minimal impact on recall, making it an ideal solution for memory-constrained environments.

Lucene's architecture, consisting of nodes, shards, and segments, which facilitates efficient distribution and management of documents for search. Each segment stores raw vectors, quantized vectors, and metadata, ensuring optimized storage and retrieval mechanisms.

Lucene's vector quantization adapts dynamically over time, adjusting quantiles during segment merge operations to maintain optimal recall. By intelligently handling quantization updates and re-quantization when necessary, Lucene ensures consistent performance while accommodating changes in data distribution.

Example of merged quantiles where segments A and B have 1000 documents and C only has 100.

Experimental results demonstrate the efficacy of scalar quantization in reducing memory footprint while maintaining search performance. Despite minor differences in recall compared to raw vectors, Lucene's quantized vectors offer significant speed improvements and recall recovery with minimal additional vectors.

Recall@10 for quantized vectors vs raw vectors. The search performance of quantized vectors is significantly faster than raw, and recall is quickly recoverable by gathering just 5 more vectors; visible by quantized@15.

Lucene's scalar quantization presents a revolutionary approach to memory optimization in vector search operations. With no need for training or optimization steps, Lucene seamlessly integrates quantization into its indexing process, automatically adapting to changes in data distribution over time. As Lucene and Elasticsearch continue to evolve, widespread adoption of scalar quantization will revolutionize memory efficiency for vector database applications, paving the way for enhanced search performance at scale.

Achieving seamless compression with minimal impact on recall

To make compression even better, we aimed to reduce each dimension from 7 bits to just 4 bits. Our main goal was to compress data further while still keeping search results accurate. By making some improvements, we managed to compress data by a factor of 8 without making search results worse. Here's how we did it.

We focused on keeping search results accurate while making data smaller. By making sure we didn't lose important information during compression, we could still find things well even with less detailed data. To make sure we didn't lose any important information, we added a smart error correction system.

We checked our compression improvements by testing them with different types of data and real search situations. This helped us see how well our searches worked with different compression levels and what we might lose in accuracy by compressing more.

Comparison of int4 dot product values to the corresponding float values for a random sample of 100 documents and their 10 nearest neighbors.

These compression features were created to easily work with existing vector search systems. They help organizations and users save space without needing to change much in their setup. With this simple compression, organizations can expand their search systems without wasting resources.

In short, moving to 4 bits per dimension for scalar quantization was a big step in making compression more efficient. It lets users compress their original vectors by 8 times. By optimizing carefully, adding error correction, testing with real data, and offering scalable deployment, organizations could save a lot of storage space without making search results worse. This opens up new chances for efficient and scalable search applications.

Paving the way for binary quantization

The optimization to reduce each dimension to 4 bits not only delivers significant compression gains but also lays the groundwork for further advancements in compression efficiency. Specifically, future advancements like binary quantization into Lucene, a development that has the potential to revolutionize vector storage and retrieval.

In an ongoing effort to push the boundaries of compression in vector search, we are actively working on integrating binary quantization into Lucene using the same techniques and principles that underpin our existing optimization strategies. The goal is to achieve binary quantization of vector dimensions, thereby reducing the size of the vector representation by a factor of 32 compared to the original floating-point format.

Through our iterations and experiments, we want to deliver the full potential of vector search while maximizing resource utilization and scalability. Stay tuned for further updates on our progress towards integrating binary quantization into Lucene and Elasticsearch, and the transformative impact it will have on vector database storage and retrieval.

Multi-vector integration in Lucene and Elasticsearch

Several real world applications rely on text embedding models and large text inputs. Most embedding models have token limits, which necessitate chunking of longer text into passages. Therefore, instead of a single document, multiple passages and embeddings must be managed, potentially complicating metadata preservation.

Now instead of having a single piece of metadata indicating, for example the first chapter of the book “Little Women”, you have to index that information data for every sentence.

Lucene's "join" functionality, integral to Elasticsearch's nested field type, offers a solution. This feature enables multiple nested documents within a top-level document, allowing searches across nested documents and subsequent joins with their parent documents. So, how do we deliver support for vectors in nested fields with Elasticsearch?

The key lies in how Lucene joins back to parent documents when searching child vector passages. The parallel concept here is the debate around pre-filtering versus post-filtering in kNN methods, as the timing of joining significantly impacts result quality and quantity. To address this, recent enhancements to Lucene enable pre-joining against parent documents while searching the HNSW graph.

Practically, pre-joining ensures that when retrieving the k nearest neighbors of a query vector, the algorithm returns the k nearest documents instead of passages. This approach diversifies results without complicating the HNSW algorithm, requiring only a minimal additional memory overhead per stored vector.

Efficiency is improved by leveraging certain restrictions, such as disjoint sets of parent and child documents and the monotonicity of document IDs. These restrictions allow for optimizations using bit sets, providing rapid identification of parent document IDs.

Searching through a vast number of documents efficiently required investing in nested fields and joins in Lucene. This work helps storage and search for dense vectors that represent passages within long texts, making document searches in Lucene more effective. Overall, these advancements represent an exciting step forward in the area of vector database retrieval within Lucene.

Wrapping up (for now)

We're dedicated to making Elasticsearch and Lucene the best vector database with every release. Our goal is to make it easier for people to search for things. With some of the investments we discuss in this blog, there is significant progress, but we're not done!

To say that the gen AI ecosystem is rapidly evolving is an understatement. At Elastic, we want to give developers the most flexible and open tools to keep up with all the innovation—with features available across recent releases until 8.13 and serverless

GCP Vector Search optimized instances: what you need to know

Elastic Cloud users benefit from having Elastic managed infrastructure across all major cloud providers (GCP, AWS and Azure) along with wide region support for GCP users. For more specific details on the instance configuration for this hardware profile, refer to our documentation for instance type: gcp.es.datahot.n2d.64x8x11

Vector Search, HNSW, and memory

Elasticsearch uses the Hierarchical Navigable Small World graph (HNSW) data structure to implement its Approximate Nearest Neighbor search (ANN). Because of its layered approach, HNSW's hierarchical aspect offers excellent query latency. To be most performant, HNSW requires the vectors to be cached in the node's memory. This caching is done automatically and uses the available RAM not taken up by the Elasticsearch JVM. Because of this, memory optimizations are important steps for scalability.

Consult our vector search tuning guide to determine the right setup for your vector search embeddings and whether you have adequate memory for your deployment.

With this in mind, the Vector Search optimized hardware profile is configured with a smaller than standard Elasticsearch JVM heap setting. This provides more RAM for caching vectors on a node, allowing users to provision fewer nodes for their vector search use cases.

If you’re using compression techniques like scalar quantization, the memory requirement is lowered by a factor of 4. To store quantized embeddings (available in versions Elasticsearch 8.12 and later) simply ensure that you’re storing in the correct element_type: byte. To utilize our automatic quantization of float vectors update your embeddings to use index type: int8_hnsw like in the following mapping example.

In upcoming versions, Elasticsearch will provide this as the default mapping, removing the need for users to adjust their mapping.

Combining this optimized hardware profile with Elasticsearch’s automatic quantization are two examples where Elastic is focused on vector search to be cost-effective while still being extremely performant.

Getting Started with Elastic Cloud vector search optimized profile for GCP

Start a free trial on Elastic Cloud and simply select the new Vector Search optimized profile to get started.

Migrating existing Elastic Cloud deployments

Migrating to this new Vector Search optimized hardware profile is a few clicks away. Simply navigate to your Elastic Cloud management UI, click to manage the specific deployment, and edit the hardware profile. In this example, we are migrating from a ‘Storage optimized’ profile to the new ‘Vector Search’ optimized profile. When choosing to do so, while there is a reduction to available storage and vCPU, what is gained is the ability to store more vectors per memory with vector search.

Migrating to a new hardware profile uses the grow and shrink approach for deployment changes. This approach adds new instances, migrates data from old instances to the new ones, and then shrinks the deployment by removing the old instances. This approach allows for high availability during configuration changes even for single availability zones.

The following image shows a typical architecture for a deployment running in Elastic Cloud, where vector search will be the primary use case.

This example deployment uses our new Vector Search optimized hardware profile, now available in GCP. This setup includes:

• Two data nodes in our hot tier with our vector search profile

• One Kibana node

• One Machine Learning node

• One integration server

• One master tiebreaker

By deploying these two “full-sized” data nodes with the Vector Search optimized hardware profile and while taking advantage of Elastic’s automatic dense vector scalar quantization, you can index roughly 60 million vectors, including one replica (with 768 dimensions).

Conclusion

Vector search is a powerful tool when building modern search applications, be it for semantic document retrieval on its own or integrating with an LLM service provider in a RAG setup. Elasticsearch provides a full-featured vector database natively integrated with a full-featured search platform. Along with improving vector search feature set and usability, Elastic continues to improve scalability. The vector search node type is the latest example, allowing users to scale their search application.

Elastic is committed to providing scalable, price effective infrastructure to support enterprise grade search experiences. Customers can depend on us for reliable and easy to maintain infrastructure and cost levers like vector compression, so you benefit from the lowest possible total cost of ownership for building search experiences powered by AI.

Custom GPTs

GPTs mark a significant advancement from the plugin system, offering an easier way for users to create custom versions of ChatGPT. Facilitated by an intuitive user interface, this enhancement simplifies the customization process, often eliminating the need for coding skills for a broad range of applications. Beyond basic personalization, those aiming to integrate ChatGPT with external data can do so through custom actions. Users have the option to share these tailored GPTs on the GPT store, keep them private for personal use, or only share them within your company’s workspace with the ChatGPT Team plan.

How ChatGPT communicates with Elasticsearch

1. ChatGPT initiates a call to the /search endpoint in the Cloud Run service.

2. The service takes this input to create an Elasticsearch search request.

3. The query response with the documentation body and URL are returned to the service.

4. The service returns the document body and URL in text form to the custom ChatGPT.

5. This response is then relayed back to the GPT in text form, ready for interpretation.

Again, this blog post assumes that you have set up your Elastic Cloud account, vectorized your content, and have an Elasticsearch cluster filled with data ready to be used. If you haven’t set all that up, see our previous post for detailed steps to follow.

Code

To bring our custom GPT to life, we create a service that acts as the intermediary between ChatGPT and our Elasticsearch data. The core of this service is a Python application which sets up a Quart app and defines the /search endpoint. In addition, we use a Dockerfile to facilitate the deployment of the app on Cloud Run.

The Python app connects to our Elastic Cloud cluster, executes a hybrid search combining BM25 and kNN queries, and returns the relevant documentation body and URL. This allows our custom GPT to access and utilize Elasticsearch data in real time.

For the complete code refer to the GitHub repository. This includes the Python app and the Dockerfile necessary for Cloud Run deployment.

Deploy a service

For detailed steps on deploying the service using Google Cloud Platform (GCP), refer to the deployment section in our previous blog post on ChatGPT Plugins. There, you’ll find a step-by-step guide for setting up and deploying your service on GCP.

Create a custom GPT

After logging into your ChatGPT Plus account, navigate to “My GPTs” via your profile to find the “Create a GPT” link. Alternatively, the “Explore GPTs” section above your conversations also leads to the GPT store, where you can find a link to create a GPT.

Configure the custom GPT

The GPT editor provides two ways to configure your GPT: the "Create" tab for a guided setup through conversational prompts and the "Configure" tab for direct configuration input. For configuring the Elastic Docs Assistant, we'll primarily use manual configuration to precisely define our GPT's settings.

Assign a name to your GPT, such as “Elastic Docs Assistant,” and add a brief description highlighting its function.

Under instructions, define the primary role of your GPT and provide it with instructions on how to present information:

Let’s switch to the “Create” tab and ask ChatGPT to generate conversation starters and a logo. Perhaps I’ll upload my own logo instead.

We won’t be uploading any knowledge files as all the data we use is in Elasticsearch. Instead, we'll define an action.

Define an action

This is where we connect our data to Elasticsearch. Clicking “Create a new action” will take us to the action editor.

First, I define my API key that I’m using in my endpoint service with a custom header name I set in my environment.

Then I copy in my OpenAPI specification:

Upon entering this information our schema will be automatically validated and display a search action, with any errors in red. If everything looks good, this is where the preview pane becomes particularly useful. Not only can you test the action to confirm its functionality, but the assistant also provides debugging information about the request. This is helpful for refining your GPT’s responses based on the service's response.

Further customization can be achieved by configuring the GPT instructions to dynamically modify its action requests, such as rewriting the user input before it's sent to the service or adding request query parameters based on some condition in the user input. This eliminates the need for traditional coding logic, assuming your endpoint is designed to support these modifications.

Publish the custom GPT

Click “Publish” in the top right corner above the preview pane to be taken to your newly created GPT.

What's next for custom GPTs

This exploration of Custom GPTs, leveraging Elasticsearch for dynamic, data-driven conversations, has only begun to reveal the potential of what's possible. By harnessing the power of ChatGPT's interface and connecting it to external data, we introduce a new dimension of customization and contextually rich interactions with state-of-the-art AI models.

You can try all of the capabilities discussed in this blog today! Get started by signing up for a free Elastic Cloud trial.

In this blog post, we may have used third party generative AI tools, which are owned and operated by their respective owners. Elastic does not have any control over the third party tools and we have no responsibility or liability for their content, operation or use, nor for any loss or damage that may arise from your use of such tools. Please exercise caution when using AI tools with personal, sensitive or confidential information. Any data you submit may be used for AI training or other purposes. There is no guarantee that information you provide will be kept secure or confidential. You should familiarize yourself with the privacy practices and terms of use of any generative AI tools prior to use.

Elastic, Elasticsearch and associated marks are trademarks, logos or registered trademarks of Elasticsearch N.V. in the United States and other countries. All other company and product names are trademarks, logos or registered trademarks of their respective owners.

So briefly, what is reranking? Rerankers take the ‘top n’ search results from existing vector search and keyword search systems, and provide a semantic boost to those results. With good reranking in place, you have better ‘top n’ results without requiring you to change your model or your data indexes – ultimately providing better search results you can send to large language models (LLMs) as context.

Recently, we collaborated with the Cohere team to make it easy for Elasticsearch developers to use Cohere’s embeddings (available in Elasticsearch 8.13 and Serverless!). It is a natural evolution to include Cohere’s incredible reranking capabilities to unlock all of the tools necessary for true refinement of results past the first-stage of retrieval.

Cohere’s Rerank 3 model can be added to any existing Elasticsearch retrieval flow without requiring any significant code changes. Given Elastic’s vector database and hybrid search capabilities, users can also bring embeddings from any 3rd party model to Elastic, to use with Rerank 3.

Elastic’s approach to hybrid search

When looking to implement RAG (Retrieval Augmented Generation), the strategy for retrieval and reranking is a key optimization for customers to ground LLMs and achieve accurate results. Customers have trusted Elastic for years with their private data, and are able to leverage several first-stage retrieval algorithms (e.g. for BM25/keyword, dense, and sparse vector retrieval). More importantly, most real-world search use cases benefit from hybrid search which we have supported since Elasticsearch 8.9.

For mid-stage reranking, we also offer native support for Learning To Rank and query rescore. In this walkthrough, we will focus on Cohere’s last stage reranking capabilities, and will cover Elastic’s mid stage reranking capabilities in a subsequent blog post!

Cohere’s approach to reranking

Cohere has seen phenomenal results with their new Rerank model. In the testing, Cohere is reporting that reranking models in particular benefit from long context. Chunking for model token limits is a necessary constraint when preparing your document for dense vector retrieval. But with Cohere’s approach for reranking, a considerable benefit to reranking can be seen based on context contained in the full document, rather than a specific chunk within the document. Rerank has a 4k token limit to enable the input of more context to unlock the full relevance benefits of incorporating this model into your Elasticsearch based search system.

(i) General retrieval based on BEIR benchmark; accuracy measured as nDCG@10

(ii) Code retrieval based on 6 common code benchmarks; accuracy measured as nDCG@10

(iii) Long context retrieval based on 7 common benchmarks; accuracy measured as nDCG@10

(iv) Semi-structured (JSON) retrieval based on 4 common benchmarks; accuracy measured as nDCG@10

If you’re interested in how to chunk with LangChain and LlamaIndex, we provide chat application reference code, integrations and more in Search Labs and our open source repository. Alternatively, you can leverage Elastic’s passage retrieval capabilities and chunk with ingest pipelines.

Building a RAG implementation with Elasticsearch and Cohere

Now that you have a general understanding of how these capabilities can be leveraged, let’s jump into an example on building a RAG implementation with Elasticsearch and Cohere.

You'll need a Cohere account and some working knowledge of the Cohere Rerank endpoint. If you’re intending to use Cohere’s newest generative model Command R+ familiarize yourself with the Chat endpoint.

In Kibana, you'll have access to a console for you to input these next steps in Elasticsearch even without an IDE set up. If you prefer to use a language client - you can revisit these steps in the provided guide.

Elasticsearch vector database

In an earlier announcement, we had some steps to get you started with the Elasticsearch vector database. You can review the steps to cover ingesting a sample books catalog, and generate embeddings using Cohere’s Embed capabilities by reading the announcement. Alternatively, if you prefer we also provide a tutorial and Jupyter notebook to get you started on this process.

Cohere reranking

The following section assumes that you’ve ingested data and have issued your first search. This will give you a baseline as to how the search results are ranked with your first dense vector retrieval.

The previous announcement concluded with a query issued against the sample books catalog, and, and generated the following results in response to the query string “Snow”. These results are returned in descending order of relevance.

You’ll next want to configure an inference endpoint for Cohere Rerank by specifying the Rerank 3 model and API key.

Once this inference endpoint is specified, you’ll now be able to rerank your results by passing in the original query used for retrieval, “Snow” along with the documents we just retrieved with the kNN search. Remember, you can repeat this with any hybrid search query as well!

To demonstrate this while still using the dev console, we’ll do a little cleanup on the JSON response above.

Take the hits from the JSON response and form the following JSON for the input, and then POST to the cohere_rerank endpoint we just configured.

And there you have it, your results have been reranked using Cohere's Rerank 3 model.

The books corpus that we used to illustrate these capabilities does not contain large passages, and is a relatively simple example. When instrumenting this for your own search experience, we recommend that you follow Cohere’s approach to populate your input with the context from the full documents returned from the first retrieved result set, not just a retrieved chunk within the documents.

Elasticsearch’s accelerated roadmap to semantic reranking and retrievers

In upcoming versions of Elasticsearch we will continue to build seamless support for mid and final stage rerankers. Our end goal is to enable developers to have the ability to use semantic reranking to improve the results from any search whether it is BM25, dense or sparse vector retrieval, or a combination with hybrid retrieval. To provide this experience, we are building a concept called retrievers into the query DSL. Retrievers will provide an intuitive way to execute semantic reranking, and will also enable direct execution of what you’ve configured in the open inference API in the Elasticsearch stack without relying on you to execute this in your application logic.

When incorporating the use of retrievers in the earlier dense vector example, this is how different the reranking experience can be:

(i) Elastic’s roadmap: The indexing step is simplified with the addition of Elastic’s future capabilities to automatically chunk indexed data

(ii) Elastic’s roadmap: The kNN retriever specifies the model (in this case Cohere’s Rerank 3) that was configured as an inference endpoint

(iii) Cohere’s roadmap: The step between sending the resulting data to Cohere’s Command R+ will benefit from a planned feature named extractive snippets which will enable the user to return a relevant chunk of the reranked document to the Command R+ model

This was our original kNN dense vector search executed on the books corpus to return the first set of results for “Snow”.

As explained in this blog, there are a few steps to retrieve the documents and pass on the correct response to the inference endpoint. At the time of this publication, this logic should be handled in your application code.

In the future, retrievers can be configured to use the Cohere rerank inference endpoint directly within a single API call.

In this case, the kNN query is exactly the same as my original, but the cleansing of the response before input to the rerank endpoint will no longer be a necessary step. A retriever will know that a kNN query has been executed and seamlessly rerank using the Cohere rerank inference endpoint specified in the configuration. This same principle can be applied to any search, BM25, dense, sparse and hybrid.

Retrievers as an enabler of great semantic reranking is on our active and near term roadmap.

Cohere’s generative model capabilities

Now you’re ready with a semantically reranked set of documents that can be used to ground the responses for the large language model of your choice! We recommend Cohere’s newest generative model Command R+. When building the full RAG pipeline, in your application code you can easily issue a command to Cohere’s Chat API with the user query and the reranked documents.

An example of how this might be achieved in your Python application code can be seen below:

This integration with Cohere is offered in Serverless and soon will be available to try in a versioned Elasticsearch release either on Elastic Cloud or on your laptop or self-managed environment. We recommend you use our Elastic Python client v0.2.0 against your Serverless project to get started!

Happy reranking!

Azure OpenAI Service On Your Data makes conversational experiences come alive for your employees, customers and users. With the addition of Elasticsearch vector database and vector search technology, LLMs are enriched by your business data, and conversations deliver superior quality responses out-of-the-box. All of this adds up to helping you better understand your data, and make more informed decisions.

Build powerful conversational chat experiences, fast

Business users, such as users on e-commerce teams, product managers, and others can add documents from an Elasticsearch index to build a conversational chat experience very quickly. All it takes is a few simple steps to configure the chat experience with parameters such as message history, and you're good to go! Customers can realize benefits pretty much right away..

• Quickly roll out conversational experiences to your users, customers, or employees--backed by context from your business data

• Common use cases include offering internal knowledge search, users self-service, or chatbots that help process common business workflows

How Elasticsearch vector database works with On Your Data

The new native experience within Azure OpenAI Studio makes adding an Elastic index a simple matter. Developers can pick Elasticsearch as their chosen vector database option from the drop-down menu..

You can bring your existing Elasticsearch indexes to On Your Data—whether those indexes live on Azure or on-prem. Just select Elasticsearch as your data source, add your Elastic endpoint and API key, add an Elastic index, and you're all set!

With the Elasticsearch vector database running in the background, users get all the Elastic advantages you'd expect.

• Precision of BM25 (text) search, the semantic understanding of vector search, and the best of both worlds with hybrid search

• Document and field level security, so users can only access information they're entitled to based on their permissions

• Filters, facets, and aggregations that add a real boost to how quickly relevant context is pulled from your organisation's data, and sent to an LLM

• Choice of leveraging a range of large language model providers, including Azure OpenAI, Hugging Face, or other 3rd party models

Elastic on Microsoft Azure: a proven combination

Elastic is a proud winner of the Worldwide Microsoft Partner of the Year award for Commercial Marketplace. Elastic and Microsoft customers have been using Elasticsearch and Azure OpenAI to build futuristic search experiences, that leverage the best of AI and machine learning, today.

Ali Dalloul, VP, Azure AI Customer eXperience Engineering had this to say about the collaboration, "By harnessing the power of Azure Cloud and OpenAI, Elastic is driving the development of AI-driven solutions that redefine customer experiences. This partnership is more than just a collaboration; it's a feedback loop of innovation, benefiting customers, Elastic, and Microsoft, while empowering the broader partner ecosystem. We're delighted to offer customers Elasticsearch's strong vector database and retrieval augmentation capabilities to store and search vector embeddings for On Your Data."

"This really helps customers connect data wherever it lives. We are happy to open the spectrum of building conversational AI solutions, agnostic to location, including Elasticsearch. We are excited to see how developers build upon this integration." Adds Pavan Li, Principal Product Manager of Azure OpenAI Service On Your Data.

Elastic's clear strengths in hybrid search--combining BM25/text search with vector search for semantic relevance, was an important differentiator. With the backing of the open source Apache Lucene community, Elastic's vector database has already been widely adopted by large companies for enterprise scale use cases.

Try On Your Data with Elasticsearch vector database today

Unlock the insights with conversational AI, using Elasticsearch and Azure OpenAI On Your Data today!

• Visit Azure OpenAI Studio to build your first conversational copilot

• Connect Elasticsearch with OpenAI models

• Read more on the Microsoft Tech Community blog

Search has evolved from simple text queries yielding straightforward results to a complex system accommodating various formats like text, images, videos, and questions.

Search not too long ago comprised of a text query and relevant results. Today's search results are enhanced with generative AI, machine learning, and interactive chat features, offering a richer, more dynamic, and contextually relevant user experience. Additionally, voice search and speech avatars have transformed traditional search, offering a more interactive and convenient user experience.

The desire for dialogue in search

In a realm where dialogue underpins every interaction, whether with fellow humans or bots, shouldn't our search experiences reflect this fundamental aspect? Envision the vast array of document corpora residing within an enterprise. Naturally, this environment sparks curiosity and a multitude of questions, leading to subsequent inquiries. This innate human trait drives us to seek answers, delve deeper following initial responses, and continuously explore. Yet, traditional question-and-answer mechanisms fall short, as they often disregard the context of preceding exchanges, leading to a disjointed and laborious process that feels unnatural and prompts users to disengage prematurely.

Beyond question and answer search

Consider the act of using a television to search for content, such as seeking action movies featuring Nicolas Cage. While most current systems adeptly provide relevant results, the inquiry rarely ends there. Subsequent questions, such as inquiring about the runtime or release dates of these movies, are a natural progression in our quest for information. However, standard search applications are not designed to facilitate a continuous dialogue; they are structured around isolated question-and-answer formats, which limits the depth of interaction and exploration.

Avatar assisted voice search experience

This is where the concept of an avatar-assisted search experience comes into play, especially in scenarios where users, myself included, prefer direct answers without the need to sift through information. Occasionally, we desire the convenience of having answers delivered to us, bypassing the effort of reading through content. The development of an avatar to generate responses could further modernize this interaction, providing a more engaging, efficient, and natural user experience.

Live demo: creating an avatar assisted voice search experience

This demo showcases a seamless integration of speech-to-text, Elasticsearch's semantic search capabilities, Azure OpenAI's RAG, and a synthesized avatar for responses.

Integration details

Speech to search

The advanced search experience begins with user voice interactions, which are converted into text by Azure Speech to Text, forming the basis of the search query. This query is then processed through Elasticsearch, using the ELSER, to retrieve relevant documents, such as TV guides listing “action movies featuring Nicolas Cage.” This ensures precision and relevance in the search results.

RAG & cache

In the enhanced search framework, merely fetching documents isn't enough. Azure OpenAI's GPT-4 refines raw data into understandable responses, ensuring smooth conversation flow. Additionally, Elasticsearch boosts efficiency as a GenAI caching layer, recycling answers for related queries, thus conserving resources. For example, if there's a cached response for "action movies featuring Nicolas Cage," the caching API will swiftly use this for similar questions like “Nicolas Cage high-intensity movies,” accelerating the search experience.

Avatar response generation

The experience is further enriched with an avatar response feature, powered by Azure Synthesizer, adding a visual and auditory dimension that surpasses traditional text-based interfaces. This creates a more engaging and interactive user experience, integrating various advanced technologies to deliver a dynamic, intuitive, and compelling search experience.

Summary

The shift from traditional Google searches to platforms like ChatGPT for answering queries illustrates a broader trend: our preference for dialogue over static information retrieval. This predilection underscores the importance for enterprises to adopt a more intuitive and conversational approach in their search functionalities. By embracing this paradigm, businesses can better align with the natural human inclination towards dialogue, thereby enhancing the overall search and discovery process within their data ecosystems.

Demo assets

Still curious, here is the link to the source code.

To the public, generative AI has seemingly appeared from nowhere. But if you dig deeper, you’ll note that the ideas underlying generative AI solutions trace their lineage back to inventions such as the Mark I perceptron in 1958 and neural networks in the late twentieth century.

Advancements in statistical techniques, the vast growth of publicly available data and advancements in Machine Learning (specifically the invention of the transformer-based neural network architecture) have led to the rise of models that contain billions of parameters or variables. To give a sense for the change in scale, the largest pre-trained model in 2019 was 330M parameters. Cohere's Command XL model, one of the leading models in Stanford’s Holistic Evaluation of Language Models (HELM) benchmark is trained on 52.4 billion parameters - ~1580x increase in size in just a few years.

Foundation Models (FMs) are ML models trained on massive quantities of structured and unstructured data, which can be fine-tuned or adapted for more specific tasks.

Large Language Models (LLMs) are a subset of FMs focused on understanding and generating human-like text. These models are ideal for needs such as translation, answering questions, summarizing information, and creating or identifying images.

LLMs can perform a wide range of tasks that span multiple domains, like writing blog posts, solving math problems, engaging in dialog, and answering questions based on a document. The size and general-purpose nature of FMs make them different from traditional ML models, which typically perform specific tasks, like analyzing text for sentiment, classifying images, and forecasting trends.

The primary goal of LLMs is to enable meaningful and engaging conversation between humans and machines and have become an immensely effective resource in countless industries, helping business to improve the customer experience.

Limitations of LLMs

LLMs have certain limitations. One notable constraint is that they are trained on general domain corpora, making them less effective on domain-specific tasks. There are scenarios when you want models to generate text based on specific data rather than generic data. For example, a health service provider company may want their chatbot to answer questions using the latest information stored in an enterprise document repository, so that the answers are specific to the health service provider’s business.

Also, these LLMs are trained offline until a knowledge cutoff date. It will be agnostic of any developments that have happened after the knowledge cutoff date. This may lead to inaccurate interpretations. For example, in 2021 San Francisco was the most expensive Bay Area City for renters. Today, it is Mountain view - source.

Approaches to enhance LLMs

There are two popular ways to reference contextual data in LLMs.

The first option is to fine-tune the base LLMs with contextual data. But, using this approach generating the correctly formatted information is time consuming. Also, it is costly to fine-tune a model. In addition to that, if the domain specific data is changing frequently, it would require frequent fine-tunings and retraining to provide accurate responses. This impacts time to market and also increases overall cost of the solution. In addition to that, not all LLMs provide an option to fine-tune.

To overcome these constraints, we can use a technique called Retrieval Augmented Generation (RAG). RAG is a process in which the model retrieves contextual documents from an external data source like Elasticsearch as part of its execution. These contextual documents are used in conjunction with the original input to produce an output.

Below are a few examples of how RAG can be used in various applications to improve the quality and relevance of generated content.

• Chatbot responses: In a chatbot system, RAG refers to combining a retrieval-based approach with a generative model. The retrieval component obtains relevant responses from a pre-defined database or knowledge base, while the generative model can add additional context or generate more fluent and diverse responses. This combination helps the chatbot provide more accurate and contextually appropriate answers to user queries.

• Content generation: RAG can be used in content generation tasks such as summarization or paraphrasing. The retrieval component can retrieve relevant sentences or paragraphs from existing documents or articles, and the generative model can then augment or rephrase the retrieved content to create new and original summaries or paraphrases.

• Recommendation systems: RAG can also be applied in recommendation systems. The retrieval component can retrieve a set of candidate items or products based on user preferences or history, and the generative model can then generate personalized recommendations or provide additional information about the recommended items to enhance the user’s decision-making process.

RAG using Elasticsearch and Cohere Command model through Amazon Bedrock

Why Cohere?

Cohere is the leading AI platform for enterprise. The company builds world-class LLMs that allow computers to search, understand meaning, and converse in text. Cohere's models are uniquely suited to the needs of business, providing ease of use and strong security and privacy controls across multiple deployment options. Companies can use the models out-of-the-box or tailor them to their particular needs using their own custom data.

Command is Cohere’s flagship text generation model. It is trained to follow user commands and to be instantly useful in practical business applications, such as text generation, summarization, RAG, and chat. Command ranks as one of the leading language models according to the Stanford’s HELM website an evaluation leaderboard comparing large language models on a wide number of tasks from Stanford University (March 2023 results). Customers can use Cohere's Command LLM through Amazon SageMaker Jumpstart and Amazon Bedrock.

Embed is Cohere’s representative model which translates text into numerical vectors that models can understand. Cohere provides industry-leading English and multilingual models (100+ languages) for a range of use cases, including semantic search, text classification, and semantic engine for RAG.

Why Elasticsearch?

To make the most of generative AI, it is essential to have a unified data platform where the organization's data is stored, making it easily (and safely) accessible and searchable in one centralized location.

Elasticsearch is a distributed, open source search and analytics engine for all types of data, including textual, numerical, geospatial, structured, and unstructured. Raw data from Enterprises flows into Elasticsearch from a variety of sources, including logs, system metrics, and web applications. Elasticsearch is built on top of Lucene and it excels at full-text search. Elasticsearch is fast and excels at delivering the most relevant responses to users.

In addition to full-text search, Elasticsearch also supports vector-search.

Vector search leverages ML to capture the meaning and context of unstructured data, including text and images, transforming it into a numeric representation. Frequently used for semantic search, vector search finds similar data using approximate nearest neighbor (ANN) algorithms. Compared to traditional keyword search, vector search yields more relevant results and executes faster. Users can enhance the search experience by combining vector search with filtering and aggregations to optimize relevance by implementing a hybrid search and combining it with traditional scoring.

Elasticsearch provides an easy-to-use and performant API to enable integration with other services. These features make Elasticsearch a preferred choice for enterprises to store business data and improve search experience.

Elasticsearch allows for the seamless integration of domain-specific context from the organization's data, thereby enhancing the performance and value of generative AI for achieving desired business objectives.

Why Amazon Bedrock?

Amazon Bedrock is a fully managed service that offers a choice of high-performing FMs from leading AI companies like AI21 Labs, Anthropic, Cohere, Meta, Stability AI, and Amazon with a single API, along with a broad set of capabilities you need to build generative AI applications, simplifying development while maintaining privacy and security. With the comprehensive capabilities of Amazon Bedrock, you can easily experiment with a variety of top FMs, privately customize them with your data using techniques such as fine-tuning and RAG, and create managed agents that execute complex business tasks—from booking travel and processing insurance claims to creating ad campaigns and managing inventory—all without writing any code. Since Amazon Bedrock is serverless, you don't have to manage any infrastructure, and you can securely integrate and deploy generative AI capabilities into your applications using the AWS services you are already familiar with.

Amazon Bedrock offers several capabilities to support security and privacy requirements and has achieved HIPAA eligibility and GDPR compliance. With Amazon Bedrock, content is not used to improve the base models and is not shared with third-party model providers. Data in Amazon Bedrock is always encrypted in transit and at rest, and can encrypt the data using your own keys. AWS PrivateLink can be used with Amazon Bedrock to establish private connectivity between FMs and your Amazon Virtual Private Cloud (Amazon VPC) without exposing your traffic to the Internet.

Solution overview

Here's how to use RAG to enable Generative AI capabilities on domain-specific business data using Elasticsearch and Cohere Generate Model -Command through Amazon Bedrock.

The below architecture diagram explains how to get domain-specific responses from Cohere Command Model through Amazon Bedrock using enterprise data hosted in Elastic Enterprise Search using a technique called RAG.

Figure 1. RAG Architecture using Elasticsearch and Amazon Bedrock

Step by step explanation

Offline Data Ingestion:

i. The documents are ingested using web crawler or any other ingestion mechanism.

ii. The Elastic Learned Sparse EncodeR (ELSER) model will embed the text and store the resulting tokens in the Elasticsearch Index

Real-time flow on user query:

1. The User provides a question via the RAG web application

2. The RAG application generates a retrieval request initialized from the vector store (Elasticsearch index). At query time, the text will be embedded using the ELSER model and the resulting tokens will be used to perform a text expansion query.

3. The Retriever component of RAG application fetches the relevant documents from Elasticsearch vector store.

4. The RAG application passes the retrieved documents (context) along with user question (prompt) to the Cohere Command Model through Amazon Bedrock

5. The Cohere Command Model through Amazon Bedrock generates a textual response and sends it back to RAG application

6. The RAG application performs any required post processing tasks. For example, it adds source to the response generated from Cohere Command Model. The User views the response in the web application.

We used the following AWS and third-party services:

1. Amazon Bedrock for interacting with LLMs from Cohere.

2. Cohere Command Model for Text Generation.

3. Elasticsearch for storing embeddings of the enterprise knowledge corpus and doing similarity search with user questions.

4. Python, LangChain and Streamlit for building the RAG application

5. Amazon EC2 for hosting the Streamlit application

6. AWS Identity and Access Management roles and policies for access management.

Prerequisites:

1. Sign up for a free trial of Elasticsearch cluster with Elastic Cloud

2. Create a new deployment on AWS following the steps

3. Reset and download the elastic user password following these steps

4. Copy the Cloud ID from the My Deployment page listed under Deployments

5. Deploy ELSER Model: In Kibana, navigate to Machine Learning> Trained models. ELSER can be found in the list of trained models. Click the Download model button under Actions. After the download is finished, start the deployment by clicking the Start deployment button. Go to the Elastic ELSER page to find more details.

1. Install packages and import modules: Firstly, we need to install modules. Make sure python is installed with min version 3.8.1. Then we need to import modules.

1. Initialize Amazon Bedrock client using the following code

1. Connect to Elasticsearch using Elastic Cloud Id, Elastic username and Elastic password. Use ElasticsearchStore to connect to our elastic cloud deployment. As we’re using ELSER we use “SparseVectorRetrievalStrategy”. This strategy uses Elasticsearch’s sparse vector retrieval to retrieve the top-k results.

1. Download the dataset, deserialize the document and split the document into passages. We’ll chunk the documents into passages in order to improve the retrieval specificity and to ensure that we can provide multiple passages within the context window of the final question answering prompt. Here we are chunking into 800 tokens with an overlap of 400 tokens. Here, we are using a simple splitter but LangChain offers more advanced splitters to reduce the chance of context being lost.

1. Index data to Elasticsearch using ElasticsearchStore.from_documents.

1. Initialize the Amazon Bedrock LLM. In the Amazon Bedrock instance, will pass bedrock_client and specific model_id. In this case model_id = cohere.command-text-v14.

1. Asking a question: Now that we have the passages stored in Elasticsearch and LLM is initialized, we can now ask a question to get the relevant passages.

1. (Optional) You can also add a reranking step to the search pipeline which can further improve the ranking of the results returned in the search step. See the Deploying with Amazon SageMaker guide on using Rerank.

Conclusion

In this post, we showed how to create a RAG web application using a combination of Elasticsearch, Amazon Bedrock, Cohere Command Model and open source python packages like LangChain and Streamlit.

We encourage you to learn more by exploring Amazon SageMaker Jumpstart, Amazon Bedrock, Cohere, and Elasticsearch and building a solution using the sample implementation provided in this post and a dataset relevant to your business. If you have questions or suggestions, leave a comment.

The blog attempts to describe some of the basic processes for storing and retrieving knowledge from LLMs. Followup blogs will describe different RAG strategies in more detail.

Pre-training

Fine-tuning

Retrieval Augmented Generation

Training duration

Days to weeks to months

Minutes to hours

Not required

Customisation

Requires large amount of domain training data Can customise model architecture, size. tokenizer etc. Creates new “foundation” LLM model

Add domain-specific data Tune for specific tasks. Updates LLM model.

No model weights. External information retrieval system can be tuned to align with LLM. Prompt can be optimised for task performance.

Objective

Next-token prediction

Increase task performance

Increase task performance for specific set of domain documents

Expertise

High

Medium

Low

Introduction to domain specific generative AI

Generative AI technologies, built on large language models (LLMs), have substantially progressed our ability to develop tools for processing, comprehending, and generating text. Furthermore, these technologies have introduced an innovative information retrieval mechanism, wherein generative AI technologies directly respond to user queries using the stored (parametric) knowledge of the model.

However, it's important to note that the parametric knowledge of the model is a condensed representation of the entire training dataset. Thus, employing these technologies for a specific knowledge base or domain beyond the original training data does come with certain limitations, such as:

• The generative AI's responses might lack context or accuracy, as they won't have access to information that wasn't present in the training data.

• There is potential for generating plausible-sounding but incorrect or misleading information (hallucinations).

Different strategies exist to overcome these limitations, such as extending the original training data, fine-tuning the model, and integrating with an external source of domain-specific knowledge. These various approaches yield distinct behaviours and carry differing implementation costs.

Strategies for integrating domain-specific knowledge into LLMs

Domain specific pre-training for LLMs

LLMs are pre-trained on huge corpora of data that represent a wide range of natural language use cases:

Model

Total dataset size

Data sources

Training cost

PaLM 540B

780 billion tokens

Social media conversations (multilingual) 50%; Filtered web pages (multilingual) 27%; Books (English) 13%; GitHub (code) 5%; Wikipedia (multilingual) 4%; News (English) 1%

8.4M TPU v2 hours

GPT-3

499 billion tokens

Common Crawl (filtered) 60%; WebText2 22%; Books1 8%; Books2 8%; Wikipedia 3%

0.8M GPU hours

LLaMA 2

2 trillion tokens

“mix of data from publicly available sources”

3.3M GPU hours 

The costs of this pre-training step are substantial, and there’s a significant amount of work required to curate and prepare the datasets. Both of these tasks require a high level of technical expertise.

In addition, pre-training is only one step in creating the model. Typically, the models are then fine-tuned on a narrower dataset that is carefully curated and tailored for specific tasks. This process also typically involves human reviewers that rank and review possible model outputs to improve the model’s performance and safety. This adds further complexity and cost to the process.

Examples of this approach applied to specific domains include:

• ESMFold, ProGen2 and others - LLM for protein sequences: protein sequences can be represented using language-like sequences but are not covered by natural language models

• Galactica - LLM for science: trained exclusively on a large collection of scientific datasets, and includes special processing to handle scientific notations

• BloombergGPT - LLM for finance: trained on 51% financial data, 49% public datasets

• StarCoder - LLM for code: trained on 6.4TB of permissively licensed source code in 384 programming languages, and included 54 GB of GitHub issues and repository-level metadata

The domain-specific models generally outperform generalist models within their respective domains, with the most significant improvements observed in domains that differ significantly from natural language (such as protein sequences and code). However, for knowledge-intensive tasks, these domain-specific models suffer from the same limitations due to their reliance on parametric knowledge. Therefore, while these models can understand the relationships and structure of the domain more effectively, they are still prone to inaccuracies and hallucinations.

Domain specific fine-tuning for LLMs

Fine-tuning for LLMs involves training a pre-trained model on a specific task or domain to enhance its performance in that area. It adapts the model's knowledge to a narrower context by updating its parameters using task-specific data, while retaining its general language understanding gained during pre-training. This approach optimises the model for specific tasks, saving significant time compared to training from scratch.

Examples

• Alpaca - fine-tuned LLaMA-7B model that behaves qualitatively similarly to OpenAI’s GPT-3.5

• xFinance - fine-tuned LLaMA-13B model for financial-specific tasks. Reportedly outperforms BloombergGPT

• ChatDoctor - fine-tuned LLaMA-7B model for medical chat.

• falcon-40b-code-alpaca - fine-tuned falcon-40b model for code generation from natural language

Costs: fine-tuning vs. pre-training LLMs

Costs for fine-tuning are significantly smaller than for pre-training. In addition, novel methods such as parameter-efficient fine-tuning (PEFT) methods (e.g. LoRA, adapters, prompt tuning, and in-context learning as described above) enable very efficient adaptation of pre-trained language models (PLMs) to various downstream applications without fine-tuning all the model's parameters. For example,

Model

Fine-tuning method

Fine-tuning dataset

Cost

Alpaca 

Self-Instruct

52K unique instructions and the corresponding outputs

3 hours on 8 80GB A100s:24 GPU hours

xFinance

Unsupervised fine-tuning and instruction fine-tuning using xTuring library

493M token text dataset; 82K instruction dataset

25 hours on 8 A100 80GB GPUs:200 GPU hours

ChatDoctor

Self-Instruct

110K patient-doctor interactions

3 hours on 6 A100 GPUS: 18 GPU hours

falcon-40b-code-alpaca

Self-Instruct

52K instruction dataset; 20K instruction-input-code triplets

4 hours on 4 A100 80GB GPUs: 16 GPU hours

Similar to domain-specific pre-trained models, these models typically exhibit better performance within their respective domains, yet they still face the limitations associated with parametric knowledge.

Enhancing LLMs with Retrieval Augmented Generation (RAG)

LLMs store factual knowledge in their parameters, and their ability to access and precisely manipulate this knowledge is still limited. This can lead to LLMs providing non-factual but seemingly plausible predictions (hallucinations) - particularly for unpopular questions. Additionally, providing references for their decisions and updating their knowledge efficiently remain open research problems.

A general purpose recipe to address these limitations is RAG, where the LLM's parametric knowledge is grounded with external or non-parametric knowledge from an information retrieval system. This knowledge is passed as additional context in the prompt to the LLM and specific instructions are given to the LLM on how to use this contextual information. This keeps it more inline with the discussion so far about parametric knowledge. The advantages of this approach are:

• Unlike fine-tuning and pre-training, LLM parameters do not change and so there are no training costs

• Expertise required to simple implementation is low (although more advanced strategies exist)

• Response can be tightly constrained to context returned from the information retrieval system, limiting hallucinations

• Smaller task specific LLMs can be used - as the LLM is being used for a specific task rather than a knowledge base.

• Knowledge base is easily updatable as it requires no changes to the LLM

• Responses can cite sources for human verification and link outs

Strategies to combine this non-parametric knowledge (i.e. retrieved text) with an LLM’s parametric knowledge is an active area of research.

Some of these approaches involve modifying the LLM in conjunction with the retrieval strategy and so can not be classified as distinctly as the definitions in this blog. We will dive into more details in further blogs.

Example

In a simple example, we utilised a fine-tuned LLaMA2 13B model, which was based on the information from this blog. This model underwent fine-tuning using AWS blog posts published after the LLaMA2 pre-training and fine-tuning data cutoff date, specifically those from July 23rd, 2023. We also ingested these documents into a Elasticsearch and established a simple RAG pipeline. In this pipeline, model responses are generated based on the retrieved documents serving as context. Red highlights indicate incorrect responses, and blue highlights correct responses.

However, it's important to note that this is just a single example and does not constitute a comprehensive evaluation of fine-tuning versus RAG, but provides an example of fine-tuning before useful for form, not facts.. We plan to conduct more thorough comparisons in upcoming blogs.

First things first, ChatGPT is awesome! It can help you work more efficiently — from summarizing a 10,000-word document to providing a list of differentiations between competing products, as well as many other tasks.

ChatGPT is the best known large language model (LLM) based on the transformer architecture. But there are other LLMs that you may have heard of, including BERT (Bidirectional Encoder Representation from Transformer), Bard (Language Model for Dialogue Applications), or LLaMA (LLM Meta AI). LLMs have multiple layers of neural networks that work together to analyze text and predict outputs. They’re trained with a left-to-right or bidirectional transformer that maximizes the probability of following and preceding words in context to figure out what might come next in a sentence. LLMs also have an attention mechanism that allows them to focus selectively on parts of text in order to identify the most relevant sections. For example, Rex is adorable and he is a cat. “He”, in this sentence, refers to “Cat” and “Rex.”

Understanding Large Language Models (LLMs)

Large Language Models are generally compared by the number of parameters — and bigger is better. The number of parameters is a measure of the size and the complexity of the model. The more parameters a model has, the more data it can process, learn from, and generate. However, having more parameters also means having more computational and memory resource demands. Parameters are learned or updated during the training process by using an optimization algorithm that tries to minimize the error or the loss between the predicted outputs and the actual outputs. By adjusting the parameters, the model can improve its performance and accuracy on a given task or domain.

LLMs are expensive to train

Modern LLMs have billions of parameters that are trained on trillions of tokens and cost millions of dollars. Training an LLM includes identifying a data set, making sure the data set is large enough for it to perform functions like a human, determining the network layer configurations, using supervised learning to learn the information in the data set, and finally, fine-tuning. Needless to say, retraining LLMs on domain specific data is also very expensive.

How does a GPT model work?

A Generative pre-trained transformer (GPT) model is a type of neural network that uses the transformer architecture to learn from large amounts of text data. The model has two main components: an encoder and a decoder. The encoder processes the input text and converts it into a sequence of vectors called embeddings that represent the meaning and context of each word/subword in numerics. However, the decoder generates the output text by predicting the next word in the sequence based on the embeddings and the previous words.

The GPT model uses a technique called “attention” to focus on the most relevant parts of the input and output texts and to capture long-range dependencies and relationships between words. The model is trained by using a large corpus of texts as both the input and the output and by minimizing the difference between the predicted words and the actual words. It can then be fine-tuned or adapted to specific tasks or domains by using smaller and more specialized data sets.

Tokens

Tokens are the basic units of text or code that an LLM uses to process and generate language. Tokens can be characters, words, subwords, or other segments of text or code, depending on the chosen tokenization method or scheme. They are assigned numerical values or identifiers and are arranged in sequences or vectors, then are fed into or outputted from the model. Tokenization is the process of splitting the input and output texts into smaller units that can be processed by the LLM models.

For example, the sentence “A quick brown fox jumps over a lazy dog” can be tokenized into the following tokens: “a,” “quick,” “brown,” “fox,” “jumps,” “over,” “a,” “lazy,” and “dog.”

Embedding

Embeddings are vectors or arrays of numbers that represent the meaning and the context of the tokens that the model processes and generates. They are derived from the parameters of the model and are used to encode and decode the input and output texts. Embeddings help the model to understand the semantic and syntactic relationships between the tokens and to generate more relevant and coherent texts. They are essential components of the transformer architecture that GPT-based models use. They can also vary in size and dimension, depending on the model and the task.

At a minimum, pre-trained LLMs contain embedding for tens of thousands of words, tokens, and terms. For example, ChatGPT-3 has a vocabulary of 14,735,746 words and a dimension of 1,536. The following is from a small model called distilbert-based-uncased. Despite the fact that this is a small model, it is still 100s mb in size.

Transformer

A transformer model is a neural network that learns context or meaning by tracking relationships in sequential data like the words in this sentence. In its simplest form, a transformer will take an input and predict an output. Within the transformer, there is an encoder stack and a decoder stack.

Let’s dig into the encoder block and the decoder block. In the encoder block, there are two important components: the self-attention neural network and the feed-forward neural network.

The self-attention layer is crucial as it builds in the “understanding” of the current token from the previous words that are relevant to the current one. For example, “it” refers to the chicken in “the chicken crossed the road because it wants to know what the jokes are all about.”

The other important layer in an encoder is the feed-forward neural network (FFNN). FFNN predicts what word comes after the current token.

Moving on to the decoder side, the encoder-decoder attention layer stands out. The encoder-decoder layer focuses on relevant parts of the input sentence, taking into account the layer below it and the output of the encoder stack.

Putting it all together:

We will take an input, tokenize the input, and obtain token IDs of the tokens before converting them into embeddings for each token. From there, we will pass the embeddings into a transformer block. At the end of the process, the transformer will predict a series of output tokens. The following image provides a detailed look at the transformer model.

The decoder stack outputs a vector of floats. The linear layer projects the vector of floats produced by the stack of decoders into a larger vector called a logits vector. If the model has a 10,000-word vocabulary, then the linear layer maps the decoder output onto a 10,000 cell vector. The softmax layer turns those scores from the logit vector into probabilities — all positive — and adding up to 100%. The cell with the highest probability is chosen, and the word associated with it is produced as the output for this step.

The challenges with ChatGPT and LLMs

• They are trained on data that has no domain knowledge and could be out of date. For example, hallucinations are incorrect answers given as if they are correct and are common with LLMs.

• The models on their own do not have a natural ability to apply or extract filters from the input. Examples include time, date, and geographical filters.

• There is no access control on what document users can see.

• There are serious privacy and sensitive data control concerns.

• It is slow and very expensive to train on your own data and keep it up to date.

• Response from ChatGPT or other LLMs can be slow. Usually, Elasticsearch would have millisecond query responses. With LLMs, it can take up to seconds to get a response. But this is expected as LLMs are performing complex tasks. Also, ChatGPT charges by the number of token processed. If you have a high velocity workload like Black Friday merchandise search for an ecommerce site, it can get very expensive very quickly. Not to mention, it probably won’t meet <10ms query SLA.

• It is difficult, if not impossible, to interpret how ChatGPT or other LLMs arrived at query results. Besides hallucinations, ChatGPT and otherLLMs may produce irrelevant responses that are difficult to determine how the model produced the erroneous answer.

Integration of Elasticsearch with LLMs

Elasticsearch supports a bag of words and BM25 information retrieval approach, in addition to vector search through kNN and aNN natively (kNN is the exact nearest neighbor distance of all documents and aNN is the approximation). For aNN, Elasticsearch uses the HNSW (hierarchical navigable small world) algorithm for calculating approximate nearest neighbor distance. Elastic can mitigate many of the problems with LLMs while letting our users take advantage of all the good things ChatGPT and other LLMs can provide.

Elasticsearch can be used as a vector database, and to perform hybrid retrieval across text and vector data. There are three patterns where Elasticsearch can provide clear benefits when used with LLMs:

1. Provide context to your data and integrate with ChatGPT or other LLMs

2. Enable you to bring your own model (any 3rd party model)

3. Use the built-in Elastic Learned Sparse Encoder model

Method 1: Provide context to your data and integrate with ChatGPT or other LLMs

The following depicts how to separate LLMs from your data while integrating with generative AI.

A customer can bring their own embedding generated by a LLM and ingest their data along with the embedding into Elasticsearch. Then, the customer can take the similarity search results from their own data stored in Elasticsearch (context to the user’s question) to ChatGPT or another LLM to construct natural-language based answers to their users.

Also, the newly released reciprocal rank fusion (RRF) allows users to perform hybrid search, which can combine and rank search results. For instance, the BM25 method can filter for the relevant documents along with vector search to provide the best documents. With RRF, customers can achieve best search results natively through Elasticsearch instead of through their own applications, which greatly reduces complexity and maintenance of their applications.

Method 2: Bring your own model

The recently announced Elasticsearch Relevance Engine

TM (ESRETM) provides the capability to bring your own LLMs. This capability has been available for a while through machine learning. The Elasticsearch machine learning team has been scaffolding infrastructure for integrating transformer-based models. Starting with the 8.8 release, you can ingest and query just like you would normally do in Elasticsearch through the search APIs. On top of that, you can use the hybrid search method with RRF, which provides even better relevance. As the models are managed and integrated into Elasticsearch, it reduces operation complexity while achieving the most relevant search results.

This approach would require the users to know what model would work well for their use case and a commercial relationship with Elastic.

Method 3: Use the built-in sparse encoder model

Elastic Learned Sparse Encoder is the Elastic out-of-the-box language model that outperforms SPLADE (SParse Lexical AnD Expansion Model), which itself is a state-of-the-art model. Elastic Learned Sparse Encoder solves the vocabulary mismatch problem where a document may be relevant to a query but does not contain any terms that appear in the query. An example of the mismatch may be if we ask “how have American corporations have assisted with Covid-19 efforts”, then manufacturers of ventilators may not appear in the query results.

Elastic Learned Sparse Encoder is accessible just like other search endpoints via the text_expansion query. Elastic Learned Sparse Encoder enables our user to begin the state-of-the-art generative AI search with a click and yield immediate results. Elastic Learned Sparse Encoder is also an Elastic commercial feature.

Here are some benchmark results using the BEIR benchmark. We used several standardized data sets (horizontal axis) and applied different retrieval metaphors (vertical axis). As you can see, a combination of BM25 and our Learned Sparse Encoder using RRF, returns the best relevance scores. These scores with RRF beat the SPLADE model and our Learned Sparse Encoder model when considered by itself. We published more details on our blog here.

Terms and definitions

Neural Network (NN)

Each node is a neuron. Think of each individual node as its own linear regression model, composed of input data, weights, a bias (or threshold), and an output. The math representation may look like:

∑wixi + bias = w1x1 + w2x2 + w3x3 + bias

output = f(x) = 1 if ∑w1x1 + b>= 0; 0 if ∑w1x1 + b < 0

Once an input layer is determined, weights (w) are assigned. These weights help determine the importance of any given variable with larger ones contributing more significantly to the output compared to other inputs. All inputs are then multiplied by their respective weights and then summed. Afterward, the output is passed through an activation function, which determines the output. If that output exceeds a given threshold, it activates the node and passes data to the next layer in the network. This results in the output of one node becoming the input of the next node. This process of passing data from one layer to the next layer is a feed-forward network. This is just one type of NNs.

LLM parameters

Weights are numerical values that define the strength of connections between neurons across different layers in the model. Biases are additional numerical values that are added to the weighted sum of inputs before being passed through an activation function.

SPLADE

SPLADE is a late interaction model. The idea behind SPLADE models is that using a pre-trained language model like BERT can identify connections between words and use the knowledge to enhance sparse vector embedding. You would use this when you have a document that covers a wide range of topics, such as a Wikipedia article about a WWII movie — it contains the plot, the actors, the history, and the studio that released the film.

With embedding retrieval techniques alone, the relevance of the document to queries becomes an issue because the document can be projected onto a large number of dimensions and render it close to none of the queries. SPLADE solves the problem by combining all token-level probability distributions into a single distribution that tells us the relevance of every token in the vocabulary to our input sentence, similar to the BM25 method. Elastic Learned Sparse Encoder is the Elastic version of the SPLADE model.

RRF

RRF is a hybrid search query that normalizes and combines multiple search result sets with different relevant indicators into a single result set. Based on our own testing, combining RRF (BM25 + Elastic Learned Sparse Encoder) produces the best search relevance.

Wrap up

By combining the creative capabilities of technologies, such as ChatGPT, and the business context of proprietary data, we can truly transform how customers, employees, and organizations search.

Retrieval augmented generation (RAG) bridges the gap between large language models that power generative AI and private data sources. Well-known limitations of large language models can be addressed with context-based retrieval, enabling you to build deeply engaging search.

• Use Elasticsearch as your vector database

• ChatGPT and Elasticsearch: OpenAI meets private data

• Generative AI search tools for developers

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

In this blog post, we may have used or referred to third party generative AI tools, which are owned and operated by their respective owners. Elastic does not have any control over the third party tools and we have no responsibility or liability for their content, operation or use, nor for any loss or damage that may arise from your use of such tools. Please exercise caution when using AI tools with personal, sensitive or confidential information. Any data you submit may be used for AI training or other purposes. There is no guarantee that information you provide will be kept secure or confidential. You should familiarize yourself with the privacy practices and terms of use of any generative AI tools prior to use.

Elastic, Elasticsearch, ESRE, Elasticsearch Relevance Engine and associated marks are trademarks, logos or registered trademarks of Elasticsearch N.V. in the United States and other countries. All other company and product names are trademarks, logos or registered trademarks of their respective owners.

Elastic and Amazon Web Services (AWS) understand this pressing need and have taken the lead in offering cutting-edge solutions to meet these demands. Using Amazon SageMaker JumpStart combined with Elasticsearch’s capabilities, businesses can now confidently explore and adopt the most suitable AI models for their specific use cases while maintaining cost-effectiveness, security, and privacy.

Elasticsearch’s integration with advanced AI models further enhances its capabilities. By leveraging Elasticsearch’s retrieval prowess, LLMs can access the most relevant documents to provide accurate and up-to-date responses. This synergy between Elasticsearch and LLMs ensures that users receive contextually relevant and factual answers to their queries, setting a new standard for information retrieval and AI-powered assistance.

Elasticsearch is a scalable data store and vector database that offers a range of features to ensure exceptional search performance. It supports traditional keyword and text-based search using the BM25 algorithm, as well as AI-ready vector search with exact match and approximate kNN (k-Nearest Neighbor) search capabilities. These advanced features allow Elasticsearch to retrieve highly relevant results for queries expressed in natural language. By combining traditional, vector, or hybrid search approaches, Elasticsearch delivers precise results, making it effortless for users to find the information customers need.

The Elasticsearch platform seamlessly incorporates robust machine learning and artificial intelligence capabilities directly into its solutions, empowering you to create highly sought-after applications and accomplish tasks with remarkable efficiency. By leveraging these advanced technologies, you can harness the full potential of Elasticsearch to deliver exceptional user experiences and expedite your workflow.

Implementing RAG using Elasticsearch and open source LLM available in Amazon SageMaker JumpStart

The solution below explains how to use Retrieval Augmented Generation (RAG) to enable GAI capabilities on domain-specific business data using Elasticsearch, Amazon SageMaker JumpStart, and your choice of open source LLMs.

Solution overview

We will start by reviewing the architecture diagram below. It explains how to get domain-specific responses from an LLM hosted in Amazon SageMaker JumpStart using enterprise data hosted in Elasticsearch using RAG.

We used the following AWS and third-party services:

1. Amazon SageMaker and Amazon SageMaker JumpStart for hosting the open source LLMs from Hugging Face

2. Falcon 40B Instruct and Flan-T5 XL LLM from Hugging Face

3. Elasticsearch for storing embeddings of the enterprise knowledge corpus and doing similarity search with user questions

4. Python, LangChain, and Streamlit for building the RAG application

5. Amazon EC2 for hosting the Streamlit application

6. AWS Identity and Access Management roles and policies for access management

Step-by-step explanation

Offline data ingestion:

We ingest data from an enterprise knowledge corpus – for example this could be internal web pages, documents describing a company’s process, or corporate financial data.

1. The documents are ingested using a web crawler or any other ingestion mechanism.

2. The textual content is converted into vectors and stored in a dense_vector field by a sentence transformer type ML model.

Real-time flow on user query:

1. The user provides a question via the Retrieval Augmented Generation (RAG) web application.

2. The RAG application generates a hybrid search request for Elasticsearch based on the user's question and sends it to Elasticsearch. The hybrid search request does a BM25 match on the text field and kNN search on the dense_vector field.

3. Elasticsearch returns the document body and source URL (if applicable) to the RAG application. The RAG application accepts only the top scored document.

4. The RAG application passes the top scored document body (context) along with user question (prompt) to the LLM hosted as Amazon SageMaker endpoint.

5. The Amazon SageMaker endpoint generates a textual response and sends it back to the RAG application.

6. The RAG application performs any required post processing tasks. For example, it adds a source url to the response generated from the LLM. The user views the response in the web application.

Let’s now look at a few setup steps and a few implementation steps to create a working search solution:

Setup steps:

1. Sign up for a free trial of an Elasticsearch cluster with Elastic Cloud.

2. Create a new deployment on AWS following these steps.

3. Add a new machine learning node following the steps below. This will enable you to run machine learning models in your deployment.

4. Click on Edit under Deployment Name in the left navigation bar.

5. Scroll down to the Machine Learning instances box.

6. Click +Add Capacity.

7. Under Size per zone, click and select 2GB RAM.

8. Click on Save and then Confirm.

9. Reset and download the elastic user password following these steps.

10. Copy the deployment ID from the Overview page under Deployment name.

11. Load an embedding model into Elasticsearch. Here, we have usedall-distilroberta-v1 model hosted in the Hugging Face model hub. You can choose other sentence transformer types based on your case. Import this Python notebook here in Amazon SageMaker and run it. Provide the Cloud Id , Elasticsearch username , and Elasticsearch password when prompted. This will download the model from Hugging Face, chunk it up, load it into Elasticsearch, and deploy the model onto the machine learning node of the Elasticsearch cluster.

12. Create an Elasticsearch index by opening Kibana from the Elastic Cloud console and navigating to Enterprise Search - Overview. Click on Create an Elasticsearch Index. Choose Web Crawler as the Ingestion method. Enter a suitable Index name and click Create Index.

13. Add an Inference Pipeline by clicking on Pipelines tab > Copy and customizing it in the Ingest Pipeline Box. Click Add Inference Pipeline in the Machine Learning Inference Pipelines box. Enter the Name for the new pipeline. Select the trained Model loaded in step 6 and Select title as source field. Click Continue in two subsequent screens and click Create Pipeline at the Review stage.

1. Update the mapping for dense vector by clicking on Dev Tools and running the following code. This will enable you to run kNN search on the title field vectors. From Elasticsearch version 8.8+, this step will be handled automatically.

1. Configure web crawler to crawl Elastic Docs (you can replace this with your Enterprise Domain corpus). Click on the relevant index under Available indices. Click on the Manage Domains tab. Click Add domain. Enter https://www.elastic.co/guide/en and click Validate Domain. Click Add domain and then Add Crawl rules. Add the following rules. Click Crawl and then Crawl all domains on this index. This will start Elasticsearch’s web crawler and it will crawl the targeted documents, generate vectors for the title field, and index the document and vector.

The implementation steps for instantiating the solution presented in this post are as follows:

1. Choose your LLM. Amazon SageMaker JumpStart offers a wide selection of proprietary and publicly available foundation models from various model providers. Log in to Amazon SageMaker Studio, open Amazon SageMaker JumpStart, and search for your preferred Foundation model. Please find the list of models available for each task here.

1. Deploy your LLM. Amazon SageMaker JumpStart studio also provides a no-code interface to deploy the model. You can easily deploy a model with few clicks. After the deployment is successful, copy the Endpoint Name.

1. Download and set up the RAG Application. Launch an EC2 instance and clone the code from this GitHub link. Set up a virtual environment following these steps. Install the required Python libraries by running the command pip install -r requirements.txt. Update the config.sh file with the following:

2. ES_CLOUD_ID: Elastic Cloud Deployment ID

3. ES_USERNAME: Elasticsearch Cluster User

4. ES_PASSWORD: Elasticsearch User password

5. FLAN_T5_ENDPOINT: Amazon SageMaker Endpoint Name pointing to Flan T5

6. FALCON_40B_ENDPOINT: Amazon SageMaker Endpoint Name pointing to Falcon 40B

7. AWS_REGION: AWS Region

8. Run the application using the command streamlit run rag_elastic_aws.py. This will start a web browser and the url will be printed to the command line.

9. Response of LLM without context.

1. Response of LLM with context derived from Elasticsearch.

Conclusion

In this post, we showed you how to create a Retrieval Augmented Generation-based search application using a combination of Elasticsearch, Amazon SageMaker JumpStart, open-source LLMs from Hugging Face, and open source Python packages like LangChain and Streamlit.

Learn more by exploring JumpStart, Amazon Titan models, Amazon Bedrock, and Elastic to build a solution using the sample implementation provided in this post and a data set relevant to your business.

Or, start your own 7-day free trial by signing up via AWS Marketplace and quickly spin up a deployment in minutes on any of the Elastic Cloud regions on AWS around the world. Your AWS Marketplace purchase of Elastic will be included in your monthly consolidated billing statement and will draw against your committed spend with AWS.

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

In this blog post, we may have used or referred to third party generative AI tools, which are owned and operated by their respective owners. Elastic does not have any control over the third party tools and we have no responsibility or liability for their content, operation or use, nor for any loss or damage that may arise from your use of such tools. Please exercise caution when using AI tools with personal, sensitive or confidential information. Any data you submit may be used for AI training or other purposes. There is no guarantee that information you provide will be kept secure or confidential. You should familiarize yourself with the privacy practices and terms of use of any generative AI tools prior to use.

Elastic, Elasticsearch, ESRE, Elasticsearch Relevance Engine and associated marks are trademarks, logos or registered trademarks of Elasticsearch N.V. in the United States and other countries. All other company and product names are trademarks, logos or registered trademarks of their respective owners.

OpenAI has discontinued the use of plugins in ChatGPT. You can read more about this here. We recommend reading this tutorial instead to learn how to build a large language model (LLM) chatbot that uses a pattern known as Retrieval-Augmented Generation. You can also read this blog to learn how to create custom GPTs with Elastic data.

You may have read this previous blog post about our journey to connect Elasticsearch’s relevance capabilities with OpenAI question-answering capabilities. The key idea in that post was to illustrate how to use Elastic with OpenAI’s GPT model to build a response and return context-relevant content to users.

The application that we built can expose a search endpoint and be called by any front-end service. The good news is that now OpenAI has released a private alpha of the future ChatGPT plugin framework.

In this blog, you will learn how to implement the plugin and extend the use of ChatGPT to any content indexed in Elasticsearch, using the Elastic documentation.

What is a ChatGPT plugin?

ChatGPT plugins are extensions that are developed to assist the model in completing its knowledge or executing actions.

For example, we know that the cutover of ChatGPT from a knowledge perspective is September 2021, so any question on recent data won’t be answered. In addition, any question that relates to something too specific beyond the boundaries of what the model has been trained on won’t be answered.

Plugins can broaden the scope of possible applications and enhance the capabilities of the models, but reciprocally, the plugin's output is augmented by the model itself.

The official list of plugins currently supported by ChatGPT are listed below. You can expect this list to expand rapidly as more organizations experiment with ChatGPT:

As you scan through the list, you’ll notice that the use cases are slowly revealing themselves here. In the case of Expedia, for example, its plugin is extending ChatGPT to assist in planning travel, making ChatGPT a trip-planning assistant.

This blog aims to achieve similar objectives for Elastic — to allow ChatGPT to access Elastic’s current knowledge base and assist you with your Elastic projects.

Architecture

We are going to bring a slight modification that has a positive cost impact in the sample code presented in part 1 by my colleague Jeff Vestal.

We will remove the call to OpenAI API, as now ChatGPT will fulfill the role of taking the content from Elasticsearch and digesting it back to the user:

1. ChatGPT makes a call to the /search endpoint of the plugin.

• This decision is based on the plugin “rules” description_for_human (see plugin-manifest below).

1. The plugin code creates a search request that is sent to Elasticsearch.

2. Documentation body and original url are returned to Python.

3. The plugin returns the document body and url, in text form to ChatGPT.

4. ChatGPT uses the information from the plugin to craft its response.

Again, this blog post assumes that you have set up your Elastic Cloud account, vectorized your content, and have an Elasticsearch cluster filled with data ready to be used. If you haven’t set all that up, see our previous post for detailed steps to follow.

Plugin code

OpenAI built a fairly simple-to-handle plugin framework for ChatGPT. It deploys a service that exposes:

• The plugin manifest, explaining what the plugin provides to the users and to ChatGPT

• The plugin openAPI definition, which is the functional description that enables ChatGPT to understand the available APIs The plugin code can be found here.

Plugin file structure

The screenshot below shows what the structure looks like:

• The plugin manifest is stored in the ai-plugin.json file under the .well-known directory as per OpenAI best practices.

• The main service code is in app.py.

• The Dockerfile will be later used to deploy the plugin to Google Cloud Compute.

• The plugin’s logo (logo.ong) as displayed in the ChatGPT plugin store, here the Elastic logo.

• The OpenAI description of the plugin.

Python code

For the full code, refer to the GitHub repository. We are going to look only at the main part of this code:

We took out all the details and kept the main parts here. There are two categories of APIs here:

1. The one required by OpenAI to build a plugin:

• /logo.png: retrieve the plugin logo

• /.well-known/ai-plugin.json: fetches the plugin manifest

• /openapi.yaml: fetches the plugin OpenAPI description

1. The plugin API:

• /search is the only one here exposed to ChatGPT that runs the search in Elasticsearch

Plugin manifest

The plugin manifest is what ChatGPT will use to validate the existence (reachable) of the plugin. The definition is the below:

There are a couple of things to point out here:

1. There are two descriptions:

• description_for_human - This is what the human sees when installing the plugin in the ChatGPT web UI.

• description_for_model - Instructions for the model to understand when to use the plugin.

1. There are some placeholders such as PLUGIN_HOSTNAME that are replaced in the Python code.

OpenAPI definition

Our code will only expose a single API endpoint to ChatGPT allowing it to search for Elastic documentation. Here is the description:

For the definition file, the key points are:

• We take the ChatGPT prompt content and pass it as a query to our Elasticsearch cluster.

• Some placeholders such as PLUGIN_HOSTNAME are replaced in the Python code.

Deploying the Elastic plugin in Google Cloud Platform (GCP)

You have a choice in picking a deployment method to expose your plugin, as well as using a different cloud provider. We use GCP in this blog post — more specifically Google Cloud Run and Google Cloud Build. The first is to expose and run the service, and the second is for continuous integration.

Setup

This setup assumes your GCP user has the right permissions to:

• Build a container image with Google Cloud Build in the Google Container Registry

• Deploy a container in Google Cloud Run

If not, you will need to update permissions on the GCP IAM page.

We are going to use the gcloud CLI to set up our environment. You can find the installation instructions here.

Once installed, run the following command to authenticate:

Then set the project identifier to your GCP project:

You are now ready to build and deploy.

Build and deploy

The first step is to build the container image using Cloud Build and push it to the Google Container Registry:

Replace PROJECT_ID with your GCP project ID and my-python-app with the name you want to give to your container image.

Export the environment required by the Python code to create the Elasticsearch client:

Finally, deploy the container image to Cloud Run:

You should see your service running in Cloud Run:

Note that you can also activate the continuous integration so that any commit in your GitHub repository will trigger a redeploy. On the service details page, click on Set up continuous deployment.

Installing the plugin in ChatGPT

Once the plugin is deployed and has a publicly accessible endpoint, it can be installed in ChatGPT. In our case, since this is deployed in Google Cloud Run, you can get the URL here:

Then in ChatGPT, go in the plugin store:

Choose to do “Develop your own plugin”:

Paste the URL you copied from the Google Cloud Run page:

Ensure the plugin is found and valid:

Follow the installation instructions until you see your plugin available in the list:

Let’s test our plugin!

OK, now for the best part! Do remember that ChatGPT decides to delegate when your prompt goes beyond its knowledge. To ensure that happens, just ask a question similar to this example:

With the steps provided in this blog, you can create your own plugin and deploy it on a cloud provider or your own hosts. This allows you to start exploring enhancing ChatGPT's knowledge and functionality, enhancing an already amazing tool with specialized and proprietary knowledge.

You can try all of the capabilities discussed in this blog today! Get started by signing up for a free Elastic Cloud trial.

Here are some other blogs you may find interesting:

• ChatGPT and Elasticsearch: OpenAI meets private data

• Monitor OpenAI API and GPT models with OpenTelemetry and Elastic

• Exploring the Future of Security with ChatGPT

In this blog post, we may have used third party generative AI tools, which are owned and operated by their respective owners. Elastic does not have any control over the third party tools and we have no responsibility or liability for their content, operation or use, nor for any loss or damage that may arise from your use of such tools. Please exercise caution when using AI tools with personal, sensitive or confidential information. Any data you submit may be used for AI training or other purposes. There is no guarantee that information you provide will be kept secure or confidential. You should familiarize yourself with the privacy practices and terms of use of any generative AI tools prior to use.

Elastic, Elasticsearch and associated marks are trademarks, logos or registered trademarks of Elasticsearch N.V. in the United States and other countries. All other company and product names are trademarks, logos or registered trademarks of their respective owners.

Combine Elasticsearch's search relevance with OpenAI's ChatGPT's question-answering capabilities to query your data. In this blog, you'll learn how to connect ChatGPT to proprietary data stores using Elasticsearch and build question/answer capabilities for your data.

What is ChatGPT?

In recent months, there has been a surge of excitement around ChatGPT, a groundbreaking AI model created by OpenAI. But what exactly is ChatGPT?

Based on the powerful GPT architecture, ChatGPT is designed to understand and generate human-like responses to text inputs. GPT stands for "Generative Pre-trained Transformer.” The Transformer is a cutting-edge model architecture that has revolutionized the field of natural language processing (NLP). These models are pre-trained on vast amounts of data and are capable of understanding context, generating relevant responses, and even carrying on a conversation. To learn more about the history of transformer models and some NLP basics in the Elastic Stack, be sure to check out the great talk by Elastic ML Engineer Josh Devins.

The primary goal of ChatGPT is to facilitate meaningful and engaging interactions between humans and machines. By leveraging the recent advancements in NLP, ChatGPT models can provide a wide range of applications, from chatbots and virtual assistants to content generation, code completion, and much more. These AI-powered tools have rapidly become an invaluable resource in countless industries, helping businesses streamline their processes and enhance their services.

Limitations of ChatGPT & how to minimize them

Despite the incredible potential of ChatGPT, there are certain limitations that users should be aware of. One notable constraint is the knowledge cutoff date. Currently, ChatGPT is trained on data up to September 2021, meaning it is unaware of events, developments, or changes that have occurred since then. Consequently, users should keep this limitation in mind while relying on ChatGPT for up-to-date information. This can lead to outdated or incorrect responses when discussing rapidly changing areas of knowledge such as software enhancements and capabilities or even world events.

ChatGPT, while an impressive AI language model, can occasionally hallucinate in its responses, often exacerbated when it lacks access to relevant information. This overconfidence can result in incorrect answers or misleading information being provided to users. It is important to be aware of this limitation and approach the responses generated by ChatGPT with a degree of skepticism, cross-checking and verifying the information when necessary to ensure accuracy and reliability.

Another limitation of ChatGPT is its lack of knowledge about domain-specific content. While it can generate coherent and contextually relevant responses based on the information it has been trained on, it is unable to access domain-specific data or provide personalized answers that depend on a user's unique knowledge base. For instance, it may not be able to provide insights into an organization’s proprietary software or internal documentation. Users should, therefore, exercise caution when seeking advice or answers on such topics from ChatGPT directly.

One way to minimize these limitations is by providing ChatGPT access to specific documents relevant to your domain and questions, and enabling ChatGPT’s language understanding capabilities to generate tailored responses.

This can be accomplished by connecting ChatGPT to a search engine like Elasticsearch.

Elasticsearch — you know, for search!

Elasticsearch is a scalable data store and vector database designed to deliver relevant document retrieval, ensuring that users can access the information they need quickly and accurately. Elasticsearch’s primary focus is on delivering the most relevant results to users, streamlining the search process, and enhancing user experience.

Elasticsearch boasts a myriad of features to ensure top-notch search performance, including support for traditional keyword and text-based search (BM25) and an AI-ready vector search with exact match and approximate kNN (k-Nearest Neighbor) search capabilities. These advanced features allow Elasticsearch to retrieve results that are not only relevant but also for queries that have been expressed using natural language. By leveraging traditional, vector, or hybrid search (BM25 + kNN), Elasticsearch can deliver results with unparalleled precision, helping users find the information they need with ease.

One of the key strengths of Elasticsearch is its robust API, which enables seamless integration with other services to extend and enhance its capabilities. By integrating Elasticsearch with various third-party tools and platforms, users can create powerful and customized search solutions tailored to their specific requirements. This flexibility and extensibility makes Elasticsearch an ideal choice for businesses looking to improve their search capabilities and stay ahead in the competitive digital landscape.

By working in tandem with advanced AI models like ChatGPT, Elasticsearch can provide the most relevant documents for ChatGPT to use in its response. This synergy between Elasticsearch and ChatGPT ensures that users receive factual, contextually relevant, and up-to-date answers to their queries. In essence, the combination of Elasticsearch's retrieval prowess and ChatGPT's natural language understanding capabilities offers an unparalleled user experience, setting a new standard for information retrieval and AI-powered assistance.

How to use ChatGPT with Elasticsearch

1. Python interface accepts user questions.

Generate a hybrid search request for Elasticsearch

• BM25 match on the title field

• kNN search on the title-vector field

• Boost kNN search results to align scores

• Set size=1 to return only the top scored document

1. Search request is sent to Elasticsearch.

2. Documentation body and original url are returned to python.

3. API call is made to OpenAI ChatCompletion.

• Prompt: "answer this question <question> using only this document <body_content from top search result>"

1. Generated response is returned to python.

2. Python adds on original documentation source url to generated response and prints it to the screen for the user.

The ElasticDoc ChatGPT process utilizes a Python interface to accept user questions and generate a hybrid search request for Elasticsearch, combining BM25 and kNN search approaches to find the most relevant document from the Elasticsearch Docs site, now indexed in Elasticsearch. However, you do not have to use hybrid search or even vector search. Elasticsearch provides the flexibility to use whichever search pattern best fits your needs and provides the most relevant results for your specific data sets.

After retrieving the top result, the program crafts a prompt for OpenAI's ChatCompletion API, instructing it to answer the user's question using only the information from the selected document. This prompt is key to ensuring the ChatGPT model only uses information from the official documentation, lessening the chance of hallucinations.

Finally, the program presents the API-generated response and a link to the source documentation to the user, offering a seamless and user-friendly experience that integrates front-end interaction, Elasticsearch querying, and OpenAI API usage for efficient question-answering.

Note that while we are only returning the top-scored document for simplicity, the best practice would be to return multiple documents to provide more context to ChatGPT. The correct answer could be found in more than one documentation page, or if we were generating vectors for the full body text, those larger bodies of text may need to be chunked up and stored across multiple Elasticsearch documents. By leveraging Elasticsearch's ability to search across numerous vector fields in tandem with traditional search methods, you can significantly enhance your top document recall.

Technical setup

The technical requirements are fairly minimal, but it takes some steps to put all the pieces together. For this example, we will configure the Elasticsearch web crawler to ingest the Elastic documentation and generate vectors for the title on ingest. You can follow along to replicate this setup or use your own data. To follow along we will need:

• Elasticsearch cluster

• Eland Python library

• OpenAI API account

• Somewhere to run our python frontend and api backend

Elastic Cloud setup

The steps in this section assume you don’t currently have an Elasticsearch cluster running in Elastic Cloud. If you do you, can skip to the next section.

Sign up If you don’t already have an Elasticsearch cluster, you can sign up for a free trial with Elastic Cloud.

Create deployment After you sign up, you will be prompted to create your first deployment.

• Create a name for your deployment.

• You can accept the default cloud provider and region or click Edit Settings and choose another location.

• Click Create deployment. Shortly a new deployment will be provisioned for you and you will be logged in to Kibana. Back to the Cloud We need to do a couple of things back in the Cloud Console before we move on: Click on the Navigation Icon in the upper left and select Manage this deployment.

Add a machine learning node.

• Back in the Cloud Console, click on Edit under your Deployment’s name in the left navigation bar.

• Scroll down to the Machine Learning instances box and click +Add Capacity.

• Under Size per zone, click and select 2 GB RAM.

• Scroll down and click on Save.

• In the pop-up summarizing the architecture changes, click Confirm.

• In a few moments, your deployment will now have the ability to run machine learning models!

Reset Elasticsearch Deployment User and password:

• Click on Security on the left navigation under your deployment’s name.

• Click on Reset Password and confirm with Reset. (Note: as this is a new cluster nothing should be using this Elastic password.)

• Download the newly created password for the “elastic” user. (We will use this to load our model from Hugging Face and in our python program.)

Copy the Elasticsearch Deployment Cloud ID.

• Click on your Deployment name to go to the overview page.

• On the right-hand side click the copy icon to copy your Cloud ID. (Save this for use later to connect to the Deployment.)

Eland

We next need to load an embedding model into Elasticsearch to generate vectors for our blog titles and later for our user’s search questions. We will be using the all-distilroberta-v1 model trained by SentenceTransformers and hosted on the Hugging Face model hub. This particular model isn’t required for this setup to work. It is good for general use as it was trained on very large data sets covering a wide range of topics. However, with vector search use cases, using a model fine-tuned to your particular data set will usually provide the best relevancy.

To do this, we will use the Eland python library created by Elastic. The library provides a wide range of data science functions, but we will be using it as a bridge to load the model into Elasticsearch from the Hugging Face model hub so it can be deployed on machine learning nodes for inference use.

Eland can either be run as part of a python script or on the command line. The repo also provides a Docker container for users looking to go that route. Today we will run Eland in a small python notebook, which can run in Google’s Colab in the web browser for free.

Open the program link and click the “Open in Colab” button at the top to launch the notebook in colab.

Set the variable hf_model_id to the model name. This model is set already in the example code but if you want to use a different model or just for future information:

• hf_model_id='sentence-transformers/all-distilroberta-v1'

• Copy model name from Hugging Face. The easiest way to do this is to click the copy icon to the right of the model name.

Run the cloud auth section, and you will be prompted to enter:

• Cloud ID (you can find this in the Elastic Cloud Console)

• Elasticsearch Username (easiest will be to use the “Elastic” user created when the deployment was created)

• Elasticsearch User Password

Run the remaining steps.

• This will download the model from Hugging face, chunk it up, and load it into Elasticsearch.

• Deploy (start) the model onto the machine learning node.

Elasticsearch index and web crawler

Next up we will create a new Elasticsearch index to store our Elastic Documentation, configure the web crawler to automatically crawl and index those docs, as well as use an ingest pipeline to generate vectors for the doc titles.

• Open Kibana from the Cloud Console if you don’t already have it open.

• In Kibana, Navigate to Enterprise Search -> Overview. Click Create an Elasticsearch Index.

• Using the Web Crawler as the ingestion method, enter elastic-docs as the index name. Then, click Create Index.

• Click on the “Pipelines” tab.

• Click Copy and customize in the Ingest Pipeline Box.

• Click Add Inference Pipeline in the Machine Learning Inference Pipelines box.

• Enter the name elastic-docs_title-vector for the New pipeline.

• Select the trained ML model you loaded in the Eland step above.

• Select title as the Source field.

• Click Continue, then click Continue again at the Test stage.

• Click Create Pipeline at the Review stage.

Update mapping for dense_vector field. (Note: with Elasticsearch version 8.8+, this step should be automatic.)

• In the navigation menu, click on Dev Tools. You may have to click Dismiss on the flyout with documentation if this is your first time opening Dev Tools.

• In Dev Tools in the Console tab, update the mapping for our dense vector target field with the following code. You simply paste it in the code box and click the little arrow to the right of line 1.

• You should see the following response on the right half of the screen:

• This will allow us to run kNN search on the title field vectors later on.

Configure web crawler to crawl Elastic Docs site:

• Click on the navigation menu one more time and click on Enterprise Search -> Overview.

• Under Content, click on Indices.

• Click on search-elastic-docs under Available indices.

• Click on the Manage Domains tab.

• Click “Add domain.”

• Enter https://www.elastic.co/guide/en, then click Validate Domain.

• After the checks run, click Add domain. Then click Crawl rules.

• Add the following crawl rules one at a time. Start with the bottom and work up. Rules are evaluated according to first match.

Disallow

Contains

release-notes

Allow

Regex

/guide/en/.*/current/.*

Disallow

Regex

.*

• With all the rules in place, click Crawl at the top of the page. Then, click Crawl all domains on this index.

Elasticsearch’s web crawler will now start crawling the documentation site, generating vectors for the title field, and indexing the documents and vectors.

The first crawl will take some time to complete. In the meantime, we can set up the OpenAI API credentials and the Python backend.

Connecting with OpenAI API

To send documents and questions to ChatGPT, we need an OpenAI API account and key. If you don’t already have an account, you can create a free account and you will be given an initial amount of free credits.

• Go to https://platform.openai.com and click on Signup. You can go through the process to use an email address and password or login with Google or Microsoft.

Once your account is created, you will need to create an API key:

• Click on API Keys.

• Click Create new secret key.

• Copy the new key and save it someplace safe as you won’t be able to view the key again.

Python backend setup

Clone or download the python program

Github Link to code

1. Install required python libraries. We are running the example program in Replit, which has isolated environments. If you are running this on a laptop or VM, best practice is to set up a virtual ENV for python.

• Run pip install -r requirements.txt

1. Set authentication and connection environment variables (e.g., if running on the command line: export openai_api=”123456abcdefg789”)

• openai_api - OpenAI API Key

• cloud_id - Elastic Cloud Deployment ID

• cloud_user - Elasticsearch Cluster User

• cloud_pass - Elasticsearch User Password

1. Run the streamlit program. More info about streamlit can be found in its docs.

• Streamlit has its own command to start: streamlit run elasticdocs_gpt.py

1. This will start a web browser and the url will be printed to the command line.

Sample chat responses

With everything ingested and the front end up and running, you can start asking questions about the Elastic Documentations.

Asking “Show me the API call for an inference processor” now returns an example API call and some information about the configuration settings.

Asking for steps to add a new integration to Elastic Agent will return:

As mentioned earlier, one of the risks of allowing ChatGPT to answer questions based purely on data it has been trained on is its tendency to hallucinate incorrect answers. One of the goals of this project is to provide ChatGPT with the data containing the correct information and let it craft an answer.

So what happens when we give ChatGPT a document that does not contain the correct information? Say, asking it to tell you how to build a boat (which isn’t currently covered by Elastic’s documentation):

When ChatGPT is unable to find an answer to the question in the document we provided, it falls back on our prompt instruction simply telling the user it is unable to answer the question.

Elasticsearch’s robust retrieval + the power of ChatGPT

In this example, we've demonstrated how integrating Elasticsearch's robust search retrieval capabilities with cutting-edge advancements in AI-generated responses from GPT models can elevate the user experience to a whole new level.

The individual components can be tailored to suit your specific requirements and adjusted to provide the best results. While we used the Elastic web crawler to ingest public data, you're not limited to this approach. Feel free to experiment with alternative embedding models, especially those fine-tuned for your domain-specific data.

You can try all of the capabilities discussed in this blog today! To build your own ElasticDocs GPT experience, sign up for an Elastic trial account, and then look at this sample code repo to get started.

If you would like ideas to experiment with search relevance, here are two to try out:

• [BLOG] Deploy NLP text embeddings and vector search using Elasticsearch

• [BLOG] Implement image similarity search with Elastic

In this blog post, we may have used third party generative AI tools, which are owned and operated by their respective owners. Elastic does not have any control over the third party tools and we have no responsibility or liability for their content, operation or use, nor for any loss or damage that may arise from your use of such tools. Please exercise caution when using AI tools with personal, sensitive or confidential information. Any data you submit may be used for AI training or other purposes. There is no guarantee that information you provide will be kept secure or confidential. You should familiarize yourself with the privacy practices and terms of use of any generative AI tools prior to use.

Elastic, Elasticsearch and associated marks are trademarks, logos or registered trademarks of Elasticsearch N.V. in the United States and other countries. All other company and product names are trademarks, logos or registered trademarks of their respective owners.

In this post, we discuss how users’ experience can be further enhanced with the addition of facets, filtering, and additional context. By providing tools like ChatGPT additional context, you can increase the likelihood of obtaining more accurate results. See how Elasticsearch's faceting and filtering framework can allow users to refine their search and reduce the cost of engaging with ChatGPT.

Comparing ChatGPT and Elasticsearch results

To improve the user experience of our sample application, we've added a feature that displays the raw results alongside the ChatGPT-created response. This will help users better understand how ChatGPT works.

Since our source data set is only crawled, the structure in the documents makes it difficult to read for a human. To show this difference and therefore the value that ChatGPT can bring, we added the raw result next to the GPT created response.

Currently, this example application only returns a single result. And even though we have hybrid scoring with vector search and BM25, this result may not be perfect. If we take this not perfect result and pass it over to ChatGPT, there’s a good chance that the response we get won’t be great either, as the context was missing important information.

Ideally, we’d just pass more context into ChatGPT, but the current 3.5-turbo models are limited to 4,096 tokens (that’s including the response you expect to get, so the actual limit is much lower). Future models will likely have a much larger limit, but this also comes with a cost.

As of today, GPT-3.5-turbo costs $0.002 per 1K tokens, while the up-and-coming GPT-4 with 32K context costs $0.06 per 1K tokens — that’s a factor of 30 more. Even with more powerful models coming in the next few years, there’s a chance that it’s not economically viable to do so for all user cases.

We will therefore not use GPT-4 right now and instead work around the max token limitation of GPT-3.5 by sending multiple concurrent requests and giving the user more flexibility in filtering the results.

Leveraging aggregations, facets, and filtering in Elasticsearch to enhance ChatGPT

To address this limitation, one of the biggest advantages of Elasticsearch is its robust faceting and filtering framework. When a user is searching for something, they may have additional preferences or context they can provide to dramatically increase the likelihood of obtaining the correct result. By leveraging Elasticsearch's faceting and filtering framework, we can allow users to refine their search based on various parameters such as date, location, or other relevant criteria.

It’s also important to note that many users have gotten used to having facet filtering options available when searching for something. Let us look at an example.

Searching for “How can I parse a message with Grok?” results in a document for ingest pipelines to be returned as the top result. This is not wrong, as ingest pipelines also support Grok expressions, but what if the user was interested in parsing his data using Logstash?

Using a simple terms aggregation as part of the request to fetch the hits, we can get a list of the top 10 product categories and offer these as a filtering option for a user.

If the user now selects “Logstash” on the left side, all results will be filtered for Logstash. It’s important to note that this all works while still using the same hybrid query model that we’ve talked about in the previous blog. We’re still using a combination of BM25 and kNN search to match our documents.

Loading multiple results in parallel

We briefly mentioned the max token limit earlier. In short, the prompt that you send to the API and its response can’t be longer than 4,096 tokens. When searching your proprietary data, you would like to provide as much specific context as possible so the model can give you the best answer. However, the 4,096 tokens aren’t that much, especially when you include things like code snippets.

A very simple first step toward mitigating the limit is to just ask multiple times in parallel, giving a different context each time. Using our approach with Elasticsearch, instead of only fetching the top 1 result and sending that to OpenAI, we can change the application to load the top 10 hits instead and then ask the question with the respective context.

This gives us 10 unique answers to our question and greatly increases our chances of presenting a relevant answer to the end user. While we are increasing the burden of the user to look at the results, it still gives them more flexibility.

Think of it like this: if you try to debug a problem and search for an exception on Google, you quickly scan the list of the top four or five results that Google displays and click on the one that seems most fitting to your question. Showing the user multiple answers to their question is similar to this.

While having a single correct answer would be ideal, having more than one to choose from initially is a great starting point. And as mentioned before, it can be cheaper compared to using a more expensive model (such as GPT-4).

We can also get more creative with our prompt and ask ChatGPT to send us a specific response if it can’t answer the question using the provided context. This will allow us to remove the results from the UI later.

One prompt that worked well in our use case is:

Working around the max token limit of ChatGPT: Answering a question from a set of answers

Since we have more than a single answer to our question now, we can attempt to summarize them into a single response. For this, we will mostly follow the same approach as before, but instead of searching Elasticsearch for the context, we will just concatenate the individual answers we’ve received so far, excluding any where the model responded that it can’t answer it based on the provided context.

Note that the prompt for this run is a little different from the earlier prompt, so the model treats our context slightly differently. The provided prompt here is by no means perfect, and depending on the data, it should be adjusted and optimized further.

With this additional “reduce phase” in place, our app will now:

• Search Elasticsearch for the top 10 hits

• 10x in parallel ask OpenAI to answer the question, providing a different context each time

• Concatenate responses from OpenAI and ask OpenAI once again to answer the question

With this setup, we can use close to 40,000 tokens of context, while only paying for the considerably cheaper GPT-3.5 model. In another blog post, we will explore the cost in more detail and use Elastic APM for tracking our spend, alongside other metrics.

It should be noted that GPT-4 may still perform much better than the approach above, so use whatever works best for you and the amount of traffic you expect.

Citations for your ChatGPT results

One downside of large language models (LLMs) is their overconfidence and tendency to hallucinate. You ask a question, you get an answer. Whether the answer is actually correct is for you to decide. The model rarely admits that it does not know something. Providing the context and telling it to respond with a specific answer as we did above helps mitigates this to some extent.

But the provided context alongside getting the model to admit that it can’t answer a question also allows us to provide more accurate citations for the responses.

In the last section, we summarized our set of 10 answers into one global answer. In addition to just providing this global answer, we can also provide a list of all source documentation pages that we used to compile the result — basically any page where the model did not respond "The provided page does not answer the question."

In this screenshot, you can see the summary answer on a set of 10 results from Elasticsearch. Even though we inspected 10 results, we are only displaying the three links to the documentation that are actually relevant to answer the question. In this case, the other seven documents returned by Elasticsearch had something to do with documents or indices, but they didn’t specifically talk about how to index something.

Searching proprietary data

We’ve mentioned in an earlier blog post that it’s great to use Elasticsearch and OpenAI to search proprietary data. However, we did use a web-crawler to crawl public documentation. That may seem a bit counterintuitive, and you’re right to think about it! OpenAI trains GPT models on web data, so we will assume it knows our documentation already. So why do we need Elasticsearch in addition to that data? Does this setup actually work on data that’s not public? It does — let’s prove it.

Using the existing setup, we will push a single super secret document about an internal project into our index.

Next we’ll then head over to our app and search for “What are the steps for the internal project?”

In summary, we used faceting and filtering to, for certain use cases, reduce the number tokens of context required to engage with ChatGPT. By providing additional context at query time, we showed it is also possible to improve the accuracy of search results.

Learn more about the possibilities with Elasticsearch and AI .

In this blog post, we may have used third party generative AI tools, which are owned and operated by their respective owners. Elastic does not have any control over the third party tools and we have no responsibility or liability for their content, operation or use, nor for any loss or damage that may arise from your use of such tools. Please exercise caution when using AI tools with personal, sensitive or confidential information. Any data you submit may be used for AI training or other purposes. There is no guarantee that information you provide will be kept secure or confidential. You should familiarize yourself with the privacy practices and terms of use of any generative AI tools prior to use.

Costs referred to herein are based on the current OpenAI API pricing and how often we call it when loading our sample app.

Elastic, Elasticsearch and associated marks are trademarks, logos or registered trademarks of Elasticsearch N.V. in the United States and other countries. All other company and product names are trademarks, logos or registered trademarks of their respective owners.

In this blog, we will instrument a Python application that uses OpenAI and analyze its performance, as well as the cost to run the application. Using the data gathered from the application, we will also show how to integrate large language models (LLMs) into your application. As a bonus, we will try to answer the question: why does ChatGPT print its output word by word?

Instrumenting the application with Elastic APM

If you’ve had a chance to give our sample application a try, you might have noticed that the result does not load as quickly as you’d expect it to, from a search interface.

The now is if this is from our two-phased approach of running a query in Elasticsearch first or if the slow behavior is emerging from OpenAI, or if it’s a combination of the two.

Using Elastic APM, we can easily instrument this application to get a better look. All we need to do for the instrumentation is the following (we will show the full example at the end of the blog post and also in a GitHub repository):

Since our sample application is using Streamlit, we will also need to start at least one transaction and eventually end it again. In addition, we can also provide information about the outcome of the transaction to APM, so we can track failures properly.

And that’s it — this would be enough to have full APM instrumentation for our application. That being said, we will be doing a little extra work here in order to get some more interesting data.

As a first step, we will add the user’s query to the APM metadata. This way we can inspect what the user was trying to search and can analyze some popular queries or reproduce errors.

In our async method, which talks to OpenAI, we will also add some more instrumentation so we can better visualize the tokens we receive, as well as to collect additional statistics.

And finally, toward the very end of our application, we will also add the number of tokens and approximate cost to our APM transaction. This will enable us to visualize these metrics later and correlate them to the application performance.

If you do not use streaming, then the OpenAI response will contain a “total_tokens” field, which is the sum of the context you sent and the response returned. If you are using the “stream=True” option, then it’s your responsibility to calculate the number of tokens or approximate them. A common recommendation is to use “(len(prompt) + len(response)) / 4” for english text, but especially code snippets can throw off this approximation. If you need more exact numbers, you can use libraries like tiktoken to calculate the number of tokens.

Analyzing the APM data — Elasticsearch vs. OpenAI performance

After instrumenting the application, a quick look at the “Dependencies” gives us a better understanding of what’s going on. It looks like our requests to Elasticsearch return within 125ms on average, while OpenAI takes 8,500ms to complete a request. (This screenshot was taken on a version of the application that does not use streaming. If you use streaming, the default instrumentation only considers the initial POST request in the dependency response time and not the time it takes to stream the full response.)

If you’ve used ChatGPT yourself already, you might have been wondering why the UI is printing every word individually, instead of just returning the full response immediately.

As it turns out, this is not actually to entice you to pay money if you’re using the free version! It’s more of a limitation of the inference model. In simple terms, in order to compute the next token, the model will need to take into consideration the last token as well. So there’s not much room for parallelization. And since every token is processed individually, this token can also be sent to the client, while the computation for the next token is running.

In order to improve the UX, it can be helpful to therefore use a streaming approach when using the ChatCompletion functionality. This way a user can start to consume the first results while the full response is being generated. You can see this behavior in the GIF below. Even though all three responses are still loading, the user can scroll down and inspect what’s there already.

As mentioned previously, we added a bit more custom instrumentation than just the bare minimum. This allows us to get detailed information on where our time is spent. Let’s take a look at a full trace and see this streaming in action.

Our application is configured to fetch the top three hits from Elasticsearch, and then run one ChatCompletion request against OpenAI in parallel.

As we can see in the screenshot, loading the individual results takes about 15s. We can also see that requests to OpenAI that return a larger response take longer to return. But this is only a single request. Does this behavior happen for all requests? Is there a clear correlation between response time and number of tokens to back up our claims from earlier?

Analyzing cost and response time

Instead of visualizing the data using Elastic APM, we can also use custom dashboards and create visualizations from our APM data. Two interesting charts that we can build show the relationship between the number of tokens in a response and the duration of the request.

We can see that the more tokens get returned (x-axis in the first chart), the higher the duration (y-axis in the first chart). In the chart to the right, we can also see that the duration per 100 tokens returned stays almost flat at around 4s, no matter the number of tokens returned in total (x-axis).

If you want to improve the responsiveness of your application that uses OpenAI models, it might be a good idea to tell the model to keep the response short.

In addition to this, we can also track our total spend and the average cost per page load, as well as other statistics.

With our sample application, the cost for a single search is around 1.1¢. This number does not sound high, but it’s likely not something that you will have on your public website as a search alternative anytime soon. For company internal data and a search interface that’s only used occasionally, this cost is negligible.

In our testing, we’ve also hit frequent errors when using the OpenAI API in Azure, which eventually made us add a retry loop to the sample app with an exponential backoff. We can also capture these errors using Elastic APM.

Any captured errors are then visible in the waterfall charts as part of the span where the failure happened.

In addition, Elastic APM also provides an overview of all the errors. In the screenshot below, you can see the occasional RateLimitError and APIConnectionError that we’ve encountered. Using our crude exponential retry mechanism, we can mitigate most of these problems.

Latency and failed transaction correlation

With all the built-in metadata that the Elastic APM agent capture, as well as the custom labels we added, we can easily analyze if there’s any correlation between the performance and any of the metadata (like services version, user query, etc.)

As we can see below, there’s a small correlation between the query “How can I mount and index on a frozen node?” and a slower response time.

Similar analysis can be done on any transaction that resulted in an error. In this example, the two queries “How do I create an ingest pipeline” and “How can I create an ingest pipeline” fail more often than other queries, causing them to bubble up in this correlation analysis.

In this blog, we instrumented an app written in Python to use OpenAI and analyze its performance. We looked at response latency and failed transactions, and we assessed the costs of running the application. We hope this how-to was useful for you!

Learn more about the possibilities with Elasticsearch and AI .

In this blog post, we may have used third party generative AI tools, which are owned and operated by their respective owners. Elastic does not have any control over the third party tools and we have no responsibility or liability for their content, operation or use, nor for any loss or damage that may arise from your use of such tools. Please exercise caution when using AI tools with personal, sensitive or confidential information. Any data you submit may be used for AI training or other purposes. There is no guarantee that information you provide will be kept secure or confidential. You should familiarize yourself with the privacy practices and terms of use of any generative AI tools prior to use.

Costs referred to herein are based on the current OpenAI API pricing and how often we call it when loading our sample app.

Elastic, Elasticsearch and associated marks are trademarks, logos or registered trademarks of Elasticsearch N.V. in the United States and other countries. All other company and product names are trademarks, logos or registered trademarks of their respective owners.

Let's run through some basic details, best practices, and how it works, so you can try it out in the Elasticsearch Service today.

Delivering speed and accuracy

Random sampling in Elasticsearch has never been easier or faster. If your query has many aggregations, you can quickly obtain results by using the random_sampler aggregation.

All the above aggregations nested under random_sampler will return sampled results. Each agg is roughly seeing only 0.1% of the documents (or 1 in every 1000th document). Where computational cost correlates with the number of documents, the aggregation speed increases. You may have also noticed the “seed” parameter. You can provide a seedto get consistent results on the same shards. Without a seed, a new random subset of documents is considered and you may get slightly different aggregated results.

How much faster is the random_sampler? The speed improves according to the provided probability as fewer documents are aggregated. The improvements relative to probability will eventually flatten out. Each aggregation has its own computational overhead regardless of the number of documents. An example of this overhead cost is comparing multi-bucket to single metric aggregations. Multi-bucket aggregations have a higher overhead due to their bucket handling logic. While speed is improved for multi-bucket aggregations, the rate of that speed increase will flatten out sooner than single metric.

Figure 1. The speedup expected for aggregations of different constant overhead.

Here are some results on expected speed and error rate over an APM data set of 64 million documents.

The calculations are from: 300 query and aggregation combinations, 5 seeds, and 9 sampling probabilities. In total, 13,500 separate experiments generated the following graphs for median speedup and median relative error as a function of the downsample factor which is 1 / sample probability.

Figure 2. Median speedup as a function of the downsample factor (or 1 / probability provided for the sampler).

Figure 3. Median relative error as a function of the downsample factor (or 1 / probability provided for the sampler).

With a probability of 0.001, for half of the scenarios tested, there was an 80x speed improvement or better with a 4% relative error or less. These tests involved a little over 64 million documents but spread across many shards. More compact shards and larger data can expect better results.

But, you may ask, do the visualizations look the same?

Below are two visualizations showing document counts for every 5 minutes over 100+ million documents. The total set loads in seconds and is sampled in milliseconds. This is with almost no discernible visual difference.

Figure 4. Sampled vs unsampled document count visualizations.

Here is another example. This time the average transaction by hour is calculated and visualized. While visually these are not exactly the same, the overall trends are still evident. For a quick overview of the data to catch trends, sampling works marvelously.

Figure 5. Sampled vs. unsampled average transaction time by hour visualization.

Best practices for using sampling aggregation

Sampling shines when you have a large data set. In these cases you might ask, should I sample before the data is indexed in Elasticsearch? Sampling at query time and before ingestion are complimentary. Each has its distinct advantages.

When sampling at ingest time, it can save disk and indexing costs. However, if your data has multiple facets, you have to stratify sampling over facets when sampling before ingestion, unless you know exactly how it will be queried. This suffers from the curse of dimensionality and you could end up with underrepresented sets of facets. Furthermore, you have to cater for the worst case when sampling before ingestion. For example, if you want to compute percentiles for two queries, one which matches 50% of the documents and one which matches 1% of documents in an index, you can get away with 7X more downsampling for the first query and achieve the same accuracy.

Here is a summary of what to expect from sampling with the random_sampler at query time.

Figure 6. Relative error for different aggregations.

Sampling accuracy varies across aggregations (see Figure 5 for some examples). Here is a list of some aggregations in order of descending accuracy: percentiles, counts, means, sums, variance, minimum, and maximum. Metric aggregation accuracy will also be affected by the underlying data variation: the lower the variation in the values, the fewer samples you need to get accurate aggregate values. The minimum and maximum will not be reliable with outliers, since there is always a reasonable chance that the sampled set misses the one very large (or small) value in the data set. If you are using terms aggregations (or some partitioning such as date histogram), aggregate values for terms (or buckets) with few values will be less accurate or missed altogether.

Aggregations also have fixed overheads (see Figure 1 for an example). This means as the sample size decreases, the performance improvement will eventually level out. Aggregations which have many buckets have higher overheads and so the speedup you will gain from sampling is smaller. For example, a terms aggregation for a high cardinality field will show less performance benefit.

If in doubt, some simple experiments will often suffice to determine good settings for your data set. For example, suppose you want to speed up a dashboard; try reducing the sample probability while the visualizations look similar enough. Chances are your data characteristics will be stable and so this setting will remain reliable.

Uncovering how sampling works

Sampling considers the entire document set within a shard. Once it creates the sampled document set, sampling applies any provided user filter. The documents that match the filter and are within the sampled set are then aggregated (see Figure 7).

Figure 7. Typical request and data flow for the random_sampler aggregation.

The key to the sampling is generating this random subset of the shard efficiently and without statistical biases. Taking geometrically distributed random steps through the document set is equivalent to uniform random sampling, meaning each document in the set has an equally likely chance of being selected into the sample set. The advantage of this approach is that the sampling cost scales with p (where p is the probability configured in the aggregation). This means no matter how small p is, the relative latency of performing the sampling adds will remain fixed.

Ensuring performance reliability and accuracy

To achieve the highest performance, accuracy, and robustness, we evaluated a range of realistic scenarios.

In the case of random_sampler, the evaluation process is complicated by two factors:

1. It cuts right across the aggregation framework and so it needs to be evaluated with many different combinations of query and aggregation,

2. The results are random numbers, so rather than running just once, you need to run multiple times and test the statistical properties of the result set.

We began with a proof of concept that showed that the overall strategy worked and the performance characteristics were remarkable. However, there are multiple factors which can affect implementation performance and accuracy. For example, we found the off-the-shelf sampling code for the geometric distribution was not fast enough. We decided to roll our own using some tricks to extract more random samples per random bit along with a very fast quantized version of the log function. You also need to be careful that you are generating statistically independent samples for different shards. In summary, as is often the case, the devil is in the details.

Undaunted, we wrote a test harness using the Elastic Python client to programmatically generate aggregations and queries, and perform statistical tests of quality.

We wanted the approximations we produce to be unbiased. This means if you run a sampled aggregation repeatedly and averaged the results it would converge towards the true value. Standard machinery allows you to test if there is statistically significant evidence of bias. We used a t-test for the difference between the statistic and true value for each aggregation. In over 300 different experiments, the minimum p-value was around 0.0003 which — given we ran 300 experiments — has about a 9% odds of occurring by chance. This is a little low, but not enough to worry about; furthermore the median p-value was 0.38.

We also tested whether various index properties affect the statistical properties. For example, we wanted to see if we could measure a statistically significant difference between the distribution of results with and without index sorting. A K-S test can be used to check if samples come from the same distribution. In our 300 experiments the smallest p-value was around 0.002 which occurs with odds of about 45% by chance.

Get started today

We're not done with this feature yet. Once you have the ability to generate fast approximate results, a key question is: how accurate are those results? We're planning to integrate a confidence interval calculation directly into the aggregation framework to answer this efficiently in a future release. Learn more about random_sampler_aggregation in this documentation. You can explore this feature and more with a free 14-day trial of Elastic Cloud.

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.