For new users, this is the fastest way to get high-quality vector search running. If you already use Elasticsearch, the new offering is vector search on the platform where your data already lives, with no new system to adopt. Elasticsearch Vector Database supports a range of scenarios, from grounding a large language model (LLM), to giving an AI agent retrieval and memory, to serving hundreds of billions of vectors. Spin up a new project and get started in minutes.

One engine, every vector use case

Elasticsearch Vector Database is built for anyone building applications using vectors:

• RAG: Retrieve the right context for your LLM with dense and sparse vector retrieval, or go with hybrid search combining both vector and lexical retrieval. The quality of your generation improves with the quality of your retrieval.

• AI agents: Give agents fast, filtered retrieval over documents and conversation memory, with the low latencies that multistep agent loops demand.

• Semantic search: Match on meaning, not keywords, with one field type and zero pipeline code.

• Recommendations and similarity: Find nearest neighbors across products, images, or whatever content you have, at scale.

Everything your vector workload needs, optimized out of the box

Building a vector-based application means wiring together several separate pieces: setting up and hosting embedding models, indexing your documents through them, storing the vectors efficiently, applying the embedding model to each query, matching against the vector store, and finally, retrieving the documents behind the matches. Elasticsearch Vector Database handles all of it for you, with no additional configuration or setup.

Vector indexing with vectordb_document index mode

vectordb_document index mode, a new index configuration purpose-built for vector-first workloads, is on by default, so you get the settings that experts would choose. Here's what it turns on:

• bfloat16 by default: Vectors are stored at half the size of float32 with negligible impact on recall, cutting your disk footprint roughly in half before quantization even enters the picture.

• Source vectors excluded: In Elasticsearch, your embeddings already live in the index structures used for search; keeping a second raw copy in _source just inflates storage and slows down fetching results. We exclude the duplicate so responses return faster and you store less.

• The right files preloaded into cache: The data structures that vector queries touch first are warmed into memory ahead of time, so your first (and your thousandth) query is lightning fast.

• Parallel merging: Merging consolidates segments into better-organized vector structures, which lifts both recall and latency, and running those merges multi-threaded means you get there faster.

Vector storage, compression, and auto-tuning

• Your vectors are compressed automatically. Better Binary Quantization (BBQ) shrinks vector memory footprints by up to 32x while preserving recall, and DiskBBQ reduces memory requirements further for large-scale workloads. 

• Opt in to auto-calibration, which tunes each segment's quantization to your data and retunes on every merge as data drifts. When tested across 18 datasets, queries per second (QPS) improved by an average of 16.7%, with recall gains in most of them.

Embeddings on managed GPU inference

• Generate embeddings with native Jina AI embedding and reranking models, or bring third-party models, all on managed GPUs via Elastic Inference Service (EIS) with no model servers to operate. Or self-host, if you prefer your own.

• The semantic_text field type automatically handles chunking and embedding, along with querying, the simplest path to semantic search in the market. 

Hybrid search and filtered vector search

• Hybrid search is built in, combining full-text and vector retrieval in a single query. Blend the results with reciprocal rank fusion (RRF) or any other blending mechanism you want. Vector search is usually the hardest part of hybrid search to configure well. With Elasticsearch Vector Database, you have it handled, and your whole hybrid stack gets better. 

• With filtered vector search, apply metadata filters as part of vector retrieval itself and not as an afterthought that wrecks recall.

Enterprise on day one

You also get role-based access control (RBAC), audit logging, and the compliance certifications that pure-play vector databases generally lack.

Affordable at scale and predictable

Elasticsearch Vector Database is built to stay affordable as you grow: BBQ and DiskBBQ compression that keeps storage linear and memory low means scaling to hundreds of billions of vectors doesn't blow up your bill. And what you do pay is built from numbers you already know: how much data you store and how much you index, along with how much search capacity you need. Estimate your document count and vector dimensions, plus your query load, and you can work out what you'll pay before you create the project. You can also understand your bill line by line at the end of the month. There are no opaque compute units and no surprise charges for background operations.

How to get started with Elasticsearch Vector Database

Create a serverless vector database project

Create a new serverless Vector Database project in Elastic Cloud. Point your data at the endpoint, and you're ready to index.

Create an index using semantic_text

Vector index mode handles the vector configuration. Using semantic_text means that embeddings and chunking setup are managed for you, as is index setup, on managed GPU inference, with no embedding pipeline to build.

Ingest documents

Index text, and the embeddings are generated for you.

Run a semantic search query

Query the same semantic field you just created:

And you get results back:

Semantic search is just the start. Run fully textual queries or combine both into hybrid queries. You can even craft your own vector queries for full control. Follow the semantic search quickstart in the docs for the full instructions.

What's next for vector search in Elasticsearch

We're already working on the next improvements:

• Better multi-tenant handling: If your data needs to stay separated per tenant, we'll give you a way to do it faster and with less code.

• Automatic index optimization: From "brand new index" to "fully optimized," with as little tinkering as possible.

• Continuous infrastructure improvements: Ongoing tuning of Vector Database's settings and infrastructure so you're always getting the best throughput and fastest responses.

Try Elasticsearch Vector Database on Elastic Cloud Serverless

Go from an empty project to a hybrid, filtered vector query in minutes, with production-grade defaults doing the tuning for you. Build fast, scalable AI apps, not infrastructure.

Start on Elastic Cloud Serverless, or dive into the full documentation and API reference. You can also access the new offering on AWS Marketplace, Google Cloud Marketplace and Microsoft Marketplace.

AI shopping agents fail not because the model is wrong but because the agent arrives at every conversation not knowing your catalog, vocabulary, or business rules. It has to discover all of this context through tool calls, and that discovery is the cost. Precomputing a structured context layer from signals you already hold (vocabulary, policies, user profiles, session behavior) cuts the exploratory work the agent does before it can answer and makes its behavior governed and predictable.

In the analogous document-retrieval case, precomputing context reduced input tokens by up to 75% on a controlled benchmark; we expect comparable savings in ecommerce because the exploration pattern is the same, though the exact figure will vary by catalog and query mix. The signals are richer in ecommerce than in almost any other domain, and most retailers already have them. The question is whether or not these signals are assembled in a form that the agent can use before it starts reasoning. Elastic’s broad mix of search capabilities, semantic, hybrid, keyword, filtering, and aggregations make it a compelling choice for not only building your core retrieval tools but also as the vital context engine.

Why AI shopping agents fail in production

Retailers investing in AI shopping assistants are discovering an uncomfortable gap between what the demos promise and what the first few months in production deliver.

The assistant takes four seconds to respond. It confidently recommends a product in a size range that doesn't exist for that item. It tells a returning customer about a coat style they bought 18 months ago and returned. It filters by a category name that doesn't match the internal taxonomy and returns zero results. The customer gives up and abandons the chat altogether.

These aren't model failures. The frontier models powering these agents are capable of extraordinary reasoning when they have the right information in front of them. The problem is that the agent arrives at the conversation knowing nothing about the retailer's catalog, customer, or business rules that govern what should and shouldn't be recommended. It has to learn all of this through the conversation itself, making exploratory tool calls to discover what departments exist, what filter values are valid, and what the brand's policies are on certain product types. Every one of those discovery calls costs latency and tokens before the agent has said a single useful thing to the customer.

Latency matters in ecommerce in a way it doesn't in many other contexts. Shoppers expect response times measured in seconds, not the minutes that enterprise knowledge-base agents routinely take. It’s well established that slower responses reduce engagement and conversion in online retail. An AI shopping assistant that thinks visibly for five seconds before answering a question about gift ideas isn't a feature; it's friction.

The fix isn't a faster model or a bigger context window. The agent's latency and cost problem is a context problem. This can be solved by carefully computing context before the agent call, not during it.

What an AI shopping agent knows before it searches

Imagine you’re the agent. You have a search tool connected to the catalog, and this query arrives:

"an outfit for an autumn wedding"

What do you actually do with that?

Start with what you don't know. An outfit for a man or a woman? Is "autumn" a color, a season, a style of fabric, or just when the wedding happens? And who is this shopper, someone who has bought from you for years, or a stranger? Do they buy expensive designer brands or do they always hunt out a bargain on sale? You have none of these answers. So you do what an agent does when it's working blind: You ask loads of follow-up questions, guess, or fire off a series of exploratory searches to find out what departments and filter values even exist, watching the seconds tick by before you've offered the customer anything at all.

Hold onto that feeling of working blind. The rest of this article is about what changes when the agent is handed the answers first.

Why ecommerce is different from general RAG

Most of the published work on reducing agent costs focuses on document retrieval: question-answering over corpora of articles, reports, or knowledge-base entries. A recent experiment from the Elastic team (Cutting agent costs with pre-computed context) demonstrated that pre-extracting structured facts from documents before the agent call reduced input token consumption by up to 75% and improved answer accuracy from 60% to 92% on a hard factual benchmark. That improvement came in stages, with the largest jump driven by feeding the agent's own wrong answers back into the extraction step rather than by precomputing context alone, which is a distinction we'll return to when we discuss governance.

Ecommerce applies the same principle to a fundamentally different structure. A product catalog isn't a document corpus. It's a highly structured index of items with strict field semantics, a domain-specific vocabulary of brand names, color codes, and category hierarchies, and a layer of business rules that override pure relevance in specific situations.

The failure modes that result are distinct from document retrieval augmented generation (RAG) failures:

• Vocabulary mismatch: A customer asks for a "navy jumper." The agent constructs a filter against a field where the canonical value is NAVY and the category is stored as Knitwear & Jumpers. Without a vocabulary mapping, the agent either guesses colors and categories and gets it wrong, or makes multiple exploratory calls to discover what values exist before it can filter correctly.

• Hallucinated filter values: Without knowing which filter dimensions are valid for a given query, agents can construct queries against fields that don't exist or with values that return zero results. A filter like category: knitwear looks reasonable; masterCategoryNames: "Knitwear & Jumpers" is what the index actually contains. If the agent doesn’t know, it either hallucinates or has to do a separate tool call to find out, causing another LLM loop, which costs time and tokens.

• Context-free personalization: The same query from two different customers, one who typically shops in the premium range and dresses for formal occasions, and one who buys primarily casualwear under £40, should return different results. Without profile context, the agent treats every query identically, which is worse than a well-tuned keyword search because it creates the impression of a personal assistant while delivering generic answers.

The context layer: What signals it needs

The reason ecommerce is particularly well suited to precomputed context is that retailers already hold an unusually rich set of signals. The challenge isn't data availability; it's assembly.

The signals fall into two groups. Two of them, the catalog vocabulary and the business policies, are the genuinely original work and the heart of this approach. The rest, live facet state, user profiles, and session history, are valuable but closer to table stakes, signals that most teams already understand how to fetch. Here's what most mid-to-large retailers have and what each signal prevents.

Signal

What it prevents

Effort to build

Catalog vocabulary

Vocabulary mismatch and hallucinated filter values; the agent guessing at colors, categories, or brand names instead of resolving them to canonical field values

One-time engineering effort (full-catalog aggregation); incremental to maintain as new categories and brands are added

Business policies

Recommendations that ignore legal or trading requirements, for example, missing age verification on alcohol queries or routing that misses a gluten-free range

Human-authored and governed, not automated; ongoing review as new policy types are added

Live facet state

Recommending filters that return zero results or out-of-stock options for the current query

Runs in parallel with vocabulary and policy lookups; leans on existing catalog and retrieval infrastructure

User profile

Making a returning customer restate sizes, budget, or brand preferences they've already given

Fastest signal to retrieve, a single document lookup by user ID

Session and purchase history

Re-recommending an item the customer already dismissed or bought and returned

Most aspirational layer; depends on customer relationship management (CRM) and analytics integration, best added once the agent is already live

The reason we say semantic metadata and not just metadata is that we’re trying to match the semantic (meaning) of the intent rather than the exact words. If a user is searching for “teal,” we should be able to understand that this is a color and which colors exist in our products that are semantically similar to teal, even if none of them are actually teal. So, if we search for “teal,” we might want to return:

ProductColours = “aquamarine, turquoise”

Hopefully, you can see how this semantic metadata is bridging the gap between user intent and the agent’s knowledge of the products.

Catalog vocabulary: The foundation of context engineering

Catalog vocabulary is the layer that does the most work, and it's the one most worth getting right first.

A vocabulary index maps natural language to the exact field values and category paths used in the product index. It answers questions like: What does "navy" map to? Which categories fall under "knitwear"? Is "Autograph" a brand or a range? What's the correct spelling of a competitor brand the agent might encounter in a query?

What makes this more than a synonym list is how it's queried. The interesting thing a shopper says is rarely an exact field value. They say "something cozy for fall," not colour: NAVY and masterCategoryNames: "Knitwear & Jumpers". So the vocabulary index needs to resolve fuzzy, natural language intent into precise, exact-match filters, and that requires both kinds of matching at once: semantic search to understand that "cozy" leans toward knitwear and fleece, and exact keyword matching to pin the result to the canonical values the product index actually stores. A metadata index that supports both on the same documents is, in effect, a translation layer between how customers talk and how the catalog is structured.

This is also where the index earns the description "semantic metadata layer" rather than "lookup table." Each entry is a small natural language description of a facet value or schema concept, so the agent can match against meaning and then read back the exact filter to use. For a typical fashion retailer, this covers hundreds of color values, brand aliases, category synonyms, and size-range conventions. Building the semantic metadata layer from a full-catalog aggregation is a one-time engineering effort; maintaining it is incremental as new categories and brands are added. Without it, an agent encountering an unfamiliar term must either guess or make exploratory tool calls to discover what's there.

Business policies in the context layer

The second original layer is policy. Some queries carry implicit business requirements that pure relevance cannot handle. A query for "wine gift for a friend" should trigger an age-verification reminder in markets where it's legally required. A query for "gluten-free food gift" should route away from general confectionery toward the specific gluten-free range. A query mentioning "wedding guest outfit" in spring should apply different weighting than the same query in November.

These are policies, and most retail search teams already write them. They just call them boost rules, merchandising overlays, or synonym configurations. The difference in an agentic context is that instead of being applied silently as query modifications, they're surfaced as readable hints the agent can use when deciding how to frame its answer and which products to surface. The agent doesn't have to infer your trading rules from the catalog; it's handed them.

The critical point: These policies encode business intent, not just relevance. A policy that routes alcohol queries through an age-appropriate flow isn't a retrieval optimization; it's a trading requirement. That's why this layer must be human-authored and governed, not generated automatically from traffic patterns, a point we return to in the governance section.

Together, vocabulary and policy are what make the agent behave like it understands your business rather than just your data. The remaining three signals sharpen the experience, but they're more familiar engineering.

Facet, profile, and session signals in the context layer

• Live facet state: Before the agent recommends filters, it should know which filters are available and how many results each returns for this specific query. An agent that suggests "filter by size 8" without knowing that size 8 is out of stock for this query undermines the customer's trust immediately. A facet state query against the product index, run in parallel with the vocabulary and policy lookups, returns the counts, ranges, and available values specific to the current query.

• User profile: A persistent profile (sizes, color preferences, budget range, brand affinities) lets a returning customer skip restating what they've already told you. It's typically the fastest signal to retrieve, a single document lookup by user ID.

• Session and purchase history: Within a session, the agent should know what the customer has already seen, dismissed, or added to their basket, so it doesn't re-recommend a dismissed item or repeat itself. Longer-term purchase history extends this, and signals like return history are richer still, but using them well depends on data most retailers hold in systems that aren't yet wired into their search path. This is the most aspirational layer and the one best approached last, once the earlier layers are delivering value. There must also be balance when building the initial context not to overinflate the size, which will slow down the first reply and increase token costs. There is, therefore, a careful balance to strike between providing information like purchase history in the initial context or providing it as a tool for the agent to use during conversation, but users will expect that, if they’re logged in, the agent should know what they’ve purchased. The precise optimal context is likely to be specific to each implementation and customer experience and will require careful testing.

How context engineering works before the LLM call

The pattern that makes this work is simple to describe and moderately involved to implement:

Without this context build, the agent makes the same discoveries on its own, but through LLM-driven tool calls that each cost a full inference round trip. As a rough rule of thumb, an exploratory tool (for example, GetFilterValues) call tends to land somewhere in the region of 600ms to 1 second of latency in practice; an agent that discovers the vocabulary, checks policy hints, and retrieves facet state through three separate tool calls before it even begins answering can, therefore, add two to three seconds to the response time, and that's before it performs the actual product search. These are order-of-magnitude estimates, not benchmarked figures, and the real numbers depend heavily on the model, the network path, and how the tools are implemented.

Replacing those discovery calls with a parallel fetch (multiple context-building queries can be done in parallel) that runs before the LLM is invoked removes most of that cost. The context build takes roughly the same wall time as a single LLM tool call, but it replaces three or four of them. The LLM would either need to repeat failed searches or do its own context-building tool calls sequentially to get enough context to be successful. Precomputing context is also deterministic rather than subject to the model's tool-selection choices, which gives the business more control to fine-tune the experience.

The token reduction follows the same logic: Each exploratory tool call returns raw data the model must process. A preassembled context summary replaces that raw data with structured facts the model can consume in one pass. At the scale of usage possible in public-facing retail websites, this token cost saving can be significant. This work on document search (Cutting agent costs with pre-computed context) showed up to 75% input token reduction on a controlled benchmark. That benchmark was document retrieval rather than ecommerce, and its authors are explicit that the multiplier isn't a fixed number you can expect everywhere. We expect the direction to hold in ecommerce, because the exploration pattern is the same; it just runs against a structured catalog rather than a document corpus. The magnitude is something each team should measure against its own traffic.

The AI shopping agent with full context

Remember the query that left you guessing: an outfit for an autumn wedding. Run it again, but this time, before you have to think, you’re handed a precomputed context as a short brief:

• This shopper’s name is Sarah, female, age 32, and she buys womenswear, size 12.

• She typically buys your mid-tier ranges.

• "Autumn" here matches these specific color labels: “rust”, “burgundy”, “forest green”, “camel”.

• Matching departments: “Womenswear”, “Menswear”.

• Matching tags: "occasion dresses”, “trouser suits”, “wedding”.

• She already has a burgundy bag in her basket.

• House rule for wedding-guest looks: Complete the outfit. Show hats and accessories, not just the dress.

Suddenly, you’re not guessing; you’re styling for this customer. And the interesting part is how those facts combine rather than just stack. The season proposes a whole autumn palette; the burgundy bag already in her basket narrows that palette to the few tones that coordinate with it; the house rule tells you to finish the look with a matching fascinator rather than stopping at the dress. Together, these facts let you answer like someone who knows both this customer and this shop, in a single pass, with nothing invented.

That short brief is exactly what the signal stack produces: the shopper's profile, the resolved vocabulary, the live basket, and the business policy. These are assembled in parallel and placed in front of the agent before its first move, so the "What do I even do with this?" problem never has to be solved one expensive tool call at a time.

Governing the context layer without automated drift

One difference between the approach described here and automated knowledge extraction systems is worth addressing directly: In ecommerce, the context index cannot self-update without human review.

The policies that govern how an agent responds to gift queries, alcohol queries, or queries from customers in certain age brackets aren't just relevance configurations; they're trading decisions with potential legal and brand implications. An automated system that generates new policies from traffic patterns, without review, is a compliance risk before it's a technical asset.

This is actually the right constraint for most retail organizations, and it aligns with how search teams already work. Merchandisers write boost rules. Search teams maintain synonym configurations. Content teams approve what language appears in automated recommendations. The context policy layer is the same kind of governed configuration; it just serves a different consumer, namely, the agent's reasoning step rather than the query pipeline.

It's worth noting where this differs from the document-retrieval work referenced earlier. In that experiment, the biggest accuracy gain came from an automated feedback loop that fed the agent's wrong answers straight back into the extractor. That works well for factual question-answering, where "right" and "wrong" are unambiguous. In ecommerce, the equivalent signals still surface automatically, but a human decides what to do with them, because the changes carry trading and compliance weight. The loop is the same shape; the publication step has a person in it.

The governance loop that works in practice has two tiers:

• Automatic signal surfacing: Zero-result queries, repeated reformulations on the same topic, and sessions that end without a purchase after an agent interaction are all signals that something in the context layer is missing or wrong. These surface automatically as candidates for improving the experience, for example: a vocabulary term that didn't resolve or a policy that didn't fire on a query type it should have covered. To do this, you need a thorough log of conversations, including the reasoning and tool call trace in a platform like Elastic. This allows you to analyze the performance of the agent using both structured tools, for example, percentage increase in thumbs-down conversations and semantically. You could also run an automated review of conversations about "gifts" around December to characterize the thumbs-up/down ratio across an AB test of two agents.

• Human authorship and review: The search or merchandising team reviews candidates and authors the appropriate vocabulary entry or policy. Policies go through approval before publication. This typically mirrors the workflow that already exists for synonym changes or boost rule modifications; the tooling is the only new part.

How to implement context engineering in phases

• Phase 1: Vocabulary layer (highest leverage, bounded engineering task).

• Phase 2: Facet state and initial policies (leans on the same catalog and retrieval primitives).

• Phase 3: User profiles and session signals (requires CRM and analytics integration; best added when the agent is active).

• Phase 4: Governed feedback loop (shifts to organizational alignment; surfaces gaps for merchandising teams).

The full signal stack described above doesn't need to be built at once, and the order isn’t arbitrary. The highest-leverage starting point is also the lowest in implementation complexity: the vocabulary layer. A semantic metadata index built from a full-catalog aggregation (canonical color values, brand aliases, category paths, field names) is a bounded engineering task, and an agent that can resolve "navy jumper" to color: NAVY, masterCategoryNames: "Knitwear & Jumpers" before its first tool call is materially better than one that discovers this through trial and error. If you build nothing else, build this.

Facet state and the first set of policies can follow close behind, often in parallel, because they lean on the same catalog and the same retrieval primitives. The later layers, such as user profiles, session signals, and the governed feedback loop, are where the work shifts from search engineering to organizational alignment. To implement CRM system integration, merchandising workflow changes, and the analytics needed to surface gaps can take a significant amount of work. Those layers are more valuable once the agent is already in regular use and generating the traffic signals that make the governed loop worth running. The important feature is that each layer stands on its own, so a retailer gets real value from phase one without committing to phase six.

What infrastructure does a context layer need?

Precomputing context at the depth described here places specific requirements on the underlying platform. It's worth being explicit about these, because the temptation in early agent builds is to reach for the simplest available tool for each capability.

• Semantic search to match natural language queries against the vocabulary index and surface the right canonical values. Fuzzy keyword matching alone won't resolve ambiguity between similar brand names or color terms.

• A percolator to implement the policy layer. A percolator reverses the usual search direction: Instead of matching a query against stored documents, it stores the queries and matches an incoming piece of text (here, the customer's message) against them. That's exactly what policy matching needs, because each policy is essentially a saved pattern that says "When a query looks like this, surface this hint."

• Real-time aggregations over the full product catalog to produce accurate facet state at query time. Precomputed facet snapshots go stale quickly in active catalogs; query-time aggregations are the more reliable source.

• Document retrieval by key for user profiles: fast, single-document lookups by user ID that must complete within the context build window.

• Structured and semantic logging and analytics over query traces and agent interactions, which are the raw material for the governed loop's automatic signal surfacing.

These are standard capabilities of a mature search and analytics platform, rather than six separate systems, and Elasticsearch provides all of them in one place. That matters less as a procurement point than as an architectural one: When semantic matching, percolation, aggregations, profile lookups, and analytics all run against the same catalog in the same cluster, the context layer stays consistent with the search layer by construction. Splitting these across a separate vector store and a separate analytics platform is a legitimate choice, but it adds operational surface and introduces a consistency problem between two systems that are reasoning about the same products. The context infrastructure is simplest to run when it lives where the product data already lives and can be updated without a separate extract, transform, load (ETL) step.

Conclusion: Context engineering is a search team's job to own

The retailers who run effective AI shopping experiences at scale aren't the ones with the largest models or the most generous token budgets. They're the ones who have done the work to make their catalog, vocabulary, and tpolicies legible to an agent before it starts reasoning.

The good news is that most of the work is already done. The vocabulary is implicit in the catalog. The policies exist as merchandising rules and compliance guidelines. The user profiles are in the CRM. The session signals are in the analytics stream. The gap isn't data; it's the assembly layer that turns those signals into a structured context the agent can consume before it starts reasoning.

The search team already owns the vocabulary, policies, and merchandising workflows. The context layer is the right home for work the search team is already doing, in a form that serves the agent as well as the query pipeline. And, because it grows every time a gap is found and filled, it behaves less like a setup cost and more like an asset that compounds.

To begin building a context layer in Elastic, you can start a trial of Elastic Cloud or run locally. You should become familiar with configuring semantic search, and if you’re interested in how to build, store, and match search policies at query time, you will enjoy this blog.

What is recall?

Recall measures on a scale from 0 to 1 how many of the documents that your users actually want appear somewhere in your search results. If a query should surface three products and your search returns only two of them in the top 10, recall@10 = 0.67 for that query. It’s a set-based metric: It doesn’t care about the position of the relevant documents within those k results. A relevant document in position 10 counts the same as one in position 1. Having a high recall means that you’re not losing relevant results.

The diagram shows two sets: all relevant documents (left) and what BM25 actually retrieved (top 10, right). Only the intersection counts toward recall, prod_1 and prod_2 were found, while prod_3, prod_4, and prod_6 were missed entirely. Result: Recall@10 = 2/5 = 0.40.

Prerequisites

Let's get down to business to better understand how recall works. This demonstration uses Python. You can follow along with it on the companion notebook (notebook.ipynb), where every code block is a cell ready to run.

The code provided uses the following:

• Elasticsearch 9.3+

• Python 3.10+

• A .env file with your Elasticsearch credentials

The dataset

We’ll use a product catalog of 1,000 products, spanning categories such as footwear, electronics, tools, and more.

Each document has four fields:

Field

Type

`title`

text

`description`

text

`brand`

keyword

`category`

keyword

The dataset is loaded from dataset.csv.

The power and limits of lexical search

BM25 is the default ranking algorithm in Elasticsearch and most search engines. It scores documents by how often your query terms appear in them, adjusted for document length and the frequency of those terms across the entire index. You get analyzers on top: lowercase normalization, stemming, and stopword removal. A query for "running shoes" will match "Running Shoes" and likely "run" as well.

This works well for a large class of queries:

• "running shoes" immediately matches products with those exact tokens in the title.

• "bluetooth speaker" surfaces portable audio products because the tokens appear verbatim.

The results are deterministic and explainable: A document ranks highly because the query terms appear in it. Debugging relevance is straightforward.

Where it breaks

Now let’s try these queries against the same catalog:

• "skincare routine": The word "routine" doesn’t appear in any product title. BM25 can partially match on "skincare," but face serums, body oils, and moisturizers are described using terms like "vitamin C," "retinol," or "brightening," none of which overlap with the query. Products that form a complete skincare routine are scattered across the index with no shared token to anchor them.

• "pet travel accessories": This is a use-case grouping, not a product category. A dog sling carrier, a pet car seat, and a travel crate are all relevant, but their descriptions talk about portability, safety, and comfort rather than "travel accessories." BM25 matches "pet" broadly but has no signal to distinguish travel-specific products from the rest of the pet catalog.

This is a recall problem. The relevant documents exist in your index. BM25 just cannot find them because the user's words and the document's words do not match closely enough.

Adding synonyms helps for known cases. But you cannot enumerate every way a user might express an intent. That is where vectors come in.

Why you should measure recall

Before fixing a problem, you need to quantify it.

Recall@k measures how many of the documents that your users actually want appear somewhere in your search results. Formally:

Precision@k measures the top k results and how many are actually relevant:

High precision means that the results you do return are good. In ecommerce, missing a relevant product (low recall) is often worse than showing a slightly imperfect result (lower precision), because a hidden product is a lost sale.

Elasticsearch's rank_eval API lets you measure both systematically. You provide a list of queries, each with a set of rated documents, and Elasticsearch computes the metrics for you across all queries.

Setting up the evaluation

The rank_eval API needs a ratings dataset: a mapping of queries to the documents that are relevant for each one, along with a relevance grade (0 = not relevant, 1 = relevant, 2 = highly relevant).

In the notebook, this is the judgments list:

The mix is intentional: q1 is a query that BM25 handles well (exact tokens in product titles), while q2, q3, and q4 are intent-based queries where the user's intent is expressed as a concept rather than specific product keywords.

Measuring BM25 baseline recall

First, set up the Elasticsearch client and index the raw text data:

Now build the rank_eval request for BM25. Each request in the list combines a query with its ratings:

Result:

0.43 means that across all four queries, BM25 finds only 43% of the documents it should find. The shortfall is concentrated in the intent-based queries: "skincare routine" misses face serums and body oils because "routine" never appears in product titles, and "pet travel accessories" retrieves off-topic pet products while missing carriers and crates described in terms of portability and safety rather than "travel accessories."

This is our baseline. Now we have a number to beat.

Adding vector search with Jina embeddings

Vector search encodes documents and queries as high-dimensional vectors, a type of vector made up of hundreds or thousands of numerical values, each encoding a specific feature of the data it represents. Documents with similar meaning end up close together in vector space, even if they share no words. "Gym equipment" and "dumbbell set" will be nearby because the concepts are related. I chose Elasticsearch as my vector database because it supports hybrid search, giving me both semantic understanding and keyword precision out of the box.

EIS includes out-of-the-box support for embedding models through its inference API.

Step 1: Using Jina embeddings v5 as an inference endpoint

If your cluster has GPU resources (available in Elastic Cloud and Elasticsearch 9.3+), the embeddings are generated on GPU, which is significantly faster than CPU inference and removes the performance trade-off that historically made vectors expensive at scale.

Why Jina embeddings specifically? jina-embeddings-v5-text is a multilingual model (119+ languages) with a 32,000-token context window and support for task-specific Low-Rank Adaptation (LoRA) adapters. It works well for short product descriptions out of the box. Read more about jina-embeddings-v5-text model here.

Step 2: Create the index with a semantic field

The semantic_text field type is the key here. It’s a higher-level abstraction over dense_vector: You point it at an inference endpoint, and Elasticsearch takes care of generating embeddings automatically.

The copy_to property on title and description means content from both fields flows into semantic_field for embedding, so a single vector captures the full product representation.

Step 3: Index the products

At index time, Elasticsearch calls the inference endpoint for each document and stores the resulting embedding in semantic_field. No extra code on your side.

Hybrid search: Combining BM25 and vectors with RRF

Adding vectors improves recall, but using vectors alone risks losing precision on exact-match queries; "running shoes" should still rank verbatim matches first. Hybrid search retains the lexical component specifically to preserve that precision.

Hybrid search with Reciprocal Rank Fusion (RRF) keeps the best of both:

• BM25 handles exact and near-exact queries with high precision.

• Semantic search handles intent-based and multilingual queries with high recall.

• RRF combines the two ranked lists into a single ranking.

The RRF formula assigns each document a score based on its rank in each result list:

A document that ranks highly in both lists gets a higher combined score. The rank_constant controls how much weight lower-ranked documents receive.

Result:

Hybrid improves substantially over BM25 (0.43) and preserves precision for exact-match queries like "running shoes."

Results: Before and after

Here’s the full comparison across all three approaches:

Result:

Method

Recall@10

BM25 (Lexical)

0.43

Hybrid (BM25 + Vectors)

0.75

Breaking it down by query:

Conclusion

Throughout this post, we saw that BM25 lexical search is reliable when users type exact queries, but it loses recall when they search by intent rather than keywords. Using rank_eval, we established a reproducible baseline to measure that gap with real numbers. From there, we added a semantic_text field powered by Jina embeddings and ran the evaluation again. The result: Hybrid search improved recall from 0.43 to 0.75 while preserving precision on exact-match queries, though the actual margin will depend on your query mix.

The pattern scales beyond this example: Collect judgments from your users' actual queries, run rank_eval as a baseline, add semantic_text, and measure again. You'll know exactly what improved and by how much.

Next steps

• Dive deeper into recall and vector search: Recall and vector search quantization by Jeff Vestal

• Add reranking for even better precision on the top results

• Explore Elasticsearch hybrid search documentation

• Read more about the rank_eval API

As we saw in the previous post, the consistency provided by function calling is not just a nice optimization; it’s essential. Once we removed structural errors from the evaluation loop, results on standard scenarios (such as those in the tier 4 dataset) improved dramatically.

Yet there’s an obvious question left to answer:

Does this approach still work when things get genuinely messy?

Real-world entity resolution rarely fails because of simple cases. It fails when names cross languages, cultures, writing systems, time periods, and organizational boundaries. It fails when people are referenced by titles instead of names, when companies change names, when transliterations aren’t consistent, and when context (not spelling) is the only thing tying a mention to a real-world entity.

So, for the final post in this series, we put the system through what we called the ultimate challenge.

What makes this the ultimate challenge?

In earlier evaluations, we tested the system using increasingly complex datasets. By the time we reached tier 4, discussed in the previous post, we were already dealing with a mix of nicknames, titles, multilingual names, and semantic references. Those tests showed that the architecture itself was sound, but that reliability issues, especially malformed JSON, were suppressing recall.

With function calling in place, we finally had a stable foundation. That gave us the opportunity to ask a more interesting question:

Can one unified pipeline handle many different kinds of entity resolution problems at once?

The ultimate challenge dataset was designed to push precisely on that dimension.

Instead of focusing on a single difficulty (like nicknames or transliteration), this dataset combines 50+ distinct challenge types, including:

• Cultural naming conventions.

• Title-based references.

• Business relationships and historical name changes.

• Multilingual and cross-script mentions.

• Compound challenges that mix several of the above.

Crucially, this isn’t about optimizing for any one narrow use case. It’s about testing whether the design pattern holds up when the rules change from entity to entity.

The dataset at a glance

The ultimate challenge dataset consists of:

• 50 entities, spanning people, organizations, and institutions.

• ~60 articles, with varying structure and linguistic complexity.

• 51 distinct challenge categories, grouped broadly into:

  • Cultural naming conventions.

  • Titles and professional context.

  • Business and organizational relationships.

  • Multilingual and transliteration challenges.

  • Combined and edge‑case scenarios.

Earlier in the series, we saw that using generative AI (GenAI) to create datasets can be a mixed blessing. Without it, assembling sufficiently large and diverse test data would be extremely difficult. But left unchecked, the model has a tendency to make things too easy.

On an early generation pass, for example, we discovered that the model had included phrases like “the Russian president” as explicit aliases for Vladimir Putin. That might seem reasonable today, but it defeats the purpose of testing contextual resolution. What happens if the article is discussing Russia in the 1990s? The system should infer the correct entity from context, not rely on a hard-coded alias.

For that reason, this dataset was deliberately designed so that shortcuts don’t work. Aliases are not explicitly listed when the system is expected to infer meaning. Descriptive phrases are not prelinked to entities. Correct matches often depend on article-level context, not just local text.

Important note: Although we demonstrate the system’s capabilities across diverse scenarios, this is still an educational prototype. Production systems handling real-world sanctioned-entity monitoring would require additional validation, compliance checks, audit trails, and specialized handling for sensitive use cases.

Why these scenarios are hard

Back in the first post in this series, we introduced a simple but ambiguous example: “The new Swift update is here!” The challenge is that “Swift” can resolve to multiple real-world entities, depending on context. That example captures a broader truth: Natural language is inherently ambiguous.

Entity resolution, therefore, is not just a string-matching problem. Humans routinely rely on shared knowledge, cultural norms, and situational context to resolve references, and we rarely even notice we’re doing it.

Consider a few common cases:

• A title like “the president” is meaningless without geopolitical and temporal context.

• A company name may refer to a parent, a subsidiary, or a former brand depending on when the article was written.

• A person’s name may appear in different orders, scripts, or transliterations, depending on language and culture.

• The same phrase can legitimately refer to different entities in different contexts, and the system must be able to reject matches just as confidently as it accepts them.

There is no single rule set that handles all of this cleanly. That’s why this prototype separates concerns so aggressively:

• Elasticsearch narrows the candidate space efficiently and transparently.

• The LLM is used only where judgment is required and is forced to explain itself.

• Retrieval and reasoning remain distinct steps.

This separation becomes even more important as the diversity of challenge types increases.

How the system handles diversity without special cases

One of the most interesting outcomes of this evaluation is what didn’t change:

• We did not add special logic for Japanese names.

• We did not add custom rules for Arabic patronymics.

• We did not add hard-coded mappings for historical company names.

Instead, the system relied on the same core ingredients introduced earlier in the series:

• Context-enriched entities indexed for semantic search.

• Hybrid retrieval (exact, alias, and semantic) in Elasticsearch.

• A small, well-defined set of candidate matches.

• LLM judgment constrained by function calling and minimal schemas.

This suggests that the system’s flexibility comes from representation and architecture, not from an ever-growing collection of rules.

When the system succeeds, it’s because the right candidates are retrieved and the LLM has enough context to explain why a reference does (or does not) map to a specific entity.

Results: How did it perform?

On the ultimate challenge dataset, the system produced the following overall results:

• Precision: ~91%

• Recall: ~86%

• F1 Score: ~89%

• LLM acceptance rate: ~72%

Performance across challenge types

Breaking down results by challenge type reveals strengths and limitations:

Strongest performance (100% F1 score) was observed in areas such as:

• Cross-script matching (Cyrillic, Korean, Chinese business entities).

• Hebrew scenarios (patronymics, professional titles, religious titles, transliteration).

• Business hierarchies (aerospace, diversified manufacturing, multidivision corporations).

• Professional titles (academic, military, political, religious).

• Combined Japanese scenarios involving multiple writing systems.

Strong performance (80–99% F1 score) included:

• International political figures (98%).

• Historical name changes (90%).

• Complex business hierarchies (89%).

• Japanese company names (93%).

• Cross-script transliteration (86%).

• Arabic patronymics (86%).

More challenging areas included:

• Advanced transliteration (Chinese, Korean): 0% F1.

• Certain Japanese scenarios (honorifics, name order, writing system variation): ~67% F1.

• Some Arabic scenarios (company names, institutional references): ~40% F1.

What’s important here is why the system struggled in these cases. The failures were not due to the overall approach breaking down, but to limitations in specific components, most notably the dense vector model used for semantic search in certain multilingual scenarios.

Because retrieval and judgment are cleanly separated, improving performance does not require rewriting the system. Swapping in a more capable multilingual embedding model, enriching entity context, or refining retrieval strategies would improve results across these categories without changing the core architecture.

From an architectural standpoint, that’s the real success metric.

What this tells us about the design

Looking back across the series, a few patterns stand out:

• Preparation matters more than clever matching. Enriching entities with context up front dramatically reduces ambiguity later.

• LLMs are most valuable as judges, not retrievers. Asking them to explain why a match makes sense is far more powerful than asking them to search.

• Reliability enables accuracy. Function calling didn’t just clean up JSON; it unlocked recall that was already latent in the retrieval step.

• Generalization beats specialization. A small number of well-chosen abstractions handled dozens of challenge types without custom logic.

This is why the prototype is intentionally Elasticsearch-native and intentionally conservative in how it uses LLMs. The goal isn’t to replace search; it’s to make search explainable in situations where meaning matters.

Final thoughts

The ultimate challenge wasn’t about chasing perfect metrics; it was about answering a more fundamental question:

Can a transparent, search-first, LLM-assisted architecture handle real-world entity ambiguity without collapsing into rules or black boxes?

For this educational prototype, the answer is yes, with clear caveats around production hardening, compliance, monitoring, and data quality. If you’re building systems that need to justify why an entity match was made, this pattern is worth serious consideration. I hope this series has shown that entity resolution doesn’t have to be mysterious. With the right separation of concerns, it becomes something you can reason about, measure, and improve.

This work also suggests a broader architectural pattern. What emerges is a slight but important evolution of classic retrieval augmented generation (RAG). Instead of allowing retrieval to feed generation directly, we introduce an explicit evaluation step. The LLM is first used to judge and sanity-check retrieved candidates, and only those approved results are allowed to augment generation. You can think of this as Generation-Augmented Retrieval-Augmented Generation with Evaluation, or GARAGE, because who doesn’t love a good acronym.

What other use cases could benefit from this pattern? Systems that require trust, transparency, and defensible reasoning are natural candidates. Future work in this area should prove as compelling as the results we’ve seen here, and I’m excited to see where the community takes it next.

Next steps: Try it yourself

Want to see the ultimate challenge in action? Check out the Ultimate Challenge notebook for a complete walkthrough, with real implementations, detailed explanations, and hands-on examples.

The complete entity resolution pipeline demonstrates the core concepts and architecture needed for production use. You can use it as a foundation to build systems that monitor news articles, track entity mentions, and answer questions about which entities appear in which articles, all while retaining transparency and explainability.

In addition to Python and JavaScript, the LangChain ecosystem also has a community-driven Java project called LangChain4j, which will be the focus of this article, showing how powerful hybrid search can be by writing a complete application using LangChain4j, Elasticsearch, and Ollama.

Setting up the environment

Running a local Elasticsearch instance

Before running the examples, you'll need Elasticsearch running locally. The easiest way is using the start-local script:

After starting, you'll have:

• Elasticsearch at http://localhost:9200.

• Kibana at http://localhost:5601.

Your API key is stored in the .env file (under the elastic-start-local folder) as ES_LOCAL_API_KEY.

> Note: This script is for local testing only. Do not use it in production. For production installations, refer to the official documentation for Elasticsearch.

Running a local Ollama instance

You’ll also need to connect your application to an embedding model. Although you can choose between any provider supported by LangChain4j (check the complete list), for this example we’ll be using Ollama, which can be easily set up locally following the quickstart.

Let’s start coding

The idea for the application is simple: Given a dataset of movies (taken from an IMDb dataset on Kaggle), we want to be able to find movies whose descriptions are relevant to our queries. This demo uses a subset of the data, which has been cleaned. You can download the dataset used for this article from our GitHub repo, along with the full code for this demo.

Step 1: Dependencies and environment

Open your favorite integrated development environment (IDE), create a new blank project, preferably with a modern Java version (we’re using Java24) and a gradle/maven version to match (in our case, Gradle 9.0).

We only need three dependencies:

The first one is needed to ingest the data that we’ll embed and query; the other two are the necessary LangChain4j dependencies to connect and manage our Elasticsearch vector store and Ollama embedding model.

The best way to connect to the external services is to set up environment variables and set them at the start of our main function:

Step 2: Ingesting the dataset

Since the dataset is a CSV, we’ll be using Jackson dataformat’s jackson-dataformat-csv to easily read the data and map it to a Java class, defined as:

Now we can create an instance of CsvSchema mapping the CSV structure and read the file into an iterator:

Each row needs to be embedded first, and then both the embedded content and the text representation will be ingested by Elasticsearch.

Let’s start by creating an instance of the Ollama embedding model class:

And then the Elasticsearch vector store, which needs an instance of the Elasticsearch Java RestClient:

For the ingestion loop, the LangChain4j library requires the data to be split in two lists for ingestion, one for the vector representation and one for the original text, so we’ll set up two lists which will be filled by the loop:

Where Embedding and TextSegment are both library specific classes.

We’ll iterate on the movie dataset iterator, use the embedding model to retrieve the vector representation for each movie information (a text representation of all the fields merged), and add the name separately as metadata so that the result will be easier to read.

Finally, the vector list and text list are passed to the vector store method addAll(), which will handle asynchronously sending the data to the vector store:

Step 3: Querying

Our goal is to find movies with time loops in the plot, so our prompt will be:

Let’s try a simple vector search first, by creating a content retriever with a k-nearest neighbor (kNN) query default configuration and then running the query and printing the results:

This outputs:

Now let’s see how hybrid search performs:

Why these results?

This query (“time loop / reliving the same day”) is a great case where hybrid search tends to shine because the dataset contains literal phrases that BM25 can match and vectors can still capture meaning.

• Vector-only (kNN) embeds the query and tries to find semantically similar plots. Using a broad sci‑fi dataset, this can drift into “trapped / altered reality / memory loss / high-stakes sci‑fi” even when there’s no time-loop concept. That’s why results like “The Witch: Part 1 – The Subversion” (amnesia) and “The Maze Runner” (trapped / escape) can appear.

• Hybrid (BM25 + kNN + reciprocal rank fusion [RRF]) rewards documents that match keywords and meaning. Movies whose descriptions explicitly mention “time loop” or “relive the same day” get a strong lexical boost, so titles like “Edge of Tomorrow” (relive the same day over and over again…) and “Boss Level” (trapped in a time loop that constantly repeats the day…) rise to the top.

Hybrid search doesn’t guarantee that every result is perfect; it balances lexical and semantic signals, so you may still see some non-time-loop sci‑fi in the tail of the top‑k.

The main takeaway is that hybrid search helps anchor semantic retrieval with exact textual evidence when the dataset contains those keywords. Check the previous article for more information on how hybrid search works.

Full code example

You can find the full demo code on GitHub.

Conclusion

In this article, we demonstrated how to use hybrid search in LangChain4j through its Elasticsearch integrations, with a complete Java example. This article is an extension of a previous article, which presents the LangChain integrations for Python and JavaScript and introduces and explains hybrid search. We’re planning to continue our collaboration with LangChain4j in the future by contributing to the embedding models with our Elasticsearch Inference API.

The root of this problem was our somewhat naive choice to use prompt-based JSON generation in which the LLM generates JSON responses in text format. If we asked the LLM to judge more than a couple of matches at a time, the generated JSON was often ill-formed. To mitigate this, we were forced to reduce the processing batch size, which simply won't scale in a production system.

So the prompt-based JSON generation helped validate our approach to entity resolution, but we need a more systematic and reliable method. OpenAI function calling provides a better path by guaranteeing structure and type safety while reducing errors and costs. We chose OpenAI's functions for the educational prototype, but other LLM providers typically provide similar functionality (for example, Claude tools).

Note: While we discuss production challenges here, this is still an educational prototype demonstrating optimization techniques. Real production systems would need additional considerations, like monitoring, alerting, fallback strategies, and comprehensive error handling.

Key concepts: Function calling, schema design, and cost benefits

What is function calling? Function calling is OpenAI's structured output API. With it, we can define schemas for LLM responses, so we always know exactly what we're going to get. By enforcing the JSON format rather than trying to define it in the LLM prompt, we should be able to eliminate parsing errors.

Why is it better than prompt-based JSON? LLMs generate nondeterministic output. One hopes that they'll at least generate content that contains the correct response, but the presentation of that response is unpredictable. With a chatbot, this is often not a problem, but our prototype is trying to programmatically process the output. Computer programs demand consistency, so when the LLM generates what we expect, everything is fine, but as soon as it goes off script, so to speak, the code errors out. We could try to account for the different possibilities, but it would be very difficult to catch everything. We could try to enforce more consistent behavior by adding something like "Always return parsable JSON". We tried this exact technique in the prototype's prompt, but we've seen that prompt-based JSON still goes off the rails pretty quickly, particularly if we try to process a batch of matches.

Function calling makes the LLM generation controllable and predictable, exactly what we need for entity resolution. To aid in the definition of the functions, we’ll also follow minimal schema design principles.

What are minimal schema design principles? Minimal schema design means defining only the fields you need, using simple types, and avoiding nested structures when possible. This reduces token usage (smaller schemas mean fewer tokens), improves reliability (simpler schemas are easier for the LLM to follow), and lowers costs (fewer tokens mean lower API costs).

What are the cost and reliability benefits? Since fewer errors means match processing is much more likely to succeed, even with large batch sizes, we don't have to retry judging matches. The elimination of retries reduces costs by reducing token usage, but using minimal schemas also keeps our token count down. This all leads to a less expensive and more reliable approach that’s much more suitable to use in production.

We need to check one more thing, though. While matches may be getting processed without error, are the errorless results actually correct? How does this new approach compare to the promising results we saw with the prompt-based approach?

Real-world results: Side-by-side comparison

As we did in the previous blog, we ran the function calling approach against the tier 4 dataset, which consists of 206 expected matches across 69 articles. The results demonstrate a dramatic improvement:

Metric

Prompt-based

Function calling

Improvement

Error rate

30.2%

0.0%

100% elimination

Precision

83.8%

90.3%

+6.5pp

Recall

62.6%

90.8%

+28.2pp

F1 score

71.7%

90.6%

+18.9pp

Acceptance rate

44.8%

60.2%

+15.4pp

True positives

129

187

+45.0%

False negatives

77

19

-75.3%

Error elimination: The key differentiator

The most striking difference is the complete elimination of JSON parsing errors. This resulted in a modest precision improvement and a far more dramatic recall improvement. The precision metric captures how often the matches the system accepts were expected in the golden document. So the prototype was decent at judging matches correctly in the prompt-based approach, but function calling does that even better.

Conversely, recall tells us how many of the expected matches were found. When a batch of matches comes back with malformed JSON, the system loses all of those matches. It's likely that Elasticsearch sends many of these matches for judgment, but we lose those matches if judgment fails. The significant recall improvement shows that this hypothesis is correct. Elasticsearch identifies the potential matches and function calling verifies which of those matches are correct.

Note: It’s expected that Elasticsearch will find some incorrect matches because we look at the top two or three results from hybrid search. Most of the time, hybrid search returns the correct match as the top result, but having the LLM judge the top few hits ensures that we see how the LLM handles incorrect matches. If we move from the educational prototype to a production system, we’ll likely tune the Elasticsearch queries more carefully so that we only send promising matches to the LLM, further optimizing our LLM costs.

What's next: The ultimate challenge

Now that we've optimized our LLM integration with function calling, we have a complete entity resolution pipeline with improved reliability and cost efficiency. However, can it handle the ultimate challenge? In the next post, we'll explore how the system handles diverse entity resolution scenarios across 50 different challenge types, including cultural naming conventions, business relationships, titles, and multilingual variations.

Try it yourself

Want to see function calling optimization in action? Check out the Function Calling Optimization notebook for a complete walkthrough with real implementations, detailed explanations, and hands-on examples. The notebook shows you exactly how to use function calling for structured output, compare it with prompt-based JSON, and analyze cost and reliability benefits.

Remember: This is an educational prototype designed to teach optimization concepts. When building production systems, consider additional factors, like multi-provider support, advanced caching strategies, monitoring and alerting, comprehensive error handling, and compliance requirements that aren't covered in this learning-focused prototype.

We’re also planning to support hybrid search in the community-driven Java integration very soon.

What is hybrid search?

Hybrid search is an information retrieval approach that combines keyword-based full-text search (lexical matching) with semantic search (vector similarity). Practically, it means a query can match documents because they contain the right terms and/or because they express the right meaning (even if the wording differs).In simple terms, you can think of it like this:

• Lexical retrieval: “Do these documents contain the words I typed (or related words)?”

• Semantic retrieval: “Do these documents mean something similar to what I typed?”

These two retrieval methods produce scores on different scales, so hybrid search systems typically use a fusion strategy to merge them into one ranking, for example, using reciprocal rank fusion (RRF).

In the figure above, we show an example: BM25 (keyword search) returns Docs A, B, and C, while semantic search returns Docs X, A, and B. The RRF algorithm then combines these two result lists into the final ranking: Doc A, Doc B, Doc X, and Doc C. With hybrid search, Doc C is included in the results thanks to BM25.

Why hybrid search matters

If you’ve built search or retrieval-augmented generation (RAG) features in production, you’ve probably seen the same failure modes show up again and again:

• Keyword search can be too literal. If the user doesn’t use the exact terms that appear in your documents, relevant content gets buried or missed.

• Semantic search can be too fuzzy. It’s great at meaning, but it can also return results that feel related while missing a critical constraint, like a product name, an error code, or a specific phrase the user actually typed.

Hybrid search exists because real user queries in production environments usually need both.

Next we’ll dive into how you get started with hybrid search in the LangChain integration for Python and JavaScript. If you want to read more about hybrid search, check out What is hybrid search?and When hybrid search truly shines.

Setting up a local Elasticsearch instance

Before running the examples, you'll need Elasticsearch running locally. The easiest way is using the start-local script:

After starting, you'll have:

• Elasticsearch at http://localhost:9200.

• Kibana at http://localhost:5601.

Your API key is stored in the .env file (under the elastic-start-local folder) as ES_LOCAL_API_KEY.

Getting started with hybrid search in LangChain (Python and JavaScript)

The dataset is a CSV with information on 1,000 science fiction movies, taken from an IMDb dataset on Kaggle. This demo uses a subset of the data, which has been cleaned. You can download the dataset used for this article from our GitHub gist, along with the full code for this demo.

Step 1: Install what you need.

First you’ll need the LangChain Elasticsearch integration and Ollama for embeddings. (You can also use some other embedding model if you wish.)

In Python:

In JavaScript:

Step 2: Configure your connection and dataset path.

In Python:

At the top of the script, we set:

• Where Elasticsearch is (ES_LOCAL_URL).

• How to authenticate (ES_LOCAL_API_KEY).

• Which demo index name to use (INDEX_NAME).

• Which CSV file we’ll ingest (scifi_1000.csv).

In JavaScript:

Notes for JavaScript:

• JavaScript uses process.env instead of os.getenv.

• Path resolution requires fileURLToPath and dirname for Elasticsearch modules.

• The class is called ElasticVectorSearch (not ElasticsearchStore as in Python).

We can now also create the client.

In Python:

In JavaScript:

Step 3: Ingest the dataset, and then compare vector-only vs. hybrid.

Step 3a: Read the CSV and build what we index.

We build three lists:

• texts: The actual text that will be embedded + searched.

• metadata: Structured fields stored alongside the document.

• ids: Stable IDs (so Elasticsearch can dedupe if needed).

In Python:

In JavaScript:

What’s important here:

• We don’t embed only the description. We embed a combined text block (title/year + director + genre + description). That makes results easier to print and sometimes improves retrieval.

• The same text is what the lexical side uses, too (in hybrid mode), because it’s indexed as searchable text.

Step 3b: Add texts to Elasticsearch using LangChain.

This is the indexing step. Here we embed texts and write them to Elasticsearch.

For asynchronous applications, please use AsyncElasticsearchStore with the same API.

You can find our reference docs for both the sync and async versions of ElasticsearchStore, along with more parameters for advanced fine-tuning RRF.

In Python:

In JavaScript:

Step 3c: Create another store for hybrid search.

We create another ElasticsearchStore object pointing at the same index but with different retrieval behavior: hybrid=False is vector-only search and hybrid=True is hybrid search (BM25 + kNN, fused with RRF).

In Python:

In JavaScript:

Step 3d: Run the same query both ways, and print results.

As an example, let’s run the query “Find movies where the main character is stuck in a time loop and reliving the same day." and compare the results from hybrid search and vector search.

In Python:

In JavaScript:

Example output

Why these results?

This query (“time loop / reliving the same day”) is a great case where hybrid search tends to shine because the dataset contains literal phrases that BM25 can match and vectors can still capture meaning.

• Vector-only (kNN) embeds the query and tries to find semantically similar plots. Using a broad sci‑fi dataset, this can drift into “trapped / altered reality / memory loss / high-stakes sci‑fi” even when there’s no time-loop concept. That’s why results like “The Witch: Part 1 – The Subversion” (amnesia) and “The Maze Runner” (trapped/escape) can appear.

• Hybrid (BM25 + kNN + RRF) rewards documents that match both keywords and meaning. Movies whose descriptions explicitly mention “time loop” or “relive the same day” get a strong lexical boost, so titles like “Edge of Tomorrow” (relive the same day over and over again…) and “Boss Level” (trapped in a time loop that constantly repeats the day…) rise to the top.

Hybrid search doesn’t guarantee that every result is perfect. It balances lexical and semantic signals so you may still see some non-time-loop sci‑fi in the tail of the top‑k.

The main takeaway is that hybrid search helps anchor semantic retrieval with exact textual evidence when the dataset contains those keywords.

Full code example

You can find our full demo code in Python and JavaScript, as well as the dataset used, hosted on GitHub gist.

Conclusion

Hybrid search provides a pragmatic and powerful retrieval strategy by combining traditional BM25 keyword search with modern vector similarity into a single, unified ranking. Instead of choosing between lexical precision and semantic understanding, you get the best of both worlds, without adding significant complexity to your application.

In real-world datasets, this approach consistently yields results that feel more intuitively correct. Exact term matches help anchor results to the user’s explicit intent, while embeddings ensure robustness against paraphrasing, synonyms, and incomplete queries. This balance is especially valuable for noisy, heterogeneous, or user-generated content, where relying on only one retrieval method often falls short.

In this article, we demonstrated how to use hybrid search in LangChain through its Elasticsearch integrations, with complete examples in both Python and JavaScript. We’re also contributing to other open-source projects, such as LangChain4j, to extend hybrid search support with Elasticsearch.

We believe hybrid search will be a key capability for generative AI (GenAI) and agentic AI applications, and we plan to continue collaborating with libraries, frameworks, and programming languages across the ecosystem to make high-quality retrieval more accessible and robust.

Multistage retrieval: The challenge of modern search

Modern search has evolved beyond simple keyword matching. Today's search applications need to understand intent, handle natural language, and combine multiple ranking signals to deliver the best results.

Retrieval of the most relevant results happens in multiple stages, with each stage gradually refining the result set. This wasn’t the case in the past, where most use cases would require one or two stages of retrieval: an initial query to get results and a potential rescoring phase.

We start with an initial retrieval, where we cast a wide net to gather results that are relevant to our query. Since we need to sieve through all the data, we should use techniques that return results fast, even when we index billions of documents.

We therefore employ trusted techniques, such as lexical search that Elasticsearch has supported and optimized since the beginning, or vector search, where Elasticsearch excels in speed and accuracy.

Lexical search using BM25 is quite fast and best at exact term matching or phrase matching, and vector or semantic search is better suited for handling natural language queries. Hybrid search combines lexical and vector search results to bring the best from both. The challenge that hybrid search solves is that vector and lexical search have completely different and incompatible scoring functions which produce values in different intervals, following different distributions. A vector search score close to 1 can mean a very close match, but it doesn’t mean the same for lexical search. Hybrid search methods, such as reciprocal rank fusion (RRF) and linear combination of scores, assign new scores that blend the original scores from lexical and vector search.

After hybrid search, we can employ techniques such as semantic reranking and Learning To Rank (LTR), which use specialized machine learning models to rerank the result.

With our most relevant results, we can use large language models (LLMs) to further enrich our response or pass the most relevant results as context to LLMs in agentic workflows in tools such as Elastic Agent Builder.

ES|QL is able to handle all these stages of retrieval. By design, ES|QL is a piped language, where each command transforms the input and sends the output to the next command. Each stage of retrieval is represented by one or more consecutive ES|QL commands. In this article, we show how each stage is supported in ES|QL.

Vector search

In Elasticsearch 9.2, we introduced tech preview support for dense vector search in ES|QL. This is as simple as calling the knn function, which only requires a dense_vector field and a query vector:

This query executes an approximate nearest neighbor search, retrieving 100 documents that are the most similar to the query_vector.

Hybrid search: Reciprocal rank fusion

In Elasticsearch 9.2, we introduced support for hybrid search using RRF and linear combination of results in ES|QL.

This allows combining vector search and lexical search results into a single result set.

To achieve this in ES|QL, we need to use the FORK and FUSE commands. FORK runs multiple branches of execution, and FUSE merges the results and assigns new relevance scores using RRF or linear combination.

In the following example, we use FORK to run two separate branches, where one is doing a lexical search using the match function, while the other is doing a vector search using the knn function. We then merge the results together using FUSE:

Let's decompose the query to better understand the execution model and first look at the output of the FORK command:

The FORK commands outputs the results from both branches and adds a _fork discriminator column:

_id

title

_score

_fork

4001

The Hobbit

0.88

fork1

3999

The Fellowship of the Ring

0.88

fork1

4005

The Two Towers

0.86

fork1

4006

The Return of the King

0.84

fork1

4123

The Silmarillion

0.78

fork1

4144

The Children of Húrin

0.79

fork1

4001

The Hobbit

4.55

fork2

3999

The Fellowship of the Ring

4.25

fork2

4123

The Silmarillion

4.11

fork2

4005

The Two Towers

3.8

fork2

4006

The Return of the King

4.1

fork2

As you’ll notice, certain documents appear twice, which is why we then use FUSE to merge rows that represent the same documents and assign new relevance scores. FUSE is executed in two stages:

• For each row, FUSE assigns a new relevance score, depending on the hybrid search algorithm that is being used.

• Rows that represent the same document are merged together, and a new score is computed.

In our example, we’re using RRF. As a first step, FUSE assigns a new score to each row using the RRF formula:

Where the rank_constant takes a default value of 60 and rank(doc)represents the position of the document in the result set.

In the first phase, our results become:

_id

title

_score

_fork

4001

The Hobbit

1 / (60 + 1) = 0.01639

fork1

3999

The Fellowship of the Ring

1 / (60 + 2) = 0.01613

fork1

4005

The Two Towers

1 / (60 + 3) = 0.01587

fork1

4006

The Return of the King

1 / (60 + 4) = 0.01563

fork1

4123

The Silmarillion

1 / (60 + 5) = 0.01538

fork1

4144

The Children of Húrin

1 / (60 + 6) = 0.01515

fork1

4001

The Hobbit

1 / (60 + 1) = 0.01639

fork2

3999

The Fellowship of the Ring

1 / (60 + 2) = 0.01613

fork2

4123

The Silmarillion

1 / (60 + 3) = 0.01587

fork2

4005

The Two Towers

1 / (60 + 4) = 0.01563

fork2

4006

The Return of the King

1 / (60 + 5) = 0.01538

fork2

Then the rows are merged together and a new score is assigned. Since a SORT _score DESC follows the FUSE command, the final results are:

_id

title

_score

4001

The Hobbit

0.01639 + 0.01639 = 0.03279

3999

The Fellowship of the Ring

0.01613 + 0.01613 = 0.03226

4005

The Two Towers

0.01587 + 0.01563 = 0.0315

4123

The Silmarillion

0.01538 + 0.01587 = 0.03125

4006

The Return of the King

0.01563 + 0.01538 = 0.03101

4144

The Children of Húrin

0.01515

Hybrid search: Linear combination of scores

Reciprocal rank fusion is the simplest way to do hybrid search, but it isn’t the only hybrid search method that we support in ES|QL.

In the following example, we use FUSE to combine lexical and semantic search results using linear combination of scores:

Let's first decompose the query and take a look at the input of the FUSE command when we only run the FORK command.

Notice that we use the match function, which is able to not only query lexical fields, such as text or keyword, but also semantic_text fields.

The first FORK branch executes a semantic query by querying a semantic_text field, while the second one executes a lexical query:

The output of the FORK command can contain rows with the same _id and _index values representing the same Elasticsearch document:

_id

title

_score

_fork

4001

The Hobbit

0.88

fork1

3999

The Fellowship of the Ring

0.88

fork1

4005

The Two Towers

0.86

fork1

4006

The Return of the King

0.84

fork1

4123

The Silmarillion

0.78

fork1

4144

The Children of Húrin

0.79

fork1

4001

The Hobbit

4.55

fork2

3999

The Fellowship of the Ring

4.25

fork2

4123

The Silmarillion

4.11

fork2

4005

The Two Towers

3.8

fork2

4006

The Return of the King

4.1

fork2

In the next step, we use FUSE to merge rows that have the same _id and _index values, and assign new relevance scores.

The new score is a linear combination of the scores the row had in each FORK branch:

Here, _score1 and _score2 represent the score a document has in the first FORK branch and the second FORK branch, respectively.

Notice that we also apply custom weights, giving more weight to the semantic score over the lexical one, resulting in this set of documents:

_id

title

_score

4001

The Hobbit

0.7 * 0.88 + 0.3 * 4.55 = 1.981

3999

The Fellowship of the Ring

0.7 * 0.88 + 0.3 * 4.25 = 1.891

4006

The Return of the King

0.7 * 0.84 + 0.3 * 4.1 = 1.818

4123

The Silmarillion

0.7 * 0.78 + 0.3 * 4.11 = 1.779

4005

The Two Towers

0.7 * 0.86 + 0.3 * 3.8 = 1.742

4144

The Children of Húrin

0.7 * 0.79 + 0.3 * 0 = 0.553

One challenge is that the semantic and lexical scores can be incompatible to apply the linear combination, since they can follow completely different distributions. To mitigate this, we first need to normalize the scores, employing score normalization methods, such as minmax. This ensures that the scores from each FORK branch are first normalized to take values between 0 and 1, before applying the linear combination formula.

To achieve this with FUSE, we need to specify the normalizer option:

Semantic reranking

At this stage, after hybrid search, we should be left with the most relevant documents. We can now use semantic reranking to reorder the results using the RERANK command. By default, RERANK uses the latest Elastic semantic reranking machine learning model, so no additional configuration is needed:

We now have our best results, sorted by relevance.

One key feature that sets the RERANK command apart from other products that offer semantic reranking integrations is that it doesn’t require the input to represent a mapped field from an index. RERANK only expects an expression that evaluates to a string value, making it possible to do semantic reranking using multiple fields:

LLM completions

Now we have a set of highly relevant, reranked results.

At this stage, you might simply decide to return the results back to your application or you might want to further enhance your results using LLM completions.

If you’re using ES|QL as part of a retrieval-augmented generation (RAG) workflow, you can choose to call your favorite LLM directly from ES|QL. To achieve this, we’ve added a new COMPLETION command that takes in a prompt, a completion inference ID which designates which LLM to call, and a column identifier to specify where to output the LLM response.

In the following example, we’re using COMPLETION to add a new _completion column that contains the summary of the content column:

Each row now contains a summary:

_id

title

_score

summary

4001

The Hobbit

0.03279

Bilbo helps dwarves reclaim Erebor from the dragon Smaug.

3999

The Fellowship of the Ring

0.03226

Frodo begins the quest to destroy the One Ring.

4005

The Two Towers

0.0315

The Fellowship splits; war comes to Rohan; Frodo nears Mordor.

4123

The Silmarillion

0.03125

Ancient myths and history of Middle-earth's First Age.

4006

The Return of the King

0.3101

Sauron is defeated and Aragorn is crowned King.

4144

The Children of Húrin

0.01515

The tragic tale of Túrin Turambar's cursed life.

In another use case, you may simply want to answer a question using the proprietary data that you have indexed in Elasticsearch. In this case, the best search results that we’ve computed in the previous stage can be used as context for the prompt:

Since the COMPLETION command unlocks the ability to send any prompt to an LLM, the possibilities are endless. Although we’re only showing a few examples, the COMPLETION command can be used in a wide range of scenarios, from security analysts using it to assign scores depending on whether a log event can represent a malicious action or data scientists using it to analyze data, to cases where you just need to generate Chuck Norris facts based on your data.

This is only the beginning

In the future, we’ll be expanding ES|QL to improve semantic reranking for long documents, better conditional execution of the ES|QL queries using multiple FORK commands, support sparse vector queries, removing close duplicate results to enhance result diversity, allowing full text search on runtime generated columns, and many other scenarios.

Additional tutorials and guides:

• ES|QL for search

• ES|QL for search tutorial

• Semantic_text field type

• FORK and FUSE documentation

• ES|QL search functions

Relevance at scale: Cypris’ search story

Cypris is a platform that helps R&D and innovation teams navigate a massive dataset of patents and research papers of over 500 million documents. Their mission is to make it easier to track innovation, find prior art, and understand the organizations driving new technologies.

But there was a problem. To get relevant results, users had to write complex boolean queries—which was fine for expert users, but a barrier for many others. Cypris needed a way to make search more intuitive and accessible.

The answer was semantic search powered by vector similarity. However, they discovered that scaling semantic search over a large corpus turned out to be a tough engineering problem. Handling 500 million high dimensional vectors wasn’t just a matter of pushing them into a system and hitting “search.” “When we first indexed all 500 million vectors, we were looking at 30- to 60-second query times in the worst case.”

It would require a series of carefully considered trade-offs between model complexity, hardware resources, and indexing strategy.

Logan Pashby is a Principal Engineer at Cypris, where he focuses on the platform's innovation intelligence features. With expertise in topics such as deep learning, distributed systems, and full-stack development, Logan solves complex data challenges and develops efficient search solutions for R&D and IP teams.

Choosing the right model

Cypris’ first attempt at vector search used 750-dimensional embeddings for every document, but they quickly realized scaling such large embeddings across 500 million documents would be unmanageable. By using the memory approximation formula without quantization, the estimated bytes of RAM required would be around 1500 GB, making it clear that they needed to adjust their strategy.

“We assumed, and we hoped, that the larger the dimension of the vector, the more information we could encode. A richer embedding space should mean better search relevance.”

They considered using sparse vectors like Elastic’s ELSER which avoids the fixed-dimension limitations of dense embeddings by representing documents as weighted lists of tokens instead. However, at the time, ELSER’s CPU-only inference seemed too slow for Cypris’s dataset. Dense vectors, on the other hand, let them leverage off-cluster GPU acceleration, which improved throughput by 10x to 50x when generating embeddings.

The team ultimately decided on lower-dimensional dense vectors that struck a balance: they were compact enough to make indexing and search feasible, yet rich enough to maintain relevance in results.

Making hybrid search work with production scale data

Challenges - disk space

Once Cypris had vectors ready to be indexed, they faced the next hurdle: efficiently storing and searching over them in Elasticsearch.

The first step was reducing disk space. “At the end of the day, vectors are just arrays of floats.... But when you have 500 million of them, the storage requirements add up quickly.” By default, vectors in Elasticsearch are stored multiple times: first in the _source field (the original JSON document), then in doc_values (columnar storage optimized for retrieval), and finally within the HNSW graph itself. Given that each 750-dimensional float32 vector takes about 3KB, storing 500 million vectors quickly becomes problematic, potentially exceeding 1.5 terabytes per storage layer.

One practical optimization Cypris used was excluding vectors from the source document in Elasticsearch. This helped reduce overhead, but it turned out disk space wasn’t the biggest challenge. The bigger challenge was memory management.

Did you know?

Elasticsearch allows you to optimize disk space by excluding vectors from the source document. This can significantly reduce storage costs, especially when dealing with large datasets. However, be aware that excluding vectors from the source will impact reindexing performance. For more details, check out the Elasticsearch documentation on source filtering.

Challenges - RAM explosion

Known nearest neighbor (kNN) search in Elasticsearch relies on HNSW graphs, which perform best when fully loaded into RAM. With 500 million high-dimensional vectors, there were significant memory demands on the system. “Trying to fit all of those vectors in memory at query time was not an easy thing to do,” Logan adds.

Cypris had to juggle multiple memory requirements: the vectors and their HNSW graphs needed to reside in off-heap memory for fast search performance, while the JVM heap had to remain available for other operations. On top of that, they still needed to support traditional keyword search, and the associated Elasticsearch inverted index would need to stay in memory as well.

Managing memory with dimensionality reduction, quantization, and segments

Cypris explored multiple approaches to better manage memory and storage, here were three that worked well:

• Lower-dimensional vectors: The Cypris team swapped to using a smaller model that reduced vector sizes, thereby lowering resource requirements.

• BBQ (Better Binary Quantization): Cypris was considering int8 quantization, but when Elastic released BBQ, Cypris adopted it quickly. “We tested it out and it didn’t have a huge hit to relevance and was significantly cheaper. So we implemented it right away”, says Logan. BBQ immediately reduced the size of their vector indexes by around 20%!

Did you know?

Elasticsearch’s Binary Quantized Vectors (BBQ) can reduce the size of vector indexes by ~20%, with minimal impact on search relevance. BBQ reduces both disk usage—by shrinking index size—and memory usage, since smaller vectors take up less space in RAM during searches. It’s especially helpful when scaling KNN search with HNSW graphs, where keeping everything in memory is critical for performance. Explore how BBQ can optimize your search infrastructure in the Elasticsearch documentation on vector search.

• Segment and shard tuning: Cypris also optimized how Elasticsearch segments and shards were managed. HNSW graphs are built per segment, so searching dense vectors means querying across all segments in a shard. As Logan explains: “HNSW graphs are independent within each segment and each dense vector field search involves finding the nearest neighbors in every segment, making the total cost dependent on the number of segments.” Fewer segments generally mean faster searches—but aggressively merging them can slow down indexing. Since Cypris ingests new documents daily, they regularly force-merge segments to keep them slightly below the default 5GB threshold, preserving automatic merging and tombstone garbage collection. To balance search speed with indexing throughput, force-merging occurs during low-traffic periods, and shard sizes are maintained within a healthy range (below 50GB) to optimize performance without sacrificing ingestion speed.

More vectors, faster searches, hybrid search and happy users

With these optimizations, Cypris brought query times down from 30–60 seconds to 5–10 seconds. They are also seeing 60–70% of their user queries shift from the previous boolean search experience to the new semantic search interface.

But the team is not stopping here! The goal is to achieve sub-second queries to support fast, iterative search and get most of their users to shift to semantic search.

What did the Cypris team learn? … and what’s next?

500 million vectors don’t scale themselves

Handling 500 million vectors isn’t just a storage problem or a search problem—it’s both. Cypris had to balance search relevance, hardware resources, and indexing performance at every step.

Did you know

Elasticsearch's _search API includes a profile feature that allows you to analyze the execution time of search queries. This can help identify bottlenecks and optimize query performance. By enabling profiling, you can gain insights into how different components of your query are processed. Learn more about using the profile feature in the Elasticsearch search profiling documentation.

With search, there’s always a trade-off

BBQ was a major win, but it didn’t eliminate the need to rethink sharding, memory allocation, and indexing strategy. Reducing the number of segments improved search speed, but made indexing slower. Excluding vectors from the source reduced disk space but complicated reindexing, as Elasticsearch doesn’t retain the original vector data needed to efficiently recreate the index. Every optimization came with a cost that had to be carefully weighed.

Prioritize your users, not the model

Cypris didn’t chase the largest models or highest dimension vectors. They focused on what made sense for their users, and working backwards. “Figure out what relevance means for your data,” Logan advises. “And work backward from there.”

Cypris is now expanding to other datasets, which could double the number of documents they have to index in Elastic. They need to move quickly to stay competitive, “We’re a small team,” Logan says. “So everything we do has to scale—and it has to work.”

To learn more, visit cypris.ai