What you'll discover

In this post, you'll learn how to:

• Build judgment lists from click data to evaluate and improve relevance.

• Apply basic search tuning (field weights, boosts, query rules) informed by analytics.

• Create rank features from behavioral signals, like popularity and conversion rate.

• Understand LTR and how click data becomes training data.

• Close the feedback loop between analytics and relevance improvement.

What you'll need

• Search analytics data from the previous blogs in this series (search, click, and conversion spans in Elastic).

• An Elasticsearch index with product data (the reference project includes sample data with rank features).

• Familiarity with Elasticsearch queries (BM25 [Elasticsearch's default text scoring algorithm], rank_feature, function scores).

Search relevance techniques at a glance

Technique

Effort

What it improves

When to use

Field weight tuning

Low

Relevance for attribute-rich queries

First step; quick wins

Query rules

Low–medium

Specific high-value queries

Known bad results for specific terms

Rank features

Medium

Blending behavioral signals with text score

Popularity, conversion, freshness boosts

LTR

High

Systematic ranking from click data

When you have ≥5k labeled query-doc pairs

From search analytics to search relevance improvements

Over the past three posts, you built a full instrumentation pipeline, including search spans with search.* attributes and click tracking with position data and click-through rate (CTR) / Mean Reciprocal Rank (MRR) metrics. This pipeline also includes conversion spans tying searches to revenue. And it all sits in traces-generic.otel-default, queryable with Elasticsearch Query Language (ES|QL).

Now you have dashboards, and you know that your CTR is 28%. You also know which queries generate revenue and which ones users abandon. Plus, you can tell your product manager exactly where the funnel leaks.

Now what?

The real value of search analytics is using behavioral data to improve relevance and close feedback loops. It’s also important to make search learn from its users. Your experience tells you that measurement is necessary but not sufficient, and a dashboard that shows poor ranking doesn't improve that ranking. 

This post covers four practical areas:

1. Judgment lists: The foundation for evaluating and improving relevance.

2. Basic search tuning: Field weights, boosts, and query rules informed by analytics.

3. Rank features for personalization: Feeding behavioral signals back into ranking.

4. LTR: Training machine learning (ML) models on click data to optimize ranking automatically.

Each one draws directly from the search.* attributes you're already collecting, and they don’t require any new instrumentation.

What are judgment lists?

Before diving into specific techniques, it's worth understanding the concept that ties them all together: judgment lists.

A judgment list is a set of query-document pairs with relevance grades: For a given query, how relevant is each document? They look like this:

Query

Document

Grade

Label

"wireless headphones"

SKU-001 (Sony WH-1000XM5)

3

Highly relevant

"wireless headphones"

SKU-042 (AirPods Max)

2

Relevant

"wireless headphones"

SKU-099 (Wired earbuds)

0

Not relevant

Judgment lists serve three purposes:

• Evaluating current relevance. Given a set of queries and known-good results, how well does your search rank them? Metrics like Normalized Discounted Cumulative Gain (NDCG) and the Rank Eval API use judgment lists to score your ranking quality. This gives you a baseline before making changes.

• Measuring the impact of changes. When you adjust field weights or add synonyms, judgment lists tell you whether the change helped or hurt. The same is true when you change boosting rules. Run the same evaluation before and after: If NDCG went up, the change improved relevance for those queries.

• Training LTR models. LTR algorithms need labeled training data, such as, For this query, these documents are relevant and those aren't. Judgment lists are that training data.

Manual vs. automated judgment lists

Traditionally, judgment lists are created by human assessors who manually rate documents for a set of test queries. This works well for a small number of high-value queries (your top 50 searches, for example), but it doesn't scale. A large catalog with thousands of distinct queries and frequent inventory changes makes manual assessment impractical.

This is where your click data becomes valuable. Every click is an implicit relevance judgment; that is, a signal that for a given query, a given document was relevant enough to engage with. Conversions are even stronger signals. By aggregating this data, you can build judgment lists automatically from real user behavior, at a scale that manual assessment can't match.

The trade-off is noise. Clicks are influenced by position bias (users click higher-ranked results more often regardless of relevance) and presentation effects. They’re also influenced by accidental clicks. We'll cover techniques to handle this noise later in this post. But the key insight is that approaches like personalization and LTR lean toward automated judgment lists because it's not scalable to manually create lists for every query or user segment, nor for every inventory change.

The judgment lists guide on Search Labs covers the concept in depth, including how to structure lists for evaluation with the Rank Eval API.

How search analytics identify search relevance problems

The most immediate use of your analytics data is to identify and fix specific relevance problems. This only requires using data to direct manual improvements, no personalization or ML.

Finding problem queries with search analytics

The ES|QL queries from the click in your application give you per-query CTR and MRR. Sort by search volume descending and CTR ascending to find your highest-impact relevance failures:

This is the CTR-by-query query re-sorted to surface problems first. The WHERE searches > 5 filter removes one-off queries that would dominate the low-CTR list with small sample noise.

• Zero-CTR queries with results.  Your ranking is returning content, but none of it’s compelling. These queries often benefit from synonym expansion. You can also improve them with boosting rules or pinned results.

• Low-CTR, high-volume queries. These are the biggest relevance investment opportunities, and they affect the most users.

• Low-MRR, high-CTR queries. Users find what they need but have to scroll for it. In these instances, the relevant documents exist, but they're ranked wrong.

Try it: Close the loop in the reference project

If you have click data in the reference project(python generate_traffic.py --blog 3 --sessions 100), you can run the problem-query query above against traces-generic.otel-default and observe your lowest-CTR queries.

The reference project's app.py already uses rank_feature boosting: rank_features.popularity, rank_features.conversion_rate, rank_features.margin_score, and rank_features.freshness are indexed on every product and blended into the BM25 score at query time. To see the effect of adjusting a boost:

1. Open reference/app.py, and find the "should" clause in _build_search_query().

2. Change the "boost" value on rank_features.popularity from 2 to 5.

3. Restart the server (python app.py) and rerun generate_traffic.py --blog 3 --sessions 50.

4. Compare the top-queries and click position distribution in ES|QL before and after.

This is informed manual tuning using the behavioral data you collected, not ML. That's the pattern for the rest of this post: First, ES|QL tells you what's wrong. Then, configuration changes and, eventually, LTR models fix it.

Field weights, boosts, and scoring for search relevance

The simplest tuning lever in Elasticsearch is adjusting how different fields contribute to the relevance score. A multi_match query across title, description, and brand fields can weight the title higher because a match there is usually more relevant. Your analytics data tells you where these weights are wrong: If users consistently click products with titles that don't match the query but with descriptions that do, your title boost may be too aggressive.

Beyond field weights, you can incorporate business metrics directly into scoring. A common example is boosting by profit margin; that is, products with higher margins rank slightly higher when relevance scores are similar. The rank_feature field type is designed for exactly this: Index a numeric signal (margin, popularity, recency) alongside each document, and the rank_feature query blends it with text relevance at query time. For more complex scoring combinations, like decay functions, weighted field values, and scripts, the function_score query gives you full control. The boosting by profit and popularity guide on Search Labs walks through this in detail, including the trade-offs between additive and multiplicative boosting approaches.

Query rules for targeted search relevance fixes

For targeted interventions, query rules let you pin, boost, or exclude specific results for specific queries, and no model training is required. They're the search equivalent of a manual override.

Your analytics data tells you exactly where to apply them. When you see a zero-CTR query, like "returns policy", that consistently returns product results instead of the returns page, you can pin the returns page at position 1 for that query. If you notice a high-revenue query where the best-selling product appears at position 4, you can boost it.

Query rules are valuable precisely because they're simple. They solve known problems immediately while you build toward more sophisticated approaches. The query rules tutorial on Search Labs walks through the setup.

Building judgment lists from click data

The basic tuning above handles individual problems. To improve relevance systematically, you need judgment lists, and your click data can build them automatically.

Converting click data into relevance grades

The simplest approach is to count clicks per query-document pair and assign graded relevance:

This gives you a table of (query, document, click_count, avg_position) tuples. The grading step maps click counts to relevance levels:

Click count

Suggested grade

Label

0

0

Not relevant (never clicked for this query)

1

1

Marginally relevant

2–3

2

Relevant

4+

3

Highly relevant

The thresholds depend on your traffic volume. For a high-traffic site, you might need 10+ clicks before calling something "highly relevant." For lower traffic, even two or three clicks is a meaningful signal. The point is that click frequency across users is a stronger signal than any single click.

You can strengthen judgments further by incorporating conversion data, a document that gets clicked and added to cart is a stronger relevance signal than one that gets clicked and abandoned:

A document with five clicks and three cart additions is a stronger candidate for grade 3 than one with five clicks and zero cart additions.

Handling position bias in click data

There's a problem with raw click counts: position bias. Users click position 1 more often because they see it first, not necessarily because it's the most relevant result. Blog 3 introduced this concept when we discussed click position distribution and referenced the foundational work by Joachims et al. (2005).

Position bias matters for judgment lists because it means raw click data over-weights whatever the current ranking happens to surface first. If you train an LTR model on biased judgments, it learns to replicate the existing ranking, which defeats the purpose.

Two practical approaches to handle this:

• Position-normalized click rates. Instead of raw click counts, calculate click-through rate per position. A document clicked 3 out of 10 times when shown at position 5 is arguably more relevant than one clicked 5 out of 10 times at position 1. Position 5 gets less visibility, so a higher click rate there is a stronger relevance signal.

• Skip-above heuristics. When a user clicks position 3 but skips positions 1 and 2, those skipped documents are implicitly judged "less relevant" for that query. This is the core insight from Joachims et al.: Skipped-above results provide negative training signals. You can extract these pairs from your click data:

Queries where the minimum click position is greater than 1 are sessions where the user skipped the top result(s). The documents at those skipped positions, for those queries, are candidates for grade 0 in your judgment list, because the user saw them and chose something lower.

For production judgment list generation, the LTR tutorial on Search Labs walks through the complete pipeline. The training LTR models with user behavior data guide covers the specific workflow of deriving training data from click-through behavior, including the Clicks Over Expected Clicks (COEC) algorithm for debiasing. And the LTR Jupyter notebooks provide working Python code for feature extraction and model training.

Rank features for search relevance personalization

Basic tuning applies the same ranking to every user. Personalization means adapting results based on who’s searching (their history, preferences, or segment). Rank features are the most accessible way to do this in Elasticsearch.

Building behavioral signals from click streams

You can aggregate your click and conversion data at the document level to produce signals that reflect overall or segment-specific popularity:

These document-level signals (click popularity, cart rate, and conversion rate) are indexed as rank_feature fields on each product document. The workflow:

1. Run the ES|QL query above periodically (daily or weekly).

2. Write the results back to a click_popularity or conversion_rate rank_feature field on each product document.

3. Use a rank_feature query to blend text relevance with behavioral signals.

The rank_feature query applies a saturation function by default: Initial popularity gains matter most, diminishing as values get large. This prevents a single viral product from dominating all queries. You can tune the function's pivot point to control how much influence the feature has relative to text relevance.

Personalizing rank features by user segment

The query above produces global popularity; that is, it’s the same for every user. Personalization comes from segmenting these signals. If you're tracking enduser.pseudo.id or user.id, you can compute features per user cohort:

• Category affinity: How often does this user click products in "electronics" versus "clothing"?

• Price sensitivity: Does this user tend to click and convert on higher- or lower-priced items?

• Brand preference: Which brands does this user engage with most?

These become additional rank features, applied at query time based on who's searching. The personalized search with LTR guide on Search Labs walks through training per-user ranking models. For a lighter approach, the cohort-aware ranking guide shows how to use multiplicative boosting to personalize at the segment level without ML, just analytics-derived weights applied to rank features at query time.

Combining multiple signals with the linear retriever

When you're blending text relevance with semantic search and behavioral rank features, you need a way to combine them. Elasticsearch's linear retriever gives you precise control over how different query types contribute to the final ranking. It computes a weighted sum of normalized scores, so you can say text relevance matters 60%, semantic similarity 30%, popularity 10% and adjust those weights based on your analytics. This is unlike Reciprocal Rank Fusion (RRF), which only considers relative rank positions.

This is particularly useful for personalization because you can vary the weights per user segment. A returning customer might get more weight on purchase history, while a first-time visitor gets more weight on global popularity.

If you're also using semantic search with embedding models, the Elastic Inference Service (EIS) now offers GPU-accelerated embedding generation, including multilingual models, directly within Elastic Cloud, making it straightforward to add a semantic retriever alongside your text and behavioral signals.

How Learn To Rank uses click streams to optimize ranking

Rank features handle individual signals, but LTR handles all of them at once. It trains an ML model to combine features (like text relevance, popularity, CTR, recency, margin, or user affinity) into a single ranking function optimized for your users.

What you need for Learn To Rank in Elasticsearch

Elasticsearch has supported native Learning to Rank since version 8.12 as an Enterprise subscription feature. The pipeline:

1. Judgment list. Query-document pairs with relevance grades (built from click data, as above).

2. Feature extraction. Numeric signals for each query-document pair (BM25 score, popularity, CTR, recency, price, margin, user segment features).

3. Model training. Typically XGBoost or LambdaMART, trained on your judgment list with features.

4. Deployment. Upload the trained model to Elasticsearch via Eland, and use it as a rescorer.

The click and conversion data from this series feeds steps 1 and 2, and the judgment list is your training labels. The document-level features from the rank features section above (click popularity, conversion rate, margin) are additional features alongside text relevance scores.

Why automated judgment lists from click data scale better

This is where the scalability argument for automated judgment lists becomes concrete. A manually curated judgment list might cover your top 100 queries well, but personalized ranking needs judgment data across thousands of queries and multiple user segments. You can't hire assessors to rate results for "wireless headphones" separately for electronics enthusiasts, budget shoppers, and professional audio engineers.

Automated judgment lists from click data scale to every query your users actually run and update as inventory and user behavior change. They can also be segmented by cohort. The training LTR models with user behavior data guide demonstrates this end-to-end workflow, showing how to go from raw click events to trained ranking models.

For teams looking to close this loop even further, the agentic autotuning approach demonstrates using an AI agent to continuously monitor search quality and generate judgment lists from user interactions. It automatically retrains LTR models, turning the feedback loop into an autonomous system.

What Learn To Rank requires: Traffic, pipeline, and evaluation

LTR requires enough traffic to generate meaningful judgment lists and engineering time to build and maintain the pipeline. It also requires ongoing evaluation to ensure that the model improves over time. But for search applications with sufficient volume, it's the most effective way to make ranking learn from user behavior. The LTR introduction on Search Labs covers the full scope of what's involved.

Elasticsearch Relevance Studio: Visual search relevance tuning

Between query rules (manual, targeted) and LTR (ML, systemic), there's a middle ground: visual relevance tuning. Elasticsearch Relevance Studio is a tool for comparing and tuning search configurations side by side. It lets you adjust boost values, field weights, and query structures while seeing the results update in real time.

Analytics data, especially CTR and MRR, tells you which queries to focus on. Start from a ranked list of problem queries (the low-CTR, high-volume queries from the ES|QL analysis above), and work through them systematically in Relevance Studio, instead of guessing which searches need tuning.

The workflow:

1. Identify problem queries. Run the CTR-by-query and MRR-by-query analyses.

2. Open those queries in Relevance Studio. See current results alongside the tuned version.

3. Adjust field weights and boosting. Experiment with configuration changes.

4. Evaluate with judgment lists. Use the Rank Eval API to confirm that the change improves NDCG for your test queries.

5. Measure the impact in production. Rerun the analytics after deploying changes, and compare CTR / MRR.

This before-and-after loop is where analytics and tuning connect. With the data analytics you can prioritize the queries that affect the most users and verify that changes actually helped. Without this data, tuning is guesswork, and you're adjusting weights without knowing which queries matter or how to measure improvement. 

If you are looking at tuning and evaluating relevance you should also investigate the Elasticsearch Relevance Studio project which lets you directly compare search strategies. 

Closing the loop between search analytics and search relevance

Search relevance improvement follows a consistent loop: measure quality with ES|QL analytics, apply changes (query rules, field weights, rank features or LTR models), then measure again.

The key insight is that the same instrumentation that measures search quality also generates the data to improve it. Your search spans produce the queries to analyze, and your click spans produce the judgment data for evaluation and LTR training. Plus, your conversion spans tell you which improvements matter most to the business.

Measuring search relevance improvements after changes

After deploying a change, like a new query rule or updated field weights, or deploying an LTR model, you need to know whether it helped:

Metric

What it measures

How to compare

CTR trend

Whether more users are clicking results after the change

Run the CTR-by-query ES|QL query for the week before and after; compare per-query percentages

MRR trend

Whether clicks are shifting toward higher-ranked positions

Compare mean reciprocal rank per query before and after using the MRR-by-query analysis

Conversion rate trend

Whether more clicks are turning into purchases or cart additions

Run the click-to-conversion ES|QL query for both periods; compare cart rate percentages

Revenue per query

Whether the business impact of the change is positive

Compare revenue attributed to affected queries via conversion spans before and after deployment

Run the same ES|QL queries before and after. If your change was a query rule for "laptop bag", compare that query's CTR and MRR from the week before to the week after.

A/B testing with experiment attributes

The feature_flag.key attribute  is designed for exactly this. Route a percentage of traffic to a new ranking configuration, and set feature_flag.key to the experiment name and optionally feature_flag.result.variant to the variant (for example, "control" or "variant-boost-v2"). Then propagate it to click spans the same way you propagate `query_id`, and compare metrics between groups:

This gives you an A/B comparison of CTR by experiment variant. You don’t need a separate experimentation platform for basic comparisons, although you’ll want a proper framework for statistical rigor on sample sizes and significance.

Monitoring for search relevance regressions

Once you've identified your high-value queries, that is, the ones that drive revenue and have been tuned for good engagement, you need to protect them. Relevance regressions on your top 20 revenue-generating queries are business-critical incidents.

This is where monitoring comes in. In the next blog in the series, we'll cover setting up alerts on these metrics: CTR drops on high-revenue queries and MRR regressions after deployments. We’ll also cover conversion rate anomalies. The same ES|QL queries that power your analytics dashboards can drive alerting rules; the feedback loop covers measurement and improvement. It also provides operational protection.

Search relevance improvement roadmap: Week 1 to Quarter 1

Here's a practical starting point. You don't need to implement LTR on day one. The approaches build on each other:

• Week 1: Evaluate and fix known problems. Run the CTR-by-query analysis. Find your zero-CTR queries with high search volume, and fix the worst ones with synonyms or pinned results via query rules. You could also use adjusted field weights. Use judgment lists (even a small manual one for your top queries) to confirm that the changes improve NDCG before deploying. You should get immediate, targeted impact.

• Month 1: Add behavioral rank features. Extract document-level click popularity and conversion rates from your analytics. Index them as rank_feature fields, and blend behavioral signals with text relevance using the linear retriever or `rank_feature` queries. Then consider simple business boosts like margin. This lifts baseline quality across all queries without manual intervention per query.

• Quarter 1: Automate with LTR. Once you have enough click data (typically several weeks of production traffic), build judgment lists automatically from click and conversion data. Train an LTR model that combines text features, behavioral features, and business features, and then deploy it as a rescorer. The ranking now learns from user behavior and improves as you collect more data.

At each stage, measure the impact with the same metrics. CTR and MRR are your scorecards, as is conversion rate. If a change doesn't move them, it didn't help, regardless of how sophisticated the approach.

What's next: Search reliability engineering and SLOs

We've covered the full sequence: instrument search, measure quality, track conversions, and improve relevance. The missing piece is making sure it all keeps working.

Next, we close the series with search reliability engineering, including Service Level Objectives (SLOs) for search quality and alerting on metric regressions, along with operational dashboards that catch problems before users notice them. The same metrics you've been building become the basis for search health monitoring, turning your analytics pipeline into a reliability system.

Get started

Working code

• Reference project: Working code for the entire blog series; clone, configure, and run.

What you'll discover

In this post, you'll learn how to:

• Add conversion tracking (add-to-cart and purchase spans) with search.* attributes that tie back to the originating search.

• Build a full search-to-revenue funnel: search → click → add-to-cart → purchase.

• Write Elasticsearch Query Language (ES|QL) queries to calculate conversion rates, revenue per query, and average order value.

• Identify where users drop off and which team should own each drop-off point.

• Attribute revenue to specific search queries for prioritizing relevance work.

What you'll need

• Click tracking from Blog 3 (search + click spans flowing to Elastic via OTel-native ingestion).

• Backend endpoints for add-to-cart and checkout events (example code provided).

• Basic understanding of ecommerce conversion funnels.

Why search revenue attribution matters

Your product manager walks into a meeting and asks, "Which searches are driving the most revenue?"

You can tell them what users search for (Blog 2) and what they click (Blog 3). But you can't tell them what they buy. The gap between "clicked a result" and "purchased a product" is where the business case for search investment lives, and right now it's invisible.

This post closes the loop. By adding two more span types (add-to-cart and purchase), you get a full funnel from search to revenue, built on the same search.* attributes and ES|QL queries you've been using since Blog 2.

What search conversion tracking answers for your team

Here's what conversion tracking lets you answer and why each question matters to different people on your team.

• Which searches drive revenue? This is the product manager's question. When you can attribute dollars to specific queries, you can prioritize relevance work by business impact. A query with mediocre click-through rate (CTR) but high conversion value is more important than a high-CTR query that never leads to a purchase.

• Where do users drop off? The funnel from search to purchase has four stages: search, click, add-to-cart, and purchase. Each drop-off points to a different problem. High click-to-cart drop-off suggests that product pages aren't convincing. High cart-to-purchase drop-off is checkout friction rather than a search problem. Knowing where users abandon tells you which team should fix it.

• Which queries to protect? Once you know that "laptop bag" generates $12,000/month in attributed revenue, you treat it differently. Any relevance change that touches high-revenue queries gets extra scrutiny. You can set up monitoring (Blog 6, coming soon) to alert when conversion rates drop for your top-earning searches.

Two new instrumentation points total about 20 lines of code, and they follow the same pattern as you’ve used before. You add attributes to spans, and query them with ES|QL.

Following along with code? The reference project has conversion tracking ready to enable. Uncomment the Blog 4 sections in app.py and app.js, restart, and then generate traffic with python generate_traffic.py --blog 4.

The four-stage search conversion funnel

Before we write any code, here's the shape of what we're building. Each stage is an instrumentation point, and each creates spans in traces-generic.otel-default:

The thread running through the entire chain is search.query_id. The same identifier you derived from the trace ID in Blog 2 and used to link clicks to searches in Blog 3 now carries through to add-to-cart and purchase events. This is what makes revenue attribution possible: You can trace a purchase back to the search that started the journey.

Add conversion tracking spans with OpenTelemetry

Add-to-cart span instrumentation

When a user adds a product to their cart from a search results page (or from a product detail page they reached via search), you create a cart.add span. This captures the moment that intent turns into action.

This follows the same pattern as click tracking in Blog 3; that is, an independent span linked to the originating search via query_id. The new attributes are cart.quantity and cart.price which let you aggregate revenue at query time.

Purchase span instrumentation

When the user completes checkout, you create a checkout.complete span. This is the revenue event, the one that answers the product manager's question.

• Spans capture errors automatically. This is a side benefit of using OTel spans for conversion events. If an add-to-cart or checkout call throws an unhandled exception, the span's status is automatically set to ERROR and the exception details are recorded. These errors are business-critical (a broken checkout flow means lost revenue), and they show up immediately in Elastic APM's error tracking, service maps, and alerting. You get conversion analytics and operational monitoring from the same instrumentation, with no extra code.

• checkout.total_amount is the revenue number. This is what you'll aggregate in ES|QL to get revenue-by-query. It represents the order total, not the price of a single item.

• search.query_id is conditional. Not every purchase originates from search. Users browse categories and follow promotional links. Then they return to their cart days later. The if event.query_id: guard ensures that you only attribute purchases to search when there's a genuine connection. Purchases without a query_id still get recorded; they just don't appear in search attribution queries.

• search.query is set on both cart and purchase spans. This is a deliberate denormalization. You could join back to searches via query_id to get the query text, but while ES|QL does support LOOKUP JOIN, it’s likely not the right choice here due to the need to optimize the lookup index. By putting the query text directly on conversion spans, your revenue-by-query and cart-by-query queries are single, straightforward aggregations.

Span attributes for cart and purchase events

Add-to-cart attributes:

Attribute

Type

Required

Purpose

search.action

string

yes

"add_to_cart"

search.result_click_id

string

yes

Product document ID

search.result_click_position

int

yes

Position in results when added

search.query_id

string

yes

Links to originating search

enduser.pseudo.id

string

yes

Client/device identifier (OTel Semantic Conventions  [SemConv])

cart.quantity

int

yes

Quantity added

cart.price

float

optional

Product price

search.query

string

recommended

The search query text (for cart-by-query analysis)

Purchase attributes:

Attribute

Type

Required

Purpose

search.action

string

yes

"purchase"

checkout.order_id

string

yes

Unique order identifier

checkout.total_amount

float

yes

Order total

checkout.item_count

int

yes

Number of items purchased

enduser.pseudo.id

string

yes

Client/device identifier (OTel SemConv)

search.query_id

string

recommended

Links to originating search (last search in journey)

search.query

string

recommended

The search query text (for revenue-by-query analysis)

These follow the same search.* namespace from Blogs 2 and 3, with new cart.* and checkout.* prefixes for conversion-specific data. With OTel-native ingestion, all attributes are stored under attributes.* with dot notation preserved. That means no more mapping strings to labels.* and numbers to numeric_labels.*. A search.query attribute is queryable as attributes.search.query. Likewise, checkout.total_amount is queryable as attributes.checkout.total_amount.

User identity: connecting search to purchase across sessions

You'll notice that enduser.pseudo.id appears on every span type in this series. It's the minimum identity level; that is, a persistent identifier stored in the browser's localStorage that ties events to a device across sessions.

For conversion tracking, identity becomes more important. You need to connect a search on Monday to a purchase on Tuesday, or correlate cart additions across tabs. Our schema supports three identity levels, aligned with OTel semantic conventions:

Attribute

Persistence

Purpose

enduser.pseudo.id

Permanent (localStorage)

Device/browser identifier (OTel SemConv)

session.id

Per-visit (sessionStorage)

Groups events within a single visit (OTel SemConv)

user.id

Account (auth system)

Authenticated user (OTel SemConv)

For the funnel queries in this post, enduser.pseudo.id is sufficient. It links the journey from search to purchase within a browser. If your users authenticate, adding user.id enables cross-device attribution (searched on mobile, purchased on desktop) and richer personalization. session.id helps disambiguate when the same client has multiple active sessions.

All three are optional on interaction spans. Start with enduser.pseudo.id, and add the others when your use case requires them. The important thing is consistency: Use the same identifiers across search, click, and conversion spans so the joins work.

Frontend integration

The front end needs to propagate query_id through the user journey. When the user clicks a search result, you already have query_id from the search response (Blog 3). The key is carrying it forward.

The front end sends client_id as an HTTP field name; and the back end maps it to the OTel semantic convention enduser.pseudo.id when setting span attributes. The query_id propagation is the critical piece. Store it alongside the product in the cart data structure so it survives navigation between pages. We'll discuss the design challenges of this in the attribution section below.

Using the reference project? The reference app's frontend/app.js wires up add-to-cart buttons, but the checkout flow isn’t implemented in the browser UI. It goes through the traffic generator. To simulate conversion events, run: python generate_traffic.py --blog 4 --sessions 100. This sends a realistic mix of searches, clicks, cart additions, and purchases to all three backend endpoints.

Verify that conversion events are arriving

Before building funnel queries, confirm that both span types are flowing to Elastic:

Add-to-cart events:

Purchase events:

If these return rows, you have the full funnel instrumented. If not, check the same things as always: OpenTelemetry Protocol (OTLP) endpoint, auth token, and span export.

Funnel analysis with ES|QL

Count search, click, cart and purchase events in a single query

You can count all four funnel stages in a single ES|QL query using the same COUNT(CASE(...))pattern from the CTR query in Blog 3:

Example output:

searches

clicked

carts

purchases

click_rate

cart_rate

purchase_rate

146

41

28

12

28.1%

19.2%

8.2%

You get all four counts and three conversion rates from a single query. The WHERE clause pulls all four span types into one result set, and COUNT(CASE(...)) counts each type separately. This is the same technique that made the CTR query in Blog 3 so clean.

Notice that we use search.first_clickfor the click stage rather than counting all click events. This gives you the number of searches that received at least one click (the same definition used for CTR in Blog 3). Without this, a search with three clicks would inflate the click count to three while only counting as one search, making the funnel numbers misleading. Each stage now represents a unique progression: how many searches happened, how many of those got clicked, how many led to a cart addition, and how many resulted in a purchase.

You can turn this into a funnel visualization using Kibana Lens. Run each stage count as a separate ES|QL query, save them as dashboard panels, and arrange them as a horizontal bar chart with the four stages on the y-axis and counts on the x-axis. The drop-off at each step becomes immediately visible.

The dashboard above (from the first blog in the series) shows this in practice: The Conversion Funnel panel in the lower left uses a horizontal bar chart to visualize the four stages, and the Top Queries by Revenue table alongside it shows revenue attribution. You can build these directly from the ES|QL queries in this post.

What drop-off rates tell you and who owns each bottleneck

Each transition in the funnel tells you something specific:

• Search to click (CTR): You measured this in Blog 3. Low CTR means that results aren't compelling. This is a relevance problem.

• Click to cart: The user engaged with a result but didn't add it to their cart. This could mean that the product page isn't persuasive or the price isn't competitive. It could also mean that the item was out of stock. It's often not a search problem. It’s possible that the search worked (the user clicked), but something downstream lost them.

• Cart to purchase: The user committed to buying but didn't complete checkout. Complicated forms, unexpected shipping costs, and payment issues cause this checkout friction. This is almost never a search problem, but it's useful to know where the funnel leaks so you don't waste time optimizing relevance when checkout is the bottleneck.

The diagnostic pattern is straightforward:

Drop-off point

Likely cause

Who owns it

Search to click

Relevance / ranking

Search team

Click to cart

Product page / pricing / availability

Product / merchandising

Cart to purchase

Checkout UX / payment / shipping

Checkout / growth

This is one of the most valuable things that full-funnel data gives you: the ability to point at the right problem. When the VP asks, "Why aren't searches converting?", you can show whether the bottleneck is relevance, product pages, or checkout.

Revenue attribution by search query

Here's the query your product manager actually wants:

This gives you a ranked list of queries by the revenue they generated. The search.query attribute on purchase spans (the denormalization we discussed earlier) makes this a single aggregation query, without joins or subqueries.

How to act on search revenue  data

• Protect high-revenue queries. If "laptop bag" generates the most revenue, any relevance change that affects that query gets extra scrutiny. You might add it to a regression test suite or pin specific results with query rules. You might even set up an alert when its conversion rate drops (Blog 6).

• Prioritize relevance investment. The queries at the top of this list are where relevance improvements have the most business impact. A 10% CTR improvement on a query that generates $500/month in revenue is worth more than a 50% improvement on one that generates $20.

• Identify missed opportunities. Cross-reference with the top-queries data from Blog 2. A query with high search volume but no purchase attribution is either a browsing query (informational intent) or a conversion gap worth investigating.

Top revenue queries: where to focus relevance investment

To focus your relevance team's efforts, pull the top revenue-generating queries:

Cross-reference this with the per-query CTR from Blog 3. A query with high revenue but low CTR is underperforming; even small relevance improvements have outsized business impact. A query with high CTR but no purchase attribution might be informational (such as users browsing but not buying). The queries at the top of both lists deserve the most attention from your relevance team.

Average order value by search query

Average order value (AOV) tells you how much purchases are worth, in addition to which queries convert to those purchases. Queries with high AOV are your premium-intent searches; ranking improvements there have the biggest per-purchase impact:

A query with high AOV but low volume is a different opportunity than high volume + low AOV. The first means premium intent from a small audience (consider featured results or dedicated landing pages); the second means broad reach with budget buyers (price sensitivity may be limiting conversion more than relevance).

Search revenue attribution: limitations and workarounds

Revenue attribution from search is valuable, but it's imperfect. Understanding the limitations helps you set appropriate expectations and design around them.

Last-touch attribution in multi-search journeys

Users rarely search once and buy. A typical journey might look like:

1. Search "laptop bag": Browse results and click a few.

2. Search "laptop bag leather": Refine the search.

3. Search "laptop sleeve 15 inch": Try a different angle.

4. Add to cart from the third search's results.

5. Purchase.

With the instrumentation above, this purchase attributes to the third search, the one whose query_id was on the cart item. The first two searches contributed to the journey but get no credit.

This is last-touch attribution, and it's the simplest model that works within a single query_id linkage. It's not perfect, but it's concrete and unambiguous. The alternative, that is,tracking every query_id in a user's session and distributing credit, adds significant complexity to both instrumentation and analysis.

For most teams, last-touch is a good starting point. If you need multi-touch attribution later, the raw data is there. You can query all searches and clicks for a given client_id within a time window and reconstruct the full journey:

This reconstructs a user's full search journey in chronological order. It's useful for debugging individual sessions, even if you don't build automated multi-touch attribution.

Cross-session attribution limits with query_id

A user searches for "wireless headphones" on Monday, clicks a few results, leaves, and comes back on Wednesday to buy. The query_id from Monday's search is long gone, since it was a property of that specific search request.

This is a fundamental limitation of query_id-based attribution. It works within a session (or more precisely, within the scope where the front end retains the query_id). It doesn't work across sessions.

For cross-session attribution, you'd need a different approach, typically a user-level event store where you associate product views, cart additions, and purchases with a persistent user ID and then look back in time to find the originating search. That's a more complex analytics pipeline and is outside the scope of what we're building here.

The practical impact is that your search-attributed revenue will be an undercount. Some purchases that were genuinely influenced by search won't carry a query_id. This is fine for relative comparisons (such as, Which queries generate more revenue than others?), even if the absolute numbers are conservative.

How to persist query_id from search to checkout

A few practical decisions affect how far your query_id propagation reaches:

• Store query_id in the cart. When a user adds a product to their cart, persist the query_id alongside the item. This way, even if the user navigates away and comes back to checkout later (within the same session), the attribution survives.

• Don't overwrite query_id on re-search. If a user adds a product from search A and then searches again and adds another product from search B, each cart item should keep its own query_id. The purchase event carries the last search's query_id as a summary, but per-item attribution gives you richer data.

• Accept the limitations. Not every purchase will have search attribution. Direct navigation, category browsing, promotional links, and returning customers who go straight to their cart will all produce purchases without a query_id. That's correct behavior, not missing data.

Spans vs. log events for conversion tracking

Conversion events can emit both an OTel span and a UBI-compatible log event, the same dual-signal pattern used for click tracking in Blog 3. The log event is actually richer for conversions. It can contain the full items list with per-item query_id attribution, which doesn't map cleanly to flat span attributes.

The span gives you the simple, aggregatable view (total revenue by query), and the log gives you the detailed, per-item view (which specific products from which specific searches). For the funnel queries in this post, spans are sufficient. If you need item-level attribution analysis, the log events in logs-generic.otel-default have the detail you need, and ES|QL queries them the same way, just against a different index pattern.

What's next: turning conversion data into relevance improvements

We now have the complete instrumentation picture, with four span types: search (Blog 2), search.result.click (Blog 3), cart.add, and checkout.complete (this post). These capture the full user journey from query to purchase. Every span lives in traces-generic.otel-default, and every metric is queryable with ES|QL. The search.query_id thread ties the entire funnel together.

But measuring the funnel is only half the story. The real payoff is using this data to make search better.

In a later blog in this series, we take everything we've built and turn it into relevance improvements. Click positions and conversion data become judgment lists for Learning To Rank, and per-query CTR and revenue become rank features for boosting. Plus, high-revenue queries get protective monitoring. And tools like Elasticsearch Relevance Studio give you a visual interface for tuning the searches that matter most, using exactly the data you're now collecting.

The instrumentation you've built in Blogs 2–4 is a feedback loop, beyond analytics: Measure, improve, and measure again.

Get started with search conversion tracking

• Reference project: Working code for the entire blog series; clone, configure, and run.

• Elastic Distribution of OpenTelemetry for Python: EDOT for Python.

• OpenTelemetry with Elastic: How to send OTel data to Elastic APM.

• ES|QL documentation: Query language reference.

• UBI Standard: Reference schema for search event structure.

• Query rules: Pin, boost, or exclude results for specific queries.

What you'll discover

In this post, you'll learn how to:

• Add client-side click tracking that links clicks back to their originating search via search.query_id.

• Calculate CTR; that is, the percentage of searches that produce at least one click.

• Calculate MRR; that is, how far down the results users click on average.

• Analyze click position distribution to see the full shape of user engagement.

• Write ES|QL queries for all three metrics against traces-generic.otel-default.

• Identify which specific queries need relevance tuning.

What you'll need

• A working OTel instrumentation setup from Blog 2 (search spans with search.* attributes flowing to Elastic via OTel-native ingestion).

• A front end that can send click events (JavaScript example provided).

• Familiarity with the attributes.* field mapping from Blog 2.

Why search logs alone can't measure search quality

In the second blog of the series, we instrumented search requests and ran six ES|QL queries against the data. We can see what users search for, which queries return nothing, and how fast search is.

But there's a blind spot. A search that returns 15 results looks healthy from the server side. Every metric we have says it worked. But if nobody clicks any of those results, your ranking has a problem, and none of the queries from Blog 2 will tell you.

This is the gap between results returned and results that are useful. Search volume, zero-results rate, and latency measure the mechanics of search, but they don't measure whether search is actually helping users find what they need.

To answer that question, you need a second instrumentation point: click tracking.

If you’re following along with code, the reference project has click tracking ready to enable. Uncomment the Blog 3 sections in app.py and frontend/app.js, restart, and then generate traffic with python generate_traffic.py --blog 3.

How click data measures search relevance

Before we write any code, here's what click data lets you measure and why each metric matters:

• CTR answers the most basic engagement question: What percentage of searches result in at least one click? If your CTR is low, users are seeing results but not finding them compelling enough to engage. Establish your own baseline once you have data; CTR varies considerably across product category, query type, and industry vertical.

• MRR goes deeper: When users do click, where in the results are they clicking? An MRR of 1.0 means every user clicks the top result (a perfect ranking). An MRR of 0.5 means the average click is at position 2. Low MRR with high CTR is particularly telling. It means that users are finding what they need, but your ranking is making them work for it.

• Click position distribution shows the full shape of where users click. A healthy search engine shows most clicks at position 1 with a sharp drop-off. A flat distribution across positions 1–5 means that your ranking isn't differentiating well. Per-query distributions reveal exactly which searches need relevance tuning.

Together, these metrics move you from Did search work? (Blog 2) to Did search work well?, and they pinpoint exactly where to invest in relevance improvements. Later in the series, we'll show how to turn these metrics into concrete actions: building judgment lists for Learning To Rank (LTR), tuning relevance with tools like Elasticsearch Relevance Studio, and evaluating changes with the Rank Eval API. But first, you need the data.

All three metrics require just one new instrumentation point: about 15 lines of code.

Add click tracking

Click tracking captures what happens after the results appear. When a user clicks a search result, we create a new span with attributes describing the interaction, including which document they clicked, where it appeared in the results, and which search produced it.

Here's the code:

Let's unpack what matters.

Click spans are separate traces

This is the key architectural difference from Blog 2. Search spans are created synchronously during the API request: The user searches, the span opens, Elasticsearch responds, and the span closes. Click spans are asynchronous. The user searches, gets results, browses the page, and might click 30 seconds later (or they might never click).

That means click spans aren't children of the search span's trace. They're independent traces, linked to the originating search through search.query_id. This is the same query_id we derived from the trace ID in Blog 2, and it now serves as the join key between searches and clicks across traces-generic.otel-default.

Choosing the right OTel signal for clicks

OTel gives you three signal types, and clicks could be modeled as any of them. Each has strengths:

Signal

Index

Weight

Best for…

Spans

traces-generic.otel-default

Full trace context

Same-index queries with search spans

Logs

logs-*

Lighter weight

Log-centric pipelines, high volume

Span events

logs-generic.otel-default

Lightest instrumentation

Attaching to existing spans

• Spans land in traces-generic.otel-default alongside search spans, are fully queryable in ES|QL, appear in Kibana APM views, and carry timing information. Since our search spans are already in traces-generic.otel-default, using spans for clicks means you can query searches and clicks together in a single ES|QL statement, and no cross-index joins are needed.

• Log records are also independently queryable in ES|QL, living in logs-*. If you use the same search.* attribute names, the analytics queries are almost identical; just change the index pattern. Logs are lighter weight (no trace context overhead) and are a natural fit if your team already has a log-centric observability pipeline. One of Elastic's strengths here is that traces, logs, and metrics all land in the same platform and are all queryable with ES|QL, so choosing logs over spans doesn't mean giving up any query capability.

• Span events are lightweight at instrumentation time (attached to an existing span in the OTel API). In Elastic's OpenTelemetry Protocol (OTLP) ingestion pipeline, span events are written as separate documents to logs-* data streams (for example logs-generic.otel-default), and they’re independently queryable in the same way as logs. They’re a good, lightweight option but might require more code changes than logs, which can even pull in logs from legacy code.

In this series, we use spans because they keep searches and clicks in the same index with the simplest query path. But if you're at high volume and want to optimize for cost, or if your organization already routes OTel logs to Elasticsearch, the log-based approach works well; the search.* attribute schema is the same either way, and ES|QL queries against logs-* follow the same patterns you'll see below.

The search.first_click attribute

search.first_click is a Boolean set only on the first click for a given query_id. It exists for one reason: accurate CTR calculation without post-processing.

CTR is defined as the percentage of searches with at least one click. Without search.first_click, you'd need to deduplicate clicks by query_id at query time: grouping, counting distinct values, and subquerying. By marking the first click at instrumentation time, the ES|QL query becomes a simple count.

The set above is demo-only. It grows unbounded and breaks with multiple API replicas. The reference implementation uses a thread-safe time-to-live (TTL) dict with a 30-minute expiry window (_is_first_click() in app.py). For production with multiple replicas, use a shared external cache (Redis, Memcached) keyed by query_id with a TTL matching your session window.

Where to track first click: Front end vs. back end

The search.first_click deduplication could live in either the front end or the back end. Both are valid, and here are the trade-offs:

• Frontend tracking is simpler to implement. The browser already knows the current query and whether the user has clicked before. No server-side state is required, you don’t have to worry about multiple replicas, and it works without any backend changes. The downside is that browser state is ephemeral; a page refresh, multiple tabs, or an ad blocker can interfere with accurate tracking.

• Backend tracking (our approach) gives you a single source of truth. All click events flow through one place, so the deduplication is consistent regardless of what the client does. It also means that the analytics logic is colocated with the instrumentation code, which simplifies reasoning about data quality. The trade-off is that the back end needs to maintain state: the _clicked_queries set. For a single-instance API, this is trivial; for multiple replicas behind a load balancer, you'd use a shared TTL cache (Redis or similar).

We chose backend tracking here because we want the analytics data to be authoritative. This click data will later feed into relevance tuning and judgment lists, where accuracy matters. But if you're starting simple or running a client-side–only setup, frontend tracking is a perfectly reasonable first step. The search.first_click attribute works the same way regardless of where you set it.

Sending click events from the browser

The browser sends click events to the back end when a user clicks a result. It needs three things from the search response: the document ID, the position, and the query_id.

CLIENT_ID is generated once and stored in localStorage. It survives page reloads and gives you a stable enduser.pseudo.id without requiring a login. The back end maps client_id → enduser.pseudo.id on the span.

Positions are 1-indexed; that is, the first result is position 1, not 0.

Click tracking OTel attributes and ES|QL field mapping

Attribute

Type

Required

Purpose

search.action

string

yes

Event type: "click"

search.result_click_id

string

yes

Document ID clicked

search.result_click_position

int

yes

Position in results (1-indexed)

search.query_id

string

yes

Links to originating search

enduser.pseudo.id

string

yes

Client/device identifier

search.first_click

boolean

recommended

true if first click for this query_id

search.result_click_type

string

recommended

Object type: "product", "article"

search.query

string

recommended

The search query text (for queryability)

These follow the same search.* namespace we established in Blog 2. With OTel-native ingestion, attributes map directly to attributes.* fields in ES|QL:

OTel attribute

ES|QL field

search.action

attributes.search.action

search.result_click_id

attributes.search.result_click_id

search.result_click_position

attributes.search.result_click_position

search.query_id

attributes.search.query_id

search.first_click

attributes.search.first_click

With OTel-native ingestion, search.first_click is stored as a native Boolean; you query it with == true, not == "true", and no string coercion is needed.

Verify that clicks are arriving

Before calculating metrics, confirm that click spans are flowing to APM:

If this returns rows, you're ready for analytics. If not, check the same things as we looked at in Blog 2: OTLP endpoint, auth token, and span export.

Note: Results in this post are illustrative, generated by running python generate_traffic.py --blog 3 --sessions 50 on the reference project. Running Blog 3 traffic adds click events and additional search sessions on top of the 62 from Blog 2, so cumulative search counts will exceed 62. Your exact numbers will vary based on session count and the random nature of the traffic simulator. The metric calculations and ES|QL patterns are what to focus on.

CTR

CTR is the primary signal for search relevance, answering the question that Blog 2 couldn't: Are users engaging with the results?

CTR = searches with at least one click / total searches * 100

CTR is a binary per-search metric: Either a search got clicked or it didn't. The maximum is 100%.

Overall search CTR with ES|QL

Result: 41 clicked searches out of 146 total. CTR: 28.1%

This is a single query that pulls both search spans and first-click spans from traces-generic.otel-default. The OR in the WHERE clause brings both into one result set. COUNT(CASE(...)) counts each type separately, and EVAL does the division.

This works because of search.first_click. Without it, you'd be counting raw clicks (a user who clicks three results on one search would inflate the count). The deduplication happened at instrumentation time; the query stays simple.

CTR by search query: Finding your worst relevance failures

The overall number is useful for dashboards. The per-query breakdown is where you find problems.

This shows CTR broken down by query text. 

What CTR tells you (and what it doesn't)

• High searches + zero clicks: These are the worst relevance failures. Fix these first.

• High searches + low CTR: Results appear, but they aren't compelling. Check ranking.

• Low CTR + high zero-results rate: This is a double problem; either no results or bad results.

• CTR trend over time: This measures the impact of relevance changes.

CTR doesn't measure satisfaction. A user who clicks position 1, bounces back, and then clicks position 3 still counts as one clicked search. For a fuller picture, you need to know where they're clicking. That's what MRR measures.

CTR vs. clicks per search

CTR (what we just calculated) is capped at 100%. This is the industry-standard definition.

Clicks per search is total clicks divided by total searches. It can exceed 1.0. For example, a search where the user clicks three results scores 3.0. It measures engagement depth, which is useful but different. If you need it, count all click spans (not just first_click) divided by search spans.

MRR

MRR tells you where users click. It measures how far down the results list users go before finding something worth clicking.

MRR = average of (1 / click_position) across all clicks

The reciprocal rank transforms click positions into a 0–to–1 scale, where higher is better:

Click position

Reciprocal rank

1

1.000

2

0.500

3

0.333

5

0.200

10

0.100

Overall search MRR with ES|QL

MRR can be calculated two ways, depending on what you want to measure:

• All-click MRR: Averages the reciprocal rank of every click and reflects overall click quality, including repeated interactions.

• First-click MRR: Averages only the first click per search (using search.first_click == true). It’s more comparable to traditional information retrieval (IR) evaluation and aligns with how you computed CTR.

For consistency with CTR and alignment with judgment-list workflows in Blog 5, we prefer first-click MRR:

Result: MRR = 0.495

This is decent but shows room for improvement. An MRR of 0.495 means the average first click lands around position 2. It isn’t a crisis, but there are queries where ranking can be improved.

MRR by search query: Finding your worst-ranked results

Like CTR, the per-query breakdown is where the actionable data lives. To surface the worst-ranked queries first, sort ascending.

This reveals which queries have the worst ranking. A query with multiple clicks and low MRR means that the ranking is consistently poor for that search; that is, users find results, but they have to dig for them.

What is a good MRR score for search?

• MRR > 0.8: This ranking is solid; users usually click position 1–2.

• MRR 0.5–0.8:  This is decent, but there’s room for improvement.

• MRR < 0.5: This is a ranking problem, and users are scrolling past top results.

• MRR drop after a change: This is ranking regression that should be investigated immediately.

• Low MRR + high CTR: Users are finding things, but they have to work for it.

That last pattern is particularly interesting. High CTR with low MRR means your results are relevant (users are clicking), but your ranking isn't surfacing the best results first. It's an optimization opportunity, not a crisis.

MRR limitations: Position bias and multi-click sessions

MRR is heavily influenced by the gap between position 1 and position 2 (1.0 versus 0.5). Positions 5 and beyond barely move the average. This means that MRR is most sensitive to whether your top result is good, which is often exactly what you want to optimize.

MRR also only measures clicks, not satisfaction. With all-click MRR, a user who clicks position 1, bounces, and then clicks position 3 contributes two data points, but only the second was useful. The first-click MRR queries above avoid this by counting only the first click per search via search.first_click == true.

Click position distribution

Click position distribution shows you the full picture of where in the results users are engaging.

Results:

Position

Clicks

1

21

2

10

3

7

4

4

5

3

This is a reasonable distribution: 21 of 45 clicks (47%) land on position 1, with a tapering tail. If you paste this query into Discover's ES|QL editor, Kibana auto-generates a bar chart that makes the shape immediately visible.

How to read click position distribution for search relevance

• Sharp dropoff after position 1: The ranking is effective, and the top result is usually right.

• Flat across positions 1–5: The ranking isn't differentiating well, and all positions are equally likely to be clicked.

• Spike at position 3+ for specific queries: Those queries have ranking problems.

• No clicks beyond position 5: Users don't scroll far. Top 5 ranking matters most.

Click position distribution by search query

To see the shape for specific queries:

A query where all clicks land on position 1 has perfect ranking. A query where clicks spread across positions 1–5 needs relevance tuning.

Position bias and click models

One caveat: Click position distribution is influenced by position bias; that is, users see position 1 first, so it gets clicked more regardless of relevance. A click at position 1 isn't necessarily more relevant, just more visible.

This is a well-studied problem in information retrieval. Click models are statistical models that attempt to separate genuine relevance from position bias in click data. The foundational work by Joachims et al. (2005) showed that users are significantly biased toward higher-ranked results, and, in proposed methods like skip-above analysis (if a user clicks position 3 but skips positions 1 and 2), those skipped results are likely less relevant for that query.

For the metrics in this post, you don't need to implement a full click model. The key insight is practical: Compare distributions between queries rather than treating absolute position counts as ground truth. If query A has 80% of clicks at position 1 and query B has clicks spread across positions 1–5, query B's ranking is worse, even accounting for position bias. Later in the series, when we look at building judgment lists for LTR, position bias correction becomes more important, and the click data you're collecting here is exactly what those models need as input.

CTR, MRR, and click distribution: Reading search quality metrics together

These three metrics offer three different angles on search result quality:

Metric

What it measures

Our value

Interpretation

CTR

Do users click at all?

28.1%

Moderate: Roughly a third of searches get engagement, but there’s room to improve.

MRR

Where do they click?

0.495

Decent: The average click is around position 2, but ranking can be improved.

Distribution

What's the shape?

47% at position 1

Reasonable drop-off: The top result wins most but isn’t dominant.

Together, they tell a coherent story. For our demo data, search is performing adequately: Users are engaging with results and can find what they need, but the ranking has room to improve. The CTR of 28% and MRR of 0.495 are realistic starting points for a new search implementation without tuning.

Where they're most valuable is in combination at the query level. The queries to fix first are those with high volume + low CTR + low MRR; that is, lots of users are searching, few are clicking, and those who do click are scrolling deep. That's where relevance investment has the highest return.

Using click data for LTR and relevance tuning

These metrics don't just tell you how search is performing; they're the foundation for making it better. The click data you're now collecting feeds directly into relevance improvement workflows:

• Judgment lists for LTR: Click positions and frequencies become graded relevance labels for training machine learning (ML) ranking models. A document clicked at position 1 across many queries is a strong positive signal.

• Relevance tuning tools: Per-query CTR and MRR tell you exactly which queries to focus on in tools like Relevance Studio or the Rank Eval API, which scores your ranking against expected results.

• Query rules and boosting: Zero-CTR queries with results are candidates for pinning, boosting, or synonym rules.

We'll cover these applications in detail in Blog 5. For now, the important thing is that the instrumentation you've built here is doing double duty: It measures search quality and provides the training data to improve it.

Next in the series: Conversion tracking from search to purchase

We can now measure whether users find results (Blog 2) and whether they engage with them (this post). But a click isn't a conversion. A user who clicks a product and then abandons the page didn't get what they needed.

In the next post, we add conversion tracking, the third instrumentation point that closes the loop from search to purchase. It’s the same pattern: Add search.* attributes to add-to-cart and checkout spans, query with ES|QL, and answer the question your product manager actually cares about: Which searches drive revenue?

Get started

• Reference project: Working code for the entire blog series (clone, configure, and run).

• Elastic Distribution of OpenTelemetry Python: EDOT Python.

• OpenTelemetry with Elastic: How to send OTel data to Elastic APM.

• ES|QL documentation: Query language reference.

• UBI standard: Reference schema for search event structure.

This is the third post in a series on search analytics with OpenTelemetry and Elastic. Next up: From clicks to conversions: Conversion tracking, funnel analysis, and revenue attribution.

What’s a reranker and how does it work?

A reranker is an AI model used near the end of an information retrieval pipeline, after other modules have assembled a short list of candidate matches to a query. It’s trained to order the candidate list from best matching to least. Using a specialized model focused purely on ranking candidate matches can improve result quality dramatically.

Jina AI’s latest rerankers use a technique called late interaction, where queries and documents are encoded separately into lists of token embeddings that reflect each token’s semantics in context and then compared to each other.

This is an AI analog of lexical and grammatical disambiguation.

For example, consider the meaning of the word match in these two sentences:

• She looked for a match to light the candle.

• She looked for a match on Tinder.

The first sentence might be a match for queries about matchboxes; the second for queries about romance.

Transformer-based models do this kind of in-context disambiguation but bring much richer information into the token embeddings they produce. The word match might have a semantic embedding near to words like fire or illumination in the first sentence, while in the second, it might be closer to smartphone or swipe.

Late interaction rerankers generate these context-enriched token embeddings for both the query and the candidate documents and then compare them to produce sortable scores. They’re completely agnostic about how candidate match lists are created. The reranker works exactly the same when combined with lexical search schemes, like BM25, AI-driven semantic embeddings-based retrieval, or hybrid and federated search systems that may retrieve multiple candidate lists from different sources or using different algorithms. Of course, the results always depend on the quality of the candidates, so a reranker can’t fix bad first-stage retrieval, but it almost always improves whatever you’ve got.

jina-reranker-v3.5 is a listwise reranker using the last-but-not-late technique developed for jina-reranker-v3. The query and a list of candidate matches are passed into the model together and processed in one pass, returning a numerical score for each candidate. This enables the model to use context information from the query and the full candidate list to make sense of the entire input, producing better results because of the richer information available to it.

jina-reranker-v3 proved that listwise rerankers with last-but-not-late interaction can compete with the largest models on general reranking benchmarks. Only jina-reranker-v3.5 and models with over four billion parameters beat it on Massive Text Embedding Benchmark (MTEB) reranker tasks. However, this approach places strict limits on candidate list sizes. The query and all candidate matches must fit in the input context window of the model.

What problems does Jina Reranker v3.5 solve?

Despite having frontier-level performance overall, jina-reranker-v3 has some notable performance gaps:

Domain-specific text retrieval

jina-reranker-v3 was trained on general text corpora and, as a result, it underperforms on important use cases, particularly:

• Legal texts, like case law and contract clauses.

• Medical literature, like clinical trials and patient records.

• Financial datasets and other texts full of important numbers.

• Computer programming and IT documentation.

• Product catalogs full of technical terminology and specifications.

Structured data: Tables, JSON, and key-value records

Vast quantities of essential, real-world data is encoded in spreadsheets, tables, key-value lists, and structured records, like JSON data. However, rerankers trained purely for textual comparison, like jina-reranker-v3, perform poorly on this kind of data.

Compute costs for long candidate lists

The self-attention architecture at the core of most text-processing AI models means that memory and compute requirements grow quadratically with the size of its input. This makes jina-reranker-v3, like other AI models, very computationally expensive to run with a full input context window. But, to make the most effective use of the model, we want to put as many match candidates as possible into its input. When it’s at its most useful, it’s also slower and more expensive to run.

We’ve developed jina-reranker-v3.5 specifically to address these issues without reducing its performance on general purpose text retrieval.

What’s new in Jina Reranker v3.5?

jina-reranker-v3.5 contains a modified self-attention mechanism that enhances performance, increases processing speed, and reduces the resources required at inference time to process a full input context window. We’ve also introduced a new three-stage self-distillation training process better suited to the sliding-window architecture of large input context models.

We’ve also curated and used training data focusing on the performance gaps we identified in jina-reranker-v3, including:

• Multilingual legal texts drawn from diverse international sources.

• Medical texts drawn largely from scientific literature and materials used for other AI projects, including a collection of Chinese medical question-answer pairs.

• Financial industry data, including investment-related question-answer pairs, regulations, and tables with numbers and associated texts.

• Structured data, especially from ecommerce sources and public corpora of tables.

• Expanded multilingual and cross-language texts.

For details on the data sources and technical innovations in jina-reranker-v3.5, see our technical report.

How Jina Reranker 3.5 performs on retrieval benchmarks

Parameters

597 million

Input modalities

Text only

Context window size

131,072 tokens

Maximum number of candidate matches

No fixed limit, but all candidates and query must fit in the context window.

Languages

Training in 52 languages

General text reranking performance (BEIR and MIRACL)

jina-reranker-v3.5 improves on jina-reranker-v3’s performance on general text reranking benchmarks. On the English-language Benchmarking Information Retrieval (BEIR) benchmark, the average score has increased enough to surpass the frontier Qwen3-Reranker-4B and 0.6B models and Mixedbread AI’s rerankers.

We’ve also improved jina-reranker-v3’s multilingual reranking performance on the Multilingual Information Retrieval Across a Continuum of Languages (MIRACL) benchmark. Only the four billion parameter Qwen3 reranker regularly beats jina-reranker-v3.5’s score.

Legal, medical, and financial reranking

The Retrieval Embedding Benchmark (RTEB) suite consists of diverse domain-specific retrieval benchmarks. jina-reranker-v3.5 outperforms jina-reranker-v3 on all RTEB tasks related to law, medicine, and finance. Only the large Qwen3 reranker, at almost seven times as many parameters, has better average performance in those three domains.

The new model shows particularly strong improvements for legal data, beating jina-reranker-v3’s score by over 50% on case law retrieval tasks.

Task

Reranker v3

Reranker v3.5

Improvement v3 to v3.5

AILA-Case

20.82

32.55

+11.73 (56%)

AILA-Statute

32.15

46.16

+14.01 (44%)

LegalQuAD

81.84

83.09

+1.25 (1.5%)

LegalSum

69.64

70.99

+1.33 (1.9%)

Structured data reranking (Struct-IR and STaRK benchmarks)

We evaluated jina-reranker-v3.5's structured data reranking on two benchmarks: Struct-IR and STaRK. Both benchmarks contain AI-generated JSON text data covering a variety of applications, including product records, scientific papers, and biomedical knowledge bases.

jina-reranker-v3.5 improves substantially on jina-reranker-v3’s score on the Struct-IR benchmark, once again only exceeded by Qwen3-Reranker-4B. On the STaRK benchmark, jina-reranker-v3.5 beats all the other models we tested, of any size.

Inference speed: Latency benchmarks for short and long documents

The longer the candidate documents get, the more significant the architectural improvements we’ve brought to jina-reranker-v3.5 are. To verify this, we used two retrieval datasets distinguished by large differences in the average document length:

Dataset

Avg. doc length

jina-reranker-v3

jina-reranker-v3.5

Speedup

BEIR Natural Questions

145.5 tokens

371.1 ms

305.3 ms

22%

RTEB AILAcasedocs

1,904.0 tokens

16,064.9 ms

10,290.9 ms

56%

jina-reranker-v3.5 is significantly faster in both cases. On the Natural Questions benchmark, there’s a 22% speedup compared to jina-reranker-v3 with average request latency falling from 371.1 ms to 305.3 ms. Each query from the AILAcasedocs benchmark is much larger (more than 10 times larger on average) so it naturally takes longer to rerank on average: 16,064.9 ms for jina-reranker-v3 and 10,290.9 ms for jina-reranker-v3.5. This represents a 56% speedup for the newer model, representing less latency for applications and lower computer costs.

When should you use Jina Reranker v3.5?

Reranking improves search precision in practically every case, and jina-reranker-v3.5 has applications in a wide variety of information retrieval contexts. However, it has some limitations. The table below summarizes our best-practice advice:

Use case

Recommendation

General text retrieval in common international languages

Use `jina-reranker-v3.5`.

Legal, financial, and medical domain retrieval

Use `jina-reranker-v3.5`.

Semi-structured data, tables, product information texts for ecommerce

Use `jina-reranker-v3.5`.

Non-text or mixed-media data

Use `jina-reranker-m0`, which supports both text and image input.

How to use Jina Reranker 3.5 with the Elastic Inference API

jina-reranker-v3.5 is available via the Jina API with free tokens to try it out. It’s also available via the Elastic Inference API and Elastic Inference Service.

If you’re already using jina-reranker-v3, all you have to do is change the name of the model in the model field of your request to the Jina API or model_id field when configuring an Elastic Inference API endpoint. The two models have completely identical interfaces.

You can install jina-reranker-v3.5 as a Jina On-Prem container to get a completely self-contained server that runs on your own hardware. The model weights are also available to download for testing and research. Follow the instructions on the model’s page at Hugging Face. In both cases, the model is available under a CC BY-NC-4.0 license, so you’re free to try it out for testing, building prototypes, or doing scientific research. For commercial use, please contact Elastic sales.

AI shopping agents fail not because the model is wrong but because the agent arrives at every conversation not knowing your catalog, vocabulary, or business rules. It has to discover all of this context through tool calls, and that discovery is the cost. Precomputing a structured context layer from signals you already hold (vocabulary, policies, user profiles, session behavior) cuts the exploratory work the agent does before it can answer and makes its behavior governed and predictable.

In the analogous document-retrieval case, precomputing context reduced input tokens by up to 75% on a controlled benchmark; we expect comparable savings in ecommerce because the exploration pattern is the same, though the exact figure will vary by catalog and query mix. The signals are richer in ecommerce than in almost any other domain, and most retailers already have them. The question is whether or not these signals are assembled in a form that the agent can use before it starts reasoning. Elastic’s broad mix of search capabilities, semantic, hybrid, keyword, filtering, and aggregations make it a compelling choice for not only building your core retrieval tools but also as the vital context engine.

Why AI shopping agents fail in production

Retailers investing in AI shopping assistants are discovering an uncomfortable gap between what the demos promise and what the first few months in production deliver.

The assistant takes four seconds to respond. It confidently recommends a product in a size range that doesn't exist for that item. It tells a returning customer about a coat style they bought 18 months ago and returned. It filters by a category name that doesn't match the internal taxonomy and returns zero results. The customer gives up and abandons the chat altogether.

These aren't model failures. The frontier models powering these agents are capable of extraordinary reasoning when they have the right information in front of them. The problem is that the agent arrives at the conversation knowing nothing about the retailer's catalog, customer, or business rules that govern what should and shouldn't be recommended. It has to learn all of this through the conversation itself, making exploratory tool calls to discover what departments exist, what filter values are valid, and what the brand's policies are on certain product types. Every one of those discovery calls costs latency and tokens before the agent has said a single useful thing to the customer.

Latency matters in ecommerce in a way it doesn't in many other contexts. Shoppers expect response times measured in seconds, not the minutes that enterprise knowledge-base agents routinely take. It’s well established that slower responses reduce engagement and conversion in online retail. An AI shopping assistant that thinks visibly for five seconds before answering a question about gift ideas isn't a feature; it's friction.

The fix isn't a faster model or a bigger context window. The agent's latency and cost problem is a context problem. This can be solved by carefully computing context before the agent call, not during it.

What an AI shopping agent knows before it searches

Imagine you’re the agent. You have a search tool connected to the catalog, and this query arrives:

"an outfit for an autumn wedding"

What do you actually do with that?

Start with what you don't know. An outfit for a man or a woman? Is "autumn" a color, a season, a style of fabric, or just when the wedding happens? And who is this shopper, someone who has bought from you for years, or a stranger? Do they buy expensive designer brands or do they always hunt out a bargain on sale? You have none of these answers. So you do what an agent does when it's working blind: You ask loads of follow-up questions, guess, or fire off a series of exploratory searches to find out what departments and filter values even exist, watching the seconds tick by before you've offered the customer anything at all.

Hold onto that feeling of working blind. The rest of this article is about what changes when the agent is handed the answers first.

Why ecommerce is different from general RAG

Most of the published work on reducing agent costs focuses on document retrieval: question-answering over corpora of articles, reports, or knowledge-base entries. A recent experiment from the Elastic team (Cutting agent costs with pre-computed context) demonstrated that pre-extracting structured facts from documents before the agent call reduced input token consumption by up to 75% and improved answer accuracy from 60% to 92% on a hard factual benchmark. That improvement came in stages, with the largest jump driven by feeding the agent's own wrong answers back into the extraction step rather than by precomputing context alone, which is a distinction we'll return to when we discuss governance.

Ecommerce applies the same principle to a fundamentally different structure. A product catalog isn't a document corpus. It's a highly structured index of items with strict field semantics, a domain-specific vocabulary of brand names, color codes, and category hierarchies, and a layer of business rules that override pure relevance in specific situations.

The failure modes that result are distinct from document retrieval augmented generation (RAG) failures:

• Vocabulary mismatch: A customer asks for a "navy jumper." The agent constructs a filter against a field where the canonical value is NAVY and the category is stored as Knitwear & Jumpers. Without a vocabulary mapping, the agent either guesses colors and categories and gets it wrong, or makes multiple exploratory calls to discover what values exist before it can filter correctly.

• Hallucinated filter values: Without knowing which filter dimensions are valid for a given query, agents can construct queries against fields that don't exist or with values that return zero results. A filter like category: knitwear looks reasonable; masterCategoryNames: "Knitwear & Jumpers" is what the index actually contains. If the agent doesn’t know, it either hallucinates or has to do a separate tool call to find out, causing another LLM loop, which costs time and tokens.

• Context-free personalization: The same query from two different customers, one who typically shops in the premium range and dresses for formal occasions, and one who buys primarily casualwear under £40, should return different results. Without profile context, the agent treats every query identically, which is worse than a well-tuned keyword search because it creates the impression of a personal assistant while delivering generic answers.

The context layer: What signals it needs

The reason ecommerce is particularly well suited to precomputed context is that retailers already hold an unusually rich set of signals. The challenge isn't data availability; it's assembly.

The signals fall into two groups. Two of them, the catalog vocabulary and the business policies, are the genuinely original work and the heart of this approach. The rest, live facet state, user profiles, and session history, are valuable but closer to table stakes, signals that most teams already understand how to fetch. Here's what most mid-to-large retailers have and what each signal prevents.

Signal

What it prevents

Effort to build

Catalog vocabulary

Vocabulary mismatch and hallucinated filter values; the agent guessing at colors, categories, or brand names instead of resolving them to canonical field values

One-time engineering effort (full-catalog aggregation); incremental to maintain as new categories and brands are added

Business policies

Recommendations that ignore legal or trading requirements, for example, missing age verification on alcohol queries or routing that misses a gluten-free range

Human-authored and governed, not automated; ongoing review as new policy types are added

Live facet state

Recommending filters that return zero results or out-of-stock options for the current query

Runs in parallel with vocabulary and policy lookups; leans on existing catalog and retrieval infrastructure

User profile

Making a returning customer restate sizes, budget, or brand preferences they've already given

Fastest signal to retrieve, a single document lookup by user ID

Session and purchase history

Re-recommending an item the customer already dismissed or bought and returned

Most aspirational layer; depends on customer relationship management (CRM) and analytics integration, best added once the agent is already live

The reason we say semantic metadata and not just metadata is that we’re trying to match the semantic (meaning) of the intent rather than the exact words. If a user is searching for “teal,” we should be able to understand that this is a color and which colors exist in our products that are semantically similar to teal, even if none of them are actually teal. So, if we search for “teal,” we might want to return:

ProductColours = “aquamarine, turquoise”

Hopefully, you can see how this semantic metadata is bridging the gap between user intent and the agent’s knowledge of the products.

Catalog vocabulary: The foundation of context engineering

Catalog vocabulary is the layer that does the most work, and it's the one most worth getting right first.

A vocabulary index maps natural language to the exact field values and category paths used in the product index. It answers questions like: What does "navy" map to? Which categories fall under "knitwear"? Is "Autograph" a brand or a range? What's the correct spelling of a competitor brand the agent might encounter in a query?

What makes this more than a synonym list is how it's queried. The interesting thing a shopper says is rarely an exact field value. They say "something cozy for fall," not colour: NAVY and masterCategoryNames: "Knitwear & Jumpers". So the vocabulary index needs to resolve fuzzy, natural language intent into precise, exact-match filters, and that requires both kinds of matching at once: semantic search to understand that "cozy" leans toward knitwear and fleece, and exact keyword matching to pin the result to the canonical values the product index actually stores. A metadata index that supports both on the same documents is, in effect, a translation layer between how customers talk and how the catalog is structured.

This is also where the index earns the description "semantic metadata layer" rather than "lookup table." Each entry is a small natural language description of a facet value or schema concept, so the agent can match against meaning and then read back the exact filter to use. For a typical fashion retailer, this covers hundreds of color values, brand aliases, category synonyms, and size-range conventions. Building the semantic metadata layer from a full-catalog aggregation is a one-time engineering effort; maintaining it is incremental as new categories and brands are added. Without it, an agent encountering an unfamiliar term must either guess or make exploratory tool calls to discover what's there.

Business policies in the context layer

The second original layer is policy. Some queries carry implicit business requirements that pure relevance cannot handle. A query for "wine gift for a friend" should trigger an age-verification reminder in markets where it's legally required. A query for "gluten-free food gift" should route away from general confectionery toward the specific gluten-free range. A query mentioning "wedding guest outfit" in spring should apply different weighting than the same query in November.

These are policies, and most retail search teams already write them. They just call them boost rules, merchandising overlays, or synonym configurations. The difference in an agentic context is that instead of being applied silently as query modifications, they're surfaced as readable hints the agent can use when deciding how to frame its answer and which products to surface. The agent doesn't have to infer your trading rules from the catalog; it's handed them.

The critical point: These policies encode business intent, not just relevance. A policy that routes alcohol queries through an age-appropriate flow isn't a retrieval optimization; it's a trading requirement. That's why this layer must be human-authored and governed, not generated automatically from traffic patterns, a point we return to in the governance section.

Together, vocabulary and policy are what make the agent behave like it understands your business rather than just your data. The remaining three signals sharpen the experience, but they're more familiar engineering.

Facet, profile, and session signals in the context layer

• Live facet state: Before the agent recommends filters, it should know which filters are available and how many results each returns for this specific query. An agent that suggests "filter by size 8" without knowing that size 8 is out of stock for this query undermines the customer's trust immediately. A facet state query against the product index, run in parallel with the vocabulary and policy lookups, returns the counts, ranges, and available values specific to the current query.

• User profile: A persistent profile (sizes, color preferences, budget range, brand affinities) lets a returning customer skip restating what they've already told you. It's typically the fastest signal to retrieve, a single document lookup by user ID.

• Session and purchase history: Within a session, the agent should know what the customer has already seen, dismissed, or added to their basket, so it doesn't re-recommend a dismissed item or repeat itself. Longer-term purchase history extends this, and signals like return history are richer still, but using them well depends on data most retailers hold in systems that aren't yet wired into their search path. This is the most aspirational layer and the one best approached last, once the earlier layers are delivering value. There must also be balance when building the initial context not to overinflate the size, which will slow down the first reply and increase token costs. There is, therefore, a careful balance to strike between providing information like purchase history in the initial context or providing it as a tool for the agent to use during conversation, but users will expect that, if they’re logged in, the agent should know what they’ve purchased. The precise optimal context is likely to be specific to each implementation and customer experience and will require careful testing.

How context engineering works before the LLM call

The pattern that makes this work is simple to describe and moderately involved to implement:

Without this context build, the agent makes the same discoveries on its own, but through LLM-driven tool calls that each cost a full inference round trip. As a rough rule of thumb, an exploratory tool (for example, GetFilterValues) call tends to land somewhere in the region of 600ms to 1 second of latency in practice; an agent that discovers the vocabulary, checks policy hints, and retrieves facet state through three separate tool calls before it even begins answering can, therefore, add two to three seconds to the response time, and that's before it performs the actual product search. These are order-of-magnitude estimates, not benchmarked figures, and the real numbers depend heavily on the model, the network path, and how the tools are implemented.

Replacing those discovery calls with a parallel fetch (multiple context-building queries can be done in parallel) that runs before the LLM is invoked removes most of that cost. The context build takes roughly the same wall time as a single LLM tool call, but it replaces three or four of them. The LLM would either need to repeat failed searches or do its own context-building tool calls sequentially to get enough context to be successful. Precomputing context is also deterministic rather than subject to the model's tool-selection choices, which gives the business more control to fine-tune the experience.

The token reduction follows the same logic: Each exploratory tool call returns raw data the model must process. A preassembled context summary replaces that raw data with structured facts the model can consume in one pass. At the scale of usage possible in public-facing retail websites, this token cost saving can be significant. This work on document search (Cutting agent costs with pre-computed context) showed up to 75% input token reduction on a controlled benchmark. That benchmark was document retrieval rather than ecommerce, and its authors are explicit that the multiplier isn't a fixed number you can expect everywhere. We expect the direction to hold in ecommerce, because the exploration pattern is the same; it just runs against a structured catalog rather than a document corpus. The magnitude is something each team should measure against its own traffic.

The AI shopping agent with full context

Remember the query that left you guessing: an outfit for an autumn wedding. Run it again, but this time, before you have to think, you’re handed a precomputed context as a short brief:

• This shopper’s name is Sarah, female, age 32, and she buys womenswear, size 12.

• She typically buys your mid-tier ranges.

• "Autumn" here matches these specific color labels: “rust”, “burgundy”, “forest green”, “camel”.

• Matching departments: “Womenswear”, “Menswear”.

• Matching tags: "occasion dresses”, “trouser suits”, “wedding”.

• She already has a burgundy bag in her basket.

• House rule for wedding-guest looks: Complete the outfit. Show hats and accessories, not just the dress.

Suddenly, you’re not guessing; you’re styling for this customer. And the interesting part is how those facts combine rather than just stack. The season proposes a whole autumn palette; the burgundy bag already in her basket narrows that palette to the few tones that coordinate with it; the house rule tells you to finish the look with a matching fascinator rather than stopping at the dress. Together, these facts let you answer like someone who knows both this customer and this shop, in a single pass, with nothing invented.

That short brief is exactly what the signal stack produces: the shopper's profile, the resolved vocabulary, the live basket, and the business policy. These are assembled in parallel and placed in front of the agent before its first move, so the "What do I even do with this?" problem never has to be solved one expensive tool call at a time.

Governing the context layer without automated drift

One difference between the approach described here and automated knowledge extraction systems is worth addressing directly: In ecommerce, the context index cannot self-update without human review.

The policies that govern how an agent responds to gift queries, alcohol queries, or queries from customers in certain age brackets aren't just relevance configurations; they're trading decisions with potential legal and brand implications. An automated system that generates new policies from traffic patterns, without review, is a compliance risk before it's a technical asset.

This is actually the right constraint for most retail organizations, and it aligns with how search teams already work. Merchandisers write boost rules. Search teams maintain synonym configurations. Content teams approve what language appears in automated recommendations. The context policy layer is the same kind of governed configuration; it just serves a different consumer, namely, the agent's reasoning step rather than the query pipeline.

It's worth noting where this differs from the document-retrieval work referenced earlier. In that experiment, the biggest accuracy gain came from an automated feedback loop that fed the agent's wrong answers straight back into the extractor. That works well for factual question-answering, where "right" and "wrong" are unambiguous. In ecommerce, the equivalent signals still surface automatically, but a human decides what to do with them, because the changes carry trading and compliance weight. The loop is the same shape; the publication step has a person in it.

The governance loop that works in practice has two tiers:

• Automatic signal surfacing: Zero-result queries, repeated reformulations on the same topic, and sessions that end without a purchase after an agent interaction are all signals that something in the context layer is missing or wrong. These surface automatically as candidates for improving the experience, for example: a vocabulary term that didn't resolve or a policy that didn't fire on a query type it should have covered. To do this, you need a thorough log of conversations, including the reasoning and tool call trace in a platform like Elastic. This allows you to analyze the performance of the agent using both structured tools, for example, percentage increase in thumbs-down conversations and semantically. You could also run an automated review of conversations about "gifts" around December to characterize the thumbs-up/down ratio across an AB test of two agents.

• Human authorship and review: The search or merchandising team reviews candidates and authors the appropriate vocabulary entry or policy. Policies go through approval before publication. This typically mirrors the workflow that already exists for synonym changes or boost rule modifications; the tooling is the only new part.

How to implement context engineering in phases

• Phase 1: Vocabulary layer (highest leverage, bounded engineering task).

• Phase 2: Facet state and initial policies (leans on the same catalog and retrieval primitives).

• Phase 3: User profiles and session signals (requires CRM and analytics integration; best added when the agent is active).

• Phase 4: Governed feedback loop (shifts to organizational alignment; surfaces gaps for merchandising teams).

The full signal stack described above doesn't need to be built at once, and the order isn’t arbitrary. The highest-leverage starting point is also the lowest in implementation complexity: the vocabulary layer. A semantic metadata index built from a full-catalog aggregation (canonical color values, brand aliases, category paths, field names) is a bounded engineering task, and an agent that can resolve "navy jumper" to color: NAVY, masterCategoryNames: "Knitwear & Jumpers" before its first tool call is materially better than one that discovers this through trial and error. If you build nothing else, build this.

Facet state and the first set of policies can follow close behind, often in parallel, because they lean on the same catalog and the same retrieval primitives. The later layers, such as user profiles, session signals, and the governed feedback loop, are where the work shifts from search engineering to organizational alignment. To implement CRM system integration, merchandising workflow changes, and the analytics needed to surface gaps can take a significant amount of work. Those layers are more valuable once the agent is already in regular use and generating the traffic signals that make the governed loop worth running. The important feature is that each layer stands on its own, so a retailer gets real value from phase one without committing to phase six.

What infrastructure does a context layer need?

Precomputing context at the depth described here places specific requirements on the underlying platform. It's worth being explicit about these, because the temptation in early agent builds is to reach for the simplest available tool for each capability.

• Semantic search to match natural language queries against the vocabulary index and surface the right canonical values. Fuzzy keyword matching alone won't resolve ambiguity between similar brand names or color terms.

• A percolator to implement the policy layer. A percolator reverses the usual search direction: Instead of matching a query against stored documents, it stores the queries and matches an incoming piece of text (here, the customer's message) against them. That's exactly what policy matching needs, because each policy is essentially a saved pattern that says "When a query looks like this, surface this hint."

• Real-time aggregations over the full product catalog to produce accurate facet state at query time. Precomputed facet snapshots go stale quickly in active catalogs; query-time aggregations are the more reliable source.

• Document retrieval by key for user profiles: fast, single-document lookups by user ID that must complete within the context build window.

• Structured and semantic logging and analytics over query traces and agent interactions, which are the raw material for the governed loop's automatic signal surfacing.

These are standard capabilities of a mature search and analytics platform, rather than six separate systems, and Elasticsearch provides all of them in one place. That matters less as a procurement point than as an architectural one: When semantic matching, percolation, aggregations, profile lookups, and analytics all run against the same catalog in the same cluster, the context layer stays consistent with the search layer by construction. Splitting these across a separate vector store and a separate analytics platform is a legitimate choice, but it adds operational surface and introduces a consistency problem between two systems that are reasoning about the same products. The context infrastructure is simplest to run when it lives where the product data already lives and can be updated without a separate extract, transform, load (ETL) step.

Conclusion: Context engineering is a search team's job to own

The retailers who run effective AI shopping experiences at scale aren't the ones with the largest models or the most generous token budgets. They're the ones who have done the work to make their catalog, vocabulary, and tpolicies legible to an agent before it starts reasoning.

The good news is that most of the work is already done. The vocabulary is implicit in the catalog. The policies exist as merchandising rules and compliance guidelines. The user profiles are in the CRM. The session signals are in the analytics stream. The gap isn't data; it's the assembly layer that turns those signals into a structured context the agent can consume before it starts reasoning.

The search team already owns the vocabulary, policies, and merchandising workflows. The context layer is the right home for work the search team is already doing, in a form that serves the agent as well as the query pipeline. And, because it grows every time a gap is found and filled, it behaves less like a setup cost and more like an asset that compounds.

To begin building a context layer in Elastic, you can start a trial of Elastic Cloud or run locally. You should become familiar with configuring semantic search, and if you’re interested in how to build, store, and match search policies at query time, you will enjoy this blog.

Combining image and text into one embedding beats either alone, and the gap isn't small. In our tests, averaged image and text embeddings put the correct product in the top spot up to 1.5 times as often as image embeddings alone. We benchmarked 5,000 real apparel and footwear products in English and German across two Jina embedding models, jina-clip-v2 and jina-embeddings-v5-omni-small, to see which model and which indexing strategy actually wins for ecommerce search. The older, narrower Contrastive Language–Image Pre-training–style (CLIP-style) model beat the newer, more general one, and that wasn't what we expected. This post walks through the data, the method, and what we'd recommend doing with it.

jina-clip-v2 vs. jina-embeddings-v5-omni-small: What's different

Multimodal embedding models work by generating representative semantic vectors for inputs of different kinds in a single high-dimensional space.

We used two models that do this:

jina-clip-v2

jina-embeddings-v5-omni-small

Architecture

Dual encoder (separate text + image towers)

Single shared backbone + frozen encoders

Parameters

~865M total

~1.74B total

Embedding dimensions

1024

1024

Max input

512×512 images, 8k tokens text

32k tokens

Language coverage

Broad multilingual

~100 languages

Modality handling

Text ↔ image alignment (purpose-built)

Text, image, audio, video via projectors

jina-clip-v2 is a CLIP-style dual encoder: a text tower (Jina XLM-RoBERTa, 561M parameters) and a separate image tower (EVA02-L14, 304M parameters), about 865M parameters in total. The two towers are independently trained but fine-tuned to output to a common semantic space. It produces 1024-dimensional embeddings, handles images up to 512×512 and up to 8k tokens of text, and has broad multilingual support. It has been engineered specifically to support text-to-image, image-to-text, and text-to-text matching.

jina-embeddings-v5-omni-small has a broader scope. It extends the jina-embeddings-v5-text model to support images, audio, and video by attaching frozen vision and audio encoders to the frozen text backbone. The encoders connect through cross-modal projectors, small trained layers that map each encoder's output into the text model's embedding space. These projectors are the only part of the model to receive additional training. The resulting model produces 1024-dimensional embeddings, supports a 32k-token input context, and covers roughly 100 languages. It encodes queries and documents asymmetrically: a query with the retrieval.query task, a document with retrieval.passage.

There’s an important functional difference between the two models: jina-clip-v2 is really two separate models trained to work together, but jina-embeddings-v5-omni-small uses a single shared backbone that produces embeddings for all its supported media types. Every modality maps into one shared vector space. In principle, it can handle text, images, audio, or video, or combine materials of different media types into one input, yielding one embedding that encompasses all the data. However, there are two important caveats when using jina-embeddings-v5-omni-small: Combining image and text into one input is a documented weak spot for the model, and the Jina API only allows users to embed one modality per request. Theoretically, it can create a joint image and text vector, but in practice, you can’t with the API (and shouldn’t anyway).

The ecommerce product dataset we used

For this article, we downloaded the Fashion Product Images dataset from Kaggle. It contains roughly 44,000 catalog entries for products from a real fashion retailer, each with a high-resolution photo and structured metadata. We only used the Apparel and Footwear categories (about 30,600 products) and sampled 5,000 from them with a fixed random seed.

For each product, the dataset contains three records:

• Each product is pictured in a 1800×2400 JPEG against a clean background.

• Metadata with the labels gender, masterCategory, subCategory, articleType, baseColour, season, year, usage, and productDisplayName.

• A collection of additional informationwith labels like Neckline, Pattern, Sleeve Length, Fit, and Fabric, and a free-text description in English.

For example, item #13885 is labelled "Scullers Men Check Black Shirts," with an accompanying image (see below) and a description that reads "Black and white checked shirt, made of 100% cotton, full length buttoned placket, long sleeves with buttoned cuffs."

How we generated the search queries

We generated our test queries without an AI language model, using rules and substitution lists.

For each product, we started from its color and article type and added one descriptive modifier, drawn at random from the product’s available metadata: neckline, pattern, sleeve length, length, surface styling, fit, fabric, season, or usage. We then reworded the query by substituting words from a fixed synonym table. This avoids making queries that reuse the catalog's own words. For example, ″t-shirt″ becomes ″tee″, ″regular fit″ becomes ″classic cut″, ″printed″ becomes ″with graphic design″, ″sleeveless″ becomes ″no sleeves″. The longest phrase with a synonym was replaced first, so we swapped ″sports shoes″ rather than ″shoes″. Because the pipeline is rule-based and seeded, query production is reproducible and all variation is accounted for. German queries were generated the same way, using a German term table, and then a native speaker corrected them for natural retail phrasing. Some examples:

The German queries used their own term table, mapping the same catalog attributes to natural German retail phrasing (later checked by a native speaker). Some examples:

The six embedding configurations we tested

We tested retrieval in six configurations, using the same text queries in each test condition and the same 5,000 product indexed dataset. For both jina-clip-v2 and jina-embeddings-v5-omni-small, we tested three different ways of generating embeddings for indexing:

1. Image-only. We generated embeddings from the images alone without any other data.

2. Text-only. We generated embeddings for the free text descriptions alone.

3. Averaged image and text. For each product, we generated embeddings for the image and text description separately and then averaged the two vectors into one.

We used the Jina API to generate document and query embeddings for product images and free text descriptions, as shown in the code below. All images were resized to fit into a 512x512px square before processing.

We combined images and texts by embedding them separately, averaging the two vectors, and then normalizing the result so we can speed up calculating cosines. This works for multimodal models because both embeddings share the same semantic space. The sum of the two vectors is a new vector with the semantic features of both.

This is very easy to do using the numpy package in Python. We used the code below:

For this article, we did exact retrieval, calculating the cosine between queries and all 5,000 stored product embeddings. In Elasticsearch, we would use a shortcut to approximate the same result. From the ranked results, we calculate Recall@1, Recall@5, Recall@10, Mean Reciprocal Rank (MRR), and Normalized Discounted Cumulative Gain at position 10 (nDCG@10).

Each query has exactly one correct answer, so Recall@K is the share of queries whose product lands in the top K. MRR scores the results by how close the correct answer is to the top. nDCG@10 is a standard metric that penalizes putting the best answer lower on the results list.

Product search benchmark results

The table below is the German cross-lingual run. We evaluated German queries to find products with English descriptions:

Query benchmark results for jina-clip-v2 and jina-embeddings-v5-omni-small using German-language text queries and the images and English descriptions of garments.

The averaged image/text embeddings score the best, both using jina-clip-v2 and jina-embeddings-v5-omni-small. Surprisingly, averaged vectors from jina-clip-v2 lead the table. It ranks the correct product first about 1.5 times as often as the image-only setup and significantly more often than the text-only setup. Furthermore, it beats every jina-embeddings-v5-omni-small condition. Results for the same tests using English-language queries:

Query benchmark results for jina-clip-v2 and jina-embeddings-v5-omni-small using English-language text queries and the images and English descriptions of garments.

The English run, on the same 5,000 products, tells the same story. The main difference is that the jina-embeddings-v5-omni-small scores are significantly closer to those of jina-clip-v2, although still lower.

What the benchmark scores actually mean

The relatively low scores in the German and English benchmark tables above are to be expected. This is a real-world dataset full of near duplicates. A search for a "black tee with classic cut" has to sort through dozens of basic black T-shirts, and even on the best ecommerce sites, you would expect a result like this. The important thing to understand is the difference in scores between the different conditions, not their absolute values. Our key finding is that combining text and image embeddings yields better performance than either one alone, highlighting how multimodal AI-driven search can use different information sources to produce better performance than non-multimodal strategies.

When a picture is worth a thousand words, and when it isn’t

The gain from adding images to text embeddings isn’t evenly distributed. We did a deep dive to see if there was a pattern to the results and discovered a few things:

• Images add a lot for visually distinctive items and attributes. For example, on footwear, image-only retrieval is on par with text-only: 0.029 versus 0.028 for Recall@1. Shoe styles have distinctive shapes, so the picture does the work (image-only 0.029 versus text 0.028). The same holds for visible attributes more broadly (color, pattern, sleeve length), where fusing image and text gives the biggest lift over text-only (0.085 versus 0.073).

• Images help the least with things the model can’t see. For example, if we query for fabric types, adding images to embeddings adds next to nothing. Humans and AI models alike struggle to see that something is or isn’t made of linen or polyester or some other fabric type. That information is only in the text description and metadata.

As a rule of thumb, we find that searches for clothing lean more heavily on accurate text descriptions, while footwear leans more on the semantics of images. But in both cases, merging the two embeddings either improves results or doesn’t make them worse. This highlights how use-case–specific considerations drive optimal search strategies.

Cross-language queries gain a lot from multimodal embeddings

The gap in performance between text-only and combined image and text embeddings using jina-clip-v2 is much larger for German queries (0.074 versus 0.065 Recall@1) than for English ones (0.076 versus 0.075). This implies that English queries can take advantage of being in the same language as the product descriptions. Whether that’s due to overlaps in the words or that the model is simply more competent with single-language semantics than cross-language doesn’t matter. But adding images to the text embeddings compensates almost completely for the model’s shortcomings in cross-language retrieval.

This gap is even larger for jina-embeddings-v5-omni-small. In any kind of cross-language or multilingual context, multimodal embeddings seem to significantly improve retrieval performance.

Can AI-generated product descriptions replace human ones?

AI-generated descriptions scored worse than human-written ones in our tests. We tried replacing human-authored product descriptions with ones written by jina-vlm based on the image. For this test, we used a 1,000-product random subset. The results were much worse than with the original human text. This was what we expected: The automatically generated description was less accurate and less oriented toward the salient features of the product than the human authored one.

So it turns out that not everyone’s job can be replaced by AI. People who write blurbs for catalogs ought to be safe for now.

How should you index your ecommerce data?

Our tests aren’t totally scientifically rigorous, but they do offer some insights into the issues you might face if you have similar data. We offer the following as provisional conclusions:

• If you have aligned texts and images (and most catalogs do), combine them in your embeddings. In every case, using a multimodal embedding model like the ones Jina AI by Elastic provides and then averaging the image and text embeddings significantly outperformed all other options. The combination adds no computing costs at inference time but does create additional costs at embedding time. For each product, you’ll need to generate two embeddings and combine them, roughly doubling the cost.

• Use the right model. You need to identify a model that supports all the modalities and languages you plan to use. All the inputs have to be embedded in the same semantic vector space or none of this will work. It won’t do to get two single-modality models or multiple language-specific text models, average their outputs, and hope for the best. Jina AI by Elastic currently supports texts in up to 100 languages, including computer code and technical terminology images of all kinds, such as scans and infographics, as well as audio and video data. You can change your mind about your models later, but only if you’re willing to reindex all your data.

• Whatever you pick, test it on your own data. The only way to know what the best model is for you, your data, and your use case is to try them out. We were very surprised that our older CLIP-style model outperformed our latest on this dataset, but it was trained almost specifically for this use case. Your data and use case could easily show the opposite. This result is from one catalog, with one style of photography and one kind of query. The ranking between two models can flip with a different domain, image style, or query mix. Run the same sort of benchmark on a sample of your own products before you commit. It’s the only way to know which model really fits your case.

• Use generated descriptions to fill gaps, not to replace good text. AI isn’t a replacement for good work done by conscientious people. Replacing human-made descriptions with machine-made ones made results worse. AI should only replace humans when it has to, like when data is missing or needs to be augmented and it’s impractical to have humans fill in the gaps. Yes, we have tools that work in those situations, but they aren’t necessarily good substitutes. They’re OK substitutes, sometimes.

Average your embeddings. Semantic embeddings are very robust, and averaging them is a relatively cheap solution that doesn’t affect inference-time costs at all. This is a real boon over methods that index each modality separately and require multiple queries to satisfy a single request. But they do require compatible multimodal models.

Limitations of this product search benchmark

A few things to keep in mind before generalizing too much from this experiment:

This article doesn’t perfectly match real-world use cases. Human users make messier queries and have more ambiguous matching criteria. The queries we used were generated specifically for this data. A test with actual customer-made queries from system logs would be a better one.

Embedding averaging isn’t the same as a true joint embedding. Embedding models rely on the different parts of their input to interact in order to extract a semantic representation of the whole. The approach used here is a bit of a hack, one that relies on the robust nature of semantic embedding spaces to get the job done. We expect future models from Jina AI to produce better embeddings by supporting more than one input modality at a time.

This is one dataset in one domain with distinctive features. Fashion photography is very foreground-focused and the descriptions are attribute-rich. Other kinds of materials, even for ecommerce, may look very different. It’s important to test as much as possible with your own data or something very similar.

How to get started with multimodal product search embeddings

jina-clip-v2 and jina-embeddings-v5-omni-small are available through the Jina API, Elastic Inference Service (EIS), and Hugging Face. The omni models are free to download under a CC-BY-NC-4.0 license and free for noncommercial use, with commercial licensing through Elastic. If you use Elasticsearch, EIS exposes both models through the semantic field type, with non-text media in Base64 encoding.

The takeaway from this article is intended to be practical and actionable: For product search, a picture and its description aren’t the same signal. Both add information, and you don’t have to pick one. Average your multimodal embeddings, and benchmark the results with your own data to get a good picture of the kinds of results you can expect.

What you'll discover

In this post, you'll learn:

• The value of a comprehensive search analytics stack.

• The case for using a modern observability standard for search analytics.

• The benefits of combining search analytics and observability.

• Practical first steps for getting started.

The search analytics challenge

Your product manager wants to know which searches are driving purchases and which ones are losing customers. To answer, you have to grep through 5GB of Nginx logs, join it with a CSV of purchase data, and pray that the timestamps align. There has to be a better way.

The questions seem simple:

• Which searches lead to revenue, and which ones lose customers?

• Are users finding what they're looking for?

• Which queries are failing?

• How do you connect search behavior to business outcomes?

Getting answers is harder. Traditional approaches involve stitching together custom event pipelines, third-party analytics tools, and hand-rolled dashboards. You end up with data silos: search logs here, click events there, and business metrics somewhere else entirely.

What if there was a simpler way?

Enter OpenTelemetry (and why it fits search)

If you come from a search engineering background, you may not have crossed paths with application performance monitoring (APM), which is the practice of instrumenting your code to understand how it behaves in production. The key idea behind modern APM is the trace, a structured record that follows a single request as it flows through your system, from the initial API call through every database query and service hop. Unlike logs, which are isolated lines of text, traces connect the dots across your entire stack.

OpenTelemetry (OTel) has become the industry standard for producing these traces. Born from the merger of OpenTracing and OpenCensus, it provides a vendor-neutral way to collect traces, metrics, and logs. The core idea is simple: Instrument your code once, and send data anywhere.

Elastic has invested heavily in this standard as a contributor. Elastic actively aligns its Elastic Common Schema (ECS) with OTel Semantic Conventions so that field names are consistent across both, and it donated the Universal Profiling Agent to the OTel project to make profiling a core OTel signal.

Because of this alignment, Elastic natively understands OTel data. OTel transmits data using the OpenTelemetry Protocol (OTLP). On Elastic Cloud, the managed OTLP endpoint (mOTLP) accepts OTLP spans directly from your SDK, without requiring an intermediate collector. For self-managed deployments, the Elastic Distribution of OpenTelemetry (EDOT) Collector provides the same path. This lets you use standard OTel traces for search analytics, without any additional infrastructure.

How traces become analytics

A traceis a collection of spans, where each span represents one operation (an API call, a database query, or a search request). Each span can carry arbitrary attributes(key-value pairs that describe what happened).

This is where it gets interesting for search. Search behavior can ride on the same spans OTel already generates for APM, with no new attributes required beyond what you add to existing calls.

• search.query: What the user searched for.

• search.result_count: How many results came back.

• search.result_click_position: Which result they clicked.

By extending OTel with attributes like these, we can capture rich behavioral data using the same infrastructure that powers application monitoring. You don’t need a new pipeline or a new vendor; you’re simply adding new attributes to existing spans.

A unified approach: OTel + Elastic + ES|QL

If you're building search, there's a good chance you're already running Elasticsearch. It powers search for thousands of ecommerce sites, content platforms, and enterprise applications. It’s often chosen for its speed, flexibility, and features like vector search, Learning To Rank (LTR), and query rules. What's less well-known in the search community is that Elastic also has over a decade of investment in observability. Elastic APM, logging, and infrastructure monitoring are used at scale across industries.

Running search and observability on the same platform is what makes this approach work. The same platform that runs your search engine can also analyze how people use it. Elastic natively ingests OTel data via its managed OTLP endpoint, and Elasticsearch Query Language (ES|QL), Elasticsearch's piped query language, makes it easy to explore and aggregate that data in Kibana.

Together, they offer a compelling approach to search analytics:

• Instrument once: Add OTel attributes to your search requests and user interactions.

• Store centrally: Traces flow to Elastic via mOTLP alongside your other application telemetry.

• Query flexibly: Use ES|QL in Kibana to calculate metrics, slice by any dimension, and explore patterns.

You don’t need a separate analytics pipeline or a dedicated click-tracking service. You just need your search application with the proper instrumentation.

What you can measure

With the right instrumentation in place, you can answer the questions that matter:

Search quality metrics

• CTR: What percentage of searches result in a click? Low CTR might indicate poor relevance or unappealing result presentation.

• Mean Reciprocal Rank (MRR): When users click, how far down the results list do they go? An MRR of 1.0 means every click is on position 1. If everybody clicks on the second position result, your MMR would be 0.5. Higher is better.

• Zero Results Rate: What percentage of searches return nothing? These are your content gaps or query parsing failures. Every zero-result search is a missed opportunity.

Query-level analysis

Beyond aggregate metrics, you can drill into specific queries:

• Top queries by volume: What are users actually searching for?

• Queries with low CTR: Where is relevance failing?

• Zero-result queries: What content is missing from your index?

• Click position distribution: Are clicks concentrated at the top, or are they scattered?

Business impact

Search doesn't exist in a vacuum. By tracking the journey from search to conversion, you can connect relevance to revenue:

• Which searches lead to add-to-cart events?

• What's the conversion rate for searches versus browsing?

• Which queries drive the most revenue?

Training data for machine learning

The same click data that measures search quality can also train models to improve it. Click positions and frequencies can be transformed into judgment lists for LTR models, turning user behavior into relevance signals.

Search analytics dashboard example in Kibana

Here's what this looks like in practice. This Kibana dashboard is powered entirely by OTel traces and ES|QL, with no custom pipeline or separate analytics service:

Every panel on this dashboard comes from the same index. The headline metrics use value-based coloring to surface health at a glance: CTR and MRR are green (healthy), while the Zero Results Rate is red (needs attention). Below, you can see which queries drive the most clicks, which ones return nothing, and how searches convert through the funnel to revenue.

Notice the SLO cards at the bottom, with targets like "99% of searches under 250ms" that let you track reliability as a measurable commitment. That's a hint at something bigger.

How does search analytics data flow into Elastic?

The data flow is straightforward:

• Browser: User searches and clicks. The front end can send events directly via OTel or relay them through the back end.

• Back end: Your search API, instrumented with OTel. Each search request becomes a span, with attributes like search.query, search.result_count, and search.query_id.

• Elastic (mOTLP): Receives and stores the OTel traces via the mOTLP endpoint. Point OTEL_EXPORTER_OTLP_ENDPOINT at your Elastic deployment with ApiKey auth. This doesn’t require a collector or transformation.

• Kibana: Where you explore and visualize your data. Use ES|QL in Discover to run ad hoc queries, use the APM UI to inspect individual traces, build dashboards for ongoing monitoring, and set up alerts when metrics degrade.

How OTel attributes become queryable data

Before we query, you need to understand how Elastic stores these OTel attributes, because it affects how you write your ES|QL.

With OTel-native ingestion, custom attributes are stored directly under attributes.*, and their original dot notation is preserved. There’s no underscore translation and no type splitting, and strings, numbers, and booleans all live in the same namespace.

OTel attribute

Type

ES|QL field

`search.query`

string

`attributes.search.query`

`search.result_count`

number

`attributes.search.result_count`

`search.first_click`

boolean

`attributes.search.first_click`

This mapping is as simple as it looks: The attribute name in your code is the field name in your query. We'll cover more examples in the next post, when you're writing queries hands-on.

ES|QL: Your search data in five lines

ES|QL is Elasticsearch's piped query language. You start with a data source, and then, one step at a time, pipe it through filters, aggregations, and calculations.

Here's a single query that calculates your CTR, the percentage of searches that result in at least one click:

Reading from the top down: Pull all search spans and first-click spans from traces-generic.otel-default, count each type separately with COUNT(CASE(...)), and then divide to get CTR. We label the first click on each search with search.first_click at instrumentation time, so the query just counts it and no deduplication is needed. One query provides one counted result, which gives you the CTR.

That's the pattern. ES|QL reads from the top down, and each pipe step transforms the data. In the next post, we'll run six queries like this, including top queries, zero-results analysis, search performance, and volume over time, all against real data. Stay tuned for it!

From search analytics to full-stack observability

By choosing OTel and Elastic for search analytics, you're investing in infrastructure that serves multiple needs.

The same traces that calculate your CTR also give you operational visibility, without requiring any extra instrumentation:

The search dashboard's SLO cards track three targets: your search latency target (99% under 250ms), search quality (85% returning results), and availability (99.9% success rate). Below them, latency percentiles use the same value-based coloring. Green means healthy, yellow means watch it, and red means needs attention.

The operational charts break down where time is spent (Elasticsearch query time versus application overhead) and how latency trends over time. All of this comes from the same search.* attributes you added for analytics, and no extra instrumentation is required.

This is the practical advantage of the OTel approach:

• Search latency: How long are queries taking? Where are the slow ones?

• Error rates: Are searches failing? If so, why?

• Dependencies: How does Elasticsearch performance affect your search API?

• SLOs: Set targets for search latency or Zero Results Rate.

• Alerting: Get notified when metrics degrade.

• Anomaly detection: Let machine learning (ML) spot unusual patterns in search behavior.

Start with search analytics, and you can extend that foundation into full-stack observability. It’s a single instrumentation investment that results in multiple returns.

What OTel attributes do you need for search analytics?

To get started, you'll want to capture a few key attributes on your search spans:

Attribute

Type

Purpose

`search.query`

string

The user's search terms

`search.result_count`

number

How many results returned (0 = zero-result search)

`search.query_id`

string

Unique query identifier (derived from trace ID)

`search.result_click_id`

string

Which result was clicked

`search.result_click_position`

number

Position of the clicked result (1-indexed)

For click tracking, add these attributes to click event spans:

Attribute

Type

Purpose

`search.result_click_id`

string

Document ID that was clicked

`search.result_click_position`

number

Position in results (1-indexed)

`search.action`

string

Event type: click, impression, add_to_cart, purchase

`search.query_id`

string

Links this interaction back to the originating search

`search.first_click`

boolean

`true` on the first click per search (for CTR without deduplication)

We use the search.* namespace following OTel's convention of domain-specific prefixes (http.*, db.*, messaging.*). While OTel doesn't yet have standardized search conventions, search.* is self-describing and vendor-neutral. Our naming is informed by the User Behavior Insights (UBI) standard, which defines a detailed schema for search events. We reference it for event structure without coupling our instrumentation to it.

If you don't have APM set up yet, you can capture the same search.* attributes as OTel log records instead of spans. The analytics concepts are identical; you just query logs-generic.otel-default instead of traces-generic.otel-default, and the attributes.* field paths are the same. See the OpenTelemetry logs with Elastic documentation for setup details.

What's next in this search analytics series

This post introduced the concept of using OTel instrumentation and ES|QL queries to build search analytics on Elastic. The rest of the series goes from concept to production. Stay tuned for it!

Get started with search analytics on Elastic

Ready to add search analytics to your application?

• Reference project: Clone, configure, and have search analytics flowing in 10 minutes.

• OpenTelemetry and Elastic: How to send OTel data to Elastic via the managed OTLP endpoint.

• ES|QL documentation: Learn the query language.

• Elastic Observability: The broader platform.

This is the first post in a six-part series on search analytics with OpenTelemetry and Elastic. Next up: Instrument your search API: Add search attributes to your back end, and run your first ES|QL queries.

If agents can call tools on demand and models can reason over massive context windows, why not just fetch data live from every system and let the LLM figure it out?

It’s a reasonable question. It’s also the wrong conclusion.

The reality is that MCP and agent tooling don’t eliminate the need for search. They make the quality of search more critical than ever. In this blog, we’ll explore why MCP, federated search, and large context windows don’t replace search engines and why indexes remain the foundational layer for scalable, accurate, enterprise-grade AI.

What MCP actually is (and what it is not)

MCP is a coordination protocol. It standardizes how an agent requests information or actions from external systems.

What MCP doesn’t do:

• Rank results across systems.

• Understand relevance across heterogeneous data.

• Normalize schemas or metadata.

• Data transformations or enrichments at scale.

• Apply consistent security and permissions.

• Optimize for latency, cost, or scale.

In other words, MCP tells agents how to ask for data, not which data matters most.

Modern retrieval requires query intelligence, not just data access

In modern enterprise search architectures, retrieval quality is determined long before a query reaches an index. Raw queries — especially those generated by agents — may be incomplete, overly literal, schema-driven rather than intent-driven, and at times syntactically invalid.

This is why mature search platforms introduce a query intelligence layer that performs query rewriting, entity normalization, synonym expansion, and intent disambiguation before retrieval even begins.

For example, an agent-generated request such as: “Show severity 2 authentication failures from last sprint” may be rewritten to include authentication synonyms (login, SSO, OAuth), normalized severity mappings, and sprint-to-date-range translation. The result is not just more matches — it is more relevant matches.

In enterprise AI, retrieval is not a single step. It is a controlled pipeline.

This distinction is crucial because once MCP-based agents start pulling information live from multiple tools, they recreate a familiar pattern under a new name: federated search.

MCP-based retrieval is federated search in disguise

Federated search isn’t new. Enterprises have tried it for decades.

The model is simple:

• Send the user’s query to multiple systems in parallel (SharePoint, GitHub, Jira, customer relationship management [CRM]).

• Collect the responses.

• Merge and present the results.

MCP-driven tool calls follow the same pattern, except that the caller is now an agent instead of a user interface.

And the same problems resurface.

Why federated search breaks down at enterprise scale

• Latency becomes unpredictable: A federated query is only as fast as its slowest system. Enterprise systems can have wildly different response times and rate limits, so federated queries tend to be slow and jittery. Agents must wait for multiple round trips before reasoning can even begin. The result is a laggy experience and unpredictable wait times.

• Relevance is fragmented: Because each system ranks results on its own, there’s no unified relevance model. Federated search cannot apply a single ranking or semantic understanding across all content, so results often seem disjointed or incomplete. Agents may retrieve correct information but not the most useful information.

• Context is shallow and incomplete: Federated systems typically expose only what’s directly accessible through an API call.They rarely surface:

  • Usage signals, like clicks, dwell time, recency of access, popularity, or authority.

  • Relationships between documents across different systems to correlate the insights.

  • Organizational knowledge beyond a single silo. This strips agents of the broader context required for high-quality reasoning.

• Limited filtering and features: In a federated setup, you can only filter on fields that every system supports (the “lowest common denominator”). If one system doesn’t support a particular filter or facet, you lose that functionality entirely. This severely limits rich search features, like date ranges, categories, or tags.

The power of an indexed search

Search engines achieve millisecond-level retrieval at massive scale by using specialized data structures, including inverted indexes for lexical search and k‑dimensional trees (k-d trees) for vector-based retrieval. The approach is to crawl or ingest every source into search engines, creating a central place of company knowledge. This brings big advantages:

• Speed by design: Searching an index is lightning fast. Queries hit inverted indexes and specialized data structures, avoiding the need to poll each backend system.

• Relevance that compounds over time: Search engines that support semantic search are capable of comprehending the intent, and machine learning models can rerank results for enterprise contexts. In one Elastic experiment, Elastic users see more accurate results when combining vector search with a question-answering (QA) model to extract answers. It gives better precision than keyword matching.

• Advanced features: Elastic’s Graph retrieval augmented generation (RAG) solution shows how structuring an index as a knowledge graph can power more contextual retrieval. In other words, indexes aren’t just backward-looking dumps of text; they can also encode relationships and ontologies that let AI connect the dots across documents.

• Permission-aware search: Enterprise AI cannot compromise on security. Indexed search allows:

  • Document-level security.

  • Role-based access control.

  • Permission-aware retrieval for RAG and agents.

Agents see only what users are allowed to see, without leaking data into model prompts or training. Elasticsearch is suitable for the indexed search layer in the diagram below, as it provides the essential components for context engineering.

Retrieval consistency through search templates and governed execution

At scale, retrieval must be predictable, secure, and repeatable. This is where search templates become critical.

Search templates act as retrieval contracts between applications, agents, and the search platform. Instead of dynamically constructing queries at runtime, agents invoke pre-defined retrieval patterns that enforce:

• Consistent relevance logic

• Mandatory security filters

• Cost and latency guardrails

• Business-specific ranking rules

• Explicit index and field scope boundaries

In MCP-driven architectures, this becomes even more important. Agents should not dynamically invent retrieval strategies. Instead, MCP tool calls can map directly to approved search templates, ensuring that every retrieval request adheres to enterprise relevance and governance standards.

This approach shifts retrieval from ad-hoc query execution to controlled retrieval orchestration.

Retrieval is now a multi-layer engineering discipline

Modern enterprise retrieval is no longer a simple query-to-index operation. It typically includes multiple coordinated layers:

• Query understanding — rewriting, expansion, entity resolution

• Retrieval strategy selection — hybrid search, vector search, graph retrieval, or synthetic query techniques such as Hypothetical Document Embeddings (HyDE), where the system generates a representative answer or expanded context first and retrieves documents using that richer semantic signal.

• Execution governance — templates, security enforcement, and performance guardrails

• Ranking and re-ranking — blending lexical precision, semantic similarity, and interaction-derived relevance signals such as click-through patterns, dwell time, and document usage frequency.

When these layers are implemented upstream, agents receive clean, high-confidence context rather than raw, fragmented data.

This is what makes large-scale agent systems reliable in production environments.

Advanced retrieval techniques improve context quality before reasoning begins

Modern retrieval systems increasingly use AI-assisted techniques to improve recall and semantic coverage before ranking is applied.

One example is Hypothetical Document Embeddings (HyDE). Instead of embedding only the original query, the system first generates a hypothetical answer or expanded context, embeds that representation, and retrieves documents based on that richer semantic signal.

This is particularly useful in enterprise environments where:

• Users or agents may not know the exact terminology

• Knowledge is distributed across silos

• Important context is implied rather than explicitly stated

Techniques like HyDE improve the probability that relevant documents are retrieved even when the original query is underspecified.

This reinforces a key principle of enterprise AI: better context retrieval produces better reasoning outcomes.

Agents aren’t data engineers; they’re reasoning systems

They shouldn’t be responsible for stitching together raw data, reconciling schemas, or compensating for poor retrieval.

This is where a search platform such as Elasticsearch becomes foundational.

By ingesting data once and normalizing it upstream (through pipelines, mappings, enrichment processors, and prebuilt indexes), Elasticsearch resolves schema mismatches, joins signals across sources, and materializes retrieval-ready views of the data. At query time, the agent receives clean, ranked, semantically enriched results rather than fragmented raw records.

For example, instead of an agent pulling independently from CRM, ticketing, and documentation systems and attempting to reconcile customer IDs, timestamps, and formats in real time, Elasticsearch can pre-index these sources into a unified customer interaction index with hybrid (keyword + vector) search and relevance ranking. The agent then queries a single, coherent interface and immediately reasons over the most relevant context.

This separation of concerns, that is, Elasticsearch handling data integration and retrieval, and agents focusing on reasoning, planning, and decision-making,is what makes agent systems scalable, reliable, and production ready.

Elastic’s role in the AI stack

Elastic sits at the intersection of search and AI by design.

• Connectors and crawlers ingest data continuously from enterprise systems.

• Semantic and vector search enable intent-based retrieval.

• Hybrid search blends lexical precision with semantic understanding.

• RAG workflows ground LLMs in authoritative, permission-aware data.

Elastic does not compete with agents or MCP. It makes them effective.

Bigger models don’t eliminate retrieval

Some have wondered whether huge new LLMs can bypass traditional search, perhaps by letting the model read everything in one go. Large context windows feel powerful, but they introduce:

• Higher latency.

• Higher cost.

• Lower precision due to noise.

• A higher propensity for confusion, context clash, and context poisoning.

RAG wins because it filters first and then reasons.In another Elastic Search Labs experiment, RAG achieved answers in about 1 second, versus 45 seconds for the raw-LM approach, at 1/1250th the cost, and with far higher accuracy. In other words, giving an LLM a million tokens of documents is slower, more expensive, and actually less precise than filtering through an index first.

Conclusion: MCP changes the interface, not the fundamentals

MCP is a meaningful step forward in how agents interact with tools. But it doesn’t replace the need for fast, relevant, governed retrieval.

In enterprise AI:

• Context quality determines answer quality.

• Indexes create that context.

• Search is the foundation, not the legacy.

Indexes aren’t obsolete in the era of MCP. They’re the reason that MCP-based agents can work at all.

Quick start

Download the relevant release or build and install (Linux build script generates Elasticsearch‑compatible zip):

Install in Elasticsearch:

Test:

Why Hebrew search is different

Hebrew is morphologically rich: Prefixes, suffixes, inflection, and clitics all collapse into a single surface form. That makes naive tokenization insufficient. Without true lemmatization, search quality suffers; users miss relevant results due to simple variations in form. This project tackles that by embedding a Hebrew lemmatization model inside the analyzer itself, so every token passes through a neural model before indexing and querying.

Example

Users may search for the lemma “בית” (house), but documents might contain:

• בית (a house)

• בבית (in the house)

• לבית (to the house)

• בבתים (in houses)

• לבתים (to houses)

Without lemmatization, these become different surface tokens; lemmatization normalizes them toward the same lemma (בית), improving recall:

What this plugin does

Rather than relying on rule-based stemming, the analyzer runs a Hebrew lemmatization model as part of the Elasticsearch analysis chain and emits one normalized lemma per token. Because the model is neural, it can use local context within each analyzed segment to choose a lemma in ambiguous cases—while still producing stable tokens that work well for indexing and querying. The analyzer:

• Runs a Hebrew lemmatization model inside Elasticsearch.

• Produces better normalized tokens for Hebrew text.

• Supports stopwords and standard analyzer pipelines.

The result: Fast, reliable lemmatization

This analyzer is optimized for real‑world throughput:

• ONNX Runtime in‑process inference.

• INT8-quantized model for lower latency and memory footprint.

• Java Foreign Function Interface (FFI) for high‑performance native inference.

The result: fast, reliable lemmatization with predictable operational behavior.

To evaluate performance, we ran a benchmark in a Docker container (4 cores, 12 GB RAM) on 1 million large documents (5.7 GB of data) from the Hebrew Wikipedia dataset. You’ll find the results below:

Metric (search)

Task

Value

Unit

Min throughput

hebrew-query-search

409.75

ops/s

Mean throughput

hebrew-query-search

490.65

ops/s

Median throughput

hebrew-query-search

491.85

ops/s

Max throughput

hebrew-query-search

496.13

ops/s

50th percentile latency

hebrew-query-search

7.02242

ms

90th percentile latency

hebrew-query-search

10.7338

ms

99th percentile latency

hebrew-query-search

19.0406

ms

99.9th percentile latency

hebrew-query-search

27.165

ms

50th percentile service time

hebrew-query-search

7.02242

ms

90th percentile service time

hebrew-query-search

10.7338

ms

99th percentile service time

hebrew-query-search

19.0406

ms

99.9th percentile service time

hebrew-query-search

27.165

ms

Error rate

hebrew-query-search

0

%

Open source and Elastic‑ready

The plugin is fully open source and works on:

• Elastic open‑source distributions.

• Elastic Cloud.

You can build it yourself or download prebuilt releases and install it like any other plugin.

To upload the analyzer plugin to Elastic Cloud, navigate to the Extensions section within your Elastic Cloud console and proceed with the upload.

Credits

This project is a fork of the Korra ai Hebrew analysis plugin (MIT), which was implemented by Korra.ai with funding and guidance from the National NLP Program led by MAFAT and the Israel Innovation Authority.

This fork focuses on Elasticsearch 9.x compatibility and running lemmatization fully in-process via ONNX Runtime, using an INT8‑quantized model and bundled Hebrew stopwords. Lemmatization is powered by DictaBERT dicta-il/dictabert-lex (CC‑BY‑4.0).

Huge thanks to the Dicta team for making high-quality Hebrew natural language processing (NLP) models available to the community.

Links

• GitHub repo

• Releases

Multistage retrieval: The challenge of modern search

Modern search has evolved beyond simple keyword matching. Today's search applications need to understand intent, handle natural language, and combine multiple ranking signals to deliver the best results.

Retrieval of the most relevant results happens in multiple stages, with each stage gradually refining the result set. This wasn’t the case in the past, where most use cases would require one or two stages of retrieval: an initial query to get results and a potential rescoring phase.

We start with an initial retrieval, where we cast a wide net to gather results that are relevant to our query. Since we need to sieve through all the data, we should use techniques that return results fast, even when we index billions of documents.

We therefore employ trusted techniques, such as lexical search that Elasticsearch has supported and optimized since the beginning, or vector search, where Elasticsearch excels in speed and accuracy.

Lexical search using BM25 is quite fast and best at exact term matching or phrase matching, and vector or semantic search is better suited for handling natural language queries. Hybrid search combines lexical and vector search results to bring the best from both. The challenge that hybrid search solves is that vector and lexical search have completely different and incompatible scoring functions which produce values in different intervals, following different distributions. A vector search score close to 1 can mean a very close match, but it doesn’t mean the same for lexical search. Hybrid search methods, such as reciprocal rank fusion (RRF) and linear combination of scores, assign new scores that blend the original scores from lexical and vector search.

After hybrid search, we can employ techniques such as semantic reranking and Learning To Rank (LTR), which use specialized machine learning models to rerank the result.

With our most relevant results, we can use large language models (LLMs) to further enrich our response or pass the most relevant results as context to LLMs in agentic workflows in tools such as Elastic Agent Builder.

ES|QL is able to handle all these stages of retrieval. By design, ES|QL is a piped language, where each command transforms the input and sends the output to the next command. Each stage of retrieval is represented by one or more consecutive ES|QL commands. In this article, we show how each stage is supported in ES|QL.

Vector search

In Elasticsearch 9.2, we introduced tech preview support for dense vector search in ES|QL. This is as simple as calling the knn function, which only requires a dense_vector field and a query vector:

This query executes an approximate nearest neighbor search, retrieving 100 documents that are the most similar to the query_vector.

Hybrid search: Reciprocal rank fusion

In Elasticsearch 9.2, we introduced support for hybrid search using RRF and linear combination of results in ES|QL.

This allows combining vector search and lexical search results into a single result set.

To achieve this in ES|QL, we need to use the FORK and FUSE commands. FORK runs multiple branches of execution, and FUSE merges the results and assigns new relevance scores using RRF or linear combination.

In the following example, we use FORK to run two separate branches, where one is doing a lexical search using the match function, while the other is doing a vector search using the knn function. We then merge the results together using FUSE:

Let's decompose the query to better understand the execution model and first look at the output of the FORK command:

The FORK commands outputs the results from both branches and adds a _fork discriminator column:

_id

title

_score

_fork

4001

The Hobbit

0.88

fork1

3999

The Fellowship of the Ring

0.88

fork1

4005

The Two Towers

0.86

fork1

4006

The Return of the King

0.84

fork1

4123

The Silmarillion

0.78

fork1

4144

The Children of Húrin

0.79

fork1

4001

The Hobbit

4.55

fork2

3999

The Fellowship of the Ring

4.25

fork2

4123

The Silmarillion

4.11

fork2

4005

The Two Towers

3.8

fork2

4006

The Return of the King

4.1

fork2

As you’ll notice, certain documents appear twice, which is why we then use FUSE to merge rows that represent the same documents and assign new relevance scores. FUSE is executed in two stages:

• For each row, FUSE assigns a new relevance score, depending on the hybrid search algorithm that is being used.

• Rows that represent the same document are merged together, and a new score is computed.

In our example, we’re using RRF. As a first step, FUSE assigns a new score to each row using the RRF formula:

Where the rank_constant takes a default value of 60 and rank(doc)represents the position of the document in the result set.

In the first phase, our results become:

_id

title

_score

_fork

4001

The Hobbit

1 / (60 + 1) = 0.01639

fork1

3999

The Fellowship of the Ring

1 / (60 + 2) = 0.01613

fork1

4005

The Two Towers

1 / (60 + 3) = 0.01587

fork1

4006

The Return of the King

1 / (60 + 4) = 0.01563

fork1

4123

The Silmarillion

1 / (60 + 5) = 0.01538

fork1

4144

The Children of Húrin

1 / (60 + 6) = 0.01515

fork1

4001

The Hobbit

1 / (60 + 1) = 0.01639

fork2

3999

The Fellowship of the Ring

1 / (60 + 2) = 0.01613

fork2

4123

The Silmarillion

1 / (60 + 3) = 0.01587

fork2

4005

The Two Towers

1 / (60 + 4) = 0.01563

fork2

4006

The Return of the King

1 / (60 + 5) = 0.01538

fork2

Then the rows are merged together and a new score is assigned. Since a SORT _score DESC follows the FUSE command, the final results are:

_id

title

_score

4001

The Hobbit

0.01639 + 0.01639 = 0.03279

3999

The Fellowship of the Ring

0.01613 + 0.01613 = 0.03226

4005

The Two Towers

0.01587 + 0.01563 = 0.0315

4123

The Silmarillion

0.01538 + 0.01587 = 0.03125

4006

The Return of the King

0.01563 + 0.01538 = 0.03101

4144

The Children of Húrin

0.01515

Hybrid search: Linear combination of scores

Reciprocal rank fusion is the simplest way to do hybrid search, but it isn’t the only hybrid search method that we support in ES|QL.

In the following example, we use FUSE to combine lexical and semantic search results using linear combination of scores:

Let's first decompose the query and take a look at the input of the FUSE command when we only run the FORK command.

Notice that we use the match function, which is able to not only query lexical fields, such as text or keyword, but also semantic_text fields.

The first FORK branch executes a semantic query by querying a semantic_text field, while the second one executes a lexical query:

The output of the FORK command can contain rows with the same _id and _index values representing the same Elasticsearch document:

_id

title

_score

_fork

4001

The Hobbit

0.88

fork1

3999

The Fellowship of the Ring

0.88

fork1

4005

The Two Towers

0.86

fork1

4006

The Return of the King

0.84

fork1

4123

The Silmarillion

0.78

fork1

4144

The Children of Húrin

0.79

fork1

4001

The Hobbit

4.55

fork2

3999

The Fellowship of the Ring

4.25

fork2

4123

The Silmarillion

4.11

fork2

4005

The Two Towers

3.8

fork2

4006

The Return of the King

4.1

fork2

In the next step, we use FUSE to merge rows that have the same _id and _index values, and assign new relevance scores.

The new score is a linear combination of the scores the row had in each FORK branch:

Here, _score1 and _score2 represent the score a document has in the first FORK branch and the second FORK branch, respectively.

Notice that we also apply custom weights, giving more weight to the semantic score over the lexical one, resulting in this set of documents:

_id

title

_score

4001

The Hobbit

0.7 * 0.88 + 0.3 * 4.55 = 1.981

3999

The Fellowship of the Ring

0.7 * 0.88 + 0.3 * 4.25 = 1.891

4006

The Return of the King

0.7 * 0.84 + 0.3 * 4.1 = 1.818

4123

The Silmarillion

0.7 * 0.78 + 0.3 * 4.11 = 1.779

4005

The Two Towers

0.7 * 0.86 + 0.3 * 3.8 = 1.742

4144

The Children of Húrin

0.7 * 0.79 + 0.3 * 0 = 0.553

One challenge is that the semantic and lexical scores can be incompatible to apply the linear combination, since they can follow completely different distributions. To mitigate this, we first need to normalize the scores, employing score normalization methods, such as minmax. This ensures that the scores from each FORK branch are first normalized to take values between 0 and 1, before applying the linear combination formula.

To achieve this with FUSE, we need to specify the normalizer option:

Semantic reranking

At this stage, after hybrid search, we should be left with the most relevant documents. We can now use semantic reranking to reorder the results using the RERANK command. By default, RERANK uses the latest Elastic semantic reranking machine learning model, so no additional configuration is needed:

We now have our best results, sorted by relevance.

One key feature that sets the RERANK command apart from other products that offer semantic reranking integrations is that it doesn’t require the input to represent a mapped field from an index. RERANK only expects an expression that evaluates to a string value, making it possible to do semantic reranking using multiple fields:

LLM completions

Now we have a set of highly relevant, reranked results.

At this stage, you might simply decide to return the results back to your application or you might want to further enhance your results using LLM completions.

If you’re using ES|QL as part of a retrieval-augmented generation (RAG) workflow, you can choose to call your favorite LLM directly from ES|QL. To achieve this, we’ve added a new COMPLETION command that takes in a prompt, a completion inference ID which designates which LLM to call, and a column identifier to specify where to output the LLM response.

In the following example, we’re using COMPLETION to add a new _completion column that contains the summary of the content column:

Each row now contains a summary:

_id

title

_score

summary

4001

The Hobbit

0.03279

Bilbo helps dwarves reclaim Erebor from the dragon Smaug.

3999

The Fellowship of the Ring

0.03226

Frodo begins the quest to destroy the One Ring.

4005

The Two Towers

0.0315

The Fellowship splits; war comes to Rohan; Frodo nears Mordor.

4123

The Silmarillion

0.03125

Ancient myths and history of Middle-earth's First Age.

4006

The Return of the King

0.3101

Sauron is defeated and Aragorn is crowned King.

4144

The Children of Húrin

0.01515

The tragic tale of Túrin Turambar's cursed life.

In another use case, you may simply want to answer a question using the proprietary data that you have indexed in Elasticsearch. In this case, the best search results that we’ve computed in the previous stage can be used as context for the prompt:

Since the COMPLETION command unlocks the ability to send any prompt to an LLM, the possibilities are endless. Although we’re only showing a few examples, the COMPLETION command can be used in a wide range of scenarios, from security analysts using it to assign scores depending on whether a log event can represent a malicious action or data scientists using it to analyze data, to cases where you just need to generate Chuck Norris facts based on your data.

This is only the beginning

In the future, we’ll be expanding ES|QL to improve semantic reranking for long documents, better conditional execution of the ES|QL queries using multiple FORK commands, support sparse vector queries, removing close duplicate results to enhance result diversity, allowing full text search on runtime generated columns, and many other scenarios.

Additional tutorials and guides:

• ES|QL for search

• ES|QL for search tutorial

• Semantic_text field type

• FORK and FUSE documentation

• ES|QL search functions

However, when you fix this one issue, you accidentally break other queries since you couldn’t test all cases manually. But how can you or your QA team test if a change in one query has a ripple effect in other queries? Or even more importantly, how can you be sure that your changes actually improved a query?

Towards a systematic evaluation

Here is where judgment lists come in useful. Instead of depending on manual and subjective testing any time you make a change, you can define a fixed set of queries that are relevant for your business case, together with their relevant results.

This set becomes your baseline. Every time you implement a change, you use it to evaluate if your search actually improved or not.

The value of this approach is that it:

• Removes uncertainty: you no longer need to wonder if your changes impact other queries; the data will tell you.

• Stops manual testing: once the judgment sets are recorded, the test is automatic.

• Supports changes: You can show clear metrics that support the benefits of a change.

How to start building your judgment list

One of the easiest ways to start is to take a representative query and manually select the relevant documents. There are two ways to do this list:

• Binary Judgments: Each document associated with a query gets a simple tag: relevant (usually with a score of “1”) and not-relevant (“0”).

• Graded Judgments: Here, each document gets a score with different levels. For example: setting a 0 to 4 scale, similar to a Likert scale, where 0 = “not at all relevant” and 4 = “totally relevant,” with variations like “relevant,” “somewhat relevant,” etc.

Binary judgments work well when the search intent has clear limits: Should this document be in the results or not?

Graded judgements are more useful when there are grey areas: some results are better than others, so you can get “very good,” “good,” and “useless” results and use metrics that value the order of the results and the user’s feedback. However, graded scales also introduce drawbacks: different reviewers may use the scoring levels differently, which makes the judgments less consistent. And because graded metrics give more weight to higher scores, even a small change (like rating something a 3 instead of a 4) can create a much bigger shift in the metric than the reviewer intended. This added subjectivity makes graded judgments noisier and harder to manage over time.

Do I need to classify the documents myself?

Not necessarily, since there are different ways to create your judgment list, each with its own advantages and disadvantages:

• Explicit Judgments: Here, SMEs go over each query/document and manually decide if (or how) relevant it is. Though this provides quality and control, it is less scalable.

• Implicit Judgments: With this method, you infer the relevant documents based on real-user behavior like clicks, bounce rate, and purchases, among others. This approach allows you to gather data automatically, but it might be biased. For example, users tend to click top results more often, even if they are not relevant.

• AI-Generated Judgments: This last option uses models (like LLMs) to automatically evaluate queries and documents, often referred to as LLM juries. It’s fast and easy to scale, but the quality of the data depends on the quality of the model you’re using and how well LLM training data aligns with your business interests. As with human grades, LLM juries can introduce their own biases or inconsistencies, so it’s important to validate their output against a smaller set of trusted judgments. LLM models are probabilistic by nature, so it is not uncommon to see an LLM model giving different grades to the same result regardless of setting temperature parameter as 0.

Below are some recommendations to choose the best method for creating your judgment set:

• Decide how critical some features are for you that only users can properly judge (like price, brand, language, style, and product details). If those are critical, you need explicit judgments for at least some part of your judgment list.

• Use implicit judgements when your search engine already has enough traffic so you can use clicks, conversions, and lingering time metrics to detect usage trends. You should still interpret these carefully, contrasting them with your explicit judgement sets to prevent any bias (e.g: users tend to click top-ranked results more often, even if lower-ranked results are more relevant)

To address this, position debiasing techniques adjust or reweight click data to better reflect true user interest. Some approaches include:

• Results shuffling: Change the order of search results for a subset of users to estimate how position affects clicks.

• Click models includeDynamic Bayesian Network DBN, User Browsing Model UBM. These Statistical models estimate the probability of a click reflects real interest rather than just position, using patterns like scrolling, dwell time, click sequence, and returning to the results page.

Example: Movie rating app

Prerequisites

To run this example, you need a running Elasticsearch 8.x cluster, locally or Elastic Cloud (Hosted or Serverless), and access to the REST API or Kibana.

Think about an app in which users can upload their opinions about movies and also search for movies to watch. As the texts are written by users themselves, they can have typos and many variations in terms of expression. So it’s essential that the search engine is able to interpret that diversity and provide helpful results for the users.

To be able to iterate queries without impacting the overall search behavior, the business team in your company created the following binary judgment set, based on the most frequent searches:

Query

DocID

Text

DiCaprio performance

doc1

DiCaprio's performance in The Revenant was breathtaking.

DiCaprio performance

doc2

Inception shows Leonardo DiCaprio in one of his most iconic roles.

DiCaprio performance

doc3

Brad Pitt delivers a solid performance in this crime thriller.

DiCaprio performance

doc4

An action-packed adventure with stunning visual effects.

sad movies that make you cry

doc5

A heartbreaking story of love and loss that made me cry for hours.

sad movies that make you cry

doc6

One of the saddest movies ever made — bring tissues!

sad movies that make you cry

doc7

A lighthearted comedy that will make you laugh

sad movies that make you cry

doc8

A science-fiction epic full of action and excitement.

Creating the index:

BULK request:

Below is the Elasticsearch query the app is using:

From judgment to metrics

By themselves, judgment lists do not provide much information; they are only an expectation of the results from our queries. Where they really shine is when we use them to calculate objective metrics to measure our search performance.

Nowadays, most of the popular metrics include

• Precision: Measures the proportion of results that are truly relevant within all search results.

• Recall: Measures the proportion of relevant results the search engine found among x results.

• Discounted Cumulative Gain (DCG): Measures the quality of the result’s ranking, considering the most relevant results should be at the top.

• Mean Reciprocal Rank (MRR): Measures the position of the first relevant result. The higher it is in the list, the higher its score.

Using the same movie rating app as an example, we’ll calculate the recall metric to see if there’s any information that is being left out of our queries.

In Elasticsearch, we can use the judgment lists to calculate metrics via the Ranking Evaluation API. This API receives as input the judgment list, the query, and the metric you want to evaluate, and returns a value, which is a comparison of the query result with the judgment list.

Let’s run the judgment list for the two queries that we have:

We’ll use two requests to _rank_eval: one for the DiCaprio query and another for sad movies. Each request includes a query and its judgment list (ratings). We don’t need to grade all documents since the ones not included in the ratings are considered as with no judgment. To do the calculations, recall only considers the “relevant set,” the documents that are considered relevant in the rating.

In this case, the DiCaprio query has a recall of 1, while the sad movies got 0. This means that for the first query, we were able to get all relevant results, while in the second query, we did not get any. The average recall is therefore 0.5.

Maybe we’re being too strict with the minimum_should_match parameter since by demanding that 100% of the words in the query are found in the documents, we’re probably leaving relevant results out. Let’s remove the minimum_should_match parameter so that a document is considered relevant if only one word in the query is found in it.

As you can see, by removing the minimum_should_match parameter in one of the two queries, we now get an average recall of 1 in both.

In summary, removing the minimum_should_match: 100% clause, allows us to got a perfect recall for both queries.

We did it! Right?

Not so fast!

By improving recall, we open the door to a wider range of results. However, each adjustment implies a trade-off. This is why defining complete test cases, using different metrics to evaluate changes.

Using judgment lists and metrics prevents you from going in blind when making changes since you now have data to back them up. Validation is no longer manual and repetitive, and you can test your changes in more than just one use case. Additionally, A/B testing allows you to test live which configuration works best for your users and business case, thus coming full circle from technical metrics and real-world metrics.

Final recommendations for using judgment lists

Working with judgment lists is not only about measuring but also about creating a framework that allows you to iterate with confidence. To achieve this, you can follow these recommendations:

1. Start small, but start. You don’t need to have 10,000 queries with 50 judgment lists each. You only need to identify the 5–10 most critical queries for your business case and define which documents you expect to see at the top of the results. This already gives you a base. You typically want to start with the top queries plus the queries with no results. You can also start testing with an easy-to-configure metric like Precision and then work your way up in complexity.

2. Validate with users. Complement the numbers with A/B testing in production. This way, you’ll know if changes that look good in the metrics are also generating a real impact.

3. Keep the list alive. Your business case will evolve, and so will your critical queries. Update your judgment periodically to reflect new needs.

4. Make it part of the flow. Integrate judgment lists into your development pipelines. Make sure each configuration change, synonym, or text analysis is automatically validated against your base list.

5. Connect technical knowledge with strategy. Don’t stop at measuring technical metrics like precision or recall. Use your evaluation results to inform business outcomes.

In this article, we explore how to make Elasticsearch search results more relevant for different e-commerce user segments using an explainable, multiplicative boosting strategy — without any machine learning post-processing.

Introduction: Why personalization matters

Elasticsearch is very good at ranking results by textual relevance (BM25) and by semantic relevance (vectors). In e-commerce, that is necessary but not sufficient. Two people can type the same query and reasonably expect different results:

• A luxury shopper searching for “red lipstick” expects prestige brands near the top.

• A budget shopper wants affordable options promoted.

• A gift buyer may prefer popular bundles.

The goal is to adjust ranking so that, for a given query, products that align with the user’s segment rise modestly in the list, without destroying the underlying relevance. This article shows how to add cohort-aware personalization on top of Elasticsearch’s relevance using only function_score, a keyword field, and small multiplicative boosts.

Multiplicative boosting for cohort personalization

The core challenge in cohort personalization is stability. You want a product that is relevant to the query to remain relevant, with a controlled, explainable uplift when it matches the user’s segment. What often goes wrong is that personalization signals are added to the score in a way that either:

• overwhelms BM25 on some queries, or

• has almost no effect on others.

This happens because most boosting approaches use additive scoring. However, BM25 scales can vary dramatically across queries and datasets, so a fixed additive adjustment (e.g., “add +2.0 for a cohort match”) is sometimes a massive change to the BM25 score, and other times is negligible. Instead, what we want is a guarantee that if a product is a good match for the query, and it aligns with the user’s cohort, then its score is increased by a controlled percentage regardless of the absolute BM25 scale. We can achieve this with a multiplicative pattern:

This article shows how to implement this pattern using Elasticsearch’s function_score query, a cohorts field on the product, and a list of user cohorts passed at query time.

Modeling cohorts in your product catalog

The simplest way to enable cohort-aware ranking is to treat cohorts as tags. For example, a product might carry tags such as:

• Lipstick: ["female", "beauty", "luxury"]

• Men’s deodorant: ["male", "personal_care", "sport"]

• Glitter gloss: ["female", "beauty", "youth", "party"].

A user or session carries a set of tags inferred from behavior and profile:

• High-income female luxury shopper: ["female", "beauty", "luxury"]

• Budget-oriented female shopper: ["female", "beauty", "budget"]

Cohort overlap is the count of shared cohort tags between the user/session and the product. No weighting, no semantic similarity — just a simple intersection. For example, if the user cohorts are [“female”, “beauty”, “budget”] and a lipstick has [“female”, “beauty”, “luxury”], the overlap is 2. If a men’s deodorant has [“male”, “personal_care”, “sport”], the overlap with that same user is 0.

The intuition is that (a) BM25 ranks documents depending on how relevant they are to the user’s query, and (b) cohort overlap boosts products based on how well each product aligns with the user's segment. To accomplish this, we transform the cohort overlap between the user cohorts and the product cohorts into a multiplicative boost that scales BM25.

To avoid field explosion, we keep all cohort tags in a single keyword field, such as follows:

This is easy for merchandisers to understand, avoids hundreds of boolean fields like is_female or is_luxury, and works efficiently with term filters.

Why additive boosts don’t work

One subtle but important point is that even a standard boolean query is additive. When Elasticsearch scores a document, the base BM25 score from the main query (typically in a must) and every matching should clause contributes an additive score. This means “additive boosting” is not just about boosts, it’s fundamental to how boolean scoring works.

Personalization built on additive logic behaves inconsistently because BM25 scales differ per query and dataset. For example the base BM25 scores for three products might be 12, 8, 4 in one instance, and might be 0.12, 0.08, 0.04 after updating your dataset or modifying your query. In this case an additive boost (e.g., +2.0) becomes a dominating force when the base BM25 scores are small (a +2.0 boost on a score of 0.12 is about 18× higher) or a rounding error when the base BM25 scores are large (+2.0 boost on a score of 12 is only about 1.17× higher). This creates inconsistent, unpredictable ranking behavior.

Why multiplicative boosting is the right shape

If we apply a multiplicative boost, the shape is consistent:

With weight_per_cohort = 0.1, an overlap of 2 gives a boost of 1.2 (20% increase), an overlap of 1 gives a boost of 1.1 (10% increase), and an overlap of 0 gives a boost of 1.0 (no change). This means that a product that is more aligned with the user cohort gets a predictable percentage uplift, regardless of whether its BM25 score is 0.01 or 10.0. BM25 remains the primary signal; cohort alignment gently reshapes the ranking.

How function_score gives us multiplicative behavior

To convert cohort overlap into a controlled percentage boost, we need a way to take the normal BM25 score and scale it up by a factor such as 1.1, 1.2, or 1.3. Elasticsearch does not support multiplying a score directly inside a standard query, but function_score provides exactly this capability: it lets us compute an additional score component and combine it with the base score using a chosen strategy, which is "multiply" for this use case.

Elasticsearch’s function_score lets us implement multiplicative cohort boosting in three steps. First, each cohort match contributes a small weight (e.g., 0.1). Second, we include a baseline weight of 1.0 so that the final multiplier never drops below 1. Third, we sum all cohort contributions using score_mode: "sum" to produce a boost factor that represents (1 + overlap × weight). Finally, we combine this boost factor with the BM25 score using boost_mode: "multiply", which gives us the exact multiplicative behavior we want.

The calculation below shows how the final score is calculated, where BM25 is the base relevance; n is the number of matched cohorts; w is weight_per_cohort (e.g., 0.1); and additive baseline = 1.0:

So, with 2 overlapping cohorts and w = 0.1:

This is exactly the multiplicative behavior we want.

Putting it together: index, data, and baseline ranking

Create a simple index:

Index a few products:

A baseline query for “red lipstick” might look like:

This returns a pure BM25 ranking (without any cohort boosting). In this example, the scores of LIP-001 and LIP-002 will be very close (or identical), because they match the same query terms with similar frequencies and have comparable lengths.

The relative ranking is what matters; the exact numeric scores may differ depending on shard configuration, analyzer differences, or Elasticsearch version.

Product ID

Description

BM25 score

LIP-001

Premium cherry red lipstick with velvet finish

0.603535

LIP-002

Affordable matte red lipstick for everyday wear

0.603535

LIP-003

Glitter red gloss for parties and festivals

0.13353139

Persona A: high-income luxury shopper

Suppose we know that Persona A fits into the following cohorts:

We translate that into a set of cohort filters, each with a small weight, plus a baseline factor:

For this persona LIP-001 (“Premium cherry red lipstick with velvet finish”) matches “female”, “beauty”, and “luxury” which means that the cohort overlap is 3 and therefore the boost factor is 1.3. On the other hand, LIP-002 and LIP-003 match “female” and “beauty” which results in a boost factor of 1.2.

Product ID

Description

Base BM25 score

Boost factor

New score

LIP-001

Premium cherry red lipstick with velvet finish

0.603535

1.3x (30%)

0.7845955

LIP-002

Affordable matte red lipstick for everyday wear

0.603535

1.2x (20%)

0.724242

LIP-003

Glitter red gloss for parties and festivals

0.13353139

1.2x (20%)

0.16023767

As desired for this luxury user, the luxury lipstick (LIP-001) receives the strongest uplift and will tend to rise above similar alternatives in the results.

Persona B: budget-oriented shopper

A budget-conscious shopper might belong to the following cohorts:

["female", "beauty", "budget"]

The query for this user is nearly identical to the previous query, except for the cohort values which now reflect “budget” rather than “luxury”:

For this persona LIP-002 (“Affordable matte red lipstick for everyday wear”) matches “female”, “beauty”, and “budget” which means that the cohort overlap is 3 and therefore the boost factor is 1.3. On the other hand, LIP-001 and LIP-003 match “female” and “beauty” which results in a boost factor of 1.2.

Product ID

Description

Base BM25 score

Boost factor

New score

LIP-002

Affordable matte red lipstick for everyday wear

0.603535

1.3x (30%)

0.7845955

LIP-001

Premium cherry red lipstick with velvet finish

0.603535

1.2x (20%)

0.724242

LIP-003

Glitter red gloss for parties and festivals

0.13353139

1.2x (20%)

0.16023767

As desired for this budget user, the budget lipstick (LIP-002) receives the strongest uplift and will tend to rise above similar alternatives in the results.

How to build the cohort filter dynamically (Python example)

You will normally inject the cohort filters at query time based on the user/session profile. For example:

Using term filters on a keyword field is fast, shard-cache friendly, and fully visible in the _explain API, which shows exactly which filters fired and which weights were applied.

How cohort assignment works

Cohort assignment is intentionally left outside Elasticsearch, and is outside the scope of this article. However, sources could include:

• browsing events (“has viewed lipstick” → beauty)

• gender inference (from preferences or marketing profile)

• device characteristics (mobile shopper)

• location (“urban buyer”)

• historical purchases

• marketing segments

• personalization cookies

All of these are input signals, but the scoring mechanism in Elasticsearch remains the same. Elasticsearch does not need to know how you inferred the segments. This separation of concerns keeps Elasticsearch focused on ranking, while your application or data science layer owns the logic for inferring segments.

How to choose the right boost weight

In our examples, we used 0.1 per cohort. This value is tunable. Staying between 0.05 and 0.20 will likely provide good results. You should A/B test weights based on:

• catalog diversity

• number of cohort tags per product

• variability in BM25

• business goals (revenue vs. discovery vs. personalization)

Limit the number of cohorts assigned to each product

Giving a product 20 cohort tags leads to:

• Noise in the signals

• Gaming by merchandisers (“tag everything as luxury”)

• Loss of explainability

• Over-boosting

As a starting point (to be confirmed by your own testing), we recommend:

• Approximately 5 cohorts per product.

• Optionally, an offline validation step (ingest pipeline, CI script, or index-time check) that warns or blocks when more than 5 tags are assigned.

Customized cohort boosting per user

So far, our examples assume every cohort contributes equally. In reality, some users strongly prefer certain segments. In some cases, you might know that certain cohorts are especially important for an individual user. For example:

• A user who almost always buys luxury brands.

• A user who consistently picks budget options.

You can encode this by assigning different weights per cohort instead of a flat 0.1. For example, if your application has detected a “super-luxury” shopper, then you could modify the function scoring as follows:

In the above example matching “female” or “beauty” each add +0.1 while matching luxury adds +0.2. In this example, a product matching all three cohorts would get:

This remains fully explainable, and you can document the configuration (“luxury is 2× as important as other cohorts for this user”). Additionally, the explain API will show exactly how those numbers contributed to the final score.

Conclusion:

This Elasticsearch-native approach to cohort personalization uses only lightweight metadata and standard query constructs, while preserving explainability, stability, and business control over the relevance model. This delivers precise, predictable relevance that ensures the business goals never sacrifice the quality of the search results.

Implementation summary

If you want to adopt this pattern in production, the high-level steps are:

• Add a single keyword field (cohorts) to each product containing 3–5 cohort tags.

• Compute user/session cohorts in your application logic (from browsing, purchase history, CRM, etc.) and pass them with the query.

• Inject dynamic function_score filters into your query with one per user cohort, and each with a small weight (e.g., 0.1), plus a baseline weight (1.0).

• Wrap your existing BM25 query in function_score with score_mode: "sum" and boost_mode: "multiply" to apply multiplicative boosting.

• Tune per-cohort weights (typically 0.05–0.20) based on A/B experiments, ensuring BM25 remains the dominant signal.

These steps let you layer cohort personalization cleanly on top of your existing search relevance, without scripts, ML models, or major architecture changes.

What’s next?

This pattern is a powerful example of how to build sophisticated relevance rules directly into your queries, ensuring speed and reliability.

1. Implement custom personalization faster: If you're ready to deploy and optimize this advanced cohort personalization strategy, or to tackle other complex relevance challenges, our team can help you build, tune, and operationalize your Elasticsearch solution quickly. Contact Elastic Services for help implementing this and other advanced search techniques.

2. Join the discussion: For general questions about advanced relevance techniques and implementation, join the broader Elastic Stack community for search discussions.

With Elasticsearch 8.18 and 9.0, ES|QL adds a host of new functionalities, including:

• support for scoring

• semantic search

• more configuration options for the match function

• a new KQL function

In this blog, we will review these 8.18 features and other exciting new features that we plan to add to ES|QL, reinforcing our investment in making ES|QL a modern search language ready to fit your needs, whether you are building a search application powered by ES|QL or analyzing your data in Kibana Discover.

Introducing scoring

In 8.17 we added the ability to filter documents using full text functions. If you are unfamiliar with full text filtering in ES|QL, we suggest reading our original blog post about it.

With 8.18 and 9.0 we introduce support for scoring, making it possible to return documents in order of their relevance. To access the score for each document, simply add the metadata _score field to your ES|QL query:

We retrieve the same scores we get from the equivalent search API query:

Full text search functions such as match, qstr and kql can only be used in the context of a WHERE condition and are the only ones that contribute to the score.

The _score column can not only be used to sort documents by relevance, but also in custom scoring formulas. In the next example, we keep only the most relevant results using a score threshold and then add a score boost based on the reader rating:

Improving the match function

In ES|QL, the match function simply translates to a Query DSL match query. In 8.18 and 9.0, we expanded the match function's capabilities to include all options that are currently available in Query DSL. It is now possible to set well-known match options such as boost, fuzziness and operator in ES|QL too:

Enter semantic search

The 8.18 release comes with the exciting announcement that semantic search is now generally available. We've expanded the match function to support querying over semantic_text field types.

In ES|QL, executing a semantic query is now as simple as performing a full-text query, as shown in this example:

In this example, we set semantic_title to use the semantic_text field type.

Mapping your index fields as semantic_text is all it takes to set up your index for semantic search.

Check our search with semantic text tutorial for more details.

Hybrid search with ES|QL

ES|QL makes it straightforward to do both semantic and lexical search at the same time. It is also possible to set different boosts, prioritizing results from semantic search or lexical search, depending on your use case:

Transitioning from KQL

If you are a long-term user of Kibana Discover and use KQL (Kibana Query Language) to query and visualize your data and you'd like to try ES|QL but don't know where to start, don't worry, we got you!

In 8.18 and 9.0, ES|QL adds a new function which allows you to use KQL inside ES|QL. This is as simple as:

ES|QL is already available in Kibana Discover.

This way, you get the best of both worlds: you can continue to use KQL and start getting more familiar with ES|QL at your own pace.

Check out our getting started with ES|QL guide for more information.

Beyond 8.18 and 9.0

In future releases, we'll be adding more and more search capabilities to ES|QL, including vector search, semantic reranking, enhanced score customization options, and additional methods for combining hybrid search results, such as Reciprocal Rank Fusion (RRF).

Try it out yourself

These changes are available starting with Elasticsearch 8.18, but they are already available in Elasticsearch Serverless. For Elasticsearch Serverless, start a free trial cloud today or try Elastic on your local machine now!

Follow the Search and filter in ES|QL tutorial for a hands-on introduction to the features described in this blog post! 

Filters vs facets

Before we begin, let's define what filters and facets are. Filters are predefined attributes used to restrict a set of results. In a marketplace, for example, filters are available even before a search is performed. The user can select a category, such as "Video games", before searching for "PS5", refining the search to a more specific subset instead of the entire database. This significantly increases the chances of obtaining more relevant results.

Facets work similarly to filters but are only available after the search is performed. In other words, the search returns results, and based on them, a new list of refinement options is generated. For example, when searching for a PS5 console, facets such as storage capacity, shipping cost, and color may be displayed to help users choose the ideal product.

Now that we have defined filters and facets, let's discuss the impact of the classical and Machine Learning (ML)-based approaches on their implementation and usage. Each method has advantages and challenges that influence search efficiency.

Classical approach to filters and facets

In this approach, filters and facets are manually defined based on predefined rules. This means that the attributes available for refining the search are fixed and planned in advance, considering the catalog structure and user needs.

For example, in a marketplace, categories such as "Electronics" or "Fashion" may have specific filters like brand, format and price range. These rules are created statically, ensuring consistency in the search experience but requiring manual adjustments whenever new products or categories emerge.

Although this approach provides predictability and control over the displayed filters and facets, it can be limited when new trends arise that demand dynamic refinement.

Pros:

• Predictability and control: Since filters and facets are manually defined, management becomes easier.

• Low complexity: No need to train models.

• Ease of maintenance: As rules are predefined, adjustments and corrections can be made quickly.

Cons:

• Reindexing required for new filters: Whenever a new attribute needs to be used as a filter, the entire dataset must be reindexed to ensure that documents contain this information.

• Lack of dynamic adaptation: Filters are static and do not automatically adjust to changes in user behavior.

Implementation of filters/facets – Classical approach

In Dev Tools, Kibana, we will create a demonstration of filters/facets using the classical approach.

First, we define the mapping to structure the index:

The brand and storage fields are set as keyword, allowing them to be used directly in aggregations (facets). The price field is of type float, enabling the creation of price ranges.

In the next step, the product data will be indexed:

Now, let's retrieve classic facets by grouping the results by brand, storage, and price range. In the query, size:0 was defined. In this scenario, the goal is to retrieve only the aggregation results without including the documents corresponding to the query.

The response will include counts for Brand, Storage, and Price, helping to create filters and facets.

Machine learning/AI-based approach to filters and facets

In this approach, Machine Learning (ML) models, including Artificial Intelligence (AI) techniques, analyze data attributes to generate relevant filters and facets. Instead of relying on predefined rules, ML/AI leverages indexed data characteristics. This enables the dynamic discovery of new facets and filters.

Pros:

• Automatic updates: New filters and facets are generated automatically, without the need for manual adjustments.

• Discovery of new attributes: It can identify previously unconsidered data characteristics as filters, enriching the search experience.

• Reduced manual effort: The team does not need to constantly define and update filtering rules as AI learns from available data.

Cons:

• Maintenance complexity: The use of models may require pre-validation to ensure the consistency of the generated filters.

• Requires ML and AI expertise: The solution demands qualified professionals to fine-tune and monitor model performance.

• Risk of irrelevant filters: If the model is not well-calibrated, it may generate facets that are not useful for users.

• Cost: The use of ML and AI may require third-party services, increasing operational costs.

It's worth noting that even with a well-calibrated model and a well-crafted prompt, the generated facets should still go through a review step. This validation can be manual or based on moderation rules, ensuring that the content is appropriate and safe. While not necessarily a drawback, it is an important consideration to ensure the quality and suitability of the facets before they are made available to users.

Implementation of filters/facets – AI approach

In this demonstration, we will use an AI model to automatically analyze product characteristics and suggest relevant attributes. With a well-structured prompt, we extract information from the catalog and transform it into filters and facets. Below, we present each step of the process.

Initially, we will use the Inference API to register an endpoint for integration with an ML service. Below is an example of integration with OpenAI's service.

Now, we define the pipeline to execute the prompt and obtain the new filters generated by the model.

Running a simulation of this pipeline for the "PlayStation 5" product, with the following description:

Stunning Gaming: Marvel at stunning graphics and experience the features of the new PS5.

Breathtaking Immersion: Discover a deeper gaming experience with support for haptic feedback, adaptive triggers, and 3D Audio technology.

Slim Design: With the PS5 Digital Edition, gamers get powerful gaming technology in a sleek, compact design.

1TB of Storage: Have your favorite games ready and waiting for you to play with 1TB of built-in SSD storage.

Backward Compatibility and Game Boost: The PS5 console can play over 4,000 PS4 games. With Game Boost, you can even enjoy faster, smoother frame rates in some of the best PS4 console games.

Let's observe the prompt output generated from this simulation.

Now a new field, dynamic_facets, will be added to the new index to store the facets generated by the AI.

Using the Reindex API, we will reindex the videogames index to videogames_1, applying the generate_filter_ai pipeline during the process. This pipeline will automatically generate dynamic facets during indexing.

Now, we will run a search and get the new filters:

Results:

To symbolize the implementation of the facets, below is a simple front-end:

The UI code presented is here.

Conclusion

Both approaches to creating filters and facets have their benefits and points of concern. The classic approach, based on manual rules, offers control and lower costs but requires constant updates and does not dynamically adapt to new products or features.

On the other hand, the AI ​​and Machine Learning-based approach automates facet extraction, making the search more flexible and allowing the discovery of new attributes without manual intervention. However, this approach can be more complex to implement and maintain, requiring calibration to ensure consistent results.

The choice between the classic and AI-based approaches depends on the needs and complexity of the business. For simpler scenarios, where data attributes are stable and predictable, the classic approach can be more efficient and easier to maintain, avoiding unnecessary costs with infrastructure and AI models. On the other hand, the use of ML/AI to extract facets can add significant value, improving the search experience and making filtering more intelligent.

The important thing is to evaluate whether automation justifies the investment or whether a more traditional solution already meets the business needs effectively.

This ease of use does come with some tradeoffs: we simplified semantic search with semantic_text by making judgments on default behavior that fit the majority of use cases. Unfortunately, this means that some customizations available for traditional vector search queries aren’t present in the semantic query. We didn’t want to add all of these options directly to the semantic query, as that would undermine the simplicity that we strive for. Instead, we expanded the queries that support the semantic_text field, leaving it up to you to choose the best query that meets your needs.

Let’s walk through these changes, starting with creating a simple index with a semantic_text field:

We made match happen in semantic search!

First and most importantly, the match query will now work with semantic_text fields!

This means that you can change your old semantic query:

Into a simple match query:

We can see the benefits of semantic search here because we’re searching for “song lyrics about love”, none of which appears in the indexed document. This is because of ELSER’s text expansion.

But wait, it gets better!

If you have multiple indices, and the same field name is semantic_text in one field and perhaps text in the other field, you can still run match queries against these fields. Let’s create another index, with the same field names, but different types (text instead of semantic_text). Here’s a simple example to illustrate:

Here, searching for “crazy” brings up both the lexical match that has “crazy” in the title, and the semantic lyric “lose my mind.”

There are some caveats to keep in mind when using the match functionality with semantic_text:

• The underlying semantic_text field has a limitation where you can’t use multiple inference IDs on the same field. This limitation extends to match—meaning that if you have two semantic_text fields with the same name, they need to have the same inference ID or you’ll get an error. You can work around this by creating different names and querying them in a boolean query or a compound retriever.

• Depending on what model you use, the scores between lexical (text) matches and semantic matches will likely be very different. In order to get the best ranking of results, we recommend using second stage rerankers such as semantic reranking or RRF.

Semantic search using the match query is also available in ES|QL! Here’s the same example as above, but using ES|QL:

Expert-level semantic search with knn and sparse_vector

Match is great, but sometimes you want to specify more vector search options than the semantic query supports. Remember, the tradeoff of making the semantic query as simple as it is involved making some decisions on default behavior.

This means that if you want to take advantage of some of the more advanced vector search features, perhaps num_candidates or filter from the knn query or token pruning in the sparse_vector query, you won’t be able to do so using the semantic query.

In the past, we provided some workarounds to this, but they were convoluted and required knowing the inner workings and architecture of the semantic_text field and constructing a corresponding nested query. If you’re doing that workaround now, it will still work—however, we now support query DSL using knn or sparse_vector queries on semantic_text fields.

All about that dense (vector), no trouble

Here’s an example script that populates a text_embedding model and queries a semantic_text field using the knn query:

The knn query can be modified with extra options to enable more advanced queries against the semantic_text field. Here, we perform the same query but add a pre-filter against the semantic_text field:

Keepin’ it sparse (vector), keepin’ it real

Similarly, sparse embedding models can be queried more specifically using semantic_text fields as well. Here’s an example script that adds a few more documents and uses the sparse_vector query:

The sparse_vector query can be modified with extra options, to enable more advanced queries against the semantic_text field. Here, we perform the same query but add token pruning against a semantic_text field:

This example significantly decreases the token frequency ratio required to pruning, which helps us show differences with such a small dataset, though they’re probably more aggressive than you’d want to see in production (remember, token pruning is about pruning irrelevant tokens to improve performance, not drastically change recall or relevance). You can see in this example that the Avril Lavigne song is no longer returned, and the scores have changed due to the pruned tokens. (Note that this is an illustrative example, and we still recommend a rescore adding pruned tokens back into scoring for most use cases).

You’ll note that with all of these queries if you’re only querying a semantic_text field, you no longer need to specify the inference ID in knn’s query_vector_builder or in the sparse_vector query. This is because it will be inferred from the semantic_text field. You can specify it if you want to override with a different (compatible!) inference ID for some reason or if you’re searching combined indices that have both semantic_text and sparse_vector or dense_vector fields though.

Try it out yourself

We’re keeping the original semantic query simple, but expanding our semantic search capabilities to power more use cases and seamlessly integrate semantic search with existing workflows. These power-ups are native to Elasticsearch and are already available in Serverless. They’ll be available in stack-hosted Elasticsearch starting with version 8.18.

Try it out today!

This blog post will explain how this new feature can help in improving your document ranking in text search and how to implement it in Elasticsearch.

Whether you are trying to optimize an eCommerce search, build the best context for a Retrieval Augmented Generation(RAG) application or craft a question answering based search on millions of academic papers, you have probably realized how challenging it can be to accurately optimize document ranking in a search engine. That's where Learning to Rank comes in.

Understanding relevance features and how to build a scoring function

Relevance features are the signals to determine how well a document matches a user's query or interest, all of which impact search relevance. These features can vary significantly depending on the context, but they generally fall into several categories. Let’s take a look at some common relevance features used across different domains:

• Text Relevance Scores (e.g., BM25, TF-IDF): Scores derived from text matching algorithms that measure the similarity of document content to the search query. These scores can be obtained from Elasticsearch.

• Document Properties (e.g., price of a product, publication date): Features that can be extracted directly from the stored document.

• Popularity Metrics (e.g., click-through rate, views): Indicators of how popular or frequently accessed a document is. Popularity metrics can be obtained with Search analytics tools, of which Elasticsearch provides out-of-the-box.

The scoring function combines these features to produce a final relevance score for each document. Documents with higher scores are ranked higher in search results.

When using the Elasticsearch Query DSL, you are implicitly writing a scoring function that weights relevance features and ultimately defines your search relevance

Scoring in the Elasticsearch Query DSL

Consider the following example query:

This query translates into the following scoring function:

While this approach works well, it has a few limitations:

• Weights are estimated: The weights assigned to each feature are often based on heuristics or intuition. These guesses may not accurately reflect the true importance of each feature in determining relevance.

• Uniform Weights Across Documents: Manually assigned weights apply uniformly to all documents, ignoring potential interactions between features and how their importance might vary across different queries or document types. For instance, the relevance of recency might be more significant for news articles but less so for academic papers.

As the number of features and documents increases, these limitations become more pronounced, making it increasingly challenging to determine accurate weights. Ultimately, the chosen weights become a compromise, potentially leading to suboptimal ranking in many scenarios.

A compelling alternative is to replace the scoring function that uses manual weights by a ML-based model that computes the score using relevance features.

Hello Learning To Rank (LTR)!

LambdaMART is a popular and effective LTR technique that uses gradient boosting decision trees (GBDT) to learn the optimal scoring function from a judgment list.

The judgment list is a dataset that contains pairs of queries and documents, along with their corresponding relevance labels or grades. Relevance labels are typically either binary, (e.g. relevant/irrelevant) or graded (e.g between 0 for completely irrelevant and 4 for highly relevant). Judgment lists can be created manually by humans or be generated from user engagement data, such as clicks or conversions.

The example below uses a graded relevance judgment.

LambdaMART treats the ranking problem as a regression task using a decision tree where the inner nodes of the tree are conditions over the relevance features, and the leaves are the predicted scores.

LambdaMART uses a gradient boosted tree approach, and in the training process it builds multiple decision trees where each tree corrects errors of its predecessors. This process aims to optimize a ranking metric like NDCG, based on examples from the judgment list. The final model is a weighted sum of individual trees.

XGBoost is a well known library that provides an implementation of LambdaMART, making it a popular choice to implement ranking based on gradient boosting decision trees.

Getting started with LTR in Elasticsearch

Starting with version 8.13, Learning To Rank is integrated directly into Elasticsearch and associated tooling as a technical preview feature.

Train and deploy an LTR model to Elasticsearch

Eland is our Python client and toolkit for DataFrames and machine learning in Elasticsearch. Eland is compatible with most of the standard Python data science tools like Pandas, scikit-learn and XGBoost.

We highly recommend using it to train and deploy your LTR XGBoost model, as it provides features to simplify this process:

1. The first step of the training process is to define the relevant features of the LTR model. Using the Python code below, you can specify the relevant features using the Elasticsearch Query DSL.

1. The second step of the process is to build your training dataset. At this step you will compute and add relevance features for each rows of your judgment list:

To help you with this task, Eland provides the FeatureLogger class:

1. When the training dataset is built, the model is trained very easily (as also shown in the notebook):

1. Deploy your model to Elasticsearch once the training process is complete:

To learn more about how our tooling can help you to train and deploy the model, check out this end-to-end notebook.

Use your LTR model as a rescorer in Elasticsearch

Once you deploy your model in Elasticsearch, you can enhance your search results through a rescorer. The rescorer allows you to refine a first-pass ranking of search results using the more sophisticated scoring provided by your LTR model:

In this example:

• First-pass query: The multi_match query retrieves documents that match the query the quick brown fox in the title and content fields. This query is designed to be fast and capture a large set of potentially relevant documents.

• Rescore phase: The learning_to_rank rescorer refines the top results from the first-pass query using the LTR model.

  • model_id: Specifies the ID of the deployed LTR model (ltr-model-xgboost in our example).

  • params: Provides any parameters required by the LTR model to extract features relevant to the query. Here query_text allows you to specify the query issued by the user that some of our features extractors expect.

  • window_size: Defines the number of top documents from the search results issued by the first-pass query to be rescored. In this example, the top 100 documents will be rescored.

By integrating LTR as a two stage retrieval process, you can can optimize both performance and accuracy of your retrieval process by combining:

• Speed of Traditional Search: The first-pass query retrieves a large number of documents with a broad match very quickly, ensuring fast response times.

• Precision of Machine Learning Models: The LTR model is applied only to the top results, refining their ranking to ensure optimal relevance. This targeted application of the model enhances precision without compromising overall performance.

Try LTR yourself!?

Whether you are struggling to configure search relevance for an eCommerce platform, aiming to improve the context relevance of your RAG application, or you are simply curious about enhancing your existing search engine's performance, you should consider LTR seriously.

To start your journey with implementing LTR, make sure to visit our notebook detailing how to train, deploy, and use an LTR model in Elasticsearch and to read our documentation. Let us know if you built anything based on this blog post or if you have questions on our Discuss forums and the community Slack channel.