Alexander Dávila - Elasticsearch Labs

Training LTR models in Elasticsearch with judgement lists based on user behavior data

Alexander Dávila — Wed, 15 Oct 2025 00:00:00 GMT

A big challenge when using Learning-to-rank models is to create a high-quality judgment list to train the model on. Traditionally, this process involves a manual evaluation of query-document relevance to assign a grade to each one. This is a slow process that does not scale well and is hard to maintain (imagine having to update a list with hundreds of entries by hand).

Now, what if we could use real user interactions with our search application to create this training data? Using UBI data lets us do just that. Creating an automatic system that can capture and use our searches, clicks, and other interactions to generate a judgment list. This process can scale and be repeated far more easily than a manual interaction and would tend to yield better results. In this blog, we will explore how we can query UBI data stored in Elasticsearch to calculate meaningful signals to generate a training dataset for an LTR model.

You can find the full experiment here.

Why UBI data can be useful to train your LTR model

UBI data offers several advantages over a manual annotation:

Volume: Given that UBI data comes from real interactions, we can collect much more data than we can generate manually. This is assuming we have enough traffic to generate this data, of course.
Real User intent: Traditionally, a manual judgment list comes from an expert evaluation of the available data. On the other hand, UBI data reflects real user behavior. This means we can generate better training data that will improve our search system's accuracy, because it's based on how users actually interact with and find value in your content rather than theoretical assumptions about what should be relevant.
Continuous updates: Judgment lists need to be refreshed over time. If we create them from UBI data, we can have current data that results in updated judgment lists.
Cost effectiveness: Without the overhead of manually creating a judgment list, the process can be repeated efficiently any number of times.
Natural query distribution: UBI data represent real user queries, which can drive deeper changes. For example, do our users use natural language to search in our system? If so, we might want to implement a semantic search or hybrid search approach.

It does come with some warnings, though:

Bias amplification: Popular content is more likely to receive clicks, just because it gets more exposure. So this might end up amplifying popular items and possibly drowning out better options.
Incomplete coverage: New content lacks any interactions, so it might be difficult for it to be high in the results. Rare queries can also lack sufficient data points to create meaningful training data.
Seasonal variations: If you expect user behaviour to change drastically over time, historical data might not tell you much about what is a good result.
Task ambiguity: A click doesn’t always guarantee that the user found what they were looking for.

Grades calculation

Grades for LTR training

To train LTR models, we need to provide some numerical representation of how relevant a document is for a query. In our implementation, this number is a continuous score going from 0.0 to 5.0+, where higher scores indicate higher relevance.

To show how this grading system works, consider this manually created example:

Query

Document content

Grade

Explanation

"best pizza recipe"

"Authentic Italian Pizza Dough Recipe with Step-by-Step Photos"

4.0

Highly relevant, exactly what the user is looking for

"best pizza recipe"

"History of Pizza in Italy"

1.0

Somewhat in topic, it is about pizza but is not a recipe

"best pizza recipe"

"Quick 15-Minute Pizza Recipe for Beginners"

3.0

Relevant, a good result but it maybe misses the mark on being the “best” recipe.

"best pizza recipe"

"Car Maintenance Guide"

0.0

Not relevant at all, completely unrelated to the query

As we can see here, the grade is a numerical representation of how relevant a document is to our sample query of “best pizza recipe”. With these scores, our LTR model can learn which documents should be presented higher in the results.

How to calculate the grades is the core of our training dataset. There are multiple approaches to do this, each with its own strengths and weaknesses. For example, we could assign a binary score of 1 for relevant 0 for not relevant or we could just count the number of clicks in a resulting document for each query.

In this blog post, we will be using a different approach, taking into account the user behavior as our input and calculating a grade number as the output. We will also be correcting bias that could occur from the fact that higher results tend to be more clicked, regardless of the relevancy of the document.

Calculating the grades - COEC algorithm

The COEC (Clicks over Expected Clicks) algorithm is a methodology for calculating judgment grades from user clicks. As we stated earlier, users tend to click on higher-positioned results even if the document is not the most relevant to the query; this is called Position Bias. The core idea for using the COEC algorithm is that not all clicks are equally significant; a click on a document at position 10 indicates that the document is much more relevant to the query than a click on a document at position 1. To quote the research paper about the COEC algorithm (linked above):

“It is well known that the click-through rate (CTR) of search results or advertisements decreases significantly depending on the position of the results.”

You can further read about position bias here.

To address this with the COEC algorithm, we follow these steps:

1. Establish position baselines: We calculate the click-through rate (CTR) for each search position from 1 to 10. This means we determine what percentage of users typically click on position 1, position 2, and so on. This step captures the users’ natural position bias. We calculate the CTR using:Where:

= Position. From 1 to 10

= Total clicks (on any document) at position p across all queries

= Total impressions: How many times any document appeared at the position p across all queries

Here, we expect higher positions to get more clicks.

2. Calculate Expected Clicks (EC):

This metric establishes how many clicks a document “should” have received based on the positions it appeared in and the CTR for those positions We calculate EC using:Where:

= All queries where the document d appeared

= Position of the document d in the query q results

3. Count actual clicks: We count the actual total clicks a document received across all queries where it appeared, hereafter called A(d).

4. Compute the COEC score: This is the ratio of Actual clicks (A(d)) over the Expected clicks (EC(d)):This metric normalizes for position bias like this:

A score of 1.0 means the document performed exactly as expected given the positions it appeared in.
A score above 1.0 means the document performed better than expected by looking at its positions. So this document is more relevant for the query.
A score under 1.0 means the document performed worse than expected by looking at its positions. So this document is less relevant for the query.

The end result is a grade number that captures what users are looking for, taking into account position-based expectations extracted from real interactions with our search system.

Technical implementation

We will be creating a script to create a judgment list to train an LTR model.

The input for this script is the UBI data indexed in Elastic (queries and events).

The output is a judgment list in a CSV file generated from these UBI documents using the COEC algorithm. This judgment list can be used with Eland to extract relevant features and train an LTR model.

Quick start

To generate a judgment list from the sample data in this blog, you can follow these steps:

1. Clone the repository:

git clone https://github.com/Alex1795/elastic-ltr-judgement_list-blog.git cd elastic-ltr-judgement_list-blog

2. Install required libraries

For this script, we need the following libraries:

pandas: to save the judgment list
elasticsearch: To get the UBI data from our Elastic deployment

We also need Python 3.11

pip install -r requirements.txt

3. Update the environment variables for your Elastic deployment in a .env file

ES_HOST
API_KEY

To add the environment variables, use:

source .env

4. Create the ubi_queries, ubi_events indices, and upload the sample data. Run the setup.py file:

python setup.py

5. Run the Python script:

python judgement_list-generator.py

If you follow these steps, you should see a new file called judgment_list.csv that looks like this:

This script calculates the grades applying the COEC algorithm discussed before using the calculate_relevance_grade() function that is shown below.

Data architecture

Ubi queries

Our UBI queries index has information about the queries executed in our search system. This is a sample document:

{ "client_id": "client_002", "query": "italian pasta recipes", "query_attributes": { "search_type": "recipe", "category": "food", "cuisine": "italian" }, "query_id": "q002", "query_response_id": "qr002", "query_response_object_ids": [ "doc_011", "doc_012", "doc_013", "doc_014", "doc_015", "doc_016", "doc_017", "doc_018", "doc_019", "doc_020" ], "timestamp": "2024-08-14T11:15:00Z", "user_query": "italian pasta recipes" }

Here we can see data from the user (client_id), from the results of the query (query_response_object_ids), and the query itself (timestamp, user_query)

Ubi click events

Our ubi_events index has data from each time a user clicked a document in the results. This is a sample document:

{ "action_name": "click", "application": "recipe_search", "client_id": "client_001", "event_attributes": { "object": { "description": "Authentic Italian Pizza Dough Recipe with Step-by-Step Photos", "device": "desktop", "object_id": "doc_001", "position": { "ordinal": 1, "page_depth": 1 }, "user": { "city": "New York", "country": "USA", "ip": "192.168.1.100", "location": { "lat": 40.7128, "lon": -74.006 }, "region": "NY" } } }, "message": "User clicked on document doc_001", "message_type": "click", "query_id": "q001", "timestamp": "2024-08-14T10:31:00Z", "user_query": "best pizza recipe" }

Judgment list generation script

General script overview

This script automates the generation of the judgment list using UBI data from Queries and Click events stored in Elasticsearch. It executes these tasks:

Fetches and processes the UBI data in Elasticsearch.
Correlates UBI events with its queries.
Calculates the CTR for each position.
Calculates the expected clicks (EC) for each document.
Counts the actual clicks for each document.
Calculates the COEC score for each query-document pair.
Generates a judgment list and writes it in a CSV file.

Let’s go over each function:

connect_to_elasticsearch()

def connect_to_elasticsearch(host, api_key): """Create and return Elasticsearch client""" try: es = Elasticsearch( hosts=[host], api_key=api_key, request_timeout=60 ) # Test the connection if es.ping(): print(f"✓ Successfully connected to Elasticsearch at {host}") return es else: print("✗ Failed to connect to Elasticsearch") return None except Exception as e: print(f"✗ Error connecting to Elasticsearch: {e}") return None

This function returns an Elasticsearch client object using the host and api key.

fetch_ubi_data()

def fetch_ubi_data(es_client: Elasticsearch, queries_index: str, events_index: str, size: int = 10000) -> Tuple[List[Dict], List[Dict]]: """ Fetch UBI queries and events data from Elasticsearch indices. Args: es_client: Elasticsearch client queries_index: Name of the UBI queries index events_index: Name of the UBI events index size: Maximum number of documents to fetch Returns: Tuple of (queries_data, events_data) """ logger.info(f"Fetching data from {queries_index} and {events_index}") # Fetch queries with error handling try: queries_response = es_client.search( index=queries_index, body={ "query": {"match_all": {}}, "size": size } ) queries_data = [hit['_source'] for hit in queries_response['hits']['hits']] logger.info(f"Fetched {len(queries_data)} queries") except Exception as e: logger.error(f"Error fetching queries from {queries_index}: {e}") raise # Fetch events (only click events for now) with error handling try: events_response = es_client.search( index=events_index, body={ "query": { "term": {"message_type.keyword": "CLICK_THROUGH"} }, "size": size } ) events_data = [hit['_source'] for hit in events_response['hits']['hits']] logger.info(f"Fetched {len(events_data)} click events") except Exception as e: logger.error(f"Error fetching events from {events_index}: {e}") raise logger.info(f"Data fetch completed successfully - Queries: {len(queries_data)}, Events: {len(events_data)}") return queries_data, events_data

This function is the data extraction layer; it connects with Elasticsearch to fetch UBI queries using a match_all query and filters UBI events to get ‘CLICK_THROUGH’ events only.

process_ubi_data()

def process_ubi_data(queries_data: List[Dict], events_data: List[Dict]) -> pd.DataFrame: """ Process UBI data and generate judgment list. Args: queries_data: List of query documents from UBI queries index events_data: List of event documents from UBI events index Returns: DataFrame with judgment list (qid, docid, grade, keywords) """ logger.info("Processing UBI data to generate judgment list") # Group events by query_id clicks_by_query = {} for event in events_data: query_id = event['query_id'] if query_id not in clicks_by_query: clicks_by_query[query_id] = {} # Extract clicked document info object_id = event['event_attributes']['object']['object_id'] position = event['event_attributes']['object']['position']['ordinal'] clicks_by_query[query_id][object_id] = { 'position': position, 'timestamp': event['timestamp'] } judgment_list = [] # Process each query for query in queries_data: query_id = query['query_id'] user_query = query['user_query'] document_ids = query['query_response_object_ids'] # Get clicks for this query query_clicks = clicks_by_query.get(query_id, {}) # Generate judgment for each document shown for doc_id in document_ids: grade = calculate_relevance_grade(doc_id, query_clicks, document_ids, queries_data, events_data) judgment_list.append({ 'qid': query_id, 'docid': doc_id, 'grade': grade, 'query': user_query }) df = pd.DataFrame(judgment_list) logger.info(f"Generated {len(df)} judgment entries for {df['qid'].nunique()} unique queries") return df

This function handles the judgment list generation. It starts processing the UBI data by associating UBI events and queries. Then it calls the calculate_relevance_grade() function for each document-query pair to obtain the entries for the judgment list. Finally, it returns the resulting list as a pandas dataframe.

calculate_relevance_grade()

def calculate_relevance_grade(document_id: str, clicks_data: Dict, query_response_ids: List[str], all_queries_data: List[Dict] = None, all_events_data: List[Dict] = None) -> float: """ Calculate COEC (Click Over Expected Clicks) relevance score for a document. Args: document_id: ID of the document clicks_data: Dictionary of clicked documents with their positions for current query query_response_ids: List of document IDs shown in search results (ordered by position) all_queries_data: All queries data for calculating position CTR averages all_events_data: All events data for calculating position CTR averages Returns: COEC relevance score (continuous value, typically 0.0 to 5.0+) """ # If no global data provided, fall back to simple position-based grading if all_queries_data is None or all_events_data is None: logger.warning("No global data provided, falling back to position-based grading") # Simple fallback logic if document_id in clicks_data: position = clicks_data[document_id]['position'] if position > 3: return 4.0 elif position >= 1 and position <= 3: return 3.0 if document_id in query_response_ids: position = query_response_ids.index(document_id) + 1 if position <= 5: return 2.0 elif position >= 6 and position <= 10: return 1.0 return 0.0 # Calculate rank-aggregated click-through rates position_ctr_averages = {} position_impression_counts = {} position_click_counts = {} # Initialize counters for pos in range(1, 11): # Positions 1-10 position_impression_counts[pos] = 0 position_click_counts[pos] = 0 # Count impressions (every document shown contributes) for query in all_queries_data: for i, doc_id in enumerate(query['query_response_object_ids'][:10]): # Top 10 positions position = i + 1 position_impression_counts[position] += 1 # Count clicks by position for event in all_events_data: if event.get('action_name') == 'click': position = event['event_attributes']['object']['position']['ordinal'] if position <= 10: position_click_counts[position] += 1 # Calculate average CTR per position for pos in range(1, 11): if position_impression_counts[pos] > 0: position_ctr_averages[pos] = position_click_counts[pos] / position_impression_counts[pos] else: position_ctr_averages[pos] = 0.0 # Calculate expected clicks for this specific document expected_clicks = 0.0 # Count how many times this document appeared at each position for any query for query in all_queries_data: if document_id in query['query_response_object_ids']: position = query['query_response_object_ids'].index(document_id) + 1 if position <= 10: expected_clicks += position_ctr_averages[position] # Count total actual clicks for this document across all queries actual_clicks = 0 for event in all_events_data: if (event.get('action_name') == 'click' and event['event_attributes']['object']['object_id'] == document_id): actual_clicks += 1 # Calculate COEC score if expected_clicks > 0: coec_score = actual_clicks / expected_clicks else: coec_score = 0.0 logger.debug( f"Document {document_id}: {actual_clicks} clicks / {expected_clicks:.3f} expected = {coec_score:.3f} COEC") return coec_score

This is the function that implements the COEC algorithm. It calculates the CTR for each position, then it compares the actual clicks for a document-query pair, and finally calculates the actual COEC score for each one.

generate_judgment_statistics()

def generate_judgment_statistics(df: pd.DataFrame) -> Dict: """Generate statistics about the judgment list.""" stats = { 'total_judgments': len(df), 'unique_queries': df['qid'].nunique(), 'unique_documents': df['docid'].nunique(), 'grade_distribution': df['grade'].value_counts().to_dict(), 'avg_judgments_per_query': len(df) / df['qid'].nunique() if df['qid'].nunique() > 0 else 0, 'queries_with_clicks': len(df[df['grade'] > 1]['qid'].unique()), 'click_through_rate': len(df[df['grade'] > 1]) / len(df) if len(df) > 0 else 0 } return stats

It generates useful statistics from the judgment list, such as total queries, total unique documents, or the grade distribution. This is purely informational and does not change the resulting judgment list.

Results and impact

If you follow the instructions in the Quick start section, you should see a resulting CSV file containing a judgment list with 320 entries (you can see a sample output in the repo). With these fields:

qid: unique ID of the query
docid: unique identifier for a resulting document
grade: the calculated grade for the query-document pair
query: The user query

Let’s look at the results for the query “Italian recipes”:

qid

docid

grade

query

q1-italian-recipes

recipe_pasta_basics

0.0

Italian recipes

q1-italian-recipes

recipe_pizza_margherita

3.333333

Italian recipes

q1-italian-recipes

recipe_risotto_guide

10.0

Italian recipes

q1-italian-recipes

recipe_french_croissant

0.0

Italian recipes

q1-italian-recipes

recipe_spanish_paella

0.0

Italian recipes

q1-italian-recipes

recipe_greek_moussaka

1.875

Italian recipes

We can see from the results that for the query “Italian recipes”:

The risotto recipe is definitely the best result for the query, receiving 10 times more clicks than expected
Pizza Margherita is a great result too.
The Greek mousaka (surprisingly) is a good result as well and performs better than its position on the results would suggest. This means a few users looking for Italian recipes got interested in this recipe instead. Maybe these users are interested in Mediterranean dishes in general. At the end, what this tells us is that this could be a good result to be shown under the other two ‘better’ matches we discussed above.

Conclusion

Using UBI data lets us automate the training of LTR models, creating high-quality judgment lists from our own users. UBI data provides a big dataset that reflects how our search system is being used.By using the COEC algorithm to generate the grades, we account for inherent bias while at the same time, it reflects what a user considers a better result. The method outlined here can be applied to real use cases to provide a better search experience that evolves with real usage trends.

Elasticsearch plugin for UBI: Analyze user data in Kibana

Eduard Martin,Alexander Dávila — Fri, 26 Sep 2025 00:00:00 GMT

In this article, we’ll show you how to capture and analyze user analytics data using the UBI (User Behavior Insights) standard in Elasticsearch.

You can learn more about UBI in this article.

Data collected with the UBI collector can be used on Kibana to build dashboards that open the window to users’ behavior in our application. In this blog, we will explore how to analyze UBI data in Kibana to gain insights into how our app is being used.

Demo set up

We can easily reproduce the demo in this blog following these steps:

1. Clone the repository

git clone https://github.com/Alex1795/ubi-dashboard-elasticsearch_blog.git cd ubi-dashboard-elasticsearch_blog

2. Install required libraries:

pip install -r requirements.txt

3. Run the setup script. Make sure to have the following environment variables set beforehand

ES_HOST
API_KEY
KIBANA_HOST

python setup.py

That’s all you need to do. If everything went well, you should see this output from the script execution:

As we can see the script:

Created two indices with the appropriate mappings
Indexed 23 documents to these indices
Uploaded some saved objects to Kibana

Now, let’s take a look at what exactly this script did behind the scenes.

Understanding the uploaded data

First, we put some data in Elasticsearch before creating our visualizations.

You can reproduce the process manually in Kibana DevTools, copying the mappings and sample data and using the PUT and PUT _bulk APIs, respectively.

Ubi_events index

User action data, documents are generated for every click (in this case), and it includes:

application: The client application that generated the event ("search-ui")
action_name: Type of user action performed ("click")
query_id: Links this event to the corresponding search query session
client_id: A generated, unique ID that represents a user or session without revealing personal data. It is generated instead of using identifiable data like email addresses or usernames. This approach allows us to have privacy advantages such as safe analytics capabilities and secure data sharing without exposing PII, while still having important functionality like session continuity, behavioral analysis, or A/B testing.
timestamp: ISO 8601 formatted timestamp when the event occurred
message_type: Category of the event for processing ("CLICK_THROUGH")
message: Human-readable description of what happened ("Clicked Fahrenheit 451")
user_query: The original search term that led to this event ("fahrenheit")
event_attributes: Nested object containing detailed event context:
- object.device: Device type used by the user ("mobile")
- object.object_id: Unique identifier of the clicked item
- object.description: Details about the clicked item (book title, date, author)
- object.position.ordinal: Ranking position of the item in search results (1st)
- object.position.page_depth: Which page of results the item appeared on (1st page)
- object.user.ip: User's IP address
- object.user.city/region/country: Geographic location data
- object.user.location: Precise latitude/longitude coordinates

Sample document:

{ "application": "search-ui", "action_name": "click", "query_id": "2dd48446-7ca8-4510-89f4-2ebb67ed240b", "client_id": "8c1915fe-8ee0-4487-b801-3b1d67c25cf6", "timestamp": "2025-07-30T14:25:52.698Z", "message_type": "CLICK_THROUGH", "message": "Clicked Fahrenheit 451", "user_query": "fahrenheit", "event_attributes": { "object": { "device": "mobile", "object_id": "ZwoTM5gBPJ218VOaBpj4", "description": "Fahrenheit 451(1953-10-15) by Ray Bradbury", "position": { "ordinal": 1, "page_depth": 1 }, "user": { "ip": "192.168.1.100", "city": "New York", "region": "New York", "country": "United States", "location": { "lat": 40.7128, "lon": -74.006 } } } } }

You can download the index mappings here

Ubi_queries index

Search data includes data relevant to each search executed by the users:

query_response_id: Unique identifier for this specific query response instance
user_query: The original search term entered by the user ("fahrenheit")
query_id: Unique identifier for the search query session
query_response_object_ids: Array of object IDs that were returned as search results (["3", "9"])
query: The complete Elasticsearch query object in JSON format, including search parameters, fields to search, result size, sorting, and metadata
client_id: A generated unique ID that represents a user or session without revealing personal data. It is generated instead of using identifiable data like email addresses or usernames. This approach allows us to have privacy advantages such as safe analytics capabilities and secure data sharing without exposing PII, while still having important functionality like session continuity, behavioral analysis, or A/B testing.
timestamp: Unix timestamp in milliseconds when the query was executed (1753885225098)

Sample document:

{ "query_response_id": "03e8af3e-8725-49d9-99ad-36bf2a8e96d1", "user_query": "fahrenheit", "query_id": "f8b2f5bc-cb3c-49d4-86bc-19212a782ba7", "query_response_object_ids": [ "3", "9" ], "query": """{"from":0,"size":20,"query":{"multi_match":{"query":"fahrenheit","fields":["author^1.0","name^1.0"]}},"_source":{"includes":["name","author","image_url","url","price","release_date"],"excludes":[]},"sort":[{"_score":{"order":"desc"}}],"ext":{"query_id":"f8b2f5bc-cb3c-49d4-86bc-19212a782ba7","user_query":"fahrenheit","client_id":"8c1915fe-8ee0-4487-b801-3b1d67c25cf6","object_id_field":null,"query_attributes":{}}}""", "client_id": "8c1915fe-8ee0-4487-b801-3b1d67c25cf6", "timestamp": 1753885225098 }

You can download the index mappings here.

Sample data

We can use the _bulk API to index some sample data in both indices

This will create 6 documents in the ubi_queries index and 16 in the ubi_events index.

Dashboard object

Before going into details of the visualizations used in this example here, you can download the Saved Object of the full example dashboard and import it into your Kibana instance. This dashboard explores the most searched terms, when searches and events took place, and where they come from (in a map).

Visualize Insights

We are going to create a Kibana dashboard to analyze the most common metrics leveraging Kibana Lens. For a reference on available visualizations, visit this page.

Ubi_events

We will start with some simple Metric visualizations created with Lens: Total events: Counts how many events were triggered in the timeframe. Uses a simple count of the documents in the index, denoted by # Records in the field list.

Event actions: Counts actions by action_name. This is a simple count of documents split by action_name.keyword. In our sample data, we have two types of actions:

click: Generated when a user clicks in the book link
search_input: Generated when a user enters text in the search box (debounce 300ms)

Now on table visualizations:

Top clicks: A table with a count of the number of events split by the query they come from. It uses a Top values function on the user_query.keyword. This can give us visibility on which queries generate more interactions on our webpage.

Finally, some other visualizations:

Device types: This visualization breaks down the percentage of events by the device they come from. The device can be one of three categories: Desktop, mobile, or tablet. This visualization is a pie that uses the top values of event_attributes.object.device.keyword, and can give us insights into which type of devices our users have. This can generate alerts if we detect an unexpected, sudden fall of events on a specific type of device, as this might indicate that a recent change in our app resulted in errors when accessing it from a device.

Events map: A map visualization that shows where the events are coming from, which allows us to see the geographical distribution of our users. Right now, this shows where individual documents come from, but this can also be used to see the density of users with a heatmap, for example.

This particular visualization can provide very interesting insights when used with different filters. For example, we can see where different search terms are coming from or where most of our clicks are originating. This can be useful information for making decisions on localization efforts or establishing differences across local markets. The map uses the location at event_attributes.object.user.location.

UBI Events: A saved search with the latest UBI events documents

Ubi_queries

Here we have visualizations from this index:

Total queries: A simple document count of the index to show how many queries have been received in total. This shows the big picture and answers the question of how many total queries we had in the selected time window.

Unique clients: A unique_count of the field client_id to show how many different clients have used our website.

Top queries (tag cloud): A Tag cloud of the top 5 most searched terms. This visualization uses the field user_query.keyword and allows us to easily see the main terms that our users are looking for.

Queries over time: A line chart of queries per hour, which uses a simple count metric in a horizontal axis of the field timestamp

Query terms over time: Similar to the last one, but broken down by the user_query.keyword. This chart can show how many different terms are searched over time.

Top queries: A Top values table showing how many times a term was searched. It uses the user_query.keyword field.

Client queries: A Top values table of the client_id field that counts the total queries and unique queries per client.

No result queries: A Top value table that shows the top query_terms that didn’t match any document, and a Unique Count of the field client_id. This can be very useful to determine what products our website is lacking. For example, in an e-commerce book store, seeing regular searches for a particular book title could lead us to buy copies to sell. Alternatively, it can also indicate shortcomings in our search implementation, for example, if people are using question-based searches that align better with semantic search approaches.

Here you can see the full dashboard:

Analysis of sample data

In our dashboard, we can get some insights:

Traffic is coming from 3 different cities in the US
Most of our users access our website from a desktop device, but we have a sizable number of users using a mobile device and even some using a tablet.
We can see the top query is “asimov,” but at the same time, we do not have any results. This might be a good indicator of what products should be prioritized for stock acquisition.

To further this analysis, we could use Kibana’s Machine Learning capabilities to understand and predict behaviours on our website. Going even one step further, we can create alerts based on these behaviors using the different available connectors.

From a search relevance perspective, user behavior is a useful input for relevance engineering tools like LTR.

Conclusion

Data collected by the UBI collector can be easily used to have a better understanding of our users. The resulting dashboard becomes a live pulse of what our users are searching for and can point to data gaps to drive improvements in our search engine.

Note: The o19s User Behavior Insights (UBI) plugin mentioned in this article is a third-party, community-maintained plugin and is not officially supported by Elastic. For questions or issues related to this plugin, please refer to the o19s UBI project repository at https://github.com/o19s/ubi.

RAG with a map: Multimodal + geospatial in Elasticsearch

Alexander Dávila — Wed, 10 Sep 2025 00:00:00 GMT

When working with RAG systems, Elasticsearch offers a significant advantage by combining a hybrid search (vector search + traditional text search) approach with hard filters to ensure the retrieved data is relevant to the user query. This makes models less prone to hallucination and, in general, improves your system quality. In this blog, we will explore how we can take a multimodal RAG system to the next level using Elastic’s geospatial search features.

Getting started

You can find the full source code used in this blog here.

Prerequisites

Elasticsearch 8.0.0+
Ollama
- cogito:3b model
Python 3.8+
Python dependencies:
- elasticsearch
- elasticsearch-dsl
- ollama
- clip_processor
  - torch
  - transformers
  - PIL
- streamlit
- json
- os
- typing

Setup

1. Clone the repository:

git clone https://github.com/Alex1795/multimodal_RAG_elasticsearch.git cd multimodal_RAG_elasticsearch

2. Install required libraries:

pip install -r requirements.txt

3. Install and set up Ollama:

Download from https://ollama.com/download/# Download and start the required model ollama pull cogito:3b ollama run cogito:3b

4. Configure Elasticsearch

Make sure to have the following environment variables set:
- ES_INDEX
- ES_HOST
- ES_API_KEY
Set the index mapping on Elasticsearch, put special attention to the geolocation and embeddings definition:

PUT mmrag_blog { "mappings": { "properties": { "title": { "type": "text", "analyzer": "standard" }, "geolocation": { "type": "geo_point" }, "image_filename": { "type": "keyword" }, "generated_description": { "type": "text", "analyzer": "standard" }, "description": { "type": "text", "analyzer": "standard" }, "text_embedding": { "type": "dense_vector", "dims": 512, "index": true, "similarity": "cosine" }, "image_embedding": { "type": "dense_vector", "dims": 512, "index": true, "similarity": "cosine" }, "photo_id": { "type": "keyword" } } } }

Run the application

1. Generate and index images’ embeddings and metadata:

python upload_documents.py

This file runs the data indexing pipeline. It processes the image metadata files and enriches them with multimodal embeddings (from the description and the image itself using the CLIP model). Finally, it uploads the documents to Elasticsearch. After executing this command, you should see the mmrag_blog index in Elasticsearch with the image metadata, geolocation, and image and text embeddings.

2. Run the streamlit app and use the UI in your browser with:

streamlit run streamlit_app.py #comment

After executing this command, you can see the project webpage at http://localhost:8501.

The webpage is the interface for the RAG application. From there, you can ask a question, and then the assistant will extract the appropriate parameters from your question, run an RRF search on Elasticsearch to find related pictures, and formulate a response. It will also show some pictures from the results.

Implementation overview

To demonstrate Elastic’s RAG capabilities, we will build an assistant that can answer questions about national parks using relevant data. The search combines 4 approaches with data inferred from the user’s text query:

Image vector search
Text vector search
Lexical text search
Geospatial filtering

This allows our assistant to answer questions that are relevant and focused on what the user needs.

Now, how can the geospatial filter improve the assistant results? For example, if the user asks, “Where can I find canyons near Salt Lake City?” Without a geospatial filter, the assistant might suggest:

Canyonlands National Park - Utah
Grand Canyon National Park - Arizona

However, since we know the user is specifically looking for sites near Salt Lake City, it makes sense to look for answers in Utah. Therefore, the correct option is Canyonlands National Park only.

The implementation in this blog uses a geo_distance query to be able to find results (the picture’s geopoint) in a particular national park area. We are also using geoshapes to draw the parks’ areas.

However, Elastic capabilities with geo queries go well beyond that:

geo_bounding_box query: Finds documents (geopoints or geoshapes) that intersect a specified rectangle
geo_grid query: Finds documents that intersect a specified geohash, map tile, or H3 bin
geo_shape query: Finds documents that are related (intersects, is contained by, is within, or a disjoint operation) to the specified geoshape

Dataset

We will use geotagged pictures from national parks obtained from Flickr. We augment these pictures with a description and vectorize both the image and description using the openai/clip-vit-base-patch32 model:

We merge these embeddings with the images’ metadata, and at the end, we get a document that looks like this:

{ "title": "Spa Geyser in Yellowstone National Park on a sunny day", "geolocation": { "lat": 44.45899722222222, "lon": -110.82573611111111 }, "image_filename": "52631363114_Spa_Geyser_in_Yellowstone_National_Park_on_a_sunny.jpg", "generated_description": "A small geyser releases steady streams of hot water and steam into the air on a clear sunny day. Colorful mineral deposits surround the thermal feature, creating vibrant orange and yellow formations. The active geothermal vent demonstrates the underground volcanic activity that powers these natural fountains.", "text_embedding": [ 0.02323250286281109, … -0.17811810970306396 ], "image_embedding": [ -0.22548234462738037, … -0.040389999747276306 ] }

A key benefit of using Elastic with geo positions is the Kibana Maps visualization. In Kibana, our dataset looks like this (note that we also added geo shapes for the national parks):

Zooming in, we can see the same document as before in Yellowstone. Additionally, Yellowstone Park’s (approximate) shape is also drawn in the layer below.

System architecture

Indexing pipeline

The indexing pipeline will handle the vectorization of both the image and description. It will also add more metadata to the image to create a document and index it to Elastic:

1. The starting point is pairs of images and metadata from national parks. This metadata includes the geolocation, title of the image, and a description.

2. We feed the image and description to the CLIP model (openai/clip-vit-base-patch32) to obtain an embedding of each in the same vectorial space of 512 dimensions. You can see the complete source code of this step here.

def create_image_embedding(image_path): image = Image.open(image_path).convert('RGB') inputs = processor(images=image, return_tensors="pt", padding=True, truncation=True, use_fast=True) with torch.no_grad(): outputs = model.get_image_features(**inputs) return outputs.numpy().flatten()

The process to generate an embedding from our image is:

Load the image in RGB format using Image.open()
Process the image by converting it into tensors, which is the format the model expects, using processor()
Extracts a dense vector representation in 512 dimensions from the image using model.get_image_features()
At the end, converts the PyTorch tensor into a flattened numpy array using outputs.numpy().flatten()

def create_text_embedding(text): # Process the text inputs = processor(text=[text], return_tensors="pt", padding=True, truncation=True) # Generate embedding with torch.no_grad(): text_features = model.get_text_features(**inputs) # Normalize the embedding (CLIP embeddings are typically normalized) text_features = text_features / text_features.norm(dim=-1, keepdim=True) # Convert to numpy array embedding = text_features.numpy().flatten() return embedding

The process to generate an embedding from text is:

Processes the input text, tokenizing it and converting it to tensors using processor()
Parses the tokenized text using the model to extract its semantic features using model.get_text_features(). The resulting embedding also has 512 dimensions.
Normalizes the embedding so the dot similarity can be computed using text_features / text_features.norm()
Finally, it converts the embedding into a flattened numpy array using text_features.numpy().flatten()

We chose this model because it is a multimodal model that maximizes the similarity between image and text. This way, a description of an image and the image itself tend to generate embeddings that are close in the vector space.

3. We merge all the metadata, the description, geoposition, and embeddings from the image and description in a JSON file

We index the JSON file to Elastic using:

es.index(document=doc, index=index)

Where doc is the metadata for each image.

Search pipeline

This stage will handle the user’s query, create the search, and generate a response from the search results. The LLM used is cogito:3b with Ollama, though it could be easily replaced by any remote model—like Claude or ChatGPT. We chose this particular model because it’s lightweight and it excels at general tasks (as is expected from an assistant) compared to similar models (like Llama 3.2 3B). This means we get proper results without a long waiting time, and everything is running locally!

The pipeline works like this:

1. We receive an input from the user: Where can I see mountains in Washington State?.

2. We feed the user input and a dictionary of the parks, including their states and geolocations (defined in the same Python file), to the LLM with instructions to extract parameters for the Elastic query. The exact prompt is:

”””You are going to extract data from a user query for a national parks search system. Available National Parks: {parks_info} Extract the following information and format it as JSON: - context_search: the main activity or interest (e.g., "hike","walk dog"," or"camping") - distance_km: estimated search radius in kilometers (default: 100 if not specified) - location_type: specific state, city, or region mentioned - reference_location: if a city is mentioned, include it (e.g., "Boston","Denver") - relevant_parks: list of park IDs that might be relevant based on location (use the exact park IDs from the list above) Examples: User query: "Where can I hike in Utah?" Response: {{"context_search": "hike", "distance_km": 100, "location_type": "Utah", "reference_location": null, "relevant_parks": ["arches_national_park", "canyonlands_national_park"]}} Only respond with valid JSON. No additional text. If a city is mentioned, use the state that city is in as the location_type. User query: {query}”””

And this is an example of data in the parks_info dictionary:

{ "mt_rainier_national_park": { "coordinates": (46.8523, -121.7603), "state": "Washington" } }

The model extracts the following data from the prompt above:

{ 'context_search': 'mountains', 'distance_km': 100, 'location_type': 'Washington', 'reference_location': None, 'relevant_parks': ['mt_rainier_national_park'] }

3. We use the context_search parameter to generate a new embedding using the same CLIP model.

4. We extract the coordinates from the parks_info dictionary.

5. We use all these parameters to create an Elasticsearch query. This is the heart of the RAG feature:

We create a geo_distance filter using the coordinates and the 'distance_km' parameter.
We create a match text query against the ‘generated_description’ field.
We create a standard retriever that uses the text query from the previous step and the geo_distance filter.
We create two knn retrievers that use the embedding created in step 4 and match it against the image embedding and text embedding indexed on each document. Each retriever also uses the geo_distance filter.
We use an RRF retriever to combine the resulting datasets of all the other retrievers.

This whole process is executed in the rrf_search() function:

def rrf_search(index_name, lat, lon, distance, text_query, k=10, num_candidates=100): """ Create an RRF search object bound to a specific index. Then executes the search. Args: index_name (str): Name of the Elasticsearch index lat (float): Latitude for geo filtering lon (float): Longitude for geo filtering distance (int/str): Distance for geo filtering text_query (str): Text to search in description fields k (int): Number of top results for KNN search num_candidates (int): Number of candidates for KNN search Returns: Search: List of results frm Elasticsearch """ embedding = create_text_embedding(text_query).tolist() # Create geo distance query geo_filter = Q('geo_distance', distance=distance, geolocation={'lat': lat, 'lon': lon}) # Create text search queries text_queries = [ Q('match', generated_description=text_query), Q('match', description=text_query) ] # Create boolean query for standard search standard_query = Q('bool', filter=[geo_filter], should=text_queries) # Create search object bound to index s = Search(index=index_name) s = s.source(["image_filename", "generated_description"]) # Build RRF configuration retrievers = [ # Standard retriever { "standard": { "query": standard_query.to_dict() } }, # Text KNN retriever { "knn": { "filter": geo_filter.to_dict(), "field": "text_embedding", "query_vector": embedding, "k": k, "num_candidates": num_candidates } }, # Image KNN retriever { "knn": { "filter": geo_filter.to_dict(), "field": "image_embedding", "query_vector": embedding, "k": k, "num_candidates": num_candidates } } ] # Apply RRF configuration s = s.extra(retriever={'rrf': {'retrievers': retrievers}}, size=3) #print(s.to_dict()) es = Elasticsearch(cloud_id=cloud_id, api_key=api_key) results = s.using(es).execute()["hits"]["hits"] return results

At the end, we obtain a query like this:

{ "retriever": { "rrf": { "retrievers": [ { "standard": { "query": { "bool": { "filter": [ { "geo_distance": { "distance": "100km", "geolocation": { "lat": 46.8523, "lon": -121.7603 } } } ], "should": [ { "match": { "generated_description": "mountains" } } ] } } } }, { "knn": { "filter": { "geo_distance": { "distance": "100km", "geolocation": { "lat": 46.8523, "lon": -121.7603 } } }, "field": "text_embedding", "query_vector": [ 0.01967986486852169, ... 0.00988344382494688], "k": 10, "num_candidates": 100 } }, { "knn": { "filter": { "geo_distance": { "distance": "100km", "geolocation": { "lat": 46.8523, "lon": -121.7603 } } }, "field": "image_embedding", "query_vector": [ 0.01967986486852169, ... 0.00988344382494688], "k": 10, "num_candidates": 100 } } ] } }, "size": 3, "_source": [ "image_filename", "generated_description" ] }

6. Afterwards, we feed the documents obtained from Elastic and the user’s original query to the LLM with this prompt:

f"""You are a helpful assistant for national parks activities. Based on the search results below, provide a comprehensive and helpful response to the user's original query. Original User Query: {original_query} Search Parameters Used: - Activity/Interest: {search_params.get('context_search', 'N/A')} - Search Distance: {search_params.get('distance_km', 'N/A')} km - Location: {search_params.get('location_type', 'N/A')} Search results: {results_text} Instructions: - Provide a natural, conversational response - Recommend specific activities and locations based on the search results only - Include practical information when available - Do not suggest alternatives if no results were found - Be enthusiastic and helpful about national parks experiences - Keep the response focused and not too lengthy - Structure your response separating your suggestions per national park - Do not include anything about national parks that are not in the results"""

7. Finally, the LLM creates a response from the search results:

I'd be happy to help you find mountains in Washington State! Based on the search results, here are some fantastic locations: Mount Rainier National Park is a must-visit destination for mountain lovers. Paradise Valley offers breathtaking views of the Tatoosh Mountain Range and Mount Rainier itself. The best time to visit is during late spring when the wildflowers bloom. This location offers incredible opportunities to see mountains up close and personal - whether you're hiking, camping, or simply taking in the breathtaking scenery. Would you like more specific information about this park?

Web application

A front-end based on Streamlit handles the user input, runs the search pipeline to obtain the LLM final response, and displays images from the search results with their descriptions.

You can find the application source code and instructions here.

Multimodal RAG and geospatial search usage example

Query: Any places to ride a boat in Oregon?

Here, the LLM extracted these parameters:

{ 'context_search': 'boat ride', 'distance_km': 100, 'location_type': 'Oregon', 'reference_location': None, 'relevant_parks': ['crater_lake_national_park'] }

And the search centered on Crater Lake National Park, so the response comes only from this national park in Oregon. This way, the system makes sure that it responds to the user under the given constraints and does not mention other parks where a boat ride is possible, but are not in Oregon.

Conclusion

In this article, we saw how integrating multimodal RAG capabilities with Elasticsearch's robust geospatial features significantly enhances the relevance and accuracy of search results in RAG systems. By combining image and text vector search with lexical search and precise geo-filtering, the system can provide highly contextualized answers. This approach not only minimizes hallucinations but also leverages Elasticsearch's diverse geo-query options and Kibana's visualization tools to deliver a comprehensive and user-centric search experience.