Elastic Inference Service (EIS) is now on Elastic Cloud, delivering GPU-accelerated ML inference for Elasticsearch. It simplifies GenAI workflows, removes manual node ops, and powers real-time insights with embeddings, reranking, and LLMs. 

State-of-the-art architecture and performance

blog-EIS-updated-highres.png

blog-eis-diagram-highres.png

What's next? 

Try EIS on Elastic Cloud

blog-EIS-fast (1) (1).png

GPUs go brrr! Elastic Inference Service (EIS): GPU-accelerated inference for Elasticsearch

In 8.0, you can now upload PyTorch machine learning models into Elasticsearch to provide modern natural language processing (NLP).  Integrate one of the most popular formats for building NLP models and incorporate them as part of a NLP data pipeline.

What is NLP?

NLP in Elasticsearch

Workflow of implementing a PyTorch NLP model

What’s next

blog-thumb-website-search.png

Introduction to modern natural language processing with PyTorch in Elasticsearch

Learn how to use a labeled relevance dataset to improve your search relevance. With hands-on examples, sample scripts and Jupyter notebooks, we provide everything you need to get started and optimize your queries!

blog-banner-search-results-orange.png

blog-thumb-search-results-orange.png

Improving search relevance with data-driven query optimization

A great search experience keeps your users and customers coming back. Learn how you can analyze search interaction metrics with Elasticsearch and Kibana to tune search relevance and boost searcher success.

Every great data-driven product starts with a series of questions. What problem are we trying to solve? Who is our target user? How are we going to measure success? Building a great search experience is no different. We need to understand what a user is looking for, translate that into a meaningful search experience, and then measure if the user’s “<a href="https://en.wikipedia.org/wiki/Information_needs">information need</a>”<a href="#1" id="ref1">1</a> has been satisfied — if they found the result(s) they needed no matter how they expressed their query.&nbsp;Maybe it’s obvious when some features of an app or website are successful, but how do you know when a search (or searcher) is successful? We might need to ask if the user found something of interest. Or how long it took them to do so. We can consider the amount of effort put into their search. Did they have to try clicking on multiple results? Did they have to retype their queries in different ways, trying different word combinations or spellings?&nbsp;These are all questions that online search relevance metrics can answer. In this post, we’ll explore how to collect events from a search application and ingest them into Elasticsearch to build granular metrics and dashboards of commonly used online <a href="https://www.elastic.co/enterprise-search/ecommerce" target="_self">search relevance</a> metrics.<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltfd5516a20b7e8f57/5efa5b51fae1631405303713/blog-relevance-metrics-dashboard.jpg" data-sys-asset-uid="bltfd5516a20b7e8f57" alt="Relevance metrics dashboard in Kibana" style="display: block;margin: auto;"/><figcaption>Figure 1. A online search relevance metrics dashboard in Kibana</figcaption>To help illustrate the concepts outlined here and to provide a framework for implementing this approach on your own, we’ve created an <a href="https://github.com/elastic/examples/blob/master/Machine%20Learning/Online%20Search%20Relevance%20Metrics">accompanying repository of code and configuration</a>. It consists of a simulation for generating behavioural events and all the configuration you need for Elasticsearch. The code should be self-contained and everything that is talked about in this post has a corresponding implementation example. Walking through the <a href="https://github.com/elastic/examples/blob/master/Machine%20Learning/Online%20Search%20Relevance%20Metrics">README</a> should help you get everything setup and running. Feedback and pull requests are always welcome.<h2>What are online search relevance metrics?</h2>Before we begin, let’s take a moment to consider just what online search relevance metrics actually are. It’s a mouthful just to say that phrase, so let’s break it down to understand what we’re focusing on here.<h3>Online</h3>Online refers to the fact that we are using behaviour from real users to calculate metrics. We typically mine logs or collect events emitted directly from a search application to track user behaviour. This is contrasted with offline metrics which are typically calculated from expert user judgments. In this scenario, we sit down a bunch of expert users of the search application (domain experts) and ask them to judge the search results. We might ask them if the result in the first position was relevant and if so how relevant was it on a scale of 0 to 4 (0 being not relevant at all). Unfortunately, human judgements are quite expensive and slow to collect since we have to train a bunch of people to actually do the job!<a href="#2" id="ref2">2</a><h3>Search</h3>Ok, of course all metrics related to search are...well...search metrics. What is important to call out is that the metrics described in this post and the accompanying project are aimed at a specific kind of search experience: a traditional one-box search with pages of list-based results. These are probably the most common search experiences, but if you happen to have a grid-based layout for results or perhaps some other type of interface, you will have to reconsider the types of metrics you will need and the semantics of the user behaviour events that you will emit and use.<h3>Relevance</h3>When we talk about relevance we’re referring to that idea of how close to answering a user’s “information need” we can get. An information need is something that the searcher has in mind about what they would like to find or know. A query is then the explicit formulation of the user’s information need. The fun part comes when there is a gap between the information need and the query. For example, if I’m searching for “shovel” at a hardware store website, but I type in “shvel” instead, the gap between my need and the query is just a misspelling which can easily and automatically be corrected for.<a href="#3" id="ref3">3</a>&nbsp;When we measure search relevance, what we’re interested in measuring is how well we fulfil the information need as expressed through the query.<a href="#4" id="ref4">4</a>We can also have non-relevance metrics and information that is also useful but in different ways. Take for example a metric showing the 95th percentile of the query duration, or the amount of time a search query took to execute. This isn’t directly tied to relevance but of course it matters. Other information we might want to look at (and will include in the simulation project) is information about the query distribution often expressed as a table of top queries or query terms. These kinds of non-relevance metrics and associated information are crucial to help understanding the search experience and can actually be used to help tune the relevance. However since they don’t relate to how well we are addressing users’ information needs they aren’t relevance metrics themselves.<h3>Metrics</h3>Metrics are our way of understanding what users are doing and how successful they are in the search experience. However, metrics are only a representation of relevance as they include many biases in the data and other sources of noise. There is a famous saying about models (statistical or scientific) stating that, “<a href="https://en.wikipedia.org/wiki/All_models_are_wrong">All models are wrong, but some are useful</a>.” The same holds true for metrics. We use metrics to guide us in relevance tuning and understanding where to make improvements to the search experience, however we will need more sophisticated tools such as A/B testing and result interleaving to accurately make statements about improvements between versions of a search algorithm. These are all topics that are unfortunately out of scope for this post but it’s important to understand that there are limitations to metrics. What we can do with relevance metrics is make statements like, “Is this search experience providing relevant results or not?” and that is a great place to start!<h2>Walkthrough</h2>Calculating online search relevance metrics starts with capturing user behavior events through the search application. For most search experiences and for the purposes of this post, we’ll be using three types of events: query, page, and click. We’ll use “search application” to encapsulate all of the systems in the stack before your search engine (e.g. Elasticsearch). It includes any backend microservices and your frontend client such as a mobile app or website.<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blta4b002f186356812/5efa5b6f89ea556d588182dd/blog-relevance-metrics-ligh-level.jpg" data-sys-asset-uid="blta4b002f186356812" style="display: block;margin: auto;"/><figcaption>Figure 2. A simplified, high-level data flow diagram. Drawings courtesy of a COVID-19 homeschooling diversion for my kids. For more information on Joe, please see <a href="https://www.youtube.com/watch?v=-VAdltGTgaU">here</a>.</figcaption><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blta0f5aa31b0a08872/5efa5b910d385c132fe65587/blog-relevance-metrics-data-flow.jpg" data-sys-asset-uid="blta0f5aa31b0a08872" style="display: block;margin: auto;"/><figcaption>Figure 3. A data flow diagram of steps inside Elasticsearch itself. Sorry, no cute drawings here.</figcaption><h3>Emitting events</h3>When a user interacts with search and issues a new query, a query event is emitted from the search application. When a user paginates to subsequent pages of results, a page event is emitted. Finally, when a user interacts with a search result in any way, such as clicking through on a search result, a click event is emitted. What ties this solution together is the use of a random and opaque ID (e.g. a UUID) that is generated with the first query event and carried through to all subsequent page and click events. This is what we’ll refer to as the search query ID (and we’ll be talking about this a lot in the future).The semantics of when to issue a query event (and thus generate a new query ID) are also important. We assume that every unique query string, set of filters, etc. are also unique query events since they alter the result set. That means, if a user types “foo” and gets search results, then reformulates the query to “foo fighters” and gets new search results, we consider those as two separate searches and thus will generate two unique query events with unique query IDs. For a single query, if the user moves to the next page, we’ll just emit a page event since the results are not changing, we just want to track pagination effort and results that the user has seen. As you can imagine, this does put some onus on the search application to generate a new query ID and emit query events when those criteria are met, but it’s well worth the effort to encode these semantics correctly.<h3>Ingesting events</h3>Once we have events being emitted by the search application, we’re ready to have them ingested into an Elasticsearch cluster. In most cases this will happen in near real time using <a href="/beats/filebeat">Filebeat</a> with <a href="/guide/en/beats/filebeat/7.8/filebeat-input-log.html">log files</a> or perhaps through an existing event streaming system. Alternatively, you can collect events in a 3rd party library or analytics system and export them in batches into Elasticsearch. The size and location (cloud or on-premises) of your Elasticsearch cluster will depend heavily on the event throughput and amount of historical data you’d like to store. Using the <a href="/elasticsearch/service">Elasticsearch Service</a> on <a href="/cloud/">Elastic Cloud</a> is a perfect way to manage an analytics cluster like this, and scale to whatever size and shape of data you might need. Any way you host it, we’re going to be using a new feature called <a href="https://www.elastic.co/guide/en/elasticsearch/reference/7.8/transforms.html">transforms</a> so make sure you’re using version 7.7 or higher.<h3>Transforming events into per-query metrics</h3>Now that we have behavioural events from the search application in Elasticsearch, we can use <a href="https://www.elastic.co/guide/en/elasticsearch/reference/7.8/transforms.html">transforms</a><a href="#5" id="ref5">5</a> to group events by the unique search query ID. The transform will output to a new index that will store the per-query relevance metrics by grouping events by the search query ID and calculating statistics of all events belonging to the same query. Aggregate metrics such as the average click-through rate (CTR) over a period of time will be done in the last step.As an example, consider the following sequence of events from a single user searching for documentation on <a href="/search">elastic.co</a> (note that these are simplified events schemas):<ul><li>query<pre>Event ID: eid-001 Query ID: qid-001 Value: "machine learning API" Page: 1 </pre></li><li>click<pre>Event ID: eid-002 Query ID: qid-001 Position: 2 Document ID: https://www.elastic.co/guide/en/machine-learning/7.7/ml-api-quickref.html </pre></li><li>page<pre>Event ID: eid-003 Query ID: qid-001 Page: 2 </pre></li><li>click<pre>Event ID: eid-004 Query ID: qid-001 Position: 12 Document ID: https://www.elastic.co/guide/en/machine-learning/7.7/create-jobs.html </pre></li></ul>Grouping the above events by query ID, counting the number of clicks (on any page), and the number of clicks at position 3 (before or in position 3) would result in the following document:<pre>Query ID: qid-001 Number of clicks: 2 Number of clicks at 3: 1 </pre><h3>Aggregate metrics</h3>In the previous step, we calculate per-query metrics. These are granular metrics based on a single user’s unique query and any subsequent actions. The next level of metrics comes from aggregating all of the per-query metrics into statistical summaries that show a metric over a period of time and over all users and queries in that timeframe.As in the example above, if we calculate the number of clicks at position 3 in the transform per-query, we can then count the number queries that have clicks at 3 over a period of time such as a day, week or month. When we find the proportion of queries with clicks at 3 vs all queries with clicks, we get the click-through rate (CTR) at position 3 (CTR@3). In Kibana, we can use a simple <a href="https://www.elastic.co/guide/en/kibana/7.8/TSVB.html">TSVB time series chart</a> and a <a href="/blog/how-to-display-data-as-a-percentage-in-kibana-visualizations">filter ratio calculation</a>.<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt961b5c52003a8cb3/5f0354bcab65482be6a3676c/blog-relevance-metrics-filter-ratio.png" data-sys-asset-uid="blt961b5c52003a8cb3" style="display: block;margin: auto;width: 650px;" width="650"/><figcaption>Figure 4. Using a Kibana TSVB time series chart to calculate CTR@3 with a filter ratio.</figcaption><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt4842ee1ca6c498d4/5f0354cd97f99e2d3aa337cc/blog-relevance-metrics-time-series.png" data-sys-asset-uid="blt4842ee1ca6c498d4" style="display: block;margin: auto;width: 650px;" width="650"/><figcaption>Figure 5. A Kibana TSVB time series chart of CTR@3.</figcaption>Aggregation is typically done ad-hoc or at analysis time rather than in offline batch processes. By aggregating at analysis time, the person analyzing the metrics can decide dynamically on time-ranges, granularity, filtering and splicing data as they see fit to better understand the success of the search experience under measurement.With the above example serving as a template, you can fill a dashboard with metrics calculated using transforms for per-query metrics, and aggregations for aggregation metrics. From CTR@n to abandonment rate and more, it’s all possible! See the project <a href="https://github.com/elastic/examples/blob/master/Machine%20Learning/Online%20Search%20Relevance%20Metrics">README</a> and some of the references cited below for more ideas on which types of metrics can be calculated.<h2>Conclusion</h2>We hope this post has helped you to understand some of the basics of online search metrics and how you can use the Elastic Stack to build a consistent and reliable system for measurement. From here, there are many ways to add custom events, dimensions to events and metrics and we encourage you to keep exploring. If you want to dive deeper and look at the implementation details, please have a read through the project <a href="https://github.com/elastic/examples/blob/master/Machine%20Learning/Online%20Search%20Relevance%20Metrics">README</a> which will definitely keep you busy! Definitely give it a try yourself, either locally or in a <a href="/elasticsearch/service">free trial of Elastic Cloud</a>.How are you measuring your search experiences? What kinds of online (and offline) metrics are you using? Join us in the <a href="http://discuss.elastic.co/">Discuss forum</a> to continue the discussion.<h3>References</h3>For more information on common challenges encountered with calculating and interpreting online search relevance metrics, please refer to:<ul><li><a href="https://dl.acm.org/doi/10.1145/1458082.1458092">How does clickthrough data reflect retrieval quality?</a></li><li><a href="https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/36363.pdf">Beyond Position Bias: Examining Result Attractiveness as a Source of Presentation Bias in Clickthrough Data</a></li></ul>For additional blog posts related to relevance tuning and measurement, please see:<ul><li><a href="/blog/made-to-measure-how-to-use-the-ranking-evaluation-api-in-elasticsearch">Made To Measure: How to Use the Ranking Evaluation API in Elasticsearch blog post</a></li><li><a href="/blog/test-driven-relevance-tuning-of-elasticsearch-using-the-ranking-evaluation-api">Test-Driven Relevance Tuning of Elasticsearch using the Ranking Evaluation API</a></li><li><a href="/blog/how-to-improve-elasticsearch-search-relevance-with-boolean-queries">Improving search relevance with boolean queries blog</a></li></ul><h3>Appendix</h3><a href="#ref1">1</a> From <a href="https://nlp.stanford.edu/IR-book/html/htmledition/boolean-retrieval-1.html">Introduction to Information Retrieval</a>, by Manning, Raghavan and Schütze: “Information retrieval (IR) is finding material (usually documents) of an unstructured nature (usually text) that satisfies an information need from within large collections (usually stored on computers).”<a href="#ref2">2</a> We also have problems with judges disagreeing and not being able to reliably judge search results. By using aggregate user behaviour from real users, we hope to overcome some of these problems. Of course using real user behaviour introduces its own problems, including topics such as position bias, where users are more likely to interact with the top-ranked items than lower-ranked items irrespective of their actual relevance. Some of these are topics of much research and definitely another blog post but it’s important to be aware of the limitations of this approach.<a href="#ref3">3</a> However things get more complex when you consider what exactly I meant when I searched for “shovel”. My actual need (since it’s spring time as I write this) is a “handheld garden shovel”. But if I was searching for “shovel” in the winter (and live somewhere where it actually regularly snows) then I’m more likely to be looking for a “snow shovel”.<a href="#ref4">4</a> In this post we’ll be using only the events on the search results page, however we may want to consider adding business events as a way of measuring “business success” (e.g. add to basket, purchased, etc.) which can sometimes be more meaningful.<a href="#ref5">5</a> Transforms are a feature that was added in 7.3 and reached GA in 7.7. Transforms allows us to perform group-by and pivot type operations on data. Since they are quite in-depth topics, you can find additional information in the <a href="/webinars/introducing-data-frame-transforms-for-elastic-machine-learning">Introducing transforms in Elastic machine learning</a> webinar as well as in the <a href="/blog/elasticsearch-7-3-0-released">7.3.0 release blog</a>.

blog-banner-search-results-dark-blue.png

blog-thumb-search-results-dark-blue.png

Analyzing online search relevance metrics with the Elastic Stack

Multilingual search using language identification in Elasticsearch

Start free trial

Blog Footer CTA

Sign up for Elastic Cloud free trial

headshot-josh-devins-300x300.jpg

Senior Principal Engineer, Search and ML

Josh Devins

Надсилаючи цю форму ви погоджуєтеся з <a href="/legal/terms-of-use">Умовами надання послуг Elastic</a>. Ваші особисті дані будуть опрацьовані відповідно з <a href="/legal/privacy-statement">Положеннями про конфіденційність Elastic</a>.

Newsletter GDPR Text - Ukrainian

By submitting you acknowledge that you've read and agree to our <a href="/legal/elastic-cloud-account-terms" target="_self">Terms of Service</a>, and you agree to receive the <a href="/legal/privacy-statement#how-we-use-the-information" target="_self">selected communications</a> at the contact details you provided above. See <a href="/legal/privacy-statement/" target="_self">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.

Communication Preferences - ONLY for /communication-preferences

By submitting you agree to <a href="/agreements/global/cloud-services-monthly" rel="nofollow">Elastic Cloud Terms of Service</a>. Your personal data will be processed in accordance with <a href="/legal/privacy-statement">Elastic's Privacy Statement</a>.

Cloud Trial GDPR Text

Intentionally Blank

By submitting you acknowledge that you've read and agree to our <a href="/legal/serverless-preview-terms" target="_blank">Terms of Service</a>, and that Elastic may <a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">contact you</a> about our related products and services, using the details you provide above. See <a href="/legal/privacy-statement/" target="_blank">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.

Serverless GDPR Text

提交即表示您確認已詳閱並同意我們的<a href="/legal/elastic-cloud-account-terms" target="_blank">服務條款</a>，且同意 Elastic 使用您在上方提供的個人資料<a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">與您聯絡</a>，為您提供相關產品和服務的資訊。如需詳細說明或隨時想停止收到產品和服務資訊，請參閱 <a href="/legal/privacy-statement/" target="_blank">Elastic 的隱私聲明</a>。

Newsletter GDPR Text - TW

By submitting you acknowledge that you've read and agree to our <a href="/legal/elastic-cloud-account-terms" target="_blank">Terms of Service</a>, and that Elastic may <a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">contact you</a> about our related products and services, using the details you provide above. See <a href="/legal/privacy-statement/" target="_blank">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.By submitting this form, I agree to Elastic sharing my contact information with Google Cloud. I also agree to Elastic giving my information to the vendor for the purpose of shipping my food package.

GDPR Text + Google Cloud

Inviando la registrazione (1) esprimi il consenso ai <a href="/legal/terms-of-use">termini e condizioni di Elastic</a> e <a href="/legal/privacy-statement">all’informativa sulla privacy</a>, nonché (2) a ricevere saltuarie e-mail.

Newsletter GDPR Text - Italian

By submitting you acknowledge that you've read and agree to our <a href="/legal/terms-of-use" target="_blank">Terms of Service</a>, and that Elastic may <a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">contact you</a> about our related products and services, using the details you provide above. See <a href="/legal/privacy-statement/" target="_blank">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.

Newsletter GDPR Text

Load more

Overview

Can't make it? Register and we'll send you the recording. You'll also receive an email with related content

Can't make it? Register and we'll send you the recording. You'll also receive an email with related content.

You'll also receive an email with related content.

You'll also receive an email with related content

Speakers

Close

See more insights

The content on this page is not available in the selected language. As Elastic grows globally, we continue to support content in multiple languages.

Author

Articles by Josh Devins

Senior Principal Engineer, Search and ML, Elastic

GPUs go brrr! Elastic Inference Service (EIS): GPU-accelerated inference for Elasticsearch

Introduction to modern natural language processing with PyTorch in Elasticsearch

Improving search relevance with data-driven query optimization

Analyzing online search relevance metrics with the Elastic Stack

Multilingual search using language identification in Elasticsearch