In this post, we discuss how users’ experience can be further enhanced with the addition of facets, filtering, and additional context. By providing tools like ChatGPT additional context, you can increase the likelihood of obtaining more accurate results. See how Elasticsearch's faceting and filtering framework can allow users to refine their search and reduce the cost of engaging with ChatGPT.

Comparing ChatGPT and Elasticsearch results

To improve the user experience of our sample application, we've added a feature that displays the raw results alongside the ChatGPT-created response. This will help users better understand how ChatGPT works.

Since our source data set is only crawled, the structure in the documents makes it difficult to read for a human. To show this difference and therefore the value that ChatGPT can bring, we added the raw result next to the GPT created response.

Currently, this example application only returns a single result. And even though we have hybrid scoring with vector search and BM25, this result may not be perfect. If we take this not perfect result and pass it over to ChatGPT, there’s a good chance that the response we get won’t be great either, as the context was missing important information.

Ideally, we’d just pass more context into ChatGPT, but the current 3.5-turbo models are limited to 4,096 tokens (that’s including the response you expect to get, so the actual limit is much lower). Future models will likely have a much larger limit, but this also comes with a cost.

As of today, GPT-3.5-turbo costs $0.002 per 1K tokens, while the up-and-coming GPT-4 with 32K context costs $0.06 per 1K tokens — that’s a factor of 30 more. Even with more powerful models coming in the next few years, there’s a chance that it’s not economically viable to do so for all user cases.

We will therefore not use GPT-4 right now and instead work around the max token limitation of GPT-3.5 by sending multiple concurrent requests and giving the user more flexibility in filtering the results.

Leveraging aggregations, facets, and filtering in Elasticsearch to enhance ChatGPT

To address this limitation, one of the biggest advantages of Elasticsearch is its robust faceting and filtering framework. When a user is searching for something, they may have additional preferences or context they can provide to dramatically increase the likelihood of obtaining the correct result. By leveraging Elasticsearch's faceting and filtering framework, we can allow users to refine their search based on various parameters such as date, location, or other relevant criteria.

It’s also important to note that many users have gotten used to having facet filtering options available when searching for something. Let us look at an example.

Searching for “How can I parse a message with Grok?” results in a document for ingest pipelines to be returned as the top result. This is not wrong, as ingest pipelines also support Grok expressions, but what if the user was interested in parsing his data using Logstash?

Using a simple terms aggregation as part of the request to fetch the hits, we can get a list of the top 10 product categories and offer these as a filtering option for a user.

If the user now selects “Logstash” on the left side, all results will be filtered for Logstash. It’s important to note that this all works while still using the same hybrid query model that we’ve talked about in the previous blog. We’re still using a combination of BM25 and kNN search to match our documents.

Loading multiple results in parallel

We briefly mentioned the max token limit earlier. In short, the prompt that you send to the API and its response can’t be longer than 4,096 tokens. When searching your proprietary data, you would like to provide as much specific context as possible so the model can give you the best answer. However, the 4,096 tokens aren’t that much, especially when you include things like code snippets.

A very simple first step toward mitigating the limit is to just ask multiple times in parallel, giving a different context each time. Using our approach with Elasticsearch, instead of only fetching the top 1 result and sending that to OpenAI, we can change the application to load the top 10 hits instead and then ask the question with the respective context.

This gives us 10 unique answers to our question and greatly increases our chances of presenting a relevant answer to the end user. While we are increasing the burden of the user to look at the results, it still gives them more flexibility.

Think of it like this: if you try to debug a problem and search for an exception on Google, you quickly scan the list of the top four or five results that Google displays and click on the one that seems most fitting to your question. Showing the user multiple answers to their question is similar to this.

While having a single correct answer would be ideal, having more than one to choose from initially is a great starting point. And as mentioned before, it can be cheaper compared to using a more expensive model (such as GPT-4).

We can also get more creative with our prompt and ask ChatGPT to send us a specific response if it can’t answer the question using the provided context. This will allow us to remove the results from the UI later.

One prompt that worked well in our use case is:

Working around the max token limit of ChatGPT: Answering a question from a set of answers

Since we have more than a single answer to our question now, we can attempt to summarize them into a single response. For this, we will mostly follow the same approach as before, but instead of searching Elasticsearch for the context, we will just concatenate the individual answers we’ve received so far, excluding any where the model responded that it can’t answer it based on the provided context.

Note that the prompt for this run is a little different from the earlier prompt, so the model treats our context slightly differently. The provided prompt here is by no means perfect, and depending on the data, it should be adjusted and optimized further.

With this additional “reduce phase” in place, our app will now:

• Search Elasticsearch for the top 10 hits

• 10x in parallel ask OpenAI to answer the question, providing a different context each time

• Concatenate responses from OpenAI and ask OpenAI once again to answer the question

With this setup, we can use close to 40,000 tokens of context, while only paying for the considerably cheaper GPT-3.5 model. In another blog post, we will explore the cost in more detail and use Elastic APM for tracking our spend, alongside other metrics.

It should be noted that GPT-4 may still perform much better than the approach above, so use whatever works best for you and the amount of traffic you expect.

Citations for your ChatGPT results

One downside of large language models (LLMs) is their overconfidence and tendency to hallucinate. You ask a question, you get an answer. Whether the answer is actually correct is for you to decide. The model rarely admits that it does not know something. Providing the context and telling it to respond with a specific answer as we did above helps mitigates this to some extent.

But the provided context alongside getting the model to admit that it can’t answer a question also allows us to provide more accurate citations for the responses.

In the last section, we summarized our set of 10 answers into one global answer. In addition to just providing this global answer, we can also provide a list of all source documentation pages that we used to compile the result — basically any page where the model did not respond "The provided page does not answer the question."

In this screenshot, you can see the summary answer on a set of 10 results from Elasticsearch. Even though we inspected 10 results, we are only displaying the three links to the documentation that are actually relevant to answer the question. In this case, the other seven documents returned by Elasticsearch had something to do with documents or indices, but they didn’t specifically talk about how to index something.

Searching proprietary data

We’ve mentioned in an earlier blog post that it’s great to use Elasticsearch and OpenAI to search proprietary data. However, we did use a web-crawler to crawl public documentation. That may seem a bit counterintuitive, and you’re right to think about it! OpenAI trains GPT models on web data, so we will assume it knows our documentation already. So why do we need Elasticsearch in addition to that data? Does this setup actually work on data that’s not public? It does — let’s prove it.

Using the existing setup, we will push a single super secret document about an internal project into our index.

Next we’ll then head over to our app and search for “What are the steps for the internal project?”

In summary, we used faceting and filtering to, for certain use cases, reduce the number tokens of context required to engage with ChatGPT. By providing additional context at query time, we showed it is also possible to improve the accuracy of search results.

Learn more about the possibilities with Elasticsearch and AI .

In this blog post, we may have used third party generative AI tools, which are owned and operated by their respective owners. Elastic does not have any control over the third party tools and we have no responsibility or liability for their content, operation or use, nor for any loss or damage that may arise from your use of such tools. Please exercise caution when using AI tools with personal, sensitive or confidential information. Any data you submit may be used for AI training or other purposes. There is no guarantee that information you provide will be kept secure or confidential. You should familiarize yourself with the privacy practices and terms of use of any generative AI tools prior to use.

Costs referred to herein are based on the current OpenAI API pricing and how often we call it when loading our sample app.

Elastic, Elasticsearch and associated marks are trademarks, logos or registered trademarks of Elasticsearch N.V. in the United States and other countries. All other company and product names are trademarks, logos or registered trademarks of their respective owners.

In this blog, we will instrument a Python application that uses OpenAI and analyze its performance, as well as the cost to run the application. Using the data gathered from the application, we will also show how to integrate large language models (LLMs) into your application. As a bonus, we will try to answer the question: why does ChatGPT print its output word by word?

Instrumenting the application with Elastic APM

If you’ve had a chance to give our sample application a try, you might have noticed that the result does not load as quickly as you’d expect it to, from a search interface.

The now is if this is from our two-phased approach of running a query in Elasticsearch first or if the slow behavior is emerging from OpenAI, or if it’s a combination of the two.

Using Elastic APM, we can easily instrument this application to get a better look. All we need to do for the instrumentation is the following (we will show the full example at the end of the blog post and also in a GitHub repository):

Since our sample application is using Streamlit, we will also need to start at least one transaction and eventually end it again. In addition, we can also provide information about the outcome of the transaction to APM, so we can track failures properly.

And that’s it — this would be enough to have full APM instrumentation for our application. That being said, we will be doing a little extra work here in order to get some more interesting data.

As a first step, we will add the user’s query to the APM metadata. This way we can inspect what the user was trying to search and can analyze some popular queries or reproduce errors.

In our async method, which talks to OpenAI, we will also add some more instrumentation so we can better visualize the tokens we receive, as well as to collect additional statistics.

And finally, toward the very end of our application, we will also add the number of tokens and approximate cost to our APM transaction. This will enable us to visualize these metrics later and correlate them to the application performance.

If you do not use streaming, then the OpenAI response will contain a “total_tokens” field, which is the sum of the context you sent and the response returned. If you are using the “stream=True” option, then it’s your responsibility to calculate the number of tokens or approximate them. A common recommendation is to use “(len(prompt) + len(response)) / 4” for english text, but especially code snippets can throw off this approximation. If you need more exact numbers, you can use libraries like tiktoken to calculate the number of tokens.

Analyzing the APM data — Elasticsearch vs. OpenAI performance

After instrumenting the application, a quick look at the “Dependencies” gives us a better understanding of what’s going on. It looks like our requests to Elasticsearch return within 125ms on average, while OpenAI takes 8,500ms to complete a request. (This screenshot was taken on a version of the application that does not use streaming. If you use streaming, the default instrumentation only considers the initial POST request in the dependency response time and not the time it takes to stream the full response.)

If you’ve used ChatGPT yourself already, you might have been wondering why the UI is printing every word individually, instead of just returning the full response immediately.

As it turns out, this is not actually to entice you to pay money if you’re using the free version! It’s more of a limitation of the inference model. In simple terms, in order to compute the next token, the model will need to take into consideration the last token as well. So there’s not much room for parallelization. And since every token is processed individually, this token can also be sent to the client, while the computation for the next token is running.

In order to improve the UX, it can be helpful to therefore use a streaming approach when using the ChatCompletion functionality. This way a user can start to consume the first results while the full response is being generated. You can see this behavior in the GIF below. Even though all three responses are still loading, the user can scroll down and inspect what’s there already.

As mentioned previously, we added a bit more custom instrumentation than just the bare minimum. This allows us to get detailed information on where our time is spent. Let’s take a look at a full trace and see this streaming in action.

Our application is configured to fetch the top three hits from Elasticsearch, and then run one ChatCompletion request against OpenAI in parallel.

As we can see in the screenshot, loading the individual results takes about 15s. We can also see that requests to OpenAI that return a larger response take longer to return. But this is only a single request. Does this behavior happen for all requests? Is there a clear correlation between response time and number of tokens to back up our claims from earlier?

Analyzing cost and response time

Instead of visualizing the data using Elastic APM, we can also use custom dashboards and create visualizations from our APM data. Two interesting charts that we can build show the relationship between the number of tokens in a response and the duration of the request.

We can see that the more tokens get returned (x-axis in the first chart), the higher the duration (y-axis in the first chart). In the chart to the right, we can also see that the duration per 100 tokens returned stays almost flat at around 4s, no matter the number of tokens returned in total (x-axis).

If you want to improve the responsiveness of your application that uses OpenAI models, it might be a good idea to tell the model to keep the response short.

In addition to this, we can also track our total spend and the average cost per page load, as well as other statistics.

With our sample application, the cost for a single search is around 1.1¢. This number does not sound high, but it’s likely not something that you will have on your public website as a search alternative anytime soon. For company internal data and a search interface that’s only used occasionally, this cost is negligible.

In our testing, we’ve also hit frequent errors when using the OpenAI API in Azure, which eventually made us add a retry loop to the sample app with an exponential backoff. We can also capture these errors using Elastic APM.

Any captured errors are then visible in the waterfall charts as part of the span where the failure happened.

In addition, Elastic APM also provides an overview of all the errors. In the screenshot below, you can see the occasional RateLimitError and APIConnectionError that we’ve encountered. Using our crude exponential retry mechanism, we can mitigate most of these problems.

Latency and failed transaction correlation

With all the built-in metadata that the Elastic APM agent capture, as well as the custom labels we added, we can easily analyze if there’s any correlation between the performance and any of the metadata (like services version, user query, etc.)

As we can see below, there’s a small correlation between the query “How can I mount and index on a frozen node?” and a slower response time.

Similar analysis can be done on any transaction that resulted in an error. In this example, the two queries “How do I create an ingest pipeline” and “How can I create an ingest pipeline” fail more often than other queries, causing them to bubble up in this correlation analysis.

In this blog, we instrumented an app written in Python to use OpenAI and analyze its performance. We looked at response latency and failed transactions, and we assessed the costs of running the application. We hope this how-to was useful for you!

Learn more about the possibilities with Elasticsearch and AI .

In this blog post, we may have used third party generative AI tools, which are owned and operated by their respective owners. Elastic does not have any control over the third party tools and we have no responsibility or liability for their content, operation or use, nor for any loss or damage that may arise from your use of such tools. Please exercise caution when using AI tools with personal, sensitive or confidential information. Any data you submit may be used for AI training or other purposes. There is no guarantee that information you provide will be kept secure or confidential. You should familiarize yourself with the privacy practices and terms of use of any generative AI tools prior to use.

Costs referred to herein are based on the current OpenAI API pricing and how often we call it when loading our sample app.

Elastic, Elasticsearch and associated marks are trademarks, logos or registered trademarks of Elasticsearch N.V. in the United States and other countries. All other company and product names are trademarks, logos or registered trademarks of their respective owners.