What you'll discover

In this post, you'll learn how to:

• Set up OpenTelemetry in an example Python FastAPI back end.

• Add custom search.* attributes to your search spans in ~20 lines of code.

• Understand how OTel-native ingestion maps attributes to queryable data.

• Write six ES|QL queries against real trace data: top queries, zero-results rate, which queries return nothing, average and max latency, slow query investigation, and search volume over time.

• Turn those queries into saved Kibana visualizations.

What you'll need

• An Elastic Cloud deployment (or self-managed with OTel-native ingestion enabled). Examples tested on Elastic Stack 9.x.

From concept to code: Building the search analytics instrumentation

In the first post, we made the case for using OpenTelemetry to capture search analytics. The idea: Add search.* attributes to your existing OTel spans, send them to Elastic APM, and query them with ES|QL.

Now let's build it.

Want working code? A companion reference project accompanies this series. It's a minimal FastAPI app with the exact instrumentation described below. Clone it, add your Elastic Cloud credentials, and you'll have search analytics data flowing in 10 minutes. Each blog stage maps to a commented-out code block you can enable as you progress.

An OpenTelemetry primer for search API developers

If you've been building search systems but haven't worked with OpenTelemetry before, here's the minimum you need to know.

OTel is an open standard for collecting observability data, traces, metrics, and logs from your applications. It's vendor-neutral: You instrument your code once, and send data to any compatible back end.

The core concept is the span. A span represents a single operation, for example, an API call, a database query, or a search request. Every span has a start time, an end time (the difference is the span duration), and attributes, which are key-value pairs that describe what happened.

Spans nest inside each other to form traces. A trace is a tree of spans that represents one end-to-end request. When a user searches, the trace might look like: browser request → API handler → search logic → Elasticsearch query. Each step is a span, and the parent-child relationships show you exactly where time was spent. This is distributed tracing, which works across services and network boundaries, so a single trace can follow a request from front end to back end to database and back.

Why traces instead of logs? You could log "search query=headphones results=15 took=120ms" and parse it later. But a log line is flat; it can't show you that the 120ms Elasticsearch time sat inside a 250ms API call, revealing 130ms of overhead in your application layer. Traces give you hierarchy, timing, and correlation across services. For search analytics, that means you can see not just what happened but also where time was spent and how operations relate to each other.

For this post, we don't need to understand the full OTel ecosystem. We just need three things:

1. Create a span when a search request happens.

2. Add attributes to that span, describing the search (such as search.query or result_count).

3. Send the span to Elastic, where we can query it with ES|QL.

That's it. If you can call span.set_attribute("key", value), you can build search analytics.

What you'll build

By the end of this post, every search request in your API will emit an OTel span that looks like this:

And you'll run six ES|QL queries against real data to answer questions that your team is already asking, using about 20 lines of instrumentation code in total.

Install the OTel SDK

We're using Python and FastAPI here. The same pattern applies to any language with an OTel SDK; the concepts are identical, only the imports change.

Elastic provides the Elastic Distribution of OpenTelemetry Python (EDOT), which bundles the standard OTel SDK with sensible defaults, early access to Elastic-contributed improvements, and a single configure_opentelemetry() call that handles all the boilerplate. We recommend it:

Three packages, two roles:

• elastic-opentelemetry EDOT: The OTel API, SDK, and OpenTelemetry Protocol (OTLP) exporter in one package, preconfigured for Elastic.

• opentelemetry-instrumentation-fastapi: Auto-instruments HTTP endpoints (automatic spans for every request).

• opentelemetry-instrumentation-elasticsearch: Auto-instruments Elasticsearch client calls (automatic spans for every query).

The auto-instrumentation packages are doing real work here. Without writing a single line of tracing code, you already get HTTP request spans and Elasticsearch query spans. What we're adding is the search-specific context that turns generic traces into analytics.

Using the standard OTel SDK instead? Replace elastic-opentelemetry with opentelemetry-api, opentelemetry-sdk, and opentelemetry-exporter-otlp-proto-http. You'll need to wire up the TracerProvider, OTLPSpanExporter, and BatchSpanProcessor manually (about 10 extra lines). Everything else in this post works the same either way. See the Elastic OTel guide for the full setup.

Configure the connection

OTel uses environment variables for connection configuration. You need four to get started:

Variable

Purpose

Example

`OTEL_EXPORTER_OTLP_ENDPOINT`

Managed OTLP (mOTLP) endpoint URL

`https://my-deployment.ingest.us-central1.gcp.elastic-cloud.com`

`OTEL_EXPORTER_OTLP_HEADERS`

Authentication

`Authorization=ApiKey <your-api-key>`

`OTEL_SERVICE_NAME`

Service name (shown in Kibana APM)

`search-analytics-demo`

`OTEL_RESOURCE_ATTRIBUTES`

Other resource attributes

`service.version=1.0.0`

Where to find these values: In Elastic Cloud, your mOTLP endpoint follows the pattern https://<deployment>.ingest.<region>.gcp.elastic-cloud.com. You can find it in the Elastic Cloud console under your deployment's details or in Kibana at the APM integration page (/app/home#/tutorial/apm) under the OpenTelemetry tab. Your API key can be created from Kibana's Stack Management > API Keys or via the Elasticsearch Create API Key API. For self-managed deployments, you can use the EDOT Collector as an intermediary that receives OTLP and forwards to Elasticsearch.

Set them in your environment or .env file:

Initialize the tracer

With EDOT and environment variables configured, initialization wires up three things: the tracer provider and two auto-instrumentation packages that automatically create spans for every HTTP request and every Elasticsearch query:

configure_opentelemetry() reads the OTEL_* environment variables and sets up the tracer provider, exporter, and batch processor with Elastic-optimized defaults, including using the HTTP exporter automatically, which is what the mOTLP endpoint requires. If you see connection errors with vanilla OTel, the most common cause is accidentally using the gRPC exporter instead of HTTP.

The two instrumentors patch the FastAPI and elasticsearch-py libraries at startup so every request and every Elasticsearch call automatically generates a span, without any code changes to individual endpoints.

Instrument your search API

Here's where it gets interesting. This is the code that turns a generic API endpoint into a search analytics source:

A few things worth unpacking.

• start_as_current_span creates the span and sets it as the active span in the current context. This matters because the Elasticsearch client instrumentation picks up the active span and nests its own spans underneath it. You get a span hierarchy automatically.

• search.query_id is derived from the trace ID. Every trace already has a unique identifier, and we're reusing it as the query identifier. There’s no UUID generation or database sequence. When we add click tracking later, clicks will reference this same query_id to link back to the search that produced the results.

• search.result_count does double duty. It tells you how many results came back, and when it's zero, you know you have a content gap. There’s no need for a separate boolean flag: Just filter on result_count == 0 in your queries.

Note that the application name is no longer set as a span attribute; it's the service.name resource attribute, configured once via OTEL_SERVICE_NAME. This is the standard OTel approach: Resource attributes describe the service, and span attributes describe the operation.

Normalize search queries before analyzing them

Notice that we're storing request.query as is. That means "Laptop Bag", "laptop bag", and " laptop bag " will be counted as three different queries when you aggregate with STATS ... BY attributes.search.query.

For cleaner analytics, normalize before setting the attribute:

Lowercasing and trimming whitespace is enough for most cases. If you need the original phrasing (for display or debugging), store it in a separate attribute, like search.query.original. But start simple. You can always add the raw version later if you find you need it.

What a search API trace looks like

Once this is running, a single search request produces this trace:

The auto-instrumented spans give you HTTP latency and Elasticsearch query detail. Your search span in the middle ties them together with the business context: what the user searched for, how many results came back, how long Elasticsearch took.

The search span attributes you need to capture

Here's the full set of attributes we're capturing on the search span:

Attribute

Type

When set

Purpose

`search.query`

string

Before query

The query as the user entered it

`search.query_id`

string

Before query

Unique identifier, derived from trace ID

`search.result_count`

int

After query

Total matching results (0 = zero-result search)

`search.took_ms`

int

After query

Elasticsearch execution time in milliseconds

`search.query_response_hit_ids`

string[]

After query

Document IDs returned (optional; enables per-result analytics)

`feature_flag.key`

string

Before query

A/B test flag name (optional; pair with `feature_flag.result.variant` for the assigned variant; enables per-variant click-through rate (CTR) comparison)

We use the search.* namespace following OTel's convention of domain-specific prefixes (http.*, db.*, messaging.*). While there aren't standardized search conventions in OTel yet, search.* is self-describing and vendor-neutral. The naming is informed by the User Behavior Insights (UBI) Standard, which defines a schema for search events. We reference it for structure without coupling to it. Where established OTel conventions exist, like feature_flag.key (flag name) and feature_flag.result.variant (assigned variant) for A/B experiments, we reuse them rather than inventing custom attributes.

You'll notice that enduser.pseudo.id isn't in the search span table above. We don't need it for query analytics, but you'll add it as soon as you introduce click tracking in our third blog; it ties click events back to a specific browser session, enabling per-user CTR and Mean Reciprocal Rank (MRR). Blog 3 adds enduser.pseudo.id (browser-generated, persistent across sessions) to link clicks back to searches. Our fourth blog focussing on revenue attribution documents session.id and user.id as optional extensions for authenticated users who want cross-device attribution.

How OpenTelemetry attributes become queryable ES|QL fields

Before we start querying, you need to understand how OTel attributes map to Elasticsearch fields. With OTel-native ingestion into Elastic, the mapping is straightforward.

OTel attribute

Type

ES|QL field

`search.query`

string

`attributes.search.query`

`search.result_count`

int

`attributes.search.result_count`

`search.took_ms`

int

`attributes.search.took_ms`

`search.query_id`

string

`attributes.search.query_id`

`feature_flag.key`

string

`attributes.feature_flag.key`

With OTel-native ingestion, attribute names preserve their dot notation under attributes.*. All types live in the same namespace; there’s no split between string and numeric fields. Booleans are stored as native booleans, not strings. If you've used Elastic APM's classic ingestion before, you'll appreciate the simplicity: What you set in code is what you query.

Running ES|QL queries in Kibana Discover

With spans flowing to Elastic, open Kibana and go to Discover (in the left sidebar under Analytics, or use the global search bar and type "Discover"). By default, you'll see the KQL query bar, a filter language familiar from Kibana dashboards. Click Try ES|QL in the top right to switch to the ES|QL editor. Unlike KQL (which filters documents) or the JSON query DSL (which requires nested objects), ES|QL is a piped language: Each | step transforms the previous output, making aggregations like STATS count BY field read naturally from left to right.

The editor gives you a full-width text area where you type piped queries. Results appear as both a table and an auto-generated chart; Kibana picks a sensible visualization based on your query shape. For STATS ... BY queries, you'll get a bar chart automatically.

Set the time range wide enough to capture your data (top-right date picker). If you're just getting started, try "Last 30 days".

Six ES|QL queries for search analytics

Everything below runs against traces-generic.otel-default, the index where Elastic's OTel-native ingestion automatically stores trace data. You don't need to create this index; it's provisioned by Elastic when the first OTLP span arrives.

These queries ran against our live demo cluster: 62 searches, ~20 distinct queries, latency range 77ms–153ms.

Note: Results below are illustrative. When you run the reference project and generate traffic with python generate_traffic.py --blog 2 --sessions 50, your exact numbers and top queries will vary based on session count and random query selection. The query patterns and ES|QL syntax are what matter here.

Query 1: Are spans arriving?

Start simple. Count your search spans.

Result: 62.

If this returns zero, your spans aren't arriving. Check your OTLP endpoint and API key and that init_otel() is being called before any requests. The name == "search" filter ensures that you're counting your custom spans, not the auto-instrumented Elasticsearch client spans (which are also named "search").

Query 2: What are users searching for?

The first question every search team asks.

Results:

Query

Searches

Average results

laptop

9

8

headphones

7

12

running shoes

6

5

"laptop" was the most popular query, with nine searches. There were around 20 distinct queries total (including zero-result ones).

The avg_results column tells you whether popular queries are actually returning content. A query with high volume and low results is a relevance problem worth investigating. If your query has high volume and high results, check whether users are actually clicking. We'll get to that in the next blog focussing on measuring search quality with click data.

Query 3: What percentage of searches return nothing?

Result: 17.7% (11 out of 62 searches returned nothing).

We're using attributes.search.result_count == 0, a straightforward numeric comparison. No separate boolean attribute is needed when you already have the count.

A zero-results rate above 10% is worth investigating. Every zero-result search is a user who asked for something and got nothing back. Some of those are junk queries, but others reveal real content gaps or query parsing failures.

Query 4: Which queries return nothing?

The rate tells you there's a problem. This query tells you where.

Results:

Query

Occurrences

quantum physics calculator

4

unicorn saddle

3

holographic projector

2

time machine parts

2

Three different failure modes: "quantum physics calculator" and "unicorn saddle" are out-of-catalog queries you'll never be able to serve. This is useful to know but nothing to fix. "holographic projector" might be a real emerging category worth considering. "time machine parts" is probably noise. In a production catalog, these would be mixed with legitimate zero-result queries that are fixable, like missing synonyms, phrasing mismatches, or product gaps.

Repeated zero-result queries are the highest-priority fixes. One-off failures are usually noise.

Query 5: How fast is search?

Result: Average 81ms, max 153ms.

search.took_ms captures Elasticsearch's self-reported execution time, the took field from the search response. This is different from span duration, the wall-clock time from when the span started to when it ended (as we covered in the primer above). Span duration measures end-to-end time, including network round trips, serialization, and application logic. You want both: Comparing them tells you where overhead lives. If took_ms is 50ms but the span duration is 200ms, the extra 150ms is network or application overhead, not a query problem.

This attribute also keeps your analytics portable. If you're using OTel log records instead of spans (a lighter-weight alternative we mention in the first blog in the series), there's no span duration. took_ms is the only timing signal you have.

We'll go deeper on search performance monitoring (Service Level Objectives [SLOs], alerting on latency regressions, and using this data for operational dashboards) in the last blog in the series focussing on Search Reliability Engineering.

Want to find the slow queries specifically?

A query with high average latency and high result count is hitting many documents; consider query optimization. High latency with low results might mean complex filters or slow aggregations. Outlier max values are often cold caches or cluster issues.

Query 6: How does search volume change over time?

Counts, rates, and latencies tell you the what. Volume over time tells you the when: When did traffic spike, when did it drop, and when did that zero-results rate jump?

DATE_TRUNC(5 minutes, @timestamp) rounds each timestamp down to the nearest 5-minute boundary. The result is a time series that Kibana's Lens can render as a bar chart or line, showing your search traffic pattern for any time window.

Narrow the bucket for higher granularity (1 minute), widen it for trend analysis (1 hour, 1 day). When you add this to a dashboard alongside your zero-results rate, you can answer: Did zero-results spike because traffic changed or because something broke?

Verify that your search API instrumentation is working

If you're using the reference project, the full setup is:

Then trigger a search:

Wait 5–10 seconds for the BatchSpanProcessor to flush, and then open Kibana → Discover → switch to ES|QL mode and run:

You should see rows with attributes.search.query, attributes.search.result_count, and attributes.search.took_ms.

If no rows appear, check in order:

1. OTEL_EXPORTER_OTLP_ENDPOINT points to the mOTLP endpoint, not your Elasticsearch URL.

2. OTEL_EXPORTER_OTLP_HEADERS includes Authorization=ApiKey <your-key>.

3. OTEL_TRACES_SAMPLER=always_on is set (default sampler may drop spans).

4. Kibana → Observability → APM → Services shows search-analytics-demo (confirms export is working).

Turn ES|QL results into Kibana visualizations

The bar chart that Discover auto-generates from your ES|QL results is a good start, but you can customize it. Click the pencil icon in the top-right corner of the chart to open the inline Lens editor.

From here you can:

• Change chart type (bar, line, area, pie, table, metric).

• Adjust axes and add breakdown dimensions.

• Save the visualization to a dashboard.

This is the path from ad hoc ES|QL exploration to a persistent dashboard panel. You don't need to build visualizations from scratch; Discover and Lens handle the chart rendering from your query results.

Lens is Elastic's drag-and-drop visualization editor, and it's more capable than this quick workflow suggests. You can build multilayer charts, combine metrics with breakdowns, add reference lines, and design full dashboards that mix ES|QL panels with traditional aggregation-based visualizations. For search analytics, that means you can put top queries, zero-results trends, and latency percentiles side by side in a single view.

To go deeper:

• Lens documentation: A full guide to the visualization editor.

• ES|QL in Lens: Using ES|QL queries as data sources for dashboard panels.

• Kibana Dashboards: Building and sharing operational dashboards.

• Use an Agent built in Kibana to make visualizations for you

We'll build a full search analytics dashboard in a later post.

How sampling affects search analytics accuracy

Most application performance monitoring (APM) configurations sample traces to control costs, capturing 10% or 25% of requests. For application monitoring, that's fine. For search analytics, it's a problem.

If you're sampling at 10%, your "total searches" count is 90% lower than reality. Your zero-results rate is still accurate (it's a ratio), but volume counts are off.

Two approaches:

Approach 1: Configure 100% sampling for search endpoints. Your search API probably handles far fewer requests than your main application, so the data volume increase is manageable. The simplest way is through environment variables:

These are head-based sampling decisions made at the start of each trace. They apply globally to the service, which is fine if your search API is a dedicated service. If search shares a service with other endpoints and you need per-endpoint sampling rules, you can implement a custom Sampler in the OTel SDK that inspects the span name or attributes before deciding.

More sophisticated routing (sampling differently per endpoint, dropping noisy spans, or making decisions after a trace completes [tail-based sampling]) typically involves deploying an OTel Collector (such as the EDOT Collector) as an intermediary between your application and Elastic. That's a valuable architecture pattern, but it's beyond the scope of this post. See the OTel Collector documentation and Elastic's EDOT Deployment for more on collector-based sampling and routing architectures.

Approach 2: Upscale in your queries. If you know the sampling rate, multiply: EVAL estimated_total = total_searches * 10. Ratios and averages stay correct; only absolute counts need adjustment.

For more on sampling strategies generally, see the OTel sampling documentation.

For the queries in this post, we used 100% sampling.

What's next: Adding click tracking to search analytics

The six ES|QL queries in this post answer: What do users search for, what returns nothing, how fast is search, and when does traffic spike? They're all derived from a single instrumentation point: the search span.

But they can't tell you whether users are finding what they need. A search that returns 15 results looks healthy from the server side. But if nobody clicks any of those results, your ranking has a problem.

In the next post, we add click tracking, a second span that captures which result the user clicked and where it appeared in the list. If you've been running the reference project, you already have 62 search spans; the next post builds directly on that data. With searches and clicks linked together, we'll calculate:

• Click-through rate (CTR): What percentage of searches result in a click.

• Mean Reciprocal Rank (MRR): How far down the results users have to scroll.

• Click position distribution: The shape of where users click.

The pattern is the same: You add attributes to spans and query them with ES|QL. You get a richer view of the same data without introducing new infrastructure.

Resources to get started with search analytics on OpenTelemetry

• Reference project: Working code for the entire blog series; clone, configure, run.

• EDOT Python: Elastic distribution of OpenTelemetry for Python.

• OpenTelemetry with Elastic: How to send OTel data to Elastic APM.

• OpenTelemetry Python SDK: Upstream SDK documentation and instrumentation guides.

• ES|QL documentation: Query language reference.

• UBI Standard: Reference schema for search event structure.

This is the second post in a series on search analytics with OpenTelemetry and Elastic. Next up: Measuring search quality: Click tracking, CTR, MRR, and click position analysis.

Now, what if we could use real user interactions with our search application to create this training data? Using UBI data lets us do just that. Creating an automatic system that can capture and use our searches, clicks, and other interactions to generate a judgment list. This process can scale and be repeated far more easily than a manual interaction and would tend to yield better results. In this blog, we will explore how we can query UBI data stored in Elasticsearch to calculate meaningful signals to generate a training dataset for an LTR model.

You can find the full experiment here.

Why UBI data can be useful to train your LTR model

UBI data offers several advantages over a manual annotation:

• Volume: Given that UBI data comes from real interactions, we can collect much more data than we can generate manually. This is assuming we have enough traffic to generate this data, of course.

• Real User intent: Traditionally, a manual judgment list comes from an expert evaluation of the available data. On the other hand, UBI data reflects real user behavior. This means we can generate better training data that will improve our search system's accuracy, because it's based on how users actually interact with and find value in your content rather than theoretical assumptions about what should be relevant.

• Continuous updates: Judgment lists need to be refreshed over time. If we create them from UBI data, we can have current data that results in updated judgment lists.

• Cost effectiveness: Without the overhead of manually creating a judgment list, the process can be repeated efficiently any number of times.

• Natural query distribution: UBI data represent real user queries, which can drive deeper changes. For example, do our users use natural language to search in our system? If so, we might want to implement a semantic search or hybrid search approach.

It does come with some warnings, though:

• Bias amplification: Popular content is more likely to receive clicks, just because it gets more exposure. So this might end up amplifying popular items and possibly drowning out better options.

• Incomplete coverage: New content lacks any interactions, so it might be difficult for it to be high in the results. Rare queries can also lack sufficient data points to create meaningful training data.

• Seasonal variations: If you expect user behaviour to change drastically over time, historical data might not tell you much about what is a good result.

• Task ambiguity: A click doesn’t always guarantee that the user found what they were looking for.

Grades calculation

Grades for LTR training

To train LTR models, we need to provide some numerical representation of how relevant a document is for a query. In our implementation, this number is a continuous score going from 0.0 to 5.0+, where higher scores indicate higher relevance.

To show how this grading system works, consider this manually created example:

Query

Document content

Grade

Explanation

"best pizza recipe"

"Authentic Italian Pizza Dough Recipe with Step-by-Step Photos"

4.0

Highly relevant, exactly what the user is looking for

"best pizza recipe"

"History of Pizza in Italy"

1.0

Somewhat in topic, it is about pizza but is not a recipe

"best pizza recipe"

"Quick 15-Minute Pizza Recipe for Beginners"

3.0

Relevant, a good result but it maybe misses the mark on being the “best” recipe.

"best pizza recipe"

"Car Maintenance Guide"

0.0

Not relevant at all, completely unrelated to the query

As we can see here, the grade is a numerical representation of how relevant a document is to our sample query of “best pizza recipe”. With these scores, our LTR model can learn which documents should be presented higher in the results.

How to calculate the grades is the core of our training dataset. There are multiple approaches to do this, each with its own strengths and weaknesses. For example, we could assign a binary score of 1 for relevant 0 for not relevant or we could just count the number of clicks in a resulting document for each query.

In this blog post, we will be using a different approach, taking into account the user behavior as our input and calculating a grade number as the output. We will also be correcting bias that could occur from the fact that higher results tend to be more clicked, regardless of the relevancy of the document.

Calculating the grades - COEC algorithm

The COEC (Clicks over Expected Clicks) algorithm is a methodology for calculating judgment grades from user clicks. As we stated earlier, users tend to click on higher-positioned results even if the document is not the most relevant to the query; this is called Position Bias. The core idea for using the COEC algorithm is that not all clicks are equally significant; a click on a document at position 10 indicates that the document is much more relevant to the query than a click on a document at position 1. To quote the research paper about the COEC algorithm (linked above):

“It is well known that the click-through rate (CTR) of search results or advertisements decreases significantly depending on the position of the results.”

You can further read about position bias here.

To address this with the COEC algorithm, we follow these steps:

1. Establish position baselines: We calculate the click-through rate (CTR) for each search position from 1 to 10. This means we determine what percentage of users typically click on position 1, position 2, and so on. This step captures the users’ natural position bias. We calculate the CTR using:Where:

= Position. From 1 to 10

= Total clicks (on any document) at position p across all queries

= Total impressions: How many times any document appeared at the position p across all queries

Here, we expect higher positions to get more clicks.

2. Calculate Expected Clicks (EC):

This metric establishes how many clicks a document “should” have received based on the positions it appeared in and the CTR for those positions We calculate EC using:Where:

= All queries where the document d appeared

= Position of the document d in the query q results

3. Count actual clicks: We count the actual total clicks a document received across all queries where it appeared, hereafter called A(d).

4. Compute the COEC score: This is the ratio of Actual clicks (A(d)) over the Expected clicks (EC(d)):This metric normalizes for position bias like this:

• A score of 1.0 means the document performed exactly as expected given the positions it appeared in.

• A score above 1.0 means the document performed better than expected by looking at its positions. So this document is more relevant for the query.

• A score under 1.0 means the document performed worse than expected by looking at its positions. So this document is less relevant for the query.

The end result is a grade number that captures what users are looking for, taking into account position-based expectations extracted from real interactions with our search system.

Technical implementation

We will be creating a script to create a judgment list to train an LTR model.

The input for this script is the UBI data indexed in Elastic (queries and events).

The output is a judgment list in a CSV file generated from these UBI documents using the COEC algorithm. This judgment list can be used with Eland to extract relevant features and train an LTR model.

Quick start

To generate a judgment list from the sample data in this blog, you can follow these steps:

1. Clone the repository:

2. Install required libraries

For this script, we need the following libraries:

• pandas: to save the judgment list

• elasticsearch: To get the UBI data from our Elastic deployment

We also need Python 3.11

3. Update the environment variables for your Elastic deployment in a .env file

• ES_HOST

• API_KEY

To add the environment variables, use:

4. Create the ubi_queries, ubi_events indices, and upload the sample data. Run the setup.py file:

5. Run the Python script:

If you follow these steps, you should see a new file called judgment_list.csv that looks like this:

This script calculates the grades applying the COEC algorithm discussed before using the calculate_relevance_grade() function that is shown below.

Data architecture

Ubi queries

Our UBI queries index has information about the queries executed in our search system. This is a sample document:

Here we can see data from the user (client_id), from the results of the query (query_response_object_ids), and the query itself (timestamp, user_query)

Ubi click events

Our ubi_events index has data from each time a user clicked a document in the results. This is a sample document:

Judgment list generation script

General script overview

This script automates the generation of the judgment list using UBI data from Queries and Click events stored in Elasticsearch. It executes these tasks:

• Fetches and processes the UBI data in Elasticsearch.

• Correlates UBI events with its queries.

• Calculates the CTR for each position.

• Calculates the expected clicks (EC) for each document.

• Counts the actual clicks for each document.

• Calculates the COEC score for each query-document pair.

• Generates a judgment list and writes it in a CSV file.

Let’s go over each function:

connect_to_elasticsearch()

This function returns an Elasticsearch client object using the host and api key.

fetch_ubi_data()

This function is the data extraction layer; it connects with Elasticsearch to fetch UBI queries using a match_all query and filters UBI events to get ‘CLICK_THROUGH’ events only.

process_ubi_data()

This function handles the judgment list generation. It starts processing the UBI data by associating UBI events and queries. Then it calls the calculate_relevance_grade() function for each document-query pair to obtain the entries for the judgment list. Finally, it returns the resulting list as a pandas dataframe.

calculate_relevance_grade()

This is the function that implements the COEC algorithm. It calculates the CTR for each position, then it compares the actual clicks for a document-query pair, and finally calculates the actual COEC score for each one.

generate_judgment_statistics()

It generates useful statistics from the judgment list, such as total queries, total unique documents, or the grade distribution. This is purely informational and does not change the resulting judgment list.

Results and impact

If you follow the instructions in the Quick start section, you should see a resulting CSV file containing a judgment list with 320 entries (you can see a sample output in the repo). With these fields:

• qid: unique ID of the query

• docid: unique identifier for a resulting document

• grade: the calculated grade for the query-document pair

• query: The user query

Let’s look at the results for the query “Italian recipes”:

qid

docid

grade

query

q1-italian-recipes

recipe_pasta_basics

0.0

Italian recipes

q1-italian-recipes

recipe_pizza_margherita

3.333333

Italian recipes

q1-italian-recipes

recipe_risotto_guide

10.0

Italian recipes

q1-italian-recipes

recipe_french_croissant

0.0

Italian recipes

q1-italian-recipes

recipe_spanish_paella

0.0

Italian recipes

q1-italian-recipes

recipe_greek_moussaka

1.875

Italian recipes

We can see from the results that for the query “Italian recipes”:

• The risotto recipe is definitely the best result for the query, receiving 10 times more clicks than expected

• Pizza Margherita is a great result too.

• The Greek mousaka (surprisingly) is a good result as well and performs better than its position on the results would suggest. This means a few users looking for Italian recipes got interested in this recipe instead. Maybe these users are interested in Mediterranean dishes in general. At the end, what this tells us is that this could be a good result to be shown under the other two ‘better’ matches we discussed above.

Conclusion

Using UBI data lets us automate the training of LTR models, creating high-quality judgment lists from our own users. UBI data provides a big dataset that reflects how our search system is being used.By using the COEC algorithm to generate the grades, we account for inherent bias while at the same time, it reflects what a user considers a better result. The method outlined here can be applied to real use cases to provide a better search experience that evolves with real usage trends.

In this article we'll take a look at the new code generation feature in the Kibana Dev Console, and how it can significantly reduce your development effort by generating ready to use code for you.

This feature is available in our Serverless platform and in Elastic Cloud and self-hosted releases 8.16 and up.

The Kibana Dev Console

This section provides a quick introduction to the Kibana Dev Console, in case you have never used it before. Skip to the next section if you are already familiar with it.

While you are in any part of the Search section in Kibana, you will notice a "Console" link at the bottom of your browser's page:

When you click this link, the Console expands to cover the page. Click it again to collapse it.

In the left-side panel of the Dev Console, you can enter Elasticsearch requests, with the help of an interactive editor that provides auto-completion and checks your syntax. Some example requests are already pre-populated so that you have something to start experimenting with.

When the cursor is on a request, a "play" button appears to its right. You can click this button to send the request to your Elasticsearch server.

After you execute a request, the response from the server appears in the panel on the right.

Code Export feature in Kibana Dev Console

The Dev Console makes it easy to prototype your requests or queries until you get exactly what you want. But what happens next? If you need to convert the request to code so that you can incorporate it into your application, then you can save time using the new code export feature.

Next to the Play button you will find the three dot or "kebab" button, which opens a menu of options. The first option provides access to the code export feature. If you've never used this feature before, it will appear with a "Copy as curl" label.

If you select this option, your clipboard will be loaded with a curl command that is equivalent to the selected request.

Now, things get more interesting when you click the "Change" link, which allows you to switch to a different target language. In this initial release, the code export adds support for Python and JavaScript. More languages are expected to be added in future releases.

You can now select your desired language and click "Copy code" to put the exported code in your clipboard. You can also change the default language that is offered in the menu.

The exported code is a complete script in the selected language, using the official Elasticsearch client for that language. Here is an example of how the PUT /my-index request shown above looks when exported to the Python language:

To use the exported code follow these steps:

• Paste the code from the clipboard to a new file with the correct extension (.py for Python, or .js for JavaScript).

• In your terminal, add an environment variable called ELASTIC_API_KEY with a valid API Key for your Elasticsearch cluster. You can create an API key right in Kibana if you don't have one yet.

• Execute the script with the python or node commands depending on your language, making sure the official Elasticsearch client is installed.

Now you are ready to adapt the exported code as needed to integrate it into your application!

Conclusion

In this article you have learned about the new Code Export feature in the Kibana Dev Console. We hope this feature will streamline your development process with Elasticsearch!

ES|QL to Pandas dataframes in Python

Importing test data

First, let's import some test data. As before, we will be using the employees sample data and mappings. The easiest way to load this dataset is to run these two Elasticsearch API requests in the Kibana Console.

Converting dataset to a Pandas DataFrame object

OK, with that out of the way, let's convert the full employees dataset to a Pandas DataFrame object using the ES|QL Arrow export:

Even though this dataset only contains 100 records, we use a LIMIT command to avoid ES|QL warning us about potentially missing records. This prints the following dataframe:

OK, so what actually happened here?

• Given format="arrow", Elasticsearch returns binary Arrow streaming data

• The Elasticsearch Python client looks at the Content-Type header and creates a PyArrow object

• Finally, PyArrow's Pandas integration converts the PyArrow object to a Pandas dataframe.

Note that the types_mapper=pd.ArrowDtype parameter asks Pandas to use a PyArrow backend instead of a NumPy backend, since the source data is PyArrow. While this backend is not enabled by default for compatibility reasons, it has many advantages: it handles missing values, is faster, more interopable and supports more types. (This is not a zero copy conversion, however.)

For this example to work, the Pandas and PyArrow optional dependencies need to be installed. If you want to use another dataframe library such as Polars instead, you don't need Pandas and can directly use polars.from_arrow to create a Polars DataFrame from the PyArrow table returned by the Elasticsearch client.

One limitation is that Elasticsearch does not currently handle multi-valued fields, which is why we had to drop the is_rehired, job_positions and salary_change columns. This limitation will be lifted in a future version of Elasticsearch.

Anyway, you now have a Pandas dataframe that you can use to analyze your data further. But you can also continue massaging the data using ES|QL, which is particularly useful when queries return more than 10,000 rows, the current maximum number of rows that ES|QL queries can return.

More complex queries

In the next example, we're counting how many employees are speaking a given language by using STATS ... BY (not unlike GROUP BY in SQL). And then we sort the result with the languages column using SORT:

Unlike with CSV, we did not have to specify any types, as Arrow data already includes types. Here's the result:

21 employees speak 5 languages, wow! And 10 employees did not declare any spoken language. The missing value is denoted by <NA>, which is consistently used for missing data with the PyArrow backend. If we had used the NumPy backend instead, this column would have been converted to floats and the missing value would have been a confusing NaN, as NumPy integers don't have any sentinel value for missing data.

Queries with parameters

Finally, suppose that you want to expand the query from the previous section to only consider employees that speak N or more languages, with N being a variable parameter. For this we can use ES|QL's built-in support for parameters, which eliminates the risk of an injection attack associated with manually assembling queries with variable parts:

which prints the following:

Conclusion

As we saw, ES|QL's native Arrow support makes working with Pandas and other DataFrame libraries even nicer than using CSV and it will continue to improve over time, with the multi-value support coming in a future version of Elasticsearch.

Additional resources

If you want to learn more about ES|QL, the ES|QL documentation is the best place to start. You can also check out this other Python example using Boston Celtics data. To know more about the Python Elasticsearch client itself, you can refer to the documentation, ask a question on Discuss with the language-clients tag or open a new issue if you found a bug or have a feature request. Thank you!

If you are reading this, you probably want to know how rich you can get before actually reaching the same odds of being hit by a bus. I can't blame you, I want to know too. Let's work out the odds so that we can make sure we win the lottery rather than get in an accident!

What we are going to see in this blog is figuring out the probability of being hit by a bus and the probability of achieving wealth. We'll then compare both and understand until what point your chances of getting rich are higher, and when you should consider getting life insurance.

So how are we going to do that? This is going to be a mix of magic numbers pulled from different articles online, some synthetics data and the power of ES|QL, the new Elasticsearch Query Language. Let's get started.

Data for the ES|QL analysis

The magic number

The challenge starts here as the dataset is going to be somewhat challenging to find. We are then going to assume for the sake of the example that ChatGPT is always right. Let’s see what we get for the following question:

Cough Cough… That sounds about right, this is going to be our magic number.

Generating the wealth data

Prerequisites

Before running any of the scripts below, make sure to install the following packages:

Now, there is one more thing we need, a representative dataset with wealth distribution to compute wealth probability. There is definitely some portion of it here and there, but again, for the example we are going to generate a 500K line dataset with the below python script. I am using python 3.11.5 in this example:

It should take some time to run depending on your configuration since we are injecting 500K documents here!

FYI, after playing with a couple of versions of the script above and the ESQL query on the synthetic data, it was obvious that the net worth generated across the population was not really representative of the real world. So I decided to use a log-normal distribution (np.random.lognormal) for income to reflect a more realistic spread where most people have lower incomes, and fewer people have very high incomes.

Net Worth Calculation: Used a combination of random multipliers (np.random.uniform(0.5, 5)) and additional noise (np.random.normal(0, 10000)) to calculate net worth. Added a check to ensure no negative net worth values by using np.maximum(0, net_worths).

Not only have we generated 500K documents, but we also used the Elasticsearch python client to bulk ingest all these documents in our deployment. Please note that you will find the endpoint to pass in as hosts Cloud ID in the code above.

For the deployment API key, open Kibana, and generate the key in Stack Management / API Keys:

The good news is that if you have a real data set, all you will need to do is to change the above code to read your dataset and write documents with the same data mapping.

Ok we're getting there! The next step is pouring our wealth distribution.

ES|QL wealth analysis

Introducing ES|QL: A powerful tool for data analysis

The arrival of Elasticsearch Query Language (ES|QL) is very exciting news for our users. It largely simplifies querying, analyzing, and visualizing data stored in Elasticsearch, making it a powerful tool for all data-driven use cases.

ES|QL comes with a variety of functions and operators, to perform aggregations, statistical analyses, and data transformations. We won’t address them all in this blog post, however our documentation is very detailed and will help you familiarize with the language and the possibilities.

To get started with ES|QL today and run the blog post queries, simply start a trial on Elastic Cloud, load the data and run your first ES|QL query.

Understanding the wealth distribution with our first query

To get familiar with the dataset, head to Discover in Kibana and switch to ES|QL in the dropdown on the left hand side:

Let’s fire our first request:

As you could expect from our indexing script earlier, we are finding the documents we bulk ingested, notice the simplicity of pulling data from a given dataset with ES|QL where every query starts with the From clause, then your index.

In the query above given we have 500K lines, we limited the amount of returned documents to 10. To do this, we are passing the output of the first segment of the query via a pipe to the limit command to only get 10 results. Pretty intuitive, right?

Alright, what would be more interesting is to understand the wealth distribution in our dataset, for this we will leverage one of the 30 functions ES|QL provides, namely percentile.

This will allow us to understand the relative position of each data point within the distribution of net worth. By calculating the median percentile (50th percentile), we can gauge where an individual’s net worth stands compared to others.

Like our first query, we are passing the output of our index to another function, Stats, which combined with the percentile function will output the median net worth:

The median is about 54K, which unfortunately is probably optimistic compared to the real world, but we are not going to solve this here. If we go a little further, we can look at the distribution in more granularity by computing more percentiles:

With the below output:

The data reveals a significant disparity in wealth distribution, with the majority of wealth being concentrated among the richest individuals. Specifically, the top 5% (95th percentile) possess a disproportionately large portion of the total wealth, with a net worth starting at $852,988.26 and increasing dramatically in the higher percentiles.

The 99th percentile individuals hold a net worth exceeding $2 million, highlighting the skewed nature of wealth distribution. This indicates that a substantial portion of the population has modest net worth, which is probably what we want for this example.

Another way to look at this is to augment the previous query and grouping by age to see if there is, (in our synthetic dataset), a relation between wealth and age:

This could be visualized in a Kibana dashboard. Simply:

• Navigate to Dashboard

• Add a new ES|QL visualization

• Copy and paste our query

• Move the age field to the horizontal axis in the visualization configuration

Which will output:

The above suggests that the data generator randomized wealth uniformly across the population age, there is no specific trend pattern we can really see.

Median Absolute Deviation (MAD)

We calculate the median absolute deviation (MAD) to measure the variability of net worth in a robust manner, less influenced by outliers.

With a median net worth of 44,205.44, we can infer the typical range of Net Worth: Most individuals’ net worth falls within a range of 9,581.78 to $97,992.66.

The statistical showdown between Net Worth and Bus Collision

Alright, this is the moment to understand how rich we can get, based on our dataset, before getting hit by a bus. To do that, we are going to leverage ES|QL to pull our entire dataset in chunks and load it into a pandas dataframe to build a net worth probability distribution. Finally, we will determine where the ends meet between the net worth and bus collision probabilities.

The entire Python notebook is available here. I also recommend you read this blog post which walks you through using ES|QL with pandas dataframes.

Helper functions

As you can see in the previously referred blog post, we introduced support for ES|QL since version 8.12 of the Elasticsearch python client. Thus our notebook first defines the below functions:

The first function is straightforward and executes an ES|QL query, the second is fetching the entire dataset from our index. Notice the trick in there that I am using a counter built-in to a field in my index to paginate through the data. This is workaround I am using while our engineering team is working on the support for pagination in ES|QL.

Next, knowing that we have 500K documents in our index, we simply call these function to load the data in a data frame:

Fit Pareto distribution

Next, we fit our data to a Pareto distribution, which is often used to model wealth distribution because it reflects the reality that a small percentage of the population controls most of the wealth. By fitting our data to this distribution, we can more accurately represent the probabilities of different net worth levels.

We can visualize the pareto distribution with the code below: ``

Breaking point

Finally, with the calculated probability, we determine the target net worth corresponding to the bus hit probability and visualize it. Remember, we use the magic number ChatGPT gave us for the probability of getting hit by a bus:

Conclusion

Based on our synthetic dataset, this chart vividly illustrates that the probability of amassing a net worth of approximately $12.5 million is as rare as the chance of being hit by a bus. For the fun of it, let’s ask ChatGPT what the probability is:

Okay… $439 million? I think ChatGPT might be hallucinating again.

In this blog, you will learn how to:

• Create an Elasticsearch Serverless project

• Create an Inference Endpoint to generate embeddings with ELSER

• Use a Semantic Text field for auto-chunking and calling the Inference Endpoint

• Use the Open Crawler to crawl blogs

• Connect to an LLM using Elastic’s Playground to test prompts and context settings for a RAG chat application.

If you want to jump right into the code, you can view the accompanying Jupyter Notebook here.

ChatGPT and Elasticsearch (April 2023)

A lot has changed since I wrote the initial ChatGPT and Elasticsearch: OpenAI meets private data. Most people were just playing around with ChatGPT, if they had tried it at all. And every booth at every tech conference didn’t feature the letters “AI” (whether it is a useful fit or not).

Updates in Elasticsearch (August 2024)

Since then, Elastic has embraced being a full featured vector database and is putting a lot of engineering effort into making it the best vector database option for anyone building a search application. So as not to spend several pages talking about all the enhancements to Elasticsearch, here is a non-exhaustive list in no particular order:

• ELSER - The Elastic Learned Sparse Encoder

• Elastic Serverless Service was built and is in public beta

• Elasticsearch open Inference API

  • Embeddings

  • Chat completion

  • Semantic rerankers

• Semantic_text type - Simplify semantic search

  • Automatic chunking

• Playground - Visually experiment with RAG application building in Elasticsearch

• Retrievers

• Open web crawler

With all that change and more, the original blog needs a rewrite. So let’s get started.

Updated flow: ChatGPT, Elasticsearch & RAG

The plan for this updated flow will be:

1. Setup

   1. Create a new Elasticsearch serverless search project

   2. Create an embedding inference API using ELSER

   3. Configure an index template with a semantic_text field

   4. Create a new LLM connector

   5. Configure a chat completion inference service using our LLM connector

2. Ingest and Test

   1. Crawl the Elastic Labs sites (Search, Observability, Security) with the Elastic Open Web Crawler.

   2. Use Playground to test prompts using our indexed Labs content

3. Configure and deploy our App

   1. Export the generated code from Playground to an application using FastAPI as the backend and React as the front end.

   2. Run it locally

   3. Optionally deploy our chatbot to Google Cloud Run

Setup

Elasticsearch Serverless Project

We will be using an Elastic serverless project for our chatbot. Serverless removes much of the complexity of running an Elasticsearch cluster and lets you focus on actually using and gaining value from your data. Read more about the architecture of Serverless here.

If you don’t have an Elastic Cloud account, you can create a free two-week trial at elastic.co (Serverless pricing available here). If you already have one, you can simply log in.

Once logged in, you will need to create a cloud API key.

NOTE: In the steps below, I will show the relevant parts of Python code. For the sake of brevity, I’m not going to show complete code that will import required libraries, wait for steps to complete, catch errors, etc.

For more robust code you can run, please see the accompanying Jypyter notebook!

Create Serverless Project

We will use our newly created API key to perform the next setup steps.

First off, create a new Elasticsearch project.

• url - This is the standard Serverless endpoint for Elastic Cloud

• project_data - Your Elasticsearch Serverless project settings

  • name - Name we want for the project

  • region_id - Region to deploy

  • optimized_for - Configuration type - We are using vector which isn’t strictly required for the ELSER model but can be suitable if you select a dense vector model such as e5.

Create Elasticsearch Python client

One nice thing about creating a programmatic project is that you will get back the connection information and credentials you need to interact with it!

ELSER Embedding API

Once the project is created, which usually takes less than a few minutes, we can prepare it to handle our labs’ data.

The first step is to configure the inference API for embedding. We will be using the Elastic Learned Sparse Encoder (ELSER).

• Command to create the inference endpoint

• Specify this endpoint will be for generating sparse embeddings

• model_config - Settings we want to use for deploying our semantic reranking model

  • service - Use the pre-defined elser inference service

  • service_settings.num_allocations - Deploy the model with 8 allocations

  • service_settings.num_threads - Deploy with one thread per allocation

• inference_id - The name you want to give to you inference endpoint

• task_type- Specifies this endpoint will be for generating sparse embeddings

This single command will trigger Elasticsearch to perform a couple of tasks:

1. It will download the ELSER model.

2. It will deploy (start) the ELSER model with eight allocations and one thread per allocation.

3. It will create an inference API we use in our field mapping in the next step.

Index Mapping

With our ELSER API created, we will create our index template.

• index_patterns - The pattern of indices we want this template to apply to.

• body - The main content of a web page the crawler collects will be written to

  • type - It is a text field

  • copy_to - We need to copy that text to our semantic text field for semantic processing

• semantic_body is our semantic text field

  • This field will automatically handle chunking of long text and generating embeddings which we will later use for semantic search

  • inference_id specifies the name of the inference endpoint we created above, allowing us to generate embeddings from our ELSER model

• headings - Heading tags from the html

• id - crawl id for this document

• meta_description - value of the description meta tag from the html

• title is the title of the web page the content is from

Other fields will be indexed but auto-mapped. The ones we are focused on pre-defining in the template will not need to be both keyword and text type, which is defined automatically otherwise.

Most importantly, for this guide, we must define our semantic_text field and set a source field to copy from with copy_to. In this case, we are interested in performing semantic search on the body of the text, which the crawler indexes into the body.

Crawl All the Labs!

We can now install and configure the crawler to crawl the Elastic * Labs. We will loosely follow the excellent guide from the Open Crawler released for tech-preview Search Labs blog.

The steps below will use docker and run on a MacBook Pro. To run this with a different setup, consult the Open Crawler Github readme.

Clone the repo

Open the command line tool of your choice. I’ll be using Iterm2. Clone the crawler repo to your machine.

Build the crawler container

Run the following command to build and run the crawler.

Configure the crawler

Create a new YAML in your favorite editor (vim):

We want to crawl all the documents on the three labs’ sites, but since blogs and tutorials on those sites tend to link out to other parts of elastic.co, we need to set a couple of runs to restrict the scope. We will allow crawling the three paths for our site and then deny anything else.

Paste the following in the file and save

Copy the configuration into the Docker container:

Validate the domain

Ensure the config file has no issues by running:

Start the crawler

When you first run the crawler, processing all the articles on the three lab sites may take several minutes.

Confirm articles have been indexed

We will confirm two ways.

First, we will look at a sample document to ensure that ELSER embeddings have been generated. We just want to look at any doc so we can search without any arguments:

Ensure you get results and then check that the field body contains text and semantic_body.inference.chunks.0.embeddings contains tokens.

We can check we are gathering data from each of the three sites with a terms aggregation:

You should see results that start with one of our three site paths.

To the Playground!

With our data ingested, chunked, and inference, we can start working on the backend application code that will interact with the LLM for our RAG app.

LLM Connection

We need to configure a connection for Playground to make API calls to an LLM. As of this writing, Playground supports chat completion connections to OpenAI, AWS Bedrock, and Google Gemini. More connections are planned, so check the docs for the latest list.

When you first enter the Playground UI, click on “Connect to an LLM”

Since I used OpenAI for the original blog, we’ll stick with that. The great thing about the Playground is that you can switch connections to a different service, and the Playground code will generate code specifically to that service’s API specification. You only need to select which one you want to use today.

In this step, you must fill out the fields depending on which LLM you wish to use. As mentioned above, since Playground will abstract away the API differences, you can use whichever supported LLM service works for you, and the rest of the steps in this guide will work the same.

If you don’t have an Azure OpenAI account or OpenAI API account, you can get one here (OpenAI now requires a $5 minimum to fund the API account).

Once you have completed that, hit “Save,” and you will get confirmation that the connector has been added. After that, you just need to select the indices we will use in our app. You can select multiple, but since all our crawler data is going into elastic-labs, you can choose that one.

Click “Add data sources” and you can start using Playground!

Select the “restaurant_reviews” index created earlier.

Playing in the Playground

After adding your data source you will be in the Playground UI.

To keep getting started as simple as possible, we will stick with all the default settings other than the prompt. However, for more details on Playground components and how to use them, check out the Playground: Experiment with RAG applications with Elasticsearch in minutes blog and the Playground documentation.

Experimenting with different settings to fit your particular data and application needs is an important part of setting up a RAG-backed application.

The defaults we will be using are:

• Querying the semantic_body chunks

• Using the three nearest semantic chunks as context to pass to the LLM

Creating a more detailed prompt

The default prompt in Playground is simply a placeholder. Prompt engineering continues to develop as LLMs become more capable. Exploring the ever-changing world of prompt engineering is a blog, but there are a few basic concepts to remember when creating a system prompt:

• Be detailed when describing the app or service the LLM response is part of. This includes what data will be provided and who will consume the responses.

• Provide example questions and responses. This technique, called few-shot-prompting, helps the LLM structure its responses.

• Clearly state how the LLM should behave.

• Specify the Desired Output Format.

• Test and Iterate on Prompts.

With this in mind, we can create a more detailed system prompt:

Feel free to to test out different prompts and context settings to see what results you feel are best for your particular data. For more examples on advanced techiques, check out the Prompt section on the two part blog Advanced RAG Techniques. Again, see the Playground blog post for more details on the various settings you can tweak.

Export the Code

Behind the scenes, Playground generates all the backend chat code we need to perform semantic search, parse the relevant contextual fields, and make a chat completion call to the LLM. No coding work from us required!

In the upper right corner click on the “View Code” button to expand the code flyout

You will see the generated python code with all the settings your configured as well as the the functions to make a semantic call to Elasticsearch, parse the results, built the complete prompt, make the call to the LLM, and parse those results.

Click the copy icon to copy the code.

You can now incorporate the code into your own chat application!

Wrapup

A lot has changed since the first iteration of this blog over a year ago, and we covered a lot in this blog. You started from a cloud API key, created an Elasticsearch Serverless project, generated a cloud API key, configured the Open Web Crawler, crawled three Elastic Lab sites, chunked the long text, generated embeddings, tested out the optimal chat settings for a RAG application, and exported the code!

Where’s the UI, Vestal?

Be on the lookout for part two where we will integrate the playground code into a python backend with a React frontend. We will also look at deploying the full chat application.

For a complete set of code for everything above, see the accompanying Jypyter notebook

The code that accompanies this article implements a database of famous quotes. It includes a back end written in Python with the FastAPI web framework, and a front end written in TypeScript and React. Regarding vector search, this application demonstrates how to:

• run a local Elasticsearch service using Docker,

• bulk-ingest a large number of documents efficiently,

• generate vector embeddings for documents as they are ingested,

• leverage the power of a GPU to accelerate the generation of vector embeddings through parallelization,

• run vector search queries using the approximate kNN algorithm,

• aggregate results from vector search,

• compare vector search results against those resulting from a standard match (BM25) query.

Below you can see a screenshot of the application. In this article you will find a detailed explanation of how the ingest and search features work. You then have the option to install and run the code on your own computer to experiment and learn!

What is the Elasticsearch-DSL client for Python?

Sometimes called the "high-level" Python client, Elasticsearch-DSL offers idiomatic (or "Pythonic") access to your Elasticsearch database, in contrast with the official (or "low-level") Python client, which provides direct access to the complete range of Elasticsearch features and endpoints.

When using Elasticsearch-DSL, the structure (or "mappings") of Elasticsearch indices are defined as classes, with a syntax that is similar to that of Python dataclasses. The documents stored in these indices are represented by instances of these classes. All the transformations that are necessary to map between Python objects and Elasticsearch documents are automatically and transparently carried out, resulting in application code that is simple and idiomatic.

To add Elasticsearch-DSL to your Python project, you can install it with pip:

If your project is asynchronous, then there are additional dependencies that need to be installed, so in that case use the following command instead:

Index definition

As stated above, with Elasticsearch-DSL the structure of an Elasticsearch index is defined as a Python class. The example application featured in this article uses a dataset of famous quotes that have the following fields:

• quote: the text of the quote, as a string

• author: the name of the author, as a string

• tags: a list of tag names that apply to the quote, each a string

As part of this application we are going to add one additional field, the vector embedding that we will use to search for quotes:

• embedding: a list of floating point numbers representing a vector embedding for the quote

Let's write an initial document class to describe our famous quotes index:

The AsyncDocument class that is used as a base class for our QuoteDoc class implements all the functionality to connect the class to an Elasticsearch index. The choice of an asynchronous document base class was made because this examples uses the FastAPI web framework, which is also asynchronous. For projects that do not use asynchronous Python, the Document base class must be used when declaring document classes.

The name attribute given in the Index inner class defines the name of the Elasticsearch index that will be used with documents of this class.

If you have used Python dataclasses before, you likely find the way fields are defined very familiar, with each field being given a Python type hint. These Python types are mapped to the closest Elasticsearch type, so for example, in the case of str, the corresponding field in the Elasticsearch index will be given the type text, the standard type that is used for text that needs to be indexed for full-text search, while float is mapped to the equally named float on the Elasticsearch side.

While it can be useful to leave the quote field as is so that we can use it for both vector and full-text searches, the author and tags fields do not really need all the extra work associated with full-text search. The best Elasticsearch type for these fields is keyword, which just stores the text, without doing any indexing. Likewise, the embedding field is not just a simple list of floating point numbers, we are going to use it for vector search, which is a behavior associated with the dense_vector type in Elasticsearch.

To assign a type override to a field, we add an assignment with the mapped_field() function, as shown in the improved version of the QuoteDoc class that follows:

As you can see in this updated version, the elasticsearch_dsl package includes classes such as Keyword and DenseVector to represent all the native Elasticsearch field types.

Did you notice the init=False argument given in this new definition of the embedding field? If you are familiar with Python dataclasses you may recognize init as one of the options available in the dataclasses field() function, used to indicate that the given attribute should be omitted from the constructor for instances of the class. The behavior is the same here, which means that when creating an instance of QuoteDoc, this argument should not be given.

How will the vector embeddings be generated if they will not be passed down to the document constructor? Elasticsearch-DSL always calls the clean() method in all documents before serializing them and sending them to Elasticsearch. This method is a convenience entry point where the application can add any custom field processing logic. For example, fields that are optional or auto-generated can be added in this method. Here is the final version of the QuoteDoc document class, including the logic that generates the embeddings:

For this example we are going to use embeddings from a SentenceTransformers model. These embeddings are easy to generate locally and being open source and free they are convenient to use when experimenting. The all-MiniLM-L6-v2 model is a great general purpose embedding model for English text. There are many other models that are also compatible with the SentenceTransformers framework, so feel free to use a different one if you prefer.

The clean() method can be used for more advanced use cases as well. For example, it is common when working with large bodies of text to split the text into smaller chunks, and then generate embeddings for each chunk. Elasticsearch accommodates this use case through nested objects. If you want to see an advanced example that implements this type of solution, check out the vectors example in the Elasticsearch-DSL repository.

Document ingestion

With the structure of the index in place, we can now create the index. This is done with the init() class method:

In many cases it is useful to delete a previously existing index to make sure an ingest process begins from a clean starting point. This can be done using the _index class attribute, which provides access to the Elasticsearch index, along with its exists() and delete() methods:

The example dataset used by the example application is a collection of almost 37,000 famous quotes. It comes as a CSV file with the quote, author and tags columns. The tags are given as a comma-separated string. The dataset is available for download from the example GitHub repository.

To ingest the data contained in this dataset, Python's csv module can be used:

The csv.DictReader class creates a CSV file importer that returns a dictionary for each row in the data file. For each row, we create a QuoteDoc instance and pass the quote, author and tags in the constructor. For the tags, the string that is read from the CSV file has to be split into a list, which is how it will be stored in the Elasticsearch index.

To write a document to the index, the save() method is invoked. This method will call the document's clean() method, which in turn will generate the vector embedding for the quote.

Starting an Elasticsearch instance

Before the above ingest script can be executed, you need to have access to a running instance of Elasticsearch. By far the easiest (and also 100% free) way to do this is with a Docker container.

To start a single-node Elasticsearch service on your computer first make sure you have Docker running, and then execute the following command:

To make sure you are running the latest and greatest version, open the release notes page to find out what is the current version, then replace the version number in the last line of the above command.

The -v option in the command above sets up a mapping between a directory named data in your local system and the data directory in the Elasticsearch container. All the data files used by Elasticsearch will be saved in this directory, so that in case you need to restart your container you do not lose any data. If you prefer to not store the data files in your computer, then you can remove the -v line and the data will be stored ephemerally in the container.

Note that deploying Elasticsearch using this method is only adequate for local experimentation. If you intend to deploy Elasticsearch on a production server, consider using our Elasticsearch on Docker Compose or Elasticsearch on Kubernetes guides.

Connecting to Elasticsearch

The ingestion script needs to know how to connect to Elasticsearch. If you are running a Docker container as demonstrated in the previous section, add the following line between the imports and the definition of the QuoteDoc class:

To complete the script, the ingest_quotes() function should be called. Add the following snippet at the bottom of your source file:

The asyncio.run() function will launch the asynchronous application. If your application is not asynchronous, then you would just call the ingest function directly.

For your convenience, below you can find the complete code for the script up to this point. You can save this file as search.py. You can find an example of this file here.

Create a virtual environment for your project using the tool of your choice, and then install the dependencies on it:

Make sure you have the quotes.csv file in the current directory, and then start the ingest by running the script:

The script does not print anything, so it will run for a while adding the quotes from the CSV file into your Elasticsearch index. The file has about 37,000 quotes, so expect the process to run for several minutes.

Luckily you do not need to wait that long. If you start the script and no error appears, that is confirmation that everything is working. You can press Ctrl-C to stop it and continue reading to learn about ingest performance.

Performance tuning part 1: bulk processing

If your dataset is small, then the above ingest solution will work just fine, and it has the benefit that it is simple to code and easy to understand.

For larger ingest jobs, however, it is necessary to sacrifice code clarity and pay attention to performance, so let's see what optimizations can be done in this application.

First of all, to evaluate performance we need to be able to measure the performance of the existing solution. Below is the updated ingest_quotes() function, which now calls ingest_progress() every 100 ingested documents to show how many documents have been ingested, along with an average document per second.

This version of the ingest is nicer than the previous one because it prints regular status updates. If you let the script run for a while you may see an output similar to the one below:

The data file has close to 37,000 quotes, so now you can have a good idea of how long the ingest will take. Assuming the average of 97 ingested documents per second holds throughout the entire ingest job, it should take less than 7 minutes to ingest the entire dataset. You can press Ctrl-C to stop this ingest process, there is no need to let it run to completion yet.

Elasticsearch offers a very flexible bulk ingest feature, which is made available in the Elasticsearch-DSL package's bulk() method. Instead of saving each document, the entire import loop can be moved into a generator function which is given to the bulk() method as an argument:

Here the get_next_quote() inner generator function yields QuoteDoc instances. The QuoteDoc.bulk() method will run the generator and issue batch updates to Elasticsearch. With this change, you can expect to see a small speed improvement:

For another small improvement, the JSON serializer used by the Elasticsearch client can be changed to the orjson library, which performs better than Python's own:

This should lead to another small performance improvement:

Performance tuning part 2: GPU accelerated embeddings

You have seen in the previous section that we have obtained some modest performance improvements by processing ingest requests in bulk. But while ingestion requests are now being grouped, the embeddings continue to be generated one by one in the clean() method of the QuoteDoc class.

Is there a way to optimize embedding generation? The SentenceTransformers model uses PyTorch, which in turn uses a GPU if one is available. But the embeddings are generated individually, which does not lead to an optimal utilization of the GPU hardware. GPUs are very good at parallelization, so we can reorganize the ingest function to generate embeddings in batches. And once again the price we pay for this comes in increased code complexity.

So we are going to stop using the clean() method to generate document embeddings, and instead we are going to accumulate the QuoteDoc instances in a list, and once we reach a good number we'll generate embeddings for all of them in a single operation.

Let's start by writing a helper function that generates embeddings for a list of QuoteDoc instances:

Note how now the model.encode() method is given a list of quotes to embed instead of a single one. When the input argument is a list, the model generates an embedding for each list element. The method accepts an optional batch_size argument (not used in the example above) that defaults to 32 that can be used to control the size of each batch of samples that are sent to the model for computation. Depending on the GPU hardware you may find that different values of this argument help tune performance to the best possible. Once the embeddings are generated, they are assigned to each quote using a for-loop.

Now the ingest function can be refactored to accumulate quotes and use the helper function to generate embeddings:

In this version of ingest_quotes(), each QuoteDoc instance is added to the quotes list, and when 512 elements have accumulated the embed_quotes() function added above is used to generate the embeddings more efficiently. Once the objects have their embeddings, they are yielded, so that the bulk() method from Elasticsearch-DSL can add them to the index as before.

What is the significance of the 512 number? There isn't any. We know that the model uses a batch size of 32, so it makes sense to accumulate at least that many documents. Starting from 32, you can try if larger powers of 2 provide better performance. With the hardware available to me, I've found 512 to give the best performance.

Here is an example run using batched embeddings:

And now the ingestion process runs much faster, with the entire dataset ingested in about 1 minutes and 16 seconds.

If you decide to try to optimize your ingest, you are encouraged to try different options and see what works best with your hardware.

Querying the index

If you are following along, by now you have an Elasticsearch index called quotes that is populated with about 37K famous quotes, each with a searchable vector embedding. Now it is time to learn how to query this index.

When using Elasticsearch-DSL, the document classes return a search object from their search() method:

The search object has a large number of methods that map to the query options in the Elasticsearch query DSL.

The simplest query that can be issued is the match all query, which returns all the elements. With the class-based approach used by Elasticsearch-DSL, this is how to run the query:

This would obviously print a listing of the entire list of quotes stored in the index, up to 10,000, which is the maximum number of results Elasticsearch returns by default.

In many cases it is useful to request a subset of the results. The search object uses Python style slicing for this. Here is how to request the first 25 results only:

Here is how to request the second page of results, at 25 results per page:

Elasticsearch offers approximate and exact vector search queries, also called k-nearest neighbor (kNN) queries. To run a vector search query with the approximate k-nearest neighbor algorithm, the Knn query should be used:

The Knn query class accepts the field that stores the embeddings and a search vector as arguments. In the above snippet the variable q has the search text entered by the user.

If instead you prefer to run a regular full-text search, the Match query class is used:

Filters

One of the most important benefits of using Elasticsearch as a vector database is that it is a robust database system, and all the options you can expect to have from a database nicely integrates with your vector search queries.

A great example of this is filters. The famous quotes database stores a list of tags for each quote, so it is only natural to have the option to restrict a query to quotes that have a specific tag.

Given a list of tag filters stored in a tags variable, the following snippet configures a search object to only return results that include the given tags using a "terms" filter:

Aggregations

Another example of a useful database function that is fully integrated with vector search is aggregations. Given a query, Elasticsearch can aggregate the tags and provide the counts of quotes per tag.

The next snippet shows how to add a Terms aggregation to an existing query, which will return the 100 most referenced tags in the results:

Recall that the tags field was declared with the Keyword() type, which means that the tags will be stored as is on the index, without any processing. This is required by the Terms aggregation, which will count the occurrences of each tag in the results.

A complete query example

You have seen a few isolated query examples. In this section you can see how they can all be integrated into a function that performs a query in the example application.

The search_quotes() function shown below accepts a query string q, a list of filters tags and a use_knn flag to choose between kNN or full-text search query. It also accepts start and size pagination arguments.

The function decides which of the three queries you've seen above to issue depending on the input arguments. If q is empty, then it selects a "match all" query, and in any other case it selects a kNN or match query depending on the use_knn flag, which the user can control from a checkbox in the application's user interface.

The function returns three results as a tuple:

• a list of QuoteDoc instances that are the search results,

• the tag aggregations as a list of tuples, each with tag name and document count,

• the total number of results, which is useful to show in paginated queries

Here is the complete code of this function:

To be able to access both the search results and the aggregation results, we now issue the request explicitly through the execute() method and store the response is stored in r. The hits attribute of the response object contains the actual search results, and the aggs attribute provides access to the aggregations. The format in which the aggregation results is provided is described in the terms aggregation documentation.

Conclusion

The complete quotes example is available in a GitHub repository that you can install and run on your computer. Follow the instructions on the README.md file to set it up.

You are welcome to use this example to experiment with vector embeddings and Elasticsearch!

Elasticsearch 8.12 and 8.13 brought basic support for geospatial types to ES|QL. This was dramatically enhanced with the addition of geospatial search capabilities in 8.14. More importantly, this support was designed to conform closely to the Simple Feature Access standard from the Open Geospatial Consortium (OGC) used by other spatial databases like PostGIS, making it much easier to use for GIS experts familiar with these standards.

In this blog, we'll show you how to use ES|QL to perform geospatial searches, and how it compares to the SQL and Query DSL equivalents. We'll also show you how to use ES|QL to perform spatial joins, and how to visualize the results in Kibana Maps. Note that all the features described here are in "technical preview", and we'd love to hear your feedback on how we can improve them.

Searching for geospatial data

Let's start with an example query:

This performs a search for any city boundary polygons that intersect with a rectangular search polygon around the Sanya Phoenix International Airport (SYX).

In a sample dataset of airports, cities and city boundaries, this search finds the intersecting polygon and returns the desired fields from the matching document:

abbrev

airport

region

city

city_location

SYX

Sanya Phoenix Int'l

天涯区

Sanya

POINT(109.5036 18.2533)

That was easy! Now compare this to the classic Elasticsearch Query DSL for the same query:

Both queries are reasonably clear in their intent, but the ES|QL query closely resembles SQL. The same query in PostGIS looks like this:

Look back at the ES|QL example. So similar, right?

We've found that existing users of the Elasticsearch API find ES|QL much easier to use. We now expect that existing SQL users, particularly Spatial SQL users, will find that ES|QL feels very familiar to what they are used to seeing.

Why not SQL?

What about Elasticsearch SQL? It has been around for a while and has some geospatial features. However, Elasticsearch SQL was written as a wrapper on top of the original Query API, which meant only queries that could be transpiled down to the original API were supported. ES|QL does not have this limitation. Being a completely new stack allows for many optimizations that were not possible in SQL. Our benchmarks show ES|QL is very often faster than the Query API, particularly with aggregations!

Differences to SQL

Clearly, from the previous example, ES|QL is somewhat similar to SQL, but there are some important differences. For example, ES|QL is a piped query language, starting with a source command like FROM and then chaining all subsequent commands together with the pipe | character. This makes it very easy to understand how each command receives a table of data and performs some action on that table, such as filtering with WHERE, adding columns with EVAL, or performing aggregations with STATS. Rather than starting with SELECT to define the final output columns, there can be one or more KEEP commands, with the last one specifying the final output results. This structure simplifies reasoning about the query.

Focusing in on the WHERE command in the above example, we can see it looks quite similar to the PostGIS example:

ES|QL

PostGIS

Aside from the difference in string quotation characters, the biggest difference is in how we type-cast the string to a spatial type. In PostGIS, we use the ::geometry suffix, while in ES|QL, we use the ::geo_shape suffix. This is because ES|QL runs within Elasticsearch, and the type-casting operator :: can be used to convert a string to any of the supported ES|QL types, in this case, a geo_shape. Additionally, the geo_shape and geo_point types in Elasticsearch imply the spatial coordinate system known as WGS84, more commonly referred to using the SRID number 4326. In PostGIS, this needs to be explicit, hence the use of the SRID=4326; prefix to the WKT string. If that prefix is removed, the SRID will be set to 0, which is more like the Elasticsearch types cartesian_point and cartesian_shape, which are not tied to any specific coordinate system.

Both ES|QL and PostGIS provide type conversion function syntax as well:

ES|QL

PostGIS

OGC functions

Elasticsearch 8.14 introduces the following four OGC spatial search functions:

ES|QL

PostGIS

Description

ST_INTERSECTS

ST_Intersects

Returns true if two geometries intersect, and false otherwise.

ST_DISJOINT

ST_Disjoint

Returns true if two geometries do not intersect, and false otherwise. The inverse of ST_INTERSECTS.

ST_CONTAINS

ST_Contains

Returns true if one geometry contains another, and false otherwise.

ST_WITHIN

ST_Within

Returns true if one geometry is within another, and false otherwise. The inverse of ST_CONTAINS.

These function behave similarly to their PostGIS counterparts, and are used in the same way. For example, ST_INTERSECTS returns true if two geometries intersect and false otherwise. If you follow the documentation links in the above table, you might notice that all the ES|QL examples are within a WHERE clause after a FROM clause, while all the PostGIS examples are using literal geometries. In fact, both platforms support using the functions in any part of the query where they make sense.

The first example in the PostGIS documentation for ST_INTERSECTS is:

The ES|QL equivalent of this would be:

Note how we did not specify the SRID in the PostGIS example. This is because in PostGIS when using the geometry type, all calculations are done on a planar coordinate system, and so if both geometries have the same SRID, it does not matter what the SRID is. In Elasticsearch, this is also true for most functions, however, there are exceptions where geo_shape and geo_point use spherical calculations, as we'll see in the next blog about spatial distance search.

ES|QL versatility

So, we've seen examples above for using spatial functions in WHERE clauses, and in ROW commands. Where else would they make sense? One very useful place is in the EVAL command. This command allows you to evaluate an expression and return the result. For example, let's determine if the centroids of all airports grouped by their country names are within a boundary outlining the country:

The results are expected, the centroid of UK airports are within the UK boundary, and not within the Iceland boundary, and vice versa:

centroid

count

in_uk

in_iceland

within_uk

within_iceland

POINT (-21.946634463965893 64.13187285885215)

1

false

true

false

true

POINT (-2.597342072712148 54.33551226578214)

17

true

false

true

false

POINT (0.04453958108176276 23.74658354606057)

873

false

false

false

false

In fact, these functions can be used in any part of the query where their signature makes sense. They all take two arguments, which are either a literal spatial object or a field of a spatial type, and they all return a boolean value. One important consideration is that the coordinate reference system (CRS) of the geometries must match, or an error will be returned. This means you cannot mix geo_shape and cartesian_shape types in the same function call. You can, however, mix geo_point and geo_shape types, as the geo_point type is a special case of the geo_shape type, and both share the same coordinate reference system. The documentation for each of the functions defined above lists the supported type combinations.

Additionally, either argument can be a spatial literal or a field, in either order. You can even specify two fields, two literals, a field and a literal, or a literal and a field. The only requirement is that the types are compatible. For example, this query compares two fields in the same index:

The query basically asks if the city location is within the city boundary, which should generally be true, but there are always exceptions:

cardinality

count

in_city

few

29

false

many

740

true

A far more interesting question would be whether the airport location is within the boundary of the city that the airport serves. However, the airport location resides in a different index than the one containing the city boundaries. This requires a method to effectively query and correlate data from these two separate indexes.

Spatial joins

ES|QL does not support JOIN commands, but you can achieve a special case of a join using the ENRICH command, which behaves similarly to a 'left join' in SQL. This command operates akin to a 'left join' in SQL, allowing you to enrich results from one index with data from another index based on a spatial relationship between the two datasets.

For example, let's enrich the results from a table of airports with additional information about the city they serve by finding the city boundary that contains the airport location, and then perform some statistics on the results:

This returns the top 5 regions with the most airports, along with the centroid of all the airports that have matching regions, and the range in length of the WKT representation of the city boundaries within those regions:

centroid

count

min_wkt

max_wkt

region

POINT (-32.56093470960719 32.598117914802714)

90

207

207

null

POINT (-73.94515332765877 40.70366442203522)

9

438

438

City of New York

POINT (-83.10398317873478 42.300230911932886)

9

473

473

Detroit

POINT (-156.3020245861262 20.176383580081165)

5

307

803

Hawaii

POINT (-73.88902732171118 45.57078813901171)

4

837

837

Montréal

So, what really happened here? Where did the supposed JOIN occur? The crux of the query lies in the ENRICH command:

This command instructs Elasticsearch to enrich the results retrieved from the airports index, and perform an intersects join between the city_location field of the original index, and the city_boundary field of the airport_city_boundaries index, which we used in a few examples earlier. But some of this information is not clearly visible in this query. What we do see is the name of an enrich policy city_boundaries, and the missing information is encapsulated within that policy definition.

Here we can see that it will perform a geo_match query (intersects is the default), the field to match against is city_boundary, and the enrich_fields are the fields we want to add to the original document. One of those fields, the region was actually used as the grouping key for the STATS command, something we could not have done without this 'left join' capability. For more information on enrich policies, see the enrich documentation. While reading those documents, you will notice that they describe using the enrich indexes for enriching data at index time, by configuring ingest pipelines. This is not required for ES|QL, as the ENRICH command works at query time. It is sufficient to prepare the enrich index with the necessary data and enrich policy, and then use the ENRICH command in your ES|QL queries.

You may also notice that the most commonly found region was null. What could this imply? Recall that I likened this command to a 'left join' in SQL, meaning if no matching city boundary is found for an airport, the airport is still returned but with null values for the fields from the airport_city_boundaries index. It turns out there were 89 airports that found no matching city_boundary, and one airport with a match where the region field was null. This lead to a count of 90 airports with no region in the results. Another interesting detail is the need for the MV_EXPAND command. This is necessary because the ENRICH command may return multiple results for each input row, and MV_EXPAND helps to separate these results into multiple rows, one for each outcome. This also clarifies why "Hawaii" shows different min_wkt and max_wkt results: there were multiple regions with the same name but different boundaries.

Kibana Maps

Kibana has added support for Spatial ES|QL in the Maps application. This means that you can now use ES|QL to search for geospatial data in Elasticsearch, and visualize the results on a map.

There is a new layer option in the add layers menu, called "ES|QL". Like all of the geospatial features described so far, this is in "technical preview". Selecting this option allows you to add a layer to the map based on the results of an ES|QL query. For example, you could add a layer to the map that shows all the airports in the world.

Or you could add a layer that shows the polygons from the airport_city_boundaries index, or even better, how about that complex ENRICH query above that generates statistics for how many airports are in each region?

What's next

You might have noticed in two of the examples above we squeezed in yet another spatial function ST_CENTROID_AGG. This is an aggregating function used in the STATS command, and the first of many spatial analytics features we plan to add to ES|QL. We'll blog about it when we've got more to show!

Before that, we want to tell you more about a particularly exciting feature we've worked on: the ability to perform spatial distance searches, one of the most used spatial search features of Elasticsearch. Can you imagine what the syntax for distance searches might look like? Perhaps similar to an OGC function? Stay tuned for the next blog in this series to find out!

Spoiler alert: Elasticsearch 8.15 has just been released, and spatial distance search with ES|QL is included!

Imagine a multinational company with support tickets and solutions in different languages and wants to take advantage of that knowledge across divisions. Currently, knowledge is limited to the language the agent speaks. Let's fix that!

In this article, I’m going to show you how to test Mixtral’s language capabilities, by creating a multilingual RAG system.

You can follow the notebook to reproduce this article's example here

Steps

1. Creating embeddings endpoint

2. Creating mappings

3. Indexing data

4. Asking questions

Creating embeddings endpoint

Our support tickets for this example will come in English, Spanish, and German. The Mistral embeddings model is not multilingual, but we can generate multilingual embeddings using the e5 model, so we can index text on different languages and manage it as a single source, giving us a much richer context.

To create e5 multilingual embeddings you can use Kibana:

Or the _inference API:

Creating Mappings

For the mappings we will use semantic_text mapping type, which is one of my favorite features. It handles the process of chunking the data, generating embeddings, and querying embeddings for you!

We call the text field super_body because with a single mapping type it will handle chunks and embeddings.

Indexing data

We will index a couple of support tickets with problems and solutions in two languages, and then ask a question about problems within many documents in a third.

The following documents will be added to the index:

1. English Support Ticket: Calendar Sync Issue

Support Ticket #EN1234 Subject: Calendar sync not working with Google Calendar

Description: I'm having trouble syncing my project deadlines with Google Calendar. Whenever I try to sync, I get an error message saying "Unable to connect to external calendar service."

Resolution: The issue was resolved by following these steps:

1. Go to Settings > Integrations

1. Disconnect the Google Calendar integration

1. Clear browser cache and cookies

1. Reconnect the Google Calendar integration

1. Authorize the app again in Google's security settings

The sync should now work correctly. If problems persist, ensure that third-party cookies are enabled in your browser settings.

2. German Support Ticket: File Upload Problem

Support-Ticket #DE5678 Betreff: Datei-Upload funktioniert nicht

Beschreibung: Ich kann keine Dateien mehr in meine Projekte hochladen. Jedes Mal, wenn ich es versuche, bleibt der Ladebalken bei 99% stehen und dann erscheint eine Fehlermeldung.

Lösung: Das Problem wurde durch folgende Schritte gelöst:

1. Überprüfen Sie die Dateigröße. Die maximale Uploadgröße beträgt 100 MB.

1. Deaktivieren Sie vorübergehend den Virenschutz oder die Firewall.

1. Versuchen Sie, die Datei im Inkognito-Modus hochzuladen.

1. Wenn das nicht funktioniert, leeren Sie den Browser-Cache und die Cookies.

1. Als letzten Ausweg, versuchen Sie einen anderen Browser zu verwenden.

In den meisten Fällen lag das Problem an zu großen Dateien oder an Interferenzen durch Sicherheitssoftware. Nach Anwendung dieser Schritte sollte der Upload funktionieren.

3. Marketing Campaign Ideas (noise)

Q3 Marketing Campaign Ideas

1. Social media contest: "Share Your Productivity Hack"

   • Users share tips using our software, best entry wins a premium subscription.

1. Webinar series: "Mastering Project Management"

   • Invite industry experts to share insights using our tool.

1. Email campaign: "Unlock Hidden Features"

   • Series of emails highlighting lesser-known but powerful features.

1. Partner with a productivity podcast for sponsored content.

1. Create a "Project Management Memes" social media account for lighter, shareable content.

4. Mitarbeiter des Monats (noise)

Mitarbeiter des Monats: Juli 2023

Wir freuen uns, bekannt zu geben, dass Sarah Schmidt zur Mitarbeiterin des Monats Juli gewählt wurde!

Sarah hat außergewöhnliche Leistungen in folgenden Bereichen gezeigt:

• Kundenbetreuung: Sarah hat durchschnittlich 95% positive Bewertungen erhalten.

• Teamarbeit: Sie hat maßgeblich zur Verbesserung unseres internen Wissensmanagementsystems beigetragen.

• Innovation: Sarah hat eine neue Methode zur Priorisierung von Support-Tickets vorgeschlagen, die unsere Reaktionszeiten um 20% verbessert hat.

Bitte gratulieren Sie Sarah zu dieser wohlverdienten Anerkennung!

This is how a document will look like inside Elasticsearch:

Asking questions

Now, we are going to ask a question in Spanish:

The expectation is retrieving documents #1 and #2, then sending them to the LLM as additional context, and finally, getting an answer in Spanish.

Retrieving documents

To retrieve the relevant documents, we can use this nice and short query that will run a search on the embeddings, and return the support tickets most relevant to the question.

Notes about the parameters set: size: 2 Because we know we want the top 2 documents. excludes For clarity in the response. Documents are short so each one will be one chunk long.

Answering the question

Now we can call the Mistral completion API using the Python library to answer the question.

The answer is in perfect Spanish and on point!

Show answer

Hola, me alegra que te hayas comunicado con nosotros. Parece que hay dos problemas distintos.

En cuanto a la sincronización del calendario, puedes seguir estos pasos para resolver el problema:

1. Ve a Configuración > Integraciones

1. Desconecta la integración del Calendario de Google

1. Borra la caché y las cookies del navegador

1. Vuelve a conectar la integración del Calendario de Google

1. Autoriza de nuevo la aplicación en la configuración de seguridad de Google

Si sigues teniendo problemas, asegúrate de que las cookies de terceros están habilitadas en la configuración de tu navegador.

En cuanto al problema de subir un archivo, hay varias cosas que puedes probar:

1. Comprueba el tamaño del archivo. El tamaño máximo de carga es de 100 MB.

1. Desactiva temporalmente el antivirus o el cortafuegos.

1. Intenta cargar el archivo en modo incógnito.

1. Si eso no funciona, borra la caché y las cookies del navegador.

1. Como último recurso, prueba a usar un navegador diferente.

En la mayoría de los casos, el problema se debe a archivos demasiado grandes o a interferencias causadas por software de seguridad. Al seguir estos pasos, deberías poder cargar el archivo correctamente.

¡Espero que esto te ayude a resolver tus problemas! Si tienes alguna otra pregunta, no dudes en preguntar.

Conclusion

Mixtral 8x22B is a powerful model that enables us to leverage data sources in different languages, being able to answer, understand, and translate in many languages. This ability– together with multilingual embeddings– allows you to have multilingual support both in the data retrieval and the answer generation stages, removing language barriers entirely.

If you are interested on reproducing the examples of this article, you can find the Python Notebook with the requests here

Understanding the BEIR benchmark in search relevance evaluation

To improve any system you need to be able to measure how well it is doing. In the context of search BEIR (or equivalently the Retrieval section of the MTEB leaderboard) is considered the “holy grail” for the information retrieval community and there is no surprise in that. It’s a very well-structured benchmark with varied datasets across different tasks. More specifically, the following areas are covered:

• Argument retrieval (ArguAna, Touche2020)

• Open-domain QA (HotpotQA, Natural Questions, FiQA)

• Passage retrieval (MSMARCO)

• Duplicate question retrieval (Quora, CQADupstack)

• Fact-checking (FEVER, Climate-FEVER, Scifact)

• Biomedical information retrieval (TREC-COVID, NFCorpus, BioASQ)

• Entity retrieval (DBPedia)

• Citation prediction (SCIDOCS)

It provides a single statistic, nDCG@10, related to how well a system matches the most relevant documents for each task example in the top results it returns. For a search system that a human interacts with relevance of top results is critical. However, there are many nuances to evaluating search that a single summary statistic misses.

Structure of a BEIR dataset

Each benchmark has three artefacts:

• the corpus or documents to retrieve

• the queries

• the relevance judgements for the queries (aka qrels).

Relevance judgments are provided as a score which is zero or greater. Non-zero scores indicate that the document is somewhat related to the query.

Dataset

Corpus size

#Queries in the test set

#qrels positively labeled

#qrels equal to zero

#duplicates in the corpus

Arguana

8,674

1,406

1,406

0

96

Climate-FEVER

5,416,593

1,535

4,681

0

0

DBPedia

4,635,922

400

15,286

28,229

0

FEVER

5,416,568

6,666

7,937

0

0

FiQA-2018

57,638

648

1,706

0

0

HotpotQA

5,233,329

7,405

14,810

0

0

Natural Questions

2,681,468

3,452

4,021

0

16,781

NFCorpus

3,633

323

12,334

0

80

Quora

522,931

10,000

15,675

0

1,092

SCIDOCS

25,657

1,000

4,928

25,000

2

Scifact

5,183

300

339

0

0

Touche2020

382,545

49

932

1,982

5,357

TREC-COVID

171,332

50

24,763

41,663

0

MSMARCO

8,841,823

6,980

7,437

0

324

CQADupstack (sum)

457,199

13,145

23,703

0

0

Table 1: Dataset statistics. The numbers were calculated on the test portion of the datasets (dev for MSMARCO).

Table 1 presents some statistics for the datasets that comprise the BEIR benchmark such as the number of documents in the corpus, the number of queries in the test dataset and the number of positive/negative (query, doc) pairs in the qrels file. From a quick a look in the data we can immediately infer the following:

• Most of the datasets do not contain any negative relationships in the qrels file, i.e. zero scores, which would explicitly denote documents as irrelevant to the given query.

• The average number of document relationships per query (#qrels / #queries) varies from 1.0 in the case of ArguAna to 493.5 (TREC-COVID) but with a value <5 for the majority of the cases.

• Some datasets suffer from duplicate documents in the corpus which in some cases may lead to incorrect evaluation i.e. when a document is considered relevant to a query but its duplicate is not. For example, in ArguAna we have identified 96 cases of duplicate doc pairs with only one doc per pair being marked as relevant to a query. By “expanding” the initial qrels list to also include the duplicates we have observed a relative increase of ~1% in the nDCG@10 score on average.

Example of duplicate pairs in ArguAna. In the qrels file only the first appears to be relevant (as counter-argument) to query (“test-economy-epiasghbf-pro02a”)

When comparing models on the MTEB leaderboard it is tempting to focus on average retrieval quality. This is a good proxy to the overall quality of the model, but it doesn't necessarily tell you how it will perform for you. Since results are reported per data set, it is worth understanding how closely the different data sets relate to your search task and rescore models using only the most relevant ones. If you want to dig deeper, you can additionally check for topic overlap with the various data set corpuses. Stratifying quality measures by topic gives a much finer-grained assessment of their specific strengths and weaknesses.

One important note here is that when a document is not marked in the qrels file then by default it is considered irrelevant to the query. We dive a little further into this area and collect some evidence to shed more light on the following question: “How often is an evaluator presented with (query, document) pairs for which there is no ground truth information?". The reason that this is important is that when only shallow markup is available (and thus not every relevant document is labeled as such) one Information Retrieval system can be judged worse than another just because it “chooses” to surface different relevant (but unmarked) documents. This is a common gotcha in creating high quality evaluation sets, particularly for large datasets. To be feasible manual labelling usually focuses on top results returned by the current system, so potentially misses relevant documents in its blind spots. Therefore, it is usually preferable to focus more resources on fuller mark up of fewer queries than broad shallow markup.

Leveraging the BEIR benchmark for search relevance evaluation

To initiate our analysis we implement the following scenario (see the notebook):

1. First, we load the corpus of each dataset into an Elasticsearch index.

2. For each query in the test set we retrieve the top-100 documents with BM25.

3. We rerank, the retrieved documents using a variety of SOTA reranking models.

4. Finally, we report the “judge rate” for the top-10 documents coming from steps 2 (after retrieval) and 3 (after reranking). In other words, we calculate the average percentage of the top-10 documents that have a score in the qrels file.

The list of reranking of models we used is the following:

• Cohere's rerank-english-v2.0 and rerank-english-v3.0

• BGE-base

• mxbai-rerank-xsmall-v1

• MiniLM-L-6-v2

Retrieval

Reranking

Dataset

BM25 (%)

Cohere Rerank v2 (%)

Cohere Rerank v3 (%)

BGE-base (%)

mxbai-rerank-xsmall-v1 (%)

MiniLM-L-6-v2 (%)

Arguana

7.54

4.87

7.87

4.52

4.53

6.84

Climate-FEVER

5.75

6.24

8.15

9.36

7.79

7.58

DBPedia

61.18

60.78

64.15

63.9

63.5

67.62

FEVER

8.89

9.97

10.08

10.19

9.88

9.88

FiQa-2018

7.02

11.02

10.77

8.43

9.1

9.44

HotpotQA

12.59

14.5

14.76

15.1

14.02

14.42

Natural Questions

5.94

8.84

8.71

8.37

8.14

8.34

NFCorpus

31.67

32.9

33.91

30.63

32.77

32.45

Quora

12.2

10.46

13.04

11.26

12.58

12.78

SCIDOCS

8.62

9.41

9.71

8.04

8.79

8.52

Scifact

9.07

9.57

9.77

9.3

9.1

9.17

Touche2020

38.78

30.41

32.24

33.06

37.96

33.67

TREC-COVID

92.4

98.4

98.2

93.8

99.6

97.4

MSMARCO

3.97

6.00

6.03

6.07

5.47

6.11

CQADupstack (avg.)

5.47

6.32

6.87

5.89

6.22

6.16

Table 2: Judge rate per (dataset, reranker) pairs calculated on the top-10 retrieved/reranked documents

From Table 2, with the exception of TREC-COVID (>90% coverage), DBPedia (~65%), Touche2020 and nfcorpus (~35%), we see that the majority of the datasets have a labeling rate between 5% and a little more than 10% after retrieval or reranking. This doesn’t mean that all these unmarked documents are relevant but there might be a subset of them -especially those placed in the top positions- that could be positive.

With the arrival of general purpose instruction tuned language models, we have a new powerful tool which can potentially automate judging relevance. These methods are typically far too computationally expensive to be used online for search, but here we are concerned with offline evaluation. In the following we use them to explore the evidence that some of the BEIR datasets suffer from shallow markup.

In order to further investigate this hypothesis we decided to focus on MSMARCO and select a subset of 100 queries along with the top-5 reranked (with Cohere v2) documents which are currently not marked as relevant. We followed two different paths of evaluation: First, we used a carefully tuned prompt (more on this in a later post) to prime the recently released Phi-3-mini-4k model to predict the relevance (or not) of a document to the query. In parallel, these cases were also manually labeled in order to also assess the agreement rate between the LLM output and human judgment. Overall, we can draw the following two conclusions:

• The agreement rate between the LLM responses and human judgments was close to 80% which seems good enough as a starting point in that direction.

• In 57.6% of the cases (based on human judgment) the returned documents were found to be actually relevant to the query. To state this in a different way: For 100 queries we have 107 documents judged to be relevant, but at least 0.576 x 5 x 100 = 288 extra documents which are actually relevant!

Here, some examples drawn from the MSMARCO/dev dataset which contain the query, the annotated positive document (from qrels) and a false negative document due to incomplete markup:

Example 1:

Example 2:

Manually evaluating specific queries like this is a generally useful technique for understanding search quality that complements quantitive measures like nDCG@10. If you have a representative set of queries you always run when you make changes to search, it gives you important qualitative information about how performance changes, which is invisible in the statistics. For example, it gives you much more insight into the false results your search returns: it can help you spot obvious howlers in retrieved results, classes of related mistakes, such as misinterpreting domain-specific terminology, and so on.

Our result is in agreement with relevant research around MSMARCO evaluation. For example, Arabzadeh et al. follow a similar procedure where they employ crowdsourced workers to make preference judgments: among other things, they show that in many cases the documents returned by the reranking modules are preferred compared to the documents in the MSMARCO qrels file. Another piece of evidence comes from the authors of the RocketQA reranker who report that more than 70% of the reranked documents were found relevant after manual inspection.

Update - September 9th: After a careful re-evaluation of the dataset we identified 15 more cases of relevant documents, increasing their total number from 273 to 288

Main takeaways & next steps

• The pursuit for better ground truth is never-ending as it is very crucial for benchmarking and model comparison. LLMs can assist in some evaluation areas if used with caution and tuned with proper instructions

• More generally, given that benchmarks will never be perfect, it might be preferable to switch from a pure score comparison to more robust techniques capturing statistically significant differences. The work of Arabzadeh et al. provides a nice of example of this where based on their findings they build 95% confidence intervals indicating significant (or not) differences between the various runs. In the accompanying notebook we provide an implementation of confidence intervals using bootstrapping.

• From the end-user perspective it’s useful to think about task alignment when reading benchmark results. For example, for an AI engineer who builds a RAG pipeline and knows that the most typical use case involves assembling multiple pieces of information from different sources, then it would be more meaningful to assess the performance of their retrieval model on multi-hop QA datasets like HotpotQA instead of the global average across the whole BEIR benchmark

In the next blog post we will dive deeper into the use of Phi-3 as LLM judge and the journey of tuning it to predict relevance.

This blog builds on that notebook and explores fetch surrounding chunking, an emerging pattern in RAG that uses intelligent chunking and Elasticsearch vector database to optimize LLM responses. The approach balances data input to enhance the accuracy and relevance of LLM-generated answers through semantic hybrid search.

The motivation: A refined approach to RAG data chunking

The primary motivation behind building this notebook was to demonstrate a refined approach to RAG by addressing the challenge of data chunking. Traditional methods often fall short in dynamically adjusting the data size fed to LLMs, either overwhelming the model with too much context or starving it with too little. This notebook aims to strike the right balance, providing just enough information for the LLM to generate precise and contextually relevant responses. However, it must be noted that there is no one-size-fits-all solution.

This method works especially well with books and similar texts where content flows within longer sections or chapters. However, it may require adaptation for texts structured into shorter, distinct sections, such as research papers or articles, where each segment might cover a different topic. In such cases, additional strategies may be necessary to effectively chunk and retrieve related content.

The methodology: Intelligent RAG data chunking

Fetch surrounding chunks

The core idea is to partition the source text into manageable chunks, ensuring each chunk contains just the right amount of information. For this demonstration, I used text from "Harry Potter and the Sorcerer's Stone." The text was partitioned into chapters, and each chapter was further divided into smaller chunks. These chunks, along with their dense and sparse (ELSER) vector representations, were indexed in the Elasticsearch vector database.

Assigning numbers to chunks

Each chunk within a chapter was assigned a sequential integer, allowing us to identify its position. When a matching chunk is found, the chapter number and chunk number are used to retrieve surrounding chunks, providing additional context for the LLM.

Vector database in Elasticsearch

These chunks and their vector representations were ingested into an Elasticsearch Cloud instance. Elasticsearch's robust vector search capabilities make it ideal for hosting these chunks, allowing for efficient retrieval of the most relevant chunks based on the semantic content or text match of a user's query.

AI search

To retrieve the relevant chunks, I employed a hybrid search strategy using dense vector comparisons, sparse vector comparisons, and text search in parallel. This multi-faceted approach ensures that the search results are both semantically rich and contextually accurate. A query is issued to find the matched chunk, which returns the chunk number and chapter. Surrounding chunks for that chapter are then fetched based on the matched chunk.

The RAG pattern

When a query is made, the search flow performs the following steps:

1. Query analysis: The user's query is translated into dense and sparse vectors to retrieve the most relevant chunks from the Elasticsearch index.

2. Chunk retrieval: Using the AI search strategy, the system retrieves the top relevant chunks.

3. Contextual expansion: Adjacent chunks (n-1 and n+1) are also retrieved to provide a more comprehensive context. If the chunk is the last in the chapter, it fetches n-1 and n-2; if it's the first, it fetches n+1 and n+2.

4. LLM response: These intelligently selected chunks are then fed into the LLM, ensuring it receives the optimal amount of information to generate a precise and contextually relevant response.

Why intelligent RAG data chunking matters

This approach addresses a critical aspect of RAG by optimizing the input data fed to LLMs. By leveraging intelligent chunking and hybrid semantic search, this method enhances the accuracy and relevance of the responses generated by LLMs. It showcases a pattern that can be widely applied in various applications within the RAG space, from customer support to content generation and beyond.

Conclusion

This notebook underscores the importance of intelligent data chunking in the RAG framework and demonstrates how Elasticsearch vector database can be leveraged to achieve optimal results. By ensuring the LLM receives just the right amount of information, this methodology paves the way for more accurate and contextually rich responses, enhancing the overall effectiveness of RAG systems.

An index inside Elasticsearch is where you can store your data in documents. While working with an index, the data can quickly grow old if you are working with a dynamic dataset. To avoid this issue, you can create a Python script to update your index and deploy it using Google Cloud Platform's (GCP) Cloud Functions and Cloud Scheduler in order to keep your index up-to-date automatically.

To keep your index current, you can first set up a Jupyter Notebook to test locally and create a framework of a script that will update your index if new information is present. You can adjust your script to make it more reusable and run it as a Cloud Function. With Cloud Scheduler, you can set the code in your Cloud Function to run on a schedule using a cron-type format.

Prerequisites for automating index updates

• This example uses Elasticsearch version 8.12; if you are new, check out our Quick Start on Elasticsearch.

• Download the latest version of Python if you don't have it installed on your machine. This example utilizes Python 3.12.1.

• An API key for NASA's APIs.

• You will use the Requests package to connect to a NASA API, Pandas to manipulate data, the Elasticsearch Python Client to load data into an index and keep it up to date, and Jupyter Notebooks to work with your data interactively while testing. You can run the following line to install these required packages:

Loading and updating your dataset

Before you can run your update script inside of GCP, you will want to upload your data and test the process you will use to keep your script updated. You will first connect to data from an API, save it as a Pandas DataFrame, connect to Elasticsearch, upload the DataFrame into an index, check to see when the index is last updated, and update it if new data is available. You can find the complete code of this section in this search labs notebook.

Loading your data

Let's start testing locally with a Jupyter Notebook to work with your data interactively. To do so, you can run the following in your terminal.

In the right-hand corner, you can select where it says “New” to create a new Jupyter Notebook.

First, you will need to import the packages you will be using. You will import all the packages you installed earlier, plus getpass to work with secrets such as API keys and datetime to work with date objects.

The dataset you will use is Near Earth Object Web Service (NeoWs), a RESTful web service that provides near-earth Asteroid information. This dataset lets you search for asteroids based on their closest approach date to Earth, look up a specific asteroid, and browse the overall dataset.

With the following function, you can connect to NASA's NeoWs API, get data from the past week, and convert your response to a JSON object.

Now, you can save the results of your API call to a variable called response.

To convert the JSON object into a pandas DataFrame, you must normalize the nested objects into one DataFrame and drop the column containing the nested JSON.

To call this function and view the first five rows of your dataset, you can run the following:

Connecting to Elasticsearch

You can access Elasticsearch from the Python Client by providing your Elastic Cloud ID and API key for authentication.

Now, you can save the results of your connection function to a variable called es.

An index in Elasticsearch is the main container for your data. You can name your index called asteroid_data_set.

The result you get back will look like the following:

Now, you can create a helper function that will allow you to convert your DataFrame to the correct format to upload into your index.

Next, you can bulk upload the contents of your DataFrame into Elastic, calling the helper function you just created.

You should get a result that looks similar to the following, which tells you how many rows you’ve uploaded:

When was the last time you updated your data?

Once you've uploaded data into Elastic, you can check the last time your index was updated and format the date so it can work with NASA API.

You can save the date your index was last updated to a variable and print out the date.

Updating your data

Now, you can create a function that checks to see if there is any new data since the last time the index was updated and the current date. If the object is valid and the data is not empty, it will update the index and let you know if there is no new data to update or if the DataFrame returns a type of None indicating that there may have been a problem.

If the DataFrame is a valid object, it will call the function you wrote and update the index if applicable. It will also print out the date of the index's last update to help you debug if needed. If not, it will tell you there may be a problem.

Keeping your index current

Now that you've created a framework for local testing, you are ready to set up an environment where you can run your script daily to check to see if any new data is available and update your index accordingly.

Creating a Cloud Function

You are now ready to deploy your Cloud Function. To do so, you will want to select the environment as a 2nd gen function, name your function, and select a cloud region. You can also tie it to a Cloud Pub/Sub trigger and choose to create a new topic if you haven't made it already. You can check out the complete code for this section on GitHub.

Creating a Pub/Sub topic

When creating a new topic, you can name your topic ID and select the encryption using a Google-managed encryption key.

Setting your Cloud Function's environment variables

Under where it says “Runtime environment variables,” you can add in the environment variables for your NASA_API_KEY, ELASTIC_CLOUD_ID, and ELASTIC_API_KEY. You will want to save these as the raw values without single quotes around them. So if you entered a value of 'xxxxlsdgzxxxxx' into your terminal earlier, you would want it to be xxxxlsdgzxxxxx.

Adjusting your code and adding it to your Cloud Function

After you enter your environment variables, you can press the button that says next, which will take you to a code editor. You will want to select the runtime of Python 3.12.1 or match the version of Python you are using. After that, update the entry point to update_index. The entry point serves a similar role to the main function in Python.

Instead of using getpass to retrieve secrets, you will want to use os to perform a more automated process. An example will look like the following:

You will want to adjust the order of your script to have the function that connects to Elasticsearch first. Afterward, you will want to know when your index was last updated, connect to the NASA API you are using, save it to DataFrame, and load any new data that might be available.

You may notice a new function at the bottom called update_index that ties your code together. In this function, you define the name of your index, connect to Elastic, figure out the last date the index was updated, connect to the NASA API, save the results into a data frame, and update the index if needed. To indicate the entry point function is a cloud event, you can denote it with the decorator @functions_framework.cloud_event.

Here is the full updated code sample:

Adding a requirements.txt file

You will also want to define a requirements.txt file with all the specified packages needed to run the code.

Scheduling your Cloud Function

In Cloud Scheduler, you can set up your function to run at a regular interval using unix cron format. I have the code set to run every morning at 8 am in my timezone.

You will also want to configure the execution to connect to the Pub/Sub topic you created previously. I currently have the message body set to say “hello.”

Now that you have set up your Pub/Sub topic and your Cloud Function and set that Cloud Function to run on a schedule, your index should automatically update whenever new data is present.

Conclusion

Using Python, Google Cloud Platform Functions, and Google Cloud Scheduler you should be able to ensure that your index is updated regularly. You can find the complete code here and the search labs notebook for local testing. We are also running an on-demand webinar with Google Cloud which might be a good next step if you are looking to build search apps. Let us know if you built anything based on this blog or if you have questions on our Discuss forums and the community Slack channel.

There is a distinction between inspiration and paraphrasing. It is possible to read a content, get inspired, and then explore the idea with your own words, even if you come to a similar conclusion.

While plagiarism has been a topic of discussion for a long time, the accelerated production and publication of content have kept it relevant and posed an ongoing challenge.

This challenge isn't limited to books, academic research, or judicial documents, where plagiarism checks are frequently conducted. It can also extend to newspapers and even social media.

With the abundance of information and easy access to publishing, how can plagiarism be effectively checked on a scalable level?

Universities, government entities, and companies employ diverse tools, but while a straightforward lexical search can effectively detect direct plagiarism, the primary challenge lies in identifying paraphrased content.

Plagiarism detection with Generative AI

A new challenge emerges with Generative AI. Is content generated by AI considered plagiarism when copied?

The OpenAI terms of use, for example, specify that OpenAI will not claim copyright over content generated by the API for users. In this case, individuals using their Generative AI can use the generated content as they prefer without citation.

However, the acceptance of using Generative AI to improve efficiency remains a topic of discussion.

In an effort to contribute to plagiarism detection, OpenAI developed a detection model but later acknowledged that its accuracy is not sufficiently high.

"We believe this is not high enough accuracy for standalone detection and needs to be paired with metadata-based approaches, human judgment, and public education to be more effective."

The challenge persists; however, with the availability of more tools, there are now increased options for detecting plagiarism, even in cases of paraphrased and AI content.

Detecting plagiarism with Elasticsearch

Recognizing this, in this blog we are exploring one more use case with Natural Language Processing (NLP) models and Vector Search, plagiarism detection, beyond metadata searches.

This is demonstrated with Python examples, where we utilize a dataset from SentenceTransformers containing NLP-related articles. We check the abstracts for plagiarism by performing 'semantic textual similarity' considering 'abstract' embeddings generated with a text embedding model previously imported into Elasticsearch. Additionally, to identify AI-generated content — AI plagiarism, an NLP model developed by OpenAI was also imported into Elasticsearch.

The following image illustrates the data flow:

During the ingest pipeline with an inference processor, the 'abstract' paragraph is mapped to a 768-dimensional vector, the 'abstract_vector.predicted_value'.

Mapping:

The similarity between vector representations is measured using a vector similarity metric, defined using the 'similarity' parameter.

Cosine is the default similarity metric, computed as '(1 + cosine(query, vector)) / 2'. Unless you need to preserve the original vectors and cannot normalize them in advance, the most efficient way to perform cosine similarity is to normalize all vectors to unit length. This helps avoid performing extra vector length computations during the search, instead use 'dot_product'.

In this same pipeline, another inference processor containing the text classification model detects whether the content is 'Real' probably written by humans, or 'Fake' probably written by AI, adding the 'openai-detector.predicted_value' to each document.

Ingest Pipeline:

At query time, the same text embedding model is also employed to generate the vector representation of the query 'model_text' in a 'query_vector_builder' object.

A k-nearest neighbor (kNN) search finds the k nearest vector to the query vector measured by the similarity metric.

The _score of each document is derived from the similarity, ensuring that a larger score corresponds to a higher ranking. This means that the document is more similar semantically. As a result, we are printing three possibilities: if score > 0.9, we are considering 'high similarity'; if < 0.7, 'low similarity’, otherwise, 'moderate similarity’. You have the flexibility to set different threshold values to determine what level of _score qualifies as plagiarism or not, based on your use case.

Additionally, text classification is performed to also check for AI-generated elements in the text query.

Query:

Output:

In this example, after utilizing one of the 'abstract' values from our dataset as the text query 'model_text', plagiarism was identified. The similarity score is 1.0, indicating a high level of similarity — direct plagiarism. The vectorized query and document were not recognized as AI-generated content, which was expected.

Query:

Output:

By updating the text query 'model_text' with an AI-generated text that conveys the same message while minimizing the repetition of similar words, the detected similarity was still high, but the score was 0.9302529 instead of 1.0 — paraphrase plagiarism. It was also expected that this query, which was generated by AI, would be detected.

Lastly, considering the text query 'model_text' as a text about Elasticsearch, which is not an abstract of one of these documents, the detected similarity was 0.68991005, indicating low similarity according to the considered threshold values.

Query:

Output:

Although plagiarism was accurately identified in the text query generated by AI, as well as in cases of paraphrasing and direct copied content, navigating the landscape of plagiarism detection involves acknowledging various aspects.

In the context of AI-generated content detection, we explored a model that makes a valuable contribution. However, it is crucial to recognize the inherent limitations in standalone detection, necessitating the incorporation of other methods to boost the accuracy.

The variability introduced by the choice of text embedding models is another consideration. Different models, trained with distinct datasets, result in varying levels of similarity, highlighting the importance of the text embeddings generated.

Lastly, in these examples, we used the document's abstract. However, plagiarism detection often involves large documents, making it essential to address the challenge of text length. It is common for the text to exceed a model's token limit, requiring segmentation into chunks before building embeddings. A practical approach to handling this involves utilizing nested structures with dense_vector.

Conclusion

In this blog, we discussed the challenges of detecting plagiarism, particularly in paraphrased and AI-generated content, and how semantic textual similarity and text classification can be used for this purpose.

By combining these methods, we provided an example of plagiarism detection where we successfully identified AI-generated content, direct and paraphrased plagiarism.

The primary goal was to establish a filtering system that simplifies detection but human assessment remains essential for validation.

If you are interested in learning more about semantic textual similarity and NLP, we encourage you to also check out these links:

• What is semantic search?

• What is natural language processing (NLP)?

• Lexical and Semantic Search with Elasticsearch

• Chunking Large Documents via Ingest pipelines plus nested vectors equals easy passage search

OpenAI has discontinued the use of plugins in ChatGPT. You can read more about this here. We recommend reading this tutorial instead to learn how to build a large language model (LLM) chatbot that uses a pattern known as Retrieval-Augmented Generation. You can also read this blog to learn how to create custom GPTs with Elastic data.

You may have read this previous blog post about our journey to connect Elasticsearch’s relevance capabilities with OpenAI question-answering capabilities. The key idea in that post was to illustrate how to use Elastic with OpenAI’s GPT model to build a response and return context-relevant content to users.

The application that we built can expose a search endpoint and be called by any front-end service. The good news is that now OpenAI has released a private alpha of the future ChatGPT plugin framework.

In this blog, you will learn how to implement the plugin and extend the use of ChatGPT to any content indexed in Elasticsearch, using the Elastic documentation.

What is a ChatGPT plugin?

ChatGPT plugins are extensions that are developed to assist the model in completing its knowledge or executing actions.

For example, we know that the cutover of ChatGPT from a knowledge perspective is September 2021, so any question on recent data won’t be answered. In addition, any question that relates to something too specific beyond the boundaries of what the model has been trained on won’t be answered.

Plugins can broaden the scope of possible applications and enhance the capabilities of the models, but reciprocally, the plugin's output is augmented by the model itself.

The official list of plugins currently supported by ChatGPT are listed below. You can expect this list to expand rapidly as more organizations experiment with ChatGPT:

As you scan through the list, you’ll notice that the use cases are slowly revealing themselves here. In the case of Expedia, for example, its plugin is extending ChatGPT to assist in planning travel, making ChatGPT a trip-planning assistant.

This blog aims to achieve similar objectives for Elastic — to allow ChatGPT to access Elastic’s current knowledge base and assist you with your Elastic projects.

Architecture

We are going to bring a slight modification that has a positive cost impact in the sample code presented in part 1 by my colleague Jeff Vestal.

We will remove the call to OpenAI API, as now ChatGPT will fulfill the role of taking the content from Elasticsearch and digesting it back to the user:

1. ChatGPT makes a call to the /search endpoint of the plugin.

• This decision is based on the plugin “rules” description_for_human (see plugin-manifest below).

1. The plugin code creates a search request that is sent to Elasticsearch.

2. Documentation body and original url are returned to Python.

3. The plugin returns the document body and url, in text form to ChatGPT.

4. ChatGPT uses the information from the plugin to craft its response.

Again, this blog post assumes that you have set up your Elastic Cloud account, vectorized your content, and have an Elasticsearch cluster filled with data ready to be used. If you haven’t set all that up, see our previous post for detailed steps to follow.

Plugin code

OpenAI built a fairly simple-to-handle plugin framework for ChatGPT. It deploys a service that exposes:

• The plugin manifest, explaining what the plugin provides to the users and to ChatGPT

• The plugin openAPI definition, which is the functional description that enables ChatGPT to understand the available APIs The plugin code can be found here.

Plugin file structure

The screenshot below shows what the structure looks like:

• The plugin manifest is stored in the ai-plugin.json file under the .well-known directory as per OpenAI best practices.

• The main service code is in app.py.

• The Dockerfile will be later used to deploy the plugin to Google Cloud Compute.

• The plugin’s logo (logo.ong) as displayed in the ChatGPT plugin store, here the Elastic logo.

• The OpenAI description of the plugin.

Python code

For the full code, refer to the GitHub repository. We are going to look only at the main part of this code:

We took out all the details and kept the main parts here. There are two categories of APIs here:

1. The one required by OpenAI to build a plugin:

• /logo.png: retrieve the plugin logo

• /.well-known/ai-plugin.json: fetches the plugin manifest

• /openapi.yaml: fetches the plugin OpenAPI description

1. The plugin API:

• /search is the only one here exposed to ChatGPT that runs the search in Elasticsearch

Plugin manifest

The plugin manifest is what ChatGPT will use to validate the existence (reachable) of the plugin. The definition is the below:

There are a couple of things to point out here:

1. There are two descriptions:

• description_for_human - This is what the human sees when installing the plugin in the ChatGPT web UI.

• description_for_model - Instructions for the model to understand when to use the plugin.

1. There are some placeholders such as PLUGIN_HOSTNAME that are replaced in the Python code.

OpenAPI definition

Our code will only expose a single API endpoint to ChatGPT allowing it to search for Elastic documentation. Here is the description:

For the definition file, the key points are:

• We take the ChatGPT prompt content and pass it as a query to our Elasticsearch cluster.

• Some placeholders such as PLUGIN_HOSTNAME are replaced in the Python code.

Deploying the Elastic plugin in Google Cloud Platform (GCP)

You have a choice in picking a deployment method to expose your plugin, as well as using a different cloud provider. We use GCP in this blog post — more specifically Google Cloud Run and Google Cloud Build. The first is to expose and run the service, and the second is for continuous integration.

Setup

This setup assumes your GCP user has the right permissions to:

• Build a container image with Google Cloud Build in the Google Container Registry

• Deploy a container in Google Cloud Run

If not, you will need to update permissions on the GCP IAM page.

We are going to use the gcloud CLI to set up our environment. You can find the installation instructions here.

Once installed, run the following command to authenticate:

Then set the project identifier to your GCP project:

You are now ready to build and deploy.

Build and deploy

The first step is to build the container image using Cloud Build and push it to the Google Container Registry:

Replace PROJECT_ID with your GCP project ID and my-python-app with the name you want to give to your container image.

Export the environment required by the Python code to create the Elasticsearch client:

Finally, deploy the container image to Cloud Run:

You should see your service running in Cloud Run:

Note that you can also activate the continuous integration so that any commit in your GitHub repository will trigger a redeploy. On the service details page, click on Set up continuous deployment.

Installing the plugin in ChatGPT

Once the plugin is deployed and has a publicly accessible endpoint, it can be installed in ChatGPT. In our case, since this is deployed in Google Cloud Run, you can get the URL here:

Then in ChatGPT, go in the plugin store:

Choose to do “Develop your own plugin”:

Paste the URL you copied from the Google Cloud Run page:

Ensure the plugin is found and valid:

Follow the installation instructions until you see your plugin available in the list:

Let’s test our plugin!

OK, now for the best part! Do remember that ChatGPT decides to delegate when your prompt goes beyond its knowledge. To ensure that happens, just ask a question similar to this example:

With the steps provided in this blog, you can create your own plugin and deploy it on a cloud provider or your own hosts. This allows you to start exploring enhancing ChatGPT's knowledge and functionality, enhancing an already amazing tool with specialized and proprietary knowledge.

You can try all of the capabilities discussed in this blog today! Get started by signing up for a free Elastic Cloud trial.

Here are some other blogs you may find interesting:

• ChatGPT and Elasticsearch: OpenAI meets private data

• Monitor OpenAI API and GPT models with OpenTelemetry and Elastic

• Exploring the Future of Security with ChatGPT

In this blog post, we may have used third party generative AI tools, which are owned and operated by their respective owners. Elastic does not have any control over the third party tools and we have no responsibility or liability for their content, operation or use, nor for any loss or damage that may arise from your use of such tools. Please exercise caution when using AI tools with personal, sensitive or confidential information. Any data you submit may be used for AI training or other purposes. There is no guarantee that information you provide will be kept secure or confidential. You should familiarize yourself with the privacy practices and terms of use of any generative AI tools prior to use.

Elastic, Elasticsearch and associated marks are trademarks, logos or registered trademarks of Elasticsearch N.V. in the United States and other countries. All other company and product names are trademarks, logos or registered trademarks of their respective owners.

Combine Elasticsearch's search relevance with OpenAI's ChatGPT's question-answering capabilities to query your data. In this blog, you'll learn how to connect ChatGPT to proprietary data stores using Elasticsearch and build question/answer capabilities for your data.

What is ChatGPT?

In recent months, there has been a surge of excitement around ChatGPT, a groundbreaking AI model created by OpenAI. But what exactly is ChatGPT?

Based on the powerful GPT architecture, ChatGPT is designed to understand and generate human-like responses to text inputs. GPT stands for "Generative Pre-trained Transformer.” The Transformer is a cutting-edge model architecture that has revolutionized the field of natural language processing (NLP). These models are pre-trained on vast amounts of data and are capable of understanding context, generating relevant responses, and even carrying on a conversation. To learn more about the history of transformer models and some NLP basics in the Elastic Stack, be sure to check out the great talk by Elastic ML Engineer Josh Devins.

The primary goal of ChatGPT is to facilitate meaningful and engaging interactions between humans and machines. By leveraging the recent advancements in NLP, ChatGPT models can provide a wide range of applications, from chatbots and virtual assistants to content generation, code completion, and much more. These AI-powered tools have rapidly become an invaluable resource in countless industries, helping businesses streamline their processes and enhance their services.

Limitations of ChatGPT & how to minimize them

Despite the incredible potential of ChatGPT, there are certain limitations that users should be aware of. One notable constraint is the knowledge cutoff date. Currently, ChatGPT is trained on data up to September 2021, meaning it is unaware of events, developments, or changes that have occurred since then. Consequently, users should keep this limitation in mind while relying on ChatGPT for up-to-date information. This can lead to outdated or incorrect responses when discussing rapidly changing areas of knowledge such as software enhancements and capabilities or even world events.

ChatGPT, while an impressive AI language model, can occasionally hallucinate in its responses, often exacerbated when it lacks access to relevant information. This overconfidence can result in incorrect answers or misleading information being provided to users. It is important to be aware of this limitation and approach the responses generated by ChatGPT with a degree of skepticism, cross-checking and verifying the information when necessary to ensure accuracy and reliability.

Another limitation of ChatGPT is its lack of knowledge about domain-specific content. While it can generate coherent and contextually relevant responses based on the information it has been trained on, it is unable to access domain-specific data or provide personalized answers that depend on a user's unique knowledge base. For instance, it may not be able to provide insights into an organization’s proprietary software or internal documentation. Users should, therefore, exercise caution when seeking advice or answers on such topics from ChatGPT directly.

One way to minimize these limitations is by providing ChatGPT access to specific documents relevant to your domain and questions, and enabling ChatGPT’s language understanding capabilities to generate tailored responses.

This can be accomplished by connecting ChatGPT to a search engine like Elasticsearch.

Elasticsearch — you know, for search!

Elasticsearch is a scalable data store and vector database designed to deliver relevant document retrieval, ensuring that users can access the information they need quickly and accurately. Elasticsearch’s primary focus is on delivering the most relevant results to users, streamlining the search process, and enhancing user experience.

Elasticsearch boasts a myriad of features to ensure top-notch search performance, including support for traditional keyword and text-based search (BM25) and an AI-ready vector search with exact match and approximate kNN (k-Nearest Neighbor) search capabilities. These advanced features allow Elasticsearch to retrieve results that are not only relevant but also for queries that have been expressed using natural language. By leveraging traditional, vector, or hybrid search (BM25 + kNN), Elasticsearch can deliver results with unparalleled precision, helping users find the information they need with ease.

One of the key strengths of Elasticsearch is its robust API, which enables seamless integration with other services to extend and enhance its capabilities. By integrating Elasticsearch with various third-party tools and platforms, users can create powerful and customized search solutions tailored to their specific requirements. This flexibility and extensibility makes Elasticsearch an ideal choice for businesses looking to improve their search capabilities and stay ahead in the competitive digital landscape.

By working in tandem with advanced AI models like ChatGPT, Elasticsearch can provide the most relevant documents for ChatGPT to use in its response. This synergy between Elasticsearch and ChatGPT ensures that users receive factual, contextually relevant, and up-to-date answers to their queries. In essence, the combination of Elasticsearch's retrieval prowess and ChatGPT's natural language understanding capabilities offers an unparalleled user experience, setting a new standard for information retrieval and AI-powered assistance.

How to use ChatGPT with Elasticsearch

1. Python interface accepts user questions.

Generate a hybrid search request for Elasticsearch

• BM25 match on the title field

• kNN search on the title-vector field

• Boost kNN search results to align scores

• Set size=1 to return only the top scored document

1. Search request is sent to Elasticsearch.

2. Documentation body and original url are returned to python.

3. API call is made to OpenAI ChatCompletion.

• Prompt: "answer this question <question> using only this document <body_content from top search result>"

1. Generated response is returned to python.

2. Python adds on original documentation source url to generated response and prints it to the screen for the user.

The ElasticDoc ChatGPT process utilizes a Python interface to accept user questions and generate a hybrid search request for Elasticsearch, combining BM25 and kNN search approaches to find the most relevant document from the Elasticsearch Docs site, now indexed in Elasticsearch. However, you do not have to use hybrid search or even vector search. Elasticsearch provides the flexibility to use whichever search pattern best fits your needs and provides the most relevant results for your specific data sets.

After retrieving the top result, the program crafts a prompt for OpenAI's ChatCompletion API, instructing it to answer the user's question using only the information from the selected document. This prompt is key to ensuring the ChatGPT model only uses information from the official documentation, lessening the chance of hallucinations.

Finally, the program presents the API-generated response and a link to the source documentation to the user, offering a seamless and user-friendly experience that integrates front-end interaction, Elasticsearch querying, and OpenAI API usage for efficient question-answering.

Note that while we are only returning the top-scored document for simplicity, the best practice would be to return multiple documents to provide more context to ChatGPT. The correct answer could be found in more than one documentation page, or if we were generating vectors for the full body text, those larger bodies of text may need to be chunked up and stored across multiple Elasticsearch documents. By leveraging Elasticsearch's ability to search across numerous vector fields in tandem with traditional search methods, you can significantly enhance your top document recall.

Technical setup

The technical requirements are fairly minimal, but it takes some steps to put all the pieces together. For this example, we will configure the Elasticsearch web crawler to ingest the Elastic documentation and generate vectors for the title on ingest. You can follow along to replicate this setup or use your own data. To follow along we will need:

• Elasticsearch cluster

• Eland Python library

• OpenAI API account

• Somewhere to run our python frontend and api backend

Elastic Cloud setup

The steps in this section assume you don’t currently have an Elasticsearch cluster running in Elastic Cloud. If you do you, can skip to the next section.

Sign up If you don’t already have an Elasticsearch cluster, you can sign up for a free trial with Elastic Cloud.

Create deployment After you sign up, you will be prompted to create your first deployment.

• Create a name for your deployment.

• You can accept the default cloud provider and region or click Edit Settings and choose another location.

• Click Create deployment. Shortly a new deployment will be provisioned for you and you will be logged in to Kibana. Back to the Cloud We need to do a couple of things back in the Cloud Console before we move on: Click on the Navigation Icon in the upper left and select Manage this deployment.

Add a machine learning node.

• Back in the Cloud Console, click on Edit under your Deployment’s name in the left navigation bar.

• Scroll down to the Machine Learning instances box and click +Add Capacity.

• Under Size per zone, click and select 2 GB RAM.

• Scroll down and click on Save.

• In the pop-up summarizing the architecture changes, click Confirm.

• In a few moments, your deployment will now have the ability to run machine learning models!

Reset Elasticsearch Deployment User and password:

• Click on Security on the left navigation under your deployment’s name.

• Click on Reset Password and confirm with Reset. (Note: as this is a new cluster nothing should be using this Elastic password.)

• Download the newly created password for the “elastic” user. (We will use this to load our model from Hugging Face and in our python program.)

Copy the Elasticsearch Deployment Cloud ID.

• Click on your Deployment name to go to the overview page.

• On the right-hand side click the copy icon to copy your Cloud ID. (Save this for use later to connect to the Deployment.)

Eland

We next need to load an embedding model into Elasticsearch to generate vectors for our blog titles and later for our user’s search questions. We will be using the all-distilroberta-v1 model trained by SentenceTransformers and hosted on the Hugging Face model hub. This particular model isn’t required for this setup to work. It is good for general use as it was trained on very large data sets covering a wide range of topics. However, with vector search use cases, using a model fine-tuned to your particular data set will usually provide the best relevancy.

To do this, we will use the Eland python library created by Elastic. The library provides a wide range of data science functions, but we will be using it as a bridge to load the model into Elasticsearch from the Hugging Face model hub so it can be deployed on machine learning nodes for inference use.

Eland can either be run as part of a python script or on the command line. The repo also provides a Docker container for users looking to go that route. Today we will run Eland in a small python notebook, which can run in Google’s Colab in the web browser for free.

Open the program link and click the “Open in Colab” button at the top to launch the notebook in colab.

Set the variable hf_model_id to the model name. This model is set already in the example code but if you want to use a different model or just for future information:

• hf_model_id='sentence-transformers/all-distilroberta-v1'

• Copy model name from Hugging Face. The easiest way to do this is to click the copy icon to the right of the model name.

Run the cloud auth section, and you will be prompted to enter:

• Cloud ID (you can find this in the Elastic Cloud Console)

• Elasticsearch Username (easiest will be to use the “Elastic” user created when the deployment was created)

• Elasticsearch User Password

Run the remaining steps.

• This will download the model from Hugging face, chunk it up, and load it into Elasticsearch.

• Deploy (start) the model onto the machine learning node.

Elasticsearch index and web crawler

Next up we will create a new Elasticsearch index to store our Elastic Documentation, configure the web crawler to automatically crawl and index those docs, as well as use an ingest pipeline to generate vectors for the doc titles.

• Open Kibana from the Cloud Console if you don’t already have it open.

• In Kibana, Navigate to Enterprise Search -> Overview. Click Create an Elasticsearch Index.

• Using the Web Crawler as the ingestion method, enter elastic-docs as the index name. Then, click Create Index.

• Click on the “Pipelines” tab.

• Click Copy and customize in the Ingest Pipeline Box.

• Click Add Inference Pipeline in the Machine Learning Inference Pipelines box.

• Enter the name elastic-docs_title-vector for the New pipeline.

• Select the trained ML model you loaded in the Eland step above.

• Select title as the Source field.

• Click Continue, then click Continue again at the Test stage.

• Click Create Pipeline at the Review stage.

Update mapping for dense_vector field. (Note: with Elasticsearch version 8.8+, this step should be automatic.)

• In the navigation menu, click on Dev Tools. You may have to click Dismiss on the flyout with documentation if this is your first time opening Dev Tools.

• In Dev Tools in the Console tab, update the mapping for our dense vector target field with the following code. You simply paste it in the code box and click the little arrow to the right of line 1.

• You should see the following response on the right half of the screen:

• This will allow us to run kNN search on the title field vectors later on.

Configure web crawler to crawl Elastic Docs site:

• Click on the navigation menu one more time and click on Enterprise Search -> Overview.

• Under Content, click on Indices.

• Click on search-elastic-docs under Available indices.

• Click on the Manage Domains tab.

• Click “Add domain.”

• Enter https://www.elastic.co/guide/en, then click Validate Domain.

• After the checks run, click Add domain. Then click Crawl rules.

• Add the following crawl rules one at a time. Start with the bottom and work up. Rules are evaluated according to first match.

Disallow

Contains

release-notes

Allow

Regex

/guide/en/.*/current/.*

Disallow

Regex

.*

• With all the rules in place, click Crawl at the top of the page. Then, click Crawl all domains on this index.

Elasticsearch’s web crawler will now start crawling the documentation site, generating vectors for the title field, and indexing the documents and vectors.

The first crawl will take some time to complete. In the meantime, we can set up the OpenAI API credentials and the Python backend.

Connecting with OpenAI API

To send documents and questions to ChatGPT, we need an OpenAI API account and key. If you don’t already have an account, you can create a free account and you will be given an initial amount of free credits.

• Go to https://platform.openai.com and click on Signup. You can go through the process to use an email address and password or login with Google or Microsoft.

Once your account is created, you will need to create an API key:

• Click on API Keys.

• Click Create new secret key.

• Copy the new key and save it someplace safe as you won’t be able to view the key again.

Python backend setup

Clone or download the python program

Github Link to code

1. Install required python libraries. We are running the example program in Replit, which has isolated environments. If you are running this on a laptop or VM, best practice is to set up a virtual ENV for python.

• Run pip install -r requirements.txt

1. Set authentication and connection environment variables (e.g., if running on the command line: export openai_api=”123456abcdefg789”)

• openai_api - OpenAI API Key

• cloud_id - Elastic Cloud Deployment ID

• cloud_user - Elasticsearch Cluster User

• cloud_pass - Elasticsearch User Password

1. Run the streamlit program. More info about streamlit can be found in its docs.

• Streamlit has its own command to start: streamlit run elasticdocs_gpt.py

1. This will start a web browser and the url will be printed to the command line.

Sample chat responses

With everything ingested and the front end up and running, you can start asking questions about the Elastic Documentations.

Asking “Show me the API call for an inference processor” now returns an example API call and some information about the configuration settings.

Asking for steps to add a new integration to Elastic Agent will return:

As mentioned earlier, one of the risks of allowing ChatGPT to answer questions based purely on data it has been trained on is its tendency to hallucinate incorrect answers. One of the goals of this project is to provide ChatGPT with the data containing the correct information and let it craft an answer.

So what happens when we give ChatGPT a document that does not contain the correct information? Say, asking it to tell you how to build a boat (which isn’t currently covered by Elastic’s documentation):

When ChatGPT is unable to find an answer to the question in the document we provided, it falls back on our prompt instruction simply telling the user it is unable to answer the question.

Elasticsearch’s robust retrieval + the power of ChatGPT

In this example, we've demonstrated how integrating Elasticsearch's robust search retrieval capabilities with cutting-edge advancements in AI-generated responses from GPT models can elevate the user experience to a whole new level.

The individual components can be tailored to suit your specific requirements and adjusted to provide the best results. While we used the Elastic web crawler to ingest public data, you're not limited to this approach. Feel free to experiment with alternative embedding models, especially those fine-tuned for your domain-specific data.

You can try all of the capabilities discussed in this blog today! To build your own ElasticDocs GPT experience, sign up for an Elastic trial account, and then look at this sample code repo to get started.

If you would like ideas to experiment with search relevance, here are two to try out:

• [BLOG] Deploy NLP text embeddings and vector search using Elasticsearch

• [BLOG] Implement image similarity search with Elastic

In this blog post, we may have used third party generative AI tools, which are owned and operated by their respective owners. Elastic does not have any control over the third party tools and we have no responsibility or liability for their content, operation or use, nor for any loss or damage that may arise from your use of such tools. Please exercise caution when using AI tools with personal, sensitive or confidential information. Any data you submit may be used for AI training or other purposes. There is no guarantee that information you provide will be kept secure or confidential. You should familiarize yourself with the privacy practices and terms of use of any generative AI tools prior to use.

Elastic, Elasticsearch and associated marks are trademarks, logos or registered trademarks of Elasticsearch N.V. in the United States and other countries. All other company and product names are trademarks, logos or registered trademarks of their respective owners.