Baha Azarmi - Elasticsearch Labs

An Elasticsearch Query Language (ES|QL) analysis: Millionaire odds vs. hit by a bus

Baha Azarmi — Tue, 20 Aug 2024 00:00:00 GMT

Elasticsearch Query Language (ES|QL) is designed for fast, efficient querying of large datasets. It has a straightforward syntax which will allow you to write complex queries easily, with a pipe based language, reducing the learning curve. We're going to use ES|QL to run statistical analysis and compare different odds.

If you are reading this, you probably want to know how rich you can get before actually reaching the same odds of being hit by a bus. I can't blame you, I want to know too. Let's work out the odds so that we can make sure we win the lottery rather than get in an accident!

What we are going to see in this blog is figuring out the probability of being hit by a bus and the probability of achieving wealth. We'll then compare both and understand until what point your chances of getting rich are higher, and when you should consider getting life insurance.

So how are we going to do that? This is going to be a mix of magic numbers pulled from different articles online, some synthetics data and the power of ES|QL, the new Elasticsearch Query Language. Let's get started.

Data for the ES|QL analysis

The magic number

The challenge starts here as the dataset is going to be somewhat challenging to find. We are then going to assume for the sake of the example that ChatGPT is always right. Let’s see what we get for the following question:

Cough Cough… That sounds about right, this is going to be our magic number.

Generating the wealth data

Prerequisites

Before running any of the scripts below, make sure to install the following packages:

elasticsearch==8.14.0 matplotlib numpy panda scipy

Now, there is one more thing we need, a representative dataset with wealth distribution to compute wealth probability. There is definitely some portion of it here and there, but again, for the example we are going to generate a 500K line dataset with the below python script. I am using python 3.11.5 in this example:

import pandas as pd import numpy as np import getpass from elasticsearch import Elasticsearch, helpers # Input the Elasticsearch host hosts = input('Enter your Elasticsearch host address : ') # Securely input the Elasticsearch API key api_key = getpass.getpass(prompt='Enter your Elasticsearch API Key: ') # Initialize Elasticsearch client client = Elasticsearch( hosts=hosts, api_key=api_key, ) # Generate synthetic data with a highly skewed distribution num_records = 500000 np.random.seed(42) # Ensure reproducibility # Generate net worth using a highly skewed distribution ages = np.random.randint(20, 80, num_records) # Random ages between 20 and 80 incomes = np.random.exponential(scale=10000, size=num_records) # Exponential distribution for income # Use a more skewed distribution for net worth with a much larger range net_worths = np.random.exponential(scale=100000000, size=num_records) # Extremely skewed net worth # Scale up the net worths to reach up to $100 billion net_worths = np.clip(net_worths, 0, 100000000000) # Create DataFrame df = pd.DataFrame({ 'id': range(1, num_records + 1), 'age': ages, 'income': incomes, 'net_worth': net_worths, 'counter': range(1, num_records + 1) # Add a counter field for pagination }) # Index the data into Elasticsearch index_name = 'raw_wealth_data_large' try: if client.indices.exists(index=index_name): client.indices.delete(index=index_name) except exceptions.NotFoundError: pass client.indices.create(index=index_name) def generator(df): for index, row in df.iterrows(): yield { "_index": index_name, "_source": row.to_dict() } helpers.bulk(client, generator(df)) print("Data indexed successfully.")

It should take some time to run depending on your configuration since we are injecting 500K documents here!

FYI, after playing with a couple of versions of the script above and the ESQL query on the synthetic data, it was obvious that the net worth generated across the population was not really representative of the real world. So I decided to use a log-normal distribution (np.random.lognormal) for income to reflect a more realistic spread where most people have lower incomes, and fewer people have very high incomes.

Net Worth Calculation: Used a combination of random multipliers (np.random.uniform(0.5, 5)) and additional noise (np.random.normal(0, 10000)) to calculate net worth. Added a check to ensure no negative net worth values by using np.maximum(0, net_worths).

Not only have we generated 500K documents, but we also used the Elasticsearch python client to bulk ingest all these documents in our deployment. Please note that you will find the endpoint to pass in as hosts Cloud ID in the code above.

For the deployment API key, open Kibana, and generate the key in Stack Management / API Keys:

The good news is that if you have a real data set, all you will need to do is to change the above code to read your dataset and write documents with the same data mapping.

Ok we're getting there! The next step is pouring our wealth distribution.

ES|QL wealth analysis

Introducing ES|QL: A powerful tool for data analysis

The arrival of Elasticsearch Query Language (ES|QL) is very exciting news for our users. It largely simplifies querying, analyzing, and visualizing data stored in Elasticsearch, making it a powerful tool for all data-driven use cases.

ES|QL comes with a variety of functions and operators, to perform aggregations, statistical analyses, and data transformations. We won’t address them all in this blog post, however our documentation is very detailed and will help you familiarize with the language and the possibilities.

To get started with ES|QL today and run the blog post queries, simply start a trial on Elastic Cloud, load the data and run your first ES|QL query.

Understanding the wealth distribution with our first query

To get familiar with the dataset, head to Discover in Kibana and switch to ES|QL in the dropdown on the left hand side:

Let’s fire our first request:

from raw_wealth_data_large | keep age, id, income, net_worth | limit 10

As you could expect from our indexing script earlier, we are finding the documents we bulk ingested, notice the simplicity of pulling data from a given dataset with ES|QL where every query starts with the From clause, then your index.

In the query above given we have 500K lines, we limited the amount of returned documents to 10. To do this, we are passing the output of the first segment of the query via a pipe to the limit command to only get 10 results. Pretty intuitive, right?

Alright, what would be more interesting is to understand the wealth distribution in our dataset, for this we will leverage one of the 30 functions ES|QL provides, namely percentile.

This will allow us to understand the relative position of each data point within the distribution of net worth. By calculating the median percentile (50th percentile), we can gauge where an individual’s net worth stands compared to others.

FROM raw_wealth_data_large | stats p50 = percentile(net_worth, 50)

Like our first query, we are passing the output of our index to another function, Stats, which combined with the percentile function will output the median net worth:

The median is about 54K, which unfortunately is probably optimistic compared to the real world, but we are not going to solve this here. If we go a little further, we can look at the distribution in more granularity by computing more percentiles:

FROM raw_wealth_data_large | STATS p25 = percentile(net_worth, 25) , p50 = percentile(net_worth, 50) , p75 = percentile(net_worth, 75) , p90 = percentile(net_worth, 90) , p95 = percentile(net_worth, 95) , p96 = percentile(net_worth, 96) , p98 = percentile(net_worth, 98) , p97 = percentile(net_worth, 97) , p99 = percentile(net_worth, 99) | keep p25, p25, p50, p75, p90, p95, p96, p97, p98, p99

With the below output:

The data reveals a significant disparity in wealth distribution, with the majority of wealth being concentrated among the richest individuals. Specifically, the top 5% (95th percentile) possess a disproportionately large portion of the total wealth, with a net worth starting at $852,988.26 and increasing dramatically in the higher percentiles.

The 99th percentile individuals hold a net worth exceeding $2 million, highlighting the skewed nature of wealth distribution. This indicates that a substantial portion of the population has modest net worth, which is probably what we want for this example.

Another way to look at this is to augment the previous query and grouping by age to see if there is, (in our synthetic dataset), a relation between wealth and age:

FROM raw_wealth_data_large | STATS p25 = percentile(net_worth, 25) , p50 = percentile(net_worth, 50) , p75 = percentile(net_worth, 75) , p90 = percentile(net_worth, 90) , p95 = percentile(net_worth, 95) , p96 = percentile(net_worth, 96) , p98 = percentile(net_worth, 98) , p97 = percentile(net_worth, 97) , p99 = percentile(net_worth, 99) by age | keep p25, p25, p50, p75, p90, p95, p96, p97, p98, p99, age

This could be visualized in a Kibana dashboard. Simply:

Navigate to Dashboard
Add a new ES|QL visualization
Copy and paste our query
Move the age field to the horizontal axis in the visualization configuration

Which will output:

The above suggests that the data generator randomized wealth uniformly across the population age, there is no specific trend pattern we can really see.

Median Absolute Deviation (MAD)

We calculate the median absolute deviation (MAD) to measure the variability of net worth in a robust manner, less influenced by outliers.

FROM raw_wealth_data_large | stats median_net_worth = MEDIAN(net_worth), mad_net_worth = MEDIAN_ABSOLUTE_DEVIATION(net_worth) | keep median_net_worth, mad_net_worth

With a median net worth of 44,205.44, we can infer the typical range of Net Worth: Most individuals’ net worth falls within a range of 9,581.78 to $97,992.66.

The statistical showdown between Net Worth and Bus Collision

Alright, this is the moment to understand how rich we can get, based on our dataset, before getting hit by a bus. To do that, we are going to leverage ES|QL to pull our entire dataset in chunks and load it into a pandas dataframe to build a net worth probability distribution. Finally, we will determine where the ends meet between the net worth and bus collision probabilities.

The entire Python notebook is available here. I also recommend you read this blog post which walks you through using ES|QL with pandas dataframes.

Helper functions

As you can see in the previously referred blog post, we introduced support for ES|QL since version 8.12 of the Elasticsearch python client. Thus our notebook first defines the below functions:

from io import StringIO # Function to execute ESQL query and fetch data in chunks def execute_esql_query(query): response = client.esql.query(query=query, format="csv") return pd.read_csv(StringIO(response.body)) # Function to fetch paginated data using the counter field def fetch_paginated_data(index, num_records, size=10000): all_data = pd.DataFrame() for start in range(1, num_records + 1, size): end = start + size - 1 query = f""" FROM {index} | WHERE counter >= {start} AND counter <= {end} | limit {size} """ data_chunk = execute_esql_query(query) all_data = pd.concat([all_data, data_chunk], ignore_index=True) return all_data

The first function is straightforward and executes an ES|QL query, the second is fetching the entire dataset from our index. Notice the trick in there that I am using a counter built-in to a field in my index to paginate through the data. This is workaround I am using while our engineering team is working on the support for pagination in ES|QL.

Next, knowing that we have 500K documents in our index, we simply call these function to load the data in a data frame:

# Fetch all data using pagination and ES|QL num_records = 500000 all_data_df = fetch_paginated_data(index_name, num_records) print(f"Total Data Retrieved: {len(all_data_df)} records")

Fit Pareto distribution

Next, we fit our data to a Pareto distribution, which is often used to model wealth distribution because it reflects the reality that a small percentage of the population controls most of the wealth. By fitting our data to this distribution, we can more accurately represent the probabilities of different net worth levels.

from scipy.stats import pareto # Fit a Pareto distribution to the data shape, loc, scale = pareto.fit(all_data_df['net_worth'], floc=0) # Calculate the probability density for each net worth all_data_df['net_worth_probability'] = pareto.pdf(all_data_df['net_worth'], shape, loc=loc, scale=scale) # Normalize the probabilities to sum to 1 all_data_df['net_worth_probability'] /= all_data_df['net_worth_probability'].sum() print("Data with Net Worth Probability:") print(all_data_df.head())

We can visualize the pareto distribution with the code below: ``

import matplotlib.pyplot as plt from scipy.stats import pareto # Assuming all_data_df contains the fetched net worth data from Elasticsearch # Fit a Pareto distribution to the data shape, loc, scale = pareto.fit(all_data_df['net_worth'], floc=0) # Plot the Net Worth Probability Distribution plt.figure(figsize=(10, 6)) # Plot histogram of empirical net worth data plt.hist(all_data_df['net_worth'], bins=100, density=True, alpha=0.6, color='g', label='Empirical Data') # Plot fitted Pareto distribution xmin, xmax = plt.xlim() x = np.linspace(xmin, xmax, 100) p = pareto.pdf(x, shape, loc=loc, scale=scale) plt.plot(x, p, 'k', linewidth=2, label='Fitted Pareto Distribution') # Show the plot plt.xlabel('Net Worth') plt.y bnblabel('Probability') plt.title('Net Worth Probability Distribution') plt.legend() plt.grid(True) plt.show()

Breaking point

Finally, with the calculated probability, we determine the target net worth corresponding to the bus hit probability and visualize it. Remember, we use the magic number ChatGPT gave us for the probability of getting hit by a bus:

# Find the Net Worth Corresponding to the Bus Hit Probability target_probability = 0.0000181 cumulative_probability = all_data_df['net_worth_probability'].cumsum() target_net_worth_df = all_data_df[cumulative_probability >= target_probability].head(1) target_net_worth = target_net_worth_df['net_worth'].iloc[0] print(f"Net Worth with Probability >= {target_probability}: {target_net_worth}") # Plot the Net Worth Probability Distribution plt.figure(figsize=(10, 6)) plt.hist(all_data_df['net_worth'], bins=100, density=True, alpha=0.6, color='g', label='Empirical Data') xmin, xmax = plt.xlim() x = np.linspace(xmin, xmax, 100) p = pareto.pdf(x, shape, loc=loc, scale=scale) plt.plot(x, p, 'k', linewidth=2, label='Fitted Pareto Distribution') plt.axhline(y=target_probability, color='r', linestyle='--', label='Bus Hit Probability') plt.axvline(x=target_net_worth, color='g', linestyle='--', label=f'Net Worth = {target_net_worth:.2f}') plt.xlabel('Net Worth') plt.ylabel('Probability') plt.title('Net Worth Probability Distribution') plt.legend() plt.grid(True) plt.show()

Conclusion

Based on our synthetic dataset, this chart vividly illustrates that the probability of amassing a net worth of approximately $12.5 million is as rare as the chance of being hit by a bus. For the fun of it, let’s ask ChatGPT what the probability is:

Okay… $439 million? I think ChatGPT might be hallucinating again.

ChatGPT and Elasticsearch: A plugin to use ChatGPT with your Elastic data

Baha Azarmi — Wed, 21 Jun 2023 00:00:00 GMT

Update: April 16th, 2024

OpenAI has discontinued the use of plugins in ChatGPT. You can read more about this here. We recommend reading this tutorial instead to learn how to build a large language model (LLM) chatbot that uses a pattern known as Retrieval-Augmented Generation. You can also read this blog to learn how to create custom GPTs with Elastic data.

You may have read this previous blog post about our journey to connect Elasticsearch’s relevance capabilities with OpenAI question-answering capabilities. The key idea in that post was to illustrate how to use Elastic with OpenAI’s GPT model to build a response and return context-relevant content to users.

The application that we built can expose a search endpoint and be called by any front-end service. The good news is that now OpenAI has released a private alpha of the future ChatGPT plugin framework.

In this blog, you will learn how to implement the plugin and extend the use of ChatGPT to any content indexed in Elasticsearch, using the Elastic documentation.

What is a ChatGPT plugin?

ChatGPT plugins are extensions that are developed to assist the model in completing its knowledge or executing actions.

For example, we know that the cutover of ChatGPT from a knowledge perspective is September 2021, so any question on recent data won’t be answered. In addition, any question that relates to something too specific beyond the boundaries of what the model has been trained on won’t be answered.

Plugins can broaden the scope of possible applications and enhance the capabilities of the models, but reciprocally, the plugin's output is augmented by the model itself.

The official list of plugins currently supported by ChatGPT are listed below. You can expect this list to expand rapidly as more organizations experiment with ChatGPT:

As you scan through the list, you’ll notice that the use cases are slowly revealing themselves here. In the case of Expedia, for example, its plugin is extending ChatGPT to assist in planning travel, making ChatGPT a trip-planning assistant.

This blog aims to achieve similar objectives for Elastic — to allow ChatGPT to access Elastic’s current knowledge base and assist you with your Elastic projects.

Architecture

We are going to bring a slight modification that has a positive cost impact in the sample code presented in part 1 by my colleague Jeff Vestal.

We will remove the call to OpenAI API, as now ChatGPT will fulfill the role of taking the content from Elasticsearch and digesting it back to the user:

ChatGPT makes a call to the /search endpoint of the plugin.

This decision is based on the plugin “rules” description_for_human (see plugin-manifest below).

The plugin code creates a search request that is sent to Elasticsearch.
Documentation body and original url are returned to Python.
The plugin returns the document body and url, in text form to ChatGPT.
ChatGPT uses the information from the plugin to craft its response.

Again, this blog post assumes that you have set up your Elastic Cloud account, vectorized your content, and have an Elasticsearch cluster filled with data ready to be used. If you haven’t set all that up, see our previous post for detailed steps to follow.

Plugin code

OpenAI built a fairly simple-to-handle plugin framework for ChatGPT. It deploys a service that exposes:

The plugin manifest, explaining what the plugin provides to the users and to ChatGPT
The plugin openAPI definition, which is the functional description that enables ChatGPT to understand the available APIs The plugin code can be found here.

Plugin file structure

The screenshot below shows what the structure looks like:

The plugin manifest is stored in the ai-plugin.json file under the .well-known directory as per OpenAI best practices.
The main service code is in app.py.
The Dockerfile will be later used to deploy the plugin to Google Cloud Compute.
The plugin’s logo (logo.ong) as displayed in the ChatGPT plugin store, here the Elastic logo.
The OpenAI description of the plugin.

Python code

For the full code, refer to the GitHub repository. We are going to look only at the main part of this code:

… @app.get("/search") … @app.get("/logo.png") … @app.get("/.well-known/ai-plugin.json") … @app.get("/openapi.yaml") …

We took out all the details and kept the main parts here. There are two categories of APIs here:

The one required by OpenAI to build a plugin:

/logo.png: retrieve the plugin logo
/.well-known/ai-plugin.json: fetches the plugin manifest
/openapi.yaml: fetches the plugin OpenAPI description

The plugin API:

/search is the only one here exposed to ChatGPT that runs the search in Elasticsearch

Plugin manifest

The plugin manifest is what ChatGPT will use to validate the existence (reachable) of the plugin. The definition is the below:

{ "schema_version": "v1", "name_for_human": "ElasticGPTDoc_Plugin", "name_for_model": "ElasticGPTDoc_Plugin", "description_for_human": "Elastic Assistant, you know, for knowledge", "description_for_model": "Get most recent elasticsearch docs post 2021 release, anything after release 7.15", "auth": { "type": "none" }, "api": { "type": "openapi", "url": "PLUGIN_HOSTNAME/openapi.yaml", "is_user_authenticated": false }, "logo_url": "PLUGIN_HOSTNAME/logo.png", "contact_email": "info@elastic.co", "legal_info_url": "http://www.example.com/legal" }

There are a couple of things to point out here:

There are two descriptions:

description_for_human - This is what the human sees when installing the plugin in the ChatGPT web UI.
description_for_model - Instructions for the model to understand when to use the plugin.

There are some placeholders such as PLUGIN_HOSTNAME that are replaced in the Python code.

OpenAPI definition

Our code will only expose a single API endpoint to ChatGPT allowing it to search for Elastic documentation. Here is the description:

openapi: 3.0.1 info: title: ElasticDocGPT description: Retrieve information front the most recent Elastic documentation version: 'v1' servers: - url: PLUGIN_HOSTNAME paths: /search: get: operationId: search summary: retrieves the document matching the query parameters: - in: query name: query schema: type: string description: use to filter relevant part of the elasticsearch documentations responses: "200": description: OK

For the definition file, the key points are:

We take the ChatGPT prompt content and pass it as a query to our Elasticsearch cluster.
Some placeholders such as PLUGIN_HOSTNAME are replaced in the Python code.

Deploying the Elastic plugin in Google Cloud Platform (GCP)

You have a choice in picking a deployment method to expose your plugin, as well as using a different cloud provider. We use GCP in this blog post — more specifically Google Cloud Run and Google Cloud Build. The first is to expose and run the service, and the second is for continuous integration.

Setup

This setup assumes your GCP user has the right permissions to:

Build a container image with Google Cloud Build in the Google Container Registry
Deploy a container in Google Cloud Run

If not, you will need to update permissions on the GCP IAM page.

We are going to use the gcloud CLI to set up our environment. You can find the installation instructions here.

Once installed, run the following command to authenticate:

gcloud auth

Then set the project identifier to your GCP project:

gcloud config set project PROJECT_ID

You are now ready to build and deploy.

Build and deploy

The first step is to build the container image using Cloud Build and push it to the Google Container Registry:

gcloud builds submit --tag gcr.io/PROJECT_ID/my-python-app

Replace PROJECT_ID with your GCP project ID and my-python-app with the name you want to give to your container image.

Export the environment required by the Python code to create the Elasticsearch client:

export YOUR_CLOUD_ID=VALUE export YOUR_CLOUD_PASS=VALUE export YOUR_CLOUD_USER=VALUE

Finally, deploy the container image to Cloud Run:

gcloud run deploy my-python-app \ --image gcr.io/PROJECT_ID/my-python-app \ --platform managed \ --region us-central1 \ --allow-unauthenticated \ --set-env-vars cloud_id=YOUR_CLOUD_ID,cloud_pass=YOUR_CLOUD_PASS,cloud_user=YOUR_CLOUD_USER

You should see your service running in Cloud Run:

Note that you can also activate the continuous integration so that any commit in your GitHub repository will trigger a redeploy. On the service details page, click on Set up continuous deployment.

Installing the plugin in ChatGPT

Once the plugin is deployed and has a publicly accessible endpoint, it can be installed in ChatGPT. In our case, since this is deployed in Google Cloud Run, you can get the URL here:

Then in ChatGPT, go in the plugin store:

Choose to do “Develop your own plugin”:

Paste the URL you copied from the Google Cloud Run page:

Ensure the plugin is found and valid:

Follow the installation instructions until you see your plugin available in the list:

Let’s test our plugin!

OK, now for the best part! Do remember that ChatGPT decides to delegate when your prompt goes beyond its knowledge. To ensure that happens, just ask a question similar to this example:

With the steps provided in this blog, you can create your own plugin and deploy it on a cloud provider or your own hosts. This allows you to start exploring enhancing ChatGPT's knowledge and functionality, enhancing an already amazing tool with specialized and proprietary knowledge.

You can try all of the capabilities discussed in this blog today! Get started by signing up for a free Elastic Cloud trial.

Here are some other blogs you may find interesting:

In this blog post, we may have used third party generative AI tools, which are owned and operated by their respective owners. Elastic does not have any control over the third party tools and we have no responsibility or liability for their content, operation or use, nor for any loss or damage that may arise from your use of such tools. Please exercise caution when using AI tools with personal, sensitive or confidential information. Any data you submit may be used for AI training or other purposes. There is no guarantee that information you provide will be kept secure or confidential. You should familiarize yourself with the privacy practices and terms of use of any generative AI tools prior to use.

Elastic, Elasticsearch and associated marks are trademarks, logos or registered trademarks of Elasticsearch N.V. in the United States and other countries. All other company and product names are trademarks, logos or registered trademarks of their respective owners.