Using Eland on Elasticsearch Serverless

This blog will show you how to use Eland to import machine learning models to Elasticsearch Serverless, and then how to explore Elasticsearch using a Pandas-like API.

NLP in Elasticsearch Serverless & Eland

Since Elasticsearch 8.0, it is possible to use NLP machine learning models directly from Elasticsearch. While some models such as ELSER (for English data) or E5 (for multilingual data) can be deployed directly from Kibana, all other compatible PyTorch models need to be uploaded using Eland.

Since Eland 8.14.0, eland_import_hub_model fully supports Serverless. To get the connection details, open your Serverless project in Kibana, select the "cURL" client, create an API key, and export the environment variables:

export ES_URL="https://[...].elastic.cloud:443"
export API_KEY="..."

You can then use those variables when running eland_import_hub_model:

$ docker run -it --rm --network host \
    docker.elastic.co/eland/eland \
    eland_import_hub_model \
      --url $ES_URL \
      --es-api-key $API_KEY \
      --hub-model-id elastic/distilbert-base-cased-finetuned-conll03-english \
      --task-type ner

Next, search for "Trained Models" in Kibana, which will offer to synchronize your trained models.

Once done, you will get the option to deploy your model:

Less than a minute later, your model should be deployed and you'll be able to test it directly from Kibana.

In this test sentence, the model successfully identified Joe as "Person" and "Reunion Island" as a location, with high probability.

For more details on using Eland for machine learning models (including scikit-learn, XGBoost and LightGBM, not covered here), consider reading the detailed Accessing machine learning models in Elastic blog post and referring to the Eland documentation.

Data frames in Eland

The other main functionality of Eland is exploring Elasticsearch data using a Pandas-like API.

Ingesting test data

Let's first index some test data to Elasticsearch. We'll use a fake flights dataset. While uploading using the Python Elasticsearch client is possible, in this post we'll use Kibana's file upload functionality instead, which is enough for quick tests.

1. First, download the dataset https://github.com/elastic/eland/blob/main/tests/flights.json.gz and decompress it (gunzip flights.json.gz).

2. Next, type "File Upload" in Kibana's search bar and import the flights.json file.

3. Kibana will show you the resulting fields, with "Cancelled" detected as a boolean, for example. Click on "Import".

4. On the next screen, choose "flights" for the index name and click "Import" again.

As in the screenshot below, you should see that the 13059 documents were successfully ingested in the "flights" index.

Connecting to Elasticsearch

Now that we have data to search, let's setup the Elasticsearch Serverless Python client. (While we could use the main client, the Serverless Elasticsearch Python client is usually easier to use, as it only supports Elasticsearch Serverless features and APIs.) From the Kibana home page, you can select Python which will explain how to install the Elasticsearch Serverless Python client, create an API key, and use it in your code. You should end up with this code:

from elasticsearch_serverless import Elasticsearch

client = Elasticsearch(
    "https://[...].es.eu-west-1.aws.elastic.cloud:443",
    api_key="your_api_key"
)

print(client.info())

Searching data with Eland

Finally, assuming that the above code worked, we can start using Eland. After having installed it with python -m pip install eland>=8.14, we can start exploring our flights dataset.

import eland as ed
from elasticsearch_serverless import Elasticsearch

client = Elasticsearch("https//...", api_key="...)
df = ed.DataFrame(client, es_index_pattern="flights")
df.head()

If you run this code in a notebook, the result will be the following table:

You can also run more complex queries such as aggregations:

df[["DistanceKilometers", "AvgTicketPrice"]].aggregate(["sum", "min", "std"])

which outputs the following:

The demo notebook in the documentation has many more examples that use the same dataset and the reference documentation lists all supported operations.