Elasticsearch에서 ELSER를 사용한 시맨틱 검색
텍스트 확장을 활용한 시맨틱 검색에 Elasticsearch ELSER 모델을 사용하는 방법을 확인해 보세요. 참고: 이 노트북은 향상된 검색 정확도를 제공하는 elser_model_2를 사용하는 방법을 보여줍니다.
Colab에서 열기노트북 다운로드Learn how to use the ELSER for text expansion-powered semantic search.
Note: This notebook demonstrates how to use ELSER model .elser_model_2 model which offers an improved retrieval accuracy.
If you have set up an index with ELSER model .elser_model_1, and would like to upgrade to ELSER v2 model - .elser_model_2, Please follow instructions from the notebook on how to upgrade an index to use elser model
To get started, we'll need to connect to our Elastic deployment using the Python client. Because we're using an Elastic Cloud deployment, we'll use the Cloud ID to identify our deployment.
First we need to pip install the following packages:
elasticsearch
!pip install -qU "elasticsearch<9"Next, we need to import the modules we need.
🔐 NOTE: getpass enables us to securely prompt the user for credentials without echoing them to the terminal, or storing it in memory.
from elasticsearch import Elasticsearch, helpers, exceptions
from urllib.request import urlopen
from getpass import getpass
import json
import timeNow we can instantiate the Python Elasticsearch client.
First we prompt the user for their password and Cloud ID.
Then we create a client object that instantiates an instance of the Elasticsearch class.
# https://www.elastic.co/search-labs/tutorials/install-elasticsearch/elastic-cloud#finding-your-cloud-id
ELASTIC_CLOUD_ID = getpass("Elastic Cloud ID: ")
# https://www.elastic.co/search-labs/tutorials/install-elasticsearch/elastic-cloud#creating-an-api-key
ELASTIC_API_KEY = getpass("Elastic Api Key: ")
# Create the client instance
client = Elasticsearch(
# For local development
# hosts=["http://localhost:9200"]
cloud_id=ELASTIC_CLOUD_ID,
api_key=ELASTIC_API_KEY,
)Enable Telemetry
Knowing that you are using this notebook helps us decide where to invest our efforts to improve our products. We would like to ask you that you run the following code to let us gather anonymous usage statistics. See telemetry.py for details. Thank you!
!curl -O -s https://raw.githubusercontent.com/elastic/elasticsearch-labs/main/telemetry/telemetry.py
from telemetry import enable_telemetry
client = enable_telemetry(client, "03-ELSER")Test the Client
Before you continue, confirm that the client has connected with this test.
print(client.info())Refer to https://www.elastic.co/guide/en/elasticsearch/client/python-api/current/connecting.html#connect-self-managed-new to learn how to connect to a self-managed deployment.
Read https://www.elastic.co/guide/en/elasticsearch/client/python-api/current/connecting.html#connect-self-managed-new to learn how to connect using API keys.
In this example, we are going to download and deploy the ELSER model in our ML node. Make sure you have an ML node in order to run the ELSER model.
# delete model if already downloaded and deployed
try:
client.ml.delete_trained_model(model_id=".elser_model_2", force=True)
print("Model deleted successfully, We will proceed with creating one")
except exceptions.NotFoundError:
print("Model doesn't exist, but We will proceed with creating one")
# Creates the ELSER model configuration. Automatically downloads the model if it doesn't exist.
client.ml.put_trained_model(
model_id=".elser_model_2", input={"field_names": ["text_field"]}
)The above command will download the ELSER model. This will take a few minutes to complete. Use the following command to check the status of the model download.
while True:
status = client.ml.get_trained_models(
model_id=".elser_model_2", include="definition_status"
)
if status["trained_model_configs"][0]["fully_defined"]:
print("ELSER Model is downloaded and ready to be deployed.")
break
else:
print("ELSER Model is downloaded but not ready to be deployed.")
time.sleep(5)Once the model is downloaded, we can deploy the model in our ML node. Use the following command to deploy the model.
# Start trained model deployment if not already deployed
client.ml.start_trained_model_deployment(
model_id=".elser_model_2", number_of_allocations=1, wait_for="starting"
)
while True:
status = client.ml.get_trained_models_stats(
model_id=".elser_model_2",
)
if status["trained_model_stats"][0]["deployment_stats"]["state"] == "started":
print("ELSER Model has been successfully deployed.")
break
else:
print("ELSER Model is currently being deployed.")
time.sleep(5)This also will take a few minutes to complete.
In order to use ELSER on our Elastic Cloud deployment we'll need to create an ingest pipeline that contains an inference processor that runs the ELSER model.
Let's add that pipeline using the put_pipeline method.
client.ingest.put_pipeline(
id="elser-ingest-pipeline",
description="Ingest pipeline for ELSER",
processors=[
{
"inference": {
"model_id": ".elser_model_2",
"input_output": [
{"input_field": "plot", "output_field": "plot_embedding"}
],
}
}
],
)ObjectApiResponse({'acknowledged': True})Let's note a few important parameters from that API call:
inference: A processor that performs inference using a machine learning model.model_id: Specifies the ID of the machine learning model to be used. In this example, the model ID is set to.elser_model_2.input_output: Specifies input and output fieldsinput_field: Field name from which thesparse_vectorrepresentation are created.output_field: Field name which contains inference results.
Create index
To use the ELSER model at index time, we'll need to create an index mapping that supports a text_expansion query.
The mapping includes a field of type sparse_vector to work with our feature vectors of interest.
This field contains the token-weight pairs the ELSER model created based on the input text.
Let's create an index named elser-example-movies with the mappings we need.
client.indices.delete(index="elser-example-movies", ignore_unavailable=True)
client.indices.create(
index="elser-example-movies",
settings={"index": {"default_pipeline": "elser-ingest-pipeline"}},
mappings={
"properties": {
"plot": {
"type": "text",
"fields": {"keyword": {"type": "keyword", "ignore_above": 256}},
},
"plot_embedding": {"type": "sparse_vector"},
}
},
)ObjectApiResponse({'acknowledged': True, 'shards_acknowledged': True, 'index': 'elser-example-movies'})Insert Documents
Let's insert our example dataset of 12 movies.
If you get an error, check the model has been deployed and is available in the ML node. In newer versions of Elastic Cloud, ML node is autoscaled and the ML node may not be ready yet. Wait for a few minutes and try again.
url = "https://raw.githubusercontent.com/elastic/elasticsearch-labs/main/notebooks/search/movies.json"
response = urlopen(url)
# Load the response data into a JSON object
data_json = json.loads(response.read())
# Prepare the documents to be indexed
documents = []
for doc in data_json:
documents.append(
{
"_index": "elser-example-movies",
"_source": doc,
}
)
# Use helpers.bulk to index
helpers.bulk(client, documents)
print("Done indexing documents into `elser-example-movies` index!")
time.sleep(3)Inspect a new document to confirm that it now has an plot_embedding field that contains a list of new, additional terms.
These terms are the text expansion of the field(s) you targeted for ELSER inference in input_field while creating the pipeline.
ELSER essentially creates a tree of expanded terms to improve the semantic searchability of your documents.
We'll be able to search these documents using a text_expansion query.
But first let's start with a simple keyword search, to see how ELSER delivers semantically relevant results out of the box.
Let's test out semantic search using ELSER.
response = client.search(
index="elser-example-movies",
size=3,
query={
"text_expansion": {
"plot_embedding": {
"model_id": ".elser_model_2",
"model_text": "fighting movie",
}
}
},
)
for hit in response["hits"]["hits"]:
doc_id = hit["_id"]
score = hit["_score"]
title = hit["_source"]["title"]
plot = hit["_source"]["plot"]
print(f"Score: {score}\nTitle: {title}\nPlot: {plot}\n")Next Steps
Now that we have a working example of semantic search using ELSER, you can try it out on your own data. Don't forget to scale down the ML node when you are done.