Elasticsearch에 임베딩 모델 로드
벡터 검색 및 텍스트 임베딩 생성을 위해 Hugging Face NLP 모델을 Elasticsearch에 배포하는 방법을 확인해 보세요.
Colab에서 열기노트북 다운로드The workbook implements NLP text search in Elasticsearch using a simple dataset consisting of Elastic blogs titles.
You will index blogs documents, and using ingest pipeline generate text embeddings. By using NLP model you will query the documents using natural language over the the blogs documents.
Prerequisities
Before we begin, create an elastic cloud deployment and autoscale to have least one machine learning (ML) node with enough (4GB) memory. Also ensure that the Elasticsearch cluster is running.
If you don't already have an Elastic deployment, you can sign up for a free Elastic Cloud trial.
Install packages and import modules
Before you start you need to install all required Python dependencies.
!python3 -m pip install sentence-transformers==2.7.0 "eland<9" "elasticsearch<9" transformers# import modules
from elasticsearch import Elasticsearch
from getpass import getpass
from urllib.request import urlopen
import json
from time import sleepDeploy an NLP model
We are using the eland tool to install a text_embedding model. For our model, We have used all-MiniLM-L6-v2 to transform the search text into the dense vector.
The model will transfer your search query into vector which will be used for the search over the set of documents stored in Elasticsearch.
Install text embedding NLP model
Using the eland_import_hub_model script, download and install all-MiniLM-L6-v2 transformer model. Setting the NLP --task-type as text_embedding.
To get the cloud id, go to Elastic cloud and On the deployment overview page, copy down the Cloud ID.
To authenticate your request, You could use API key. Alternatively, you can use your cloud deployment username and password.
# https://www.elastic.co/search-labs/tutorials/install-elasticsearch/elastic-cloud#finding-your-cloud-id
ELASTIC_CLOUD_ID = getpass("Elastic Cloud ID: ")
# https://www.elastic.co/search-labs/tutorials/install-elasticsearch/elastic-cloud#creating-an-api-key
ELASTIC_API_KEY = getpass("Elastic Api Key: ")!eland_import_hub_model --cloud-id $ELASTIC_CLOUD_ID --hub-model-id sentence-transformers/all-MiniLM-L6-v2 --task-type text_embedding --es-api-key $ELASTIC_API_KEY --start --clear-previousConnect to Elasticsearch cluster
Create a elasticsearch client instance with your deployment Cloud Id and API Key. In this example, we are using the API_KEY and CLOUD_ID value from previous step.
Alternately you could use your deployment Username and Password to authenticate your instance.
es = Elasticsearch(
cloud_id=ELASTIC_CLOUD_ID, api_key=ELASTIC_API_KEY, request_timeout=600
)
es.info() # should return cluster infoCreate an Ingest pipeline
We need to create a text embedding ingest pipeline to generate vector (text) embeddings for title field.
The pipeline below is defining a processor for the inference to the NLP model.
# ingest pipeline definition
PIPELINE_ID = "vectorize_blogs"
es.ingest.put_pipeline(
id=PIPELINE_ID,
processors=[
{
"inference": {
"model_id": "sentence-transformers__all-minilm-l6-v2",
"target_field": "text_embedding",
"field_map": {"title": "text_field"},
}
}
],
)Create Index with mappings
We will now create an elasticsearch index with correct mapping before we index documents.
We are adding text_embedding to include the model_id and predicted_value to store the embeddings.
# define index name
INDEX_NAME = "blogs"
# flag to check if index has to be deleted before creating
SHOULD_DELETE_INDEX = True
# define index mapping
INDEX_MAPPING = {
"properties": {
"title": {
"type": "text",
"fields": {"keyword": {"type": "keyword", "ignore_above": 256}},
},
"text_embedding": {
"properties": {
"is_truncated": {"type": "boolean"},
"model_id": {
"type": "text",
"fields": {"keyword": {"type": "keyword", "ignore_above": 256}},
},
"predicted_value": {
"type": "dense_vector",
"dims": 384,
"index": True,
"similarity": "l2_norm",
},
}
},
}
}
INDEX_SETTINGS = {
"index": {
"number_of_replicas": "1",
"number_of_shards": "1",
"default_pipeline": PIPELINE_ID,
}
}
# check if we want to delete index before creating the index
if SHOULD_DELETE_INDEX:
if es.indices.exists(index=INDEX_NAME):
print("Deleting existing %s" % INDEX_NAME)
es.indices.delete(index=INDEX_NAME, ignore=[400, 404])
print("Creating index %s" % INDEX_NAME)
es.indices.create(
index=INDEX_NAME, mappings=INDEX_MAPPING, settings=INDEX_SETTINGS, ignore=[400, 404]
)Index data to elasticsearch index
Let's index sample blogs data using the ingest pipeline.
Note: Before we begin indexing, ensure you have started your trained model deployment.
url = "https://raw.githubusercontent.com/elastic/elasticsearch-labs/main/notebooks/integrations/hugging-face/blogs.json"
response = urlopen(url)
titles = json.loads(response.read())
actions = []
for title in titles:
actions.append({"index": {"_index": "blogs"}})
actions.append(title)
es.bulk(index="blogs", operations=actions)
sleep(5)Querying the dataset
The next step is to run a query to search for relevant blogs. The example query searches for "model_text": "how to track network connections" using the model we uploaded to Elasticsearch sentence-transformers__all-minilm-l6-v2.
The process is one query even though it internally consists of two tasks. First, the query will generate an vector for your search text using the NLP model and then pass this vector to search over the dataset.
As a result, the output shows the list of query documents sorted by their proximity to the search query.
INDEX_NAME = "blogs"
source_fields = ["id", "title"]
query = {
"field": "text_embedding.predicted_value",
"k": 5,
"num_candidates": 50,
"query_vector_builder": {
"text_embedding": {
"model_id": "sentence-transformers__all-minilm-l6-v2",
"model_text": "how to track network connections",
}
},
}
response = es.search(index=INDEX_NAME, fields=source_fields, knn=query, source=False)
def show_results(results):
for result in results:
print(f'{result["fields"]["title"]}\nScore: {result["_score"]}\n')
show_results(response.body["hits"]["hits"])