LlamaIndex를 통해 Elasticsearch로 데이터를 수집하는 방법

File Name: what-is-elasticsearch-service.txt
Content: Elasticsearch Service is hosted and managed Elasticsearch and Kibana brought to you by the creators of Elasticsearch. Elasticsearch Service is part of Elastic Cloud and ships with features that you can only get from the company behind Elasticsearch, Kibana, Beats, and Logstash. Elasticsearch is a full text search engine that suits a range of uses, from search on websites to big data analytics and more.

llama-index-vector-stores-elasticsearch
llama-index
openai

version: '3.8'
services:

 elasticsearch:
   image: docker.elastic.co/elasticsearch/elasticsearch:8.16.2
   container_name: elasticsearch-8.16.2
   environment:
     - node.name=elasticsearch
     - xpack.security.enabled=false
     - discovery.type=single-node
     - "ES_JAVA_OPTS=-Xms1024m -Xmx1024m"
   ports:
     - 9200:9200
   networks:
     - shared_network

 kibana:
   image: docker.elastic.co/kibana/kibana:8.16.2
   container_name: kibana-8.16.2
   restart: always
   environment:
     - ELASTICSEARCH_URL=http://elasticsearch:9200
   ports:
     - 5601:5601
   depends_on:
     - elasticsearch
   networks:
     - shared_network

networks:
 shared_network:

documents = SimpleDirectoryReader("./faq").load_data()

storage_context = StorageContext.from_defaults(vector_store=es)
index = VectorStoreIndex(documents, storage_context=storage_context, embed_model=embed_model)

import openai
from llama_index.core import VectorStoreIndex, SimpleDirectoryReader, StorageContext
from llama_index.embeddings.openai import OpenAIEmbedding
from llama_index.vector_stores.elasticsearch import ElasticsearchStore

openai.api_key = os.environ["OPENAI_API_KEY"]

es = ElasticsearchStore(
   index_name="faq",
   es_url="http://localhost:9200"
)

def format_title(filename):
   filename_without_ext = filename.replace('.txt', '')
   text_with_spaces = filename_without_ext.replace('-', ' ')
   formatted_text = text_with_spaces.title()

   return formatted_text


embed_model = OpenAIEmbedding(model="text-embedding-3-small")

documents = SimpleDirectoryReader("./faq").load_data()

for doc in documents:
   doc.metadata['title'] = format_title(doc.metadata['file_name'])

storage_context = StorageContext.from_defaults(vector_store=es)
index = VectorStoreIndex(documents, storage_context=storage_context, embed_model=embed_model)

es = ElasticsearchStore(
   index_name="faq",
   es_url="http://localhost:9200",
   retrieval_strategy=AsyncDenseVectorStrategy(
   )
)

   index = VectorStoreIndex.from_vector_store(vector_store=es)
   retriever = index.as_retriever(similarity_top_k=5)
   results = retriever.retrieve(query)

qa_prompt = PromptTemplate(
   "You are a helpful and knowledgeable assistant."
   "Your task is to answer the user's query based solely on the context provided below."
   "Do not use any prior knowledge or external information.\n"
   "---------------------\n"
   "Context:\n"
   "{context_str}\n"
   "---------------------\n"
   "Query: {query_str}\n"
   "Instructions:\n"
   "1. Carefully read and understand the context provided.\n"
   "2. If the context contains enough information to answer the query, provide a clear and concise answer.\n"
   "3. Do not make up or guess any information.\n"
   "Answer: "
)

llm = OpenAI(model="gpt-4o")
context_str = "\n\n".join([n.node.get_content() for n in results])
response = llm.complete(
   qa_prompt.format(context_str=context_str, query_str=query)
)

print("Answer:")
print(response)

es = ElasticsearchStore(
   index_name="faq",
   es_url="http://localhost:9200",
   retrieval_strategy=AsyncDenseVectorStrategy(
   )
)


def print_results(results):
   for rank, result in enumerate(results, start=1):
       title = result.metadata.get("title")
       score = result.get_score()
       text = result.get_text()
       print(f"{rank}. title={title} \nscore={score} \ncontent={text}")


def search(query: str):
   index = VectorStoreIndex.from_vector_store(vector_store=es)

   retriever = index.as_retriever(similarity_top_k=10)
   results = retriever.retrieve(QueryBundle(query_str=query))
   print_results(results)

   qa_prompt = PromptTemplate(
       "You are a helpful and knowledgeable assistant."
       "Your task is to answer the user's query based solely on the context provided below."
       "Do not use any prior knowledge or external information.\n"
       "---------------------\n"
       "Context:\n"
       "{context_str}\n"
       "---------------------\n"
       "Query: {query_str}\n"
       "Instructions:\n"
       "1. Carefully read and understand the context provided.\n"
       "2. If the context contains enough information to answer the query, provide a clear and concise answer.\n"
       "3. Do not make up or guess any information.\n"
       "Answer: "
   )

   llm = OpenAI(model="gpt-4o")
   context_str = "\n\n".join([n.node.get_content() for n in results])
   response = llm.complete(
       qa_prompt.format(context_str=context_str, query_str=query)
   )

   print("Answer:")
   print(response)


question = "Elastic services are free?"
print(f"Question: {question}")
search(question)

Question: Elastic services are free?
Answer:
Elastic services are not entirely free. However, there is a 14-day free trial available for exploring Elastic solutions. After the trial, access to features and services depends on the subscription level.

1. title=Can I Try Elasticsearch Service For Free 
score=1.0 
content=Yes, sign up for a 14-day free trial. The trial starts the moment a cluster is created.
During the free trial period get access to a deployment to explore Elastic solutions for Enterprise Search, Observability, Security, or the latest version of the Elastic Stack.

2. title=Do You Offer Elastic S Commercial Products 
score=0.9941274512218439 
content=Yes, all Elasticsearch Service customers have access to basic authentication, role-based access control, and monitoring.
Elasticsearch Service Gold, Platinum and Enterprise customers get complete access to all the capabilities in X-Pack: Security, Alerting, Monitoring, Reporting, Graph Analysis & Visualization. Contact us to learn more.

3. title=What Is Elasticsearch Service 
score=0.9896776845746571 
content=Elasticsearch Service is hosted and managed Elasticsearch and Kibana brought to you by the creators of Elasticsearch. Elasticsearch Service is part of Elastic Cloud and ships with features that you can only get from the company behind Elasticsearch, Kibana, Beats, and Logstash. Elasticsearch is a full text search engine that suits a range of uses, from search on websites to big data analytics and more.

4. title=Can I Run The Full Elastic Stack In Elasticsearch Service 
score=0.9880631561979476 
content=Many of the products that are part of the Elastic Stack are readily available in Elasticsearch Service, including Elasticsearch, Kibana, plugins, and features such as monitoring and security. Use other Elastic Stack products directly with Elasticsearch Service. For example, both Logstash and Beats can send their data to Elasticsearch Service. What is run is determined by the subscription level.

5. title=What Is The Difference Between Elasticsearch Service And The Amazon Elasticsearch Service 
score=0.9835054890793161 
content=Elasticsearch Service is the only hosted and managed Elasticsearch service built, managed, and supported by the company behind Elasticsearch, Kibana, Beats, and Logstash. With Elasticsearch Service, you always get the latest versions of the software. Our service is built on best practices and years of experience hosting and managing thousands of Elasticsearch clusters in the Cloud and on premise. For more information, check the following Amazon and Elastic Elasticsearch Service comparison page.
Please note that there is no formal partnership between Elastic and Amazon Web Services (AWS), and Elastic does not provide any support on the AWS Elasticsearch Service.