블로그

고급 RAG 기술 2부: 쿼리 및 테스트

RAG 성능을 향상시킬 수 있는 기술을 논의하고 구현합니다. 2부 2부에서는 고급 RAG 파이프라인 쿼리 및 테스트에 중점을 둡니다.

모든 코드는 Searchlabs 리포지토리의 고급-걸레-기술 브랜치에서찾을 수 있습니다.

고급 RAG 기법에 대한 글 2부에 오신 것을 환영합니다! 이 시리즈의 1부에서는 고급 RAG 파이프라인의 데이터 처리 구성 요소를 설정하고, 논의하고, 구현하는 방법을 살펴봤습니다:

고급 RAG 파이프라인

저자가 사용한 RAG 파이프라인입니다.

이 부분에서는 쿼리 및 구현 테스트를 진행하겠습니다. 바로 본론으로 들어가 보겠습니다!

목차

검색 및 검색, 답변 생성

첫 번째 질문은 주로 연례 보고서에서 찾을 수 있는 정보에 대해 물어보겠습니다. 어때요?

Who audits Elastic?"

이제 몇 가지 기술을 적용하여 쿼리를 개선해 보겠습니다.

동의어로 쿼리 강화하기

먼저, 쿼리 문구의 다양성을 높이고 이를 Elasticsearch 쿼리로 쉽게 처리할 수 있는 형태로 바꿔보겠습니다. 쿼리를 OR 절의 목록으로 변환하기 위해 GPT-4o의 도움을 받겠습니다. 이 프롬프트를 작성해 보겠습니다:


ELASTIC_SEARCH_QUERY_GENERATOR_PROMPT = '''
You are an AI assistant specialized in generating Elasticsearch query strings. Your task is to create the most effective query string for the given user question. This query string will be used to search for relevant documents in an Elasticsearch index.

Guidelines:
1. Analyze the user's question carefully.
2. Generate ONLY a query string suitable for Elasticsearch's match query.
3. Focus on key terms and concepts from the question.
4. Include synonyms or related terms that might be in relevant documents.
5. Use simple Elasticsearch query string syntax if helpful (e.g., OR, AND).
6. Do not use advanced Elasticsearch features or syntax.
7. Do not include any explanations, comments, or additional text.
8. Provide only the query string, nothing else.

For the question "What is Clickthrough Data?", we would expect a response like:
clickthrough data OR click-through data OR click through rate OR CTR OR user clicks OR ad clicks OR search engine results OR web analytics

AND operator is not allowed. Use only OR.

User Question:
[The user's question will be inserted here]

Generate the Elasticsearch query string:
'''

쿼리에 적용하면 GPT-4o는 기본 쿼리의 동의어와 관련 어휘를 생성합니다.

'audits elastic OR 
elasticsearch audits OR 
elastic auditor OR 
elasticsearch auditor OR 
elastic audit firm OR 
elastic audit company OR 
elastic audit organization OR 
elastic audit service'

ESQueryMaker 클래스에서 쿼리를 분할하는 함수를 정의했습니다:

def parse_or_query(self, query_text: str) -> List[str]:
    # Split the query by 'OR' and strip whitespace from each term
    # This converts a string like "term1 OR term2 OR term3" into a list ["term1", "term2", "term3"]
    return [term.strip() for term in query_text.split(' OR ')]

이 OR 절의 문자열을 가져와 용어 목록으로 분할하여 주요 문서 필드에서 다중 일치를 수행할 수 있도록 하는 역할을 합니다:

["original_text", 'keyphrases', 'potential_questions', 'entities']

마지막으로 이 쿼리로 마무리합니다:

 'query': {
    'bool': {
        'must': [
            {
                'multi_match': {
                'query': 'audits Elastic Elastic auditing Elastic audit process Elastic compliance Elastic security audit Elasticsearch auditing Elasticsearch compliance Elasticsearch security audit',
                'fields': [
                    'original_text',
                'keyphrases',
                'potential_questions',
                'entities'
                ],
                'type': 'best_fields',
                'operator': 'or'
                }
            }
      ]

이렇게 하면 원래 쿼리보다 더 많은 기반을 포함하므로 동의어를 잊어버려 검색 결과가 누락되는 위험을 줄일 수 있습니다. 하지만 더 많은 일을 할 수 있습니다.

맨 위로 돌아가기

HyDE(가상의 문서 임베딩)

이번에는 HyDE를 구현하기 위해 GPT-4o를 다시 사용해 보겠습니다.

HyDE의 기본 전제는 가상의 문서, 즉 원래 쿼리에 대한 답변이 포함될 가능성이 있는 문서를 생성하는 것입니다. 문서의 사실 여부나 정확성은 문제가 되지 않습니다. 이를 염두에 두고 다음 프롬프트를 작성해 보겠습니다:

HYDE_DOCUMENT_GENERATOR_PROMPT = '''
You are an AI assistant specialized in generating hypothetical documents based on user queries. Your task is to create a detailed, factual document that would likely contain the answer to the user's question. This hypothetical document will be used to enhance the retrieval process in a Retrieval-Augmented Generation (RAG) system.

Guidelines:
1. Carefully analyze the user's query to understand the topic and the type of information being sought.
2. Generate a hypothetical document that:
   a. Is directly relevant to the query
   b. Contains factual information that would answer the query
   c. Includes additional context and related information
   d. Uses a formal, informative tone similar to an encyclopedia or textbook entry
3. Structure the document with clear paragraphs, covering different aspects of the topic.
4. Include specific details, examples, or data points that would be relevant to the query.
5. Aim for a document length of 200-300 words.
6. Do not use citations or references, as this is a hypothetical document.
7. Avoid using phrases like "In this document" or "This text discusses" - write as if it's a real, standalone document.
8. Do not mention or refer to the original query in the generated document.
9. Ensure the content is factual and objective, avoiding opinions or speculative information.
10. Output only the generated document, without any additional explanations or meta-text.

User Question:
[The user's question will be inserted here]

Generate a hypothetical document that would likely contain the answer to this query:
'''

벡터 검색은 일반적으로 코사인 벡터 유사성을 기반으로 작동하므로, HyDE의 전제는 쿼리와 문서가 아닌 문서와 문서를 일치시킴으로써 더 나은 결과를 얻을 수 있다는 것입니다.

우리가 신경 쓰는 것은 구조, 흐름, 용어입니다. 사실과 다릅니다. GPT-4o는 다음과 같이 HyDE 문서를 출력합니다:

'Elastic N.V., the parent company of Elastic, the organization known for developing Elasticsearch, is subject to audits to ensure financial accuracy, 
regulatory compliance, and the integrity of its financial statements. The auditing of Elastic N.V. is typically conducted by an external, 
independent auditing firm. This is common practice for publicly traded companies to provide stakeholders with assurance regarding the company\'s 
financial position and operations.\n\nThe primary external auditor for Elastic is the audit firm Ernst & Young LLP (EY). Ernst & Young is one of the 
four largest professional services networks in the world, commonly referred to as the "Big Four" audit firms. These firms handle a substantial number 
of audits for major corporations around the globe, ensuring adherence to generally accepted accounting principles (GAAP) and international financial 
reporting standards (IFRS).\n\nThe audit process conducted by EY involves several steps. Initially, the auditors perform a risk assessment to identify 
areas where misstatements due to error or fraud could occur. They then design audit procedures to test the accuracy and completeness of financial statements,
 which include examining financial transactions, assessing internal controls, and reviewing compliance with relevant laws and regulations. Upon completion of 
 the audit, Ernst & Young issues an audit report, which includes the auditor’s opinion on whether the financial statements are free from material misstatement 
 and are presented fairly in accordance with the applicable financial reporting framework.\n\nIn addition to external audits by firms like Ernst & Young, 
 Elastic may also be subject to internal audits. Internal audits are performed by the company’s own internal auditors to evaluate the effectiveness of internal 
 controls, risk management, and governance processes.\n\nOverall, the auditing process plays a crucial role in maintaining the transparency and reliability of 
 Elastic\'s financial information, providing confidence to investors, regulators, and other stakeholders.'

색인하려는 종류의 문서에 이상적인 후보처럼 꽤 그럴듯해 보입니다. 이를 임베드하여 하이브리드 검색에 사용하겠습니다.

맨 위로 돌아가기

하이브리드 검색

이것이 바로 검색 로직의 핵심입니다. 어휘 검색 구성 요소는 생성된 OR 절 문자열이 됩니다. 고밀도 벡터 컴포넌트에는 HyDE 문서(일명 검색 벡터)가 내장됩니다. KNN을 사용하여 검색 벡터에 가장 가까운 여러 후보 문서를 효율적으로 식별합니다. 기본적으로 어휘 검색 컴포넌트를 TF-IDF 및 BM25를 사용한 점수화라고 부릅니다. 마지막으로, 어휘와 밀도 벡터 점수는 Wang 등이 권장하는 30/70 비율을 사용하여 결합됩니다.

def hybrid_vector_search(self, index_name: str, query_text: str, query_vector: List[float], 
                         text_fields: List[str], vector_field: str, 
                         num_candidates: int = 100, num_results: int = 10) -> Dict:
    """
    Perform a hybrid search combining text-based and vector-based similarity.

    Args:
        index_name (str): The name of the Elasticsearch index to search.
        query_text (str): The text query string, which may contain 'OR' separated terms.
        query_vector (List[float]): The query vector for semantic similarity search.
        text_fields (List[str]): List of text fields to search in the index.
        vector_field (str): The name of the field containing document vectors.
        num_candidates (int): Number of candidates to consider in the initial KNN search.
        num_results (int): Number of final results to return.

    Returns:
        Dict: A tuple containing the Elasticsearch response and the search body used.
    """
    try:
        # Parse the query_text into a list of individual search terms
        # This splits terms separated by 'OR' and removes any leading/trailing whitespace
        query_terms = self.parse_or_query(query_text)

        # Construct the search body for Elasticsearch
        search_body = {
            # KNN search component for vector similarity
            "knn": {
                "field": vector_field,  # The field containing document vectors
                "query_vector": query_vector,  # The query vector to compare against
                "k": num_candidates,  # Number of nearest neighbors to retrieve
                "num_candidates": num_candidates  # Number of candidates to consider in the KNN search
            },
            "query": {
                "bool": {
                    # The 'must' clause ensures that matching documents must satisfy this condition
                    # Documents that don't match this clause are excluded from the results
                    "must": [
                        {
                            # Multi-match query to search across multiple text fields
                            "multi_match": {
                                "query": " ".join(query_terms),  # Join all query terms into a single space-separated string
                                "fields": text_fields,  # List of fields to search in
                                "type": "best_fields",  # Use the best matching field for scoring
                                "operator": "or"  # Match any of the terms (equivalent to the original OR query)
                            }
                        }
                    ],
                    # The 'should' clause boosts relevance but doesn't exclude documents
                    # It's used here to combine vector similarity with text relevance
                    "should": [
                        {
                            # Custom scoring using a script to combine vector and text scores
                            "script_score": {
                                "query": {"match_all": {}},  # Apply this scoring to all documents that matched the 'must' clause
                                "script": {
                                    # Script to combine vector similarity and text relevance
                                    "source": """
                                    # Calculate vector similarity (cosine similarity + 1)
                                    # Adding 1 ensures the score is always positive
                                    double vector_score = cosineSimilarity(params.query_vector, params.vector_field) + 1.0;
                                    # Get the text-based relevance score from the multi_match query
                                    double text_score = _score;
                                    # Combine scores: 70% vector similarity, 30% text relevance
                                    # This weighting can be adjusted based on the importance of semantic vs keyword matching
                                    return 0.7 * vector_score + 0.3 * text_score;
                                    """,
                                    # Parameters passed to the script
                                    "params": {
                                        "query_vector": query_vector,  # Query vector for similarity calculation
                                        "vector_field": vector_field  # Field containing document vectors
                                    }
                                }
                            }
                        }
                    ]
                }
            }
        }

        # Execute the search request against the Elasticsearch index
        response = self.conn.search(index=index_name, body=search_body, size=num_results)
        # Log the successful execution of the search for monitoring and debugging
        logger.info(f"Hybrid search executed on index: {index_name} with text query: {query_text}")
        # Return both the response and the search body (useful for debugging and result analysis)
        return response, search_body
    except Exception as e:
        # Log any errors that occur during the search process
        logger.error(f"Error executing hybrid search on index: {index_name}. Error: {e}")
        # Re-raise the exception for further handling in the calling code
        raise e

마지막으로 RAG 함수를 조합할 수 있습니다. 쿼리부터 답변까지 RAG는 이 흐름을 따릅니다:

  1. 쿼리를 OR 절로 변환합니다.

  2. HyDE 문서를 생성하고 임베드합니다.

  3. 둘 다 하이브리드 검색에 입력으로 전달합니다.

  4. LLM의 컨텍스트 메모리(역 패킹) 역 패킹 예제에서 가장 관련성이 높은 점수가 "가장 최근의" 이 되도록 상위 n개의 결과를 검색하여 역 패킹합니다: 쿼리: "Elasticsearch 쿼리 최적화 기술" 검색된 문서(관련성 순으로 정렬): LLM 컨텍스트에 대한 순서가 역순입니다: 순서를 반대로 하면 가장 관련성이 높은 정보(1)가 컨텍스트의 마지막에 표시되어 답변 생성 중에 LLM으로부터 더 많은 관심을 받을 수 있습니다.

    1. "부울 쿼리를 사용하여 여러 검색 기준을 효율적으로 결합할 수 있습니다."

    2. "캐싱 전략을 구현하여 쿼리 응답 시간을 개선하세요."

    3. "더 빠른 검색 성능을 위해 인덱스 매핑을 최적화하세요."

    4. "더 빠른 검색 성능을 위해 인덱스 매핑을 최적화하세요."

    5. "캐싱 전략을 구현하여 쿼리 응답 시간을 개선하세요."

    6. "부울 쿼리를 사용하여 여러 검색 기준을 효율적으로 결합할 수 있습니다."

  5. 생성할 컨텍스트를 LLM에 전달합니다.

def get_context(index_name, 
                match_query, 
                text_query, 
                fields, 
                num_candidates=100, 
                num_results=20, 
                text_fields=["original_text", 'keyphrases', 'potential_questions', 'entities'], 
                embedding_field="primary_embedding"):

    embedding=embedder.get_embeddings_from_text(text_query)

    results, search_body = es_query_maker.hybrid_vector_search(
        index_name=index_name,
        query_text=match_query,
        query_vector=embedding[0][0],
        text_fields=text_fields,
        vector_field=embedding_field,
        num_candidates=num_candidates,
        num_results=num_results
    )

    # Concatenates the text in each 'field' key of the search result objects into a single block of text.
    context_docs=['\n\n'.join([field+":\n\n"+j['_source'][field] for field in fields]) for j in results['hits']['hits']]

    # Reverse Packing to ensure that the highest ranking document is seen first by the LLM.
    context_docs.reverse()
    return context_docs, search_body

def retrieval_augmented_generation(query_text):
    match_query= gpt4o.generate_query(query_text)
    fields=['original_text']

    hyde_document=gpt4o.generate_HyDE(query_text)

    context, search_body=get_context(index_name, match_query, hyde_document, fields)

    answer= gpt4o.basic_qa(query=query_text, context=context)
    return answer, match_query, hyde_document, context, search_body

쿼리를 실행하여 답변을 확인해 보겠습니다:

According to the context, Elastic N.V. is audited by an independent registered public accounting firm, PricewaterhouseCoopers (PwC). 
This information is found in the section titled "report of independent registered public accounting firm," which states:

"We have audited the accompanying consolidated balance sheets of Elastic N.V. [...] / s / pricewaterhouseco."

멋지네요. 맞습니다.

맨 위로 돌아가기

실험

지금 대답해야 할 중요한 질문이 있습니다. 이러한 구현에 많은 노력과 추가적인 복잡성을 투자하여 얻은 것은 무엇일까요?

간단한 비교를 해보겠습니다. 개선 사항 없이 기본 하이브리드 검색과 비교하여 구현한 RAG 파이프라인. 몇 가지 테스트를 실행하여 실질적인 차이를 발견할 수 있는지 확인해 보겠습니다. 방금 구현한 RAG를 AdvancedRAG라고 하고 기본 파이프라인을 SimpleRAG라고 하겠습니다.

간단한 RAG 파이프라인

종소리와 휘파람이 없는 간단한 RAG 파이프라인

결과 요약

이 표에는 두 RAG 파이프라인에 대한 5가지 테스트 결과가 요약되어 있습니다. 답변의 세부 사항과 품질을 기준으로 각 방법의 상대적 우열을 판단했지만 이는 전적으로 주관적인 판단입니다. 이 표 아래에 실제 답변이 재현되어 있으므로 참고하시기 바랍니다. 그럼 이제 그 결과를 살펴보도록 하겠습니다!

SimpleRAG는 질문 1에 답할 수 없었습니다 & 5. AdvancedRAG는 질문 2, 3, 4에 대해서도 훨씬 더 자세히 설명했습니다. 더 자세히 설명한 것을 바탕으로 저는 AdvancedRAG의 답변 품질이 더 좋다고 판단했습니다.

테스트

질문

고급RAG 성능

SimpleRAG 성능

AdvancedRAG 지연 시간

SimpleRAG 지연 시간

우승자

1

누가 Elastic을 감사하나요?

감사인으로 PwC를 올바르게 식별했습니다.

감사자를 식별하지 못했습니다.

11.6s

4.4s

AdvancedRAG

2

2023년 총 수익은 얼마였나요?

정확한 수익 수치를 제공했습니다. 전년도 수익에 대한 추가 컨텍스트가 포함되어 있습니다.

정확한 수익 수치를 제공했습니다.

13.3s

2.8s

AdvancedRAG

3

성장은 주로 어떤 제품에 의존하나요? 얼마예요?

핵심 동인으로 Elastic Cloud를 올바르게 식별했습니다. 전체 수익 컨텍스트 포함 & 더 자세한 내용.

핵심 동인으로 Elastic Cloud를 올바르게 식별했습니다.

14.1s

12.8s

AdvancedRAG

4

직원 복리후생 계획 설명

퇴직 계획, 건강 프로그램 및 기타 혜택에 대한 포괄적인 설명을 제공합니다. 연도별 구체적인 기부 금액이 포함되어 있습니다.

보상, 퇴직 계획, 근무 환경, Elastic Cares 프로그램을 포함한 복리후생에 대한 좋은 개요를 제공했습니다.

26.6s

11.6s

AdvancedRAG

5

Elastic은 어떤 회사를 인수했나요?

보고서에 언급된 최근 인수를 올바르게 나열했습니다(CmdWatch, 빌드 시큐리티, 옵티마이즈). 일부 인수 날짜와 구매 가격을 제공했습니다.

제공된 컨텍스트에서 관련 정보를 검색하지 못했습니다.

11.9s

2.7s

AdvancedRAG

테스트 1: 누가 Elastic을 감사하나요?

AdvancedRAG

According to the context, Elastic N.V. is audited by an independent 
registered public accounting firm, PricewaterhouseCoopers (PwC). 
This information is found in the section titled "report of 
independent registered public accounting firm," which states:
"We have audited the accompanying consolidated balance sheets 
of Elastic N.V. [...] / s / pricewaterhouseco."

SimpleRAG

According to the provided context, no specific auditing firm 
or company is mentioned as the auditor for Elastic. 
The context provided talks about various features, strengths, 
deployment options, and other details related to Elastic and 
its operations but does not specify which auditing firm is 
responsible for auditing Elastic's financial statements or 
internal controls.

요약: SimpleRAG는 PWC를 감사인으로 지정하지 않았습니다.

사실 꽤 놀랍습니다. SimpleRAG 측의 검색 실패로 보입니다. 감사 관련 문서가 검색되지 않았습니다. 다음 테스트에서는 난이도를 조금 낮춰보겠습니다.

테스트 2: 총 수익 2023년

AdvancedRAG

### Total Revenue for 2023

According to the provided context, the total revenue for the 
year ended April 30, 2023, was **$1,068,989,000**. This 
information can be found several times in the provided text. 
Specifically, one part states:

"The following table sets forth our results of operations 
for the periods presented in dollars and as a percentage of 
our total revenue. ... year ended April 30, 2023, ... 
total revenue 1,068,989".

Additionally, it is mentioned here as well:

"For the years ended April 30, 2023, 2022, and 2021, 
representing year-over-year growth of 24% for the year 
ended April 30, 2023... total revenue $1,068,989".

Therefore, the total revenue for 2023 is $1.069 billion.

SimpleRAG

According to the context provided, the total revenue for the fiscal year ended April 30, 2023, is:

$1,068,989,000 (or approximately $1.1 billion).

요약: 두 RAG 모두 2023년 총 수익 1,068,989,000달러라는 정답을 얻었습니다.

둘 다 바로 여기에 있었습니다. AdvancedRAG가 더 광범위한 문서를 확보한 것 같나요? 물론 답변은 더 상세하고 지난 몇 년간의 정보를 포함하고 있습니다. 개선 사항을 고려할 때 예상되는 부분이지만, 아직 확정하기에는 이르다고 할 수 있습니다.

난이도를 높여 보겠습니다.

테스트 3: 성장은 주로 어떤 제품에 의존하나요? 얼마예요?

AdvancedRAG

### Product Growth Primarily Depends On:
According to the context, "our revenue is derived primarily from 
subscriptions of search, observability and security built into 
the elastic stack." The document mentions various strategies and 
factors contributing to the company’s growth which include investment 
in "elastic cloud," "elastic stack," and the conversion of free 
users to paid subscribers.

### Quantitative Data:
- **Revenue**: The document states, "Our total revenue was $1.1 billion, 
$862.4 million, and $608.5 million for the years ended April 30, 2023, 
2022, and 2021, respectively."
- **Subscription Revenue**: "Subscriptions accounted for 92%, 93%, 
and 93% of our total revenue for the years ended April 30, 2023, 
2022, and 2021, respectively."
- **Elastic Cloud Contribution**: "For the years ended April 30, 2023, 
2022, and 2021, Elastic Cloud contributed 40%, 35%, and 27% of our 
total revenue, respectively."
- **Customer Growth**: The number of customers increased from 
"over 15,000 as of April 30, 2021," to "approximately 20,200 as of 
April 30, 2023."

### Summary:
Product growth for the company primarily depends on the "Elastic Stack" 
(which includes search, observability, and security solutions) and 
particularly the "Elastic Cloud." The Elastic Cloud's growing contribution 
to revenue highlights its significance, which grew from 27% in 2021 to 40% in 2023.

SimpleRAG

According to the provided context, growth primarily depends on **Elastic Cloud**, 
which is described as the company's "family of cloud-based offerings." 
The context also emphasizes the significance of Elastic Cloud in achieving 
the company's long-term growth potential and its role in increasing subscription revenue.

From the context:

- "Increasing adoption of Elastic Cloud... is an important growth opportunity 
for our business. Organizations are increasingly looking for hosted 
deployment alternatives with reduced administrative burdens."
- "We believe that offering Elastic Cloud is important for achieving 
our long-term growth potential, and we expect Elastic Cloud’s contribution 
to our subscription revenue to continue to increase."

In terms of the contribution to total revenue, here are the specific figures provided:

- Elastic Cloud contributed 40% of the total revenue for the year ended April 30, 2023.
- Elastic Cloud contributed 35% of the total revenue for the year ended April 30, 2022.
- Elastic Cloud contributed 27% of the total revenue for the year ended April 30, 2021.

This progression shows a clear upward trend in the importance and 
contribution of Elastic Cloud to the company's overall revenue.

요약: 두 RAG 모두 Elastic Cloud를 핵심 성장 동력으로 정확하게 파악했습니다. 그러나 AdvancedRAG에는 구독 수익과 고객 성장을 고려한 더 자세한 내용이 포함되어 있으며, 다른 Elastic 제품에 대해서도 명시적으로 언급하고 있습니다.

테스트 4: 직원 복리후생 계획 설명

AdvancedRAG

### Employee Benefit Plan Overview

The provided context outlines various elements of the employee benefit plan at 
Elastic, often referred to as "Elasticians." Here are the key components 
described in the texts:

1. **401(k) Plan**:
   - **Coverage**: The 401(k) plan is available to substantially all U.S. 
   employees who meet minimum age and service requirements.
   - **Contributions**: Elastic makes contributions to the 401(k) plan up to 
   6% of the participating employee’s W-2 earnings and wages.
   - **Expenses**: For the fiscal years ended April 30, Elastic recorded 
   expenses of $17.9 million (2023), $15.2 million (2022), and $11.4 million (2021) 
   related to the 401(k) plan.
   - **Defined-Contribution Plans in Other Countries**: Elastic has 
   defined-contribution plans in various other countries and recorded respective 
   expenses of $9.4 million (2023), $7.2 million (2022), and $5.1 million (2021).

2. **Stock-Based Compensation**:
   - **Types of Awards**: Stock options, restricted stock units (RSUs), 
   and shares under the Employee Stock Purchase Plan (ESPP).
   - **Fair Value Measurement**: Fair value of these stock awards is 
   measured using models like Black-Scholes.
   - **Employee Stock Purchase Plan (2022 ESPP)**: 
     - Started in 2022, it allows employees to acquire ordinary 
     shares at a discount (85% of the market value at the beginning 
     or end of the offering period).
     - Offering periods are approximately six months long.

3. **Total Rewards Compensation**:
   - **Components**: Includes cash compensation as well as equity awards, 
   reflecting a comprehensive interest in the well-being of the employees.
   - **Programs**: Market competitive medical and dental programs, 
   mental health, and holistic well-being programs.
   - **Paid Time Off**: Features 16 weeks of paid leave for new 
   parents and life-planning benefits.

4. **Retirement and Income Protection Plans**:
   - **401(k) Match**: Elastic provides a dollar-for-dollar 
   match up to 6% of eligible earnings up to a plan-limit 
   maximum for U.S.-based employees.
   - **Similar Plans Outside the U.S.**: Competitive retirement 
   plans available for employees based outside the United States.

5. **Flexible Work Environment**: 
   - **Distributed Workforce**: Elastic operates with a global 
   distributed workforce model, allowing employees to work remotely 
   and providing a broad candidate pool.
   
6. **Learning and Development**:
   - **Programs**: Access to on-demand video-based learning, 
   leadership development programs, and other specific training 
   to develop managers and leaders.

7. **Employee Engagement and Surveys**:
   - **Mechanisms**: Annual employee engagement survey and 
   mid-year pulse survey.
   - **Objectives**: To monitor morale, develop action plans, 
   and maintain high engagement scores.

8. **Onboarding Program**:
   - **X-School**: An extensive new-hire orientation program to 
   help new employees meet peers and learn about the company's 
   products and solutions.

In summary, Elastic's employee benefit plan encompasses a broad 
spectrum of financial, health, well-being, and professional development 
benefits designed to support and enhance the lives of its employees. 
The plan is both competitive and comprehensive, aiming to attract,
retain, and engage its global workforce.

SimpleRAG

The employee benefit plan at the Company includes several key 
components aimed at enhancing the overall well-being and 
professional development of employees, referred to as "Elasticians." 

1. **Compensation and Benefits:**
   - The Company provides market-competitive compensation 
   packages which typically include cash compensation and equity awards.
   - Comprehensive medical and dental programs are offered, 
   emphasizing mental health and holistic well-being.
   - Paid time off programs feature:
     - 16 weeks of paid leave for new parents.
     - Life-planning benefits.
     - Travel reimbursements for certain healthcare services.

2. **Retirement and Income Protection Plans:**
   - In the United States, employees are eligible for a 401k 
   plan with a dollar-for-dollar match by the Company up to 6% 
   of eligible earnings, subject to a plan limit.
   - Similar competitive retirement and income protection plans 
   are available for employees outside the United States.

3. **Work Environment:**
   - The Company offers a flexible work environment, allowing most 
   employees to work from anywhere as often as they would like.
   - Physical offices around the world are available for employees 
   who prefer to work on-site or need a space for face-to-face interactions.

4. **Community Involvement:**
   - The Elastic Cares Program supports charitable organizations 
   that matter to employees on both local and global levels.
   - The program includes:
     - Donation matching.
     - Providing the Company's technology for free to certain 
     nonprofit organizations.
     - Volunteer time off initiative.

요약: AdvancedRAG는 미국 거주 직원을 위한 401K 플랜을 언급하고 미국 외 지역의 기여금 플랜을 정의하는 등 훨씬 더 깊이 있고 자세하게 설명합니다. 또한 건강 및 웰빙 플랜에 대해서는 언급하고 있지만 SimpleRAG에서 언급하는 Elastic Cares 프로그램은 누락되어 있습니다.

테스트 5: Elastic은 어떤 회사를 인수했나요?

AdvancedRAG

Elastic acquired several companies over the years to enhance its 
technology and market presence. According to the provided context, 
the specific acquisitions include:

1. **CmdWatch Security Inc.**: Acquired on September 17, 2021. 
The total purchase consideration was $77.8 million.
2. **Build Security Ltd.**: Acquired on September 2, 2021. The 
combined purchase price for Build Security Ltd. and another acquisition was $57.2 million.
3. **Optimyze Cloud Inc.**: Acquired on November 1, 2021. Part 
of the combined purchase price mentioned above with Build Security Ltd.
4. **Endgame, Inc.**: Although the context mentions an amended 
and restated stock incentive plan, it indicates that Endgame, Inc. was 
associated with Elastic, suggesting it could have been acquired at some point.

These acquisitions are part of Elastic's strategy to 
enhance the technology underlying its security and 
observability offerings.

SimpleRAG

The provided context does not specify any companies that Elastic has acquired. 
Therefore, based on the context, there is no information available about the companies acquired by Elastic.

요약: SimpleRAG가 인수에 대한 관련 정보를 검색하지 않아 답변이 실패했습니다. AdvancedRAG는 보고서에 나열된 주요 인수 기업인 CmdWatch, Build Security 및 Optimyze를 올바르게 나열합니다.

맨 위로 돌아가기

결론

테스트 결과, 고급 기술은 제시되는 정보의 범위와 깊이를 증가시켜 잠재적으로 RAG 답변의 품질을 향상시키는 것으로 나타났습니다.

또한 Which companies did Elastic acquire?, Who audits Elastic 같은 모호한 단어로 된 질문은 AdvancedRAG에서는 정답을 맞혔지만 SimpleRAG에서는 그렇지 않았기 때문에 신뢰도가 개선될 수 있습니다.

그러나 5건 중 3건의 경우, 하이브리드 검색을 포함하지만 다른 기술은 사용하지 않는 기본 RAG 파이프라인이 대부분의 핵심 정보를 포착하는 답변을 생성했다는 점을 유념할 필요가 있습니다.

데이터 준비 및 쿼리 단계에 LLM이 통합되어 있기 때문에 AdvancedRAG의 지연 시간은 일반적으로 SimpleRAG보다 2~5배 더 길다는 점에 유의해야 합니다. 이는 상당한 비용이므로 지연 시간보다 응답 품질이 우선시되는 상황에서만 AdvancedRAG가 적합할 수 있습니다.

데이터 준비 단계에서 클로드 하이쿠나 GPT-4o-mini와 같은 더 작고 저렴한 LLM을 사용하면 상당한 지연 비용을 줄일 수 있습니다. 답변 생성을 위해 고급 모델을 저장합니다.

이는 Wang 등의 연구 결과와 일치합니다. 연구 결과에서 알 수 있듯이 개선 사항은 비교적 점진적으로 이루어졌습니다. 간단히 말해, 간단한 기본 RAG를 사용하면 저렴하고 빠르게 부팅할 수 있으면서도 괜찮은 최종 제품을 만들 수 있습니다. 저에게는 흥미로운 결론입니다. 속도와 효율성이 중요한 사용 사례의 경우 SimpleRAG가 현명한 선택입니다. 마지막 한 방울의 성능까지 끌어올려야 하는 사용 사례의 경우 AdvancedRAG에 통합된 기술이 한 가지 방법을 제시할 수 있습니다.

왕 파이프라인

Wang 등의 연구 결과에 따르면 고급 기술을 사용하면 일관적이면서도 점진적인 개선 효과를 얻을 수 있습니다.

맨 위로 돌아가기

부록

프롬프트

RAG 질문 답변 프롬프트

쿼리 및 컨텍스트에 따라 LLM이 답변을 생성하도록 하기 위한 프롬프트입니다.

BASIC_RAG_PROMPT = '''
You are an AI assistant tasked with answering questions based primarily on the provided context, while also drawing on your own knowledge when appropriate. Your role is to accurately and comprehensively respond to queries, prioritizing the information given in the context but supplementing it with your own understanding when beneficial. Follow these guidelines:

1. Carefully read and analyze the entire context provided.
2. Primarily focus on the information present in the context to formulate your answer.
3. If the context doesn't contain sufficient information to fully answer the query, state this clearly and then supplement with your own knowledge if possible.
4. Use your own knowledge to provide additional context, explanations, or examples that enhance the answer.
5. Clearly distinguish between information from the provided context and your own knowledge. Use phrases like "According to the context..." or "The provided information states..." for context-based information, and "Based on my knowledge..." or "Drawing from my understanding..." for your own knowledge.
6. Provide comprehensive answers that address the query specifically, balancing conciseness with thoroughness.
7. When using information from the context, cite or quote relevant parts using quotation marks.
8. Maintain objectivity and clearly identify any opinions or interpretations as such.
9. If the context contains conflicting information, acknowledge this and use your knowledge to provide clarity if possible.
10. Make reasonable inferences based on the context and your knowledge, but clearly identify these as inferences.
11. If asked about the source of information, distinguish between the provided context and your own knowledge base.
12. If the query is ambiguous, ask for clarification before attempting to answer.
13. Use your judgment to determine when additional information from your knowledge base would be helpful or necessary to provide a complete and accurate answer.

Remember, your goal is to provide accurate, context-based responses, supplemented by your own knowledge when it adds value to the answer. Always prioritize the provided context, but don't hesitate to enhance it with your broader understanding when appropriate. Clearly differentiate between the two sources of information in your response.

Context:
[The concatenated documents will be inserted here]

Query:
[The user's question will be inserted here]

Please provide your answer based on the above guidelines, the given context, and your own knowledge where appropriate, clearly distinguishing between the two:
'''

Elastic 쿼리 생성기 프롬프트

동의어로 쿼리를 보강하고 OR 형식으로 변환하는 프롬프트입니다.

ELASTIC_SEARCH_QUERY_GENERATOR_PROMPT = '''
You are an AI assistant specialized in generating Elasticsearch query strings. Your task is to create the most effective query string for the given user question. This query string will be used to search for relevant documents in an Elasticsearch index.

Guidelines:
1. Analyze the user's question carefully.
2. Generate ONLY a query string suitable for Elasticsearch's match query.
3. Focus on key terms and concepts from the question.
4. Include synonyms or related terms that might be in relevant documents.
5. Use simple Elasticsearch query string syntax if helpful (e.g., OR, AND).
6. Do not use advanced Elasticsearch features or syntax.
7. Do not include any explanations, comments, or additional text.
8. Provide only the query string, nothing else.

For the question "What is Clickthrough Data?", we would expect a response like:
clickthrough data OR click-through data OR click through rate OR CTR OR user clicks OR ad clicks OR search engine results OR web analytics

AND operator is not allowed. Use only OR.

User Question:
[The user's question will be inserted here]

Generate the Elasticsearch query string:
'''

잠재적 질문 생성기 프롬프트

잠재적인 질문을 생성하고 문서 메타데이터를 보강하는 프롬프트입니다.

RAG_QUESTION_GENERATOR_PROMPT = '''
You are an AI assistant specialized in generating questions for Retrieval-Augmented Generation (RAG) systems. Your task is to analyze a given document and create 10 diverse questions that would effectively test a RAG system's ability to retrieve and synthesize information from this document.

Guidelines:
1. Thoroughly analyze the entire document.
2. Generate exactly 10 questions that cover various aspects and levels of complexity within the document's content.
3. Create questions that specifically target:
   a. Key facts and information
   b. Main concepts and ideas
   c. Relationships between different parts of the content
   d. Potential applications or implications of the information
   e. Comparisons or contrasts within the document
4. Ensure questions require answers of varying lengths and complexity, from simple retrieval to more complex synthesis.
5. Include questions that might require combining information from different parts of the document.
6. Frame questions to test both literal comprehension and inferential understanding.
7. Avoid yes/no questions; focus on open-ended questions that promote comprehensive answers.
8. Consider including questions that might require additional context or knowledge to fully answer, to test the RAG system's ability to combine retrieved information with broader knowledge.
9. Number the questions from 1 to 10.
10. Output only the ten questions, without any additional text, explanations, or answers.

Document:
[The document content will be inserted here]

Generate 10 questions optimized for testing a RAG system based on this document:
'''

HyDE 생성기 프롬프트

HyDE를 사용하여 가상의 문서를 생성하는 프롬프트

HYDE_DOCUMENT_GENERATOR_PROMPT = '''
You are an AI assistant specialized in generating hypothetical documents based on user queries. Your task is to create a detailed, factual document that would likely contain the answer to the user's question. This hypothetical document will be used to enhance the retrieval process in a Retrieval-Augmented Generation (RAG) system.

Guidelines:
1. Carefully analyze the user's query to understand the topic and the type of information being sought.
2. Generate a hypothetical document that:
   a. Is directly relevant to the query
   b. Contains factual information that would answer the query
   c. Includes additional context and related information
   d. Uses a formal, informative tone similar to an encyclopedia or textbook entry
3. Structure the document with clear paragraphs, covering different aspects of the topic.
4. Include specific details, examples, or data points that would be relevant to the query.
5. Aim for a document length of 200-300 words.
6. Do not use citations or references, as this is a hypothetical document.
7. Avoid using phrases like "In this document" or "This text discusses" - write as if it's a real, standalone document.
8. Do not mention or refer to the original query in the generated document.
9. Ensure the content is factual and objective, avoiding opinions or speculative information.
10. Output only the generated document, without any additional explanations or meta-text.

User Question:
[The user's question will be inserted here]

Generate a hypothetical document that would likely contain the answer to this query:
'''

하이브리드 검색 쿼리 샘플

{'knn': {'field': 'primary_embedding',
  'query_vector': [0.4265527129173279,
   -0.1712949573993683,
   -0.042020395398139954,
   ...],
  'k': 100,
  'num_candidates': 100},
 'query': {'bool': {'must': [{'multi_match': {'query': 'audits Elastic Elastic auditing Elastic audit process Elastic compliance Elastic security audit Elasticsearch auditing Elasticsearch compliance Elasticsearch security audit',
      'fields': ['original_text',
       'keyphrases',
       'potential_questions',
       'entities'],
      'type': 'best_fields',
      'operator': 'or'}}],
   'should': [{'script_score': {'query': {'match_all': {}},
      'script': {'source': '\n                                        double vector_score = cosineSimilarity(params.query_vector, params.vector_field) + 1.0;\n                                        double text_score = _score;\n                                        return 0.7 * vector_score + 0.3 * text_score;\n                                        ',
       'params': {'query_vector': [0.4265527129173279,
         -0.1712949573993683,
         -0.042020395398139954,
        ...],
        'vector_field': 'primary_embedding'}}}}]}},
 'size': 10}

관련 콘텐츠

LINQ to Elasticsearch ES|QL: C# 작성, Elasticsearch 쿼리

Florian Bernd

Elasticsearch에서 NLP와 벡터 검색으로 챗봇 기능 강화하기

Priscilla Parodi

Elasticsearch와 OpenSearch: 벡터 검색 성능 비교

Ugo Sangiorgi

AI 표절: Elasticsearch를 통한 표절 탐지

Priscilla Parodi

Elasticsearch의 HNSW를 위한 적응형 조기 종료

Tommaso Teofili

최첨단 검색 환경을 구축할 준비가 되셨나요?

충분히 고급화된 검색은 한 사람의 노력만으로는 달성할 수 없습니다. Elasticsearch는 여러분과 마찬가지로 검색에 대한 열정을 가진 데이터 과학자, ML 운영팀, 엔지니어 등 많은 사람들이 지원합니다. 서로 연결하고 협력하여 원하는 결과를 얻을 수 있는 마법 같은 검색 환경을 구축해 보세요.

직접 사용해 보세요