This blog builds on that notebook and explores fetch surrounding chunking, an emerging pattern in RAG that uses intelligent chunking and Elasticsearch vector database to optimize LLM responses. The approach balances data input to enhance the accuracy and relevance of LLM-generated answers through semantic hybrid search.

The motivation: A refined approach to RAG data chunking

The primary motivation behind building this notebook was to demonstrate a refined approach to RAG by addressing the challenge of data chunking. Traditional methods often fall short in dynamically adjusting the data size fed to LLMs, either overwhelming the model with too much context or starving it with too little. This notebook aims to strike the right balance, providing just enough information for the LLM to generate precise and contextually relevant responses. However, it must be noted that there is no one-size-fits-all solution.

This method works especially well with books and similar texts where content flows within longer sections or chapters. However, it may require adaptation for texts structured into shorter, distinct sections, such as research papers or articles, where each segment might cover a different topic. In such cases, additional strategies may be necessary to effectively chunk and retrieve related content.

The methodology: Intelligent RAG data chunking

Fetch surrounding chunks

The core idea is to partition the source text into manageable chunks, ensuring each chunk contains just the right amount of information. For this demonstration, I used text from "Harry Potter and the Sorcerer's Stone." The text was partitioned into chapters, and each chapter was further divided into smaller chunks. These chunks, along with their dense and sparse (ELSER) vector representations, were indexed in the Elasticsearch vector database.

Assigning numbers to chunks

Each chunk within a chapter was assigned a sequential integer, allowing us to identify its position. When a matching chunk is found, the chapter number and chunk number are used to retrieve surrounding chunks, providing additional context for the LLM.

Vector database in Elasticsearch

These chunks and their vector representations were ingested into an Elasticsearch Cloud instance. Elasticsearch's robust vector search capabilities make it ideal for hosting these chunks, allowing for efficient retrieval of the most relevant chunks based on the semantic content or text match of a user's query.

AI search

To retrieve the relevant chunks, I employed a hybrid search strategy using dense vector comparisons, sparse vector comparisons, and text search in parallel. This multi-faceted approach ensures that the search results are both semantically rich and contextually accurate. A query is issued to find the matched chunk, which returns the chunk number and chapter. Surrounding chunks for that chapter are then fetched based on the matched chunk.

The RAG pattern

When a query is made, the search flow performs the following steps:

1. Query analysis: The user's query is translated into dense and sparse vectors to retrieve the most relevant chunks from the Elasticsearch index.

2. Chunk retrieval: Using the AI search strategy, the system retrieves the top relevant chunks.

3. Contextual expansion: Adjacent chunks (n-1 and n+1) are also retrieved to provide a more comprehensive context. If the chunk is the last in the chapter, it fetches n-1 and n-2; if it's the first, it fetches n+1 and n+2.

4. LLM response: These intelligently selected chunks are then fed into the LLM, ensuring it receives the optimal amount of information to generate a precise and contextually relevant response.

Why intelligent RAG data chunking matters

This approach addresses a critical aspect of RAG by optimizing the input data fed to LLMs. By leveraging intelligent chunking and hybrid semantic search, this method enhances the accuracy and relevance of the responses generated by LLMs. It showcases a pattern that can be widely applied in various applications within the RAG space, from customer support to content generation and beyond.

Conclusion

This notebook underscores the importance of intelligent data chunking in the RAG framework and demonstrates how Elasticsearch vector database can be leveraged to achieve optimal results. By ensuring the LLM receives just the right amount of information, this methodology paves the way for more accurate and contextually rich responses, enhancing the overall effectiveness of RAG systems.

Search has evolved from simple text queries yielding straightforward results to a complex system accommodating various formats like text, images, videos, and questions.

Search not too long ago comprised of a text query and relevant results. Today's search results are enhanced with generative AI, machine learning, and interactive chat features, offering a richer, more dynamic, and contextually relevant user experience. Additionally, voice search and speech avatars have transformed traditional search, offering a more interactive and convenient user experience.

The desire for dialogue in search

In a realm where dialogue underpins every interaction, whether with fellow humans or bots, shouldn't our search experiences reflect this fundamental aspect? Envision the vast array of document corpora residing within an enterprise. Naturally, this environment sparks curiosity and a multitude of questions, leading to subsequent inquiries. This innate human trait drives us to seek answers, delve deeper following initial responses, and continuously explore. Yet, traditional question-and-answer mechanisms fall short, as they often disregard the context of preceding exchanges, leading to a disjointed and laborious process that feels unnatural and prompts users to disengage prematurely.

Beyond question and answer search

Consider the act of using a television to search for content, such as seeking action movies featuring Nicolas Cage. While most current systems adeptly provide relevant results, the inquiry rarely ends there. Subsequent questions, such as inquiring about the runtime or release dates of these movies, are a natural progression in our quest for information. However, standard search applications are not designed to facilitate a continuous dialogue; they are structured around isolated question-and-answer formats, which limits the depth of interaction and exploration.

Avatar assisted voice search experience

This is where the concept of an avatar-assisted search experience comes into play, especially in scenarios where users, myself included, prefer direct answers without the need to sift through information. Occasionally, we desire the convenience of having answers delivered to us, bypassing the effort of reading through content. The development of an avatar to generate responses could further modernize this interaction, providing a more engaging, efficient, and natural user experience.

Live demo: creating an avatar assisted voice search experience

This demo showcases a seamless integration of speech-to-text, Elasticsearch's semantic search capabilities, Azure OpenAI's RAG, and a synthesized avatar for responses.

Integration details

Speech to search

The advanced search experience begins with user voice interactions, which are converted into text by Azure Speech to Text, forming the basis of the search query. This query is then processed through Elasticsearch, using the ELSER, to retrieve relevant documents, such as TV guides listing “action movies featuring Nicolas Cage.” This ensures precision and relevance in the search results.

RAG & cache

In the enhanced search framework, merely fetching documents isn't enough. Azure OpenAI's GPT-4 refines raw data into understandable responses, ensuring smooth conversation flow. Additionally, Elasticsearch boosts efficiency as a GenAI caching layer, recycling answers for related queries, thus conserving resources. For example, if there's a cached response for "action movies featuring Nicolas Cage," the caching API will swiftly use this for similar questions like “Nicolas Cage high-intensity movies,” accelerating the search experience.

Avatar response generation

The experience is further enriched with an avatar response feature, powered by Azure Synthesizer, adding a visual and auditory dimension that surpasses traditional text-based interfaces. This creates a more engaging and interactive user experience, integrating various advanced technologies to deliver a dynamic, intuitive, and compelling search experience.

Summary

The shift from traditional Google searches to platforms like ChatGPT for answering queries illustrates a broader trend: our preference for dialogue over static information retrieval. This predilection underscores the importance for enterprises to adopt a more intuitive and conversational approach in their search functionalities. By embracing this paradigm, businesses can better align with the natural human inclination towards dialogue, thereby enhancing the overall search and discovery process within their data ecosystems.

Demo assets

Still curious, here is the link to the source code.