Elastic Search 8.13: Simplifying embedding and ranking for developers

Search-release-header-image.jpg

Elastic Search 8.13 extends the capabilities that enable developers to use artificial intelligence and machine learning models to create fast and elevated search experiences. Integrated with Apache Lucene 9.10, measured vector search performance has exceeded 2x in benchmarks, extending the sophistication of searches that can be performed in near real time. We have added native Learning to Rank (LTR) capabilities that simplify the reranking for Top-N results, which is critical for retrieval augmented generation (RAG) use cases — where the large language model (LLM) needs to operate on the best possible context. 

Creating elevated search experiences is also about the corpus of data you need to search. Connectors simplify bringing data from multiple sources to Elasticsearch, and we have extended the catalog to include Redis (Preview) and Notion (Beta) to the 20+ native connectors that encompass database (PostgreSQL, MongoDB, etc.), workplace (SharePoint, Dropbox, Gmail, etc.), connectivity (Slack, Teams, Zoom, etc.), and productivity tools (ServiceNow, Jira, etc.). Synchronization of these sources is now simple to set up with the ability to bring the data into an existing index.

Elastic Search 8.13 is available now on Elastic Cloud — the only hosted Elasticsearch offering to include all of the new features in this latest release. You can also download the Elastic Stack and our cloud orchestration products, Elastic Cloud Enterprise and Elastic Cloud for Kubernetes, for a self-managed experience.

What else is new? Check out the Elastic 8.13 announcement post to learn more.

Continuing to build upon Lucene 9.10

We are continuing our commitment to making Lucene the best vector database. In this release, we see significant performance improvements. For example, in benchmarking the Cohere data set (1m vectors of 768 float dimensions), we saw a 2x performance improvement in the nightly benchmarks. You can read more about this innovation that contributed to Lucene in the Elasticsearch Platform highlights blog.

Learning to Rank is available as tech preview

LTR is now available natively in Elasticsearch as a rescorer. It creates better search results by reranking result sets using a machine learning model trained with your input. Reranking is an important task for semantic search with or without vector search. It can account for signals beyond the query to surface the most relevant results at the top. This is particularly useful for RAG use cases, which rely on feeding only the top few results to an LLM as context.

8.13 brings LTR as part of the core Elastic Search product. We are pleased to have this in technical preview for you to try out. Try native LTR now with this example notebook. We look forward to your feedback!

Better, faster, and simpler relevance

Application code is complex enough for developers to build and maintain. The added complexity of generating embeddings and orchestrating more systems is an additional and unnecessary burden. In Elastic Search 8.13, we have simplified the integration of Cohere for embeddings, so the developer gets the advantage of great embedding models without the complexity of orchestrating many components and systems. We have refined the query performance of ELSER, allowing performance targets to be reached with fewer compute resources.

Developers often face the challenge of not controlling the source data. When it comes to large documents, they often need to be broken up (or chunked) into multiple embeddings. The same is true when you have multiple images or other artifacts associated with a single document. Multiple results are now possible with multiple inner_hits on nested kNN queries where these embeddings are stored as nested documents.

Many use cases require active management of synonyms to ensure the relevance of results. The use of product names, SKUs, and other linguistic pitfalls in data encouraged us to create an API to programmatically manage synonyms. This API is now promoted to general availability and complements the ability to manage synonyms through the UI. 

Elasticsearch Python client now supports orjson. orjson is a fast, correct JSON library for Python. In benchmarks, it exceeds the performance of standard built-in json modules. Speed improvements in the serialization/deserialization improve vector use cases where there is bulk indexing of vectors.

Use a connector with an existing index

Since the lifeblood of search is the corpus of data you operate upon, we continue to enhance the existing catalog of connectors. In this release, we have added Redis (Preview) and Notion (Beta) to the catalog. Google Drive, Gmail, OneDrive, Outlook, and Salesforce have now been promoted to general availability.

It's not just a question of what you connect to, it's also about how and what data is retrieved. We have simplified the connector flow and improved the synchronization capabilities to bring the data into Elasticsearch and have added document-level security for ServiceNow and Salesforce connectors.

More significantly, you can bring that data into an existing index. It is no longer a requirement that the connector has to use one predefined index. This gives the developer greater flexibility on how and where the data should reside in Elasticsearch, allowing them to use a pre-existing index or an index they create for a specific project or use-case, or switch to a new index at any time. 

Try it out

Read about these capabilities and more in the release notes. You can also find code references, notebooks, and the latest research in Search Labs.  

Existing Elastic Cloud customers can access many of these features directly from the Elastic Cloud console. Not taking advantage of Elastic on cloud? Start a free trial.

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

In this blog post, we may have used or referred to third party generative AI tools, which are owned and operated by their respective owners. Elastic does not have any control over the third party tools and we have no responsibility or liability for their content, operation or use, nor for any loss or damage that may arise from your use of such tools. Please exercise caution when using AI tools with personal, sensitive or confidential information. Any data you submit may be used for AI training or other purposes. There is no guarantee that information you provide will be kept secure or confidential. You should familiarize yourself with the privacy practices and terms of use of any generative AI tools prior to use. 

Elastic, Elasticsearch, ESRE, Elasticsearch Relevance Engine and associated marks are trademarks, logos or registered trademarks of Elasticsearch N.V. in the United States and other countries. All other company and product names are trademarks, logos or registered trademarks of their respective owners.