Elastic Search 8.9: Hybrid search with RRF, faster vector search, and public-facing search endpoints

By

Nick Chow,

Dana Juratoni,

Gilad Gal

blog-thumb-release-enterprise-search.png

Elastic Search 8.9 introduces hybrid search with Reciprocal Rank Fusion (RRF) to combine vector, keyword, and semantic techniques for better results. This release also brings performance improvements in vector search and ingestion with response times that are up to 30%+ faster. Users also have more ingestion options with the new SharePoint Online connector, which includes document-level security. In addition, 8.9 brings a new public search endpoint for your Elasticsearch® indices with the search applications beta.

These new features allow users to:

  • Improve search results by easily adding multiple retrieval techniques
  • Quickly ingest SharePoint Online data with document-level security
  • Create public-facing search endpoints

Elastic Search 8.9 is available now on Elastic Cloud — the only hosted Elasticsearch offering to include all of the new features in this latest release. You can also download the Elastic Stack and our cloud orchestration products, Elastic Cloud Enterprise and Elastic Cloud for Kubernetes, for a self-managed experience.

What else is new in Elastic 8.9? Check out the 8.9 announcement post to learn more >>

Hybrid search with Reciprocal Rank Fusion (RRF) combines multiple search techniques for better results

Elasticsearch users increasingly use search retrieval with different types of information — BM25 for text, vector search for dense vectors. Blended search techniques often provide better results: benchmarking over several BIER data sets show improvements in relevance when combining BM25 and ELSER-based ranking, and now users even more easily combine all those retrieval methods. In 8.9, hybrid search with Reciprocal Rank Fusion (RRF) now allows users to combine searches from any combination of:

And it works out of the box without having to normalize or configure weights, so you can quickly and seamlessly incorporate vector or semantic search with the ELSER model into existing search experiences. RRF is available as a technical preview for Platinum+ users. Learn more about RRF in the docs.

Video thumbnail

Accelerated vector hardware instructions

In 8.9, we made use of an incubating Java API (Panama vector API) that allows using native instructions (SIMD) to significantly improve performance for vector computations. This resulted in 30% (!) performance improvement for both vector searches and vector indexing, for both HNSW (kNN) and brute force search. Note that this is possible on CPUs that support it.

nightly dense vector

View examples of the performance improvement from our nightly benchmark.

To further improve query latency in brute force vector searches, we changed to the encoding of the float elements in the vector from big endian to little endian. This change resulted in faster decoding, leading to significant query latency improvements of tens of percent in brute force vector searches. These improvements are particularly significant in the slower searches, where it matters most, in which the improvement can get to around 50% of the time. This improvement comes on top of the performance improvements due to the use of the Panama API described above.

Easily ingest with new connectors and features

Users can ingest directly into Elasticsearch data from more data sources. A new SharePoint Online connector is generally available in 8.9 for Platinum+ users with improved performance, document-level security for fine-grained permissions, an incremental sync option to increase data freshness, and local binary content extraction for large files.

New ServiceNow & Dropbox connector clients are now available in beta. These connector clients enable customers to integrate more sources of proprietary data, while the following connectors are available as native on Elastic Cloud as of this version: Azure Blob Storage, Atlassian Confluence Cloud & Server, Atlassian Jira Cloud & Server and Network Drive.

Check out the complete list of available native connectors and connector clients.

Create publicly accessible search endpoints with search applications

8.9 brings search applications into beta for Platinum+ users with a public search API, so users can now create public-facing search endpoints for their Elasticsearch indices. The _applications/search-application/<app-name>/_search endpoint only allows the queries defined in an associated search template, so you can provide search without a passthrough query system between the client and your Elasticsearch deployment.

Search applications can then be used to create publicly accessible search when paired with a public search endpoint-only API key using the new API key "search_application_query" workflow role restriction, which provides read-only access to the search application endpoint. And development is even easier with a new Search Application Javascript client!

Try it out

Read about these capabilities and more in the release notes.

Existing Elastic Cloud customers can access many of these features directly from the Elastic Cloud console. Not taking advantage of Elastic on cloud? Start a free trial.

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all. 

Sign up for Elastic Cloud free trial

Spin up a fully loaded deployment on the cloud provider you choose. As the company behind Elasticsearch, we bring our features and support to your Elastic clusters in the cloud.

Start free trial