Elastic Search 8.9: Hybrid search with RRF, faster vector search, and public-facing search endpoints
Elastic Search 8.9 introduces hybrid search with Reciprocal Rank Fusion (RRF) to combine vector, keyword, and semantic techniques for better results. This release also brings performance improvements in vector search and ingestion with response times that are up to 30%+ faster. Users also have more ingestion options with the new SharePoint Online connector, which includes document-level security. In addition, 8.9 brings a new public search endpoint for your Elasticsearch® indices with the search applications beta.
These new features allow users to:
- Improve search results by easily adding multiple retrieval techniques
- Quickly ingest SharePoint Online data with document-level security
- Create public-facing search endpoints
Elastic Search 8.9 is available now on Elastic Cloud — the only hosted Elasticsearch offering to include all of the new features in this latest release. You can also download the Elastic Stack and our cloud orchestration products, Elastic Cloud Enterprise and Elastic Cloud for Kubernetes, for a self-managed experience.
What else is new in Elastic 8.9? Check out the 8.9 announcement post to learn more >>
Hybrid search with Reciprocal Rank Fusion (RRF) combines multiple search techniques for better results
Elasticsearch users increasingly use search retrieval with different types of information — BM25 for text, vector search for dense vectors. Blended search techniques often provide better results: benchmarking over several BIER data sets show improvements in relevance when combining BM25 and ELSER-based ranking, and now users even more easily combine all those retrieval methods. In 8.9, hybrid search with Reciprocal Rank Fusion (RRF) now allows users to combine searches from any combination of:
- Elastic® vector database search
- Keyword-based BM25f search
- Text expansion search with the Elastic Learned Sparse EncodeR (ELSER) model
And it works out of the box without having to normalize or configure weights, so you can quickly and seamlessly incorporate vector or semantic search with the ELSER model into existing search experiences. RRF is available as a technical preview for Platinum+ users. Learn more about RRF in the docs.
Accelerated vector hardware instructions
In 8.9, we made use of an incubating Java API (Panama vector API) that allows using native instructions (SIMD) to significantly improve performance for vector computations. This resulted in 30% (!) performance improvement for both vector searches and vector indexing, for both HNSW (kNN) and brute force search. Note that this is possible on CPUs that support it.
View examples of the performance improvement from our nightly benchmark.
To further improve query latency in brute force vector searches, we changed to the encoding of the float elements in the vector from big endian to little endian. This change resulted in faster decoding, leading to significant query latency improvements of tens of percent in brute force vector searches. These improvements are particularly significant in the slower searches, where it matters most, in which the improvement can get to around 50% of the time. This improvement comes on top of the performance improvements due to the use of the Panama API described above.
Easily ingest with new connectors and features
Users can ingest directly into Elasticsearch data from more data sources. A new SharePoint Online connector is generally available in 8.9 for Platinum+ users with improved performance, document-level security for fine-grained permissions, an incremental sync option to increase data freshness, and local binary content extraction for large files.
New ServiceNow & Dropbox connector clients are now available in beta. These connector clients enable customers to integrate more sources of proprietary data, while the following connectors are available as native on Elastic Cloud as of this version: Azure Blob Storage, Atlassian Confluence Cloud & Server, Atlassian Jira Cloud & Server and Network Drive.
Create publicly accessible search endpoints with search applications
8.9 brings search applications into beta for Platinum+ users with a public search API, so users can now create public-facing search endpoints for their Elasticsearch indices. The _applications/search-application/<app-name>/_search endpoint only allows the queries defined in an associated search template, so you can provide search without a passthrough query system between the client and your Elasticsearch deployment.
The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.