Elasticsearch 8.11 improves how it manages its indexing buffer, resulting in less segment merging.

How it works in Elasticsearch 8.10 and earlier

Change 1: flush a single segment at a time

Change 2: flush shards in a round-robin fashion

Conclusion

Less merging and faster ingestion in Elasticsearch 8.11

There are different ways to implement a vector database, which have different trade-offs. In this blog, you'll learn more about how vector search has been integrated into Elastisearch and the trade-offs that we made.

Vector search is integrated in Elasticsearch through Apache Lucene

Cons

Merges need to recompute HNSW graphs

Searches need to combine results from multiple segments

RAM needs to scale with the size of the data set to retain optimal performance

Pros

Data sets can scale beyond the total RAM size

Lock-free search

Support for incremental changes

Visibility consistency with other data structures

Incremental snapshots

Filtering and hybrid support

Compatibility with other features

Looking ahead: Separation of indexing and search

Vector search in Elasticsearch: The rationale behind the design

Author

Adrien Grand

Articles

Less merging and faster ingestion in Elasticsearch 8.11

Vector search in Elasticsearch: The rationale behind the design