Discover how we added passage vectors to Apache Lucene, the benefits of doing so, and how existing Lucene structures were used to create an efficient retrieval experience.

The (kNN) problem with parents and children

Pre-joining to the rescue

Conclusion

Adding passage vector search to Lucene

How we brought maximum-inner-product into Lucene

Negative Values and Lucene Optimizations

The Triangle Problem

Bringing Maximum-Inner-Product into Lucene

What is scalar quantization and how does it work?

Buckets of fun

Time to remember your Algebra

But, how is this accurate?

Scalar quantization 101

How did we introduce scalar quantization into Lucene

Automatic Byte Quantization in Lucene

Scalar Quantization 101

Nodes, Shards, Segments, oh my!

Quantization per segment

Quantization that grows with you

Performance & Numbers

Introducing Scalar Quantization in Lucene

Author

Benjamin Trent

Articles

Adding passage vector search to Lucene

Bringing Maximum-Inner-Product into Lucene

Scalar quantization 101

Introducing Scalar Quantization in Lucene