Optimizing scalar quantization in Lucene and adding int4 support.

Bringing Int4 Quantization in Lucene

Storing and scoring the quantized vectors

Calculating the error correction

Finding the optimal bucketing

Speed vs. Size for quantization

Speed part 2: more SIMD

The end?

Int4: More Scalar Quantization in Lucene

Optimizing scalar quantization for the vector database use case allows us to achieve significantly better performance for the same retrieval quality at high compression ratios.

Introduction

Background

Error correcting the scalar dot product

Optimizing the truncation interval

Proof of principle for int4 quantization

Conclusion

Vector Database Optimized Scalar Quantization

Scalar Quantization Optimized for Vector Databases

N-gram metrics

Intrinsic metrics

Model-based metrics

Elastic’s choice: UniEval

Real-world usage

Evaluating RAG: A journey through metrics

How did we introduce scalar quantization into Lucene

Automatic Byte Quantization in Lucene

Scalar Quantization 101

Nodes, Shards, Segments, oh my!

Quantization per segment

Quantization that grows with you

Performance & Numbers

Introducing Scalar Quantization in Lucene

What is scalar quantization and how does it work?

Buckets of fun

It's just statistics

Time to remember your Algebra

But, how is this accurate?

Scalar quantization 101

Learn about the improvements we've made to the inference performance of ELSER v2.

Improved Relevance

Quantization

Block Layout of Linear Layers

Improving information retrieval in the Elastic Stack: Improved inference performance with ELSER v2

Learn about how we're reducing retrieval costs for ELSER v2.

Retrieval Cost Aware Training

Optimizing ELSER Queries

Implementing with an Elasticsearch Query

Improving information retrieval in the Elastic Stack: Optimizing retrieval with ELSER v2

Here's how generative AI works from the ground up, including embeddings, transformer-encoder architecture, training/fine-tuning models & more.

The semantic stage of GenAI: Understanding natural language

1. Understanding embeddings, vector similarity and language models

1.1 Learning dense vectors

2. Context: The bloodstream of GenAI

3. The transformer-encoder architecture

3.1 Self-attention

3.2 Multi-headed self-attention

3.3 The feed-forward neural network

4. Training and fine tuning language models for AI search and NLP

4.1 Training patterns and language model transfer learning

4.2 The AI barrier

5. Elastic Learned Sparse Encoder for AI search out of the box

References

BERT is the most prominent encoder architecture. It was introduced in 2018 and revolutionized NLP by outperforming most benchmarks for natural language understanding and search. Encoders like BERT are the basis for modern AI: translation, AI search, GenAI and other NLP applications.

What is BERT (Bidirectional Encoder Representation from Transformers)?

Vectors are the fundamental construct with which AI understands language. Vectors represent words and are simply long arrays of numerical values. Technically, vectors may also represent sequences of words or sub-word parts.

What are vectors in generative AI?

Understanding generative AI architectures with transformers

Generative AI architectures with transformers explained from the ground up

There are different ways to implement a vector database, which have different trade-offs. In this blog, you'll learn more about how vector search has been integrated into Elastisearch and the trade-offs that we made.

Vector search is integrated in Elasticsearch through Apache Lucene

Cons

Merges need to recompute HNSW graphs

Searches need to combine results from multiple segments

RAM needs to scale with the size of the data set to retain optimal performance

Pros

Data sets can scale beyond the total RAM size

Lock-free search

Support for incremental changes

Visibility consistency with other data structures

Incremental snapshots

Filtering and hybrid support

Compatibility with other features

Looking ahead: Separation of indexing and search

Vector search in Elasticsearch: The rationale behind the design

Learn how scalar quantization can be used to reduce the memory footprint of vector embeddings in Elasticsearch.

ML Research

Evaluating scalar quantization in Elasticsearch