In our previous blog, we laid out the statistical model behind that calibration: a manifold model for how nearest-neighbor distances scale with index size, a Gaussian error model for quantization noise, and a closed-form way to combine the two into an expected recall@k for a given rerank depth. If you haven't read it, the one thing you need going in is this: given a candidate quantization encoding and a rerank depth, we can predict recall@k without building an index and benchmarking it, by fitting two small models to a sample of the corpus.

In this post we’ll go through how to score candidate configurations cheaply and how to leverage that to make merge-time decisions that are themselves cheap, correct, and consistent across real, constantly-merging indexes. This led to some pretty impressive improvements: we see almost 17% average improvement in QPS across a broad range of datasets all while increasing recall (in one case by a factor of 3). What’s more you get this immediately by adding one line to your index options, "auto_calibrate": true, and our plan is to make this our default once it has had the chance to bake a bit.

Why manual vector quantization tuning is unreliable

bbq_disk exposes several knobs: quantization bits for documents (1, 2, 4 or 7), a separate bit width for queries, an oversampling factor for reranking, and whether to precondition vectors before quantizing. None of these act independently, and their effect on recall depends on the data: a 4-bit/1-bit encoding might be plenty for one embedding model and clearly insufficient for another. A single index is also built out of many segments, merged over time, each with an eventually different vector distribution. Hand-tuning one configuration for an entire index is, at best, a compromise, which is the motivation for automatic calibration: let each segment have its own configuration, re-evaluated every time it is involved in a merge operation.

How Elasticsearch runs auto calibration at merge time

When a number of segments are merged and automatic calibration is enabled, Elasticsearch samples documents and queries from the vectors being merged and:

1. fits the manifold model over a sequence of nested samples of the merged corpus;

2. fits the error model, predicting the quantization error's standard deviation for each candidate (query bits, document bits, precondition) combination;

3. sweeps candidate configurations in ascending cost order; the candidate encodings are (1,1), (4,1), (4,2), (4,4) and (7,7) (query bits, document bits), each tried across oversampling factors of 1.25, 1.5, 1.75, 2.0, 2.5 and 3.0;

4. estimates recall@10 for each candidate using the model described in our first post, and stops at the first (cheapest) configuration predicted to hit the target of 90% recall@10.

The winning configuration (encoding, oversample factor, precondition flag) is stored directly in the segment's metadata, so it travels with the segment and is picked up automatically at query time unless a request explicitly overrides it.

Small segments skip this altogether: below 10,000 merged vectors, there isn't enough data to fit a reliable model, so Elasticsearch just uses the current DiskBBQ defaults (4-bit query / 1-bit document encoding, no preconditioning, 3x oversampling).

How the vector quantization cost model works

Following the principles described in our first post, we started by picking candidates with three nested loops that are essentially how you might imagine hand jamming a lookup table. Start with the quantization scheme as the outer loop, ordered cheapest to most expensive by document bits ((1,1) → (4,1) → (4,2) → (4,4) → (7,7)). Then we set the rerank depth within the middle loop, ordered shallow to deep (1.25× → 3.0×). Finally we set preconditioning within the inner loop (off → on).

That ordering has a cost model baked into it, it's just implicit rather than written down: exhaust every rerank depth at the current bit tier before ever trying more bits. Document bits were effectively the only resource priced as expensive; oversampling was treated as nearly free by comparison, since the sweep would always max out rerank depth on a cheap encoding before considering a pricier one.

The current implementation replaces that with an explicit, continuous cost function:

Query bits still don't factor into cost at all, only document bits (which drive index size) and rerank depth (which drives how many candidates get rescored per query). Preconditioning also stays outside the formula: Elasticsearch runs the whole cost-ordered sweep once with preconditioning off, and only if nothing meets the recall target does it re-run the sweep with preconditioning on, treating it as a fallback lever rather than something priced bit-for-bit against the other two.

With this cost model, rerank depth costs noticeably more per unit than a document bit, so the sweep will often prefer stepping up a bit tier over pushing oversampling deeper.

The main reason for this is that once you're running in a serverless deployment, compute and storage are billed and scaled independently, on very different clocks. An extra document bit is mostly a one-time, indexing-time cost; it makes the segment marginally bigger on object storage, which is cheap and doesn't need to be pre-provisioned against a spike in query traffic. It does carry a smaller recurring cost too, since quantized vectors sitting in page cache or loaded for scoring take proportionally more RAM per document as bit width grows, but that scales linearly and predictably with corpus size, and doesn't spike with query load. Rerank depth is the opposite: it's a recurring, per-query cost.

Every extra unit of oversample factor means fetching and rescoring that many more full-precision candidate vectors from disk, on every search request, for as long as the index is queried. That's compute and DRAM pressure on the search-serving tier, which has to autoscale in close to real time to match query concurrency. It sits on the hot path of the latency-and-cost budget in a way storage capacity, and the RAM footprint of the bits themselves, does not. Weighting rerank depth higher than document bits in the cost formula is what makes the sweep reflect that asymmetry.

Efficiently estimating vector quantization error

The cost model above works with the premise that the recall estimate behind it is trustworthy. The manifold and error models need to be accurate for the recall assessment to be trustworthy. While the manifold model of the k-th to N-th nearest neighbors distance is cheap to compute, the standard deviation of the quantization noise for a given candidate encoding is a bit more expensive in principle.

DiskBBQ uses fixed count clusters to accelerate nearest neighbor queries. Our quantization procedure takes advantage of this by only quantizing the vector residuals from the cluster centroids. This means as the data scales, the magnitude of vectors we quantize relative to the various components of the similarity calculation shrinks. As such, quantization accuracy increases. We need to account for this when converting our sample estimates to the segment as a whole.

Clustering the corpus at several sample sizes and fitting how the error scales with cluster size requires re-clustering a real sample of the corpus at several different sizes and fitting a regression model to see how the error shrinks as the effective cluster size grows. We also add a conservative +3σ margin on top of the fitted estimate to guard against noise in the fit itself. This is accurate and appropriately cautious; however, while benchmarking on common dense retrieval datasets, we found that performing several hierarchical k-means passes per candidate was expensive.

To speed things up, we tried approximating residuals with a synthetic isotropic-Gaussian formula. Instead of clustering increasing-size samples, this approach generated synthetic residuals from the manifold model's local density estimate. It was fast and fit for background merges, with the full repeated clustering approach reserved for force-merges only. However, it turned out to inflate error when embeddings (residuals) are anisotropic (some directions carry a lot more variance than others). As a result, the estimated error could grow significantly on strongly anisotropic data (e.g., Fashion-MNIST-style image embeddings).

So instead we looked for a still fast but more accurate way of calculating residuals. We opted for using a single clustering pass over a smaller sample (2,048 vectors). The clustering runs once per merge and is then warm-started for every candidate encoding evaluated afterward, instead of re-clustering from scratch each time. To get the error's dependence on corpus size, which the baseline learns by re-clustering at multiple sizes, this approach instead reuses the manifold model's invDim as a plug-in for that dependence, extrapolating from the single real measurement rather than fitting the size relationship separately.

We also trimmed the query sample used during calibration from 1,024 to 256 vectors, on the reasoning that a smaller sample is enough once the error is being measured from real data rather than synthesized (and validated by benchmarks). The net effect was comparable wall-clock cost to the synthetic residual formula it replaced, but grounded in real per-cluster residuals, accurate enough that force-merge and background merge could be unified onto one path.

As an example, we take five different benchmark datasets and calculate the quantization error standard deviation (SD) by directly measuring the gap between exact and quantized dot products on a sample of real (or, for the synthetic residual formula, fabricated) residuals, then extrapolating that measurement to the full corpus size. They differ only in how much sampling and regression goes into that extrapolation: the multi-sample scaling fit sweeps fifteen sample sizes and fits how error scales with cluster size, the single-pass real residual measurement takes one larger real residual sample and reuses the manifold's intrinsic dimension to estimate the size dependency, and the synthetic residual formula skips real residuals altogether and samples from a synthetic Gaussian from the manifold's expected rank distance. We treat the multi-sample scaling fit as ground truth in this comparison because it's the most sample rich of the three, not because it's a zero variance measurement of the "true" corpus-wide error (it has its own sampling noise too). The table below summarises the methods and findings.

Method

How it works

Speed

Accuracy

When used

Multi-sample scaling fit

Clusters at 15 sample sizes, fits regression

Slow

Gold standard

Ground truth baseline

Single-pass real residual

One clustering pass + manifold invDim plugin

Fast

Near gold standard

Background + force merge

Synthetic residual formula

Gaussian from manifold density estimate

Fast

Inflated on anisotropic data

Deprecated

In order to exchange methods, we only need to be confident that they agree. This question can be answered independently of the correctness of the actual estimates, which we verified in our previous post for the multi-sample scaling fit. The figures below report the predicted quantization SD and the predicted recall@10, which is influenced by how we estimate the error. We report the analytical recall the manifold model predicts as a function of the quantization parameters, given the estimated error distribution perturbing the true distance ordering. This way, we isolate the quantization error's effect on ranking from any separate recall loss the IVF index itself might introduce, which is a distinct error.

The single-pass real residual measurement's calculated error SD is closer to the multi-sample scaling fit (our gold standard), with respect to the synthetic Gaussian residuals. Consequently, the predicted recall is closer when using the single-pass + manifold plugin method. Indeed, we found the models to be essentially interchangeable regarding the indexing decisions they lead to. Critically, we lower the calibration overhead by an order of magnitude.

Auto calibration overhead on indexing performance

We compared the cost of auto calibration on indexing, when compared with ES defaults, over 18 public benchmarks. We noticed that more than 50% of the datasets report an auto calibration overhead below 2%. Three datasets report 16-27% overhead, while two datasets sit in the 31-35% overhead.

The merge overhead is larger for smaller datasets (Fashion-MNIST, FiQA) that get indexed in a few seconds; that is expected as the size of the vector samples being used for calibration is fixed and therefore more noticeable with tiny datasets. In fact, for larger datasets like DBPedia-Entity and HotpotQA (5M doc vectors) the overhead is sometimes not noticeable and within 11% in the worst case.

What quantization parameters does auto calibration choose?

Looking at the encoding auto-calibration landed on for each of the real datasets:

Query bits were 4 in every dataset. While query bits aren't priced into the cost formula, we still iterate through lower query bits first (e.g., at 1 bit doc vectors, we first evaluate recall for 1 bit query vectors, then for 4 bit query vectors); so it’s possible for some datasets to even choose symmetric 1-bit quantization. The center of mass is a 2-bit document encoding with somewhere between 1.5x and 1.75x oversampling; 4-bit only shows up for two genuinely harder datasets (Fashion-MNIST's image embeddings, GIST-1M), and 1-bit only for a handful of the text-embedding models that are most robust to quantization. In fact, our own models are among those that quantize best: we selected 1 bit documents for all three corpuses we tested with Jina v3.

Recall and QPS improvements from automatic calibration

Auto-calibration is a broad win across the eighteen datasets: QPS improves in 15 of 18 cases (often substantially, double digits on about ten, and over +50% on FiQA GTE, Fashion MNIST, and Glove-200), and recall improves in 15 of 18 cases too, including a dramatic +295.7% rescue on Fashion MNIST. Most datasets see gains on both metrics simultaneously, and even the more modest cases still land solidly positive, recall improvements are commonly in the high single digits to double digits, QPS gains follow a similar pattern. Where either metric does dip, the drops are small and contained: the three QPS regressions all stay under 1.5%, and the three recall regressions all stay under 2%.

How to enable auto-calibrated vector quantization in Elasticsearch

The feature is not enabled by default for now, and opt-in via auto_calibrate on bbq_disk index options:

With this set, you no longer need to guess at bits, oversampling, or preconditioning: each segment picks the cheapest configuration that's predicted to hit 90% recall@10 for its own vector distribution, and re-evaluates that choice every time it's merged.

What's next for automatic vector quantization in Elasticsearch

Our first post showed that recall could be predicted in closed form from a small sample. Turning that into something running inside a real merge path meant a second round of engineering decisions that the model itself doesn't answer: how to order a sweep over candidates so it's cheap in the common case, how to price oversampling against document bits given how each is actually paid for at query time, and how to estimate the error term itself cheaply without quietly wrecking its accuracy.

In the end, we have a feature that allows us to tailor indexing choices to the data characteristics, with less than 11% overhead to index time for large indices. This gives us the ability to accurately control recall while optimizing quantization and oversampling choices for query performance. We got an average increase of 16.7% in QPS when we enabled this feature compared to our previous default settings for DiskBBQ. All while reliably achieving our target recall. Taking away the configuration burden from the user actually allows us to make better choices; it is a win-win.

This is the beginning of a longer journey that we’re working on to bring automatic configuration based on a combination of better understanding of the operating environment and better understanding of the data characteristics. We look forward to sharing more of this work with you in the near future.

A vector store that achieves good performance without tuning is more valuable than one that requires expert tuning. In fact, our contention is a data store that can be coaxed to exceptional performance by an expert who spends a week hand-tuning it is less useful than one that beats it consistently out of the box. In other words, easily achieving good performance is a first class property, not a nice to have. We can see this clearly in our telemetry. The great majority of users will never tune the internals of vector search at all, and why should they: it is just an enabler for what they're trying to build.

This is the imperative behind features like auto-calibration. The system as a whole should look at your data and your quality target and choose good parameters for you. Indeed we think this is a win-win, since it has far more nuanced information available to it to make these choices than we expose.

To make "good performance" precise, it helps to name the three attributes that characterize any vector search system, because they trade off against one another and you can't talk about one without fixing the others:

1. Performance: throughput (QPS), latency, and so on.

2. Hardware cost: a fair comparison always holds cost fixed. It's trivial to buy your way to more QPS or better recall by throwing hardware at the problem; the interesting question is what you achieve per dollar.

3. Search quality: recall, nDCG, and related measures of whether you're returning the right results.

The three form a frontier. Push one and, at fixed budget, you pay in another. Any honest comparison of approaches pins two down and measures the third. What we describe in this post is the mechanism we're introducing to pick quantization parameters for a fixed recall budget. It is a step on a longer journey towards a vector store that configures itself well across the board.

Why recall is the right quality metric for vector search

Search quality is tricky, because the "right" results depend on relevance labels you usually don't have at index time. So we lean on recall as a safe proxy. The argument is simple: recall measures how well the approximate index reproduces the results of exact search over the same embeddings. If recall is high, you have not degraded search quality relative to what the underlying model can do; you can be confident you’ve faithfully preserved the baseline. You might still wish for a better embedding model, we've got you covered, but that's a separate concern from the index not damaging what the model already gives you.

This is why controlling recall matters so much, and why you should be wary of any system that doesn't reliably control it. If a vendor can't control recall, they can silently degrade your search experience, achieving impressive QPS numbers while quietly returning worse results, and you'd have no way to know without a labeled evaluation set. The method in this post is about maximizing performance while keeping a firm, predictable grip on quality.

Why vector quantization parameters must be chosen at index time

What makes the problem genuinely hard is that vectors are quantized as they are indexed, so the parameters that govern quality (how many bits, how deep to rerank, whether to precondition) have to be evaluated before we've seen the data laid out in its final form. We can't index everything, measure recall, and iterate; by then the quantization is baked in.

So we need to estimate what we'll need from a small sample, cheaply and in advance. Fortunately the Elasticsearch gives us natural moments to do this: segment merges are exactly such an opportunity. When segments are combined we have to rewrite the data anyway and can assess the data and (re)choose parameters. And as we'll see, models fit to small random samples give excellent estimates of the quantities we actually need to control. They’re typically good enough to set parameters once, with a small margin, and trust them as the index grows.

How vector quantization affects nearest-neighbor recall

With that motivation in place, let's start to dig into the details.

Vector quantization is a critical component for making approximate nearest-neighbor (ANN) search affordable at scale; it's an area we've innovated in the past. Instead of storing and comparing full-precision embeddings, we store a lossy, compressed representation and search over that. The catch is the one above: lossy representations move distances around, so the "nearest" neighbors under quantized distances are not always the true nearest neighbors and recall suffers.

The standard fix is to over-retrieve and rerank. We use the cheap quantized distances to pull back the top candidates, then recompute exact distances for those and keep the best . As long as the true top- are present somewhere in the retrieved top-, reranking recovers them exactly.

Reranking isn’t free, we have to fetch high precision vectors from disk. However, we can precisely characterize the performance of reranking based on hardware characteristics alone. This reframes the whole problem. The question is no longer "how much does quantization distort distances?" in the abstract, but something which relates back to the attributes we care about:

This post derives a model that answers exactly that. The core of it is a single, surprisingly clean idea: if we can characterize the distribution of distances to the -th nearest neighbor, and we have a model of the quantization error distribution, then we can compute expected recall after reranking in closed form (up to a one-dimensional integral). Everything else – bit counts, rerank depth, whether to precondition – becomes a search over a model we can fit cheaply from a small sample, instead of an expensive empirical sweep over full indices built with those parameters.

We build it up to this in three stages: the geometry of nearest-neighbor distances, the scaling law that falls out of it, and then the recall model that ties quantization error to recall given a reranking budget. Be warned, the following gets a little bit involved, but to give you intuition about what is happening see the video below.

Quantization error vs. the nearest-neighbor distance gap

Fix a query and rank the database vectors by their true distance to it: , so is the distance to the -th nearest neighbor. Reranking the top succeeds for the true -th neighbor whenever it is not pushed past rank by quantization noise.

Two competing quantities govern this:

• The quantization error that is essentially fixed for a given scheme and dataset: it depends on the embedding dimension, the vector distribution, and the number of bits, but not on how big the index is.

• The criticality gap , which is the distance between the -th and the -th nearest neighbor. This is the margin we have to absorb error. Crucially, it shrinks as the index grows: pack more vectors into the same region and neighbors crowd together.

There’s a detail here we’ll gloss over for the sake of presentation: for IVF style indices, we’re quantizing the residual from a cluster’s centroid. This does in fact couple the quantization error to the index size, but we can handle it much the same way we handle the distance to the -th nearest neighbor.

For reranking to recover the recall lost to quantization, we need the error to only rarely exceed the gap. If we can write down the distribution of and the distribution of the error, we can make that statement quantitative. The first job is to estimate the distribution of nearest-neighbor distances.

Deriving the nearest-neighbor distance distribution

Real embeddings don't fill their ambient space; they concentrate on a lower-dimensional manifold. Near a query, though, we can make a mild local assumption: in a small neighborhood around the query, the data density is roughly uniform. Here is the intrinsic dimension of the manifold; it is unknown and generally far smaller than the embedding dimension. How to estimate it is the subject of Section 4.

Let be the vectors falling in , modeled as i.i.d. uniform on , and define the distance from to its nearest neighbor:

To get the distribution of we use the standard order-statistics trick: rather than ask where the minimum is, ask for the probability it exceeds some radius . The event is exactly the event that every point lands outside the -ball centered on the query .

A single point lands inside with probability equal to the ratio of the ball's volume to the region's volume

where is the volume of the unit -ball. (We assume is large enough that the relevant is small, so the ball doesn't spill outside and boundary effects are negligible.) Because the points positions are assumed to be independent, the survival function is

What we're really interested in is how R behaves on average. To compute this, we use the identity that the expectation of a non-negative random variable is the integral of its survival function, . Evaluating this with (2) gives the headline result:

(The exact integral carries an extra factor; it's an constant that we can fold into a fitted coefficient later, so we drop it here.)

Glacial scaling: why neighbor distances barely change as your index grows

It is interesting to consider what this formula tells us about how distances change with dataset size: . The exponent is , and in high intrinsic dimensions that is a very small number. This is a property the method leans on, so it's worth plugging in some numbers:

• If then doubling multiplies by , so distances drop by ~30%.

• If then doubling multiplies by , so distances drop by a little over 1%.

In high dimensions, neighbor distances barely move even if you add a lot of data; call it glacial scaling. It's the reason we can choose quantization parameters once from a tiny sample, with a small safety margin, and trust them to remain valid even after the index grows substantially before the next re-quantization.

Expected distance to the k-th neighbor and the criticality gap

We actually care about the whole sequence of order statistics , , not just the minimum. There's a simple way to get them.

Map each radius to the cumulative volume it encloses by defining

By (1), each is exactly the probability of landing within radius , so the are uniform on . The order statistics of uniforms are textbook: the -th smallest of uniforms follows a Beta distribution,

Inverting the volume map, , gives the scaling of the -th neighbor distance:

That's all we need for the expected gap:

The last form is the intuitive one: the gap between the -th and -th neighbors is the distance to the -th neighbor, scaled by . Widening the rerank depth relative to opens the gap; higher intrinsic dimension closes it (the exponent pushes toward 1).

Why the expected gap is sufficient to predict recall

Working with an expectation is only legitimate if the gap doesn't fluctuate wildly around it. It doesn't because concentration of measure saves us. Applying the delta method to and using from the Beta distribution, a little algebra gives

So the coefficient of variation is about . For any reasonable intrinsic dimension this is negligible, which justifies modeling only the expected distances. (If you're worried about the delta method approximation, you can check the results numerically: the delta-method variance and the resulting coefficient of variation match the exact expressions to several significant figures.)

Extending the model to cosine similarity and inner product search

The derivation is for the Euclidean metric, but the other common metrics reduce to it:

• For cosine similarity, the equidistant surface is the intersection of a sphere around the query with the unit sphere. This is called a hyperspherical cap, whose volume scales as for small . Therefore, the analysis carries over unchanged up to constants, with the dimension reduced by one.

• For MIPS (maximum inner product), some extra care is needed, because nearest neighbors aren't confined to a compact region. A distant vector can still win on inner product if its norm is large enough, so the gap is really governed by the tail of the norm distribution. However, there is a clean fix, which is to use the Neyshabur–Srebro transformation. This lifts vectors onto a unit hypersphere in dimensions. After this operation, it's just the cosine case.

Fitting intrinsic dimension and scale from a small sample

Equation (3) has a known functional form but two unknown parameters: the intrinsic dimension and the scale . Both are easy to fit, and it's more convenient to fit them from raw neighbor distances than from gaps directly.

Sample several subsets of database vectors of sizes and a set of query vectors . For each query and each subset, measure , the distance to the -th nearest neighbor of within . Taking logs of the scaling law linearises it:

Specifically, this is linear in and , so ordinary least squares recovers and . Varying the subset size is what makes it possible to estimate : it's precisely the rate at which distances shrink with data volume. With the fitted parameters, the whole-index expected gap is

Figure 1 shows how well this fits in practice (and it’s remarkably good): predicted versus actual average distance to the -th neighbor, across a range of datasets and metrics, have between 0.996 and 0.999.

Modeling vector quantization error as Gaussian

With the nearest-neighbor distance model established, the second component is the quantization error distribution. For every metric we use, the quantized distance estimate differs from the true distance by an error that is a sum of many independent per-dimension contributions. By the Central Limit Theorem that sum tends to Gaussian, so we model the error as normal with a variance we estimate empirically:

where is the quantized distance estimate using -bit vectors and is the total number of (query, neighbor) pairs in our sample set. In other words: sample, quantize, measure the squared distance errors, average.

Figure 2 shows the empirical basis for the Gaussian assumption: measured quantization error densities against best-fit Gaussians across a variety of datasets. The fit is good, which is what lets the rest of the model stay analytic.

We could stop here and take a minimax view: threshold the probability that the -th and -th neighbors swap, using the expected gap (4) against the error scale . But that controls a worst-case event, and what we actually want to control is average recall. The outcome would be overly conservative quantization parameters and we'd pay some performance. The next section estimates expected recall properly.

Predicting expected recall after reranking

Combining the distance model and the error model gives a closed-form estimate of expected recall after reranking. Model the noisy distance of the -th true neighbor as a Gaussian centered on its true distance:

The -th neighbor survives reranking, i.e., lands in the retrieved top , if fewer than other vectors have a smaller noisy distance. Condition on and count the competitors closer than :

Then the probability of recalling neighbor integrates over where its own noisy distance lands:

The terms of are independent Bernoullis but not identically distributed, since every neighbor sits at a different true distance , so each has its own probability of intruding on the top- set:

with the standard normal CDF. This makes a Poisson-binomial variable. Since we sum many of them (because ), the Lyapunov CLT applies and we approximate

with the standard Poisson-binomial moments

The survival probability then has a clean closed form:

This is where the two halves of the post so far finally meet. We don't need to know the individual because the manifold scaling law from Section 3 supplies them: . So the moments become explicit sums over ranks, which we truncate at a safe cutoff (say , since distant neighbors contribute negligibly):

Finally, average recall@ given rerank depth sums the per-neighbor recall over the top :

Here is the standard normal density. Each integral is smooth and one-dimensional, so Gauss–Legendre quadrature evaluates it in microseconds. The entire recall prediction for a set of candidate parameters costs a handful of quadrature evaluations, not index build and benchmark run.

Figure 3 validates the end-to-end model: predicted average recall against measured recall across many parameter settings and multiple datasets has .

How the recall model selects vector quantization parameters

With a fast recall predictor available, parameter selection becomes a cheap ordered search. Given a target recall and a rerank budget (typically expressed as a multiple of ), we can find the minimum document and query bit counts, and other knobs, that clear the target. There are a few things to note that are practically important:

1. Glacial scaling gives us some safety because moves so slowly with for even moderate intrinsic dimension. A small margin in the calculation means the chosen parameters stay valid if a lot of vectors are added before parameters are restimated.

2. Small is the worst case if is a fixed multiple of . The gap \mathbb{E}[R_{(k)}]( is smallest for small so if a parameter choice satisfies the recall target at then it will for larger will too.

3. We can treat quantization as a black box because the error model only needs the empirical error variance. This means we can test any configuration, including preconditioning, the same way and we can simply order candidate parameter tuples by increasing index and query cost, and stop at the first choice that hits the target recall. For tuples of (query bits, doc bits, rerank depth, precondition) a sensible search sequence increases query precision first, then document precision , , , , , , , , and each combined (via an outer product ) with rerank depths like and precondition , exiting as soon as the target is met.

Results: auto-selected quantization parameters and recall across datasets

In this section, we discuss the results of the initial experiments on the end-to-end behavior. We’ve made some further refinements as part of the work to fully integrate with Elasticsearch that we discuss in our other post.

The table below shows auto-selected parameters targeting recall 0.97, measured with brute-force search, so the number reflects loss due to quantization alone (64 query clusters, targeting document clusters of size 384, which matches the settings of DiskBBQ).

Dataset

Query bits

Doc bits

Precondition

Depth

Recall

FiQA E5 small

4

2

false

30

0.97

FiQA arctic

2

2

false

30

0.95

FiQA GTE

2

1

true

30

0.98

MNIST

3

1

true

30

0.99

Fashion MNIST

3

1

true

30

0.99

Quora E5 small

2

2

false

30

0.99

Quora arctic

2

1

false

30

0.97

Quora GTE

1

1

false

30

0.98

Dbpedia E5 small

4

2

false

30

0.99

Dbpedia arctic

2

1

false

30

0.94

Dbpedia GTE

2

1

false

30

0.96

Wiki Cohere

2

2

false

30

0.99

Hotpot E5 small

4

2

false

30

0.97

Hotpot GTE

2

1

false

30

0.96

Glove 100

4

2

false

30

0.87

Glove 200

4

2

false

30

0.89

SIFT128

4

4

false

20

0.99

There are a few things worth highlighting:

• The recall is very sensitive to rerank depth. This is why we nearly always end up choosing the maximum depth available: a step up in rerank depth from 20 to 30 is typically what pushes us to hit the recall target for fewer bits and we prefer fewer bits. In the real system, we tuned this behavior based on a more representative reranking cost.

• Glove underperforms partly we approximate the query distribution with random samples from the corpus, but Glove is also less well characterized by the model than the other datasets. A plausible explanation is that the approximately uniform local density assumption from Section 2 is less reliable for Glove embeddings, which would show up as higher recall variance between queries. However, Glove embeddings are not representative of the actual vectors we need to store.

• The FiQA GTE preconditioning choice is a knife-edge case: preconditioning produced only a tiny expected recall improvement, but the prediction sat right at the recall cutoff and allows us to drop the query from 3 to 2 bits. If we'd rather only keep preconditioning where its benefit is clear-cut, we can enforce a minimum uplift threshold. This sort of fine-tuning of the decision logic leaves all the heavy lifting to estimate recall unaffected.

Key takeaways: auto-tuning vector quantization from first principles

We presented a method to pick optimal quantization parameters to achieve a target recall. It rests on two models that compose cleanly:

1. A geometric model of neighbor distances that follows from a local uniform density assumption. We use this to derive the nearest-neighbor distance, the glacial scaling law of the expected distance, and the expected distance profile . We show that fitting and by a simple log-linear regression to average distances in small random samples from the corpus gives an extremely accurate predictive model.

2. A Gaussian quantization error model that is justified by the CLT. Its only parameter is an empirical variance we estimate by comparing quantized and raw vector similarities for a sample of the corpus.

Finally, we show that it is possible to feed the estimated distance model into a Poisson-binomial count of neighbors that intrude on the top- set. Applying the Lyapunov CLT the expected recall@ after reranking to depth falls out as a one-dimensional integral we evaluate by quadrature.

The outcome is an accurate () predictive model of recall as a function of the quantization parameters. Choosing quantization parameters then becomes an ordered search with a predictive model telling us if we’ve hit the recall constraint. And nicely one that also comes with a built-in argument (glacial scaling) for why the chosen parameters remain safe even when estimated from a relatively small fraction of the data.

We’ve built this entire mechanism into Elasticsearch using segment merges as an opportunity to reassess our quantization choices. Aside from the peace of mind this brings (that you’ll achieve good recall whatever vectors you throw at it), it also allows us to chose near optimal parameters from a performance perspective. This closes the loop on our original objective: near optimal performance out of the box, at least as far as quantization goes. We’re pretty excited about the advantages that model based tuning can bring to vector search and look forward to sharing other work we have in this direction in the near future.

In HNSW, search proceeds by iteratively expanding candidate nodes in the graph, maintaining a bounded set of nearest neighbors discovered so far. Each expansion has a cost (vector operations, random seeks to disk, and more), and the marginal benefit of that cost tends to decrease as the search progresses.

One way to optimize HNSW graph traversal is to stop searching when the marginal likelihood of finding new true neighbors doesn’t increase. For this reason, in Elasticsearch 9.2 we introduced a new early termination mechanism. This stops the search process when visiting graph nodes doesn’t provide enough new nearest neighbors, consecutively, for a fixed number of times.

This article guides you through how we improved over the mentioned early termination mechanism in HNSW to make it better suited for different datasets and data distributions.

Early termination in HNSW

In HNSW, search proceeds by iteratively expanding candidate nodes in the proximity graph, maintaining a bounded set of nearest neighbors discovered so far, until it either has visited the whole graph or meets some early stop criteria.

Early termination is therefore not necessarily always an optimization, it’s part of the search algorithm itself. The moment we decide to stop determines the balance between efficiency and recall. In Elasticsearch, there are already a number of ways a query on HNSW can early terminate:

• A fixed maximum number of nodes is visited.

• A fixed timeout is reached.

While simple and predictable, these rules are largely agnostic to what the search is actually doing. Also they’re used mostly to make sure that the query finishes in reasonable time for the end user.

In a previous blogpost, we introduced the concept of redundancy in HNSW. In short, redundant computations occur when HNSW continues to evaluate new candidate nodes that don’t result in finding more nearest neighbors.

Patience: Measuring progress instead of effort

The notion of patience reframes early termination around progress rather than effort.

Instead of asking:

“How many steps have we taken?”

The new question becomes:

“What is the amount of computation we accept to waste, until we lose hope?”

During HNSW search, early exploration typically produces peak improvements to the top-k candidate set. During first steps of the HNSW graph exploration, the set of neighbors is continuously updated as the algorithm keeps discovering nearer and nearer neighbors to the query vector. Over time, these improvements become rarer as the search converges. Patience-based termination monitors this pattern and terminates the search once improvements have ceased for a sustained period.

In practice, while visiting the HNSW graph we also compute the queue saturation ratio as we hop through candidate nodes. This measures the percentage of nearest neighbors that were left unchanged while visiting the most recent graph node (or the inverse of the number of new neighbors introduced during the last iteration). When such a ratio becomes too big for too many consecutive iterations, we stop visiting the graph.

Conceptually, patience treats HNSW search as a diminishing returns process. When returns flatten out, continuing to explore the graph yields little benefit.

This framing is powerful because it ties termination directly to observable outcomes rather than to arbitrary fixed limits.

The benefit of using this smart early termination technique is that HNSW graph explorations tend to visit a smaller number of graph nodes while retaining an almost perfect relative recall.

To visualize this, we can plot the amount of recall per visited node that we got with the patience based early termination (labeled as et=static), when compared to the default HNSW behavior (labeled as et=no) on a couple of datasets, FinancialQA and Quora, and models, JinaV3 and E5-small.

Static thresholds and HNSW dynamics

In practice, in Elasticsearch this is implemented using static thresholds. One threshold refers to the saturation threshold: that is, the ratio of saturation that we consider suboptimal. The other threshold refers to the number of consecutive graph nodes that we allow to be visited while still having a suboptimal queue saturation: that is, the patience threshold.

When we introduced this early termination strategy in Elasticsearch 9.2, we decided to opt for conservative defaults, so as to let the recall as much as possible, while still gaining in terms of latency and memory consumption. For this reason, we set the saturation threshold to be 100% and the patience threshold to be set as a (bounded) 30% of the num_candidates in the KNN query.

In many scenarios, these settings resulted to work nicely; however, two queries requesting the same number of neighbors might have radically different convergence behaviors. Some queries encounter dense local neighborhoods and saturate quickly; others must traverse long, sparse paths before finding competitive candidates. The latter resulted to be the most difficult to handle effectively.

As a result, we sometimes noticed:

• Over-exploration for easy queries.

• Premature termination for hard queries.

Therefore, we figured that fixed threshold values encode global assumptions about convergence, whereas we could make HNSW better adapt to different dynamics.

Making HNSW early termination adaptive

Adaptive early termination approaches this problem from a different angle. Instead of enforcing predefined stopping thresholds, the algorithm infers when to stop from the search dynamics themselves.

So instead of comparing the queue saturation ratio between two consecutive candidates, we decided to introduce both an instant smoothed discovery rate (how many new neighbors were introduced for a query q, in the last visit i) together with rolling mean and standard deviation of such a discovery rate during the graph visit (using Welford’s algorithm). These statistics about the discovery rate are calculated per query, so that this information can be used to decide different degrees of patience for each query.

The previously static thresholds become adaptive to the discovery rate statistics: The saturation threshold becomes the rolling mean plus the standard deviation; whereas we make the patience adapt and scale inversely with the standard deviation.

The early exit rules remain the same; the saturation happens when the instant discovery rate is lower than the adaptive saturation threshold. The graph visit stops if the saturation persists for a number of consecutive candidate visits that’s larger than the adaptive patience.

This way, we obtain a behavior that doesn’t depend on the num_candidates parameter in the KNN query (which might be always set or left as the default, regardless of early exit) and that better adapts to each query and vector distribution dynamically.

The recall per visited node on FinancialQA and Quora with the adaptive strategy (labeled as et=adaptive) reports a higher recall per visited node, when compared to the static strategy (et=static) and the default HNSW behavior (et=no).

Adaptive early termination is turned on by default in Elasticsearch 9.3 for HNSW dense vector fields (and it can eventually be turned off via the same index level setting).