Fewer bits, same recall: More conservative auto-calibration sweeps for DiskBBQ

Try out vector search for yourself using this self-paced hands-on learning for Search AI. You can start a free cloud trial or try Elastic on your local machine now.

Auto-calibration in Elasticsearch now picks cheaper vector quantization for DiskBBQ at the same recall target. Across our 11 benchmark datasets, none of the segments end up on a 4-bit symmetric encoding. Plus, preconditioning gets chosen for three of them, which previously was almost never the case. And several datasets that had needed two document bits now need just one. Recall@10 held within three points, and when queries scan fewer clusters, it stayed the same or improved (by up to 11 points on one benchmark). Calibration still runs in one to two seconds per merge.

We got there by tripling the weight on document bits in the cost model and moving preconditioning into the inner loop, so it competes at every step of the sweep rather than waiting outside as a fallback. The recall model was never making this decision. It tells us which configurations clear the bar. The cost model picks which of those we ship, so changing one coefficient moved the chosen encoding on several of our 11 datasets and removed 4-bit entirely.

The recall model comes from our first blog post, and the merge-time sweep is from our second one.

How DiskBBQ auto-calibration previously chose quantization bits

As a quick recap, each candidate configuration has three parts:

• An encoding, written as (query bits, document bits): one of (1,1), (4,1), (4,2), (4,4), or (7,7).

• A rerank depth (oversampling factor): 1.25, 1.5, 1.75, 2.0, 2.5, or 3.0.

• Whether to precondition vectors before quantizing them.

The previous sweep sorted the 30 (encoding, rerank depth)pairs by:

cost = document_bits + 1.3 × rerank_depth

The 1.3 weight was set empirically. It prices rerank depth above a document bit because oversampling is a recurring per-query cost (more full-precision vectors fetched and rescored on every search), while a document bit is mostly paid once at indexing time.

That sweep ran the whole sorted list once with preconditioning off. It only ran the list again with preconditioning on if nothing reached the recall target.

This had two side effects that we didn't love:

• Bit tiers interleaved. With a weight of 1 on document bits and rerank depths spanning 1.25–3.0, the rerank term can swing the cost by up to 1.3 × 1.75 ≈ 2.3, which is more than one document bit. So the sweep tried a 2-bit encoding with shallow reranking before a 1-bit encoding with deep reranking:

Whenever a 2-bit encoding with shallow reranking was good enough, we committed to twice the document bits without first checking whether one bit with a little more reranking would also do.

• Preconditioning was a last resort. Since preconditioning was the outer loop, a segment could only end up preconditioned if no non-preconditioned configuration (all the way up to (7,7) at 3.0x) reached the target. In practice, that rarely happened, so preconditioning was almost never selected. When it would have let a 1-bit encoding hit the target, the sweep picked a much larger non-preconditioned encoding instead, sometimes 4-bit symmetric.

What changed: A sweep that spends document bits last

The new sweep follows one rule: don't spend more bits per document until everything cheaper has been tried. That means exhausting the cheap asymmetric encodings first, across every rerank depth and both preconditioning options, before moving on to the more expensive 4-bit and 7-bit symmetric encodings.

Two small changes get us there.

A heavier weight on document bits in the cost model

The cost function now has an explicit weight on document bits:

cost = 3.0 × document_bits + 1.3 × rerank_depth

To keep bit tiers from interleaving, one extra document bit must cost more than the widest possible swing in the rerank term:

DOC_BITS_WEIGHT > RERANK_COST_WEIGHT × (max_rerank − min_rerank)
               = 1.3 × (3.0 − 1.25) ≈ 2.28

The cost ordering now tiers by document bits, and it does so by construction; that is, every 1-bit configuration sorts before every 2-bit one, every 2-bit one before any 4-bit one, and so on. 

Vector quantization preconditioning moves to the inner loop

Instead of running the full sweep twice, we now try preconditioning on and off for each (encoding, rerank depth) pair before moving to the next, more expensive pair:

for (encoding, rerank_depth) in cost order:      # tiered by document bits
    for precondition in [false, true]:
        if predicted_recall@10 >= 0.9:
            return (encoding, rerank_depth, precondition)

Preconditioning now competes on equal terms at every step of the sweep. If a preconditioned 1-bit encoding hits the target, we pick it instead of escalating to more bits first.

The figure below shows the full sweep before and after the change. Each cell is one candidate, evaluated left to right, and the first one predicted to reach the target wins. Before, 2-bit and 4-bit candidates appear among the 1-bit ones, and the first preconditioned candidate doesn't show up until step 31. After, the sweep goes through the tiers in order, and preconditioning is tried from step 2:

The total number of candidates that the sweep can evaluate stays the same (30 pairs × 2 preconditioning options), so the worst-case calibration cost doesn't change. Since we now find acceptable configurations in the cheaper tiers more often, the sweep tends to stop earlier.

Why do extra quantization bits cost more than deeper reranking?

In our previous post, we argued that rerank depth should be priced above a document bit, because reranking is a recurring per-query cost, but bits are mostly paid once at indexing time. That argument still holds within a tier; for a given encoding, the sweep still tries the shallowest rerank first. What we revisited is the relative cost across tiers.

Document bits are a permanent, per-segment cost. They set the on-disk size of the quantized vectors and, more importantly for DiskBBQ, how much memory is needed to keep the posting lists that get scanned hot in the page cache. Theoretically, moving from one to two document bits doubles that footprint, and moving to 4-bit symmetric quadruples it, for every vector in the segment and for as long as the segment lives. (Practically, however, the cost is more constrained; for example, ~7% more disk and ~1.8x memory moving from one to two bits per dimension).  A slightly deeper rerank, or a cheap preconditioning step, buys the same recall without permanently growing the index.

Preconditioning, in particular, is a bargain. It reduces quantization error by spreading variance more evenly across dimensions. It doesn't change the size of the quantized vectors, and its query-time cost is a single transformation of the query vector. Treating it as a last resort left that headroom unused.

What auto-calibration now picks across 11 benchmark datasets

We reran the calibration over our benchmark suite and compared the previously picked configurations with those picked by the new sweep:

The shift is clear:

• Preconditioning gets selected. It’s now chosen for three of 11 datasets, but previously,  it was effectively never picked.

• No more 4-bit symmetric encodings. The datasets that previously needed (4,4) now reach the target with fewer document bits, once preconditioning is in play.

• Fewer 2-bit encodings and more 1-bit encodings. Several datasets that settled on (4,2) now land on a 1-bit asymmetric encoding with preconditioning and/or a deeper rerank.

Every one of those moves shrinks the quantized vectors; going from two document bits to one halves their size, and going from (4,4) to a 1-bit encoding cuts it by 4x.

Does auto-calibration slow down merges? 

The new sweep evaluates more candidates before it settles, but the numbers barely move:

• Calibration time. This is still one to two seconds per merge. The added work is less than one second.

• Indexing and merge time. There’s no measurable change.

• Recall where the configuration stayed the same (the majority of our benchmarks. This is unchanged.

• Recall where the configuration changed. Recall@10 moved by at most three points in either direction, with every cluster visited.

• Recall at a low visit budget. This stayed the same or improved (by up to 11 points on one benchmark).

How to enable auto-calibration on a bbq_disk field

There’s nothing to change. If you've already enabled auto-calibration on a bbq_disk field, newly merged segments get the new sweep automatically. The selected encoding, rerank depth, and preconditioning flag are stored in each segment's metadata as before, so existing segments keep their configuration until they take part in a merge.

What's next for auto_calibrate defaults

This change is small in code, but it says something about how we think about automatic configuration. The recall model tells us what's good enough. The cost model encodes what we care about among the good-enough options. When that preference is explicit, it’s much easier to reason about and evolve.

A more conservative sweep is a step toward turning auto_calibrate on by default.