However, with the rapid progression of coding agents, performance optimization has become a task we can tackle semiautomatically. Unlike many software engineering challenges, optimizing code offers a cheap and objective verifier. If you ask an AI model to make code faster, there’s a hard number at the end telling you exactly what happened, backed by profiling tools that explain why. This makes performance a perfect candidate for automation, provided you can actually trust the numbers.

If you simply point a coding agent at a benchmark, you typically get low signal-to-noise: wins that fall inside the variance of the environment, or variations caused by thermal throttling rather than better code. To capture optimizations that actually benefit Elasticsearch users, we had to bridge the gap between "checkable in principle" and "checked in practice." We built a highly trustworthy measurement loop: a harness that assumes the agent will be wrong a good fraction of the time but reliably catches and proves it when it’s right and then helps guide it where to look next.

Once the machinery is in place, the results speak for themselves. By letting this harness loose on the codebase, we've already begun uncovering meaningful wins across the stack. In part 2 of this post, we’ll dive into some examples it has found so far, including string conversion inefficiencies in Elasticsearch Query Language (ES|QL), an improvement to our NEON vector dot product implementation, and an upgrade opportunity for the gzip library we were using. In this part, we’ll take a look at the design choices we made and how they relate to the broader topic of effective harness development.

The AI code optimization pipeline architecture

The first step in any software engineering problem is to identify the correct high-level components. We made an architectural choice that turned out to be very helpful for this problem: separate understanding where opportunities exist from the loop making code changes. The agent starts with a real workload but only uses it to mine information about where to seek performance improvements. At this stage, it’s instructed to go broad and consider a range of performance-related signals. Once it has found and classified the hot spots, the agent reads the context of the code around them to understand the optimization opportunities. We use a separate task to condense the ranked list of hot spots into artifacts that a loop can iterate against in minutes: a microbenchmark that we prove exercises the hot path in its real operating regime. Finally, we use a proposer-verifier loop to actually make changes to the codebase to improve performance on the benchmark. This hands off to validation to assess the impact on real workloads at the end. Our CLI (atune) supplies the tools this process needs, and the rest is largely automated by a set of task-specific instructions.

For context, our high-level architecture looks like the following. Pink boxes are the humans, and teal boxes are the agent. There are three task types, one skeleton loop, and one referee.

An exploration task profiles a real workload and produces ranked opportunities; a human promotes one into an exploitation task, which iterates against an approved microbenchmark and commits each accepted experiment; a validation run on the real workload guards the result before a human reviews and opens the PR. Where no benchmark covers the hot path, a benchmark task authors one and a human approves it into a registry. A performance atlas informs every task and accumulates what each one learns.

Why performance optimization suits autonomous agents

Three properties make a task ideally suited for autonomous work, and it's worth being explicit about them because they provide a checklist you can use to evaluate automation candidates. You want:

1. An objective verdict so that the agent can be held to something other than its own opinion.

2. A dense guiding signal so that it knows where to look next instead of guessing.

3. A bounded blast radius so that being wrong is affordable.

Performance gives you all three. Benchmarks provide the verdict and profilers provide the gradient, while a rejected patch costs you wall-clock time rather than correctness. The change gets reverted, and the reason gets recorded. Life goes on. Regarding the first two, we've come to think the gradient matters more than the verdict. ″This got faster″ is binary, whereas a profile hints at what to try next. An agent can generate its next hypothesis conditioned on a rich guiding signal, the richer the better, rather than grinding through a list.

Signals are what the agent gets to see

A useful mental model is that the CLI is the agent's sensory apparatus. That changes how you design each command. Rather than exposing a capability, you design it to return a clear and concise answer to a question about the task at hand, and, when relevant, an explanation the model can reason over, instead of raw data it has to parse and interpret. These are the signals that the agent acts on, and we ended up with the following for our harness:

Signal

Question it answers

Facet-decomposed macro profile

Where in the code does real workload time go, per query type?

Allocation and lock sampling, in the same capture

Is the cost cycles, garbage, or contention?

Cost-composition classification

Is this in scope compute, other product code, GC, JIT tax, or parked threads?

Input-shape instrumentation

What does the workload actually feed this code?

Statistical verdict

Did this change help, at this measured noise floor?

Allocation-rate comparison

Did the new code end up allocating more?

Interpreted disassembly

Why did that result happen?

End-to-end A/B guard, with differential profile attribution

Did anything appear to break, and was it us?

Environment check

Is this machine even fit to measure right now?

Upstream duplicate search

Has somebody already reported or fixed this?

Four of these signals are worth dwelling on, because in each case the tool encodes a judgment that the agent would otherwise have had to keep making by hand.

Facet-decomposition is the clearest example. Blended CPU shares hide breadth: if you profile a mixed query workload, the grouping hash map insert and the percentiles sketch update can both show up as single-digit percentages of the total run time and look comparable. They aren't comparable at all, because the hash map insert is paid for by nearly every aggregation query, while the sketch update is only paid for when somebody asks for percentiles. So the profiler runs each named query facet as its own race, and every opportunity the agent records carries a breadth field (universal, broad, or narrow) and gets ranked by headroom × tractability × breadth. A universal 3% beats a narrow 10%. Putting the ranking function in the tool prevents it from having to be rediscovered on every run.

The cost-composition classification works as a router. Rather than handing the agent a flat top-N list of frame names, it buckets every sampled stack into ″in scope compute,″ ″other product compute,″ ″GC,″ ″JIT and safepoint overhead,″ ″off CPU waiting,″ and ″parked threads.″ Each of those buckets implies a different kind of investigation. If GC is above about 15%, the real target is allocation rate, and the CPU top frames will actively mislead you, because they show where objects were collected rather than where they were created. If JIT and safepoint overhead is above about 25%, you're looking at a ceiling rather than an opportunity, since no in-scope code change will move it. If threads are parked and core utilization is low, this is a concurrency problem and CPU flame graphs are the wrong instrument entirely. We wrote that mapping into the playbook as a table, so the model (even a cheap one) reads a profile the way that an experienced engineer would, rather than reaching straight for the top frame. The classification is a prior for forming a hypothesis, though, not a substitute for evidence, so the agent still has to cite specific frames when it proposes an experiment.

Interpreted disassembly is a tool we hadn't originally provided, but it most definitely earns its place. It helps answer why, the question that unblocks the next hypothesis. Flame graphs tell you where the time goes; they rarely tell you why a change made things worse. So atune asm runs the benchmark briefly with the JIT told to print the assembly for one hot method, captures both sides of the working-tree diff, reduces each to the final C2 compilation, normalizes the addresses, and diffs them. The diff alone would likely still be 4,000 lines of aarch64, so on top of it sits an interpretation layer: a per-mnemonic delta, a net instruction count, the compilation tier that was actually captured, and a vectorization signal that counts vector register references on each side and raises a warning when they're eliminated, halved, or narrowed from Advanced Vector Extensions (AVX) to Streaming SIMD Extensions (SSE) width. The playbook then maps mnemonic patterns to causes:

Pattern in the diff

Likely cause

b.eq/b.ne up, csel down

New unpredictable branches

Clusters of str/ldr against the stack pointer

The compiler ran out of registers

NEON loads replaced by scalar compares

The vector path degraded

In one experiment, the agent fused two SIMD mask extractions into one, and the benchmark regressed by 26%. The vectorization warning explained it in about 10 seconds. Without that tool, the agent has a dead end and no working model of the machine; with it, it has a corrected model and several new ideas.

The fourth signal is less a single tool than a habit; the instruments check themselves. Core utilization is derived two independent ways, from sample density and from process sampling, so the two can be compared. The classification is rejected if the unclassified bucket exceeds a budget, on the grounds that a breakdown which can't account for its own samples shouldn't be reasoned over. The disassembly capture warns when the compilation it caught isn't the steady-state one. The upstream duplicate search is restricted to read-only commands, and that restriction is enforced by a test that greps the source, so no future edit can quietly reintroduce the ability to file anything. Each of these exists because a tool that can be confidently wrong is worse than a tool that is merely absent.

One small piece of design is worth highlighting as a specific instance of good return practice. atune compare returns 0 for improved, 1 for no change, 2 for regressed, and 3 for error, and the loop branches on this code. That means no parsing and no ambiguity about what the verdict was. Plus, no tokens are spent interpreting prose.

If you take one thing away from our CLI design, it’s a broader design principle. In a general setting, the interesting thing isn't the individual signals we found useful to understand performance; it's that the CLI is capturing and packaging the judgment of an experienced performance engineer into tools that return answers rather than raw data. Structurally imposing good judgment about the problem an agent is tasked with improves outcomes. The right CLI is as much part of that story as the instructions. Furthermore, tokens are saved by tools that return decisions and digests rather than data. That means a comparison verdict instead of raw JMH output; a triage summary sitting on top of a 4,000-line disassembly diff.

From 20-second probes to hours of validation

Building a verifier that you can afford to consult is a separate problem from building one that you can trust. This covers the affordability part. Or, if you like aphorisms, real workloads are where truth lives and where iteration goes to die. A macro profile takes 45 to 60 minutes, and an end-to-end validation run takes hours. But a microbenchmark takes minutes. That's why the exploration-then-exploitation split works; you go broad on the real workload once and then hand off to a microbenchmark that you can iterate against in minutes.

The catch is that the handoff is only sound if the microbenchmark exercises the hot path in a realistic operating regime; that is, with the right cardinality and right data distribution. If you get that wrong, your fast loop spins fast but in the wrong direction. Until we finalized the handoff procedure, we saw cases where the agent accepted changes on a benchmark whose key distributions happened to flatter it, and only the end-to-end run caught the problem.

Handing off from exploration to exploitation

The handoff between the two phases is structured rather than informal. An exploration task's primary deliverable is a set of opportunity records, and each one carries the scope paths that an exploitation task would be allowed to edit, a headroom estimate, a classification (constant factor, structural,

allocation, or concurrency), and the benchmark it would gate on (or an explicit "no benchmark coverage" flag, if none exists). Each also carries a narrow test pattern so that the correctness gate stays cheap. The test pattern field exists because of a specific incident; an exploration task omitted it, and the resulting exploitation task ran a very heavy test suite on every experiment. The fix was to change the upstream artifact rather than add an instruction downstream. This is a pattern we use repeatedly; make and record decisions as early as possible rather than re-derive them each time.

Validating a new benchmark before it can gate anything

Where benchmark coverage is genuinely missing, a dedicated benchmark task authors one, and that new benchmark has to pass validity checks before anything can rely on it. Two of them are mechanical: what fraction of the benchmark’s hot self-time comes from frames that actually appear in the production profile and whether the parameters fall inside the input shapes we measured. The third is a checklist that the agent has to attest item by item. It exists to avoid the JIT getting a simpler world than production.

1. Inputs have to be reshuffled rather than fixed or sorted, so the branch predictor doesn’t get too good.

2. Results have to be consumed, or dead-code elimination deletes the thing that you meant to measure.

3. Inputs must not be compile-time constants, or they get folded away.

4. Call sites have to see roughly the product mix of types, because a monomorphic call site inlines, whereas a megamorphic one doesn’t.

A human then approves it into a hash-pinned registry. Until that happens, it's inert, because task setup refuses any task citing an unapproved benchmark. We think of that approval as the strongest gate in the system, and it's deliberately placed. An approved benchmark can decide accept or reject in every future task, so it's the one place where we ask for a human signature on an artifact rather than on a decision.

The validation ladder

Underneath all of this sits a hierarchy of feedback mechanisms with the property that each rung is cheaper and weaker than the one below it, and the cheap tiers are for rejection only.

Tier

Cost

Role

probe

~20–60 s

Directionally right? Can never accept

codegen capture

~4 min

Why did that happen?

screen

~5–15 min

Cheap statistical filter

confirm

~20–60 min

The accept decision

end-to-end

hours

Regression guard, advisory

The asymmetry between accept and reject is doing real work here. A probe is a single paired fork, whereas the accept predicate requires a full confirm run with a matching calibration record. Agents are very good at telling believable stories; indeed, they're trained on many tasks judged by both LLMs and by humans, so being convincing is actively rewarded. We don't want an agent to be able to promote a cheap signal into a decision by being persuasive about it.

For this task benchmark, wall-clock is a real cost consideration. A confirm run might take an hour. So one has to weigh carefully all the costs involved when choosing the setup. A model that lands one hypothesis in four typically beats a cheaper one landing one in 10 by a margin on end-to-end metrics. The usual instinct to down-spec the model on a long-running loop is exactly backward here. When your loop has an uncertain outcome and significant costs beyond the tokens it consumes, you may well find yourself in the same situation.

How do you know a performance improvement is real?

Is it faster? is a statistical question. So we made the accept predicate code rather than judgment and put it somewhere the agent can't bypass. There are four ideas in the accept decision, and in each case, the alternative we rejected is as informative as the choice we made.

Forks are the statistical unit

Each JMH fork collapses to its mean, and verdicts come from an exact two-sided Mann-Whitney U test at α = 0.05 plus a seeded bootstrap confidence interval over three to five fork means per side. The reason is that iterations within a fork share JIT and heap state and are therefore autocorrelated, so treating them as independent samples manufactures significance out of nothing. We rejected comparing single-run scores, which is pure noise, and iteration-level t-tests, which can be confidently wrong. Seeding the bootstrap means that a rerun reproduces the verdict exactly because the agent needs to be able to tell the difference between a result that changed and a result that was never stable.

Pair candidate and baseline in time

The screen (three forks, optionally over a subset of parameters) exists only to kill bad hypotheses in maybe 10 minutes instead of an hour. The accept decision itself comes from a confirm run that measures candidate and baseline back to back using a stash-flip. In an unstable environment, thermal and background drift only cancels if both sides ran under the same conditions, so an hours-old baseline is really a different experiment.

The noise floor is measured, not assumed

The minimum effect size that a task will accept has to clear the A/A-calibrated coefficient of variation for that specific benchmark on that specific machine, and both the confirm run and the comparison refuse to proceed without a matching calibration record. A 1–2% improvement on a laptop is indistinguishable from noise, and because the floor varies by benchmark and by JDK, any global constant you pick will be too loose somewhere and too tight somewhere else.

The accept rule is composite and deliberately conservative

A parameter combination counts as improved only if p < α, the effect clears the calibrated floor, and the confidence interval excludes zero. Overall acceptance then requires that nothing regressed (not the primary benchmarks and not the guards) and that at least one primary combination improved. The headline figure is the geometric mean of the per-combination speedup ratios, which is always positive and composes across experiments, so a task's cumulative improvement is a meaningful number rather than a sum of incomparable percentages.

What must not get slower

Guards deserve a note of their own, because they answer a different question to the primary benchmarks; not Did this get faster? but What must not get slower while it does? The task definition lists them separately for that reason, and they're typically the operations and the input regimes that the change isn't aimed at. If you're optimizing insert throughput on a hash table, iteration is a guard and so is a collision-heavy key distribution. A change that improves the common case by weakening the hash function will look good on uniformly distributed keys while catastrophically degrading more adversarial inputs. We know that because it happened; the collision distribution was missing from the matrix that accepted one of our early experiments, and the task now carries a comment telling future readers never to drop it again for a hash-quality-sensitive scope. While guards cost wall-clock on every confirm, they also surface edge-case regressions, and omitting them can be much more costly in the long run.

Runtime isn't the only thing worth guarding, either. The confirm run also captures normalized allocation rate on both sides and flags any change that buys speed with more than about 15% extra garbage. That one is advisory rather than blocking, because sometimes the trade is the right one. However, it's the kind of regression a purely time-based accept rule would happily wave through but might raise a red flag to an experienced performance engineer with better understanding of the calling context.

The end-to-end gate is one-sided and default open

The end-to-end gate is a different statistical problem: small n, high noise, and a very strong prior that we should accept based on our microbenchmark results. Our first design treated accept and reject on an equal footing, and it produced multiple clearly spurious rejections, so the redesign is one-sided and default open. An operation is flagged only when the median regression exceeds the threshold and every candidate repetition is slower than every base repetition. That full-separation criterion is the nonparametric one-sided test at this sample size, and it's robust to the single outlier repetition that would occasionally fool us otherwise. A flag then also has to be corroborated against a differential CPU flame graph, where only a rise in the task's own in-scope CPU share counts as real. Near misses get reported for transparency but don't trigger triage, and nothing is ever auto-rejected; a flag is a request for human attention rather than a verdict. We have a final backstop which is the large suite of performance tests that we already run against Elasticsearch on a daily basis.

The one-sided gate lesson generalizes beyond benchmarking. A noisy gate should be one-sided and default open where there is strong prior reason to accept. A symmetric threshold on a noisy signal doesn't just cost you real wins, it teaches the loop to distrust its own instruments, and that’s a much more expensive failure.

Exploration and exploitation need different permissions

Exploration and exploitation might look like two phases of one activity, but they have different inputs (a macro workload versus a pinned scope) and different outputs (ranked opportunities versus commits). They also have different failure modes, which means they want different permissions. We made the split a first-class property of a task, which lets us enforce it; an exploration task literally cannot commit. The baselining, comparison, checkpointing, and validation CLI all refuse exploration tasks, benchmarking allows probes only, and every probe diff is always reverted. A broad, speculative survey is safe because nothing it does can edit the code.

A nice ancillary benefit is prompt focus. Each type reads one playbook, in full, with the others explicitly not loaded. If you try to write a single document covering both "find where the headroom is" and "land a validated win inside this scope", you get something that does neither well, because the instructions for good exploration (follow the profile, widen the net, a broad survey is preferred) are close to the opposite of the instructions for good exploitation (one hypothesis at a time, minimal diff, never widen scope).

AI agent memory: Journals, knowledge bases, and postmortems

Sessions are ephemeral, but what you can learn from them isn't, so the harness accumulates three durable assets, plus one disposable view derived from them. These are a journal of the code changes we’ve tried, a knowledge base of how the code performs, postmortems of when the harness failed and, because sessions can be stopped and resumed, a session summary. What makes them work together is a clean ownership rule about which kind of fact goes where.

The journal records what we tried and measured. It's append-only, one file per task, and one record per experiment, and it's written before the code is edited. Rejections carry a forward-looking note in the form "do not retry X because Y", which is probably the highest value line, because it's what stops the next session re-deriving a dead end. Records also carry the environment and the driving model, which means that hypothesis hit rates are comparable across models.

The knowledge base records how the code works and how it performs. It's an indexed collection of per-area summaries, each stamped with the commit it was written against, and every playbook ends with an upkeep step that appends whatever durable facts the run turned up. Because the performance characteristics of the JDK also change from time to time, for example, a new Vector API might implement vector masking more efficiently on AArch64, findings from profile data are also tagged with the JVM version they apply to.

The postmortems record mistakes that the agent has made in the past, and they're indexed by symptom rather than by date. The question a session actually has when a number looks wrong is Have I seen this shape of wrongness before?, and a chronological list doesn't answer it. So the rows read like "validation fails on operations structurally unrelated to your diff", or "screen reports no matching combinations".

Keeping the journal and the knowledge base distinct sounds pedantic but isn't because without the rule, both of them turn into a diary that has a tendency to bloat the context window or miss critical information in context.

The disposable state is a session handoff, and our advice is to never write it by hand and to avoid asking a model for a session summary, if possible. Our harness regenerates a one-page digest mechanically from the journal, so a resumed session doesn’t spend time and tokens reconstructing where the task had got to. Because it’s derived rather than authored, it can’t drift from the record in the way hand-maintained content does. That’s also why it isn’t part of the audit trail; because it’s cheap to regenerate it from the journal.

Mechanical session handoffs point to two lessons, and they turn out to be the same one. Every time a person appears in the loop, it’s a source of friction and an opportunity for error. And every time you reach for a model, ask whether code can do the same job. This seems like an odd thing to advocate in a project whose primary premise is delegating to a model, but the habit is easy to fall into once you have one to hand. Judgment is expensive, wherever it happens to sit, so the person and model both have to earn their place on merit.

Two further things are critical for durable agent memory. The first is that knowledge rots, so you have to lint it. Elasticsearch's main branch moves daily, which means the knowledge base's file citations go stale, so one linter checks them. Another checks that every command, flag, and path cited in the agent-facing docs actually exists, that every postmortem is linked from the index, and that every task type has a playbook, and it runs as part of the test suite. Treating prose written for an agent as a testable artifact is the reason it stays true. The second is that loading discipline is half of memory. The instruction is to load the index and then the one or two summaries matching the task's scope; never to bulk-load the rest. Memory you can't afford to read isn't memory.

Coding agent guardrails: Containment, scope, and stop conditions

Elasticsearch is millions of lines of code, and an unscoped "make it faster" run against a codebase that size is unreviewable and unfalsifiable. It’s also expensive. The converse is that a verifier only protects what it can see, so we have to apply the same boundary to what it can change. The outcome is a task that fixes its goal, allowed paths, benchmarks, thresholds, and stop conditions before any code is edited, and none of those are things the agent may change during a run.

The human owns the safety envelope, defining scope, thresholds, and stop conditions before any code is edited, and owning every handoff that crosses a trust boundary. The agent session owns judgment: reading profiles, forming one hypothesis at a time, editing inside scope. The atune CLI owns mechanical enforcement: scope checks, correctness gates, statistics, calibration, stop conditions, environment checks, and the signals. Persisted state is the record: a worktree pinned at a base commit, an append-only journal, and generated reports.

Containment happens in two layers. The first layer is coarse and task-agnostic; it’s a single static permissions file lets the session edit the per-task worktree (which includes a local branch of Elasticsearch, all journal entries, and CLI artifacts) and the knowledge base, and it denies the pristine Elasticsearch clone, the task definitions, the harness config, and the audit trail. The second is precise and per-task: a git-level scope check that runs over tracked and untracked files before any build, so an out-of-scope edit is rejected before it can be benchmarked or committed.

Two containment layers have a nice corollary. Widening the first layer to the whole worktree costs nothing, because nothing out of scope survives the second one. Coarse containment plus precise scope beats trying to make a single mechanism do both jobs, which is what we tried first and which left us with a permissions file that needed editing for every new task.

Stop conditions are mechanical. There's a maximum number of experiments and of consecutive rejections, along with a cumulative improvement target, and proposing a new experiment is refused once one of them fires. A human can override; the agent can't. That asymmetry is what makes the gates independent of which model happens to be driving.

Tests are add-only. New test files ship with the checkpoint, and modifying or deleting an existing test is blocked by the scope check. This closes the single most tempting shortcut in the entire problem space by construction rather than by instruction, which seems like the right way to handle any shortcut you'd otherwise have to keep asking an agent not to take.

The harness lives outside the code it optimizes. The Elasticsearch clone is a separate, gitignored directory, and each task gets a worktree pinned at a base commit. The payoffs compound; the subject repo stays pristine and upstream mergeable with nothing related to the harness leaking into a PR, and the audit trail is versioned independently of a codebase that moves daily. Also, tasks are isolated from each other and from any developer checkout. Targeting a newer Elasticsearch means creating a new task rather than repointing an old one, because that task's numbers are tied to its base. It also means that the harness is retargetable in principle, since the Elasticsearch-specific parts are configuration, knowledge base, and benchmark registry rather than architecture.

The decisions the agent never makes

The rule we settled on is that the agent runs the loops and a human owns every step that crosses a trust boundary. That is creating work, blessing a measurement instrument, publishing a branch, or acting outside the repo. None of those are in the agent's allowlist, and each has a reason worth stating:

• Humans have to sign off on the task because the thing being constrained can't set its own limits.

• Calibrating the noise floor needs to be done once per benchmark, per machine, and by default is measured rather than assumed. However, it's rather expensive and we allow a human to override if they know the environment well.

• Deciding what to optimize next by promoting an opportunity is a judgment and a new scope.

• Since benchmarks go on to gate other tasks, we consider reviewing this artifact part of the correctness safety net.

• We leave outward-facing actions, such as pushing a branch or filing an issue, to a human until we're confident in the process.

• We allow actions to be forced, but the override has to sit outside the thing being overridden.

How the human actions get surfaced in the workflow matters. The generated report and the session handoff both print the human actions currently due, at the moment they become due, rather than leaving them to be inferred from the playbook.

We're deliberately not taking a position on how permanent the manual processes are. The right amount of supervision for a new technology is an empirical question, and we'd rather measure it than argue about it. We’ve started with a relatively high degree of supervision because that's the cheap direction in which to be wrong (a gate you never needed is easier to remove than a regression you shipped) and because the harness makes the question answerable. Every gate is a named, logged transition, so over time we can see which of them ever changed an outcome and which only ever cost friction. In summary, measure first, and then refine.

Building the harness is the same kind of loop

A lot of the harness design didn't fall out of an initial design document. The signal set, the ranking function, the shape of a task, and the exact wording of a playbook rule each came from watching a run go wrong. If there's one piece of advice here that generalizes, it's to use the thing before it's ready and to instrument your own disappointment.

The clearest example is a rule we now call distrust surprising results. A validation run reported that every operation had regressed, the worst of them by 14.8%. It was wrong twice over. A target operation pattern had overmatched a completely different code path, and a stale output directory from an earlier run was being read alongside the new one. Offered a coherent story, the agent took it and reverted a change that was actually good.

What went into the playbook after that incident is not "be careful." It's a three-step check to run before acting on a surprising verdict:

1. Trace the code path, and confirm that the thing which moved can even reach your diff.

2. Read the raw per-repetition data rather than the summary, and recompute one headline number by hand.

3. Compare the report's shape against a known good run, because a structurally different report implicates the pipeline rather than the code.

Alongside that, there’s another important rule, which is if the agent concludes that the harness is buggy, it must not fix it mid-run, because a mid-run harness change makes every result in that run incomparable. It should journal the evidence and stop.

Improving the harness is itself a loop worth describing. Asking the model to review its own transcripts and the harness documents, and to propose the rule itself, usually works well. It's good at spotting where its own instructions were ambiguous, in a way that's hard to reproduce by rereading the instructions yourself. What makes that output useful is having somewhere for it to land: a terse rule in the playbook, the narrative in a dated postmortem, a symptom keyed index row, and a linter that keeps the citations honest.

Restraint turns out to be part of the same discipline. The design document carries an explicit list of extension points that we've deliberately not built, because the need for them is still speculative. That's the same "don't guess, wait for evidence" rule we impose on the optimization loop, applied to ourselves.

How this applies beyond performance optimization

A few of these themes aren't specific to performance work or to Elasticsearch.

Verifiable work is the current frontier. The same insight drives reinforcement learning with verifiable rewards, and it’s what AlphaEvolve is built around. The tasks agents are consistently good at are the ones that come with a cheap oracle (tests, compilers, benchmarks), and the interesting move isn't finding more such domains but manufacturing oracles for domains that lack them. Performance is an instructive case precisely because the oracle is, in some senses, obvious, and yet building it still took most of the engineering.

The referee pattern generalizes, too. Separating a fallible optimizer from mechanical enforcement is the same shape as sandboxed execution and policy engines: judgment in the model, invariants in code. The practical consequence is that the system's safety doesn't depend on which model drives it. A weaker model wastes benchmark time, but it can't corrupt the code or accept a bogus win.

Goodhart is a standing adversary for any optimization task that uses an agent, and it has a formal treatment worth reading. An agent optimizes exactly what you tell it to, so the two-tier benchmark structure, add-only tests, benchmark approval registry, adversarial guard distributions, and a marker file that makes timing an instrumented build mechanically impossible are all one design theme wearing different clothes.

Tools are context engineering. The consensus is drifting away from "expose everything" and toward a few well-shaped tools that return digests: the principles of progressive disclosure, self-documenting interfaces, and verdicts rather than payloads.

Memory is becoming architecture. A rules file, per-task playbooks, a durable knowledge base, and an append-only journal form a hierarchy with different lifetimes, owners, and loading rules, and the hard part is eviction and staleness rather than storage; hence, the linters.

Finally, human-in-the-loop is a dial rather than a switch, so where it should sit is something to measure per domain rather than assert.

What's in part 2 of this post

The harness design is a set of hypotheses about what autonomous performance work needs, and the harness was built so that we could test them. Part 2 is that test: the first four PRs we raised using it, their gains, how many hypotheses it took to get each one, which gates actually caught something, and where the harness got in its own way. That includes the changes which didn't survive end-to-end validation, since as usual, the rejections are as informative.

In our previous blog, we laid out the statistical model behind that calibration: a manifold model for how nearest-neighbor distances scale with index size, a Gaussian error model for quantization noise, and a closed-form way to combine the two into an expected recall@k for a given rerank depth. If you haven't read it, the one thing you need going in is this: given a candidate quantization encoding and a rerank depth, we can predict recall@k without building an index and benchmarking it, by fitting two small models to a sample of the corpus.

In this post we’ll go through how to score candidate configurations cheaply and how to leverage that to make merge-time decisions that are themselves cheap, correct, and consistent across real, constantly-merging indexes. This led to some pretty impressive improvements: we see almost 17% average improvement in QPS across a broad range of datasets all while increasing recall (in one case by a factor of 3). What’s more you get this immediately by adding one line to your index options, "auto_calibrate": true, and our plan is to make this our default once it has had the chance to bake a bit.

Why manual vector quantization tuning is unreliable

bbq_disk exposes several knobs: quantization bits for documents (1, 2, 4 or 7), a separate bit width for queries, an oversampling factor for reranking, and whether to precondition vectors before quantizing. None of these act independently, and their effect on recall depends on the data: a 4-bit/1-bit encoding might be plenty for one embedding model and clearly insufficient for another. A single index is also built out of many segments, merged over time, each with an eventually different vector distribution. Hand-tuning one configuration for an entire index is, at best, a compromise, which is the motivation for automatic calibration: let each segment have its own configuration, re-evaluated every time it is involved in a merge operation.

How Elasticsearch runs auto calibration at merge time

When a number of segments are merged and automatic calibration is enabled, Elasticsearch samples documents and queries from the vectors being merged and:

1. fits the manifold model over a sequence of nested samples of the merged corpus;

2. fits the error model, predicting the quantization error's standard deviation for each candidate (query bits, document bits, precondition) combination;

3. sweeps candidate configurations in ascending cost order; the candidate encodings are (1,1), (4,1), (4,2), (4,4) and (7,7) (query bits, document bits), each tried across oversampling factors of 1.25, 1.5, 1.75, 2.0, 2.5 and 3.0;

4. estimates recall@10 for each candidate using the model described in our first post, and stops at the first (cheapest) configuration predicted to hit the target of 90% recall@10.

The winning configuration (encoding, oversample factor, precondition flag) is stored directly in the segment's metadata, so it travels with the segment and is picked up automatically at query time unless a request explicitly overrides it.

Small segments skip this altogether: below 10,000 merged vectors, there isn't enough data to fit a reliable model, so Elasticsearch just uses the current DiskBBQ defaults (4-bit query / 1-bit document encoding, no preconditioning, 3x oversampling).

How the vector quantization cost model works

Following the principles described in our first post, we started by picking candidates with three nested loops that are essentially how you might imagine hand jamming a lookup table. Start with the quantization scheme as the outer loop, ordered cheapest to most expensive by document bits ((1,1) → (4,1) → (4,2) → (4,4) → (7,7)). Then we set the rerank depth within the middle loop, ordered shallow to deep (1.25× → 3.0×). Finally we set preconditioning within the inner loop (off → on).

That ordering has a cost model baked into it, it's just implicit rather than written down: exhaust every rerank depth at the current bit tier before ever trying more bits. Document bits were effectively the only resource priced as expensive; oversampling was treated as nearly free by comparison, since the sweep would always max out rerank depth on a cheap encoding before considering a pricier one.

The current implementation replaces that with an explicit, continuous cost function:

Query bits still don't factor into cost at all, only document bits (which drive index size) and rerank depth (which drives how many candidates get rescored per query). Preconditioning also stays outside the formula: Elasticsearch runs the whole cost-ordered sweep once with preconditioning off, and only if nothing meets the recall target does it re-run the sweep with preconditioning on, treating it as a fallback lever rather than something priced bit-for-bit against the other two.

With this cost model, rerank depth costs noticeably more per unit than a document bit, so the sweep will often prefer stepping up a bit tier over pushing oversampling deeper.

The main reason for this is that once you're running in a serverless deployment, compute and storage are billed and scaled independently, on very different clocks. An extra document bit is mostly a one-time, indexing-time cost; it makes the segment marginally bigger on object storage, which is cheap and doesn't need to be pre-provisioned against a spike in query traffic. It does carry a smaller recurring cost too, since quantized vectors sitting in page cache or loaded for scoring take proportionally more RAM per document as bit width grows, but that scales linearly and predictably with corpus size, and doesn't spike with query load. Rerank depth is the opposite: it's a recurring, per-query cost.

Every extra unit of oversample factor means fetching and rescoring that many more full-precision candidate vectors from disk, on every search request, for as long as the index is queried. That's compute and DRAM pressure on the search-serving tier, which has to autoscale in close to real time to match query concurrency. It sits on the hot path of the latency-and-cost budget in a way storage capacity, and the RAM footprint of the bits themselves, does not. Weighting rerank depth higher than document bits in the cost formula is what makes the sweep reflect that asymmetry.

Efficiently estimating vector quantization error

The cost model above works with the premise that the recall estimate behind it is trustworthy. The manifold and error models need to be accurate for the recall assessment to be trustworthy. While the manifold model of the k-th to N-th nearest neighbors distance is cheap to compute, the standard deviation of the quantization noise for a given candidate encoding is a bit more expensive in principle.

DiskBBQ uses fixed count clusters to accelerate nearest neighbor queries. Our quantization procedure takes advantage of this by only quantizing the vector residuals from the cluster centroids. This means as the data scales, the magnitude of vectors we quantize relative to the various components of the similarity calculation shrinks. As such, quantization accuracy increases. We need to account for this when converting our sample estimates to the segment as a whole.

Clustering the corpus at several sample sizes and fitting how the error scales with cluster size requires re-clustering a real sample of the corpus at several different sizes and fitting a regression model to see how the error shrinks as the effective cluster size grows. We also add a conservative +3σ margin on top of the fitted estimate to guard against noise in the fit itself. This is accurate and appropriately cautious; however, while benchmarking on common dense retrieval datasets, we found that performing several hierarchical k-means passes per candidate was expensive.

To speed things up, we tried approximating residuals with a synthetic isotropic-Gaussian formula. Instead of clustering increasing-size samples, this approach generated synthetic residuals from the manifold model's local density estimate. It was fast and fit for background merges, with the full repeated clustering approach reserved for force-merges only. However, it turned out to inflate error when embeddings (residuals) are anisotropic (some directions carry a lot more variance than others). As a result, the estimated error could grow significantly on strongly anisotropic data (e.g., Fashion-MNIST-style image embeddings).

So instead we looked for a still fast but more accurate way of calculating residuals. We opted for using a single clustering pass over a smaller sample (2,048 vectors). The clustering runs once per merge and is then warm-started for every candidate encoding evaluated afterward, instead of re-clustering from scratch each time. To get the error's dependence on corpus size, which the baseline learns by re-clustering at multiple sizes, this approach instead reuses the manifold model's invDim as a plug-in for that dependence, extrapolating from the single real measurement rather than fitting the size relationship separately.

We also trimmed the query sample used during calibration from 1,024 to 256 vectors, on the reasoning that a smaller sample is enough once the error is being measured from real data rather than synthesized (and validated by benchmarks). The net effect was comparable wall-clock cost to the synthetic residual formula it replaced, but grounded in real per-cluster residuals, accurate enough that force-merge and background merge could be unified onto one path.

As an example, we take five different benchmark datasets and calculate the quantization error standard deviation (SD) by directly measuring the gap between exact and quantized dot products on a sample of real (or, for the synthetic residual formula, fabricated) residuals, then extrapolating that measurement to the full corpus size. They differ only in how much sampling and regression goes into that extrapolation: the multi-sample scaling fit sweeps fifteen sample sizes and fits how error scales with cluster size, the single-pass real residual measurement takes one larger real residual sample and reuses the manifold's intrinsic dimension to estimate the size dependency, and the synthetic residual formula skips real residuals altogether and samples from a synthetic Gaussian from the manifold's expected rank distance. We treat the multi-sample scaling fit as ground truth in this comparison because it's the most sample rich of the three, not because it's a zero variance measurement of the "true" corpus-wide error (it has its own sampling noise too). The table below summarises the methods and findings.

Method

How it works

Speed

Accuracy

When used

Multi-sample scaling fit

Clusters at 15 sample sizes, fits regression

Slow

Gold standard

Ground truth baseline

Single-pass real residual

One clustering pass + manifold invDim plugin

Fast

Near gold standard

Background + force merge

Synthetic residual formula

Gaussian from manifold density estimate

Fast

Inflated on anisotropic data

Deprecated

In order to exchange methods, we only need to be confident that they agree. This question can be answered independently of the correctness of the actual estimates, which we verified in our previous post for the multi-sample scaling fit. The figures below report the predicted quantization SD and the predicted recall@10, which is influenced by how we estimate the error. We report the analytical recall the manifold model predicts as a function of the quantization parameters, given the estimated error distribution perturbing the true distance ordering. This way, we isolate the quantization error's effect on ranking from any separate recall loss the IVF index itself might introduce, which is a distinct error.

The single-pass real residual measurement's calculated error SD is closer to the multi-sample scaling fit (our gold standard), with respect to the synthetic Gaussian residuals. Consequently, the predicted recall is closer when using the single-pass + manifold plugin method. Indeed, we found the models to be essentially interchangeable regarding the indexing decisions they lead to. Critically, we lower the calibration overhead by an order of magnitude.

Auto calibration overhead on indexing performance

We compared the cost of auto calibration on indexing, when compared with ES defaults, over 18 public benchmarks. We noticed that more than 50% of the datasets report an auto calibration overhead below 2%. Three datasets report 16-27% overhead, while two datasets sit in the 31-35% overhead.

The merge overhead is larger for smaller datasets (Fashion-MNIST, FiQA) that get indexed in a few seconds; that is expected as the size of the vector samples being used for calibration is fixed and therefore more noticeable with tiny datasets. In fact, for larger datasets like DBPedia-Entity and HotpotQA (5M doc vectors) the overhead is sometimes not noticeable and within 11% in the worst case.

What quantization parameters does auto calibration choose?

Looking at the encoding auto-calibration landed on for each of the real datasets:

Query bits were 4 in every dataset. While query bits aren't priced into the cost formula, we still iterate through lower query bits first (e.g., at 1 bit doc vectors, we first evaluate recall for 1 bit query vectors, then for 4 bit query vectors); so it’s possible for some datasets to even choose symmetric 1-bit quantization. The center of mass is a 2-bit document encoding with somewhere between 1.5x and 1.75x oversampling; 4-bit only shows up for two genuinely harder datasets (Fashion-MNIST's image embeddings, GIST-1M), and 1-bit only for a handful of the text-embedding models that are most robust to quantization. In fact, our own models are among those that quantize best: we selected 1 bit documents for all three corpuses we tested with Jina v3.

Recall and QPS improvements from automatic calibration

Auto-calibration is a broad win across the eighteen datasets: QPS improves in 15 of 18 cases (often substantially, double digits on about ten, and over +50% on FiQA GTE, Fashion MNIST, and Glove-200), and recall improves in 15 of 18 cases too, including a dramatic +295.7% rescue on Fashion MNIST. Most datasets see gains on both metrics simultaneously, and even the more modest cases still land solidly positive, recall improvements are commonly in the high single digits to double digits, QPS gains follow a similar pattern. Where either metric does dip, the drops are small and contained: the three QPS regressions all stay under 1.5%, and the three recall regressions all stay under 2%.

How to enable auto-calibrated vector quantization in Elasticsearch

The feature is not enabled by default for now, and opt-in via auto_calibrate on bbq_disk index options:

With this set, you no longer need to guess at bits, oversampling, or preconditioning: each segment picks the cheapest configuration that's predicted to hit 90% recall@10 for its own vector distribution, and re-evaluates that choice every time it's merged.

What's next for automatic vector quantization in Elasticsearch

Our first post showed that recall could be predicted in closed form from a small sample. Turning that into something running inside a real merge path meant a second round of engineering decisions that the model itself doesn't answer: how to order a sweep over candidates so it's cheap in the common case, how to price oversampling against document bits given how each is actually paid for at query time, and how to estimate the error term itself cheaply without quietly wrecking its accuracy.

In the end, we have a feature that allows us to tailor indexing choices to the data characteristics, with less than 11% overhead to index time for large indices. This gives us the ability to accurately control recall while optimizing quantization and oversampling choices for query performance. We got an average increase of 16.7% in QPS when we enabled this feature compared to our previous default settings for DiskBBQ. All while reliably achieving our target recall. Taking away the configuration burden from the user actually allows us to make better choices; it is a win-win.

This is the beginning of a longer journey that we’re working on to bring automatic configuration based on a combination of better understanding of the operating environment and better understanding of the data characteristics. We look forward to sharing more of this work with you in the near future.

This is where Elasticsearch earns its place in the stack. It is already where a great deal of operational telemetry lives (logs, metrics, traces, events), and it exposes them through an expressive query and aggregation layer. That makes it a natural tool for an agent debugging a live system: it can slice by attribute, aggregate over time, correlate across signals, and drill from a symptom down to the documents that produced it.

We've been building an agentic layer on top of Elasticsearch that continuously monitors a system and root-causes issues as they arise. A recurring primitive in that workflow is time series event analysis. For example, given an error rate, a p99 latency, a queue depth and a throughput counter, tell me whether something happened, what it was and when. A transient spike in errors, a regime change in latency, and a step up in CPU usage are the signatures of the underlying fault, and they're typically what an agent examines first as it forms and tests hypotheses.

Elasticsearch has shipped a single-change-point aggregation for some time. It answers "did this series change?" with one verdict and the most significant change it found. That's a good fit for a dashboard, but less so for an agent, which often wants to interrogate a long window and enumerate everything of interest in it: the error spike at 02:14, the latency regime shift at 02:30, the throughput dip while the pod was being rescheduled. So we upgraded the capability to detect and report multiple events of multiple kinds in a single series. At the same time, we took the opportunity to further harden it to work reliably against whatever telemetry the agent points it at. This post describes how it works.

Why single change point detection isn't enough for agents

Concretely, we want a single entry point that takes a numeric time series and returns a small list of interesting events, each with a type, a location, a significance, and some key characteristics. We care about three classes of event, because they map onto three different kinds of underlying fault:

Event type

What changes

Detection channel

Example fault

Structural change

Level (step) or slope (trend) shifts to a new sustained regime

Value channel

Config push doubles baseline latency; memory leak turns a flat curve into a ramp

Distribution change

Noise level (variance) shifts while the mean holds steady

Dispersion channel

Service responds erratically at the same average latency

Point anomaly

Isolated spike or dip against a stable background

Value channel (pulse detector)

Single burst of errors; one-minute throughput drop during GC pause

The hard part is not detecting any one of these on clean, well-behaved data. The hard part is doing it on arbitrary telemetry without per-series tuning. The agent does not know in advance whether the series it is examining is near-constant, smoothly drifting, heteroscedastic (quiet in places and noisy in others), sparsely populated, or has a magnitude of . It is not scalable to hand-pick parameters for every series it needs to analyze. Whatever we build has to be robust to all of that while maintaining excellent recall and precision. If it fails to detect important events it runs the risk of missing key corroborating evidence for a working hypothesis. Conversely, an analysis tool that reports an event for every minor fluctuation will pollute the context the agent reasons over.

The design goals, in priority order, are: correct on diverse data out of the box, parsimonious (report only what matters), and cheap enough to run interactively across many series.

How PELT and BIC power change point detection

Change-point detection is a well studied field. The classical offline formulation searches for the segmentation of a series that minimizes a penalized cost: a per-segment goodness-of-fit term plus a penalty for each added break to stop the optimizer from putting a boundary between every pair of points. Solved naively, this is combinatorial, but PELT (Pruned Exact Linear Time, Killick et al.) finds the optimal partition in roughly linear time by using a dynamic program to prune candidate boundaries that can never be optimal. On the labeling side, comparing nested models by an information criterion such as the Bayesian Information Criterion (BIC) gives a principled, scale-aware way to decide whether a candidate break is really important and what sort of change it constitutes.

These are good building blocks, and we use them. However, the textbook recipe assumes more than telemetry gives you. It typically assumes a single change type (a mean shift), a known and stationary noise level, and reasonably benign numerics. Real telemetry violates all three: variance changes matter as much as mean changes, the noise level is unknown, often heavy-tailed and changing, and the data spans extreme magnitudes and degenerate cases, such as perfectly constant segments, that wreck an ill-conditioned polynomial fit or a fixed-variance cost. Most of the engineering I describe below is about closing that gap.

Splitting one time series into three detection channels

Rather than trying to find one detector that does everything, we run three focused detectors and then merge their findings. Two of the three are the same structural detector applied to two different views of the data, or channels.

Two detectors run on the value channel: a structural detector that flags step and trend changes, and a pulse detector that identifies point spikes and dips. The dispersion channel, a windowed measure of spread, is fed to a second copy of the structural detector, where a variance change shows up as a level shift and is relabeled a distribution change. The two channels are complementary by construction: a step is a level shift the value channel flags but only has a single large first difference the dispersion channel ignores, while a variance change is invisible to the value channel yet shows up clearly in the dispersion channel. A thin orchestration layer then merges and de-duplicates the streams.

Keeping the three concerns separate makes each one tractable. A mean-shift detector and a variance-shift detector pull in opposite directions if you try to fuse them; a point-anomaly detector and a regime detector need opposite robustness settings. Separated, each can be tuned to its job.

PELT with a scale-free cost

For the structural channel, we fit each candidate segment with a low-order polynomial (constant or linear) and score it with the profiled-variance Gaussian cost. If a segment of length has residual sum of squares , the cost is

This is the negative log-likelihood of a Gaussian segment after profiling out the variance, i.e., after substituting the maximum-likelihood estimate back into the log-likelihood. The total objective PELT minimizes is the sum of segment costs plus a per-break penalty,

Here, is the BIC complexity term (number of parameters times where is the number of values in the time series) and the scale factor lets us trade sensitivity against parsimony in one place.

Using the profiled cost rather than a cost against a fixed global variance is a deliberate and important choice. The global noise level of telemetry is unreliable: on a smoothly varying series the natural estimate (the spread of first differences) can collapse toward zero, and a fixed-variance cost then treats every wiggle as enormously significant and over-segments. The profiled cost depends only on the ratio , so it is invariant to the absolute scale and immune to that failure. A small floor on keeps the logarithm finite, so a zero-residual segment is not rewarded without bound.

Keeping the fit stable using robust local weighting

Before PELT runs, we robustly down-weight points so that an excursion does not create spurious breaks or drag a segment boundary onto itself. Crucially, these weights enter into the weighted residual moments, so they shape not just the segment fit but its residual variance, and so the segment costs themselves. Each point gets a Cauchy weight, , of its residual from a rolling-median baseline against a robust scale : points near the local median keep full weight and points far from it are progressively discounted. A nice side effect of measuring excursions from the median on a window centered on each point is that clean structural breaks do not get down-weighted at all, because the majority of values land on the same side of the break as the point whose residual is being computed. So a sustained regime keeps full weight, but a lone spike does not.

There is a nice justification for scoring a weighted Gaussian cost when what we really want is robustness to a heavy tail. The Cauchy weight is exactly the iteratively reweighted least squares (IRLS) weight of its loss: , so the weighted normal equations are identical to the Cauchy M-estimator's estimating equations . A weighted-mean (or weighted-line) fit at those weights is therefore a Cauchy M-estimate, not a Gaussian one. The cost we actually evaluate inherits the same properties. Because is concave in , its tangent at the current residual lies above it. This gives a pointwise bound with the weight at the tangent point; summing, the weighted residual sum of squares is a tangent upper bound on the total Cauchy loss, touching it in both value and gradient at the weights' anchor point. Minimizing the weighted RSS is thus one step of a majorize–minimize scheme that provably decreases the true Cauchy objective, and the profiled-variance cost we feed the BIC is that majorizer standing in for the Cauchy deviance. The only approximation is that we anchor the weights once, at the rolling-median baseline, rather than iterating IRLS to its fixed point; this is exact for the inliers that sit near the baseline, and correct for gross outliers, whose vanishing weight removes them from the cost wherever the bound is loosest.

Finally, the trick that makes this work on heteroscedastic data is that the residual is judged primarily against a local robust scale, not a global one: the MAD of residuals in the same sliding window. On a series that is quiet in one stretch and noisy in another, this means a spike in the quiet stretch that is multiple local sigmas is correctly suppressed. The local MAD can collapse on quiet stretches, so we use a backstop that is a fraction of a global composite of robust scales and a floor related to the quantization error for discrete series and numerical precision otherwise.

From candidates to labeled events using BIC verification

PELT gives a globally optimal penalized segmentation, so we take its boundaries as candidates and verify each one. For a candidate at index we look at the window of length spanning to its nearest neighboring candidates and compare a no-change null against step and trend alternatives by BIC,

where counts the fitted parameters (the same parameter count as in the PELT penalty above). We map the BIC gain of an alternative over the null to a significance via , to turn a threshold into a decision boundary). We treat this as a significance score for ranking and thresholding, not as a calibrated tail probability.

At this stage we allow higher-degree models to avoid splitting smoothly varying trends. These are problematic in PELT itself because it considers short segments, which they overfit. We keep the most parsimonious alternative that clears the significance threshold and survives a persistence check. The persistence check re-scores with the weights immediately around the candidate muted, and if the evidence collapses, the "change" was driven by a few extreme points – an excursion, not a regime change – and we reject it. The polynomial order is applied symmetrically to the null and to each side of the split, so the alternative is always the same model class merely split at the candidate, and therefore strictly more flexible. The whole process can be thought of as Bayesian model selection with a preference for the null.

When no candidate survives, we still say something useful: we report the series as "stationary" (best no-change model is a constant) or "non-stationary" (best model has a slope), with the trend direction. For an agent, "this series is cleanly trending up over the window" is itself a finding.

Detecting distribution changes with a dispersion channel

A variance change is indirectly visible to the mean channel – worse, the robust weighting there actively mutes the excursions that signal it. So we detect it on a separate dispersion channel and reuse the exact same structural detector, because on this channel a variance change is just an ordinary level change.

The channel is built from one sample per non-overlapping window. Within a window, we take the inter-quartile range of the first differences, rescaled to a standard-deviation equivalent (), and pass it through :

Then . Three choices matter here. First-differencing cancels level and slope, so a mean step contributes a single large difference rather than inflating the whole window, and a steady ramp produces a flat channel. Non-overlapping windows keep the samples independent; overlapping windows autocorrelates the channel and makes the segmenter over-detect. And the IQR is used rather than the median (which is too robust and will miss a window that is 40% noisy then flatlines) or the raw standard deviation (which is not robust enough since one spike's two large differences inflate the window). Because the dispersion channel is a fraction of the original length, the verifier there is restricted to a lower-order null so a genuine low-high-low variance bump is not absorbed.

The functional form is worth dwelling on, because each part earns its place. The log makes the channel respond to ratios of noise level rather than absolute differences. Variance changes in telemetry are typically multiplicative: a regime is "twice as noisy". On a raw-scale channel, a doubling shows up as an enormous absolute jump at a high baseline and a negligible one at a low baseline, so an additive step-cost detector would find variance changes trivially in loud series and miss them in quiet ones. Under a log, a factor- change in scale is the same offset wherever it occurs, which is exactly the additive-step behavior the structural detector is built for. The " is a soft floor. A bare diverges to as the scale goes to zero, which is precisely what happens on a near constant stretch, and would manufacture a huge spurious step at the first noisy window after it. Conversely, is finite and smooth at zero, behaves linearly () while the noise is small, and recovers the multiplicative behavior once the noise is appreciable. This gives graceful degradation instead of a singularity, and with no tuned epsilon to pick. Note that squaring the scale (using a variance instead) only doubles the dynamic range; it makes no difference to the detector either way, since differs only by a constant the threshold absorbs.

Detecting point anomalies as excursions from a local baseline

Spikes and dips are detected as point excursions from the local rolling-median baseline. Working from the local residual rather than raw values means level structure is removed and smooth curvature is tracked; even for time series that change significantly the detector is sensitive to significant local deviations.

The pipeline is a generous proposer followed by a strict gate:

1. Propose every point whose residual exceeds a threshold number of robust sigmas. The scale is the larger of the global first-difference noise (which stays meaningful on smooth data where most residuals are exactly zero) and a composite of robust scales of the residuals (which inflates once a frequent large-residual population appears).

2. Merge adjacent same-sign candidates into excursions, dropping any that span a full minimum segment: that is a regime, and is owned by the structural channels.

3. Rank and cap the excursions by peak z-score, keeping the top , so a pathological series cannot drown the output.

4. Gate using one shared null: build a Gaussian KDE from the series with all of the retained excursions removed, and keep an excursion only if its peak's Bonferroni-corrected upper-/lower-tail probability under that null clears the threshold.

Removing all the tested excursions from the single null at once is a key trick. The leave-one-out alternative – score each excursion against a null containing the others – lets the largest spike and dip mask everything else. Removing them together means several genuinely distinct excursions are each judged against the remainder and all survive, while a recurring population is still rejected.

The proposer and the gate ask deliberately different questions, and that distinction drives two further choices. The proposer works on residuals from the rolling median since it wants recall, and a residual is what tells you a point stands out from its local neighborhood. The gate is value-based: it asks "is this magnitude one we see at other times in the series?", so a spike to a level that recurs elsewhere — such as periodic batch jobs — is suppressed even though it is a large local residual. Those are the right semantics for an agent, but they expose a heteroscedasticity problem, because telemetry noise is almost always a function of magnitude. Periodic spikes can sit orders of magnitude above the background, and a single KDE bandwidth fitted to the whole value range is then far too narrow up in the high tail. So ordinary large values come back as significant, a steady source of false positives.

The fix is a variance-stabilizing transform. We run the value gate in space, where is a robust measure of the spread of the background. is linear for and logarithmic for , which turns a multiplicative (magnitude-dependent) spread into a roughly constant one, so a single bandwidth is valid across orders of magnitude. It is also odd and finite at zero, so exact zeros and sign changes (dips below a small baseline) need no special handling, unlike a bare log. Crucially, it is monotone and so does not change what is tested (for any monotone function , ) so it only fixes the estimate of that tail probability.

One subtlety closes the loop. The KDE null and the kernel bandwidth are taken from different scales, on purpose. The null is the stabilized background values: so it models any mode in the data, which is what makes a recurring large magnitude unsurprising. However, the bandwidth is taken from the stabilized residuals, not the stabilized values, because a genuine level change makes the value distribution bimodal, and a bandwidth computed from that bimodal spread would balloon, masking a real spike sitting on top of a shifted regime. The residual removes the step, so the bandwidth always reflects within-regime noise and the gate stays sensitive to a deviation that is extreme relative to its own neighborhood if it is also outside the envelope for the series as a whole.

Merging structural, distribution, and point anomaly events

Finally, an orchestration layer merges the structural, distribution, and point-anomaly event streams. Structural and distribution events that mark the same regime boundary are de-duplicated to the more significant one (a boundary that shifts both level and spread is one event, not two). Pulses are a separate stream added after de-duplication, because a spike that lands on a structural boundary is a real, separate finding and must not be suppressed. Everything is then mapped back from the internal value-array index space to source-bucket indices.

Handling extreme magnitudes and edge cases

What makes this usable as an unattended tool is a collection of defensive choices for the cases that break naive implementations:

• Variance computed as loses all precision at large magnitudes: a constant series at can manufacture phantom change points purely from floating-point error. We center every PELT input by a constant offset first; the polynomial RSS is invariant to that shift in exact arithmetic, but the working magnitudes drop from to .

• Using raw indices as the regressor results in poor condition polynomial fits: the largest moment is , which is about for a cubic over a 2000-point window. This trips the SVD singularity guard and silently degrades the fit. Mapping affinely onto leaves the fit identical (RSS is invariant under reparametrization) but every moment becomes .

• Scale-free cost, as described, means the segmentation cost doesn't depend on tuning a noise estimate.

• Down-weighting wants a primarily local scale: suppress whatever is anomalous in its own neighbourhood. It uses the maximum of MAD and a small global floor on the differences from the rolling median. Conversely, spike/dip detection wants a global scale: we care about global outliers. It uses a composite of robust scales of all differences. Using the wrong one in either place produces characteristic failures – irrelevant spikes in a quiet segment, or locally large excursions creating spurious breaks – and maintaining separate channels allows us to pick appropriately.

• Using one p-value threshold, Bonferroni-corrected by the number of candidates, applied consistently across all three detectors, means that "how surprised should I be" is consistent everywhere.

The recurring theme is that the difference between a detector that works in a notebook and one that works on a firehose of production time series is mainly in handling the edge cases gracefully.

Performance: ~1ms per series on a single core

Detection cost is dominated by PELT. Its segment cost is a profiled-variance linear fit, which we evaluate in constant time from prefix-summed weighted moments rather than maintaining a regression per candidate boundary, so a single segment cost is a handful of array reads and a 2×2 solve. Cost grows a little faster than linearly with series length since PELT's pruned candidate set does not stay constant on noisy data. Therefore, to bound the worst case on very long series, we downsample ahead of detection: above a cap (2000 samples), the series is collapsed into macro-buckets, keeping two samples per bucket: the median and the largest local deviation. This is inspired by the M4 downsampling scheme, but because we need only the median and the largest excursion for structural-change and outlier detection, respectively, we can then afford to double the bucket resolution. The downsampled series carries its original bucket indices, so every reported event still maps back to a real source bucket; below the cap it is a no-op. The whole analysis is a single pass over the (possibly downsampled) series with no per-series configuration. This lets the agent call it freely across many signals.

In absolute terms, this means the analysis is comfortably interactive. On a single core, post-warmup, a typical series of 140–350 buckets is analysed in about 1 ms (≈220,000 buckets/s), and a series long enough to hit the downsample cap (say 5,000 buckets, collapsed to 2,000) takes about 40 ms, which is the effective worst case per call. For an agent issuing a handful of these calls per investigation, and parallelising across the many series in a `BY` query, the latency is negligible.

Evaluation on synthetic and production telemetry

Synthetic benchmark

We evaluate first on a synthetic generator with known ground truth, because it lets us measure the things that matter precisely. The generator produces random time series with diverse behaviors, which we group into three families. The positive family injects known events of each type: clean and noisy step changes (up and down, SNR around 10, at several positions), trend onsets and ramps (including a flat–ramp–flat sequence with two boundaries), variance changes with a constant mean (single steps and a low–high–low bump), and isolated spikes and dips. The null family ideally produces no event: stationary noise, perfectly constant series, smooth quadratic drift and clean ramps, and periodic signals. (A variance change with a constant mean is not null: it is a distribution change, an abrupt step on the dispersion channel, and so it belongs in the positive family above. The related null requirement, that such a change does not surface on the value channel as a step, is checked separately.) The adversarial family stresses the robustness machinery: a perfectly flat series at a magnitude of (for which a naive variance arithmetic manufactures phantom breaks here through catastrophic cancellation) and, conversely, a genuine spike or step in a noisy baseline as high as (which must still be found and located, the high baseline notwithstanding); a spike on top of a step; a within-regime spike after a 100 level jump; a recurring train of equal peaks (a population, not individual spikes); wide sustained excursions (a structural change, not a spike or dip); and fuzzed random level-shift series. On these series we track recall per event type, precision and the false-positive rate on the null family, localization error, parsimony (events per series and adherence to the count limit), and invariance under constant offsets, rescaling and extreme magnitudes.

The table below summarizes what the suite tests.

Event family

Representative scenarios

Required outcome

Localization tolerance

Step

clean / noisy, up / down, single and multiple, several positions

detected

≤ 4–8 buckets

Trend

slope change and flat–ramp–flat, clean and in noise

detected

≤ 8–12 buckets

Distribution

variance step (mean constant), low–high–low bump

detected

≤ 1 dispersion window

Spike / dip

isolated, multiple distinct, on heavy-tailed and high-magnitude series, within-regime after a step

detected, capped at max(5, 2% of n)

≤ 2 buckets

Null series

stationary noise, constant, smooth drift / ramp, periodic

no event reported

n/a

Invariance / robustness

constant offset, rescaling, 10^5–10^9 magnitudes, spike-on-step, recurring population, wide excursion

result unchanged / no spurious event

n/a

Running this over 400 series per scenario, just over half a million buckets in total, and scoring with the same two categories we use for the real-data evaluation below (any regime change versus point spikes and dips) gives:

Event type

Recall

Precision

Median localization error

Structural change

0.994

0.664

0

Spike / dip

0.847

0.746

0

Two things stand out. When an event is detected, it is placed essentially exactly: the median localization error is zero buckets for both categories; and the point-wise accuracy, 0.998, is directly comparable to the 0.995 we report on real telemetry below: the overwhelming majority of buckets are correctly left unmarked.

The precision figures are lower than on real data, and understandably so. A sixth of this population is a hostile null family (periodic signals, smooth drift, clean ramps) chosen precisely because they tempt a detector into a spurious break, and every false alarm on them counts against precision. The resulting false-positive rate is 0.5% per bucket, with 15% of null series carrying at least one spurious event. We treat this as the conservative end of the range: on the real-telemetry mix below, where the null series are less adversarial, precision rises to 0.85 (structural) and 0.90 (spikes/dips). Finally, offset and scale invariance holds on all 400 series: the same events, to within a few buckets, whether the series is shifted by a constant or rescaled by up to three orders of magnitude.

That raw precision also misses how the result is consumed. The agent reads events most-significant-first, so a false positive only does harm if it outranks a genuine one, and by and large it does not. The median p-value of a true positive is about , against about for a false positive: the genuine events are typically overwhelmingly more significant. Concretely, if we keep only the top events per series by significance (with the true count) precision rises to 0.94, and a randomly chosen true positive is more significant than a randomly chosen false positive 93% of the time. It is not a perfectly clean separation: a strong periodicity or a sharp curve genuinely can contain a significant-looking break, which is why that figure is 0.94 rather than 1. However, the ranking is reliable enough that an agent reading from the top, or applying a stricter significance cut-off, sees the real events first and the false alarms as a lower-significance tail. This is also why exposing the detector's significance to the agent (see What's next) matters more than squeezing the raw precision higher.

Real cloud telemetry

To evaluate on real data, we scraped around 300 metrics from our production cloud environment. These cover HTTP status-code counts, failed memory allocations, memory usage, network usage, page faults, CPU usage, and throttling metrics, measured both per instance and aggregated across the fleet as a whole. Their values range over more than 12 orders of magnitude, and they display a variety of behaviors including ramps, periodicity, step changes, distribution changes, and trend changes. The figure below shows a sample of series together with the detections we make on them.

To get a sense of the accuracy on this set data, we labeled a subset of 150 most interesting time series, marking the visually clearest features in each. This labeling is not necessarily optimized for our target use case, where false negatives are typically more problematic than false positives: an agent consumes these results as part of a broader investigation and can pull additional information to corroborate them. Even so, we find excellent agreement with human judgment on these series. Since each series comprises between 140 and 350 points, the point-wise accuracy, at 0.995, is extremely high: the great majority of points are correctly identified as neither a change, a spike, nor a dip. But the more telling metrics are recall and precision on the human-labeled points, shown in the table below. The human labels did not attempt to categorize each change, so we break the results down only into "structural changes" and "spikes / dips" — the same two categories, and the same point-wise accuracy and recall/precision metrics, as the synthetic benchmark above.

Event type

Recall

Precision

Structural change

0.94

0.89

Spike / dip

0.97

0.92

It is worth covering exactly why we get disagreements. These largely fall into three categories: spikes and dips in context, isolated breaks, and small-magnitude breaks in stable series. We deliberately do not try to detect spikes and dips that are unusual only in their immediate context; that is, not globally unusual but visually significant relative to an inferred periodicity in the data, for example. Trying to account for these without fully modeling the seasonality in the data hurt precision more than it helped recall, and we have a separate persistent anomaly-detection process that builds more complete models of baseline behavior over time. Isolated breaks are an artifact we decided to live with: PELT's cost function tends to isolate a change point with a few values intermediate between the two regimes, because absorbing it into either neighboring span inflates that span's cost. Humans are good at judging such situations visually and assign a single change point. Finally, small-magnitude changes are simply not visually obvious. We detect them deliberately and regard this as a strength of a quantitative approach, since they are often the early precursors of an incident whose later, larger effects drown them out.

How the agent uses change point results in ES|QL

To the agent, all of this is one tool call: it points it at a collection of time series and gets back a typed, located, ranked list of events. That list is small by construction, which means it drops cleanly into the model's context without crowding out everything else it is reasoning about. Because the result contains multiple events, a single call over a window can hand the agent the whole local story – "error spike at 02:14, latency regime change at 02:30, throughput dip at 02:31" – and let it correlate across signals to a root cause.

Operationally, we expose this through both the ES|QL CHANGE_POINT command and the change_point aggregation. Currently, we have not extended the change_point aggregation to return multiple change points since it breaks backwards compatibility of the output schema. It just returns the most significant event. We don't have the same restriction for ES|QL since it returns change points annotated onto the table rows to which they apply. We do plan to revisit the output schema for both ES|QL and the aggregation in a later version. We'd like to migrate to optionally returning significance in log-space, which doesn't underflow, and including a short verbal description of each change, which we expect to help agents when seeing just the change points themselves.

ES|QL is Elasticsearch's piped query language, and CHANGE_POINT runs the detector as one stage in a pipeline. Its BY clause enables it to analyze many series at once (one per group) so the agent can, in a single query, segment every service's latency or every host's error rate side by side rather than issuing a call per series. The actual leverage, compared to the change_point aggregation, is composability: the events come back as ordinary rows in the pipeline, so the agent then has the entire ES|QL language to manipulate them downstream. It can filter to a window, join change points against deploy markers, count events per service, rank by significance, feed the survivors into a further aggregation, and so on.

For example, suppose the agent wants to find out whether any servers have recently seen a sudden CPU spike or a prolonged step change in CPU usage over the last 12 hours, and whether that might point to a load-balancing issue. It could use the following query:

Here it's using a STATS ... BY host.pod to see how the detected events cluster across other dimensions of the data, such as the Kubernetes pod, and so judge whether they share a common cause.

What's next for ES|QL time series analysis

As far as detecting events of interest in time series, the foundation is in place: a single, robust, parsimonious tool that turns a raw telemetry series into the short list of events that actually matter, which is exactly the kind of reach an agentic SRE needs. Going forward, we plan to explore the best mechanism for feeding the detector's uncertainty to the agent, so that a borderline event can be flagged as "worth a second look" rather than silently included or dropped. Also, this is the first of several analytical tools we plan to build into the ES|QL query language to enable agents to triage and RCA issues more effectively; so stay tuned for further updates.

A vector store that achieves good performance without tuning is more valuable than one that requires expert tuning. In fact, our contention is a data store that can be coaxed to exceptional performance by an expert who spends a week hand-tuning it is less useful than one that beats it consistently out of the box. In other words, easily achieving good performance is a first class property, not a nice to have. We can see this clearly in our telemetry. The great majority of users will never tune the internals of vector search at all, and why should they: it is just an enabler for what they're trying to build.

This is the imperative behind features like auto-calibration. The system as a whole should look at your data and your quality target and choose good parameters for you. Indeed we think this is a win-win, since it has far more nuanced information available to it to make these choices than we expose.

To make "good performance" precise, it helps to name the three attributes that characterize any vector search system, because they trade off against one another and you can't talk about one without fixing the others:

1. Performance: throughput (QPS), latency, and so on.

2. Hardware cost: a fair comparison always holds cost fixed. It's trivial to buy your way to more QPS or better recall by throwing hardware at the problem; the interesting question is what you achieve per dollar.

3. Search quality: recall, nDCG, and related measures of whether you're returning the right results.

The three form a frontier. Push one and, at fixed budget, you pay in another. Any honest comparison of approaches pins two down and measures the third. What we describe in this post is the mechanism we're introducing to pick quantization parameters for a fixed recall budget. It is a step on a longer journey towards a vector store that configures itself well across the board.

Why recall is the right quality metric for vector search

Search quality is tricky, because the "right" results depend on relevance labels you usually don't have at index time. So we lean on recall as a safe proxy. The argument is simple: recall measures how well the approximate index reproduces the results of exact search over the same embeddings. If recall is high, you have not degraded search quality relative to what the underlying model can do; you can be confident you’ve faithfully preserved the baseline. You might still wish for a better embedding model, we've got you covered, but that's a separate concern from the index not damaging what the model already gives you.

This is why controlling recall matters so much, and why you should be wary of any system that doesn't reliably control it. If a vendor can't control recall, they can silently degrade your search experience, achieving impressive QPS numbers while quietly returning worse results, and you'd have no way to know without a labeled evaluation set. The method in this post is about maximizing performance while keeping a firm, predictable grip on quality.

Why vector quantization parameters must be chosen at index time

What makes the problem genuinely hard is that vectors are quantized as they are indexed, so the parameters that govern quality (how many bits, how deep to rerank, whether to precondition) have to be evaluated before we've seen the data laid out in its final form. We can't index everything, measure recall, and iterate; by then the quantization is baked in.

So we need to estimate what we'll need from a small sample, cheaply and in advance. Fortunately the Elasticsearch gives us natural moments to do this: segment merges are exactly such an opportunity. When segments are combined we have to rewrite the data anyway and can assess the data and (re)choose parameters. And as we'll see, models fit to small random samples give excellent estimates of the quantities we actually need to control. They’re typically good enough to set parameters once, with a small margin, and trust them as the index grows.

How vector quantization affects nearest-neighbor recall

With that motivation in place, let's start to dig into the details.

Vector quantization is a critical component for making approximate nearest-neighbor (ANN) search affordable at scale; it's an area we've innovated in the past. Instead of storing and comparing full-precision embeddings, we store a lossy, compressed representation and search over that. The catch is the one above: lossy representations move distances around, so the "nearest" neighbors under quantized distances are not always the true nearest neighbors and recall suffers.

The standard fix is to over-retrieve and rerank. We use the cheap quantized distances to pull back the top candidates, then recompute exact distances for those and keep the best . As long as the true top- are present somewhere in the retrieved top-, reranking recovers them exactly.

Reranking isn’t free, we have to fetch high precision vectors from disk. However, we can precisely characterize the performance of reranking based on hardware characteristics alone. This reframes the whole problem. The question is no longer "how much does quantization distort distances?" in the abstract, but something which relates back to the attributes we care about:

This post derives a model that answers exactly that. The core of it is a single, surprisingly clean idea: if we can characterize the distribution of distances to the -th nearest neighbor, and we have a model of the quantization error distribution, then we can compute expected recall after reranking in closed form (up to a one-dimensional integral). Everything else – bit counts, rerank depth, whether to precondition – becomes a search over a model we can fit cheaply from a small sample, instead of an expensive empirical sweep over full indices built with those parameters.

We build it up to this in three stages: the geometry of nearest-neighbor distances, the scaling law that falls out of it, and then the recall model that ties quantization error to recall given a reranking budget. Be warned, the following gets a little bit involved, but to give you intuition about what is happening see the video below.

Quantization error vs. the nearest-neighbor distance gap

Fix a query and rank the database vectors by their true distance to it: , so is the distance to the -th nearest neighbor. Reranking the top succeeds for the true -th neighbor whenever it is not pushed past rank by quantization noise.

Two competing quantities govern this:

• The quantization error that is essentially fixed for a given scheme and dataset: it depends on the embedding dimension, the vector distribution, and the number of bits, but not on how big the index is.

• The criticality gap , which is the distance between the -th and the -th nearest neighbor. This is the margin we have to absorb error. Crucially, it shrinks as the index grows: pack more vectors into the same region and neighbors crowd together.

There’s a detail here we’ll gloss over for the sake of presentation: for IVF style indices, we’re quantizing the residual from a cluster’s centroid. This does in fact couple the quantization error to the index size, but we can handle it much the same way we handle the distance to the -th nearest neighbor.

For reranking to recover the recall lost to quantization, we need the error to only rarely exceed the gap. If we can write down the distribution of and the distribution of the error, we can make that statement quantitative. The first job is to estimate the distribution of nearest-neighbor distances.

Deriving the nearest-neighbor distance distribution

Real embeddings don't fill their ambient space; they concentrate on a lower-dimensional manifold. Near a query, though, we can make a mild local assumption: in a small neighborhood around the query, the data density is roughly uniform. Here is the intrinsic dimension of the manifold; it is unknown and generally far smaller than the embedding dimension. How to estimate it is the subject of Section 4.

Let be the vectors falling in , modeled as i.i.d. uniform on , and define the distance from to its nearest neighbor:

To get the distribution of we use the standard order-statistics trick: rather than ask where the minimum is, ask for the probability it exceeds some radius . The event is exactly the event that every point lands outside the -ball centered on the query .

A single point lands inside with probability equal to the ratio of the ball's volume to the region's volume

where is the volume of the unit -ball. (We assume is large enough that the relevant is small, so the ball doesn't spill outside and boundary effects are negligible.) Because the points positions are assumed to be independent, the survival function is

What we're really interested in is how R behaves on average. To compute this, we use the identity that the expectation of a non-negative random variable is the integral of its survival function, . Evaluating this with (2) gives the headline result:

(The exact integral carries an extra factor; it's an constant that we can fold into a fitted coefficient later, so we drop it here.)

Glacial scaling: why neighbor distances barely change as your index grows

It is interesting to consider what this formula tells us about how distances change with dataset size: . The exponent is , and in high intrinsic dimensions that is a very small number. This is a property the method leans on, so it's worth plugging in some numbers:

• If then doubling multiplies by , so distances drop by ~30%.

• If then doubling multiplies by , so distances drop by a little over 1%.

In high dimensions, neighbor distances barely move even if you add a lot of data; call it glacial scaling. It's the reason we can choose quantization parameters once from a tiny sample, with a small safety margin, and trust them to remain valid even after the index grows substantially before the next re-quantization.

Expected distance to the k-th neighbor and the criticality gap

We actually care about the whole sequence of order statistics , , not just the minimum. There's a simple way to get them.

Map each radius to the cumulative volume it encloses by defining

By (1), each is exactly the probability of landing within radius , so the are uniform on . The order statistics of uniforms are textbook: the -th smallest of uniforms follows a Beta distribution,

Inverting the volume map, , gives the scaling of the -th neighbor distance:

That's all we need for the expected gap:

The last form is the intuitive one: the gap between the -th and -th neighbors is the distance to the -th neighbor, scaled by . Widening the rerank depth relative to opens the gap; higher intrinsic dimension closes it (the exponent pushes toward 1).

Why the expected gap is sufficient to predict recall

Working with an expectation is only legitimate if the gap doesn't fluctuate wildly around it. It doesn't because concentration of measure saves us. Applying the delta method to and using from the Beta distribution, a little algebra gives

So the coefficient of variation is about . For any reasonable intrinsic dimension this is negligible, which justifies modeling only the expected distances. (If you're worried about the delta method approximation, you can check the results numerically: the delta-method variance and the resulting coefficient of variation match the exact expressions to several significant figures.)

Extending the model to cosine similarity and inner product search

The derivation is for the Euclidean metric, but the other common metrics reduce to it:

• For cosine similarity, the equidistant surface is the intersection of a sphere around the query with the unit sphere. This is called a hyperspherical cap, whose volume scales as for small . Therefore, the analysis carries over unchanged up to constants, with the dimension reduced by one.

• For MIPS (maximum inner product), some extra care is needed, because nearest neighbors aren't confined to a compact region. A distant vector can still win on inner product if its norm is large enough, so the gap is really governed by the tail of the norm distribution. However, there is a clean fix, which is to use the Neyshabur–Srebro transformation. This lifts vectors onto a unit hypersphere in dimensions. After this operation, it's just the cosine case.

Fitting intrinsic dimension and scale from a small sample

Equation (3) has a known functional form but two unknown parameters: the intrinsic dimension and the scale . Both are easy to fit, and it's more convenient to fit them from raw neighbor distances than from gaps directly.

Sample several subsets of database vectors of sizes and a set of query vectors . For each query and each subset, measure , the distance to the -th nearest neighbor of within . Taking logs of the scaling law linearises it:

Specifically, this is linear in and , so ordinary least squares recovers and . Varying the subset size is what makes it possible to estimate : it's precisely the rate at which distances shrink with data volume. With the fitted parameters, the whole-index expected gap is

Figure 1 shows how well this fits in practice (and it’s remarkably good): predicted versus actual average distance to the -th neighbor, across a range of datasets and metrics, have between 0.996 and 0.999.

Modeling vector quantization error as Gaussian

With the nearest-neighbor distance model established, the second component is the quantization error distribution. For every metric we use, the quantized distance estimate differs from the true distance by an error that is a sum of many independent per-dimension contributions. By the Central Limit Theorem that sum tends to Gaussian, so we model the error as normal with a variance we estimate empirically:

where is the quantized distance estimate using -bit vectors and is the total number of (query, neighbor) pairs in our sample set. In other words: sample, quantize, measure the squared distance errors, average.

Figure 2 shows the empirical basis for the Gaussian assumption: measured quantization error densities against best-fit Gaussians across a variety of datasets. The fit is good, which is what lets the rest of the model stay analytic.

We could stop here and take a minimax view: threshold the probability that the -th and -th neighbors swap, using the expected gap (4) against the error scale . But that controls a worst-case event, and what we actually want to control is average recall. The outcome would be overly conservative quantization parameters and we'd pay some performance. The next section estimates expected recall properly.

Predicting expected recall after reranking

Combining the distance model and the error model gives a closed-form estimate of expected recall after reranking. Model the noisy distance of the -th true neighbor as a Gaussian centered on its true distance:

The -th neighbor survives reranking, i.e., lands in the retrieved top , if fewer than other vectors have a smaller noisy distance. Condition on and count the competitors closer than :

Then the probability of recalling neighbor integrates over where its own noisy distance lands:

The terms of are independent Bernoullis but not identically distributed, since every neighbor sits at a different true distance , so each has its own probability of intruding on the top- set:

with the standard normal CDF. This makes a Poisson-binomial variable. Since we sum many of them (because ), the Lyapunov CLT applies and we approximate

with the standard Poisson-binomial moments

The survival probability then has a clean closed form:

This is where the two halves of the post so far finally meet. We don't need to know the individual because the manifold scaling law from Section 3 supplies them: . So the moments become explicit sums over ranks, which we truncate at a safe cutoff (say , since distant neighbors contribute negligibly):

Finally, average recall@ given rerank depth sums the per-neighbor recall over the top :

Here is the standard normal density. Each integral is smooth and one-dimensional, so Gauss–Legendre quadrature evaluates it in microseconds. The entire recall prediction for a set of candidate parameters costs a handful of quadrature evaluations, not index build and benchmark run.

Figure 3 validates the end-to-end model: predicted average recall against measured recall across many parameter settings and multiple datasets has .

How the recall model selects vector quantization parameters

With a fast recall predictor available, parameter selection becomes a cheap ordered search. Given a target recall and a rerank budget (typically expressed as a multiple of ), we can find the minimum document and query bit counts, and other knobs, that clear the target. There are a few things to note that are practically important:

1. Glacial scaling gives us some safety because moves so slowly with for even moderate intrinsic dimension. A small margin in the calculation means the chosen parameters stay valid if a lot of vectors are added before parameters are restimated.

2. Small is the worst case if is a fixed multiple of . The gap \mathbb{E}[R_{(k)}]( is smallest for small so if a parameter choice satisfies the recall target at then it will for larger will too.

3. We can treat quantization as a black box because the error model only needs the empirical error variance. This means we can test any configuration, including preconditioning, the same way and we can simply order candidate parameter tuples by increasing index and query cost, and stop at the first choice that hits the target recall. For tuples of (query bits, doc bits, rerank depth, precondition) a sensible search sequence increases query precision first, then document precision , , , , , , , , and each combined (via an outer product ) with rerank depths like and precondition , exiting as soon as the target is met.

Results: auto-selected quantization parameters and recall across datasets

In this section, we discuss the results of the initial experiments on the end-to-end behavior. We’ve made some further refinements as part of the work to fully integrate with Elasticsearch that we discuss in our other post.

The table below shows auto-selected parameters targeting recall 0.97, measured with brute-force search, so the number reflects loss due to quantization alone (64 query clusters, targeting document clusters of size 384, which matches the settings of DiskBBQ).

Dataset

Query bits

Doc bits

Precondition

Depth

Recall

FiQA E5 small

4

2

false

30

0.97

FiQA arctic

2

2

false

30

0.95

FiQA GTE

2

1

true

30

0.98

MNIST

3

1

true

30

0.99

Fashion MNIST

3

1

true

30

0.99

Quora E5 small

2

2

false

30

0.99

Quora arctic

2

1

false

30

0.97

Quora GTE

1

1

false

30

0.98

Dbpedia E5 small

4

2

false

30

0.99

Dbpedia arctic

2

1

false

30

0.94

Dbpedia GTE

2

1

false

30

0.96

Wiki Cohere

2

2

false

30

0.99

Hotpot E5 small

4

2

false

30

0.97

Hotpot GTE

2

1

false

30

0.96

Glove 100

4

2

false

30

0.87

Glove 200

4

2

false

30

0.89

SIFT128

4

4

false

20

0.99

There are a few things worth highlighting:

• The recall is very sensitive to rerank depth. This is why we nearly always end up choosing the maximum depth available: a step up in rerank depth from 20 to 30 is typically what pushes us to hit the recall target for fewer bits and we prefer fewer bits. In the real system, we tuned this behavior based on a more representative reranking cost.

• Glove underperforms partly we approximate the query distribution with random samples from the corpus, but Glove is also less well characterized by the model than the other datasets. A plausible explanation is that the approximately uniform local density assumption from Section 2 is less reliable for Glove embeddings, which would show up as higher recall variance between queries. However, Glove embeddings are not representative of the actual vectors we need to store.

• The FiQA GTE preconditioning choice is a knife-edge case: preconditioning produced only a tiny expected recall improvement, but the prediction sat right at the recall cutoff and allows us to drop the query from 3 to 2 bits. If we'd rather only keep preconditioning where its benefit is clear-cut, we can enforce a minimum uplift threshold. This sort of fine-tuning of the decision logic leaves all the heavy lifting to estimate recall unaffected.

Key takeaways: auto-tuning vector quantization from first principles

We presented a method to pick optimal quantization parameters to achieve a target recall. It rests on two models that compose cleanly:

1. A geometric model of neighbor distances that follows from a local uniform density assumption. We use this to derive the nearest-neighbor distance, the glacial scaling law of the expected distance, and the expected distance profile . We show that fitting and by a simple log-linear regression to average distances in small random samples from the corpus gives an extremely accurate predictive model.

2. A Gaussian quantization error model that is justified by the CLT. Its only parameter is an empirical variance we estimate by comparing quantized and raw vector similarities for a sample of the corpus.

Finally, we show that it is possible to feed the estimated distance model into a Poisson-binomial count of neighbors that intrude on the top- set. Applying the Lyapunov CLT the expected recall@ after reranking to depth falls out as a one-dimensional integral we evaluate by quadrature.

The outcome is an accurate () predictive model of recall as a function of the quantization parameters. Choosing quantization parameters then becomes an ordered search with a predictive model telling us if we’ve hit the recall constraint. And nicely one that also comes with a built-in argument (glacial scaling) for why the chosen parameters remain safe even when estimated from a relatively small fraction of the data.

We’ve built this entire mechanism into Elasticsearch using segment merges as an opportunity to reassess our quantization choices. Aside from the peace of mind this brings (that you’ll achieve good recall whatever vectors you throw at it), it also allows us to chose near optimal parameters from a performance perspective. This closes the loop on our original objective: near optimal performance out of the box, at least as far as quantization goes. We’re pretty excited about the advantages that model based tuning can bring to vector search and look forward to sharing other work we have in this direction in the near future.

How DiskBBQ uses two centroid tiers for asymmetric quantization

DiskBBQ now uses two centroid tiers (fine-grained document centroids and coarser query centroids) so queries are quantized once per parent centroid instead of once per document centroid.

The old mental model is "one centroid does everything for a posting list." The new model splits responsibilities:

• Document centroids (fine-grained): Still used for posting-list structure and document centering.

• Query centroids (coarser): A parent centroid reused across multiple document centroids.

So instead of quantizing the query independently for every document centroid we visit, we quantize per parent centroid and reuse that work across all of its children. Since we were already using two-tier clustering logic as the index size grew, it was a natural fit. We can reuse the work we already do during querying.

These images are a simple representation of our goal: Quantizing per centroid gives us overhead per centroid. Let’s get rid of it!

The goal is to significantly reduce the number of times we actually need to quantize a given query.

The math behind asymmetric BBQ in Elasticsearch

To center the data prior to computing quantized query and document vectors, and , we rewrite the dot product as and expand. We can perform exactly the same operation but using different centroids for the query vector and document vector . Specifically,

As for standard Better Binary Quantization (BBQ), we quantize and in order to estimate the per (document, query) pair component of the dot product. The quantities and are scalars so just two extra additions per dot product we compute. For , we compute naturally when finding the nearest centroid. For , this can be stored with the quantized document vectors, which are just 4 bytes overhead. Below, we’ll discuss how to manage the other term on the fly.

Asymmetric BBQ in DiskBBQ

We cluster the document centroids (using k-means, for example) into clusters, for and the query and document centroid count, respectively. This means there’s a many-to-one mapping from document centroids to query centroids. We’ll denote the document centroids by their index and define this mapping to the query centroids as →.

Since there’s a unique query centroid for each document centroid, we only need to cache one value for per quantized document vector, that is, for each document vector in posting list , we need to cache with the quantized document vector.

When we come to compute the dot products between a query and the document vectors in a cluster, we look up the quantized query vector corresponding to and we compute once and use it to process the whole posting list. The quantization process is significantly more expensive than computing the dot product, so this is a big net win.

The term is estimated using the usual BBQ machinery, that is, these vectors will be quantized and the dot product value estimated from the quantized vectors. Then we can use (1) to compute the final dot product estimate. Notice that this means we only need to quantize the query at most times. Furthermore, we typically visit many centroids from the same parent centroid in a search because they’re close to one another.

Euclidean distance corrections for asymmetric quantization

For Euclidean, we can write and treat the term exactly as above. In fact, there’s a slightly nicer form. Substituting, we have that:

We can rewrite this as follows:

The corrective terms are the norm of query vector minus the document centroid , the norm of the document vector minus the query centroid , and the norm of the difference of query and document centroids. As before, can be stored as a single float with each document.

What changed in DiskBBQ indexing and scoring

At indexing/merge time, centroids can be clustered into parent groups when centroid count is large enough. Posting metadata moved from "centroid ordinal + centroid score" to a shape that explicitly carries query-centroid ordinal and document-centroid score. That decoupling is what lets scoring read documents and query centering from different places. For Euclidean, let’s break it down further by our mathematics above:

<- This is the distance from a “query vector ” to “document centroid ”. We already gather this when we find the nearest centroids during querying. No new work.

<- This is the distance from “document vector ” to “query centroid ”. However, recalling our original quantization work, this can simply replace a previously stored float value. No new storage is required.

<- This is just the distance between query centroid and document centroid . This is just a single extra floating point value per postings list.

The practical change for dot product spaces is even simpler; the only correction value change is being stored instead of .

These changes don’t introduce new computation costs and marginally reduce storage costs because we no longer quantize queries with document centroids. Those raw centroids don’t need to be present with the posting lists.

One cost we did add is a small cache of quantized query values. This is to account for clustering edge cases. For example, it's possible that query is very close to query centroid but not quite as close as . That said, the actual nearest three document centroids could have a relative order: . So, to prevent the query from being quantized twice, we keep a limited cache of the most recent quantized values for a given query.

Here’s a visualization of the situation described above. In the typical iteration scenario, we don’t want to risk unnecessarily quantizing the query against the same query centroid multiple times.

DiskBBQ asymmetric quantization: performance results

The flame graphs below show a before and after comparison. Before, about 20% of the time was spent quantizing queries when we visited each cluster. After our adjustment, it dropped to about 4%.

Of course, the bulk of the cost is still just scoring the vectors in each cluster. But every little bit helps.

Here’s a better view of the full end-to-end performance and recall. The data set was 1 million DBpedia docs encoded with the GTE-Base model. Here, “sec” indicates the number of clusters per secondary (parent) cluster. Note that symmetric quantization is still impacted by the secondary cluster size as it also impacts the two-tier clustering indexing we do already.

However, the impact on our current index structure is still dominated by centroid scoring and scoring vectors in the cluster. Asymmetric quantization removes a frustratingly expensive part of our scoring overhead, but the impact isn’t dramatic given our current structure.

What's next for DiskBBQ quantization

This simple piece of mathematics decouples our query quantization from our document quantization, giving us better storage efficiency and faster queries. This is in Elasticsearch Serverless now and will be in Elastic Stack version 9.4.0.

This now means that query quantization time isn’t a direct concern for future decisions. We can make larger index changes without worrying about the consistent overhead of quantization directly with document centroids.

This was a nerdy one. I hope you survived all the math (and that I copied it all down correctly). It’s always fun to be able to tackle complex problems with simple mathematics, and the results are actually positive in real use cases and data.

In order for you to be able to trust our estimates, we provide error estimates. Furthermore, since there are edge cases in error estimation, we certify when the estimated value and error are trustworthy. In this blog post, we’ll dive into the theory for approximating and estimating the error in such queries, as well as discuss the testing we’ve done.

Background

In order to estimate ES|QL STATS queries efficiently, we make use of a property that’s shared by many statistics: Their estimates computed from a large number of independent samples from a dataset approach their true value. In the case of an index with some field we can think of the true value of a statistic as its value computed for a random variable with uniform discrete distribution on . In the following we denote this quantity ; it can be things like AVG, MEDIAN, and so on. If we make independent draws from , denoted , such that each value is selected with probability , we have independent copies of this random variable. The property we rely on means that a sample statistic value computed from approaches as becomes large. For example, if is the mean of some metric values then as becomes large. Indeed, for many statistics the limiting error distribution is known to be normal. Furthermore, it only depends on the distribution of , the size of the sample and the type of the aggregation . This means supported STATS queries can be approximated with fixed accuracy independent of the index size .

It is easy to pick values at random from a Lucene index: create a filter that takes exponentially distributed jumps through the dataset, where the expected jump size is controlled by the desired sample probability. The AND of this filter and any other Lucene query can be performed extremely efficiently, since AND’ing filter queries is one of the things for which it is well optimized. In our other post, we discussed some real-world query examples to give a sense of the speedup we obtain for different levels of accuracy.

So far, we've only discussed obtaining an estimate of a query. While such a point estimator can be useful, without knowing anything about its error those uses are limited. We found that ES|QL has existing capabilities that make it relatively easy to incorporate cheap, flexible, and accurate error estimation at the same time. We'll discuss this next.

Error estimates

We view providing an accurate understanding of the uncertainty in our estimates as crucial for users to be able to trust the approximation. While having the option to quickly estimate an ES|QL query alone can be useful in certain situations, we wanted to provide a richer feature that allows clients to make intelligent choices. For example, if an approximate query is being used to preview a chart and the error is only a couple of pixels, there’s little point in running another expensive query to redraw it.

The way we've chosen to represent error is by a confidence interval: the -central confidence interval, to be precise. This can be expressed in terms of the cumulative density, , of the statistic being estimated. Specifically, it's the interval which contains the true value of the statistic with probability whose endpoints are and . Confidence interval calculations are surprisingly subtle. There are also important constraints for our use case that make standard approaches undesirable. Next, we’ll take a look in more detail at the motivation and the design for the approach we’ve adopted.

A key requirement of the whole project is to dramatically accelerate expensive analytics queries. It’s therefore vital that the overhead of estimating uncertainty isn’t too large compared to estimating the query result itself. We also want the feature to be as general as possible, but “isolated” within the language. In other words, ES|QL is a flexible language, and we want estimation to work with as much of it as possible. At the same time, we don’t want to introduce a cross-cutting feature that incurs development costs on every new feature we ship.

With these considerations in mind, we chose to estimate confidence intervals by partitioning the sample set and computing the query output on each subsample. This is reminiscent of bootstrap; however, since we ensure that each partition receives a disjoint random subset of the sample data, we know that they comprise true estimates of the statistic distribution. To achieve the best possible estimate of the statistic itself, we still compute its value on the full sample. For example, to estimate the mean and its distribution the process can be expressed as follows:

This introduces a complication to account for the discrepancy between the count of values used to estimate a query statistic and used to sample its distribution. This is a downside; however, there are some significant advantages.

Most of the work in analytic queries resides in computing the aggregate statistics: post-processing after a STATS reduction acts on a far smaller table, and the cost is often relatively small. In this scheme, every row in the input data to the STATS command is processed exactly twice compared to just estimating the statistic. Therefore, roughly speaking we pay a fixed overhead that's the same order of magnitude as the cost of estimating the query in order to estimate its uncertainty. Since we often achieve multiple orders of magnitude speedup on the exact query, this is acceptable.

Because this process uses a plain old table, with extra columns for the distribution samples, we can pass the whole table through any ES|QL pipeline and compute confidence intervals on the final results. For example, if we include EVAL square_avg = avg * avg in the pipeline above, we'd have exactly the same square_avg, square_avg_0, …, square_avg_B-1 extra values. At the end of the pipeline, we have samples from the distribution of the original statistics and all quantities that are computed using them. Therefore, we can apply our standard confidence interval machinery to reduce the table and convert samples into confidence intervals for derived quantities as well. This whole process is essentially transparent to the rest of the ES|QL language, and as we showed above, can be achieved by query rewriting.

The confidence interval calculation

We have independent samples of the statistic distribution . However, they're computed with fewer values than our estimate . We also have a relatively small number of distribution samples, to avoid the count discrepancy being too large, and so we don’t inflate the table too much. We therefore prefer a parametric approach for estimating confidence intervals.

The errors in the statistics for which we support estimation tend to normal distributions in the limit they're computed from many values. So a natural choice, the standard interval, is to estimate the mean and standard deviation from the samples and report the corresponding normal confidence intervals . Here, denotes the standard normal distribution function. For heavy-tailed data and statistical functions that are sensitive to outliers, such as STD_DEV, convergence to normality can be slow, resulting in poorly calibrated intervals.

Briefly, in order to assess the quality of the intervals, one can examine their calibration. Specifically, one computes a quantity called the coverage. For a central confidence interval, it should contain the true statistic value roughly times for trials. In fact, since we seek the central confidence interval, we can make the stronger statement that the true value should be above, or below, the confidence interval endpoints in roughly out trials. The empirical coverage is this fraction computed for a large number of trials. It allows us to compare alternative approaches by simulation. We return to this when we report our test results.

In order to obtain better confidence intervals, we tried a couple of different approaches: the Cornish-Fisher correction of quantiles and an adaptation of bias-corrected accelerated (BCa) confidence intervals. Simulation showed BCa provided more robust calibration across a range of confidences, so this is the approach we selected. The basic idea, which was introduced by Efron, is to assume that there exists a monotonic transformation of the underlying statistic which, when applied to a distribution sample normalizes its distribution:

Here, , and is the standard normal random variable. This is clearly a relaxation of the assumption that the statistic itself is normally distributed, which is used to derive the standard interval. In fact, this family includes many distributions, since is only constrained to be monotonic. (You can think of as a first-order Taylor expansion of the case that the variance is an arbitrary function of the true parameter value. This further relaxes the assumption that the normalizing transformation also stabilizes the variance.) The nice thing about this ansatz is that never needs to be explicitly computed, and there exist standard approaches for estimating the parameters and from the distribution samples.

To handle one simply arranges for the estimate to land at the median of transformed distribution. If we assume the cumulative distribution function in theta space is then , where is the estimated statistic value, and as before is the standard normal distribution function. Typically, is approximated by the empirical distribution function, computed indirectly by bootstrap. However, somewhat surprisingly, extensive simulation showed that we obtained better calibrated intervals using a normal approximation to our sample values, i.e. with and their empirical mean and standard deviation, respectively.

To complete the procedure, one can rearrange (1) to derive quantiles for as follows:

where is the standard normal z-score for quantile . Typically, one uses the inverse empirical cumulative density estimate of to convert quantiles back to a confidence interval. However, because we have a mismatch between the count of values used to compute distribution samples and the query estimate, we need to do some sort of scaling. Exploring options by simulation, we again found it best to use a normal approximation, , where is the number of distribution samples we use. This is just applying the usual scaling of variance by .

Efron showed that in the case is distributed as , i.e. that it depends only on the true value , then the acceleration can be estimated without any knowledge of . In particular, . By assumption, our statistics tend to normal distributions with mean . Since skew is translation and scale invariant, this gives that , i.e. one sixth of the skew of our distribution samples. One thing this glosses over is the dependence of skew, and therefore acceleration, on sample size. We know it tends to zero as the count increases. In fact, skew also asymptotes to zero as and so we also adjust acceleration to be to account for the count mismatch between the samples and estimate .

Although we significantly improve the calibration of confidence intervals by using a better methodology, we still see issues in the case that the underlying distribution has very heavy tails for some of the supported STATS functions. Therefore, we introduce some additional guard rails we discuss next.

Guard rails

To avoid the user having to understand too much about edge cases, we provide additional safeguards that surface when we've been unable to confirm that the distribution samples behave as we expect. This typically happens when the statistic isn’t computed from a sufficient number of values given the metric distribution. It's exacerbated by very skewed metric data and certain aggregation functions, such as the STD_DEV, which are sensitive to outliers.

We have some global constraints on the minimum count of values used to estimate a statistic for which we'll certify it. For example, if any bucket is empty, then we can’t rely on the distribution samples. This is because ES|QL allows mixing approximate statistics, which treat empty buckets differently. For example, consider the following query:

There is no self-contained way of correctly assigning a value to mix for empty buckets, since summing requires that we treat them as zero, in which case we bias our estimate of avg. Alternatively, ignoring empty buckets introduces bias in the sum. There is also a global minimum count of values for which we’ve verified our certification method is sufficiently reliable; this is 10.

We explored a variety of additional tests to certify the results. These were based on both tests of the underlying data distribution, specifically Hill’s estimator, as well as the statistic’s distribution properties. If the true distribution of the statistic is sufficiently normal, then our estimate and confidence interval calculation behaves as we expect: The interval is well calibrated and the interval width is representative of the actual error. Therefore, in the end, we chose to use a test based on the p-value for distribution samples’ skewness and kurtosis versus a normal distribution null hypothesis. To certify a result, we require that the two tail p-values are greater than 0.05 for both tests. As we show below, we found this test was well aligned to our actual needs: to distinguish results for which the estimate and its confidence interval are more and less reliable.

There's a simple trick we can use to boost the accuracy of the accuracy of the test: Create multiple independent distribution samples and use a vote. Given a test to certify results with a failure rate , the distribution of the count of failures for tests is for the case the null hypothesis, that the estimate is trustworthy, is true. For example, for the majority vote assuming and then the significance of the test is , i.e. we fail to certify fewer than 1% of trustworthy results. Note that we can compute multiple trials relatively easily using different seeds for the RANDOM bucket identifier.

This additional check allows us to certify that we trust our estimates and their errors. We surface this information in the approximate query results. When we can’t certify results, they won’t necessarily be inaccurate, but they should be treated with more caution.

Testing

The two main aims of the testing we discuss here were to understand the calibration of the confidence intervals and to see how well they characterize the statistics' estimation errors. The count function is particularly well behaved, its error distribution is binomial, so the majority of our testing focused on metric aggregations. We study smooth distributions but make sure we cover a range of tail behaviors. The presence of outliers is the key factor that reduces the accuracy of estimated statistics. For example, if an outlier isn’t sampled at all, it can significantly affect the value of some statistics.

We explored a range of light-tailed distributions, such as uniform and normal, and skewed and heavy-tailed distributions, such as exponential, log-normal, Cauchy, and Pareto. For each family of distribution, we used multiple parameterizations, focusing primarily on varying the scale parameter. In total, we had 24 distinct data distributions. Figure 1 shows some example sample distributions from this set. Note that we’ve truncated the charts to remove extreme outliers, which are present for both the Cauchy and log-normal distributions.

For each data distribution, we evaluated 14 different sample sizes, ranging from 1000 to 500000. Then, for each sample set, we evaluated AVG, COUNT, MEDIAN_ABSOLUTE_DEVIATION, MEDIAN, PERCENTILE([25, 75, 90, 95, 99]), SUM and STD_DEV at two levels of confidence, 50% and 90%. In total, we have around 7500 distinct experiments. For each experiment, we assessed the interval calibration using 100 runs and counting the number of times the true statistic lands in the confidence interval. This gives us a binomially distributed estimate for the true confidence interval coverage. The variation we expect in the estimated coverage changes slightly with the level of confidence; for example, at 50% we expect to see values mainly between 0.44 and 0.56, and for 90% we expect to see values mainly between 0.86 and 0.94 using 100 trials.

Figure 2 shows box plots for the empirical coverage for the two confidence levels computed from all experiments. In all cases, the confidence intervals are reasonably well calibrated. Extreme percentiles are biased for small sample sizes, which leads to increased outlier counts for small sample sizes. As a rule of thumb, you’d want roughly samples to ensure that you have enough samples in the appropriate tail.

Next, we examine the degree to which the confidence intervals capture the typical size of the estimate error. To do this, we examine the distribution of the ratio of the estimated statistics' error and half the confidence interval width for all certified results. The higher the confidence, the wider the interval, so different confidence levels shift the mean of this distribution. Figure 3 shows this distribution computed for the 90% confidence interval. As expected, the distribution is roughly normal, albeit with a tail of some larger errors. We see in all cases the confidence interval width gives the order of magnitude of the estimated statistics' actual errors.

We’ve shown that certified results are nearly always reliable; however, we’d also like some insight into the proportion of results which we fail to certify that are actually reliable, to confirm that the test aligns with our objective. We use reliable here in the fairly strong sense that the confidence interval is well calibrated. Specifically, for the 50% and 90% confidence intervals, we count the proportion of uncertified results for which the confidence interval empirical calibration has an acceptable margin of error, given the number of trials used to estimate it. Using this procedure, the false positive rate across all experiments is around 1%. This agrees well with the failure rate we expect by chance, given our test parameters, and confirms the assumption underlying the test.

Finally, to better understand the difference between certified and uncertified results, Figure 4 shows the error distribution of the ratio of the estimated statistics' errors and half the 90% confidence interval for the reliable and unreliable results separately. Note that we truncated the range for uncertified intervals.

Wrapping up

In this post, we present the background behind our approach for quickly estimating ES|QL queries and providing an indication of their errors. To do this, we developed an effective confidence interval mechanism that allows us to provide error estimates. Our approach also allows us to estimate confidence intervals for quantities derived from sampled statistics via other pipeline operations. Quantifying the error comes with a relatively small overhead compared to just estimating the query. Finally, we developed a statistical test to certify results we return. Values that aren’t certified can still be accurate, but we’re less confident in them.

As well as testing the feature on a range of real-world use cases, which we discuss in our companion post, we tested the error estimation by extensive simulation across a range of data characteristics, sample sizes, aggregation functions, and confidence levels. This showed confidence intervals are well calibrated, and the interval itself provides a good approximation of the actual error we observe in the estimates. Finally, we showed that we were able to certify intervals with a low false negative rate.

We’re planning to integrate this feature into other stack capabilities in the future, so stay tuned.

One of the key requirements for a pleasant user experience is that these operations are performed quickly. Large language model–based (LLM) agents also introduce new higher bandwidth and speculative query patterns that can potentially benefit from different optimization strategies.

In this two-part blog series, we discuss an optimization approach we’re introducing to ES|QL in version 9.4 of Elasticsearch and the Elastic Stack, which exploits a relaxation of the problem. Rather than trying to get exact values for aggregates, we allow ourselves to return approximate values, together with some characterization of their error. A key benefit of approximation is that it breaks the dependency between performance and dataset size: The accuracy with which one can approximate a query doesn’t depend on the original dataset size but, principally, its data characteristics and the query itself. As we’ll see later, this allows us to achieve some dramatic performance improvements.

In our next blog post, we will discuss the theory behind our approach and the validation we’ve done of its statistical properties. Here, we introduce the syntax and give a sense of how it’s achieved using standard ES|QL and query rewriting. You can explore its performance on a subset of the popular ClickBench benchmark. Finally, we discuss some limitations and gotchas that are worth understanding when you use query approximation.

Syntax and behavior

So how do you actually use it?

That’s it. You simply introduce the new line SET approximation=true; and write your STATS query pipeline as usual. Below, we discuss some advanced configuration options and some limitations around the agg(...) and commands. However, essentially, we choose defaults so that this will typically provide useful approximations while achieving significant speedups.

With this change, you’ll see some differences in the query results. Let’s look at a concrete example to illustrate this. Suppose the raw query is as follows:

The results might look something like this:

Approximating this query introduces some extra columns for each quantity that’s estimated:

The count column now contains an estimate, and you’ll see it’s somewhat different from the exact values above. The _approximation_confidence_interval(count) column defaults to the central 90% confidence interval for the count estimate and the _approximation_certified(count) column indicates if we’re highly confident that the results and their confidence interval are trustworthy. In outline, the confidence interval is an interval we expect has a high probability (0.9) of containing the true value for the quantity being estimated. The certified column indicates the distribution of the approximation is behaving as we expect. When the result isn’t certified, it’s often still accurate, but our test of the properties of its distribution hasn’t been able to confirm this. These quantities are discussed in more detail in our second post.

Implementation

An approximate query is rewritten before query execution using random sampling and extrapolation. Let’s take a look at the query of the previous section. The part of the rewritten query responsible for obtaining the best estimate looks like:

The query samples a fraction of the data, and therefore the final count has to be extrapolated by scaling up with the inverse of the sample probability. Extrapolation clearly depends on the underlying aggregation function, and we handle this appropriately for all functions we support.

To obtain the sample probability, we're setting a fixed number_of_rows to be processed by the STATS command. In this case, the probability is calculated as follows:

This query is executed before the final approximate query is executed.

As well as this best estimate, confidence intervals and a statistical test used to certify that the value distribution is behaving as we expect also need to be computed. The intervals are computed using a variant of the bias-corrected and accelerated bootstrap confidence interval (BCa) method. Therefore, the data needs to be partitioned into B buckets, which are used in turn to compute the intervals. Omitting some implementation details, this approximate query looks like:

To certify the estimate and confidence interval, there should be enough data, and the distribution of the bucket values should tend to normality.

Some queries can be efficiently computed using only summary statistics maintained in the index. To handle these correctly, where sampling is both slower and inaccurate, we updated the physical query planner, since detecting this case requires information that’s only available where the data resides. When the planner detects this is possible, it simply executes the query as normal. Such queries are typically fast anyway, and there’s no real side effect, so you don’t need to worry about this when using approximation; however, you’ll see that confidence intervals for such queries always have zero length, indicating the results are exact.

Results

To explore the performance improvements, we use ClickBench. This is a benchmark for analytics workloads for database management systems (DBMS). It comprises approximately 100 million rows, with a focus on clickstream and traffic analysis, web analytics, machine-generated data, structured logs, and events data. The benchmark also defines 43 queries that are typical of ad-hoc analytics and real-time dashboards.

Some of the queries aren’t suitable for approximation. For example, we don’t support approximating the unique count of a categorical value or computing the minimum and maximum of a metric value. We also don’t care about queries targeting search alone, for which Elasticsearch has excellent performance in any case. We therefore exclude these types of query from our evaluation. Finally, we also want to test a few additional aggregation functions, such as percentiles, which are not well represented in the original query set, so add some variants of the original metric queries to this end.

Queries in the benchmark are written using standard SQL and so need porting to use ES|QL syntax. This translation is fairly straightforward. Here’s an example:

becomes:

when rewritten in ES|QL.

For running all benchmarks, we use an Elastic Cloud Hosted instance with 870GB disk, 29GB Ram, and 4 vCPUs, in effect, an Amazon Elastic Compute Cloud (EC2) i3.xlarge instance. In the following results, we simply compare ES|QL with and without query approximation. Extensive results on a range of different hardware setups and datastores can be found here. Even with significantly constrained test hardware (matching the vCPUs of the smallest setup), our approximation approach achieves competitive results against much larger systems.

We run each query and its approximation five times in a random order, clearing the query cache between each run. We report the average run time over all five runs. While clearing the cache should be sufficient to avoid most of the advantage of running second, we wanted to avoid any possible accidental prewarming effects, which is why we alternate.

The results break down into four categories:

1. Queries which are rewritten to use index summary statistics (three queries).

2. Queries that perform well (13 queries).

3. Queries with high cardinality partitioning (seven queries).

4. Queries with restrictive filters (12 queries).

Roughly speaking, for these four categories, approximate querying is: equivalent (1); faster and accurate (2); faster but unreliable (3); and slightly slower (4), compared to exact querying, respectively.

For category 1, the planner automatically detects that we’re able to perform the query using summary statistics, and we end up executing the queries in the same way. To do this, we need information that’s only available on the data nodes, so we perform the rewrite only after we've estimated the sample probability. Because we're able to do this very efficiently, the overhead is small (around 10–15%). In both cases, the results are exact.

Queries in category 2 run on average 23 faster if estimating the values and computing confidence intervals and 72 faster if just estimating the values, which you can select as follows: SET approximation={"confidence_level":null}. These headline figures hide quite some variation in the impact of approximation on performance. The table below shows some queries sampled from the range of speedups we see:

Query

Baseline / ms

Approximate with CI / ms

Approximate without CI / ms

3

1725

145

15

10

4340

1721

56

13

32912

6106

3821

21

46739

3284

2139

22

252505

6478

5019

Here are the corresponding queries:

We'll return to the accuracy of the approximation in the next blog post, but to give a sense of this, we plot below the exact and approximate values for a sample run for query 13:

For category 3, we get an average speedup of . However, the results of queries in this category can miss some partitions and often have large estimation errors. Approximation can still be valuable for such queries, particularly in the context of agentic workflows, but requires larger sample sizes than out default if accuracy is important. As we discuss in the next section, we provide an API to explicitly control the sample size. If the source dataset is sufficiently large, this can be increased and approximation will still yield significant performance improvements. The table below shows a couple of query examples for this category:

Query

Baseline / ms

Approximate with CI / ms

Approximate without CI / ms

15

8256

1187

124

17

70641

2109

982

Here are the corresponding queries:

Finally, category 4 queries use selective filters and end up being executed exactly, but they run slightly slower because of the work done in the query rewrite stage. Typically, all these queries run fast anyway, so the absolute slowdown is small. On average, they run approximately 14% or 370ms slower than the “without” sampling for our test setup.

Limitations and best practices

It’s worth explicitly mentioning some limitations. In particular, the following queries are not currently supported:

1. Queries using the TS source command.

2. Queries using the FORK or JOIN processing command.

3. Pipelines which use two or more STATS commands.

4. The ABSENT, PRESENT, DISTINCT_COUNT, MIN, MAX, TOP, ST_CENTROID_AGG and ST_EXTENT_AGG aggregation functions.

We plan to lift some of these restrictions in future releases, such as approximating queries using TS, FORK and JOIN; however, some are intrinsic. For example, while there’s prior art for estimating the minimum and maximum of a metric dataset or the count of unique values of a categorical dataset (see, for example, this paper), they require making certain distributional assumptions, either explicitly or implicitly. In summary, we view trying to automatically provide estimates of these statistics as being too open to accidental misuse.

For the expert user, we provide another route: ES|QL supports using the SAMPLE command directly. This allows one to obtain “point estimates” of any query, albeit with no attempt to correct for the impact of sampling or quantify error. For example:

computes the unique count of the value field on a sample of roughly 1/100th of the dataset. The sample probability can be adjusted to get a sense of how this is asymptoting, or more sophisticated estimation procedures can use STATS COUNT() BY value to estimate the frequency profile of the data.

There are a couple of cases that are more problematic for sampling. If a very restrictive filter is applied in the query, then sampling is of little value, since few rows match anyway. In such cases, we discover that we’d have to sample too large a proportion of the rows to estimate the query in the rewrite phase. In this case, we revert to running the query without sampling and its result is exact. However, the search procedure to determine the fraction of rows to sample comes with some overhead. One therefore pays a penalty, albeit less than the original query cost, for no benefit. If you know in advance that the query is expected to match relatively few rows, it's best to run it without approximation.

The second case only applies when computing STATS partitioned by some expression. If the cardinality of this expression is very high, then even if many rows are searched, individual statistics may be computed from a small number of rows. Some cases are more problematic than others. Sorting by ascending count, that is, finding the rarest partitions, can be impossible to estimate in a single query if heavy hitters would require us to sample most of the dataset to find them. For this particular case, heavy hitting partitions can be estimated first and sometimes efficiently excluded by updating the query. In general, infrequent partitions may be lost in the sampling process, and their statistics' estimation errors can be high. It’s worth noting that we won’t attempt to estimate any statistic for which we have fewer than 10 samples, and we simply drop them from the result set. In the case of very high cardinality BY clause, for example, a field whose value is unique for every row, this means the query can return no results. If you find approximate query results are too inaccurate, you have the option to increase the sample size, which by default is 1,000,000 for STATS, which uses grouping and 100,000 otherwise. Currently, this needs to be done manually, and we provide the following API for this:

Occasionally, functions significantly alter the distribution characteristics of the quantities they act on. A contrived example is the following:

If the variation in the estimate sl is much larger than we expect the distribution of csl to be mainly flat in the interval with peaks near both endpoints. In this particular case, it’s not clear that the central confidence interval is a particularly useful concept, since the modes of the distribution lie outside almost all central confidence intervals. In any case, just observing the samples of csl, our standard confidence interval machinery won’t reliably characterize this distribution and it will underestimate the variability of csl. However, our statistical test should detect this problem, and the result won’t be certified.

Finally, we note that Elasticsearch implements some query optimization strategies that ideally need to account for the fact that sampling is taking place. These rewrite the query at the Lucene level and the preprocessing involved in this rewrite can be relatively expensive. Accelerating an expensive string matching operation by first building a suitable data structure makes sense if the query needs to process every row, but if it processes only a small fraction of them, the trade-off is different. This is something we plan to enhance in future.

Conclusions

In this blog post, we introduced a new form of query optimization we’re bringing to ES|QL that enables dramatically faster querying by relaxing the constraint that the results are exact. We found on ClickBench that we were able to accurately estimate query values and their confidence intervals up to 100 times faster and values alone up to 250 times faster than we can compute them exactly. Furthermore, we expect this advantage to grow as the dataset size increases, because the approximation accuracy is independent of the dataset size. This feature works with many features of the ES|QL language and is enabled by simply prepending SET approximation=true; to the query to estimate.

As well as providing a point estimate, we also estimate confidence intervals and indicate whether we think that the underlying assumptions used to compute these are satisfied. This allows us to certify the results if the results are reliable. We explain the theory behind this feature and discuss the testing of its accuracy in our next post.

Understanding the BEIR benchmark in search relevance evaluation

To improve any system you need to be able to measure how well it is doing. In the context of search BEIR (or equivalently the Retrieval section of the MTEB leaderboard) is considered the “holy grail” for the information retrieval community and there is no surprise in that. It’s a very well-structured benchmark with varied datasets across different tasks. More specifically, the following areas are covered:

• Argument retrieval (ArguAna, Touche2020)

• Open-domain QA (HotpotQA, Natural Questions, FiQA)

• Passage retrieval (MSMARCO)

• Duplicate question retrieval (Quora, CQADupstack)

• Fact-checking (FEVER, Climate-FEVER, Scifact)

• Biomedical information retrieval (TREC-COVID, NFCorpus, BioASQ)

• Entity retrieval (DBPedia)

• Citation prediction (SCIDOCS)

It provides a single statistic, nDCG@10, related to how well a system matches the most relevant documents for each task example in the top results it returns. For a search system that a human interacts with relevance of top results is critical. However, there are many nuances to evaluating search that a single summary statistic misses.

Structure of a BEIR dataset

Each benchmark has three artefacts:

• the corpus or documents to retrieve

• the queries

• the relevance judgements for the queries (aka qrels).

Relevance judgments are provided as a score which is zero or greater. Non-zero scores indicate that the document is somewhat related to the query.

Dataset

Corpus size

#Queries in the test set

#qrels positively labeled

#qrels equal to zero

#duplicates in the corpus

Arguana

8,674

1,406

1,406

0

96

Climate-FEVER

5,416,593

1,535

4,681

0

0

DBPedia

4,635,922

400

15,286

28,229

0

FEVER

5,416,568

6,666

7,937

0

0

FiQA-2018

57,638

648

1,706

0

0

HotpotQA

5,233,329

7,405

14,810

0

0

Natural Questions

2,681,468

3,452

4,021

0

16,781

NFCorpus

3,633

323

12,334

0

80

Quora

522,931

10,000

15,675

0

1,092

SCIDOCS

25,657

1,000

4,928

25,000

2

Scifact

5,183

300

339

0

0

Touche2020

382,545

49

932

1,982

5,357

TREC-COVID

171,332

50

24,763

41,663

0

MSMARCO

8,841,823

6,980

7,437

0

324

CQADupstack (sum)

457,199

13,145

23,703

0

0

Table 1: Dataset statistics. The numbers were calculated on the test portion of the datasets (dev for MSMARCO).

Table 1 presents some statistics for the datasets that comprise the BEIR benchmark such as the number of documents in the corpus, the number of queries in the test dataset and the number of positive/negative (query, doc) pairs in the qrels file. From a quick a look in the data we can immediately infer the following:

• Most of the datasets do not contain any negative relationships in the qrels file, i.e. zero scores, which would explicitly denote documents as irrelevant to the given query.

• The average number of document relationships per query (#qrels / #queries) varies from 1.0 in the case of ArguAna to 493.5 (TREC-COVID) but with a value <5 for the majority of the cases.

• Some datasets suffer from duplicate documents in the corpus which in some cases may lead to incorrect evaluation i.e. when a document is considered relevant to a query but its duplicate is not. For example, in ArguAna we have identified 96 cases of duplicate doc pairs with only one doc per pair being marked as relevant to a query. By “expanding” the initial qrels list to also include the duplicates we have observed a relative increase of ~1% in the nDCG@10 score on average.

Example of duplicate pairs in ArguAna. In the qrels file only the first appears to be relevant (as counter-argument) to query (“test-economy-epiasghbf-pro02a”)

When comparing models on the MTEB leaderboard it is tempting to focus on average retrieval quality. This is a good proxy to the overall quality of the model, but it doesn't necessarily tell you how it will perform for you. Since results are reported per data set, it is worth understanding how closely the different data sets relate to your search task and rescore models using only the most relevant ones. If you want to dig deeper, you can additionally check for topic overlap with the various data set corpuses. Stratifying quality measures by topic gives a much finer-grained assessment of their specific strengths and weaknesses.

One important note here is that when a document is not marked in the qrels file then by default it is considered irrelevant to the query. We dive a little further into this area and collect some evidence to shed more light on the following question: “How often is an evaluator presented with (query, document) pairs for which there is no ground truth information?". The reason that this is important is that when only shallow markup is available (and thus not every relevant document is labeled as such) one Information Retrieval system can be judged worse than another just because it “chooses” to surface different relevant (but unmarked) documents. This is a common gotcha in creating high quality evaluation sets, particularly for large datasets. To be feasible manual labelling usually focuses on top results returned by the current system, so potentially misses relevant documents in its blind spots. Therefore, it is usually preferable to focus more resources on fuller mark up of fewer queries than broad shallow markup.

Leveraging the BEIR benchmark for search relevance evaluation

To initiate our analysis we implement the following scenario (see the notebook):

1. First, we load the corpus of each dataset into an Elasticsearch index.

2. For each query in the test set we retrieve the top-100 documents with BM25.

3. We rerank, the retrieved documents using a variety of SOTA reranking models.

4. Finally, we report the “judge rate” for the top-10 documents coming from steps 2 (after retrieval) and 3 (after reranking). In other words, we calculate the average percentage of the top-10 documents that have a score in the qrels file.

The list of reranking of models we used is the following:

• Cohere's rerank-english-v2.0 and rerank-english-v3.0

• BGE-base

• mxbai-rerank-xsmall-v1

• MiniLM-L-6-v2

Retrieval

Reranking

Dataset

BM25 (%)

Cohere Rerank v2 (%)

Cohere Rerank v3 (%)

BGE-base (%)

mxbai-rerank-xsmall-v1 (%)

MiniLM-L-6-v2 (%)

Arguana

7.54

4.87

7.87

4.52

4.53

6.84

Climate-FEVER

5.75

6.24

8.15

9.36

7.79

7.58

DBPedia

61.18

60.78

64.15

63.9

63.5

67.62

FEVER

8.89

9.97

10.08

10.19

9.88

9.88

FiQa-2018

7.02

11.02

10.77

8.43

9.1

9.44

HotpotQA

12.59

14.5

14.76

15.1

14.02

14.42

Natural Questions

5.94

8.84

8.71

8.37

8.14

8.34

NFCorpus

31.67

32.9

33.91

30.63

32.77

32.45

Quora

12.2

10.46

13.04

11.26

12.58

12.78

SCIDOCS

8.62

9.41

9.71

8.04

8.79

8.52

Scifact

9.07

9.57

9.77

9.3

9.1

9.17

Touche2020

38.78

30.41

32.24

33.06

37.96

33.67

TREC-COVID

92.4

98.4

98.2

93.8

99.6

97.4

MSMARCO

3.97

6.00

6.03

6.07

5.47

6.11

CQADupstack (avg.)

5.47

6.32

6.87

5.89

6.22

6.16

Table 2: Judge rate per (dataset, reranker) pairs calculated on the top-10 retrieved/reranked documents

From Table 2, with the exception of TREC-COVID (>90% coverage), DBPedia (~65%), Touche2020 and nfcorpus (~35%), we see that the majority of the datasets have a labeling rate between 5% and a little more than 10% after retrieval or reranking. This doesn’t mean that all these unmarked documents are relevant but there might be a subset of them -especially those placed in the top positions- that could be positive.

With the arrival of general purpose instruction tuned language models, we have a new powerful tool which can potentially automate judging relevance. These methods are typically far too computationally expensive to be used online for search, but here we are concerned with offline evaluation. In the following we use them to explore the evidence that some of the BEIR datasets suffer from shallow markup.

In order to further investigate this hypothesis we decided to focus on MSMARCO and select a subset of 100 queries along with the top-5 reranked (with Cohere v2) documents which are currently not marked as relevant. We followed two different paths of evaluation: First, we used a carefully tuned prompt (more on this in a later post) to prime the recently released Phi-3-mini-4k model to predict the relevance (or not) of a document to the query. In parallel, these cases were also manually labeled in order to also assess the agreement rate between the LLM output and human judgment. Overall, we can draw the following two conclusions:

• The agreement rate between the LLM responses and human judgments was close to 80% which seems good enough as a starting point in that direction.

• In 57.6% of the cases (based on human judgment) the returned documents were found to be actually relevant to the query. To state this in a different way: For 100 queries we have 107 documents judged to be relevant, but at least 0.576 x 5 x 100 = 288 extra documents which are actually relevant!

Here, some examples drawn from the MSMARCO/dev dataset which contain the query, the annotated positive document (from qrels) and a false negative document due to incomplete markup:

Example 1:

Example 2:

Manually evaluating specific queries like this is a generally useful technique for understanding search quality that complements quantitive measures like nDCG@10. If you have a representative set of queries you always run when you make changes to search, it gives you important qualitative information about how performance changes, which is invisible in the statistics. For example, it gives you much more insight into the false results your search returns: it can help you spot obvious howlers in retrieved results, classes of related mistakes, such as misinterpreting domain-specific terminology, and so on.

Our result is in agreement with relevant research around MSMARCO evaluation. For example, Arabzadeh et al. follow a similar procedure where they employ crowdsourced workers to make preference judgments: among other things, they show that in many cases the documents returned by the reranking modules are preferred compared to the documents in the MSMARCO qrels file. Another piece of evidence comes from the authors of the RocketQA reranker who report that more than 70% of the reranked documents were found relevant after manual inspection.

Update - September 9th: After a careful re-evaluation of the dataset we identified 15 more cases of relevant documents, increasing their total number from 273 to 288

Main takeaways & next steps

• The pursuit for better ground truth is never-ending as it is very crucial for benchmarking and model comparison. LLMs can assist in some evaluation areas if used with caution and tuned with proper instructions

• More generally, given that benchmarks will never be perfect, it might be preferable to switch from a pure score comparison to more robust techniques capturing statistically significant differences. The work of Arabzadeh et al. provides a nice of example of this where based on their findings they build 95% confidence intervals indicating significant (or not) differences between the various runs. In the accompanying notebook we provide an implementation of confidence intervals using bootstrapping.

• From the end-user perspective it’s useful to think about task alignment when reading benchmark results. For example, for an AI engineer who builds a RAG pipeline and knows that the most typical use case involves assembling multiple pieces of information from different sources, then it would be more meaningful to assess the performance of their retrieval model on multi-hop QA datasets like HotpotQA instead of the global average across the whole BEIR benchmark

In the next blog post we will dive deeper into the use of Phi-3 as LLM judge and the journey of tuning it to predict relevance.

Let's run through some basic details, best practices, and how it works, so you can try it out in the Elasticsearch Service today.

Delivering speed and accuracy

Random sampling in Elasticsearch has never been easier or faster. If your query has many aggregations, you can quickly obtain results by using the random_sampler aggregation.

All the above aggregations nested under random_sampler will return sampled results. Each agg is roughly seeing only 0.1% of the documents (or 1 in every 1000th document). Where computational cost correlates with the number of documents, the aggregation speed increases. You may have also noticed the “seed” parameter. You can provide a seedto get consistent results on the same shards. Without a seed, a new random subset of documents is considered and you may get slightly different aggregated results.

How much faster is the random_sampler? The speed improves according to the provided probability as fewer documents are aggregated. The improvements relative to probability will eventually flatten out. Each aggregation has its own computational overhead regardless of the number of documents. An example of this overhead cost is comparing multi-bucket to single metric aggregations. Multi-bucket aggregations have a higher overhead due to their bucket handling logic. While speed is improved for multi-bucket aggregations, the rate of that speed increase will flatten out sooner than single metric.

Figure 1. The speedup expected for aggregations of different constant overhead.

Here are some results on expected speed and error rate over an APM data set of 64 million documents.

The calculations are from: 300 query and aggregation combinations, 5 seeds, and 9 sampling probabilities. In total, 13,500 separate experiments generated the following graphs for median speedup and median relative error as a function of the downsample factor which is 1 / sample probability.

Figure 2. Median speedup as a function of the downsample factor (or 1 / probability provided for the sampler).

Figure 3. Median relative error as a function of the downsample factor (or 1 / probability provided for the sampler).

With a probability of 0.001, for half of the scenarios tested, there was an 80x speed improvement or better with a 4% relative error or less. These tests involved a little over 64 million documents but spread across many shards. More compact shards and larger data can expect better results.

But, you may ask, do the visualizations look the same?

Below are two visualizations showing document counts for every 5 minutes over 100+ million documents. The total set loads in seconds and is sampled in milliseconds. This is with almost no discernible visual difference.

Figure 4. Sampled vs unsampled document count visualizations.

Here is another example. This time the average transaction by hour is calculated and visualized. While visually these are not exactly the same, the overall trends are still evident. For a quick overview of the data to catch trends, sampling works marvelously.

Figure 5. Sampled vs. unsampled average transaction time by hour visualization.

Best practices for using sampling aggregation

Sampling shines when you have a large data set. In these cases you might ask, should I sample before the data is indexed in Elasticsearch? Sampling at query time and before ingestion are complimentary. Each has its distinct advantages.

When sampling at ingest time, it can save disk and indexing costs. However, if your data has multiple facets, you have to stratify sampling over facets when sampling before ingestion, unless you know exactly how it will be queried. This suffers from the curse of dimensionality and you could end up with underrepresented sets of facets. Furthermore, you have to cater for the worst case when sampling before ingestion. For example, if you want to compute percentiles for two queries, one which matches 50% of the documents and one which matches 1% of documents in an index, you can get away with 7X more downsampling for the first query and achieve the same accuracy.

Here is a summary of what to expect from sampling with the random_sampler at query time.

Figure 6. Relative error for different aggregations.

Sampling accuracy varies across aggregations (see Figure 5 for some examples). Here is a list of some aggregations in order of descending accuracy: percentiles, counts, means, sums, variance, minimum, and maximum. Metric aggregation accuracy will also be affected by the underlying data variation: the lower the variation in the values, the fewer samples you need to get accurate aggregate values. The minimum and maximum will not be reliable with outliers, since there is always a reasonable chance that the sampled set misses the one very large (or small) value in the data set. If you are using terms aggregations (or some partitioning such as date histogram), aggregate values for terms (or buckets) with few values will be less accurate or missed altogether.

Aggregations also have fixed overheads (see Figure 1 for an example). This means as the sample size decreases, the performance improvement will eventually level out. Aggregations which have many buckets have higher overheads and so the speedup you will gain from sampling is smaller. For example, a terms aggregation for a high cardinality field will show less performance benefit.

If in doubt, some simple experiments will often suffice to determine good settings for your data set. For example, suppose you want to speed up a dashboard; try reducing the sample probability while the visualizations look similar enough. Chances are your data characteristics will be stable and so this setting will remain reliable.

Uncovering how sampling works

Sampling considers the entire document set within a shard. Once it creates the sampled document set, sampling applies any provided user filter. The documents that match the filter and are within the sampled set are then aggregated (see Figure 7).

Figure 7. Typical request and data flow for the random_sampler aggregation.

The key to the sampling is generating this random subset of the shard efficiently and without statistical biases. Taking geometrically distributed random steps through the document set is equivalent to uniform random sampling, meaning each document in the set has an equally likely chance of being selected into the sample set. The advantage of this approach is that the sampling cost scales with p (where p is the probability configured in the aggregation). This means no matter how small p is, the relative latency of performing the sampling adds will remain fixed.

Ensuring performance reliability and accuracy

To achieve the highest performance, accuracy, and robustness, we evaluated a range of realistic scenarios.

In the case of random_sampler, the evaluation process is complicated by two factors:

1. It cuts right across the aggregation framework and so it needs to be evaluated with many different combinations of query and aggregation,

2. The results are random numbers, so rather than running just once, you need to run multiple times and test the statistical properties of the result set.

We began with a proof of concept that showed that the overall strategy worked and the performance characteristics were remarkable. However, there are multiple factors which can affect implementation performance and accuracy. For example, we found the off-the-shelf sampling code for the geometric distribution was not fast enough. We decided to roll our own using some tricks to extract more random samples per random bit along with a very fast quantized version of the log function. You also need to be careful that you are generating statistically independent samples for different shards. In summary, as is often the case, the devil is in the details.

Undaunted, we wrote a test harness using the Elastic Python client to programmatically generate aggregations and queries, and perform statistical tests of quality.

We wanted the approximations we produce to be unbiased. This means if you run a sampled aggregation repeatedly and averaged the results it would converge towards the true value. Standard machinery allows you to test if there is statistically significant evidence of bias. We used a t-test for the difference between the statistic and true value for each aggregation. In over 300 different experiments, the minimum p-value was around 0.0003 which — given we ran 300 experiments — has about a 9% odds of occurring by chance. This is a little low, but not enough to worry about; furthermore the median p-value was 0.38.

We also tested whether various index properties affect the statistical properties. For example, we wanted to see if we could measure a statistically significant difference between the distribution of results with and without index sorting. A K-S test can be used to check if samples come from the same distribution. In our 300 experiments the smallest p-value was around 0.002 which occurs with odds of about 45% by chance.

Get started today

We're not done with this feature yet. Once you have the ability to generate fast approximate results, a key question is: how accurate are those results? We're planning to integrate a confidence interval calculation directly into the aggregation framework to answer this efficiently in a future release. Learn more about random_sampler_aggregation in this documentation. You can explore this feature and more with a free 14-day trial of Elastic Cloud.

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.