However, with the rapid progression of coding agents, performance optimization has become a task we can tackle semiautomatically. Unlike many software engineering challenges, optimizing code offers a cheap and objective verifier. If you ask an AI model to make code faster, there’s a hard number at the end telling you exactly what happened, backed by profiling tools that explain why. This makes performance a perfect candidate for automation, provided you can actually trust the numbers.

If you simply point a coding agent at a benchmark, you typically get low signal-to-noise: wins that fall inside the variance of the environment, or variations caused by thermal throttling rather than better code. To capture optimizations that actually benefit Elasticsearch users, we had to bridge the gap between "checkable in principle" and "checked in practice." We built a highly trustworthy measurement loop: a harness that assumes the agent will be wrong a good fraction of the time but reliably catches and proves it when it’s right and then helps guide it where to look next.

Once the machinery is in place, the results speak for themselves. By letting this harness loose on the codebase, we've already begun uncovering meaningful wins across the stack. In part 2 of this post, we’ll dive into some examples it has found so far, including string conversion inefficiencies in Elasticsearch Query Language (ES|QL), an improvement to our NEON vector dot product implementation, and an upgrade opportunity for the gzip library we were using. In this part, we’ll take a look at the design choices we made and how they relate to the broader topic of effective harness development.

The AI code optimization pipeline architecture

The first step in any software engineering problem is to identify the correct high-level components. We made an architectural choice that turned out to be very helpful for this problem: separate understanding where opportunities exist from the loop making code changes. The agent starts with a real workload but only uses it to mine information about where to seek performance improvements. At this stage, it’s instructed to go broad and consider a range of performance-related signals. Once it has found and classified the hot spots, the agent reads the context of the code around them to understand the optimization opportunities. We use a separate task to condense the ranked list of hot spots into artifacts that a loop can iterate against in minutes: a microbenchmark that we prove exercises the hot path in its real operating regime. Finally, we use a proposer-verifier loop to actually make changes to the codebase to improve performance on the benchmark. This hands off to validation to assess the impact on real workloads at the end. Our CLI (atune) supplies the tools this process needs, and the rest is largely automated by a set of task-specific instructions.

For context, our high-level architecture looks like the following. Pink boxes are the humans, and teal boxes are the agent. There are three task types, one skeleton loop, and one referee.

An exploration task profiles a real workload and produces ranked opportunities; a human promotes one into an exploitation task, which iterates against an approved microbenchmark and commits each accepted experiment; a validation run on the real workload guards the result before a human reviews and opens the PR. Where no benchmark covers the hot path, a benchmark task authors one and a human approves it into a registry. A performance atlas informs every task and accumulates what each one learns.

Why performance optimization suits autonomous agents

Three properties make a task ideally suited for autonomous work, and it's worth being explicit about them because they provide a checklist you can use to evaluate automation candidates. You want:

1. An objective verdict so that the agent can be held to something other than its own opinion.

2. A dense guiding signal so that it knows where to look next instead of guessing.

3. A bounded blast radius so that being wrong is affordable.

Performance gives you all three. Benchmarks provide the verdict and profilers provide the gradient, while a rejected patch costs you wall-clock time rather than correctness. The change gets reverted, and the reason gets recorded. Life goes on. Regarding the first two, we've come to think the gradient matters more than the verdict. ″This got faster″ is binary, whereas a profile hints at what to try next. An agent can generate its next hypothesis conditioned on a rich guiding signal, the richer the better, rather than grinding through a list.

Signals are what the agent gets to see

A useful mental model is that the CLI is the agent's sensory apparatus. That changes how you design each command. Rather than exposing a capability, you design it to return a clear and concise answer to a question about the task at hand, and, when relevant, an explanation the model can reason over, instead of raw data it has to parse and interpret. These are the signals that the agent acts on, and we ended up with the following for our harness:

Signal

Question it answers

Facet-decomposed macro profile

Where in the code does real workload time go, per query type?

Allocation and lock sampling, in the same capture

Is the cost cycles, garbage, or contention?

Cost-composition classification

Is this in scope compute, other product code, GC, JIT tax, or parked threads?

Input-shape instrumentation

What does the workload actually feed this code?

Statistical verdict

Did this change help, at this measured noise floor?

Allocation-rate comparison

Did the new code end up allocating more?

Interpreted disassembly

Why did that result happen?

End-to-end A/B guard, with differential profile attribution

Did anything appear to break, and was it us?

Environment check

Is this machine even fit to measure right now?

Upstream duplicate search

Has somebody already reported or fixed this?

Four of these signals are worth dwelling on, because in each case the tool encodes a judgment that the agent would otherwise have had to keep making by hand.

Facet-decomposition is the clearest example. Blended CPU shares hide breadth: if you profile a mixed query workload, the grouping hash map insert and the percentiles sketch update can both show up as single-digit percentages of the total run time and look comparable. They aren't comparable at all, because the hash map insert is paid for by nearly every aggregation query, while the sketch update is only paid for when somebody asks for percentiles. So the profiler runs each named query facet as its own race, and every opportunity the agent records carries a breadth field (universal, broad, or narrow) and gets ranked by headroom × tractability × breadth. A universal 3% beats a narrow 10%. Putting the ranking function in the tool prevents it from having to be rediscovered on every run.

The cost-composition classification works as a router. Rather than handing the agent a flat top-N list of frame names, it buckets every sampled stack into ″in scope compute,″ ″other product compute,″ ″GC,″ ″JIT and safepoint overhead,″ ″off CPU waiting,″ and ″parked threads.″ Each of those buckets implies a different kind of investigation. If GC is above about 15%, the real target is allocation rate, and the CPU top frames will actively mislead you, because they show where objects were collected rather than where they were created. If JIT and safepoint overhead is above about 25%, you're looking at a ceiling rather than an opportunity, since no in-scope code change will move it. If threads are parked and core utilization is low, this is a concurrency problem and CPU flame graphs are the wrong instrument entirely. We wrote that mapping into the playbook as a table, so the model (even a cheap one) reads a profile the way that an experienced engineer would, rather than reaching straight for the top frame. The classification is a prior for forming a hypothesis, though, not a substitute for evidence, so the agent still has to cite specific frames when it proposes an experiment.

Interpreted disassembly is a tool we hadn't originally provided, but it most definitely earns its place. It helps answer why, the question that unblocks the next hypothesis. Flame graphs tell you where the time goes; they rarely tell you why a change made things worse. So atune asm runs the benchmark briefly with the JIT told to print the assembly for one hot method, captures both sides of the working-tree diff, reduces each to the final C2 compilation, normalizes the addresses, and diffs them. The diff alone would likely still be 4,000 lines of aarch64, so on top of it sits an interpretation layer: a per-mnemonic delta, a net instruction count, the compilation tier that was actually captured, and a vectorization signal that counts vector register references on each side and raises a warning when they're eliminated, halved, or narrowed from Advanced Vector Extensions (AVX) to Streaming SIMD Extensions (SSE) width. The playbook then maps mnemonic patterns to causes:

Pattern in the diff

Likely cause

b.eq/b.ne up, csel down

New unpredictable branches

Clusters of str/ldr against the stack pointer

The compiler ran out of registers

NEON loads replaced by scalar compares

The vector path degraded

In one experiment, the agent fused two SIMD mask extractions into one, and the benchmark regressed by 26%. The vectorization warning explained it in about 10 seconds. Without that tool, the agent has a dead end and no working model of the machine; with it, it has a corrected model and several new ideas.

The fourth signal is less a single tool than a habit; the instruments check themselves. Core utilization is derived two independent ways, from sample density and from process sampling, so the two can be compared. The classification is rejected if the unclassified bucket exceeds a budget, on the grounds that a breakdown which can't account for its own samples shouldn't be reasoned over. The disassembly capture warns when the compilation it caught isn't the steady-state one. The upstream duplicate search is restricted to read-only commands, and that restriction is enforced by a test that greps the source, so no future edit can quietly reintroduce the ability to file anything. Each of these exists because a tool that can be confidently wrong is worse than a tool that is merely absent.

One small piece of design is worth highlighting as a specific instance of good return practice. atune compare returns 0 for improved, 1 for no change, 2 for regressed, and 3 for error, and the loop branches on this code. That means no parsing and no ambiguity about what the verdict was. Plus, no tokens are spent interpreting prose.

If you take one thing away from our CLI design, it’s a broader design principle. In a general setting, the interesting thing isn't the individual signals we found useful to understand performance; it's that the CLI is capturing and packaging the judgment of an experienced performance engineer into tools that return answers rather than raw data. Structurally imposing good judgment about the problem an agent is tasked with improves outcomes. The right CLI is as much part of that story as the instructions. Furthermore, tokens are saved by tools that return decisions and digests rather than data. That means a comparison verdict instead of raw JMH output; a triage summary sitting on top of a 4,000-line disassembly diff.

From 20-second probes to hours of validation

Building a verifier that you can afford to consult is a separate problem from building one that you can trust. This covers the affordability part. Or, if you like aphorisms, real workloads are where truth lives and where iteration goes to die. A macro profile takes 45 to 60 minutes, and an end-to-end validation run takes hours. But a microbenchmark takes minutes. That's why the exploration-then-exploitation split works; you go broad on the real workload once and then hand off to a microbenchmark that you can iterate against in minutes.

The catch is that the handoff is only sound if the microbenchmark exercises the hot path in a realistic operating regime; that is, with the right cardinality and right data distribution. If you get that wrong, your fast loop spins fast but in the wrong direction. Until we finalized the handoff procedure, we saw cases where the agent accepted changes on a benchmark whose key distributions happened to flatter it, and only the end-to-end run caught the problem.

Handing off from exploration to exploitation

The handoff between the two phases is structured rather than informal. An exploration task's primary deliverable is a set of opportunity records, and each one carries the scope paths that an exploitation task would be allowed to edit, a headroom estimate, a classification (constant factor, structural,

allocation, or concurrency), and the benchmark it would gate on (or an explicit "no benchmark coverage" flag, if none exists). Each also carries a narrow test pattern so that the correctness gate stays cheap. The test pattern field exists because of a specific incident; an exploration task omitted it, and the resulting exploitation task ran a very heavy test suite on every experiment. The fix was to change the upstream artifact rather than add an instruction downstream. This is a pattern we use repeatedly; make and record decisions as early as possible rather than re-derive them each time.

Validating a new benchmark before it can gate anything

Where benchmark coverage is genuinely missing, a dedicated benchmark task authors one, and that new benchmark has to pass validity checks before anything can rely on it. Two of them are mechanical: what fraction of the benchmark’s hot self-time comes from frames that actually appear in the production profile and whether the parameters fall inside the input shapes we measured. The third is a checklist that the agent has to attest item by item. It exists to avoid the JIT getting a simpler world than production.

1. Inputs have to be reshuffled rather than fixed or sorted, so the branch predictor doesn’t get too good.

2. Results have to be consumed, or dead-code elimination deletes the thing that you meant to measure.

3. Inputs must not be compile-time constants, or they get folded away.

4. Call sites have to see roughly the product mix of types, because a monomorphic call site inlines, whereas a megamorphic one doesn’t.

A human then approves it into a hash-pinned registry. Until that happens, it's inert, because task setup refuses any task citing an unapproved benchmark. We think of that approval as the strongest gate in the system, and it's deliberately placed. An approved benchmark can decide accept or reject in every future task, so it's the one place where we ask for a human signature on an artifact rather than on a decision.

The validation ladder

Underneath all of this sits a hierarchy of feedback mechanisms with the property that each rung is cheaper and weaker than the one below it, and the cheap tiers are for rejection only.

Tier

Cost

Role

probe

~20–60 s

Directionally right? Can never accept

codegen capture

~4 min

Why did that happen?

screen

~5–15 min

Cheap statistical filter

confirm

~20–60 min

The accept decision

end-to-end

hours

Regression guard, advisory

The asymmetry between accept and reject is doing real work here. A probe is a single paired fork, whereas the accept predicate requires a full confirm run with a matching calibration record. Agents are very good at telling believable stories; indeed, they're trained on many tasks judged by both LLMs and by humans, so being convincing is actively rewarded. We don't want an agent to be able to promote a cheap signal into a decision by being persuasive about it.

For this task benchmark, wall-clock is a real cost consideration. A confirm run might take an hour. So one has to weigh carefully all the costs involved when choosing the setup. A model that lands one hypothesis in four typically beats a cheaper one landing one in 10 by a margin on end-to-end metrics. The usual instinct to down-spec the model on a long-running loop is exactly backward here. When your loop has an uncertain outcome and significant costs beyond the tokens it consumes, you may well find yourself in the same situation.

How do you know a performance improvement is real?

Is it faster? is a statistical question. So we made the accept predicate code rather than judgment and put it somewhere the agent can't bypass. There are four ideas in the accept decision, and in each case, the alternative we rejected is as informative as the choice we made.

Forks are the statistical unit

Each JMH fork collapses to its mean, and verdicts come from an exact two-sided Mann-Whitney U test at α = 0.05 plus a seeded bootstrap confidence interval over three to five fork means per side. The reason is that iterations within a fork share JIT and heap state and are therefore autocorrelated, so treating them as independent samples manufactures significance out of nothing. We rejected comparing single-run scores, which is pure noise, and iteration-level t-tests, which can be confidently wrong. Seeding the bootstrap means that a rerun reproduces the verdict exactly because the agent needs to be able to tell the difference between a result that changed and a result that was never stable.

Pair candidate and baseline in time

The screen (three forks, optionally over a subset of parameters) exists only to kill bad hypotheses in maybe 10 minutes instead of an hour. The accept decision itself comes from a confirm run that measures candidate and baseline back to back using a stash-flip. In an unstable environment, thermal and background drift only cancels if both sides ran under the same conditions, so an hours-old baseline is really a different experiment.

The noise floor is measured, not assumed

The minimum effect size that a task will accept has to clear the A/A-calibrated coefficient of variation for that specific benchmark on that specific machine, and both the confirm run and the comparison refuse to proceed without a matching calibration record. A 1–2% improvement on a laptop is indistinguishable from noise, and because the floor varies by benchmark and by JDK, any global constant you pick will be too loose somewhere and too tight somewhere else.

The accept rule is composite and deliberately conservative

A parameter combination counts as improved only if p < α, the effect clears the calibrated floor, and the confidence interval excludes zero. Overall acceptance then requires that nothing regressed (not the primary benchmarks and not the guards) and that at least one primary combination improved. The headline figure is the geometric mean of the per-combination speedup ratios, which is always positive and composes across experiments, so a task's cumulative improvement is a meaningful number rather than a sum of incomparable percentages.

What must not get slower

Guards deserve a note of their own, because they answer a different question to the primary benchmarks; not Did this get faster? but What must not get slower while it does? The task definition lists them separately for that reason, and they're typically the operations and the input regimes that the change isn't aimed at. If you're optimizing insert throughput on a hash table, iteration is a guard and so is a collision-heavy key distribution. A change that improves the common case by weakening the hash function will look good on uniformly distributed keys while catastrophically degrading more adversarial inputs. We know that because it happened; the collision distribution was missing from the matrix that accepted one of our early experiments, and the task now carries a comment telling future readers never to drop it again for a hash-quality-sensitive scope. While guards cost wall-clock on every confirm, they also surface edge-case regressions, and omitting them can be much more costly in the long run.

Runtime isn't the only thing worth guarding, either. The confirm run also captures normalized allocation rate on both sides and flags any change that buys speed with more than about 15% extra garbage. That one is advisory rather than blocking, because sometimes the trade is the right one. However, it's the kind of regression a purely time-based accept rule would happily wave through but might raise a red flag to an experienced performance engineer with better understanding of the calling context.

The end-to-end gate is one-sided and default open

The end-to-end gate is a different statistical problem: small n, high noise, and a very strong prior that we should accept based on our microbenchmark results. Our first design treated accept and reject on an equal footing, and it produced multiple clearly spurious rejections, so the redesign is one-sided and default open. An operation is flagged only when the median regression exceeds the threshold and every candidate repetition is slower than every base repetition. That full-separation criterion is the nonparametric one-sided test at this sample size, and it's robust to the single outlier repetition that would occasionally fool us otherwise. A flag then also has to be corroborated against a differential CPU flame graph, where only a rise in the task's own in-scope CPU share counts as real. Near misses get reported for transparency but don't trigger triage, and nothing is ever auto-rejected; a flag is a request for human attention rather than a verdict. We have a final backstop which is the large suite of performance tests that we already run against Elasticsearch on a daily basis.

The one-sided gate lesson generalizes beyond benchmarking. A noisy gate should be one-sided and default open where there is strong prior reason to accept. A symmetric threshold on a noisy signal doesn't just cost you real wins, it teaches the loop to distrust its own instruments, and that’s a much more expensive failure.

Exploration and exploitation need different permissions

Exploration and exploitation might look like two phases of one activity, but they have different inputs (a macro workload versus a pinned scope) and different outputs (ranked opportunities versus commits). They also have different failure modes, which means they want different permissions. We made the split a first-class property of a task, which lets us enforce it; an exploration task literally cannot commit. The baselining, comparison, checkpointing, and validation CLI all refuse exploration tasks, benchmarking allows probes only, and every probe diff is always reverted. A broad, speculative survey is safe because nothing it does can edit the code.

A nice ancillary benefit is prompt focus. Each type reads one playbook, in full, with the others explicitly not loaded. If you try to write a single document covering both "find where the headroom is" and "land a validated win inside this scope", you get something that does neither well, because the instructions for good exploration (follow the profile, widen the net, a broad survey is preferred) are close to the opposite of the instructions for good exploitation (one hypothesis at a time, minimal diff, never widen scope).

AI agent memory: Journals, knowledge bases, and postmortems

Sessions are ephemeral, but what you can learn from them isn't, so the harness accumulates three durable assets, plus one disposable view derived from them. These are a journal of the code changes we’ve tried, a knowledge base of how the code performs, postmortems of when the harness failed and, because sessions can be stopped and resumed, a session summary. What makes them work together is a clean ownership rule about which kind of fact goes where.

The journal records what we tried and measured. It's append-only, one file per task, and one record per experiment, and it's written before the code is edited. Rejections carry a forward-looking note in the form "do not retry X because Y", which is probably the highest value line, because it's what stops the next session re-deriving a dead end. Records also carry the environment and the driving model, which means that hypothesis hit rates are comparable across models.

The knowledge base records how the code works and how it performs. It's an indexed collection of per-area summaries, each stamped with the commit it was written against, and every playbook ends with an upkeep step that appends whatever durable facts the run turned up. Because the performance characteristics of the JDK also change from time to time, for example, a new Vector API might implement vector masking more efficiently on AArch64, findings from profile data are also tagged with the JVM version they apply to.

The postmortems record mistakes that the agent has made in the past, and they're indexed by symptom rather than by date. The question a session actually has when a number looks wrong is Have I seen this shape of wrongness before?, and a chronological list doesn't answer it. So the rows read like "validation fails on operations structurally unrelated to your diff", or "screen reports no matching combinations".

Keeping the journal and the knowledge base distinct sounds pedantic but isn't because without the rule, both of them turn into a diary that has a tendency to bloat the context window or miss critical information in context.

The disposable state is a session handoff, and our advice is to never write it by hand and to avoid asking a model for a session summary, if possible. Our harness regenerates a one-page digest mechanically from the journal, so a resumed session doesn’t spend time and tokens reconstructing where the task had got to. Because it’s derived rather than authored, it can’t drift from the record in the way hand-maintained content does. That’s also why it isn’t part of the audit trail; because it’s cheap to regenerate it from the journal.

Mechanical session handoffs point to two lessons, and they turn out to be the same one. Every time a person appears in the loop, it’s a source of friction and an opportunity for error. And every time you reach for a model, ask whether code can do the same job. This seems like an odd thing to advocate in a project whose primary premise is delegating to a model, but the habit is easy to fall into once you have one to hand. Judgment is expensive, wherever it happens to sit, so the person and model both have to earn their place on merit.

Two further things are critical for durable agent memory. The first is that knowledge rots, so you have to lint it. Elasticsearch's main branch moves daily, which means the knowledge base's file citations go stale, so one linter checks them. Another checks that every command, flag, and path cited in the agent-facing docs actually exists, that every postmortem is linked from the index, and that every task type has a playbook, and it runs as part of the test suite. Treating prose written for an agent as a testable artifact is the reason it stays true. The second is that loading discipline is half of memory. The instruction is to load the index and then the one or two summaries matching the task's scope; never to bulk-load the rest. Memory you can't afford to read isn't memory.

Coding agent guardrails: Containment, scope, and stop conditions

Elasticsearch is millions of lines of code, and an unscoped "make it faster" run against a codebase that size is unreviewable and unfalsifiable. It’s also expensive. The converse is that a verifier only protects what it can see, so we have to apply the same boundary to what it can change. The outcome is a task that fixes its goal, allowed paths, benchmarks, thresholds, and stop conditions before any code is edited, and none of those are things the agent may change during a run.

The human owns the safety envelope, defining scope, thresholds, and stop conditions before any code is edited, and owning every handoff that crosses a trust boundary. The agent session owns judgment: reading profiles, forming one hypothesis at a time, editing inside scope. The atune CLI owns mechanical enforcement: scope checks, correctness gates, statistics, calibration, stop conditions, environment checks, and the signals. Persisted state is the record: a worktree pinned at a base commit, an append-only journal, and generated reports.

Containment happens in two layers. The first layer is coarse and task-agnostic; it’s a single static permissions file lets the session edit the per-task worktree (which includes a local branch of Elasticsearch, all journal entries, and CLI artifacts) and the knowledge base, and it denies the pristine Elasticsearch clone, the task definitions, the harness config, and the audit trail. The second is precise and per-task: a git-level scope check that runs over tracked and untracked files before any build, so an out-of-scope edit is rejected before it can be benchmarked or committed.

Two containment layers have a nice corollary. Widening the first layer to the whole worktree costs nothing, because nothing out of scope survives the second one. Coarse containment plus precise scope beats trying to make a single mechanism do both jobs, which is what we tried first and which left us with a permissions file that needed editing for every new task.

Stop conditions are mechanical. There's a maximum number of experiments and of consecutive rejections, along with a cumulative improvement target, and proposing a new experiment is refused once one of them fires. A human can override; the agent can't. That asymmetry is what makes the gates independent of which model happens to be driving.

Tests are add-only. New test files ship with the checkpoint, and modifying or deleting an existing test is blocked by the scope check. This closes the single most tempting shortcut in the entire problem space by construction rather than by instruction, which seems like the right way to handle any shortcut you'd otherwise have to keep asking an agent not to take.

The harness lives outside the code it optimizes. The Elasticsearch clone is a separate, gitignored directory, and each task gets a worktree pinned at a base commit. The payoffs compound; the subject repo stays pristine and upstream mergeable with nothing related to the harness leaking into a PR, and the audit trail is versioned independently of a codebase that moves daily. Also, tasks are isolated from each other and from any developer checkout. Targeting a newer Elasticsearch means creating a new task rather than repointing an old one, because that task's numbers are tied to its base. It also means that the harness is retargetable in principle, since the Elasticsearch-specific parts are configuration, knowledge base, and benchmark registry rather than architecture.

The decisions the agent never makes

The rule we settled on is that the agent runs the loops and a human owns every step that crosses a trust boundary. That is creating work, blessing a measurement instrument, publishing a branch, or acting outside the repo. None of those are in the agent's allowlist, and each has a reason worth stating:

• Humans have to sign off on the task because the thing being constrained can't set its own limits.

• Calibrating the noise floor needs to be done once per benchmark, per machine, and by default is measured rather than assumed. However, it's rather expensive and we allow a human to override if they know the environment well.

• Deciding what to optimize next by promoting an opportunity is a judgment and a new scope.

• Since benchmarks go on to gate other tasks, we consider reviewing this artifact part of the correctness safety net.

• We leave outward-facing actions, such as pushing a branch or filing an issue, to a human until we're confident in the process.

• We allow actions to be forced, but the override has to sit outside the thing being overridden.

How the human actions get surfaced in the workflow matters. The generated report and the session handoff both print the human actions currently due, at the moment they become due, rather than leaving them to be inferred from the playbook.

We're deliberately not taking a position on how permanent the manual processes are. The right amount of supervision for a new technology is an empirical question, and we'd rather measure it than argue about it. We’ve started with a relatively high degree of supervision because that's the cheap direction in which to be wrong (a gate you never needed is easier to remove than a regression you shipped) and because the harness makes the question answerable. Every gate is a named, logged transition, so over time we can see which of them ever changed an outcome and which only ever cost friction. In summary, measure first, and then refine.

Building the harness is the same kind of loop

A lot of the harness design didn't fall out of an initial design document. The signal set, the ranking function, the shape of a task, and the exact wording of a playbook rule each came from watching a run go wrong. If there's one piece of advice here that generalizes, it's to use the thing before it's ready and to instrument your own disappointment.

The clearest example is a rule we now call distrust surprising results. A validation run reported that every operation had regressed, the worst of them by 14.8%. It was wrong twice over. A target operation pattern had overmatched a completely different code path, and a stale output directory from an earlier run was being read alongside the new one. Offered a coherent story, the agent took it and reverted a change that was actually good.

What went into the playbook after that incident is not "be careful." It's a three-step check to run before acting on a surprising verdict:

1. Trace the code path, and confirm that the thing which moved can even reach your diff.

2. Read the raw per-repetition data rather than the summary, and recompute one headline number by hand.

3. Compare the report's shape against a known good run, because a structurally different report implicates the pipeline rather than the code.

Alongside that, there’s another important rule, which is if the agent concludes that the harness is buggy, it must not fix it mid-run, because a mid-run harness change makes every result in that run incomparable. It should journal the evidence and stop.

Improving the harness is itself a loop worth describing. Asking the model to review its own transcripts and the harness documents, and to propose the rule itself, usually works well. It's good at spotting where its own instructions were ambiguous, in a way that's hard to reproduce by rereading the instructions yourself. What makes that output useful is having somewhere for it to land: a terse rule in the playbook, the narrative in a dated postmortem, a symptom keyed index row, and a linter that keeps the citations honest.

Restraint turns out to be part of the same discipline. The design document carries an explicit list of extension points that we've deliberately not built, because the need for them is still speculative. That's the same "don't guess, wait for evidence" rule we impose on the optimization loop, applied to ourselves.

How this applies beyond performance optimization

A few of these themes aren't specific to performance work or to Elasticsearch.

Verifiable work is the current frontier. The same insight drives reinforcement learning with verifiable rewards, and it’s what AlphaEvolve is built around. The tasks agents are consistently good at are the ones that come with a cheap oracle (tests, compilers, benchmarks), and the interesting move isn't finding more such domains but manufacturing oracles for domains that lack them. Performance is an instructive case precisely because the oracle is, in some senses, obvious, and yet building it still took most of the engineering.

The referee pattern generalizes, too. Separating a fallible optimizer from mechanical enforcement is the same shape as sandboxed execution and policy engines: judgment in the model, invariants in code. The practical consequence is that the system's safety doesn't depend on which model drives it. A weaker model wastes benchmark time, but it can't corrupt the code or accept a bogus win.

Goodhart is a standing adversary for any optimization task that uses an agent, and it has a formal treatment worth reading. An agent optimizes exactly what you tell it to, so the two-tier benchmark structure, add-only tests, benchmark approval registry, adversarial guard distributions, and a marker file that makes timing an instrumented build mechanically impossible are all one design theme wearing different clothes.

Tools are context engineering. The consensus is drifting away from "expose everything" and toward a few well-shaped tools that return digests: the principles of progressive disclosure, self-documenting interfaces, and verdicts rather than payloads.

Memory is becoming architecture. A rules file, per-task playbooks, a durable knowledge base, and an append-only journal form a hierarchy with different lifetimes, owners, and loading rules, and the hard part is eviction and staleness rather than storage; hence, the linters.

Finally, human-in-the-loop is a dial rather than a switch, so where it should sit is something to measure per domain rather than assert.

What's in part 2 of this post

The harness design is a set of hypotheses about what autonomous performance work needs, and the harness was built so that we could test them. Part 2 is that test: the first four PRs we raised using it, their gains, how many hypotheses it took to get each one, which gates actually caught something, and where the harness got in its own way. That includes the changes which didn't survive end-to-end validation, since as usual, the rejections are as informative.

Vector Search on Elasticsearch Serverless

Elasticsearch Serverless is built on Stateless Elasticsearch, a fully decoupled compute and storage architecture where index data lives in remote object storage and search nodes maintain only a local cache. For vector search to be fast on this architecture, the scoring engine needs to work directly with the local cache, not copy it to the heap first.

Elasticsearch simdvec is the engine behind every vector distance computation in Elasticsearch. It provides hand-tuned AVX-512 and NEON kernels, bulk scoring with explicit prefetching, and off-heap memory access that keeps data flowing from storage straight to CPU registers. On stateful Elasticsearch, simdvec has always had a direct fuel line: Memory-mapped files feed native pointers straight into SIMD intrinsics. On serverless, the data was sitting right there in the blob cache's memory-mapped regions, in exactly the right form, but there was no path connecting it to the scoring engine.

We've now built that path. simdvec runs on Serverless with the same off-heap, native SIMD scoring as stateful. And because serverless gives us control over the entire storage layer, this is just the beginning.

Premium fuel only: why simdvec requires off-heap memory for vector scoring

simdvec's speed comes from working directly with off-heap memory. It takes a native pointer to memory-mapped data and passes it straight to C++ SIMD intrinsics. No intermediate copies, no heap allocations. The data flows from storage straight to CPU registers. This matters more than it sounds: simdvec's kernels process vectors faster than the data can be copied, so any copy in the path becomes the bottleneck, not the scoring itself.

On stateful Elasticsearch, this just works. Lucene memory-maps index files from local disk, and the scorer extracts a native pointer directly from the mapped region. This is the path that delivers the benchmark numbers we've published, and it's what we wanted to bring to serverless. To see how, we first need to understand how serverless stores and accesses data.

The serverless blob cache: how Elasticsearch stores vector data

In the stateless architecture, the primary copy of all index data lives in remote object storage, such as S3. Each search node maintains a local cache (called the blob cache) that keeps recently and frequently accessed portions of the index data on local SSD. The frozen tier on stateful Elasticsearch uses the same architecture: Searchable snapshots are backed by a similar blob cache that memory-maps regions from remote storage onto local disk. When a search hits cached data, it's served from fast local storage. When it misses, the blob cache fetches the data from the remote store and caches it for future queries.

The blob cache is organized into fixed-size memory-mapped regions, 16MB by default. It manages its own lifecycle: tracking which regions are in use, applying a least-frequently-used eviction policy when the cache is full, and reference counting to ensure regions aren't evicted while being read. The regions are still memory-mapped through the OS, but the blob cache controls which regions exist, which are populated, and when they're reclaimed. On stateful, those decisions are left entirely to the OS.

Crucially, because each region is memory-mapped, the blob cache already holds vector data in exactly the form simdvec needs. But before we built the connection, there was no way to get at it. Every vector comparison was copied into a heap array and handed to a slower scorer. No direct memory pointers, no SIMD, and garbage collection pressure on every call.

Unified scoring: one SIMD path for all storage tiers

We introduced a new abstraction that lets the scorer safely borrow direct memory from whatever storage layer is underneath, just long enough to run the SIMD computation. If the data is available as direct memory, simdvec's native kernels run. If not (data not yet cached or spanning a region boundary), the scorer falls back to a heap copy. In practice, the fallback is rare.

This gave us a single scoring entry point across all tiers:

1. Stateful (local disk): The scorer extracts a native pointer from the OS memory map.

2. Blob cache (serverless, frozen tier): The scorer borrows a direct memory slice from a cache region.

3. Fallback: The scorer copies bytes to the heap. Rare in practice.

The scorer doesn't know which tier it's running on, and it doesn't need to. It also means we no longer maintain separate scoring implementations; previously, there was a fast native path for stateful and a slower path for everything else. Now every improvement to simdvec benefits all tiers automatically, including its most powerful capability: bulk scoring.

Bulk vector scoring across blob cache regions

A single query may score thousands of candidate vectors. simdvec's bulk scoring processes these in batches with multi-accumulator inner loops, query amortization, and cache-line prefetching, up to 4x faster than single-vector alternatives when data exceeds CPU cache.

Search over an Inverted file (IVF) index is where bulk scoring has the most impact. The query selects a set of candidate posting lists and sweeps through the quantized vectors, scoring them in large batches against the query vector. On stateful, those vectors live in one contiguous memory-mapped file, so bulk scoring resolves them with straightforward pointer arithmetic and scores a batch in a single native call.

On serverless, a sweep through a posting list may cross blob cache region boundaries. We extended the direct memory abstraction with a bulk access method that resolves multiple vector offsets to their respective cache regions in a single call. If all vectors in the batch are cached and none cross a region boundary, the scorer gets a direct memory slice and passes the whole batch to simdvec's native bulk kernel with the same prefetching and pipelining as stateful. When a vector does cross a boundary, the system falls back to per-vector scoring: still zero-copy, just without the batching benefit. With 16MB regions and 1024-byte vectors, that happens roughly once every 16,000 vectors.

simdvec's bulk scoring architecture, the key differentiator highlighted in the simdvec benchmarks, now operates on serverless with the same characteristics that make it fast on stateful. So how does it perform in practice?

simdvec on Elasticsearch Serverless: vector search lap times

We benchmarked with an 18 million vector MSMARCO dataset at 1024 dimensions, using IVF with Better Binary Quantization (BBQ) 1-bit quantization. All results are on a warm blob cache with the full dataset resident in local cache regions, so we're measuring the scoring path rather than remote fetch latency.

Throughput. Under concurrent load, search throughput nearly doubled, jumping from 398 to 739 ops/s. Single-client gains were 23-39%, but the real difference shows up under concurrency: The improvement was 2-3x larger because eliminating heap copies removes the GC pressure and allocation contention that previously throttled concurrent scoring.

Tail latency. The direct memory path transformed tail latency under load:

• p99.9 dropped from 237 ms to 30 ms (87% reduction).

• p99.99 dropped from 9.1 seconds to 55 ms (99.4% reduction).

p100 dropped from 11.4 seconds to under 100 ms.

The worst-case outliers that previously took seconds now complete in tens of milliseconds. The heap-copy-induced queueing that caused latency spikes is gone.

Recall is identical. The same vectors are scored, producing the same results. And we're just getting started.

Beyond parity: what Elasticsearch Serverless can do for vector search that stateful can't

Reaching parity with stateful was the goal. But the more interesting realization is what the stateless architecture lets us do that stateful can’t.

On stateful, the OS controls memory-mapped file behavior: which pages stay resident, when to evict, how aggressively to read ahead. The application can offer hints, but they apply to entire file mappings, and the kernel may ignore them. Worse, search and indexing happen concurrently on the same node, so a hint that benefits one access pattern can hurt another. In practice, to balance different needs, you have to be conservative.

On serverless, two things are fundamentally different. The blob cache manages its own memory-mapped regions with full application-level control. And serverless separates indexing and search onto dedicated tiers: Search nodes never merge, indexing nodes never serve queries. No conflicting access patterns means we can be aggressive with memory advice. Here’s what we’re working on:

• Per-region memory advice. The blob cache knows what type of data each region holds. It can issue random-access hints for rescoring regions, where raw float32 vectors are read in unpredictable order and the kernel’s default readahead would waste memory on pages that will never be used. It can apply sequential readahead for scans through quantized vectors. On the indexing tier, merges read data sequentially, so aggressive readahead brings pages in before they're needed, with no risk of harming concurrent random reads that simply aren't happening on that node.

• Cache-aware prefetching. simdvec already prefetches at the CPU cache-line level. On serverless, we can coordinate this with the blob cache's knowledge of region residency, prefetching at multiple levels: remote store to cache, OS pages to RAM, and cache lines to CPU. The blob cache can tell the scorer which regions are resident before scoring begins, avoiding work on data that would trigger a remote fetch.

• Workload-aware eviction. The blob cache can prioritize retaining data that vector search depends on: IVF centroid indexes that are checked on every query or quantized vectors that are scored in bulk, over data that's accessed infrequently. The OS page cache evicts based on generic heuristics with no understanding of what the data represents. On serverless, eviction policy can be tuned to the workload.

The blob cache gives us a level of control over the memory hierarchy that the OS page cache simply can’t. This is why we see serverless as the most promising platform for the next generation of vector search performance work. Not just matching stateful, but surpassing it. And vectors are just the beginning.

Vector search on Elasticsearch Serverless: what we shipped and what's next

simdvec now runs everywhere Elasticsearch runs (stateful, serverless, and frozen tier) with the same native SIMD scoring, the same bulk scoring, and the same off-heap efficiency. The abstraction we built is general-purpose and already wired through every layer in the storage chain, so the same approach could benefit term lookups, aggregations, sorting, and stored field retrieval in the future.

Elasticsearch Serverless is where we're investing most heavily in vector search performance. Every improvement to simdvec, every optimization to the blob cache, and every new storage-level improvement lands here first. If you're choosing where to run your vector workloads, serverless is the platform that keeps getting faster. You can get started with a free Elastic Cloud trial.

Why this matters

Most typical analytical workflows eventually boil down to grouping data. Whether it’s computing average bytes per host, counting events per user, or aggregating metrics across dimensions, the core operation is the same — map keys to groups and update running aggregates.

At a small scale, almost any reasonable hash table works fine. At the large scale (hundreds of millions of documents and millions of distinct groups) details start to matter. Load factors, probing strategy, memory layout, and cache behavior can make the difference between linear performance and a wall of cache misses.

Elasticsearch has supported these workloads for years, but we’re always looking for opportunities to modernize core algorithms. As such, we evaluated a newer approach inspired by Swiss tables and applied it to how ES|QL computes statistics.

What are Swiss tables, really?

Swiss tables are a family of modern hash tables popularized by Google’s SwissTable and later adopted in Abseil and other libraries.

Traditional hash tables spend a lot of time chasing pointers or loading keys just to discover that they don’t match. Swiss tables’ defining feature is the ability to reject most probes using a tiny cache-resident array structure, stored separately from the keys and values, called control bytes, to dramatically reduce memory traffic.

Each control byte represents a single slot and, in our case, encodes two things: whether the slot is empty, and a short fingerprint derived from the hash. These control bytes are laid out contiguously in memory, typically in groups of 16, making them ideal for single instruction, multiple data (SIMD) processing.

Instead of probing one slot at a time, Swiss tables scan an entire control-byte block using vector instructions. In a single operation, the CPU compares the fingerprint of the incoming key against 16 slots and filters out empty entries. Only the few candidates that survive this fast path require loading and comparing the actual keys.

This design trades a small amount of extra metadata for much better cache locality and far fewer random loads. As the table grows and probe chains lengthen, those properties become increasingly valuable.

SIMD at the center

The real star of the show is SIMD.

Control bytes are not just compact, they’re also explicitly designed to be processed with vector instructions. A single SIMD compare can check 16 fingerprints at once, turning what would normally be a loop into a handful of wide operations. For example:

In practice, this means:

• Fewer branches.

• Shorter probe chains.

• Fewer loads from key and value memory.

• Much better utilization of the CPU’s execution units.

Most lookups never make it past the control-byte scan. When they do, the remaining work is focused and predictable. This is exactly the kind of workload that modern CPUs are good at.

SIMD under the hood

For readers who like to peek under the hood, here’s what happens when inserting a new key into the table. We use the Panama Vector API with 128-bit vectors, thus operating on 16 control bytes in parallel.

The following snippet shows the code generated on an Intel Rocket Lake with AVX-512. While the instructions reflect that environment, the design does not depend on AVX-512. The same high-level vector operations are emitted on other platforms using equivalent instructions (for example, AVX2, SSE, or NEON).

Each instruction has a clear role in the insertion process:

• vmovdqu: Loads 16 consecutive control bytes into the 128-bit xmm0 register.

• vpbroadcastb: Replicates the 7-bit fingerprint of the new key across all lanes of the xmm1 register.

• vpcmpeqb: Compares each control byte against the broadcasted fingerprint, producing a mask of potential matches.

• kmovq + test: Moves the mask to a general-purposes register and quickly checks whether a match exists.

Finally, we settled on probing groups of 16 control bytes at a time, as benchmarking showed that expanding to 32 or 64 bytes with wider registers provided no measurable performance benefit.

Integration in ES|QL

Adopting Swiss-style hashing in Elasticsearch was not just a drop-in replacement. ES|QL has strong requirements around memory accounting, safety, and integration with the rest of the compute engine.

We integrated the new hash table tightly with Elasticsearch’s memory management, including the page recycler and circuit breaker accounting, ensuring that allocations remain visible and bounded. Elasticsearch's aggregations are stored densely and indexed by a group ID, keeping the memory layout compact and fast for iteration, as well as enabling certain performance optimizations by allowing random access.

For variable-length byte keys, we cache the full hash alongside the group ID. This avoids recomputing expensive hash codes during probing and improves cache locality by keeping related metadata close together. During rehashing, we can rely on the cached hash and control bytes without inspecting the values themselves, keeping resizing costs low.

One important simplification in our implementation is that entries are never deleted. This removes the need for tombstones (markers to identify previously occupied slots) and allows empty slots to remain truly empty, which further improves probe behavior and keeps control-byte scans efficient.

The result is a design that fits naturally into Elasticsearch’s execution model while preserving the performance characteristics that make Swiss tables attractive.

How does it perform?

At small cardinalities, Swiss tables perform roughly on par with the existing implementation. This is expected: When tables are small, cache effects dominate less and there is little probing to optimize.

As cardinality increases, the picture changes quickly.

The heatmap above plots time improvement factors for different key sizes (8, 32, 64, and 128 bytes) across cardinalities from 1,000 up to 10,000,000 groups. As cardinality grows, the improvement factor steadily increases, reaching up to 2–3x for uniform distributions.

This trend is exactly what the design predicts. Higher cardinality leads to longer probe chains in traditional hash tables, while Swiss-style probing continues to resolve most lookups inside SIMD-friendly control-byte blocks.

Cache behavior tells the story

To better understand the speedups, we ran the same JMH benchmarks under Linux perf and captured cache and TLB statistics.

Compared to the original implementation, the Swiss version performs about 60% fewer cache references overall. Last-level cache loads drop by more than 4x, and LLC load misses fall by over 6x. Since LLC misses often translate directly into main-memory accesses, this reduction alone explains a large portion of the end-to-end improvement.

Closer to the CPU, we see fewer L1 data cache misses and nearly 6x fewer data TLB misses, pointing to tighter spatial locality and more predictable memory access patterns.

This is the practical payoff of SIMD-friendly control bytes. Instead of repeatedly loading keys and values from scattered memory locations, most probes are resolved by scanning a compact, cache-resident structure. Less memory touched means fewer misses, and fewer misses mean faster queries.

Wrapping up

By adopting a Swiss-style hash table design and leaning hard into SIMD-friendly probing, we achieved 2–3x speedups for high-cardinality ES|QL stats workloads, along with more stable and predictable performance.

This work highlights how modern CPU-aware data structures can unlock substantial gains, even for well-trodded problems, like hash tables. There is more room to explore here, like additional primitive type specializations and use in other high-cardinality paths, like joins, all of which are just part of the broader and ongoing effort to continually modernize Elasticsearch internals.

If you’re interested in the details or want to follow the work, check out this pull request and meta issue tracking progress on Github.

Happy hashing!

Recap

First, let’s bring you up to speed. Elasticsearch has established itself as a powerful vector database, offering a rich set of features and strong performance for large-scale similarity search. With capabilities such as scalar quantization, Better Binary Quantization (BBQ), SIMD vector operations, and more disk-efficient algorithms like DiskBBQ, it already provides efficient and flexible options for managing vector workloads.

By integrating NVIDIA cuVS as a callable module for vector search tasks, we aim to deliver significant gains in vector indexing performance and efficiency to better support large-scale vector workloads.

The challenge

One of the toughest challenges in building a high-performance vector database is constructing the vector index - the HNSW graph. Index building quickly becomes dominated by millions or even billions of arithmetic operations as every vector is compared against many others. In addition, index lifecycle operations, such as compaction and merges, can further increase the overall compute overhead of indexing. As data volumes and associated vector embeddings grow exponentially, accelerated computing GPUs, built for massive parallelism and high-throughput math, are ideally positioned to handle these workloads.

Enter the Elasticsearch-GPU Plugin

NVIDIA cuVS is an open-source CUDA-X library for GPU-accelerated vector search and data clustering that enables fast index building and embedding retrieval for AI and recommendation workloads.

Elasticsearch uses cuVS through cuvs-java, an open-source library developed by the community and maintained by NVIDIA. The cuvs-java library is lightweight and builds on the cuVS C API using Panama Foreign Function to expose cuVS features in an idiomatic Java way, while remaining modern and performant.

The cuvs-java library is integrated into a new Elasticsearch plugin; therefore, vector indexing on the GPU can occur on the same Elasticsearch node and process, without the need to provision any external code or hardware. During index building, if the cuVS library is installed and a GPU is present and configured, Elasticsearch will use the GPU to accelerate the vector indexing process. The vectors are given to the GPU, which constructs a CAGRA graph. This graph is then converted to the HNSW format, making it immediately available for vector search on the CPU. The final format of the built graph is the same as what would be built on the CPU; this allows Elasticsearch to leverage GPUs for high-throughput vector indexing when the underlying hardware supports it, while freeing CPU power for other tasks (concurrent search, data processing, etc.).

Index build acceleration

As part of integrating GPU acceleration into Elasticsearch, several enhancements were made to cuvs-java, focusing on efficient data input/output and function invocation. A key enhancement is the use of cuVSMatrix to transparently model vectors, whether they reside on the Java heap, off-heap, or in GPU memory. This enables data to move efficiently between memory and the GPU, avoiding unnecessary copies of potentially billions of vectors.

Thanks to this underlying zero-copy abstraction, both transferring to GPU memory and retrieving the graph can occur directly. During indexing, vectors are first buffered in memory on the Java heap, then sent to the GPU to construct the CAGRA graph. The graph is subsequently retrieved from the GPU, converted into HNSW format, and persisted to disk.

At merge time, the vectors are already stored on disk, bypassing the Java heap entirely. Index files are memory-mapped, and data is transferred directly into GPU memory. The design also easily accommodates different bit-widths, such as float32 or int8, and naturally extends to other quantization schemes.

Drumroll…so, how does it perform?

Before we get into the numbers, a bit of context is helpful. Segment merging in Elasticsearch typically runs automatically in the background during indexing, which makes it difficult to benchmark in isolation. To obtain reproducible results, we used force-merge to explicitly trigger segment merging in a controlled experiment. Since force-merge performs the same underlying merge operations as background merging, its performance serves as a useful indicator of expected improvements, even though the exact gains may differ in real-world indexing workloads.

Now, let’s see the numbers.

Our initial benchmark results are very promising. We ran the benchmark on an AWS g6.4xlarge instance with locally attached NVMe storage. A single node of Elasticsearch was configured to use the default, optimal number of indexing threads (8 - one for each physical core), and to disable merge throttling (which is less applicable with fast NVMe disks).

For the dataset, we used 2.6 million vectors with 1,536 dimensions from the OpenAI Rally vector track, encoded as base64 strings, and indexed as float32 hnsw. In all scenarios, the constructed graphs achieve recall levels of up to 95%. Here’s what we found:

• Indexing Throughput: By moving graph construction to the GPU during in-memory buffer flushes, we increase throughput by ~12x.

• Force-merge: After indexing completes, the GPU continues to accelerate segment merging, speeding up the force-merge phase by ~7x.

• CPU usage: Offloading graph construction to the GPU significantly reduces both average and peak CPU utilization. The graphs below illustrate CPU usage during indexing and merging, highlighting how much lower it is when these operations run on the GPU. Lower CPU utilization during GPU indexing frees up CPU cycles that can be redirected to improve search performance.

• Recall: Accuracy remains effectively the same between CPU and GPU runs, with the GPU-built graph reaching marginally higher recall.

Comparing along another dimension: Price

The earlier comparison intentionally used identical hardware, with the only difference being whether the GPU was used during indexing. That setup is useful for isolating raw compute effects, but we can also look at the comparison from a cost perspective.

At roughly the same hourly price as the GPU-accelerated configuration, one can provision a CPU-only setup with approximately twice the comparable CPU and memory resources: 32 vCPUs (AMD EPYC) and 64 GB of RAM, allowing to double the number of indexing threads to 16.

To keep the comparison fair and consistent, we ran this CPU-only experiment on an AWS g6.8xlarge instance, with the GPU explicitly disabled. This allowed us to hold all other hardware characteristics constant while evaluating the cost–performance trade-off of GPU acceleration versus CPU-only indexing.

The more powerful CPU instance does show improved performance compared to the benchmarks in the above section, as you would expect. However, when we compare this more powerful CPU instance against the original GPU-accelerated results, the GPU still delivers substantial performance gains: ~5x improvement in indexing throughput, and ~6x in force merge, all while building graphs that achieve recall levels of up to 95%.

Conclusion

In end-to-end scenarios, GPU acceleration with NVIDIA cuVS delivers nearly a 12x improvement in indexing throughput and a 7x decrease in force-merge latency, with significantly lower CPU utilization. This shows that vector indexing and merge workloads benefit significantly from GPU acceleration. On a cost-adjusted comparison, GPU acceleration continues to yield substantial performance gains, with approximately 5x higher indexing throughput and 6x faster force-merge operations.

GPU-accelerated vector indexing is currently planned for Tech Preview in Elasticsearch 9.3, which is scheduled to be released early in 2026.

Stay tuned for more.

As part of our mission to leave no efficiencies behind, we are exploring acceleration opportunities with these curious computer chips, which you may have heard of—NVIDIA GPUs! (Seriously, have you not?).

When obsessing over performance, we have several problem spaces to explore—how to index exponentially more data, how to retrieve insights from it, and how to do it when your ML models are involved. You should be able to eke out every last benefit available when you have GPUs.

In this post, we dive into our collaboration with the NVIDIA vector search team as we explore GPU-accelerated vector search in Elasticsearch. This work paves the way for use cases where developers could use a mix of GPUs and CPUs for real-world Elasticsearch-powered apps. Exciting times!

Elasticsearch GPUs

We are excited to share that the Elasticsearch engineering team is helping build the open-source cuVS Java API experience for developers, which exposes bindings for vector search algorithms. This work leverages our previous experience with Panama FFI. Elasticsearch and Apache Lucene use the NVIDIA cuVS API to build the graph during indexing. Okay, we are jumping ahead; let’s rewind a bit.

NVIDIA cuVS, an open-source C++ library, is at the heart of this collaboration. It aims to bring GPU acceleration to vector search by providing higher throughput, lower latency, and faster index build times. But Elasticsearch and Apache Lucene are written in Java; how will this work?

Enter lucene-cuvs and the Elastic-NVIDIA-SearchScale collaboration to bring it into the Lucene ecosystem to explore GPU-accelerated vector search in Elasticsearch. In the recent NVIDIA cuVS 25.02 release, we added a Java API for cuVS. The new API is experimental and will continue to evolve, but it’s currently available for use. The question may arise: aren’t Java to native function calls slow? Not anymore! We’re using the new Panama FFI (Foreign Function Interface) for the bindings, which has minimal overhead for Java to native downcalls.

We’ve been using Panama FFI in Elasticsearch and Lucene for a while now. It’s awesome! But... there is always a “but”, isn’t there? FFI has availability challenges across Java versions. We overcame this by compiling the cuVS API to Java 21 and encapsulating the implementation within a multi-release jar targeting Java 22. This allows the use of cuVS Java directly in Lucene and Elasticsearch.

Ok, now that we have the cuVS Java API, what else would we need?

A tale of two algorithms for CPU

Elasticsearch supports the HNSW algorithm for scalable approximate KNN search. However, to get the most out of the GPU, we use a different algorithm, CAGRA [CUDA ANN GRAph], which has been specifically designed for the high levels of parallelism offered by the GPU.

Before we get into how we look to add support for CAGRA, let’s look at how Elasticsearch and Lucene access index data through a “codec format”. This consists of

1. the on-disk representation,

2. the interfaces for reading and writing data,

3. and the machinery for dealing with Lucene’s segment-based architecture.

We are implementing a new KNN (k-nearest neighbors) vector format that internally uses the cuVS Java API to index and search on the GPU. From here, we “plumb” this codec type through Elasticsearch’s mappings to a field type in the index. As a result, your existing KNN queries continue to work regardless of whether the backing index is using a CAGRA or HNSW graph. Of course, this glosses over many details, which we plan to cover in a future blog. The following is the high-level architecture for a GPU-accelerated Elasticsearch.

This new codec format defaults to CAGRA. However, it also supports converting a CAGRA graph to an HNSW graph for search on the CPU.

Indexing and searching on the GPU: Making some “core” decisions

With the stateless architecture for Elasticsearch Serverless, which separates indexing and search, there is now a clear delineation of responsibilities. We pick the best hardware profile to fulfill each of these independent responsibilities.

We anticipate users to consider two main deployment strategies:

1. Index and search on the GPU: During indexing, build a CAGRA graph and use it during search - ideal when extremely low latency search is required.

2. Index on GPU and search on CPU: During indexing, build a CAGRA graph and convert it to an HNSW graph. The HNSW graph is stored in the index, which can later be used on the CPU for searching.

This flexibility provides different deployment models, offering tradeoffs between cost and performance. For example, an indexing service could use GPU to efficiently build and merge graphs in a timely manner while using a lower-powered CPU for searching.

So here is the plan for GPU-accelerated vector search in Elasticsearch

We are looking forward to bringing performance gains and flexibility with deployment strategies to users, offering various knobs to balance cost and performance. Here is the NVIDIA GTC 2025 session where this work was presented in detail.

We’d like to thank the engineering teams at NVIDIA and SearchScale for their fantastic collaboration. In an upcoming blog, we will explore the implementation details and performance analysis in greater depth. Hold on to your curiosity hats 🎩!