A sorted ES|QL query over 346 million log documents

Using real-world Nginx logs, the benchmark searched 346 million docs across 432 GB of total index data. It returned 500 results and took ~500 seconds. That's more than eight minutes.

What’s happening under the hood?

Before this optimization, candidate rows were loaded, decoded, and evaluated by the user filter before TopN could determine which of them could no longer contribute to the final result.

How min-competitive TopN prunes work in ES|QL

TopN learns something useful while the query is still running. Once its heap is full, the worst value currently in the heap becomes a competitive threshold: that is, any row that cannot beat that value cannot make the final result.

ES|QL can feed that threshold back into earlier stages of execution. Two changes make this work:

1. Dynamic Lucene pushdown. As TopN finds better candidates, its competitive threshold improves. ES|QL pushes that threshold down to Lucene, allowing Lucene to skip candidates that cannot beat the current worst TopN value. This avoids loading, decoding, and evaluating rows that have no chance of making the result.

2. Shared threshold across drivers. ES|QL processes data in parallel across multiple drivers. If each driver only uses its own local threshold, one driver cannot benefit from better candidates discovered by another. Sharing the best-known competitive threshold means a strong result found by one driver immediately helps all the others prune their remaining work.

Together, these changes move TopN from a late-stage filter to something that shapes query execution much earlier in the pipeline.

How does the min-competitive TopN optimization work?

The query asks for the top three rows matching a wildcard filter, sorted in descending order, across three segments of 10 rows each. Without the optimization, all 30 rows are loaded, decoded, and evaluated against the filter before TopN sees any of them, at a cost of 30 row loads and 15 TopN operations.

With dynamic pushdown, ES|QL maintains the TopN heap as it scans. The heap fills at the end of the first segment with [98, 51, 47], and its worst entry is pushed down to Lucene as the competitive threshold. That cuts the second segment to four rows out of 10, which improves the heap to [98, 83, 77] and tightens the threshold again, leaving just two rows to load in the third segment. This produces the same three results, with 16 row loads and 10 TopN operations.

Sharing the TopN threshold across drivers

One driver may find excellent recent candidates immediately. Without coordination, another driver doesn't know that and continues working with a weak threshold. Sharing the global worst competitive value lets that discovery benefit all drivers.

Benchmark results on a local node and Elastic Cloud Serverless

The min-competitive TopN optimization ships as generally available (GA) in Elasticsearch 9.6. For real-world Nginx data benchmarks (same query, same dataset, p50 of 51 measurements, with 20 warmup runs discarded):

Local node

Serverless (GCP)

Before optimization

~500s

~520s

Production implementation

~5s

~14s

Runtime improvement

~99%

~97%

Speedup

100x

37x

Benchmark hardware

Local node

Serverless (GCP)

CPU

Apple M4

Small GCP host

RAM

48 GB

8 GB

Java Virtual Machine (JVM) heap

24 GB

4 GB

Storage

Local external TB5 NVMe SSD

Cloud storage

Shards

100

100

When does this query optimization technique help most?

The biggest gains happen when TopN becomes competitive early. If good candidates appear at the start of execution, the threshold rises quickly and subsequent scans get pruned aggressively. Three conditions amplify this:

1. N is small relative to the candidate population. Finding 500 rows out of 346 million documents gives the optimizer far more room to eliminate work than returning a large fraction of the matches does.

2. The post-Lucene work is expensive enough that skipping a row saves something measurable, like avoiding loading large strings, decompression, or wildcard evaluation.

3. Many candidates pass the coarse Lucene scan in the first place. If almost nothing matches the initial filter, there’s not much unnecessary work left to eliminate.

If matching rows are rare or if good candidates are discovered late, the threshold takes longer to become useful and the gain can be much smaller.

The current implementation applies to @timestamp sorted queries. Queries sorted by other fields don’t yet benefit from this optimization.

What's next for ES|QL query performance

The same idea has room to run in several directions, including:

• Other numeric sort fields. The underlying approach isn’t inherently limited to timestamps, so extending it to other numeric sort fields could benefit a much broader range of sorted TopN queries.

• Processing promising data first. How quickly the threshold becomes useful depends on how early TopN sees good candidates. Processing more promising data slices first could establish a strong threshold sooner, allowing more aggressive pruning of subsequent work.

• Early termination. As the threshold strengthens during execution, it becomes possible to recognize when remaining ranges cannot produce a better result and to stop processing them entirely, rather than running through to completion.

• Coordination across nodes. The current optimization shares competitive information across drivers within a node. Broader coordination across nodes or clusters could let strong candidates discovered in one part of a distributed query reduce work elsewhere.

• Late materialization and result streaming. Competitive filtering and late materialization attack the same problem from different angles. Competitive filtering eliminates rows that cannot reach the final TopN as early as possible, while a fetch phase can defer loading expensive field values until the set of likely winners is much smaller. Result streaming could extend this further. If the engine can determine that some leading results can no longer be displaced, those rows could be returned before the full query completes. For interactive investigations, getting the first useful result fast matters as much as total query runtime.

What this means for Elasticsearch performance tuning

The min-competitive TopN optimization in Elasticsearch 9.6 shows how much work can be avoided when information discovered during query execution is fed back into earlier stages. In our Nginx log benchmark, that turned a roughly 500-second query into a roughly 5-second query on a single node.

This is also a starting point. Extending competitive filtering to more sort fields, finding strong candidates earlier, terminating work sooner, and eventually coordinating these decisions across nodes offer opportunities to push the same idea further.

The earlier we know what can still win, the less work we need to do for everything that cannot.