PromQL compiles to the same compute engine that runs ES|QL and inherits its planner and distributed execution, along with its release process. We didn’t build a second engine for this, and there’s no plugin to install.

This post is about how we built it.

Key takeaways:

• One engine: The implementation combines Elasticsearch’s mature distributed planning, storage, and testing infrastructure with its newer compute engine, which provides a columnar execution runtime. This lets PromQL reuse proven Elasticsearch capabilities while executing through a modern, native vectorized pipeline rather than introducing a separate runtime.

• One server: Elasticsearch implements the Prometheus remote write and query APIs directly, so Prometheus-compatible ingest and queries run without any additional plugins.

• Engineered for efficiency: Supporting PromQL required new engine primitives for range-aligned evaluation grids, backward-looking windows, dynamic label grouping, pipeline result reshaping, and compact wide aggregation keys. These primitives allow PromQL queries to execute efficiently end to end, with the relevant semantics implemented directly in the compute engine rather than through external post-processing.

• Compatibility measured in real use: In addition to Prometheus compliance tests, we built a differential-testing and quality-control pipeline over 2,000 PromQL queries collected from public repositories. 

Read more:

• Query Prometheus Metrics in Elasticsearch with PromQL

• Ship Prometheus Metrics to Elasticsearch with Remote Write

• Bringing Fire to Elasticsearch: Adding Native Prometheus APIs

Why run PromQL on Elasticsearch

Many teams already store logs and traces in Elasticsearch while running metrics in Prometheus or another dedicated metrics back end.

Prometheus and its ecosystem are strong and widely adopted, but large deployments can also bring operational sprawl and scaling challenges, along with limited retention. 

So we set out to combine Elastic’s highly optimized time-series database (TSDB) with a best-in-class metrics ecosystem. The result is a smaller observability stack, with fewer systems to operate and metrics storage that scales horizontally and supports long-term retention.

One engine: PromQL and ES|QL share the same compute engine

We made an early architectural decision not to run a separate PromQL engine next to Elasticsearch.

PromQL is instead another front end to the Elasticsearch compute engine.

This puts PromQL in the normal Elasticsearch development lifecycle. It uses the same planner, distributed execution engine, testing infrastructure, and release process as Elasticsearch itself.

To learn more about Elasticsearch’s query engine, check our blog.

Like ES|QL’s time series queries that use the TS source command, PromQL is translated into a highly optimized query plan and executed across the cluster. Nodes process columnar batches through vectorized operators, while partial results move through exchanges until the final result is assembled. 

This also means that PromQL and ES|QL operate over the same execution engine and time-series data. ES|QL can additionally extend a PromQL computation with post-processing that PromQL doesn’t support, such as lookup joins and inline aggregations.

For example, assume Prometheus request counters are stored in metrics-* and keyed by the service label. A lookup index, service_registry, maps each instance to its owning team and environment and to its service tier:

This architecture requires the execution engine to support PromQL semantics natively and efficiently rather than ES|QL syntax sugar. The following sections describe the changes and new execution primitives we introduced to achieve that.

One server: Prometheus remote write and HTTP API built into Elasticsearch

Query execution is only half of the story. The Prometheus ecosystem also expects familiar ingest and query APIs.

Prometheus protocols are the de facto standard for everything metrics in almost every team’s observability stack. So we built the HTTP API directly in the Elasticsearch server, which eliminated the need for a third component and tightened integration stability and performance.

On the ingest side, we added an endpoint for the Prometheus remote write protocol. It accepts Snappy-compressed Protocol Buffer messages, maps labels to TSDS dimensions, maps the metric name/value into metric fields, infers counter versus gauge mappings, and writes directly into TSDS. The built-in template is dynamic, so users don’t have to predeclare every Prometheus label or metric.

On the query side, Elasticsearch exposes Prometheus query APIs. A request enters through the Prometheus endpoint and is executed in the compute engine.

Running PromQL efficiently in a columnar engine

Sharing an execution engine doesn’t mean treating PromQL as syntax sugar over ES|QL. PromQL has different time, grouping, and response semantics, as well as workload characteristics that matter at scale. Supporting it efficiently required extending the compute engine rather than compensating in the API layer.

PromQL time grids: Aligning evaluation steps with TSTEP

Time-series query engines are optimized for grouping over time.

Elasticsearch normally groups timestamps with TBUCKET(...), which truncates each timestamp to a fixed interval boundary. Truncation is cheap and produces deterministic bucket boundaries. It also makes intermediate results easier to reuse.

Prometheus defines evaluation points differently. For a range query, timestamps are laid out as fixed steps anchored to the query range, rather than derived by truncating each sample timestamp. Two queries with the same step but different range boundaries can therefore produce different evaluation grids.

To preserve these semantics, we introduced TSTEP(...), which derives its grouping grid from the query range and step,  rather than truncating timestamps to globally aligned boundaries.

PromQL uses TSTEP(...) internally, preserving Prometheus timestamp semantics while still lowering the operation to a native Elasticsearch execution primitive.

Some might see this as a simple problem, but the nuances matter. 

Take, for example, a query that finds a 5m rolling average of a metric: 

At evaluation time T, the result represents the average over the preceding five-minute range: 

(T - 5m, T]

When the query is executed with a five-minute step, each output value is labeled with the upper end of its corresponding five-minute window.

Elasticsearch previously lacked this semantic and supported only forward-looking window aggregation functions:

We rewrote the window-evaluation path so that both ES|QL and PromQL use a common backward-looking windowing implementation:

Together, TSTEP(...) and backward-looking windows preserve the two time semantics that matter for PromQL range evaluation.

Dynamic label grouping: How PromQL without() resolves at runtime

Time grids determine when a PromQL expression is evaluated. Aggregation determines which input series are combined and which labels identify each output series.

For most analytical query engines, that identity is known when the query is planned. The planner can allocate grouping columns and choose an aggregation strategy. It also carries a fixed key through the execution pipeline.

That’s how ES|QL works:

The output series are grouped by an explicit key (cluster, namespace).

PromQL can express the same operation in the opposite direction:

Now we know which dimensions not to use. We don’t necessarily know the full grouping key until the query is executed. This is a small language difference with significant execution consequences. 

One possible implementation is to discover every label used by the metric, subtract instance and pod, and rewrite the expression into an ordinary by(...) aggregation. That adds a discovery phase before planning. It also becomes inefficient for high-dimensional metrics where only a subset of all possible dimensions may have useful value in a particular series. Most queries need only a small subset of the available dimensions, so carrying the entire dimension universe as an aggregation key wastes memory and adds bookkeeping overhead.

We instead extended the time-series execution path with dynamic grouping columns. The engine loads dimensions as the series are read and applies the exclusions per time series. This avoids making the grouping schema a prerequisite for planning and avoids carrying a large sparse set of grouping columns through aggregation.

Dimension packing: Keeping wide PromQL grouping keys cheap

The idiomatic way of writing PromQL aggregations involves heavy use of without(...) over by(...):

Excluding labels rather than explicitly listing them makes dashboards and alerts resilient to schema evolution. If a new label is added to the metric, the query continues to preserve it unless it’s explicitly excluded.

The consequence for the engine is that effective grouping keys can be wide. Many of those labels often have low cardinality, yet each still participates in every aggregation stage.

In a columnar engine, each grouping column is normally represented as a separate vector. Ten grouping labels therefore mean 10 vectors flowing through every aggregation operator:

Dimension fields are declared in the index mapping; the planner knows the schema up front, and grouping keys stay narrow and predictable.

In the columnar engine, each grouping label is carried as a separate vector or block. A key with 10 labels therefore requires 10 vectors to be read, hashed, compared, and retained by aggregation operators. As key width grows, so does the amount of data and bookkeeping that must move through the pipeline:

To avoid paying that per-column cost for every additional label, we introduced dimension packing. Before aggregation begins, the engine encodes the full grouping key into a single compact representation. Hash and comparison operations run on the packed key rather than on each block independently:

Packing lets hashing and comparison operate on a single compact key rather than an increasing number of grouping blocks, making aggregation overhead less sensitive to key width. Because the engine is shared, ES|QL time-series queries will benefit from this optimization as well.

Building the Prometheus HTTP API response inside the pipeline

Unlike Elasticsearch's ES|QL column-oriented response format, the Prometheus response is row-oriented.  The Prometheus API returns one result row per time series, with its samples represented as timestamp-value pairs. 

To support a compatible API layer, we had to regroup in the HTTP layer converting the columnar results into boxed row objects and accumulate them in map- and list-based structures until the complete Prometheus response could be produced. 

We replaced this with the TimeSeriesCollapse compute operator. It groups rows by series and aligns samples to the query’s fixed step grid. It emits the reshaped result as ordinary columnar pages containing one row per series, with aligned multi-valued timestamp and value blocks. And it preserves compact, vectorized block representation throughout the pipeline: 

The HTTP layer can now serialize those blocks directly, avoiding the maps, lists, boxed objects, and associated allocations required by the earlier implementation.

Testing PromQL compatibility against 2,000 real queries

Prometheus compliance tests were our starting point.

Even though they gave us a strong baseline, they didn’t tell us how frequently individual PromQL features appear in real workloads. To complement that baseline, we built a second test corpus from over 2,000 PromQL queries collected from public repositories. 

We then classified those queries by the language features and expression patterns they exercise:

For each compatible query shape, we run the same query against Elasticsearch and Prometheus and compare the results. 

In addition to that, we actively rely on fuzz testing, which catches issues that unit tests alone are unlikely to expose, including differences in timestamp alignment, label retention, aggregation behavior, range-vector evaluation, and response encoding.

Which PromQL functions and APIs are supported in 9.5

Since 9.4 (technical preview), PromQL support in Elasticsearch has expanded substantially. In Elasticsearch 9.5, both the PROMQL command in ES|QL and the Prometheus HTTP APIs are generally available (GA), with more than 80% of the PromQL workflows in our real-world corpus now running without modification:

The main additions since technical preview are:

Feature

Example

Status

Prometheus remote write ingest

POST /_prometheus/api/v1/write

GA in 9.5

Range queries

/api/v1/query_range

GA in 9.5

Instant queries

/api/v1/query

GA in 9.5

Metric metadata and build info

/api/v1/metadata, /api/v1/status/buildinfo

GA in 9.5

Native histogram functions

histogram_quantile, histogram_count, histogram_sum

GA in 9.5

Per-selector offset modifiers

[5m] offset 1h

GA in 9.5

Top-level or operator

rate(a[5m]) or rate(b[5m])

GA in 9.5, up to eight operands

Prometheus remote write ingest

Elasticsearch accepts Prometheus remote write (v1) messages directly:

Snappy-compressed Protocol Buffer messages are decoded, and labels are mapped to TSDS dimensions. Metric names and values are written into the time-series index. The built-in template is dynamic, so users don’t have to predeclare every Prometheus label or metric.

Range and instant queries through the Prometheus HTTP API

Both range and instant query endpoints are available:

Range queries return matrices evaluated over a time window, and instant queries return vectors evaluated at a single timestamp. These endpoints can be used by Kibana, Grafana, or Prometheus-compatible alerting tools, and custom dashboards.

Metric metadata and build info endpoints

Elasticsearch exposes metadata about available metrics and a build-info endpoint:

The metadata endpoint returns metric types and help text, and the build-info endpoint returns the Prometheus-compatible server version. Grafana and other tools use these endpoints for feature detection and UI behavior.

Native histogram functions: histogram_quantile, count, and sum

Elasticsearch supports the main PromQL operations over native histograms:

Native histograms adapt their bucket layout to the data, providing useful precision across a wide value range without requiring users to configure every bucket boundary in advance. Classic histograms continue to work alongside native histograms.

Per-selector offset modifiers in PromQL

Offset modifiers shift a selector’s time window backward:

This returns the request rate from one hour earlier. Per-selector offsets are commonly used to compare current traffic, latency, or resource usage with an earlier baseline, such as the same period one week ago.

Top-level or operator in PromQL

Elasticsearch supports the top-level PromQL or operator:

In PromQL, or isn’t a Boolean operation. It performs a union between two sets of time series. Results from the left side are retained; a series from the right side is added only when its label set doesn’t match a series already returned by the left side. This is useful during migrations where the same logical metric may exist under an old and a new name.

The implementation follows Prometheus’s left-side precedence rules and preserves the __name__ label. Top-level chains of up to eight operands are supported.

PromQL features not yet supported in Elasticsearch

GA doesn’t mean complete PromQL compatibility. Some less common and more complex parts of PromQL remain unsupported. These gaps now define the next phase of the work: 

Feature

Example

Status

Advanced vector matching

on(instance) group_left

Planned

Sorting and ranking

topk, bottomk, limitk, sort, sort_desc

Planned

Label manipulation

label_replace, label_join

Planned

Absolute time modifier

@ 1710000000

Planned

Mixed-offset compound expressions

rate(...) - rate(... offset 1h)

Planned

Alerting and target endpoints

/api/v1/alerts, /api/v1/targets

Out of scope

Advanced vector matching with on() and group_left

Some binary operations that require Prometheus vector matching aren’t yet part of GA.

For example, this query divides per-instance request rates by a per-instance capacity metric:

The on(instance) clause specifies which labels identify matching series. group_left permits many request-rate series to match a single per-instance capacity series, while retaining the labels from the higher-cardinality left-hand side.

These expressions are common when joining a detailed metric with metadata or a lower-cardinality capacity metric. Basic binary expressions are supported where applicable, while the remaining vector-matching forms are planned work.

Sorting and ranking: topk, bottomk, and sort

Prometheus sorting and ranking functions are also not yet part of GA:

This returns the 10 services with the highest request rate. Similar queries are widely used in “top offenders” dashboards for traffic, latency, errors, and resource consumption.

The remaining functions include:

Label manipulation with label_replace and label_join

PromQL can construct or rewrite labels during query evaluation. These functions are particularly useful when dashboard variables, naming conventions, or label schemas don’t match exactly:

This creates an environment label from the cluster label.

Another common example combines existing labels into a display-oriented label:

This produces a target label, such as payments/api-7f6d9. label_replace(...) and label_join(...) aren’t yet included in GA.

Advanced time modifiers: The @ modifier and mixed offsets

Several advanced time modifiers and expression forms remain outside the GA scope.

For example, an absolute @ modifier evaluates a selector at a fixed Unix timestamp rather than at the query’s normal evaluation time:

This is useful for comparisons against a fixed historical point.

PromQL also permits expressions in which the two sides use different offsets:

This compares current traffic with traffic one hour earlier. Per-selector offset is available in GA, but not every combination of offsets and compound expressions is part of GA yet.

Prometheus API endpoints not yet implemented

In addition, the Prometheus HTTP API surface isn’t yet fully complete. Notably:

Alerting metadata through:

used by tools that inspect active alert state.

Target discovery through:

used to inspect scrape targets, health, and labels.

These endpoints concern Prometheus server and scrape-target state rather than querying metrics stored in Elasticsearch.

For the full list of limitations, see the PromQL limitations page. 

Try PromQL in Elasticsearch 9.5

To query Prometheus metrics in Elasticsearch 9.5 or Serverless, see the PromQL documentation and the Prometheus HTTP API reference.

Static ID lists vs. dynamic filtering with ES|QL's WHERE clause

Static ID list filtering

Dynamic filtering with WHERE IN subquery

Run query A to find the IDs you care about

Outer query asks the main question

Copy the values by hand

Subquery builds the filter list from live data

Paste them into a static WHERE IN list

WHERE field IN (subquery) applies the filter live

Run query B with the hardcoded list

Results stay current with the data

Repeat the whole process when the data changes

No copied list to maintain

Figure 1. The old copy-paste loop collapses into a single dynamic filter.

How the WHERE IN subquery replaces static filter lists

Traditional IN filtering is still useful when the list is small and static:

But many real investigations don’t start with a tidy list. They start with a question: Which users are suspicious, which hosts are noisy, which services are failing, or which accounts crossed a threshold?

That’s where the WHERE IN subquery becomes useful. The list is produced by ES|QL rather than typed by hand.

WHERE IN subquery example: filtering logs by suspicious users

Read it like this: Show me log events for users who appear in the list of users with at least 10 failed login attempts.

The outer query asks the main question, and the subquery builds the dynamic filter list, eliminating the need to copy and paste.

The subquery can target a different index or index pattern from the outer query.

Filtering without subqueries: the manual ID copy workflow

Imagine that you want to inspect traffic for the top failing services. First you run:

Then you copy the five service names and paste them into another query:

That’s fine once, but less fine when the top five change every hour.

Dynamic filtering with a WHERE IN subquery

The subquery finds the top failing services from the last two days, and the outer query returns the log events for those services. One query builds the full picture.

Excluding values with NOT IN subqueries in ES|QL

Sometimes the interesting question is about what doesn’t belong:

That pattern is useful for exclusion checks, gap analysis, and workflows that ask for the things outside an approved or expected set.

Nested subquery chains in ES|QL's WHERE clause

An IN subquery replaces the literal value list with a query in parentheses. The inner query runs first and returns a single column, and the outer WHERE filters against it. Because that inner query is a full pipeline, it can contain its own IN subquery, which lets you express a chain of lookups that would otherwise require three separate queries and two rounds of copy-paste.

Read it inside out. The innermost query finds priority regions, and the middle query finds customers in those regions, while the outer query sums revenue for those customers. Each layer is a normal ES|QL pipeline, so each one can filter, aggregate, or sort on its own before handing a clean column up to the layer above.

Combining WHERE IN subqueries with AND and OR conditions

Because an IN subquery is a Boolean condition, it composes with AND and OR like any other predicate. You can require membership in two independent sets or accept membership in either:

The AND combination finds orders placed by VIP customers for products that are being discontinued. Swap AND for OR, and you get orders that match either condition. Each subquery runs its own pipeline, so the two sets are computed independently and then combined by the Boolean operator.

Merging multiple indices into one WHERE IN subquery

The query inside an IN subquery is a full pipeline, so its FROM command can reference more than one subquery. Each branch runs its own pipeline, and the FROM command merges the rows from all branches into one result set. KEEP host_id selects the single column that the outer filter needs. This is useful when the values you want to filter against live in several indices with different schemas. For more details on how subqueries in the FROM command handle indices with different schemas, see Three indices walk into a FROM clause: ES|QL subqueries in Elasticsearch.

The IN subquery combines matching host IDs from three indices, prod, staging, and edge, into one value list. The outer query then counts alerts for any host in that combined set. Adding a fourth source means adding one more branch, with no change to the outer query.

Using an Elasticsearch subquery inside each FROM branch

A FROM subquery gives each index its own branch with its own WHERE, and that WHERE can use an IN subquery. This is how you apply the same dynamic filter across several indices that each have their own schema.

Each branch filters its index down to VIP customers before the two branches combine, so the final aggregation runs over a single normalized set of rows.

When to use ES|QL WHERE IN subqueries

• Investigations that start by finding risky users, hosts, accounts, or services.

• Operational dashboards where the interesting entities change over time.

• Top-N follow-up queries, such as events for the five noisiest services.

• Set comparison workflows, especially with NOT IN.

• Queries that would otherwise need glue code just to pass values from one step to the next.

Requirements and constraints for WHERE IN subqueries

• Return exactly one column from the IN subquery.

• Use KEEP at the end of the subquery so the comparison field is obvious.

• Make sure the outer field and the subquery field have compatible types.

• If the subquery uses SORT, add an explicit LIMIT, as unbounded SORT isn’t supported in ES|QL yet.

• Use this for membership filtering. If you need columns from both sides, a JOIN` may be the better tool.

Why ES|QL dynamic filtering replaces manual query workflows

The WHERE IN subquery turns a manual workflow into a declarative one. You can let one query build the filter for another query directly inside the WHERE command, instead of asking ES|QL for a list, copying it somewhere else, and hoping it stays fresh. 

Your WHERE clause now has a better way to handle Filter this by whatever that query finds.

If you've been exploring ES|QL, you might have noticed that a subquery used to feel like it had exactly one front door: FROM. And it made sense. Most of the time, queries start with a simple directive to go fetch documents from an index. In an earlier post, we taught the WHERE command a new trick: IN subqueries. And before that, subqueries showed up in the FROM command to combine data sources. In both instances, every subquery started the same way, with FROM.

But not every useful query starts with a bulk document fetch. Sometimes you need to evaluate time-series semantics. Other times, you just need to whip up a tiny inline row for testing. Sometimes, the absolute best input to a filter is a dynamic query that builds the list for you on the fly, rather than a static, hard-coded list.

Source command

Reads from

Best for

Requirements

FROM

Indexes, data streams, aliases, views

Stored document lookups, live filter lists

None (works with any index)

TS

Time-series data streams

Metric aggregations with counter-reset handling

TSDS with index.mode: time_series

ROW

Inline literal values

Test cases, seed values, synthetic placeholders

None (no index needed)

The FROM, TS and ROW source commands are generally available (GA) in Elasticsearch 9.5, while the WHERE IN subquery remains in technical preview in 9.5.

How ES|QL subquery source commands work

Think of subqueries as having two specific placements and three different engines.

Where subqueries go: FROM and WHERE placements

• Inside the FROM command: The subquery is an independent result source, contributing its rows to the outer query. Fields that exist in one source but not the other are gracefully filled with null values.

• Inside the WHERE command: The IN subquery contributes dynamic values to be used as a predicate or filter.

Three source commands for starting a subquery

Door #1: FROM (the classic door)

FROM is the familiar workhorse. You use it when the subquery should read stored data from indices, data streams, aliases, or views. One of its best use cases is the "stop-copy-pasting-IDs" pattern. Instead of running one query, manually copying the output values, and pasting them into the filter of another query, the subquery becomes your live filter list.

Door #2: ROW (the tiny door)

ROW is the lightweight option that requires absolutely no index setup. It allows you to build rows completely out of literal, inline values. This makes ROW useful for small seed values, test cases, allow/deny lists, or one-off "what if?" scenarios. In the query below, ROW is the perfect way to staple a synthetic sentinel row or placeholder directly onto real data.

Door #3: TS (the time-series door)

The TS command targets time-series data streams and enables time-series aggregation functions. Why not just use FROM for metrics? TS is uniquely optimized for time-series data, and it natively handles tricky scenarios, like counter resets on process restarts and uneven metric publish intervals. Using TS as a subquery lets you filter your application logs by what your metrics are saying. For example, imagine asking ES|QL: Show me log events only from clusters whose peak metric throughput crossed 800:

In this scenario, the filter is reacting to live metric behavior rather than being hard-coded.

Combining FROM, TS and ROW in one query

Because subquery placements and source commands are independent, you can freely mix and match them. You can throw all three doors into a single FROM union to generate a cohesive table containing real logs, a live metrics summary, and a synthetic row.

ES|QL subquery constraints

A few constraints to keep in mind before using subqueries:

• IN subqueries demand one column: If a subquery feeds an IN operator, it must project exactly one column. Use the KEEP command to make that explicitly clear.

• TS requires a time series data stream (TSDS): The TS command only works on data stored in a TSDS, which uses index.mode: time_series.

The takeaway: when to use each source command

Subqueries in ES|QL provide a structured way to compose queries by using the results of one query as the input to another. Choosing the appropriate source command (FROM, ROW, or TS) lets you combine data and generate inline values. It also lets you filter dynamically without duplicating query logic. For more details and additional examples, see the ES|QL subqueries documentation.

• Subquery in FROM command

• Subquery in WHERE command

They’re ECS-aligned, ready for Discover and out-of-the-box dashboards once you ship the log, no custom schema project. Below: why we built this, how it differs from slow logs, what each line contains, and how to turn it on.

Why we built this (you asked, a lot!)

Coordinator-level query logging has been a very popular request; we listened and delivered! The same pain kept showing up: You want the response duration for Service Level Objectives (SLOs) and dashboards. You want to know the execution time of queries executed in your cluster, and you want to be able to see the full query.

If using cross-cluster search, a search that fans out across clusters looks like one operation from the app or Kibana, but operationally it’s a chain of work: coordination, remote execution, merges, timeouts, and partial results. When something is slow or flaky, teams need to know not only how long the request took but also which clusters contributed and whether the outcome was success, partial, or a hard failure.

What you get: One log stream, one entry per query! Every entry has the coordinator duration (the very same took time that actually matches your search API response), success or failure, and the full query text. Elastic Common Schema–compliant (ECS) JSON, optional duration threshold and user/audit fields, plus X-Opaque-Id that lets you trace a hot query back to the saved object it originates from, and the trace ID so you can correlate with Kibana or your own tooling.

What’s more: Logs follow a stable, ECS-aligned schema, which means you don’t need to design your own ingestion pipelines or field mappings. This consistency enables out-of-the-box dashboards and analytics that work immediately once logs are shipped.

Slow logs vs. query logs: The 30-second version

Slow logs have been the go-to tool for years. They tell you which search operation is slow, but they emit one line per shard that took part, where each line reflects that shard’s piece of the work. This means that they don’t provide a single row that says how long the query execution took, from the client’s perspective. Query logs do exactly that: one line per query, with the end-to-end (wall clock) duration that lines up with the took time in the search API response. This makes them much better suited for understanding workload patterns and identifying problematic queries quickly.

Slow and query logs also differ in when they fire and what they cover. Slow logs only write when a shard’s slice breaches a duration threshold; that is, you’re optimized for “show me unusually slow shard work.” Query logs can record every query (or only those above a configurable threshold you set at the cluster level), so you can tune volume for analytics versus troubleshooting. Slow logs only support DSL queries, while query logs cover ES|QL, DSL, SQL, and EQL, which matches how you reason about “what ran on my cluster” in a modern stack. Both provide the same support in terms of correlation with headers, traces, and audit information (when you turn on user context).

The table below summarizes the main differences between the historical slow logs and the new query logs features.

Slow logs

Query logs

What they’re for

Finding hot shards / slow index operations on specific indices and classic performance tuning inside one cluster.

Understanding what query ran, how long the operation took end to end from the coordinator, and whether it succeeded, which is better for SLOs, analytics, and incident investigations.

Granularity

Per shard (and per phase) for searching slow logs: One user search can produce many lines across shards/replicas.

Per coordinator-level query: One query maps to one log event.

Scope of work

Query + indexing

Query only; indexing will come soon.

What you learn

“This shard on this index exceeded N ms in query/fetch phase.”

“This query (full text), this duration, this outcome, and (when relevant) federation/cross-cluster summary fields.”

Query types

DSL only

ES|QL, DSL, SQL, and EQL

Threshold model

Often tiered (for example, multiple time thresholds per log levels) and per index.

Single duration gate at the cluster level (for example, “log if duration ≥ 500ms”)

What you get in each log line

Every line is one JSON object (one request) in a dedicated file (for example, *_querylog.json under your Elasticsearch log directory). Below is what you can do with the data:

Did it succeed, how long did it take, and what broke? Outcome (whether the request was successful or not), duration (took / took_millis, in line with the API), and a clear failure or timeout when something goes wrong. That’s the core signal for alerting, SLOs, and dashboards: “Are we green? If not, what’s the error?” You also get how many rows or hits came back (result_count), so you can separate “slow but empty” from “slow and huge.”

What actually ran? Query type (esql, dsl, sql, eql) plus the full query text. That answers “Which dashboard rule, saved search, or client pattern is hammering us?” Mix it with duration and outcome to find the worst offenders to fix or throttle.

Who asked for it, and how do I trace it end to end? X-Opaque-Id and trace ID tie a line back to Kibana or your own headers. Task and optional parent task IDs help follow work that was enqueued or chained (async or nested operations).

Cross-cluster search: Who participated, and did anyone misbehave? When cross-cluster search (CCS) is in play, the log can carry remote cluster aliases, per-cluster duration, and status (successful, failed, partial, skipped). You can see at a glance whether a slow search was local or a specific remote dragging the response. DSL can also record that a search was served from a remote alias; ES|QL exposes the richer cluster map; EQL logs a lighter view (for example, which remotes and how many) when remotes are involved.

Security (optional). With elasticsearch.querylog.include.user, you get the usual identity and realm fields (plus effective user when run-as applies), and API key metadata when applicable. Pair with query text and duration for governance and capacity conversations that use names, not only IPs.

There’s more available than we covered here, including additional execution details, shard-level outcomes, and optional profiling information depending on the query type. For every field path and setting, see the Elasticsearch documentation on query logs.

Where the logs live (and how to use them)

Logs land in your Elasticsearch log directory as *_querylog.json (for example, mycluster_querylog.json) on the coordinating node. Ship them with the querylog fileset in the Filebeat Elasticsearch module, so you can then inspect them in Discover (filter by event.dataset: elasticsearch.querylog). On Elastic Cloud, you need to enable Logs on your deployment, and the query logs are shipped as soon as you enable them.

Two workflows. If you need a one-off look to find out who’s hammering the cluster, what the query mix is, or a quick audit, just turn logging on, set a duration threshold so you only log what matters (for example, ≥ 1 s or ≥ 5 min), and then turn it off when you’re done. If you want ongoing query analytics, simply enable logging, point Filebeat at the log, and open a dashboard on the monitoring cluster. Two very simple steps, enable + ship, and you’re done. One request per line, one duration per request, no custom pipeline.

The dashboard below builds upon the new query logs and is provided out of the box. On the top row, you can find the P95/P99 query latencies (with an optional “acceptable latency” bar), the query type breakdown, the success and failure ratio, the user and system queries ratio, and (for DSL) hits versus aggregations. Underneath that, the latency over time (avg, p50, p95, p99, max) with a reference line so you can spot regressions, query volume over time (stacked by type), and tables for top indices, top users, and top error types. Filtering for cluster, user, or index lets you zoom into exactly what you want to focus on.

Heads up. Logging of queries is asynchronous, so it doesn’t block the query execution. Use the duration threshold to cap volume. Also worth noting that at very high queries per second (QPS), we may drop some lines rather than slow your cluster down. For analytics, shipping to a separate monitoring cluster keeps the cluster you’re debugging from taking the extra load.

Some configuration and code samples

Query logging is off by default. Flip it on in elasticsearch.yml or via the cluster settings API. Here’s how.

Enable query logging

In elasticsearch.yml:

Or dynamically via the cluster settings API:

Only log queries above a duration threshold

If you don’t want to log every health check or tiny request, simply set a threshold so only queries that run at least this long get an entry. Duration is in time units:

Include user/audit information

If you use the Security plugin and want to see who ran each query:

Log DSL searches that hit only system indices

By default, searches that target only system indices aren’t logged. To include them, enable query logging and set:

Example log entries

One line = one JSON object = one request with the same shape for ES|QL, DSL, SQL, EQL. Below: a successful DSL search and afailed EQL query with timestamp, duration, query type, and full query. On success, you get result count and shard stats, on failure an error block. User-inclusion and X-Opaque-Id show up when you’ve enabled them.

Success (DSL search):

Failure (EQL query):

Wrapping up

Elasticsearch query logs provide you with one single coordinator-level log for every query (ES|QL, DSL, SQL, EQL). One line per request, coordinator duration, full query, optional user and X-Opaque-Id. Enable it, set a duration threshold and user-inclusion if you want them, and you’re done. Logs live in your log dir (*_querylog.json), and when shipped with Filebeat, you can find them in Discover under the elasticsearch.querylog dataset.

Head to the Elasticsearch documentation on query logs for the full list of configuration settings, and field references. Slow or broken queries can also be found in AutoOps, which leverages the X-Opaque-Id to tie a long-running search back to its origin, such as a dashboard, a saved search, or an alerting rule.

Finally, it’s also worth noting that this new query log is an evolution of the ES|QL-only query log that we released in 9.2. We recommend adopting the new query log since it not only supports ES|QL queries, but also all your other queries.

Now, go see what’s actually running in your cluster.

Prerequisites

• Elasticsearch 8.15+ (for :: cast operator support; core ES|QL features available from 8.11)

Runtime fields versus ES|QL

Runtime fields were introduced in Elasticsearch 7.11 as a way to define fields at query time. Instead of reindexing data, you could write a Painless script that computes values on the fly:

This works, but comes with trade-offs:

• Painless scripting overhead: Every runtime field requires scripting knowledge, and the syntax is Java-like, not query-like.

• Performance cost: Runtime fields evaluate per document at query time. Elasticsearch classifies them as "expensive queries" that can be rejected by cluster settings.

• Isolated computation: Each runtime field computes independently. There’s no way to chain transforms or use the output of one field in another within the same query.

ES|QL changes the equation. It has its own execution engine (not translated to Query DSL), runs queries concurrently across nodes, and provides a complete toolkit for field computation: EVAL, GROK, DISSECT, type casting, and pipeline chaining.

Let's see how each runtime field pattern maps to ES|QL.

Setting up the example data

All the code snippets in this article can be executed in the Kibana Dev Tools console.

To follow along, create a sample index with data that exercises all five patterns. This simulates a server logs scenario with mixed field types, raw messages, and some intentional data quality issues:

Now index some sample documents:

Notice that response_time is stored as a keyword (a common real-world mistake), and the last document has "US-EAST" instead of "us-east" (a data quality issue we’ll fix later).

Pattern 1: Field concatenation

A common runtime field use case is combining two fields into one. For example, creating a host:port identifier.

The runtime field approach

You can define it inline at query time. Query-time approach avoids modifying the mapping, but you still need Painless scripting, scoping it to a single search request:

The ES|QL approach

You can run ES|QL queries using the _query API endpoint:

Response:

CONCAT accepts two or more arguments and always returns a keyword.

Note: For brevity, the remaining ES|QL examples in this article show just the query. Wrap them in POST _query { "query": "..." } to run them in Kibana Dev Tools.

When to use

If you need endpoint to persist across all queries and be available in Kibana dashboards, use a mapping-level runtime field. If you need it for a single search request within Query DSL, use a query-time runtime field. If you need it for ad-hoc analysis or exploratory work, ES|QL is simpler.

Pattern 2: Data extraction from unstructured text

Extracting structured data from raw log messages is another classic runtime field pattern.

The runtime field approach

Painless uses Java's regex Matcher class:

This is verbose. You need to know Painless regex syntax, handle the Matcher object, and call emit() correctly.

The ES|QL approach: GROK

ES|QL provides two purpose-built commands for text extraction. GROK uses regex-based patterns:

Response:

GROK uses the %{SYNTAX:SEMANTIC} pattern format. It extracts multiple fields in a single and readable command.

The ES|QL approach: DISSECT

For structured data with consistent delimiters, DISSECT is faster because it doesn’t use regular expressions:

The syntax is nearly identical to GROK, but DISSECT works by splitting on delimiters rather than matching regex patterns. This makes it faster for data that follows a consistent format.

When to use GROK vs DISSECT

Use DISSECT when your data has a predictable structure (same delimiters, same field order). Use GROK when you need regex flexibility, for example when fields may be optional or formats vary.

Pattern 3: Dynamic type conversion

When a field is mapped as keyword but contains numeric data (a surprisingly common scenario), runtime fields can cast it at query time.

The runtime field approach

You need to handle parsing exceptions manually. If Long.parseLong fails on an unexpected value, the script throws an error.

The ES|QL approach

ES|QL provides explicit conversion functions and a shorthand cast operator:

Or with the :: cast operator (available since 8.15):

Response:

Both produce the same result. The key difference from Painless: Failed conversions return null instead of throwing exceptions. The document with "not_available" simply gets null for response_ms, and ES|QL emits a warning.

Common conversion functions include:

Function

Converts to

`TO_LONG()`

Long integer

`TO_INTEGER()`

Integer

`TO_DOUBLE()`

Double

`TO_DATETIME()`

Date

`TO_BOOLEAN()`

Boolean

`TO_IP()`

IP address

`TO_VERSION()`

Version

The :: operator works with all these types (for example, field::double, field::datetime).

When to use

ES|QL's graceful null handling makes it safer for dirty data. Runtime fields with Painless give you fine-grained control over error handling but require more code. For type conversion specifically, ES|QL is almost always the better choice.

Pattern 4: Dynamic field handling

Runtime fields support "dynamic": "runtime" in mappings, which prevents mapping explosion by creating all new fields as runtime fields instead of indexed fields:

Any new field sent to this index becomes a runtime field automatically. This is useful when you ingest semi-structured data with unpredictable field names.

Where ES|QL fits

ES|QL provides query-time flexibility, but it still needs fields to be visible in the mapping. This is where runtime fields and ES|QL complement each other rather than compete.

If a field exists in _source but isn’t mapped, ES|QL cannot access it directly. The current workaround is to define a runtime field to make the unmapped field visible:

Once defined, ES|QL can query it:

This is one scenario where runtime fields remain essential. They act as a bridge, making unmapped data accessible to ES|QL.

Pattern 5: Field shadowing for error correction

Runtime fields can shadow (override) indexed fields by defining a runtime field with the same name as an existing field. This is useful for correcting data without reindexing.

The runtime field approach

Remember our data quality issue, where region has inconsistent casing ("US-EAST" versus "us-east")?

This overrides the indexed region field for all queries. Every search, aggregation, and Kibana visualization will see the lowercase version.

When you use EVAL with an existing column name, ES|QL drops the original column and replaces it with the computed value. This is the exact equivalent of field shadowing, but scoped to the current query.

You can also chain multiple corrections in a pipeline:

When to use

If the correction should apply to all queries and Kibana dashboards, use runtime field shadowing. If you need to correct data for a specific analysis, ES|QL is more flexible since you can apply different transformations in different queries without modifying the mapping.

The ES|QL pipeline advantage: Going beyond runtime fields

This is where ES|QL fundamentally surpasses runtime fields. Runtime fields are isolated: each one computes independently, and you cannot use the output of one runtime field as input for another in the same query.

ES|QL pipelines chain transforms. Here’s a single query that combines multiple patterns:

This single query:

• Extracts fields from raw text (GROK).

• Converts the duration to a number (EVAL with cast).

• Normalizes region casing (EVAL with TO_LOWER).

• Filters for errors with high duration (WHERE).

• Aggregates by region (STATS).

To achieve the same result with runtime fields, you would need to define at least three separate runtime fields (for extraction, conversion, and normalization) and then write a Query DSL query with filters and aggregations. The ES|QL version is a single, readable pipeline.

You can even use expressions directly inside aggregations:

Conclusion

What we covered:

• ES|QL provides a full toolkit (EVAL, GROK, DISSECT, type casting with ::) that replaces most runtime field patterns without any Painless scripting.

• Failed type conversions in ES|QL return null instead of throwing exceptions, making it safer for real-world data.

• Pipeline processing (chaining GROK into EVAL into WHERE into STATS) goes beyond what runtime fields can do in isolation.

• Runtime fields remain valuable for persistent computed fields, field shadowing across all queries, and as a bridge for unmapped data in ES|QL.

One important caveat: Both runtime fields and ES|QL compute values at query time, which means they pay the cost on every query. If you find yourself applying the same transformation repeatedly (type corrections, field extraction, data normalization), consider using ingest pipelines to fix the data at index time instead. Ingest pipelines let you parse, enrich, and transform documents before they’re stored, so queries can work with clean, properly typed fields directly. Runtime fields and ES|QL are great for exploration and ad-hoc analysis, but for production workloads, indexing the right data from the start is almost always the better choice.

The key takeaway: Runtime fields aren’t deprecated, and they aren’t going away. But for most query-time computation patterns, ES|QL offers a simpler, more powerful, and more performant approach. And when the transformation is known up front, an ingest pipeline is the most efficient option of all.

Next steps

• ES|QL documentation

• Runtime fields reference

• ES|QL timeline of improvements

• Getting started with runtime fields

• ES|QL processing data with DISSECT and GROK