If you run Observability on Elastic Cloud Hosted (ECH), this will feel familiar. Cross-cluster search (CCS) has long enabled teams to query across remote clusters, and it's a mainstay in large scale ECH environments. CPS takes the same idea to Serverless, and makes linking even simpler: no topology mapping, certificate configuration or per connection credentials. You link projects in the Cloud console and Elastic manages the rest.

What is cross-project search?

Serverless organizes your data into projects: Elasticsearch, Observability, or Security, each with its own scaling, permissions, and lifecycle. That isolation is a feature: it preserves data locality where needed, simplifies cost attribution, and lets teams own their own data. The tradeoff, historically, has been visibility. If your logs live in one project and your APM data lives in another, or if every business unit runs its own Observability project, getting a unified view has typically meant duplicating data into a shared store.

CPS removes that tradeoff. You designate an origin project (the project you search from) and link it to other projects you want included in that search. Once linked, an unqualified query like FROM logs* or a Discover session automatically spans the origin plus every linked project; no special syntax, no remote index prefixes. Kibana's project picker lets anyone narrow a query to just the current project or expand it to everything in scope, and administrators can set a default scope per Kibana space so a local incident-response space stays local by default while a global operations space starts broad. Access is enforced per project using your existing Elastic Cloud roles, so linking a project for search doesn't quietly grant broader permissions than a user already has.

What's the difference between cross-project search and cross-cluster search?

Cross-cluster search (ECH, self-managed)

Cross-project search (Serverless)

Where you link

Remote cluster connections, configured per cluster

Link projects in the Elastic Cloud console

Credentials

Certificates or API keys you issue and rotate

Managed by Elastic

Network setup

Network topology you map and maintain

None to configure

Query syntax

Remote cluster name referenced in the query

Unqualified query, no remote index prefix

Scope control

Set per query

Project picker, per-space defaults, and tag-based routing

Where CCS asks administrators to configure remote cluster connections, exchange certificates or API keys, and reference remote cluster names in every query, CPS asks you to link projects in the UI and query the way you already do. It's the same underlying idea: remote search without moving data, adapted to a control plane where infrastructure is meant to disappear.

Why cross-project search matters for Observability teams

Cross-project search matters most the moment your observability data stops living in one place. Observability organizations typically split their data by team, environment, region, or customer, usually for good reasons: isolating noisy neighbors, controlling blast radius, meeting data residency requirements, or giving each team ownership over its own ingest and retention. The problem shows up during an incident or a cross-cutting investigation, when someone needs to see across those boundaries without waiting for a data migration or standing up a consolidated cluster.

A few examples of where CPS changes the equation:

Keep observability data in its region and still search globally

Teams operating in multiple regions or clouds often need to keep observability data in the region it was generated, whether for regulatory reasons or to limit cross-region egress. CPS lets each region keep its own Observability project while a central project links to all of them, so an SRE team can run one query to check whether an anomaly in one region is also showing up elsewhere, without pulling all that data into one place first.

Isolate environments without losing the incident view

Splitting production, staging, and per-service or per-team projects makes scaling and cost predictable, and it limits how far a bad deploy or a runaway job can spread. It also means an on-call engineer used to have to check several places during an incident. With CPS, on-call can work from a single origin project and pull in whichever linked environments matter to the investigation, then narrow back down once things are resolved.

Central teams monitoring many business units or customers

Platform or Observability Center of Excellence (CoE) teams that operate on behalf of many business units, or Managed Service Providers (MSPs) monitoring several customer environments, can give each unit or customer its own project, with its own retention, access, and cost, while the central team links across all of them for cross-cutting reporting and anomaly detection. Each project owner keeps control of their own data; the center gets the aggregate view it needs.

Avoid ingesting the same data twice

When two teams need the same data for different purposes, the old options were duplicate ingestion pipelines or a shared, less isolated deployment. With linked projects, data is ingested once and queried from wherever it's needed, which keeps both storage costs and pipeline maintenance down.

This applies across solution types too: CPS links Elasticsearch, Observability, and Security projects interchangeably, so a Security team and an Observability team can each keep their own project, with their own retention and access controls, while querying the same underlying logs. A security analyst investigating an alert can query infrastructure or service context from a linked Observability project in Discover, dashboards, or ES|QL, and an SRE debugging an incident can query related signals from a linked Security project the same way, all without either team re-ingesting the other's data.

What does cross-project search support in Elastic Observability?

At GA, CPS is available across Elasticsearch, Observability, and Security project types (Observability and Security projects require the Complete tier). For Elastic Observability specifically, application and infrastructure monitoring, alerting rules, SLOs, machine learning anomaly detection (including APM anomaly detection), and Synthetics work across linked projects, alongside the platform-level capabilities every project gets: Discover, dashboards, and ES|QL. Project routing lets you target a query to a subset of linked projects using predefined tags (alias, cloud provider, region) or your own custom tags, combined with boolean logic, so "all EU projects" or "every environment except staging" can be expressed once and reused.

This release also raises the default number of projects you can link to a single origin, well beyond the tech preview limit, with room to go higher for larger deployments. Any existing project can now act as an origin rather than only newly created ones. For programmatic access, you only need a Cloud API key to call CPS from your own tooling. A new dedicated cloud role can now grant those keys for programmatic access.

A few nuances are worth knowing before you design around CPS:

• Alerting: Alerting rules defined in the origin project can evaluate data across linked projects. The alerts they generate stay local to the project where the alerting rule lives.
• SLOs: SLOs don't follow the project picker in the Kibana header. Instead, you set scope in the SLO create and edit form, choosing this project, all linked projects, or a specific subset, since changing scope rebuilds the SLO's underlying computation. You can't view SLOs created in linked projects from the current project, but you can create an SLO here based on data that lives in those projects.
• Synthetics: Overview, monitor details, certificates, errors, and journeys honor the Kibana header project picker and show linked-project monitors as read-only. Creating, editing, and managing monitors stay in the origin project, and Synthetics status and TLS rules still evaluate origin monitors only.
• Streams: Not CPS-aware yet; Streams stay scoped to the project where they were created.

How do you set up cross-project search?

Setting up cross-project search takes a few minutes:

1. From your origin project's settings in the Elastic Cloud console, open Link projects.
2. Select the projects you want to link, individually or in bulk.
3. Save.
4. Open Discover, run an ES|QL query, or build a dashboard as you normally would: linked project data is included automatically, and the project picker lets you narrow or widen the scope at any time.

For the full configuration walkthrough, including linking topologies (hub-and-spoke, shared data project, or full mesh) and how to scope project routing with tags and expressions, see the cross-project search documentation and the Observability-specific guide.

Cross-project search pricing

CPS pricing has two components: a charge for data retained in each linked project you query, and a charge for data egress generated by cross-project query traffic, tracked as its own line item so you can see what CPS is adding to your transfer costs. There's no separate per-link fee. Because actual cost depends on your data volumes, retention settings, and how many projects you link, use the Elastic Cloud Serverless pricing estimator to model costs for your own environment against your own ingest and retention numbers.

Start using cross-project search

If you're already running Elastic Observability on Serverless, cross-project search is available in your project settings today: link your projects and run your first cross-project query in Discover. If you haven't tried Elastic Cloud Serverless yet, start a free trial and see how far a single query can reach.

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

Elastic, Elasticsearch, and associated marks are trademarks, logos, or registered trademarks of Elasticsearch B.V. in the United States and other countries. All other company and product names are trademarks, logos, or registered trademarks of their respective owners.

The gain is in where the intermediate set lives. When you ask what the saturated hosts logged, the list of saturated hosts is computed and consumed inside Elasticsearch. Six host names or 500 trace IDs never land in a clipboard or in an AI agent's context window, and the set is recomputed from current data every time the query runs.

That changes the unit of investigation. You go from "one slow request stalled on a lock" to "most of the slowest requests did," and only the second answer tells you which team to page. It matters more the further apart your signals are: in many observability stacks, logs, metrics, and traces live in three separate systems, each with its own query language, its own time picker, and its own idea of what a host is, so the same question has to be asked two or three times and the answers joined by hand.

In this post we walk through four investigations, each of which is self-contained. For every one of them, we set out the scenario that started it, the query that answers it, the table it returns, and a note on the difficulties you run into when you try to get the same answer any other way.

| Pattern | Starts from | Question it answers | |---|---|---| | Metrics to logs | CPU saturation | Which error patterns show up only on the saturated hosts? | | Logs to metrics | Error logs | Do the erroring hosts look any different from the healthy ones? | | Traces to logs | Slow spans | What did every service log during those specific requests? | | All three signals | Pod memory pressure | Which log lines sit behind the requests that failed under that pressure? |

The data was collected with OpenTelemetry and lands in the logs-*.otel-*, traces-*.otel-*, and metrics-*.otel-* data streams, where fields keep their semantic convention names rather than being rewritten into another schema. The correlation pattern works just as well on Elastic Agent integrations, though the queries need translating rather than just renaming: ECS carries log severity as the text field log.level instead of a numeric severity_number, the System integration reports CPU as separate system.cpu.*.pct fields instead of one metric with a state dimension, and APM records durations in microseconds. All of it lands in the same cluster either way, which is the part the subquery depends on.

In Discover, the time picker already applies the range, so the examples below omit an explicit @timestamp filter. Outside Discover, add a filter by time yourself, either with literal timestamps in the query or with ?_tstart and ?_tend in the query and values in the params array of your _query request. Every result below comes from a one hour window over a synthetic fleet of 300 hosts.

Metrics to logs: what are the saturated hosts complaining about?

An infrastructure alert tells you a handful of hosts in a fleet of a few hundred sat above 90% CPU over the last hour. That tells you which hosts are hot and nothing about why. The question worth answering is whether those hosts share a failure mode, or whether they are busy for unrelated reasons and the alert is a coincidence.

FROM logs-*.otel-*
| WHERE severity_number >= 17
  AND resource.attributes.host.name IN (
      TS metrics-hostmetrics.otel-*
      | WHERE attributes.state == "idle"
      | STATS idle = AVG(AVG_OVER_TIME(metrics.system.cpu.utilization))
          BY resource.attributes.host.name
      | WHERE idle < 0.1
      | KEEP resource.attributes.host.name
    )
| STATS errors = COUNT(*), hosts = COUNT_DISTINCT(resource.attributes.host.name)
    BY pattern = CATEGORIZE(body.text), service = resource.attributes.service.name
| SORT errors DESC

The two halves of the query map onto the two halves of the question. The subquery works out which hosts were saturated, averaging CPU utilization per host and keeping the ones that averaged under 10% idle, which is another way of saying above 90% busy for the window. The outer query then works out what those hosts were complaining about, pulling their error logs and using CATEGORIZE to collapse thousands of individual lines into a handful of error classes.

Two choices in there are worth pausing on.

The subquery uses TS rather than FROM because a host does not report one CPU number. It reports a separate time series per CPU state, and per logical core as well if your collector is configured to break them out, so the reduction has to happen in two stages. AVG_OVER_TIME collapses each series to a single value first, and the outer AVG then combines those into one figure per host.

Naming that inner function matters more than it looks. Write AVG(metrics.system.cpu.utilization) on its own and TS supplies LAST_OVER_TIME for you, averaging each series' final sample rather than its average over the window. In this dataset that one substitution moves a host from 7% idle to 10% idle, which is the difference between appearing in the results and not. Querying metrics with the TS command goes into the two aggregation phases in more depth.

The log filter tests severity_number (17 is the ERROR floor on the OpenTelemetry scale) rather than the severity text, because the numeric scale is fixed by the spec while the text is whatever the emitting library decided to write. That is not a hypothetical distinction here: the error logs in this cluster carry four different labels, including SEVERE from a Java service. Matching on the text alone returns 2,754 of checkout's errors and misses the billing service entirely, while the numeric filter returns all 5,910 of them and keeps billing too.

When you run the query in Discover, the result is a short table of error classes, scoped to the saturated hosts:

The subquery returned six saturated hosts, and on all six the same service is timing out against an upstream dependency and draining its connection pool. The hosts column is what lets you set the other two rows aside without opening anything: the certificate errors reach only two of the six, and the gateway declines amount to five lines in an hour. Neither tracks the cohort the way the checkout patterns do.

Having all three signals in one store already removed the exports from this investigation. The subquery removes the step after that, and closing that last gap matters more than it sounds. By the time you have read six host names off a chart and typed them into a log search, the set has moved: a host that crossed the threshold a minute ago is missing from your list, and one that has since recovered is still in it. Here the host list is derived from current data on every run, so re-running the query during an incident gives you the current cohort.

The stale list is only half of it. A correlation done by hand exists only in the head of the person who did it, so nobody else can check it, save it, or run it again tomorrow.

Logs to metrics: do the erroring hosts look different from the healthy ones?

Filtering metrics by a log-derived host set answers the opposite question. The payments service is throwing errors on some hosts and not others, and you want to know whether resource pressure explains the split before you start reading deploy history.

That is a comparison, so the query needs both cohorts. FORK runs two branches over the same input, and IN and NOT IN against the same log-derived host set divide the fleet between them.

TS metrics-hostmetrics.otel-*
| WHERE attributes.state == "idle"
| STATS idle = AVG(AVG_OVER_TIME(metrics.system.cpu.utilization))
    BY host = resource.attributes.host.name
| EVAL busy = 1 - idle
| FORK
    ( WHERE host IN (
        FROM logs-*.otel-*
        | WHERE severity_number >= 17
          AND resource.attributes.service.name == "payments"
          AND resource.attributes.host.name IS NOT NULL
        | STATS errors = COUNT(*) BY resource.attributes.host.name
        | KEEP resource.attributes.host.name )
      | EVAL cohort = "logging errors" )
    ( WHERE host NOT IN (
        FROM logs-*.otel-*
        | WHERE severity_number >= 17
          AND resource.attributes.service.name == "payments"
          AND resource.attributes.host.name IS NOT NULL
        | STATS errors = COUNT(*) BY resource.attributes.host.name
        | KEEP resource.attributes.host.name )
      | EVAL cohort = "no errors" )
| STATS hosts = COUNT(*), mean_busy = AVG(busy), busiest_host = MAX(busy)
    BY cohort

The query reads top to bottom as three stages. The metrics query runs first and reduces the whole fleet to one busy figure per host. FORK then splits that fleet in two using the same log query in both branches, separating the hosts that appear in it from the hosts that do not. The final STATS summarizes each group, so both cohorts come back as two rows of one table, measured the same way over the same window.

This query is longer than the others, and three parts of it are less obvious than they look.

The natural way to label the two cohorts would be EVAL cohort = CASE(host IN (...), "erroring", "healthy"), and ES|QL rejects it. In 9.5 an IN subquery has to be a top-level predicate in a WHERE condition rather than an argument to a scalar function, which is why the split happens at the command level with FORK.

The STATS ... BY inside each subquery looks redundant, since KEEP alone would return the same host names. It is not: without it, the subquery returns one row per matching log document instead of one row per host, and those rows are all held in memory for the outer query to filter against. Aggregating first turns millions of rows into a few hundred host names.

The IS NOT NULL filter guards the sharpest edge here, and this dataset is a live example rather than a hypothetical. NOT IN follows SQL null semantics, so a single null in the subquery result makes the predicate match nothing at all. Five of the payments error logs in this cluster came through a sidecar that dropped the host name. Remove that one line from both branches and the query still succeeds, but it returns a single row: the nulls quietly delete the entire 286-host "no errors" cohort, and what is left looks like a perfectly plausible answer to a different question.

Run the query in Discover and you get two rows, one per cohort:

CPU does not explain the split, and the numbers say so twice. The 14 erroring hosts run slightly cooler on average than the 286 quiet ones, and the busiest machine among them averaged 50% over the hour while the quiet cohort contains a host that averaged 95%. Whatever is failing on those 14, they had headroom the entire time, and deploy history is a better place to spend the next ten minutes.

A negative result like this is worth as much as a positive one, and it is usually the one people skip. Getting it the long way means running the metrics query twice against two hand-built host lists and lining the numbers up afterwards, which is enough friction that the check often just does not happen. Both cohorts here come from the same log query in the same execution, over identical time windows, so there is nothing to reconcile and no reason not to check.

Traces to logs: what did every service log during the slow requests?

A service level objective (SLO) burn alert fires on checkout latency. Tracing gives you the slow requests and their spans, and the next question is what the services involved were writing to their logs while those specific requests were in flight.

The trace ID is the join key, and there are far too many of them to move by hand.

FROM logs-*.otel-*
| WHERE trace_id IN (
      FROM traces-*.otel-*
      | WHERE kind == "Server"
        AND resource.attributes.service.name == "checkout"
        AND name == "POST /api/orders"
        AND duration > 2000000000
      | SORT duration DESC
      | LIMIT 500
      | KEEP trace_id
    )
| STATS lines = COUNT(*), traces = COUNT_DISTINCT(trace_id)
    BY pattern = CATEGORIZE(body.text),
       service = resource.attributes.service.name,
       severity_text
| SORT traces DESC

The subquery answers "which requests were slow." It looks at the inbound request span for the checkout endpoint rather than the client and internal spans beneath it, then keeps the 500 slowest requests over two seconds. Durations are recorded in nanoseconds, which is why the threshold has so many zeros.

The outer query answers "what got logged while they were running," gathering every log line that shares one of those trace IDs and grouping them into patterns.

Counting distinct traces per pattern is what makes the output readable. A log pattern that appears 30,000 times across four traces is one chatty request, while a pattern that shows up in 470 of the 500 slowest traces is a property of being slow.

From the results above, the top row is there by construction and can be set aside: checkout writes one order submitted line per order, so it appears in all 500 traces and says nothing about why these particular 500 were slow.

From the results above, the third row is the answer, and it points at a service nobody was looking at. A lock wait timeout in inventory, two hops downstream from where the alert fired, shows up in 470 of the 500 slowest requests, and the 947 lines behind those 470 traces mean a good share of them retried more than once. The payment gateway declines are real failures, and at 19 traces out of 500 they are not what is burning the SLO.

Done by hand, this means opening slow traces one at a time and reading the correlated logs for each, which is tedious at ten traces and nobody's idea of a plan at 500. When the spans and the logs are held in different systems, every trace you check is a copied ID and a context switch, and the sample size you can afford drops to about three. Three traces is enough to form a theory and not enough to test one. Treating the slow requests as a population is what turns "this trace had a lock wait" into "470 of the 500 slowest requests had a lock wait," and that difference decides whether you page the inventory team.

All three signals: from pod memory pressure to the log lines behind the failures

IN subqueries nest, so the pattern extends to as many signal types as the question needs.

A node pool starts reporting memory pressure after a rollout. You want the log lines from the requests that actually failed on the pods under pressure, which means going from metrics to traces to logs without stopping in between.

FROM logs-*.otel-*
| WHERE trace_id IN (
      FROM traces-*.otel-*
      | WHERE kind == "Server"
        AND status.code == "Error"
        AND resource.attributes.k8s.pod.uid IN (
            TS metrics-kubeletstats.otel-*
            | STATS peak = MAX(MAX_OVER_TIME(metrics.k8s.pod.memory_limit_utilization))
                BY resource.attributes.k8s.pod.uid
            | WHERE peak > 0.95
            | KEEP resource.attributes.k8s.pod.uid
          )
      | STATS failures = COUNT(*) BY trace_id
      | SORT failures DESC
      | LIMIT 1000
      | KEEP trace_id
    )
| STATS lines = COUNT(*), traces = COUNT_DISTINCT(trace_id)
    BY pattern = CATEGORIZE(body.text), service = resource.attributes.service.name
| SORT traces DESC

Reading the query inside out, you can see each layer answering one part of the question. The innermost subquery identifies the pods whose memory peaked above 95% of their limit, using MAX_OVER_TIME to take each pod's peak rather than its average. The middle one narrows to the requests that actually failed on those pods and reduces them to at most 1,000 trace IDs. The outer query then collects the logs for those traces from every service that took part, including services running on pods that were entirely healthy, and that last part turns out to be where the answer is.

The pods are matched on their UID rather than their name, because names repeat across namespaces and restarts. The query assumes Kubernetes metadata reaches your spans, which is what the collector's k8sattributes processor is for; if it does not, the host or container ID works the same way.

From the results above, the first row sits at exactly 1,000 because that is the subquery's LIMIT, so it describes the size of the sample rather than the size of the incident. Every trace in that sample carries the cart deadline line, which is the symptom you already knew about when you started.

Look at the second row instead. product-catalog is rejecting oversized payloads across 946 of the same 1,000 traces, and it never appeared in the pod subquery at all: its pods peaked at 75% of their memory limit, well under the 95% threshold. The rollout started sending larger payloads, which would account for both cart's memory climb and the failures. Checkout's retry budget gives out in about half of them, which is how the failure became visible to users.

Filtering logs directly by the pressured pods would have shown you the cart line and hidden the product-catalog one, which is to say it would have confirmed the symptom and buried the cause. Doing it without subqueries means three queries and two hand-built lists, and the second list is a thousand trace IDs. That is usually the point at which people stop after the first hop and go with the cart theory. The reason the second hop is cheap here is that all three signals sit in the same store behind the same query language, so widening from pods to traces to every service in the trace is a clause, not a project.

Why do ES|QL subqueries matter for AI agents?

Keeping the intermediate set inside the cluster is convenient for a person and close to essential for an agent querying on your behalf.

Split across two tool calls, the intermediate result has to travel. A list of 500 trace IDs comes back in a tool response and occupies the model's context, and the agent then has to rewrite every one of them into the next query. That costs tokens on every hop, and it is where truncation and transcription errors come from. With a subquery, the intermediate set stays inside Elasticsearch and the agent only ever sees the final table.

The problem compounds when the signals are spread across systems. An agent then needs credentials, a client, and a working knowledge of the query language for each one, plus the judgment to join results that use different names for the same host. One store and one query language reduce that to a single skill the agent has to be good at.

One ES|QL string is also a complete description of the correlation, which makes the investigation reproducible: an agent can put the query in its summary, and a human can paste it into Discover and get the same logic evaluated against current data. A two-call sequence with a hardcoded host list in the middle gives you neither. The one thing to watch is that an agent calling the _query API has to filter @timestamp itself, since nothing is binding a time picker.

Four things to know before writing ES|QL IN subqueries

IN subqueries are in technical preview in Elastic Stack 9.5 and on Elastic Cloud Serverless, while TS and FORK have been generally available since 9.4. CATEGORIZE has been generally available since 9.1 and requires a Platinum license; every query above works without it if you group by an existing field instead.

Four things are worth knowing before you write your own:

• In 9.5 the subquery returns exactly one column, which is what the trailing KEEP does in each example.
• Aggregate the subquery down to distinct values with STATS ... BY before returning them. Its result is materialized for the outer query to filter against, so handing back a few hundred host names instead of a few million rows is both faster and safer.
• Filter nulls out of any NOT IN subquery, because SQL null semantics mean one null makes the predicate match nothing.
• Subqueries are non-correlated. They run independently and cannot reference columns from the outer query, so this is a set filter rather than a row-by-row join. Reach for LOOKUP JOIN when you need per-row enrichment, which we covered in ES|QL joins for richer observability.

Try ES|QL signal correlation on your own data

The pattern under all four examples is the same. You start with a set you can describe in one signal and a question you can only answer in another, and the subquery carries that set across the boundary for you.

The syntax is the smaller part of what makes that work. It works because logs, metrics, and traces sit in one store behind one query engine, under field names they kept on the way in, so crossing from one signal to another is a clause in a query rather than an integration to build and maintain. Where that is not true, the same four investigations turn into a sequence of exports, translations, and manual joins, and that costs more than slower answers. It quietly shrinks the number of questions anyone is willing to ask, and the negative results are the first to go.

Each pattern here replaces two or three queries with one, and no host list or trace ID list has to move between them. Fewer steps mean fewer places to be wrong, and a correlation you can save as a single string and hand to someone else.

To try it:

1. Open an Observability project on Elastic Cloud Serverless, or upgrade to Elastic Stack 9.5.
2. Send data with the Elastic Distributions of OpenTelemetry, or point an existing collector at Elasticsearch.
3. In Discover, switch to ES|QL and start from the metrics to logs query above, swapping in your own data streams and thresholds.
4. Read the IN subquery reference for the full set of commands you can use inside a subquery.

• Elasticsearch is a Prometheus-compatible metrics backend — Prometheus Remote Write and PromQL now works natively in Kibana, no translation layer required.

• Metrics land in Elasticsearch's columnar TSDS architecture storing data up to 2.5× more efficient than Prometheus and 2× more efficient than ClickHouse.

• ES|QL time series queries run up to 30× faster than Prometheus on gauge averages and counter rates, including high-cardinality workloads.

• Elastic costs approximately 50% less than Datadog, with no custom metric classification and no cardinality-based billing.

• Grafana can query Elasticsearch directly through the native Prometheus API, keeping your visualization layer while replacing the backend.

• Kubernetes and AWS monitoring ship with pre-built dashboards, alert templates, ML anomaly jobs, and agentic investigation content ready at ingest. Additionally skills and MCP apps are available.

• Unified backend for Metrics, logs, and traces enabling agentic investigations without stitching context across tools.

• Metrics exploration in Discover lets anyone start querying and analyzing metrics immediately, no query language expertise required.

• Custom dashboarding is fast and flexible — dashboards-as-code, AI-assisted dashboard creation, variable controls, and collapsible panels mean less time building and more time investigating.

• Migration tooling to help easily migrate dashboards and alerting rules / monitors from Datadog and Grafana.

Elasticsearch metrics now competes on every dimension that matters to SREs: you can afford to keep every metric at full resolution, query it up to 30x faster than Prometheus, pay 50% less than Datadog, migrate dashboards and alerting rules from Grafana or Datadog easily, and go from alert to root cause without stitching context across disconnected tools. The rest of this post walks through each of these in detail.

Elasticsearch metrics performance: 30× faster than Prometheus and Mimir

Datadog and Prometheus force the same tradeoff: drop high-cardinality data or watch costs spiral. SREs managing Kubernetes, AWS, or any high-cardinality infrastructure know the specific shape of this problem. The Kubernetes labels, ephemeral pod data, and fine-grained OTel dimensions that matter most during an incident are the first to go when budgets tighten.

Elastic rebuilt the time series data store and ES|QL compute engine into a fully columnar metrics engine. Adding a new Kubernetes label, a new AWS instance tag, or a new application dimension doesn't strain the system; it adds far less cost than systems that index every label. OTel, Prometheus, and application-defined metrics all land in the same columnar backend at full resolution, with logs, traces, and metrics in a single store. No data dropped, no retention shortened.

Elasticsearch stores metrics up to 2.5× more efficiently than Prometheus (results may differ due to factors like compaction), and 2× more efficiently than ClickHouse. Query performance via ES|QL runs up to 30× faster than Prometheus on gauge averages and counter rates, including high-cardinality workloads where competitors stall. The architecture post covers how TSDS is organized and why the columnar layout produces these results.

| | | | | | :------------------------: | :----------------: | :--------------: | :----------------: | | Dimension | vs. Prometheus | vs. Mimir | vs. ClickHouse | | Query performance (ES|QL) | Up to 30× faster | Up to 30× faster | Up to 8× faster | | Storage efficiency | Up to 2.5× better | On par | 2× better |

The key architectural difference is that Elasticsearch metrics does not maintain a per-series in-memory state that scales with cardinality, so adding thousands of new Kubernetes pod labels or OTel dimensions doesn't drive up memory pressure.

OTel, Prometheus-native, and application-defined metrics are all stored the same way at full resolution, queried fast, at half the cost of Datadog.

Elastic Observability metrics pricing without the Datadog custom metric penalties

Observability cost is the #1 reason teams switch platforms. For Datadog customers, the pain comes down to one pricing mechanic: custom metrics. Any user-defined value outside of Datadog's built-in integrations is classified as a custom metric and billed at a premium rate. That includes the high-cardinality data that Kubernetes, OpenTelemetry, and cloud-native workloads generate by default. The more granular your instrumentation, the faster the bill compounds. Teams running modern infrastructure hit this ceiling quickly, and the response is predictable: drop data, shorten retention, lose the context that matters most when an incident happens.

Elasticsearch metrics removes that classification. Every metric is priced the same, with no per-metric penalties, no cardinality-based billing, and no forced rollups. You keep every metric at full resolution without a surprise invoice at the end of the month. And because Elastic is 50% the cost of Datadog, the conversation with finance changes: not what data you had to drop to stay on budget, but what you found because you kept everything. It's also why the AI investigation works. Unlike Grafana's fragmented LGTM stack, the context is already unified when the alert fires, not assembled by hand across disconnected tools.

Native Prometheus and PromQL support in Elasticsearch

Most SRE teams aren't running a clean, single-format telemetry pipeline. Prometheus is deeply embedded in applications, services, platforms, and automations. Migrating metrics backends historically meant rewriting queries, rebuilding dashboards, and retraining engineers — enough friction that teams stay on platforms they've outgrown rather than go through it.

Elasticsearch metrics has removed most of that friction. Prometheus metrics arrive via Prometheus Remote Write and land in the same columnar store without semantic changes, preserving full metric fidelity end to end. Point them at Elasticsearch instead of Mimir and the data flows. No translation layer, no changes to existing scrape configs.

PromQL now works natively in Kibana, so engineers who live in PromQL don't have to change how they work. Existing PromQL queries, dashboards, and alert rules migrate into Kibana directly. 

PromQL queries work unchanged on Elasticsearch

If your team already writes PromQL, nothing needs to change. These queries run as-is against Elasticsearch as your backend — copy, paste, and go.

CPU usage rate (container-level) The per-second CPU rate across containers, grouped by pod. Useful for spotting which pods are burning CPU during an incident.

PROMQL sum by (pod) (rate(container_cpu_usage_seconds_total[5m]))

Memory working set (container-level) Current memory in active use per container — the number that matters for OOM risk, not total allocated memory.

PROMQL sum by (container) (avg_over_time(container_memory_working_set_bytes[5m]))

HTTP request rate (application-level) Per-second request throughput grouped by instance. A standard first signal when investigating latency or error spikes.

PROMQL sum by (instance) (rate(http_requests_total[5m]))

All three follow standard PromQL syntax. If you use Elasticsearch as your backend, they run without modification. For the full syntax reference and what's covered, see the PromQL support documentation.

The native Prometheus API makes Elasticsearch a fully Prometheus-compatible backend. Any Prometheus-compatible frontend (Grafana included) can query Elasticsearch directly, so teams that want to keep Grafana as their visualization layer while consolidating onto Elasticsearch can do exactly that without modifying existing dashboards or alert rules.

When SREs need to go deeper than PromQL allows, ES|QL works across metrics, logs, and traces in a single interface. The TS command handles the time series specifics: counter rates, gauge averages, window functions, and multilevel aggregations across high-cardinality dimensions. The same query that pulls a CPU counter rate can join against logs from the same host and surface the deployment event that preceded the spike. No tool switching, no new query language. The query language, the dashboards, the alert rules, the visualization layer — all of it carries over. The only thing that changes is that Elasticsearch is the single backend powering everything.

Elastic Observability: out-of-the-box dashboards, alerts, and infrastructure content

Most Observability vendors require you to build everything from scratch. Elastic Observability has reduced this need across three areas:

Metrics exploration in Discover. The new Elasticsearch metrics exploration experience lets SREs explore metrics in the same interface used for logs — no tab switching, no duplicate queries. Connect an OTel pipeline or Prometheus scrape config, open Streams, and every metric in the data stream renders as a time series chart immediately. No dashboard to build, no query to write. This is where teams can validate data, spot patterns, and start building alerts and SLOs from a live view of what's flowing and cross correlate with logs, traces and other indexed data in Elasticsearch.

Dashboards. Kibana dashboards have gained collapsible panels with lazy loading, so panels that aren't immediately visible don't generate queries until they're needed and ES|QL control variables that let SREs manipulate visualizations through dropdowns without writing new queries. Dashboards-as-code is also shipping, enabling version-controlled dashboard definitions that can be templated, shared, and deployed programmatically across environments.

Out-of-the-box infrastructure content.  Elastic is shipping with two new infrastructure OOTB experiences:

• The new Kubernetes integration ships with hierarchical dashboards, alert rule templates, ML anomaly detection jobs, and the context and prompts needed for AI-assisted root cause analysis — all pre-configured and ready the moment data starts flowing. 

• AWS infrastructure monitoring follows the same pattern: OOTB content for core AWS services activates at ingest, so teams aren't starting from scratch every time a new service or account comes online. The same approach extends to databases and other core infrastructure — the platform arrives opinionated, not blank.

Agentic investigations across your infrastructure with Elastic Observability

Elasticsearch correlates metrics, logs, and traces in a single backend, so the investigation context is assembled before an engineer is paged.

The hard part is 2am. An RDS instance hitting connection limits, starving services upstream. An Auto Scaling group failing health checks for a reason buried in application logs. A pod restart cascading across a namespace.

In a Grafana LGTM stack, you're opening three tabs before you have enough context to form a hypothesis.

In Datadog, the context is unified but the AI is a black box: no BYO-LLM, no data residency options.

In Elastic, metrics, logs, and traces share a single backend and a common schema, so the investigation context is already assembled when the alert fires — no manual correlation across tools, no context lost in translation between query languages. ML anomaly detection runs automatically against infrastructure metrics (Kubernetes, AWS, databases), so the investigation starts from a scored anomaly with context about what's typical, what changed, and how severe the deviation is, not just a raw threshold breach.

When an alert fires, Elastic's investigation workflow correlates signals, assembles root cause context, and surfaces recommended next steps before anyone is paged. The agentic Kubernetes observability post walks through a complete example end to end. The EKS troubleshooting walkthrough shows how Agent Builder and MCP work together for a full root cause loop across EC2, EKS, and related AWS services.

In addition to investigating issues in Elastic Observability, you can use Claude, Cursor, VS Code, or your favorite tool to analyze issues using MCP Apps and agent skills from Elastic. The Observability MCP App extends the analysis to wherever your team already works. If your team investigates in Claude, Cursor, or VS Code, the same investigation capabilities (infrastructure health rollup, service dependency graph, anomaly detail, blast radius analysis) render as interactive views directly in the conversation. Neither Grafana nor Datadog offer this.

• Observability MCP App — Connects Claude, Cursor, VS Code, or any MCP-compatible tool directly to your Elasticsearch data, so infrastructure health, service dependencies, and anomaly context surface as interactive views inside the conversation without leaving your tool of choice. See how it works with Kubernetes.

• Agent Skills — Pre-built skills for Kubernetes, AWS, and other core infrastructure let any agent — in Elastic or your own — run structured investigations against your observability data without custom prompt engineering. Drop them into Claude, Cursor, or your own agent pipeline and they work out of the box. Explore the observability skills or browse the skills library on GitHub.

Migrating from Datadog or Grafana to Elastic Observability

The most common reason SRE teams don't switch observability platforms is migration. Moving years of alert rules, hundreds of dashboards, and runbook-embedded PromQL queries is a daunting operational task, and the cost of maintaining parallel stacks while doing it compounds every day.

The Observability Migration Platform handles the translation automatically. Point the CLI or Claude/Cursor (with Elastic’s agent skills) at your Datadog org or Grafana instance and it converts supported dashboards, alert rules, and PromQL queries into Kibana-native outputs. The tool allows you to see what was fully migrated, what needed tweaks and what is needed from you to migrate everything. You move what you've already built.

On the ingest side, Prometheus Remote Write means the pipeline requires no changes. Scrape configs point to Elasticsearch instead of another Prometheus-compatible backend and the data lands in the same columnar store. Workflows, queries, and alert configurations carry over without change. For teams that want to keep Grafana as a visualization layer during or after migration, the native Prometheus API and PromQL support in Kibana mean the transition can be phased rather than cut over all at once.

Elasticsearch as a backend for Grafana

For teams not ready to leave Grafana, replacing the backend is a migration path in its own right, and there are two ways to do it depending on your workflow.

If your team runs Prometheus today, the lowest-friction path is Grafana's Prometheus data source. Elasticsearch now exposes a native Prometheus-compatible API, so you can point Grafana's existing Prometheus plugin directly at Elasticsearch. No sidecars, no adapters, no pipeline changes required. Existing PromQL dashboards, alert rules, and variable dropdowns work without modification, including Grafana's Metrics Drilldown explorer. Add Elasticsearch as a remote_write target in your Prometheus config and swap the data source URL. That's the full migration for most teams. See the end-to-end setup guide.

For teams that want to go further and query logs, metrics, and traces together from a single Grafana query editor, the official Grafana Elasticsearch plugin now ships with ES|QL support. This unlocks cross-signal correlation directly in Grafana, with Elasticsearch handling all three data types in a unified columnar backend. See how to set it up.

Either way, keep Grafana, replace Mimir and Loki, and gain the full benefit of Elasticsearch's columnar storage and query performance underneath. Years of operational work, preserved. The migration that teams have been putting off becomes a backend swap.

What's GA and what's in tech preview

| Capability | Status | | ----------------------------------------- | ------------ | | Columnar metrics engine (TSDS) | GA | | ES|QL time series support | GA | | PromQL support in Kibana | GA | | Prometheus Remote Write ingest | GA | | Kubernetes infrastructure OOTB experience | GA | | AWS infrastructure OOTB experience | Tech Preview | | Observability MCP App | Tech Preview | | Agent skills | Tech Preview | | Observability Migration Platform | Tech Preview |

The individual posts linked throughout cover GA versus preview specifics and known limitations.

All of this (the columnar metrics engine, native PromQL, agentic investigations, and migration tooling) runs across Elastic's three deployment modes: serverless, Elastic Cloud, and self-managed. Datadog has no on-prem option; Grafana Cloud limits its highest-value features to hosted deployments. With Elastic, you choose where your data lives.

Elastic Observability: lower cost without dropping data

Modern cloud infrastructure broke the observability model built around separate tools for separate signals. The cost is real: duplicate tooling bills, manual correlation during incidents, and data dropped just to stay on budget.

A single backend that stores every signal efficiently means you keep what you need without the bill that usually comes with it. That's a different kind of conversation to have with finance: not "we had to drop data to stay on budget," but "here's what we found." The AI gets the full picture because there's only one picture, and the platform arrives with enough pre-built content to be useful on day one, not after weeks of dashboard toil.

That's possible because of how Elasticsearch is built differently from the platforms you're likely replacing:

• Columnar metrics storage stores stores metrics data highly efficiently in TSDS index mode. 

• Native Prometheus compatibility means existing scrape configs, PromQL queries, and dashboards work without rewriting.

• Unified metrics, logs, and traces in a single backend means investigation context is assembled at query time, not manually across tabs.

• Search and analytics in the same engine — an inverted index for logs, a columnar index for metrics, queried together with ES|QL.

• Agentic investigations that correlate signals, surface anomalies, and suggest remediation before anyone is paged.

• Serverless, Elastic Cloud, or self-managed — you choose where your data lives, which Datadog cannot offer.

The cost conversation with finance becomes about what you found, not what you spent.

Get started

• Start a free trial

• Elastic Observability documentation

• Elastic Observability Labs

Frequently asked questions

Is Elasticsearch now a production-ready metrics platform?

Yes. As of June 2026, Elasticsearch ships a rebuilt columnar storage engine purpose-built for time series data, native Prometheus Remote Write ingest, PromQL support in Kibana, ES|QL time series querying, and out-of-the-box infrastructure dashboards for Kubernetes and AWS. The columnar metrics engine, ES|QL time series support, PromQL, and Prometheus ingest are all generally available in Elastic Serverless and soon GA in Elastic Cloud Hosted.

How does Elasticsearch compare to Datadog for metrics cost?

In comparable metrics workloads, Elastic Observability Serverless costs significantly less than Datadog — in illustrative examples based on published list pricing, more than 50% less, and often closer to two-thirds less. The gap is structural: Datadog bills primarily per host, then adds charges for custom metrics and containers as instrumentation grows. The cost difference is largest for exactly the workloads where Datadog bills most: high-cardinality, densely instrumented environments like Kubernetes and OTel.

How does Elasticsearch metrics performance compare to Prometheus and Grafana Mimir?

ES|QL queries on Elasticsearch run up to 30× faster than Prometheus and Mimir on gauge averages and counter rates, including high-cardinality workloads. Elasticsearch stores OTel metrics at 3.75 bytes per data point; up to 2.5× more efficiently than Prometheus and 2× more efficiently than ClickHouse.

Can teams migrate from Datadog or Grafana to Elasticsearch without rebuilding everything?

Yes. Elastic's Observability Migration Platform converts Datadog and Grafana dashboards, alert rules, and migrates PromQL queries into Kibana as-is. Teams can also keep Grafana as a visualization layer while replacing the backend with Elasticsearch, using the native Prometheus API and PromQL support in Kibana.

What makes Elasticsearch different from Grafana for metrics observability?

Elasticsearch stores metrics, logs, and traces in a single unified backend with one query language (ES|QL), while Grafana's LGTM stack splits metrics (Mimir/Prometheus) and logs (Loki) across separate backends requiring separate query languages. Elasticsearch also ships agentic investigation capabilities, which includes AI Agent, Workflows, MCP App, and Agent skills, a more comprehensive set of capabilities than Grafana. 

Does Elasticsearch support Prometheus and PromQL natively?

Yes, in two distinct ways. First, Elasticsearch accepts Prometheus metrics via Prometheus Remote Write and exposes a native Prometheus-compatible API, so it can serve as a backend for any Prometheus-compatible frontend, including Grafana. Second, Kibana supports PromQL natively, meaning existing queries, dashboards, and alert rules run directly in Kibana without a translation layer or modification.

What infrastructure monitoring content ships out of the box with Elastic Observability?

Elastic ships pre-built dashboards, alert templates, and ML anomaly detection jobs across hundreds of infrastructure integrations covering hosts, containers, cloud services, databases, network devices, and more. For Kubernetes and AWS specifically, the platform also includes agentic investigation content such as agent skills and an Observability MCP App that lets teams run investigations directly from Claude, Cursor, or VS Code. All of this is available at ingest with no configuration required.

Migrations covered by the Observability Migration Platform

The current scope covers Datadog and Grafana. The platform can work from exported assets or live APIs, and it focuses on dashboards and alerting content on the Datadog and Grafana paths it currently covers.

Support is not identical across the two sources. Datadog has end-to-end extraction, validation, compile, upload, smoke, and verification workflows, but it currently covers a narrower slice of widgets and monitors. Grafana coverage is broader. The platform provides a practical translation pipeline for the supported paths.

The screenshots below show examples of dashboards after migration.

How the Observability Migration Platform works

At a high level, the workflow has two halves: source-aware translation on the way in and target-aware validation and delivery on the way out. That split matters because Grafana and Datadog differ not only in JSON shape, but also in query languages, panel types, controls, and alerting models.

A run starts with exported assets or live source APIs. From there, the workflow normalizes source-specific objects, chooses a translation path for each supported dashboard, panel, and alerting artifact, and emits Kibana-native output. This is where most of the source-specific logic lives: translating queries or Datadog formulas, mapping panel semantics, carrying forward controls and links where possible, and deciding when an exact translation is not the right answer.

The second half is target-aware. The emitted output can be validated against an Elastic target, compiled, and uploaded to Kibana through the shared runtime. In the happy path, that yields a working translated dashboard. In rougher cases, validation may show that a panel cannot run safely as emitted. When that happens, the workflow is designed to fail conservatively: it can mark the panel for manual review or replace it with an upload-safe placeholder instead of shipping a broken runtime panel.

Just as important, the outcome is not simply "a dashboard showed up in Kibana." The workflow also produces reviewer-facing evidence such as a migration report, manifest, verification packets, and rollout plan so you can see what translated cleanly, what was downgraded or manualized, and what still needs human judgment. Those artifacts are what make the process operationally credible: they give teams something concrete to inspect, compare, and act on.

Running the migration

The platform is CLI-driven, and a good fit for migration work that needs to be repeatable, reviewable, and easy to automate. Users can start with a representative slice of dashboards and alerting content from Grafana or Datadog, point the workflow at an Elastic target, and use that first run to understand translation quality, validation results, and how much follow-up review is required.

To run the full path against Elastic, create an Elastic Observability Serverless project, generate a Serverless project API key, and point the CLI at your Elasticsearch and Kibana endpoints:

obs-migrate migrate \
  --source grafana \
  --input-mode files \
  --input-dir ./grafana_exports \
  --output-dir ./migration_output \
  --assets all \
  --native-promql \
  --data-view "metrics-*" \
  --validate \
  --es-url "$ELASTICSEARCH_ENDPOINT" \
  --es-api-key "$KEY" \
  --kibana-url "$KIBANA_ENDPOINT" \
  --kibana-api-key "$KEY" \
  --upload

The run validates the emitted queries against Elastic, compiles the generated dashboards, uploads them to Kibana, and produces the standard migration artifacts for review.

A typical run looks like this:

1. Start with exported assets or live source APIs from Grafana or Datadog.
2. Choose the asset scope with --assets dashboards, --assets alerts, or --assets all.
3. Translate the supported dashboards, queries, controls, and alerting artifacts into Kibana-native output.
4. Validate the emitted content against an Elastic target (if configured), then compile and upload the translated dashboards for dashboard-capable runs.
5. Review the migration evidence, including migration_report.json, verification_packets.json, run_summary.json, etc., to understand what translated cleanly, where semantic gaps remain, and which dashboards, panels, or alert rules still require human review.
6. If alert rule creation is enabled, review the migrated rules (which are disabled by default) in Kibana before deciding which ones to enable or redesign.

What's next

The platform is still evolving, and will continue to gain depth and self-service capabilities. The biggest open areas are stronger measured source-to-target semantic verification, further coverage for Datadog, deeper coverage for harder query families and non-dashboard surfaces, and cleaner shared runtime contracts across the workflow.

It is also built to grow over time. The source and target boundaries are explicit by design, which gives the platform room to expand coverage and support additional source paths in the future.

In conclusion

If you are planning a move into Elastic, a good starting point is to create an Elastic Observability Serverless project. That gives you the target environment where translated dashboards and alerting content can be validated and reviewed.

To learn more about the migration workflow, talk to your Elastic representative about current access, supported coverage, and how it can help with your migration needs.

Running PromQL queries in Kibana

In the ES|QL editor in Kibana, enter the PROMQL command, and type your PromQL in that block. PROMQL marks that segment so Elasticsearch parses it as PromQL inside the wider ES|QL request Kibana sends.

What you can query

Here are a few patterns to get started.

Raw metric

PROMQL container.cpu.usage

Average across all containers

PROMQL avg(container.cpu.usage)

rate() on a counter

PROMQL rate(docker.network.inbound.bytes)

Aggregated rate

PROMQL sum(rate(docker.network.inbound.bytes))

Group by a label

PROMQL sum by (agent.id) (rate(docker.network.inbound.bytes))

You may notice that none of these examples include start, end, step, or a lookback window on every rate(). Those parameters are optional: the time picker and Kibana defaults handle most of it for you.

Optionally, you can include the data stream name using the index= parameter. For example: PROMQL index=metrics-docker.cpu-default container.cpu.usage. Adding the parameter helps narrow down the scope of what data the query scans.

The current release of PromQL tech preview has over 80% query coverage benchmarked against top Grafana dashboards. Advanced modifiers and specific functions are in consideration for future releases.

Find your streams and metric names

If you have existing PromQL queries, you can use them directly in the PROMQL command without changes. If you are writing a query from scratch and need to find the exact field names, run TS metrics-* in Discover to see every metrics data stream. Each metric appears as a small chart so you can tell at a glance what is active. Hover over a metric and click the "View details" action to see the field name and the data stream it belongs to.

For a deeper walkthrough, see Explore metrics data with Discover in Kibana.

Time picker and query time handling

The time picker in Kibana sets the time window for the query. Dashboard panels and Alerting rules work the same way using their own time range, so you do not need to write start= or end= in the query itself.

Step is the gap between two consecutive data points on the chart. A smaller step means more data points across the same span. If you do not set step= or buckets=, the default is buckets=100. You can set step= to a fixed width such as 1m, or set buckets= to a different target maximum number of data points.

Discover and Dashboards

In Discover, switch to ES|QL mode and run your PROMQL query so you can see how the metric behaves over the range you pick, as a time-series chart. When you want to save that visualization, choose "Save visualization to dashboard" and add it to a new or existing dashboard.

Or go to Dashboards directly: add a panel, choose ES|QL, and write your PROMQL query.

Alerting

You can create alert rules using PromQL. Go to Alerts, open Manage rules, and create a rule. Search for Elasticsearch query and select it. Choose ES|QL as the query type.

Write your PROMQL query, but assign the metric to a variable so you can use it in a WHERE clause for the alert condition:

PROMQL metric_value=(sum by (agent.id) (rate(docker.network.inbound.bytes)))
| WHERE metric_value >= 500

Select @timestamp for the time field and continue defining the rest of the rule configuration.

Try it

1. Open an Observability project on Elastic Cloud Serverless, or use Elastic Stack 9.4.
2. Write your query: in the ES|QL editor in Kibana, run your PromQL via PROMQL. You can also go to Dashboards, add a panel, choose ES|QL, and write the query there.
3. If you are writing from scratch and need to find metric names, run TS metrics-* in Discover (see "Find your streams and metric names" above).
4. Check the results and adapt the query if needed.

PromQL support in Elasticsearch and Kibana will continue to evolve. Follow the Observability Labs feed for follow-up posts as coverage and ergonomics improve.

These latest updates, available in v9.3 and in Serverless, introduce significant performance gains, sophisticated time series functions, and native OpenTelemetry exponential histogram support that directly benefit SREs and Observability practitioners.

Query Performance and Storage Optimizations

Speed is paramount when diagnosing incidents. Compared to prior releases, we have achieved a 5x+ improvement in query latency when wildcarding or filtering by dimensions. Additionally, storage efficiency for OpenTelemetry metrics data has improved by approximately 2x, significantly reducing the infrastructure footprint required to retain high-volume observability data. If you’re hungry to learn more about what architectural updates are driving these optimizations, stay tuned… Tech blogs are on their way! 

Expanded Time Series Analytics in ES|QL

The ESQL TS source command, which targets time series indices and enables time series aggregation functions, has been significantly enhanced to support complex analytics capabilities.

We have expanded the library of time series functions to include essential tools for identifying anomalies and trends.

• PERCENTILE_OVER_TIME, STDDEV_OVER_TIME, VARIANCE_OVER_TIME: Calculate the percentile, standard deviation, or variance of a field over time, which is critical for understanding distribution and variability in service latency or resource usage.

Example: Seeing the worst-case latency in 5-minute intervals.

TS metrics*  | STATS MAX(PERCENTILE_OVER_TIME(kafka.consumer.fetch_latency_avg, 99))
  BY TBUCKET(5m)

• DERIV: This command calculates the derivative of a numeric field over time using linear regression, useful for analyzing the rate of change in system metrics.

Example: trending gauge values over time.

TS metrics*  | STATS AVG(DERIV(container.memory.available))
  BY TBUCKET(1 hour)

• CLAMP: To handle noisy data or outliers, this function limits sample values to a specified lower and upper bound.

Example: handling saturation metrics (like CPU or Memory utilization) where spikes or measurement errors can occasionally report values over 100%, making the rest of the data look like a flat line at the bottom of the chart.\

TS metrics*  | STATS AVG(CLAMP(k8s.pod.memory.node.utilization, 0, 100))
  BY k8s.pod.name

• TRANGE: This new filter function allows you to filter data for a specific time range using the @timestamp attribute, simplifying query syntax for time-bound investigations.

Example: Filtering and showing metrics for the last 4 hours.

TS metrics*  | WHERE TRANGE(4h) | STATS AVG(host.cpu.pct)
  BY TBUCKET(5m)

Window Functions To smoothen results over specific periods, ES|QL now introduces window functions. Most time series aggregation functions now accept an optional second argument that specifies a sliding time window. For example, you can calculate a rate over a 10-minute sliding window while bucketing results by minute.

Example: Calculating the average rate of requests per host for every minute, using values over a sliding window of 5 minutes.

TS metrics*  | STATS AVG(RATE(app.frontend.requests, 5m))
  BY TBUCKET(1m)

Accepted window values are currently limited to multiples of the time bucket interval in the BY clause. Windows that are smaller than the time bucket interval or larger but not a multiple of the time bucket interval will be supported in feature releases. 

Native OpenTelemetry Exponential Histograms

Elastic now provides native support for OpenTelemetry exponential histograms, enabling efficient ingest, querying, and downsampling of high-fidelity distribution data.

We have introduced a new exponential_histogram field type designed to capture distributions with fixed, exponentially spaced bucket boundaries. Because these fields are primarily intended for aggregations, the histogram is stored as compact doc values and is not indexed, optimizing storage efficiency. These fields are fully supported in ES|QL aggregation functions such as PERCENTILES, AVG, MIN, MAX, and SUM.

You can index documents with exponential histograms automatically through our OTLP endpoint or manually. For example, let’s create an index with an exponential histogram field and a keyword field:

PUT my-index-000001
{
  "settings": {
    "index": {
      "mode": "time_series",
      "routing_path": ["http.path"],
      "time_series": {
        "start_time": "2026-01-21T00:00:00Z",
        "end_time": "2026-01-25T00:00:00Z"
     }
    }
  },
  "mappings": {
    "properties": {
      "@timestamp": {
        "type": "date"
      },
      "http.path": {
        "type": "keyword",
        "time_series_dimension": true
      },
      "responseTime": {
        "type": "exponential_histogram",
        "time_series_metric": "histogram"
      }
    }
  }
}

Index a document with a full exponential histogram payload:

POST my-index-000001/_doc
{
  "@timestamp": "2026-01-22T21:25:00.000Z",
  "http.path": "/foo",
  "responseTime": {
    "scale":3,
    "sum":73.2,
    "min":3.12,
    "max":7.02,
    "positive": {
      "indices":[13,14,15,16,17,18,19,20,21,22],
      "counts":[1,1,2,2,1,2,1,3,1,1]
    }
  }
}

POST my-index-000001/_doc
{
  "@timestamp": "2026-01-22T21:26:00.000Z",
  "http.path": "/bar",
  "responseTime": {
    "scale":3,
    "sum":45.86,
    "min":2.15,
    "max":5.1,
    "positive": {
      "indices":[8,9,10,11,12,13,14,15,16,17,18],
      "counts":[1,1,1,1,1,1,1,2,1,1,2]
    }
  }
}

And finally, query the time series index using ES|QL and the TS source command:

TS my-index-000001  | STATS MIN(responseTime), MAX(responseTime),
        AVG(responseTime), MEDIAN(responseTime),
        PERCENTILE(responseTime, 90)
  BY http.path

Enhanced Downsampling

Downsampling is essential for long-term data retention. We have introduced a new "last value" downsampling mode. This method exchanges accuracy for storage efficiency and performance by keeping only the last sample value, providing a lightweight alternative to calculating aggregate metrics.

You can configure a time series data stream for last value downsampling in a similar way as regular downsampling, just by setting the downsampling_method to last_value. For example, by using a data stream lifecycle:

PUT _data_stream/my-data-stream/_lifecycle
{
  "data_retention": "7d",
  "downsampling_method": "last_value",
  "downsampling": [
     {
       "after": "1m",
       "fixed_interval": "10m"
      },
      {
        "after": "1d",
        "fixed_interval": "1h"
      }
   ]
}

In Conclusion

These enhancements mark a significant step forward in Elastic's metrics analytics capabilities, delivering 5x+ faster query latency, 2x storage efficiency and specialized commands like DERIV, CLAMP, and PERCENTILE_OVER_TIME. With native support for OpenTelemetry exponential histograms and expanded downsampling options, SREs can now perform richer, more cost-effective analysis on their observability data. This release empowers teams to detect anomalies faster and manage long-term metrics retention with greater efficiency.

We welcome you to try the new features today!

As a core pillar of Observability, metrics offer a highly structured, quantitative view of system performance and health. They provide a crucial symptomatic perspective—revealing what is happening, such as high application latency, increasing service errors, or spiking container CPU utilization, which is essential for initiating alerting and triaging efforts. This capability for effective monitoring, alerting, and triaging is paramount to ensuring robust service delivery and achieving successful business outcomes.

Elastic Observability provides a comprehensive, end-to-end experience for metrics data. Elastic ensures that metrics data can be collected from numerous sources, enriched as needed and shipped to the Elastic Stack. Elastic efficiently stores this time series data, including high-cardinality metrics, utilizing the TSDS index mode (Time Series Data Stream), introduced in prior versions and used across Elastic time series integrations. This foundation ensures comprehensive observability through out-of-the-box dashboards, alerts, SLOs, and streamlined data management.

Elastic Observability 9.2 provides enhancements to metrics exploration and analysis through powerful query language extensions and expanded UI capabilities. These enhancements focus on making analysis on TSDS data via counter rates and common aggregations over time easier and faster than ever before.

The main metrics enhancements center on these key features, offered as Tech Preview:

1. Metrics analytics with TSDS and ES|QL
2. Interactive metrics exploration in Discover
3. OTLP endpoint for metrics

Metrics analytics with TSDS and ES|QL

The introduction of the new TS source command in ES|QL (Elasticsearch Query Language) on TSDS metrics dramatically simplifies time series analysis.

The TS command is specifically designed to target only time series indices, differentiating it from the general FROM command. Its core power lies in enabling a dedicated suite of time series aggregation functions within the STATS command.

This mechanism utilizes a dual aggregation paradigm, which is standard for time series querying. These queries involve two aggregation functions:

• Inner (Time Series) function: Applied implicitly per time series, often over bucketed time intervals.

• Outer (Regular) function: Used to aggregate the results of the inner function across groups. For instance, if you use STATS SUM(RATE(search_requests)) BY TBUCKET(1 hour), host, the RATE() function is the inner function applied per time series in hourly buckets, and SUM() is the outer function, summing these rates for each host and hourly bucket.

If an ES|QL query using the TS command is missing an inner (time series) aggregation function, LAST_OVER_TIME() is implicitly assumed and used. For example, TS metrics | STATS AVG(memory_usage) is equivalent to TS metrics | STATS AVG(LAST_OVER_TIME(memory_usage)).

Key time series aggregation functions available in ES|QL via TS command

These functions allow for powerful analysis on time-series data:

| | | | | :----------------------------------------------------: | :-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: | :-----------------------------------------------------------: | | Function | Description | Example Use Case | | RATE() / IRATE() | Calculates the per-second average rate of increase of a counter (RATE), accounting for non-monotonic breaks like counter resets, making it the most appropriate function for counters, or the per-second rate of increase between the last two data points (IRATE), ignoring all but the last two points for high responsiveness. | Calculating request per second (RPS) or throughput. | | AVG_OVER_TIME() | Calculates the average of a numeric field over the defined time range. | Determining average resource usage over an hour. | | SUM_OVER_TIME() | Calculates the sum of a field over the time range. | Total errors over a specific time window. | | MAX_OVER_TIME() / MIN_OVER_TIME() | Calculates the maximum or minimum value of a field over time. | Identifying peak resource consumption. | | DELTA() / IDELTA() | Calculates the absolute change of a gauge field over a time window (DELTA) or specifically between the last two data points (IDELTA), making IDELTA more responsive to recent changes. | Tracking changes in system gauge metrics (e.g., buffer size). | | INCREASE() | Calculates the absolute increase of a counter (INCREASE). | Analyzing immediate rate changes in fast-moving counters. | | FIRST_OVER_TIME() / LAST_OVER_TIME() | Calculates the earliest or latest recorded value of a field, determined by the @timestamp field. | Inspecting initial and final metric states within a bucket. | | ABSENT_OVER_TIME() / PRESENT_OVER_TIME() | Calculates the absence or presence of a field in the result over the time range. | Identifying monitoring coverage gaps. | | COUNT_OVER_TIME() / COUNT_DISTINCT_OVER_TIME() | Calculates the total count or the count of distinct values of a field over time. | Measuring frequency or cardinality changes. |

These functions, available with the TS command, allow SREs and Ops teams to easily perform rate calculations and other common aggregations, enabling efficient metrics analysis as a routine part of observability workflows. And it’s much faster, too! Internal performance testing has revealed that TS commands outperform other ways of querying metrics data by an order of magnitude or more, and consistently! 

Interactive metrics exploration in Discover

The 9.2 release introduces the capability to explore and analyze metrics directly and interactively within the Discover interface. In addition to exploring and analyzing logs and raw events, Discover now provides a dedicated environment for metrics exploration:

• Easy start: Begin exploration simply by querying metrics ingested via TS metrics-*.

• Grid view and pre-applied aggregations: This command displays all metrics in a grid format at a glance, immediately applying the appropriate aggregations based on the metric type, such as rate versus avg.

• Search and group-by: Quickly search for specific metrics by name. Also easily group and analyze metrics by dimensions (labels) and specific values. This allows narrowing down to metrics and dimensions of choice for targeted analysis.

• Quick access to details: Furthermore, the interface provides access to crucial details, including query and response details, the underlying ES|QL commands, the metric field type, and applicable dimensions, for each metric.

• Easy tweaking and dashboarding: The system automatically populates ES|QL queries, aiding in making easy tweaks, slicing, and dicing the data. Once analyzed, metrics and resulting analyses can be added to new or existing dashboards with ease.

OTLP endpoint for metrics

We are also introducing a native OpenTelemetry Protocol (OTLP) endpoint specifically for metrics ingest directly into Elasticsearch. The endpoint especially benefits self-managed customers, and will be integrated into our Elastic Cloud Managed OTLP Endpoint for Elastic-managed offerings. The native endpoint and related updates improve ingest performance and scalability of OTel metrics, providing up to 60% higher throughput via _otlp, and up to 25% higher throughput when using classic _bulk methods. 

In Conclusion

By merging the power of ES|QL's new time series aggregations with the familiar interactive experience of Discover, Elastic 9.2 enables a potent set of metrics analytics tools. The tools significantly boost the exploration and analysis phase of any observability workflow. And we’re just getting started on unleashing the full power of metrics in Elastic Observability!

We welcome you to try the new features today!

Also learn more about how we provide metrics analytics for AWS, Azure, GCP, Kubernetes, and LLMs on Observability Labs