What is metric temporality in OpenTelemetry?

Metrics stores usually receive client-side, pre-aggregated metrics. For example, if an application records request response times, it won’t send each individual response time as a single data point to your metrics back end. Instead, the application (or rather the OTel SDK) pre-aggregates those raw response times into counters or histograms. These pre-aggregated values are then exported at a periodic interval, dramatically reducing the number of data points. Temporality is about how this pre-aggregation works. There are two temporality models: cumulative and delta.

Cumulative temporality in OTel metrics

With cumulative temporality, each data point represents the totalamount of change in the metric value since the process started. Values monotonically increase, with occasional reset to 0 (for example, when the process restarts).

Take a counter tracking the total CPU time consumed by a Java Virtual Machine (JVM):

Timestamp

Value

Meaning

10:01

12.4s

12.4s total CPU time since start

10:02

13.1s

13.1s total CPU time since start

10:03

13.9s

13.9s total CPU time since start

To compute the rate of change between 10:01 and 10:02, we subtract: 13.1 - 12.4 = 0.7s of CPU time was consumed in that interval. Dividing by the time range of the interval gives us the rate. This is the default temporality for counters in both Prometheus and OTel.

Delta temporality in OTel metrics

With delta temporality, each data point represents the change since the last measurement. Values are independent of each other. In other words, after each export, the OTel SDK resets all values for all series.

The same raw observations from the cumulative example above would look as follows with delta temporality.

Timestamp

Value

Meaning

10:01

0.5s

0.5s of CPU time in this interval

10:02

0.7s

0.7s of CPU time in this interval

10:03

0.8s

0.8s of CPU time in this interval

To compute the rate or increase, we can use the value directly, without any subtraction.

Trade-offs between cumulative and delta OpenTelemetry metrics

Both temporalities have practical trade-offs:

• Resilience to data loss: Cumulative counters are self-describing: If you miss an export, the next data point still gives you the correct total. Delta values are incremental, so a lost data point means that the corresponding increase is lost.

• Metric producer memory footprint: For cumulative temporality, the OTel SDKs need to keep a state for every series in memory. For delta temporality, the footprint is much lower. There, the SDKs only need to keep track of counters or histograms which changed since the last export. If there are a lot of counters or histograms and many of them don’t increase each period, this difference can be quite substantial.

• Aggregation across restarts: Cumulative counters require reset detection logic, which in edge cases can fail: If the metric value decreases, it’s detected as a reset. We assume that the application was restarted and the counter started from 0 again. This can be missed if the first reported counter value after the restart is higher than before the restart. A concrete example:

• The service consumes 1 second CPU time and restarts.

• After the restart, the service performs a CPU-intensive task and consumes 2 seconds of CPU time before the metric is exported again.

• The metric back end just sees 1 followed by 2 as the metric value. It never observes a decrease and therefore misses the reset.

Delta values don't have this problem since each value is independent.

If you’re using histograms, the trade-offs have an even bigger effect:

Trade-off

Cumulative

Delta

Histogram size

Buckets accumulate across exports, consuming more storage

Buckets reset each export, producing smaller histograms

Min/max accuracy

Approximated from buckets for custom time ranges (tracked values represent extremes since process start)

Exact per-export minimum and maximum values

Query performance

Faster: only the first and last value in a time range plus resets are needed

Slower: all histograms in the queried range must be combined

OpenTelemetry supports both models and lets you choose per SDK via the OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE environment variable.

Why native temporality support eliminates OTel Collector workarounds

Prometheus and most other metrics back ends pick a side: All metrics have to be either cumulative or delta. Elasticsearch previously followed that pattern, too, with native storage of cumulative counters and delta histograms, and workarounds for everything else. Delta counters were stored as gauges, functional but without native counter semantics for rate queries. And cumulative histograms were unsupported.

One workaround for unsupported temporalities is to configure your metric producers (for example, OTel SDKs) to produce data with the temporality that your back end supports. In large-scale deployments, this can be a very challenging task. And sometimes this isn’t even possible (for example, if you consume OTLP metrics from third-party services).

Another workaround is to convert the temporality prior to ingestion. In the OTel Collector, you would typically use the cumulative-to-delta processor, which comes with a big warning sign about statefulness. The conversion is inherently stateful, requiring ordered delivery of metric series to the same collector and persisted state across restarts. In practice, it works, but at scale, it comes with a lot of deployment headaches.

With Elasticsearch 9.5, you can skip the conversion pipeline entirely. Elasticsearch natively stores and queries metric data with both temporalities. It doesn’t require any stateful conversion required or explicit configuration of your OTel SDKs.

Demo: ingesting cumulative and delta OTel metrics side by side

To demonstrate the temporality support, we'll reuse a demo setup from our OTel histogram metrics ES|QL blog post: a Java Renaissance benchmark instrumented with the OTel Java agent. The twist this time: We run two instances of the benchmark, each configured with a different temporality:

• renaissance-delta: Exports metrics with delta temporality.

• renaissance-cumulative: Exports metrics with cumulative temporality.

Both instances report the same metrics under the same service name renaissance, but with different service.instance.id values. Here’s the relevant section of the docker-compose.yml that can be found in the companion code:

To run the demo yourself, you'll also have to fill out the managed OTLP endpoint URL and the corresponding API key:

After starting the demo with docker compose up --build, both instances will start reporting metrics to Elasticsearch.

Querying OTel counter metrics with ES|QL and PromQL

Let's query the first few raw data points of jvm.cpu.time for both instances to see the different temporalities in action:

This gives us the first five data points for each service instance:

The benchmark consumes CPU at a nearly constant rate. This is directly visible based on the delta temporality data: The values are nearly constant between exports. In contrast, the cumulative temporality values grow over time, as they represent the total CPU usage of the benchmark instance.

Now let's have a look at how to properly query this metric using PromQL:

The screenshot shows that both benchmark instances consume a nearly constant of 1 to 1.2 number of CPU cores with some variance. This query works because we made our rate implementation respect the temporality: Every time series (so every service instance in our case) stores the temporality as a metric dimension. The rate implementation looks at this dimension and interprets the data accordingly: For delta temporality, values are summed up; for cumulative temporality, a difference computation is done. This all happens automatically in the background, without requiring any changes to your queries.

We’ve adapted rate, increase, and irate to work this way. The same applies when using those functions in ES|QL TS queries:

Because Elasticsearch tracks the temporality as a dimension, you can have multiple series with different temporalities for the same metric, just like in the demo use case. Aggregating across series also works as expected, because at that point rate, increase, or irate already took care of normalizing the data:

Querying OTel histogram metrics across temporalities

Metric temporality applies to histograms in the same way it applies to counters: histogram buckets are effectively a set of counters, each tracking values in a specific range.As in our histogram demo, we use exponential histograms, where bucket boundaries adapt automatically to minimize relative error.

Due to this similarity, histograms can also be cumulative or delta. Either the counter per bucket is reset after each metric export or the cumulative count carries over between exports.

Let's query the median major garbage collection (GC) duration for our benchmark instances, which is a histogram metric:

Or the equivalent ES|QL query:

Again, both queries will automatically load the temporality per series and interpret the histograms accordingly. In PromQL, this is handled by the increase function. Note that in ES|QL, you don't explicitly call increase on histograms. The TS command automatically handles the temporality-aware merging of histograms when you use aggregation functions, like PERCENTILE, MEDIAN, or AVG.

How Elasticsearch stores metric temporality in TSDB

Elasticsearch's time series database (TSDB) stores metric temporality in a dedicated dimension field on each document. The index.time_series.temporality_field index setting lets you specify which field carries the temporality information. The field must be a keyword field with time_series_dimension: true and the permissible values "delta" or "cumulative".

As soon as this setting is present on a time series index, ES|QL and PromQL will load the corresponding field when performing temporality-dependent aggregations. If the field isn’t present or has no value on a document, we fall back to defaults based on the type of the corresponding metric: counters default to cumulative, and histograms default to delta. This matches the historical behavior and ensures existing queries and existing data continue to work without changes.

When you ingest metrics via the OTLP endpoint, Elasticsearch automatically adds a temporality dimension field to each document, populated from the OTLP AggregationTemporality metadata. For custom (neither OTLP nor Prometheus remote write) ingestion, you’ll have to manually set up the index.time_series.temporality_field setting and populate your temporality dimension.

The temporality is also respected during downsampling: As it’s a dimension, it’s preserved automatically and used to compute the aggregate values.

Getting started with mixed-temporality OTel metrics in Elasticsearch

With Elasticsearch 9.5, cumulative versus delta is no longer a decision you have to get correct at the start. Ingest both temporalities side by side, even for the same metric name, and let ES|QL and PromQL handle the rest. You can switch between both without having to touch your queries. For more details, see the metric temporality documentation.

Just send your OTel metrics to the Elasticsearch OTLP/HTTP endpoint and they're stored using the new exponential_histogram type and queryable immediately. Already have historical data stored in the classic histogram type? A simple ::exponential_histogram cast in your ES|QL queries handles the migration transparently. Already using downsampling? Both histogram field types are now fully supported.

Histogram metrics

When dealing with metrics (in OpenTelemetry or Prometheus, for instance), counters and gauges are the most common metric types. Gauges allow you to monitor values that rise or fall (e.g., CPU utilization). Counters allow you to, well, count things, such as the total number of HTTP requests your service is handling. Counters normally just increase in value, with a few exceptions when they reset, like when a server reboots.

In the case of counters, you can additionally collect a counter measuring the total sum of your HTTP response times, which allows you to derive the average response time by dividing that sum by the total number of requests. However, average response times provide limited insights into the collected data and the system behavior. The best insights are gained by analyzing the collected metric distribution, e.g., through median and percentile calculations. This is where counters fall short.

In the past, workarounds have been applied: For example, classic Prometheus-style histograms attempt to capture the distribution using a set of counters. By defining fixed buckets (e.g., one for response times in the range [0s, 1s), one for [1s, 4s), and so on) and associating a counter with each, we can at least estimate percentiles broadly. However, the key problem here is that we have to know the distribution of our data up front to properly define these buckets.

To that end, the OpenTelemetry community has come up with a better solution: exponential histograms. Exponential histograms assign collected values to buckets, just like classic Prometheus-style histograms. The key differentiator is that these buckets vary dynamically based on the collected values. The name "exponential" comes from the fact that the bucket sizes increase exponentially: we use small buckets for small values and wider buckets for larger values. You can find an excellent introduction in the OpenTelemetry exponential histograms introduction.

Note that in addition to classic histograms, Prometheus also added native histograms, which directly map to OTel exponential histograms. Native histograms have their own PromQL syntax. We are actively working on adding support for that syntax to the Elasticsearch PromQL implementation, so that you can directly query exponential histograms using PromQL.

Demo setup

Let's start by collecting some histogram metrics to show how they can be stored and analyzed in Elasticsearch using ES|QL.

We'll focus on a Java JVM metric: garbage collection durations. OpenTelemetry defines the jvm.gc.duration, which is a histogram-typed metric. The OpenTelemetry Java agent natively supports collecting this metric.

We'll spin up a JVM running a Renaissance benchmark to put it under stress. We'll start that JVM with the vanilla OpenTelemetry Java agent attached and have it send the metrics directly to Elasticsearch.

You can find the ready-to-run Docker-compose file here. You'll just need to insert your Elasticsearch OTLP/HTTP endpoint and API key in the docker-compose.yml:

Note that you don't have to use this demo setup. We even encourage you to try it with your own application. Here are the other important OpenTelemetry agent settings the demo already includes, which you should include too if you're bringing your own app:

Let's step through them:

• Temporality preference: OpenTelemetry supports both cumulative and delta-based histograms. Cumulative means that the histogram is only cleared after an application restart, while delta clears it after each export. At the time of writing, Elasticsearch only supports delta temporality for histograms. We are actively working on supporting cumulative histograms as well.

• Default Histogram Aggregation: By default, OpenTelemetry exports histograms in the Prometheus-style fixed bucket format. Since we want to reap the benefits of exponential histograms, we tell the agent to use them instead.

• Runtime Telemetry enabled: This tells the agent to actually collect the detailed JVM metrics, which include jvm.gc.duration.

Now we are ready to go! We'll let the application run in the background and switch over to Kibana to analyze the GC metric.

Querying with ES|QL

Now let's open up Kibana and navigate to "Discover". There we'll switch to ES|QL mode, and start querying the collected data:

As a response, we now see the metric panel shown below. If you don't see any data, make sure to double-check the Kibana time range filter.

This number represents the total number of garbage collection operations that happened in our test application during the selected time frame.

Similarly, we can query the total time spent on those garbage collection operations:

So we have roughly 270k garbage collections, which in total took 713 seconds. Given these two numbers, we can now compute the average if we are still fluent in primary school-level math. Even if not, you can just let ES|QL do that for you:

Now we know that the average garbage collection operation took about 3 milliseconds. However, Java experts might know that there are different kinds of garbage collections happening, which can have significantly different pause times. Fortunately the OpenTelemetry metric comes with attributes, which allow us to slice the data accordingly:

As expected, major garbage collections take a lot more time per collection than minor ones, at least on average. So far, we have done nothing you couldn't also achieve by just using counters. Let's now use histograms to understand the actual distribution of the GC latency. We'll look at the data over time (by grouping using TBUCKET) and focus on the major garbage collections:

The graph now shows us the minimum, maximum, median and 99th percentile for major garbage collections. Note that we aren't bound to only querying the median and the 99th percentile. We can query any percentile we'd like to see, as these are estimated at query time from the raw exponential histograms.

A note on backwards compatibility

So far, we have seen how you can use the new shiny toy in Elasticsearch and ES|QL: exponential histograms. However, since this has just reached general availability (GA) in the 9.4 release, what about your historical data?

Before exponential histograms were added, Elasticsearch was already capable of storing OpenTelemetry histograms in the histogram field type. To do so, we converted them to a different data structure supported by the histogram field type: T-Digest. T-Digest provides good accuracy for extreme percentiles (e.g., 99th percentile) at the cost of accuracy for percentiles in the middle of the distribution, such as the median. In contrast, exponential histograms provide a guaranteed upper bound on the relative error for every percentile. As conversions always introduce errors, we are happy to now have native support for exponential histograms, allowing you to collect and analyze your metrics end-to-end without unnecessary conversions.

But still, what should you do if you have historical data and still want to query it? Thanks to ES|QL union types, the answer is actually easy: You just have to add a ::exponential_histogram suffix to the histogram metrics in your queries:

When this query encounters histogram fields, it will attempt to convert them to exponential histograms. When operating on exponential_histogram fields, the ::exponential_histogram cast has no effect. Note that this also works with mixed data sets: if your backing indices use both types, the query will just do the right thing.

So if you are building queries or dashboards that you expect to run on pre-9.4 ingested data, we recommend that you simply add: ::exponential_histogram casts.

Wrapping up

Native support for OpenTelemetry exponential histograms in Elasticsearch gives you better metric fidelity and more flexible analysis in ES|QL. In this blog post, we have shown you how to easily ingest and analyze your histogram metrics with ES|QL using various aggregations and the impact exponential histograms have.

Exponential histograms are generally available in Elasticsearch basic starting with the 9.4.0 release. They will be available in Elastic Cloud Serverless a few weeks after the 9.4.0 release, once mOTLP (the managed observability OTLP intake) switches to use the Elasticsearch OTLP endpoint. We'll update this blog post and add a note on the Elastic Cloud Serverless release notes when that happens.