Why dense vectors create off-heap memory pressure that heap charts miss

Semantic search and kNN rely on dense_vector fields. Elasticsearch stores much of that data in off-heap memory. It’s related to how the Java Virtual Machine (JVM) operates, but it isn’t the same thing as heap usage.

In production, the heap versus off-heap split shows up in a familiar pattern:

• Heap looks fine for weeks, while the dense vector off-heap footprint quietly grows.

• Heap circuit breakers stay quiet or only spike late because the pressure sits outside the JVM.

• kNN search and bulk ingest slow down, queues build, and nothing on the dashboard points at vector RAM as the cause.

Heap limits protect Java allocations. They don’t tell you whether vector off-heap footprint still fits the RAM envelope that your deployment actually runs in. AutoOps already watches cluster health broadly; Vector memory pressure adds a focused read for vector-heavy nodes when memory and load signals line up.

How AutoOps measures vector RAM, heap, and headroom

AutoOps works from the same node stats metrics you already use for Stack Monitoring. For each node, it tracks three derived numbers:

Symbol

Meaning

Source (typical)

Chart (see below)

V

Vector off-heap footprint

indices.dense_vector.off_heap.total_size_bytes

First, green line

A

Available RAM in the product view

Delta between os.mem.total_in_bytes and os.mem.used_in_bytes

Second, green line

H

Headroom

A − V (headroom_bytes)

First, blue line

H > 0 means there’s a modeled runway: Vector use still fits comfortably in that accounting. H ≤ 0 means that you’re in a compression regime: Vector footprint (V) meets or exceeds the free RAM (A) picture that AutoOps can align in telemetry. On small tiers, that can be common under load. The insight emphasizes trends, growth in vector off-heap footprint, and corroborating stress, not a single negative snapshot.

AutoOps also tracks a compression regime flag (fraction of recent samples where H ≤ 0), so brief flickers don’t dominate the story (see third chart below):

How vector memory pressure detection works: Expansion and compression

Vector memory pressure is a single HIGH severity AutoOps event, which adapts to the compression regime:

• Expansion (H > 0): Emphasize shrinking headroom, hence a growing vector off-heap footprint.

• Compression (H ≤ 0): Emphasize ΔV, heap context, I/O, and latency. "Hours until H hits zero" isn’t the main narrative when headroom is already gone.

The detector requires three layers before it fires:

1. Memory carriers: Compression regime, shrinking headroom, or sustained growth in vector off-heap footprint.

2. Operational latch: Search or indexing latency versus rolling baselines, filesystem read stress (paired with latency or heap), indexing throttle, thread-pool queues or rejections, segment creep, or heap circuit breakers when paired with other stress, as circuit breakers alone don’t provide enough evidence to be escalated without corroborating stress.

3. Heap hot: Heap usage elevated versus a 24-hour rolling median on that node, so compression alone on a calm heap doesn’t fire the insight.

That pairing is intentional. Vector pressure without load might be capacity planning, and load without vector pressure might be a different root cause. Together, vector memory, operational stress, and heap heat surface the vector RAM story when the node is actually in trouble, not on every compressed mapping while the heap stays normal.

Validation: Memory pressure detection on a 4 GiB node under kNN load

We stress-tested vector memory pressure detection on 4 GiB Elastic Cloud Hosted deployments with throttled dense-vector ingest (~2,000 docs per minute) and steady kNN search (~8 queries per second). Across Hierarchical Navigable Small World (HNSW), Better Binary Quantization (BBQ) HNSW, and DiskBBQ mapping profiles over 24–48 hours:

• Vector off-heap footprint grew from near zero to about 6 GiB on the tightest runs (more than 4.7 million vectors indexed) in both HNSW test runs.

• Nodes spent most of each run in compression (H ≤ 0), which is expected when vector footprint exceeds total RAM in this model.

• Vector memory pressure stayed off while heap held near 50%, even with compression and pool stress building.

• On both HNSW and BBQ HNSW, the insight fired once heap climbed past ~75% with memory compression and thread-pool queue stress, roughly an hour before heap neared saturation. Node out of memory (OOM) and circuit breakers followed in the same window, as did slow search/indexing. As we can see on the dashboard below, performance drops drastically due to corroborating stress toward the end of the test run:

• On DiskBBQ, where compression was chronic but heap stayed normal, the insight didn’t fire,since storage rather than memory was the limiting factor. Disk and watermark signals are the right path to follow for that profile. As we can see on the screenshot below, all indicators stayed steady with constant performance throughout the test, even though we filled up the disk with more than 70 million vectors on the same instance type:

That timing is the point: Operators get a vector-first read tied to real RAM stress, with subsystem context, rather than an alert on every compressed index or only a red heap chart after the node is fighting on every front.

What to do when AutoOps raises vector memory pressure

Here’s the insight that AutoOps now raises when it detects vector memory pressure:

Recommendations in the product map to concrete actions:

1. Reduce vector footprint where quality allows: Fewer dimensions, quantized mappings, archive or split indices, reindex with a leaner mapping.

2. Tune kNN load: Lower num_candidates, reduce concurrent query rate, narrow filtered kNN where possible.

3. Consider DiskBBQ when HNSW in RAM is the bottleneck (evaluate recall/latency trade-offs for your use case). If you’re already on DiskBBQ and the heap memory is calm, treat disk and watermark insights as the primary signals. Note that DiskBBQ requires an Enterprise license.

4. Right-size RAM when vector off-heap footprint (V) trends up and headroom stays tight.

AutoOps links affected nodes and summarizes regime and stress in plain language. Treat it as “act now, rather than waiting for red on every chart.”

Where AutoOps vector memory pressure monitoring is available

Vector memory pressure is available wherever AutoOps runs against Elasticsearch 9.2+, including:

• Elastic Cloud Hosted (ECH).

• Elastic Cloud Serverless (coming soon).

• Self-managed via Cloud Connect.

AutoOps is included at all subscription levels for supported deployment types and doesn’t consume ECUs on ECH.

They’re ECS-aligned, ready for Discover and out-of-the-box dashboards once you ship the log, no custom schema project. Below: why we built this, how it differs from slow logs, what each line contains, and how to turn it on.

Why we built this (you asked, a lot!)

Coordinator-level query logging has been a very popular request; we listened and delivered! The same pain kept showing up: You want the response duration for Service Level Objectives (SLOs) and dashboards. You want to know the execution time of queries executed in your cluster, and you want to be able to see the full query.

If using cross-cluster search, a search that fans out across clusters looks like one operation from the app or Kibana, but operationally it’s a chain of work: coordination, remote execution, merges, timeouts, and partial results. When something is slow or flaky, teams need to know not only how long the request took but also which clusters contributed and whether the outcome was success, partial, or a hard failure.

What you get: One log stream, one entry per query! Every entry has the coordinator duration (the very same took time that actually matches your search API response), success or failure, and the full query text. Elastic Common Schema–compliant (ECS) JSON, optional duration threshold and user/audit fields, plus X-Opaque-Id that lets you trace a hot query back to the saved object it originates from, and the trace ID so you can correlate with Kibana or your own tooling.

What’s more: Logs follow a stable, ECS-aligned schema, which means you don’t need to design your own ingestion pipelines or field mappings. This consistency enables out-of-the-box dashboards and analytics that work immediately once logs are shipped.

Slow logs vs. query logs: The 30-second version

Slow logs have been the go-to tool for years. They tell you which search operation is slow, but they emit one line per shard that took part, where each line reflects that shard’s piece of the work. This means that they don’t provide a single row that says how long the query execution took, from the client’s perspective. Query logs do exactly that: one line per query, with the end-to-end (wall clock) duration that lines up with the took time in the search API response. This makes them much better suited for understanding workload patterns and identifying problematic queries quickly.

Slow and query logs also differ in when they fire and what they cover. Slow logs only write when a shard’s slice breaches a duration threshold; that is, you’re optimized for “show me unusually slow shard work.” Query logs can record every query (or only those above a configurable threshold you set at the cluster level), so you can tune volume for analytics versus troubleshooting. Slow logs only support DSL queries, while query logs cover ES|QL, DSL, SQL, and EQL, which matches how you reason about “what ran on my cluster” in a modern stack. Both provide the same support in terms of correlation with headers, traces, and audit information (when you turn on user context).

The table below summarizes the main differences between the historical slow logs and the new query logs features.

Slow logs

Query logs

What they’re for

Finding hot shards / slow index operations on specific indices and classic performance tuning inside one cluster.

Understanding what query ran, how long the operation took end to end from the coordinator, and whether it succeeded, which is better for SLOs, analytics, and incident investigations.

Granularity

Per shard (and per phase) for searching slow logs: One user search can produce many lines across shards/replicas.

Per coordinator-level query: One query maps to one log event.

Scope of work

Query + indexing

Query only; indexing will come soon.

What you learn

“This shard on this index exceeded N ms in query/fetch phase.”

“This query (full text), this duration, this outcome, and (when relevant) federation/cross-cluster summary fields.”

Query types

DSL only

ES|QL, DSL, SQL, and EQL

Threshold model

Often tiered (for example, multiple time thresholds per log levels) and per index.

Single duration gate at the cluster level (for example, “log if duration ≥ 500ms”)

What you get in each log line

Every line is one JSON object (one request) in a dedicated file (for example, *_querylog.json under your Elasticsearch log directory). Below is what you can do with the data:

Did it succeed, how long did it take, and what broke? Outcome (whether the request was successful or not), duration (took / took_millis, in line with the API), and a clear failure or timeout when something goes wrong. That’s the core signal for alerting, SLOs, and dashboards: “Are we green? If not, what’s the error?” You also get how many rows or hits came back (result_count), so you can separate “slow but empty” from “slow and huge.”

What actually ran? Query type (esql, dsl, sql, eql) plus the full query text. That answers “Which dashboard rule, saved search, or client pattern is hammering us?” Mix it with duration and outcome to find the worst offenders to fix or throttle.

Who asked for it, and how do I trace it end to end? X-Opaque-Id and trace ID tie a line back to Kibana or your own headers. Task and optional parent task IDs help follow work that was enqueued or chained (async or nested operations).

Cross-cluster search: Who participated, and did anyone misbehave? When cross-cluster search (CCS) is in play, the log can carry remote cluster aliases, per-cluster duration, and status (successful, failed, partial, skipped). You can see at a glance whether a slow search was local or a specific remote dragging the response. DSL can also record that a search was served from a remote alias; ES|QL exposes the richer cluster map; EQL logs a lighter view (for example, which remotes and how many) when remotes are involved.

Security (optional). With elasticsearch.querylog.include.user, you get the usual identity and realm fields (plus effective user when run-as applies), and API key metadata when applicable. Pair with query text and duration for governance and capacity conversations that use names, not only IPs.

There’s more available than we covered here, including additional execution details, shard-level outcomes, and optional profiling information depending on the query type. For every field path and setting, see the Elasticsearch documentation on query logs.

Where the logs live (and how to use them)

Logs land in your Elasticsearch log directory as *_querylog.json (for example, mycluster_querylog.json) on the coordinating node. Ship them with the querylog fileset in the Filebeat Elasticsearch module, so you can then inspect them in Discover (filter by event.dataset: elasticsearch.querylog). On Elastic Cloud, you need to enable Logs on your deployment, and the query logs are shipped as soon as you enable them.

Two workflows. If you need a one-off look to find out who’s hammering the cluster, what the query mix is, or a quick audit, just turn logging on, set a duration threshold so you only log what matters (for example, ≥ 1 s or ≥ 5 min), and then turn it off when you’re done. If you want ongoing query analytics, simply enable logging, point Filebeat at the log, and open a dashboard on the monitoring cluster. Two very simple steps, enable + ship, and you’re done. One request per line, one duration per request, no custom pipeline.

The dashboard below builds upon the new query logs and is provided out of the box. On the top row, you can find the P95/P99 query latencies (with an optional “acceptable latency” bar), the query type breakdown, the success and failure ratio, the user and system queries ratio, and (for DSL) hits versus aggregations. Underneath that, the latency over time (avg, p50, p95, p99, max) with a reference line so you can spot regressions, query volume over time (stacked by type), and tables for top indices, top users, and top error types. Filtering for cluster, user, or index lets you zoom into exactly what you want to focus on.

Heads up. Logging of queries is asynchronous, so it doesn’t block the query execution. Use the duration threshold to cap volume. Also worth noting that at very high queries per second (QPS), we may drop some lines rather than slow your cluster down. For analytics, shipping to a separate monitoring cluster keeps the cluster you’re debugging from taking the extra load.

Some configuration and code samples

Query logging is off by default. Flip it on in elasticsearch.yml or via the cluster settings API. Here’s how.

Enable query logging

In elasticsearch.yml:

Or dynamically via the cluster settings API:

Only log queries above a duration threshold

If you don’t want to log every health check or tiny request, simply set a threshold so only queries that run at least this long get an entry. Duration is in time units:

Include user/audit information

If you use the Security plugin and want to see who ran each query:

Log DSL searches that hit only system indices

By default, searches that target only system indices aren’t logged. To include them, enable query logging and set:

Example log entries

One line = one JSON object = one request with the same shape for ES|QL, DSL, SQL, EQL. Below: a successful DSL search and afailed EQL query with timestamp, duration, query type, and full query. On success, you get result count and shard stats, on failure an error block. User-inclusion and X-Opaque-Id show up when you’ve enabled them.

Success (DSL search):

Failure (EQL query):

Wrapping up

Elasticsearch query logs provide you with one single coordinator-level log for every query (ES|QL, DSL, SQL, EQL). One line per request, coordinator duration, full query, optional user and X-Opaque-Id. Enable it, set a duration threshold and user-inclusion if you want them, and you’re done. Logs live in your log dir (*_querylog.json), and when shipped with Filebeat, you can find them in Discover under the elasticsearch.querylog dataset.

Head to the Elasticsearch documentation on query logs for the full list of configuration settings, and field references. Slow or broken queries can also be found in AutoOps, which leverages the X-Opaque-Id to tie a long-running search back to its origin, such as a dashboard, a saved search, or an alerting rule.

Finally, it’s also worth noting that this new query log is an evolution of the ES|QL-only query log that we released in 9.2. We recommend adopting the new query log since it not only supports ES|QL queries, but also all your other queries.

Now, go see what’s actually running in your cluster.

1. Ingesting JSON data into Elasticsearch

2. Using an Ingest Pipeline to parse JSON fields

3. Querying and aggregating JSON fields

1. Ingesting JSON data into Elasticsearch

When ingesting JSON data into Elasticsearch, it is essential to ensure that the data is properly formatted and structured. Elasticsearch can automatically detect and map JSON fields, but it is recommended to define an explicit mapping for better control over the indexing process.

To create an index with a custom mapping, you can use the following API call:

In this example, we create an index called my_index with a custom mapping for a JSON field named json_field.

2. Using an Ingest Pipeline to parse JSON fields

If your JSON data is stored as a string within a field, you can use the Ingest Pipeline feature in Elasticsearch to parse the JSON string and extract the relevant fields. The Ingest Pipeline provides a set of built-in processors, including the json processor, which can be used to parse JSON data.

To create an ingest pipeline with the json processor, use the following API call:

In this example, we create an ingest pipeline called json_parser that parses the JSON string stored in the message field and stores the resulting JSON object in a new field called json_field.

To index a document using this pipeline, use the following API call:

The document will be indexed with the parsed JSON fields:

3. Querying and aggregating JSON fields

Once the JSON fields are indexed, you can query and aggregate them using the Elasticsearch Query DSL. For example, to search for documents with a specific value in the field1 subfield, you can use the following query:

To aggregate the values of the field2 subfield, you can use the following aggregation:

Bonus: How to deal with unparsed JSON data?

If you are in the situation where you have already ingested unparsed JSON data into a text/keyword field, there’s a way to extract the JSON data without having to reindex everything from scratch.

You can leverage the Update by Query API with the ingest pipeline developed in section 2. But before running that update, you’ll first need to update your index mapping similarly to what we did in section 1 to add the json_field mapping, by running the command below:

When done, you can simply run the command below, which will iterate over all documents in your index, extract the JSON from the message field and index the parsed JSON data into the json_field object.

Conclusion

In conclusion, parsing JSON fields in Elasticsearch can be achieved using custom mappings, the Ingest Pipeline feature, and the Elasticsearch Query DSL. By following these steps, you can efficiently index, query, and aggregate JSON data in your Elasticsearch cluster.