The changes are about storage and performance, along with the out-of-the-box (OOTB) experience. Up until 9.5.0, Elasticsearch operated as a document-based search engine by default. It could be set up to behave like a columnar system storage-wise, but that wasn’t the OOTB experience. Columnar index modes make a number of fundamental changes that allow Elasticsearch to optimize columnar analytic and search use cases:

• Fields are stored once as doc values only and are no longer indexed by default.

• New multi-value semantics. The original ordering of multiple values per field per document (for example, in arrays) is preserved by default.

• Mappings are always flat, and object and passthrough fields in mappings are always auto-flattened.

How columnar index modes fit into Elasticsearch

Many of the columnar index mode changes originate from time series data streams (TSDSs). As part of making TSDB a competitive metrics solution, we improved doc values format on disk and only store dimensions and metric fields once as doc values. We also improved query performance. TSDB is already columnar today. Essentially, this makes TSDB’s storage mode columnar. The lessons learned from TSDB are now being applied more broadly to Elasticsearch. 

Note that columnar index modes are opt-in and columnar, and document-based indices can coexist in the same cluster. An enterprise search use case can use a document-oriented index mode, while a logging use case can use logsdb_columnar index mode and be fully columnar, all in the same cluster. In fact, there are currently seven index modes, and indices can all use them in the same cluster.

How columnar storage stays fast without an inverted index

Indexed fields, either an inverted index for string-based fields or block k-dimensional (BKD) tree for numeric fields, allow Elasticsearch to query or filter by field very efficiently. However, the cost for this is an additional expensive data structure that uses a lot of disk space and is expensive to build at index time and at merge time. With the columnar index modes, fields are no longer indexed by default, so what did we do for query performance to be still acceptable on fields that were no longer indexed?

One major change was improving doc values scanning performance. This is key and is the cornerstone that any columnar system relies on. Previously, the scanning of doc values was essentially document by document. Lucene’s doc values API only allowed for looking up one value at a time. Historically, this fit the execution model of a search engine. In our own doc value format, we build the capability to allow bulk reading of values for Elasticsearch Query Language (ES|QL) queries. Also over recent minor Lucene releases, Lucene doc values API added support for bulk reading. Without this, fast columnar scanning wouldn’t have been possible.

Secondly, we fully adopted doc value skippers, a hierarchical skiplist over doc values. Contrary to an inverted index or a BKD tree, doc values skippers are lightweight data structures. At its core, a skipper allows queries to skip over a range of documents that don’t match a query. It can do this because it stores information, like min and max values. So, for example, when a range query is executed, an interval of documents can be skipped based on the intermediate result and a doc value skipper’s min and max values. The effectiveness of doc value skippers depends on the order in which documents are laid out on disk. This is why index sorting should be enabled or altered to match the use case.

By significantly improving our columnar scanning and doubling down on doc values skippers, we’re able to avoid indexing fields by default. Note that a field can still be indexed; the index mapping attribute just defaults to false in columnar mode. The exceptions to this rule are text-based fields, which are still indexed by default. This is because text fields provide free text search, which includes text analysis, along with phrase and wildcard matching. This is different from just filtering.

How Elasticsearch handles high and low cardinality fields

When setting up a schema with string fields, an important configuration parameter is often cardinality; that is, whether many unique values or a few unique string values are expected. Many systems have dedicated field or column types that target low and high cardinality string fields.

Fields that have low cardinality are typically stored with a dictionary, containing all unique values. Then, for each row offset, the offset into the dictionary containing the term the row has is stored. This is often called an ordinal. For low cardinality fields, this works well, as storing an ordinal per document takes up much less space. Encoding techniques, like delta encoding, offset encoding, and bitpacking, work well for ordinals to compact the per-document storage to just a few bits. 

However, for high cardinality fields, the dictionary and ordinal approach can work counterintuitively. If a larger percentage of the documents have a unique value, building the dictionary becomes expensive and storage savings diminish. The dictionary then becomes another level of indirection for reading values. This is why most systems in that case store values in a columnar fashion using block-based compression. For example, values of multiple rows are stored in 128KB blocks using a sliding-window dictionary-based compression algorithm (like zstandard or lz4). This, in general, is a simple and effective method to store higher cardinality fields and avoids building and maintaining a dictionary. 

With document-based Elasticsearch, there are two ways to map a string: using either the keyword field mapping or one of the text-based field mappings. The former stores an inverted index and dictionary-based doc values. The latter only stores an inverted index. This is why, typically, a text field mapper is often used in combination with a keyword mapper as a multi-field. Also, keyword field mapper (as the name suggests) is meant for keywords or fields that have a lower cardinality and uses dictionary-based doc values implementation. However, in practice, keyword field mappers are also used for high cardinality.

In columnar mode, every field is only stored once by default. For keyword fields, this means only doc values are stored with no inverted index. Text-based fields now also store doc values and an inverted index by default. Text-based field mappers are different from the keyword field mapper, as these are not automatically used via Elasticsearch’s dynamic mapping logic for columnar indices, and therefore text fields keep storing an inverted index by default.

For both keyword- and text-based fields, we didn’t choose to expose a cardinality mapping attribute. It’s not always possible to know ahead of time whether a field is low or high cardinality. When flushing and merging segments to disk, Elasticsearch sees all values and can determine the cardinality of a field. This is why we’re choosing to automatically determine whether the usage of a dictionary and ordinal-based encoding is beneficial over block-based compression using a simple cardinality threshold. If a field is below this threshold, dictionary and ordinal-based encoding is used; otherwise block-based compression is used. This simplifies configuration of the mappings and makes it possible to automatically optimize storage as data evolves, since some segments may use dictionaries while others may use blocks for the same field. However, this is currently not ready yet and so, as part of 9.5.0, in columnar mode, both keyword- and text-based field mappers store values in doc values in a block-based compressed layout on disk.

The two approaches compare as follows:

Dictionary and ordinal encoding

Block-based compression

Suits

Low cardinality fields

High cardinality fields

What’s stored

A dictionary of unique values, plus one ordinal per document

Values for many documents compressed together in blocks

Compression

Delta encoding, offset encoding, and bitpacking reduce each ordinal to a few bits

Sliding-window dictionary compression, such as zstandard or lz4, typically over 128KB blocks

Read path

Resolve the ordinal, and then look up the value in the dictionary

Decompress the block, and then read the value directly

Cost as cardinality rises

Dictionary grows large, savings shrink, and the extra indirection stays

Stable, with no dictionary to build or maintain

Used in columnar mode tech preview

Not yet

Yes, for both keyword and text fields

Columnar mapping attributes: multi_value, nullability, on_failure

The columnar index modes provide more control over how data is stored as doc values. By default, Elasticsearch is lenient and accepts all non-malformed values (for example, nulls and multiple values per field and document). If documents have fields with multiple values per document or no value, doc values store additional data structures to deal with them and therefore implicitly increase costs. 

With columnar, new mapping attributes provide additional control. Note that these new mapping attributes are currently only available with the columnar index modes but will eventually also be available for all index modes.

Three new mapping attributes are involved:

Attribute

Default

Enforces

On violation

Available

multi_value

true

One value per document per field

Document indexing fails

9.5.0

nullability

true

Field must have a value

Document indexing fails

9.5.0

on_failure

fail

How the above failures are handled

Sets fail or ignore behavior

Next minor release

multi_value: Enforcing single-valued fields

By default, Elasticsearch accepts multiple values per document. To understand the implications of this, we first should take a look at how Elasticsearch (using Lucene’s doc values) stores a dense numeric field where all documents have a single value:

With this layout, all values are stored in blocks. The number of values per block depends on the index mode but is typically 128 values and is always the same within an index. All values in a block are encoded using various encoding techniques, like delta encoding and bit packing, so each block can have a different size, depending on how well the encoding techniques compress the values. This is why a block index is required. 

Lucene has the notion of a docid (internal numbering for a document), which is essentially a row identifier.

1. Queries produce matching docids.

2. In case of a dense field, the block id can be resolved from the docid directly.

3. The offset of a block can be resolved from the block index.

4. Once that has been looked up, the target block gets decoded and all values are available.

5. Finally, from docid, the ordinal within the decoded values array can be resolved, which produces the final value.

Now let’s have a look at how the data layout changes when documents have multiple values per document:

To determine how many values belong to a single docid, an offset lookup is required.

In this case, a docid has one or more offsets. Each offset points to a block index. Values for a single document are adjacent but can stretch over blocks. In general, compaction of values works well in blocks because values are similar. However, multi-value fields can cause the compaction of values to be less efficient if the number of values per field and document is large and values aren’t similar. This and the additional storage of offsets result in multi-value fields typically having a higher storage footprint on disk.

If a field is truly single-valued, you may want to enforce this property. The multi_value mapping attribute makes that possible now. An example is a log level field. Logs typically have one log level (such as debug, info, or error). Enforcing that this field is single-valued in your mappings can help avoid accidentally using more storage than anticipated. A mapping snippet example that disallows the field log.level to have multiple values per document:

Note that even if a field allows multiple values, this doesn’t mean an offset lookup is stored. This only happens when a Lucene segment has at least one document with two or more values. The multi_value mapping attribute exists just for enforcement.

nullability: Requiring every document to have a value

By default, Elasticsearch accepts documents with fields that have no value or null value. Just as  multiple values per document require additional accounting, documents with no value require additional accounting to identify which of them have at least one value.

Doc values store a docid to offset lookup (known as IndexedDISI in Lucene) in case not all documents have a value in a segment. The offset either points directly to the block index for single-valued fields or the offset lookup in case of multi-valued fields. This lookup is compact compared to the value blocks being stored. However, if it were to be created for fields that should have at least one value per document, that would be a waste.

The nullability mapping attribute allows you to control whether documents are allowed to have no value. Just like the multi_value mapping attribute, the nullability attribute exists for enforcement.  Following is a mapping snippet example that requires the log.level field to have a value:

on_failure: What happens when validation fails

What happens if a document has multiple values for a field and if the multi_value mapping attribute is set to false or when a field is mapped with nullability set to false and a document doesn’t have that field? At the moment, indexing such documents will fail with a bad request error.

As part of the next minor release, the on_failure mapping attribute will be available. This allows you to indicate how to handle these validation failures, on a per-mapped field basis. This will support two values:

1. Fail: Fail indexing of the entire document with a client error. This is the current behavior in Elasticsearch 9.5.0.

2. Ignore: Ignore the validation error, mark the field as ignored, and store values for that field in a hidden field so that it can be introspected when requesting the source. 

Trying the columnar index modes

The columnar index modes are still under active development, but we encourage you to give them a test drive. As we prepare the columnar index modes for general availability (GA), we’ll add more performance and efficiency improvements. We believe that by adapting a columnar mindset, many use cases will benefit from being more cost effective or having better performance characteristics.

How historical metrics were loaded before backfill

Even if loading historical metrics isn’t a very common use case, it’s an important step when teams are adopting TSDB. Two scenarios have been the most prominent: bootstrapping a new time series data stream and migrating data from a different system or data stream to a time series one.

Bootstrapping a new time series data stream

You want to start a new time series data stream with a week of historical data so you have something meaningful to query from the start. With existing tooling, you had to set index.look_back_time to the seven-day maximum in the index template, and all historical data would land in a single backing index. For anything beyond seven days, you needed to create past backing indices manually.

Migrating metrics from another system

You have months worth of metrics stored on a different system and want to move your full dataset to TSDB. You need to load months of metrics history alongside live ingestion. The workaround was to manually create all the necessary past backing indices with the right time_series.start_time and time_series.end_time and to index into it directly using the index name. You then attached it to the data stream via the modify data stream API. It worked, but it required understanding the index time semantics and repeating the steps for each time window, along with coordinating that process around ongoing writes.

We wanted both scenarios to feel as close to normal bulk indexing as possible.

What time series data backfill changes

In 9.5, Elasticsearch can create backing indices covering past time ranges, which extends the eligible write window backward.

The eligible write windowis the range of @timestamp values that a time series data stream accepts for new documents. 

In the past, the eligible write window was determined only by the existing writable backing indices at the moment the request was received by Elasticsearch.

In 9.5, Elasticsearch can expand the eligible write window in the past by creating backing indices. This converts the eligible write window to a sliding window extending from the present back to the first read-only or destructive lifecycle action. Common examples of these actions, which are typically defined within your lifecycle configuration, are downsampling or searchable snapshots. Examples also include retention configurations. 

So, given that loading historical data is enabled in a cluster, the eligible write window of the data stream with the following lifecycle configuration is determined by the downsampling action, because it’s the first action that makes backing indices read-only. So, for this data stream Elasticsearch accepts documents whose @timestamp is no older than three months.

Why loading historical data into TSDB is hard

TSDB consists of data streams optimized for timestamped measurements. It uses a columnar storage layout and enforces immutable dimensions. It also organizes data into time-bound backing indices; each index covers a specific time range and accepts only documents whose @timestamp falls within it.

As time passes, rollover creates new backing indices to cover upcoming ranges. Until this release, there was no corresponding mechanism for the past. Creating indices in the past is tricky because historical data might span over a long period of time and can arrive at Elasticsearch out of order. Consequently, Elasticsearch cannot determine the write timeframe that its backing index should cover. Our solution to this is to use a preconfigured interval and lazily create past backing indices.

How Elasticsearch creates past backing indices

When a document is detected whose timestamp isn't covered by any existing backing index, Elasticsearch determines the time boundaries for the missing indices and creates them. It then adds them to the data stream in a single atomic operation. 

Lazily creating the indices ensures that a single request in the past won’t overwhelm the cluster by requiring the creation of 300 indices all at once. It also doesn’t create indices before there are docs to write into them.

Proactive vs. reactive: How we chose the index creation approach

We explored two ways to detect when a past backing index needs to be created.

The first is proactive. Inspect each incoming document's timestamp before routing, and create any missing past backing indices up front. This keeps the write path clean. By the time a document is routed, the index it needs already exists. It does require the data stream to already exist with at least one time series backing index, since that's what we inspect to determine the eligible write window and the time boundaries of the new index. The downside is that it adds work to every bulk request targeting a time series data stream, even requests that contain no past timestamps and need no backfill at all.

The second is reactive. Let the document fail the normal indexing, intercept that failure, create the missing index, and retry. This avoids any overhead on the common case, since the extra work only happens when a mismatch actually occurs. The tradeoff is more complexity in the failure handling path and a retry on every backfill document.

We ran performance tests on the proactive approach against bulk requests with no past timestamps and found no measurable regression. The overhead of inspecting timestamps turned out to be negligible. That settled it. Proactive creation is simpler and consistent with how index auto-creation already works in Elasticsearch. Plus, it adds no measurable cost to the workloads that don't use backfill.

How Elasticsearch determines past index boundaries

Each new past backing index has three properties to compute: its duration, its start time, and its end time.

Property

How it's set

Constraint

Duration

Defaults to one day, configurable via the cluster setting data_streams.past_tsdb_index_interval

Minimum one hour. If the triggering timestamp falls in a gap up to 1.3 times the configured duration, Elasticsearch collapses it into a single bridging index rather than creating many tiny ones.

Start time

Anchored to the start of the next existing backing index, working backward in multiples of the configured duration

Increased to match the end time of the previous neighboring index, where they would otherwise overlap.

End time

Start time plus the configured duration

Reduced to match the start time of the next index, where they would otherwise overlap.

Handling concurrent writes

In a distributed setup, multiple nodes can receive bulk requests with overlapping past timestamps at the same time. Each node collects the timestamps that aren’t matching any of the existing indices and sends a request to the master node. 

The master node executes a cluster update that sorts them and then, one by one, checks whether the timestamp is covered by an existing or newly created index. Otherwise, it issues a new create index request with the time boundaries calculated as described above. The cluster updates are always sequential and guaranteed to produce valid cluster states, so new indices are guaranteed to not overlap with existing indices.

How lifecycle age works for backfilled indices

Past backing indices hold old data but are new indices. Without an adjustment, lifecycle features would apply downsampling and retention based on when the index was created rather than when the data is from. We account for this by using the index.time_series.end_time as the index.lifecycle.origination_date. As a result, the age of the index as perceived by both data stream lifecycle and index lifecycle management (ILM) is based on the age of its data and not its creation time.

How to use time series data backfill

How to enable time series data backfill

Backfill support ships disabled by default. Enable it at the cluster level:

Bootstrapping with historical metrics

To load historical data into a new time series data stream:

1. Create your index template. 

2. Initialize your data stream. (This is an important step because an existing data stream is a requirement for creating past backing indices.)

3. Start indexing. 

Past backing indices are created automatically as documents with historical timestamps arrive, each covering one day of data by default. No additional configuration is needed.

Data migration into an existing data stream

Migrating data within the eligible write window

For data that falls within the eligible write window of your data stream, point your migration pipeline at the data stream and let Elasticsearch manage the rest.

Migrating data beyond a read-only action

For data older than the write window (for example, you're migrating 18 months of metrics but downsampling kicks in after seven days), you need a separate data stream without read-only lifecycle actions. Retention isn’t an issue since the data would be deleted anyway. The pattern is:

1. Create an index template for the historical data stream, using the same mappings as the original but without a lifecycle:

2. Create the historical data stream. If this step isn’t executed, the first indexing request might fail. During the first indexing request, Elasticsearch can create the data stream but it cannot yet create any past backing indices, so indexing a historical document might fail. Creating the data stream explicitly ensures that all indexing requests will be accepted:

3. Index historical data into the historical data stream while current data continues flowing into the original.

4. When the load is complete, add lifecycle. This is only supported by data stream lifecycle since this feature functions on a data stream level:

5. Query across both data streams with a wildcard pattern (my-metrics*) or a data stream alias.

6. If retention is configured, delete the historical data streams when their data expires. Data stream lifecycle will delete the data but it won't clean up the data stream itself.

As you see, the historical data needs to fit on the target tier as a whole because lifecycle will be enabled after the data is loaded. If you have a large historical import, you might choose to split it into batches. Make sure each batch can fit on the target tier as a whole at the time of indexing, to avoid running your cluster out of disk space. Lifecycle will start processing the batch's indices as soon as it's enabled, but it will need time to process the whole backlog.

Protecting the cluster during large migrations: Downsampling floodgate

When data stream lifecycle runs against a data stream with many indices that all qualify for downsampling, it queues them simultaneously. Downsampling is CPU and I/O intensive; it reads and rewrites all data in an index. Queuing dozens of operations at once can overwhelm the master node with persistent task updates while it coordinates them.

The downsampling floodgate scenario could occur before backfill support (for example, when adding a lifecycle policy to an existing data stream with months of accumulated data). Backfill makes it more likely by design.

In 9.5 and serverless, we added flood protection to data stream lifecycle. It now tracks how many indices per data stream are actively being downsampled. If that count reaches a threshold, data stream lifecycle pauses queuing further operations for that data stream until the count drops. The threshold is configurable via the cluster setting data_streams.lifecycle.downsampling.max_indices_in_progress. Other data streams aren't affected.

Limitations and prerequisites of time series data backfill

• Backfill doesn’t apply to read-only indices. If downsampling or a searchable snapshot transition has already run on a time period, documents for that period are still rejected.

• The feature requires a preexisting time series data stream with at least one time series backing index.

• System data streams are excluded.

• Replicated data streams rely on the leader data stream, so no direct backfilling is possible.

• Scaling remains your responsibility. Loading months of data can trigger significant storage usage, force merge operations, and lifecycle activity in parallel. Check that your cluster has the headroom to manage it before starting.

Conclusion

Prior to the Elasticsearch 9.5 release, loading historical data into TSDB was a manual process. By automating the generation and management of past backing indices, we aim to transform historical data migration to a native capability of your standard ingest pipelines. The inherent complexity of managing time-bound indices remains, but it has transitioned from a user responsibility into an internal Elasticsearch function. Whether you’re bootstrapping a fresh data stream or migrating extensive historical datasets, the platform now handles the heavy lifting, allowing you to focus on analyzing your metrics. We look forward to seeing how these improvements streamline your adoption of TSDB.

Observability workloads are storage-intensive by nature, and the composition of that storage determines both cost and query performance. ES95 introduces adaptive compression: rather than applying the same encoding to every numeric field, it automatically selects the encoding that best matches each field's structure. The result is a 33.6% reduction in total doc-values storage, 19% to 74% reduction on floating-point gauge metrics and a 92% reduction in @timestamp. No configuration or migration required.

Observability data is storage-intensive

Data processing systems are rarely limited by how fast they can compute. They’re limited by how fast they can move bytes: off disk, across the network, and through the memory hierarchy. Compression is how a storage engine trades CPU time for memory bandwidth, spending comparatively cheap CPU cycles so fewer bytes have to travel through the parts of the system that are usually constrained. In a read-heavy system like Elasticsearch, that trade-off pays back every time data is queried, often long after it was written.

Storage size and query performance move together; fewer bytes on disk means fewer bytes to read on every range query, every aggregation and every dashboard load. Compression is not just about saving storage. Every byte that is never written is also a byte that never has to be read.

The right encoding depends on the structure of the values themselves, and the largest wins come from exploiting the structure already present in the data rather than squeezing an opaque stream of bytes. Few workloads expose that structure more clearly than observability metrics.

A single host reports hundreds of metrics every few seconds, including CPU utilization, memory ratios, request latencies, and network throughput. Multiply that by thousands of hosts across weeks of retention, and the bytes accumulate fast. Most of that volume is structured but not uniform: timestamps arrive at near-constant intervals from thousands of concurrent series, counters increase monotonically, while gauges like 23.47 or1.15 are short decimal measurements.

A fixed compression approach cannot adapt to that variety. A timestamp column and a floating-point gauge column compress through fundamentally different techniques, but a codec that applies the same approach to both will necessarily handle one of them poorly. For most of Elasticsearch's time-series codec history, gauges were on the losing end of that trade-off.

The structure the old codec was not built to exploit

Elasticsearch stores time-series numeric values in doc values: a column-oriented structure where all values for the same field sit adjacent on disk. That adjacency makes compression possible: the codec compares consecutive values of the same field, finds patterns, and exploits them.

The time-series codec before ES95 applied the same fixed encoding to every numeric field: delta encoding followed by normalization, GCD (greatest common divisor) reduction, and bit-packing. Each encoding technique activated where it helped and skipped where it would not. For timestamps and integer counters, this approach was remarkably effective. For floating-point gauges, it could find almost nothing to work with.

The reason is how they’re stored. To support range queries, Elasticsearch stores floating-point values as integers that preserve numeric ordering. A change of 0.01 in a CPU percentage reading translates to a jump of trillions in that integer space. The codec sees those large jumps and has no strategy to further reduce their footprint. Storage stays near the original eight bytes per value.

The codec was doing the right thing with the representation it had, but the latter was chosen for querying, not compression, and the goals conflict at the bit level.

The cost of a fixed format

A compression stage for floating-point values was already on the roadmap, so the interesting part wasn’t the algorithm. The obstacle was architectural.

The previous codec baked its compression approach into the storage format. Adding a new encoding meant changing the meaning of existing bytes on disk, which forced a format migration, a rollout that can last weeks or months in large production clusters. Over time, that migration burden constrains codec development itself. The question stops being Is this a good compression idea? and becomes Is it worth another format migration? That rigidity limits the cadence of codec evolution and leads to missed compression improvements.

The right encoding without configuration

ES95 solves this at the architecture level for time-series indices. Each field's encoding is no longer baked into the format. It is selected automatically at write time, based on what the field mapping already declares: the field's name, its data type, and its metric role. Timestamps are encoded differently than counters. Counters are encoded differently than gauges. ES95 encodes all of them, and it chooses the right strategy for each.

Users already tell Elasticsearch everything the codec needs to know. The mapping describes the data; the codec chooses the compression strategy.

Compression strategy is a codec concern, not a user concern.

The alternative would have been to expose per-field encoding selection as a configuration parameter, letting users opt in to better compression for specific fields. That would shift the burden of knowing which encoding fits which data type onto those least equipped to make that call and would guarantee that most deployments never see the benefit. ES95 keeps that decision inside the codec, where it belongs. This matters most in managed and serverless deployments, where users expect the system to automatically make optimal storage decisions.

The timestamp result nobody planned for

With the adaptive architecture in place, the team set out to ship the planned float-compression algorithm. Before it arrived, the architecture proved itself by substantially improving compression for timestamps.

A time-series index is sorted first by its time-series identifier (_tsid constructed by the metric’s dimensions) and then by timestamp within each series. Timestamps on disk aren’t one smooth sequence; there are many smooth sequences laid end to end, one per series, with a large jump at every boundary where one series ends and the next begins.

The codec compresses data in fixed-size blocks without regard to those series boundaries. A block straddling a series boundary holds timestamps from two different series. The jump between them breaks monotonicity, reducing delta encoding effectiveness on blocks spanning different time series. A block that would otherwise compress to near-zero bits per value ended up needing nine or more, because bit-packing encodes every value in a block using the same fixed number of bits, so one large jump sets the cost for all of them.

In the ideal case, every block belongs to a single series: timestamps increase at near-constant intervals, delta encoding captures the regularity, and bit-packing compresses the result to near-zero bits per value. With few series, boundary blocks are rare and the overhead barely registers. On an observability cluster ingesting millions of documents across thousands of series, that changes. The cost scales along two dimensions: series count and data density. More series means more boundary events; sparser series means multiple jumps packed into single blocks. In high-churn environments, both compound, and boundary blocks accumulate into a standing tax on the most-read field in any time-series workload. It’s why @timestamp storage grew faster than the data that produced it.

The fix was to detect series boundaries and treat each run as its own independent sequence. Instead of trying to encode across the jump between two series, which forces every value in the block to pay the storage cost of that one large jump, each run is compressed on its own terms. The boundary simply becomes a seam: the jump is never seen by the encoder on either side.

That encoding is called SplitDelta. @timestamp and monotonic long counters now use it by default. No format change. No migration. Existing segments retain legacy encoding.

On the high-cardinality Elasticsearch Rally benchmark, that single unplanned encoding cut counters storage by 20%–30% and @timestamp storage by 92.4%, from 1.03 GB to 79 MB. Gigabytes to megabytes, and no, that isn’t a typo.

The pluggable pipeline had already paid for itself. SplitDelta, which wasn’t part of the original plan, slotted in without a format change or migration before ALP even shipped.

ALP: recovering the decimal that was always there

Most floating-point metrics can be expressed as short decimals with no loss of accuracy: CPU utilization at 23.47, load average at 1.15. ALP (Adaptive Lossless floating-Point compression) recovers that decimal structure from the floating-point representation, converting values into integers that the existing pipeline already handles well. ES95 feeds ALP's output into the same mature integer compression pipeline used for timestamps and counters, extracting additional savings rather than treating ALP as a standalone encoding. Values that don’t fit ALP's model (such as irregular high-precision floats or special values) fall back to direct bit-packing or the original representation without degrading the rest of the block.

ALP lets Elasticsearch treat floating-point metrics according to the structure they actually contain rather than the binary representation they happen to use. It’s applied automatically to double-valued gauge fields, selected by field type and metric role, through exactly the door the architecture had built for it.

What the numbers say

Here’s what the Elasticsearch Rally benchmark looks like on a high-cardinality workload containing 2.26 billion data points. Results are from an internal tsdb-metricsgen benchmark.

Field or metric

Storage reduction (%)

@timestamp

−92.4%

cpu.load_average.5m

−74.3%

system.cpu.utilization

−63%

memory.utilization

-19%

Total doc values

−33.6%

That overall 33.6% reduction deserves context.

A time-series index contains more than metrics. Every data point also carries the labels that identify the series: host names, IP addresses, regions, container IDs. Those dimension fields are stored as keywords. ES95 doesn’t target dimensions.

On this benchmark, two dimension fields, host.ip and host.mac, accounted for 44% of doc-values storage after ES95 ran. The 33.6% total reflects that mix. The per-field breakdown is the honest picture. Compression for dimension fields is an active area of work.

The per-field variation is the most convincing result. Some gauges shrank by nearly three quarters, while others moved by less than a fifth. That spread is direct evidence that ES95 matches compression to the structure actually present in each field. A fixed encoding treats every field identically and misses most of those wins.

Better compression without extra configuration

The storage reductions from SplitDelta and ALP are the most visible results of ES95. The more consequential result is the architecture that produced them.

Before ES95, every new compression technique required a format evolution. That reality shaped which ideas were practical to pursue. Today, new encodings become implementation decisions inside the codec rather than migration projects. Existing data never needs to move, and users gain better compression on newly written data simply by upgrading Elasticsearch. SplitDelta and ALP are the first encodings to benefit from this architecture. They will not be the last.

Asking users to choose compression algorithms would only duplicate information Elasticsearch already has. There are no per-field compression parameters to tune, and no expert knowledge is required to get good storage efficiency. Different fields get different strategies because ES95 understands what kind of data each field contains, not because a user configured it. As the codec evolves, those decisions evolve with it. The API does not.

In Elasticsearch Serverless, good defaults are part of the product. Users expect the system, not configuration, to make storage decisions. ES95 is designed to honor that expectation: encoding that starts right and gets better over time.

The compression was always there

ES95 establishes a new standard for how time-series codec evolution works. New encodings become implementation decisions, not migration projects. Users get better compression on newly written data with every Elasticsearch upgrade.

The compression was already in the data. ES95 just removed what was hiding it.

How to configure frozen_after in data stream lifecycle

frozen_after sits at the top level of the lifecycle, next to data_retention and downsampling.

That's the whole feature, at the API level. Indices in my-data-stream stay on hot for 30 days, then move to frozen for the remaining 60. After 90 days they're deleted, and the backing snapshot goes with them.

It composes with the rest of the lifecycle, including downsampling:

Same options in an index template:

The order of values is enforced: frozen_after has to be less than data_retention and greater than any downsampling.after. The API rejects configurations that don't make physical sense.

Where frozen tier data is stored: the default snapshot repository

Frozen tier data is held as partially-mounted searchable snapshots, which means DLM needs a snapshot repository to write into. Rather than make you choose a repository per lifecycle, 9.5 introduces a cluster-level default snapshot repository.

DLM uses this repository for every frozen tier index in the cluster. On Elastic Cloud Hosted (ECH), the default is pre-populated with found-snapshots so existing clusters work out of the box. You can change it to a repository you control if you'd rather keep your frozen data in a bucket you own (useful if you want object versioning, lifecycle backups to Glacier, or anything else that needs bucket-level access). Wherever you can set frozen_after in Kibana, the UI shows the current default repository inline and links to the place to change it, so you can see where frozen data will be written.

If the cluster doesn't have a default repository configured, you can still write a lifecycle with frozen_after. The API accepts it but returns a warning:

The data stays on hot until a default repository is configured and exists. The same logic applies if the cluster lacks a valid Enterprise license. Errors are visible in the data stream lifecycle status API for that stream.

Configuring frozen_after in Kibana

Kibana in 9.5 lets you set frozen_after and the default snapshot repository from the UI. In Streams, the Retention tab shows the frozen phase on the lifecycle timeline alongside hot and any downsampling steps. Click the timeline to open the data lifecycle flyout, set frozen_after, and see the timeline update before you save. Index Management's Data Streams page opens the same flyout.

How frozen tier conversion works in data stream lifecycle

When a backing index ages past frozen_after, DLM walks through five steps in order:

1. Clone. Mark the index read-only and clone it to a zero-replica copy so the original stays available while conversion runs.

2. Force merge. Merge the clone to a single segment. On completion a cluster state marker is written; duplicate force-merge requests (for instance after a master failover) are deduplicated, so a restart doesn't repeat the work.

3. Snapshot. Write the merged clone to the default repository, and record the snapshot name in cluster state on success. If a stalled snapshot from a previous attempt is detected, DLM deletes it and re-runs the step.

4. Mount. Create a partially-mounted searchable snapshot index from the snapshot.

5. Swap and delete.Once the mounted index's shards are fully allocated, atomically swap it in for the original in the data stream, then delete the original. The swap is atomic, so query results don't see a data volume dip during the transition.

On failure, DLM retries from the last successful step on the next run.

Each step is idempotent, and the cluster state markers make sure work already done isn't repeated after a master failover. Throttling caps concurrent conversions so a lifecycle change covering thousands of indices doesn't overwhelm the cluster. Errors surface in the data stream lifecycle status API and roll up to the lifecycle health indicator.

Scope and limitations of frozen_after

• Data indices only. frozen_after applies to a data stream's data indices. Failure store indices don't currently support the frozen tier and continue to be governed by their own data_retention setting.

• Enterprise license required. Frozen tier in DLM is implemented with searchable snapshots and requires an Enterprise license. You can write frozen_after on any license, but data won't move to frozen until the license is valid.

• Serverless ignores the field. In Elastic Cloud Serverless, frozen_after is accepted but ignored - Serverless manages tiering on your behalf. Built-in templates may include the field, so we don't reject it, but the step is skipped.

Getting started with frozen_after

1. In Kibana, open Streams or Index Management and choose a data stream backed by data stream lifecycle.

2. Open the Edit data lifecycle flyout, set a frozen-after value, and save. The lifecycle timeline shows the new phase.

3. On a self-managed cluster, set repositories.default_repository to a repository you control. On ECH, found-snapshots is already configured if you want zero setup.

4. For declarative workflows, write the same configuration into your index templates so new data streams pick up the lifecycle automatically.

Learn more

• Data stream lifecycle

• Frozen tier overview

• Searchable snapshots

• Manage data retention for Streams

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

The palette takes shape: how multimodal search in Elasticsearch evolved from semantic_text

The semantic field is a convergence of several complementary features we've introduced over the past couple of years, bringing them together to create a cohesive multimodal search experience. Each solved an important piece of the semantic search puzzle on its own; together they enable native multimodal search.

The first brushstroke was semantic_text. Before it, running semantic search meant manually configuring mappings, wiring up ingest pipelines with an ML model, manually chunking content, and generating query-time embeddings yourself. The semantic_text field folds all of that away: it performs inference automatically at ingest time, chunks long documents for you, and simplifies the queries you write against it. Introduced in Elasticsearch 8.15 and released as GA in Elasticsearch 8.18, it has become the foundation for semantic search on the platform.

Next came the model to power multimodal search. jina-embeddings-v5-omni is our family of multimodal embedding models, capable of embedding text, images, video, audio, and PDFs into a shared vector space. Because those embeddings are semantically compatible across modalities, you can store diverse media in a single index and query across all of it at once, such as retrieving an image via a text description or matching audio against a written phrase, all without maintaining a separate pipeline for each content type. For more detailed information about how these embeddings are generated, see the model documentation.

We added the embedding query vector builder in Elasticsearch 9.4 to handle multimodal inputs at query time. Query vector builders are general-purpose tools you can use to convert input to a vector at query time as part of your request. For example, we have the text_embedding query vector builder for text-only models and input, and the lookup query vector builder for getting a vector from an existing document. The embedding query vector builder is a new type that works with multimodal models and accepts multimodal input, including text or base64-encoded binaries. This allows you to pose a query in whatever modality fits, and Elasticsearch generates the matching vector on the fly.

The final piece was multimodal ingest. The semantic_text field brought automatic embedding to text; the semantic field extends that same automatic experience to images, audio, video, and PDFs from ingest through query.

Painting the picture: creating an index with the semantic field

Let’s create an index with a semantic field. This is as simple as setting the field type to semantic and defining the inference endpoint you want to use:

In this example, we use the .jina-embeddings-v5-omni-small inference endpoint. This is our built-in jina-embeddings-v5-omni inference service, and it is available in all environments with access to the Elastic Inference Service (EIS). This includes:

• Serverless.

• Elastic Cloud Hosted (ECH).

• Self-managed with Cloud Connected Mode (CCM).

Indexing images, audio, video and PDFs

To index an image, provide an object with a type of image and a value containing the image as a base64-encoded data URL:

Arrays of objects are also accepted, allowing you to index multiple images in a single field value:

The semantic field also supports text values, just like semantic_text. You can provide such values standalone or intermix them with image values:

Text values are handled just like they are with semantic_text: long passages are chunked according to the chunking settings configured on either the inference service or field mapping. Multimodal values, such as images, are not chunked. Each multimodal value is represented as one chunk.

Other modalities are supported as well. Change the type value to match your content’s modality. Currently we support:

• image

• audio

• video

• pdf

For example, to index a video, the request would look like:

Image search and cross-modal retrieval with a text query

To find multimodal content using a text description, run a match query on the semantic field:

Just like with semantic_text, Elasticsearch automatically generates an embedding for the query text using the inference endpoint associated with the field. That query embedding is used to return semantically similar matches.

This query pattern enables easy text-to-image search. Just index an image and use a match query to retrieve it via text description! It also works for any other modality: index the multimodal input and search by description to retrieve it.

Querying with images, video, and other multimodal inputs

We can also search using a multimodal input by using the knn query with an embedding query vector builder. For example, we can search using an image:

The input object format is the same as when providing an image to index: set the type to image and value to a base64-encoded data URL.

Similar to when querying by text description, Elasticsearch automatically generates an embedding for the query image using the inference endpoint associated with the field. That query embedding is used to return semantically similar matches.

Just like with indexing, other modalities are supported, but are limited to those supported by your inference endpoint. For example, a search using a video clip would look like:

Extending the composition: highlighting, retrievers, and other semantic field features

The semantic field didn't start from a blank canvas. It's built on the same foundation as semantic_text, inheriting its behavior and its ergonomics, and extending them to multimodal content. In practice, that means nearly everything you already know about working with semantic_text carries over unchanged. If you've built with semantic_text before, the semantic field will feel immediately familiar.

Here’s a selection of the features that come along for the ride. See the documentation for a complete list.

Highlighting the best-matching chunks

If you index multiple values in a semantic field, you may want to know which value best matches the query. The semantic highlighter can be used to return the most relevant chunks as highlight fragments:

Setting order to score returns the fragments ranked by relevance, while number_of_fragments caps how many chunks come back. The response looks like:

Note how highlighted multimodal values are represented using their data URLs.

Controlling vector quantisation with index options

The semantic field stores its embeddings in an underlying vector field, and index_options lets you control how that vector field is indexed. For example, choosing a non-default quantization strategy:

Multi-field retrievers

The semantic field participates in the multi-field query format supported by the linear and rrf retrievers. Rather than hand-writing an inner retriever per field, you supply a single query and a list of fields, mixing lexical fields and semantic fields freely:

The retriever automatically separates lexical fields from semantic fields, queries each group, and normalizes the results so that each group contributes equally to the final ranking, preventing lexical matches from drowning out semantic ones.

Cross-cluster search

The semantic field supports cross-cluster search (CCS), enabling use of the field in large, multi-cluster deployments. Simply list the indices to query using the standard <cluster>:<index> format:

The fields queried across indices and clusters can use a mix of different inference endpoints that produce different query embeddings. The search request will automatically apply the proper query embedding to each individual field queried.

Off the easel, into the world: optimising multimodal embeddings for production

When you move multimodal search from experiment to production, the size of your multimodal inputs becomes a practical concern. Multimodal data is supplied as base64-encoded data URLs, and that data is stored in the index. Those strings can grow large in a hurry: a single high-resolution file can balloon into several megabytes of encoded text, which has several side effects:

• The index size on disk can increase significantly.

• Requests and responses containing multimodal data are larger, increasing transmission time and ingress/egress costs.

• Inference on larger multimodal inputs is slower.

The good news is that you don’t need that much fidelity. Multimodal embedding models reduce each input to a compact representation before generating a vector anyway, so a smaller, lower-fidelity version of a multimodal input (such as a downscaled image or a lower-bitrate audio clip) produces a very similar embedding, and similar search quality, to its full-size original. This also applies to PDF input. PDFs are generally processed visually by multimodal models, so the quality only needs to be good enough to perform operations like image embedding and OCR. Long PDFs should be broken up into chunks of smaller inputs, so the embeddings generated more accurately represent each chunk. Feeding the model small inputs keeps your documents lean, trims index and response sizes, and speeds up ingestion, all without meaningfully affecting relevance. 

Elasticsearch reinforces this practice with a guardrail: the indices.inference.max_binary_input_size cluster setting caps the size of each binary input, defaulting to 1 MB. Any individual value that exceeds the limit is rejected with a clear error, so oversized inputs surface as an actionable problem at index time rather than as silent bloat. This setting is adjustable in self-hosted and ECH through the cluster settings API. It is not adjustable in our serverless offering, where 1 MB is the hard limit for binary sizes.

When possible, it is also advised to use source filtering to exclude semantic fields from responses. For example:

This makes responses smaller, more performant, and easier to parse because multimodal data is not returned with each.

Try out the semantic field

The semantic field is available in Elasticsearch 9.5 and Serverless. Start a free trial and try it out today.

Elastic Connectors are a type of Elastic integrations that sync data from an original data source to an Elasticsearch index. Connectors enable you to create searchable, read-only replicas of your data sources.

There are a number of connectors that are supported for variety of 3rd-parties, such as:

• MongoDB

• Various SQL DBMS such as MySQL, PostgreSQL, MSSQL and OracleDB

• Sharepoint Online

• Amazon S3

• And many more. The full list is available here.

Connectors content synchronization jobs

Connectors support two types of content synchronization jobs: full syncs and incremental syncs.

1. Full syncs

Full sync is a sync that extracts all desired documents from a 3rd-party service and ingests them into Elasticsearch. So if you've set up your Network Drive connector to ingest all documents from a folder "\Documents/Reports\2022**.docx", during a full sync the connector will fetch all the documents that match this criteria and send all of them to Elasticsearch. Simplified pseudocode for this would look like:

This works well until the sync starts to take too long. This could happen because the connector fetches more data than needed. For instance, why fetch old files that have not changed and send them to Elasticsearch? One could argue that the metadata for files could be unreliable, so all files need to be fetched again and sent to Elasticsearch. Indeed, that could be the case, but if we can trust the metadata of the data fetched from 3rd-party, we can ingest less data. Incremental sync is the way to do so.

2. Incremental syncs

Most of the time, if written well, connector spends doing IO operations. Returning to the example code there are 3 places where IO happens:

Each of these places can become a bottleneck and take a significant amount of time during the sync.

Here's where incremental sync comes into play. Its purpose is to decrease the amount of IO on any of the stages, if possible.

Potential optimizations for incremental sync

Fetch fewer documents from 3rd-party systems

Modifying the example above, the code could look like this:

In cases where only a small number of documents change in our 3rd-party system, we can speed up the ingestion process significantly. However, for Network Drive it's not possible - its API does not support filtering documents by metadata. We won't be able to avoid scanning through the full content of Network Drive.

Skip download of content of files that haven't changed since previous sync

Downloading file content takes a significant amount of time in the syncs. If files are reasonably large, the connection is unstable or throughput is low, downloading the content of files would take most of the time when syncing the content from the 3rd-party. If we skip downloading some of them, it can already significantly speed up the connector.

Consider the following example pseudocode:

If no documents were updated, the sync will actually be magnitudes faster than fully syncing the content.

Skip ingestion of non-modified documents into Elasticsearch

While it may seem minor, ingestion of data into Elasticsearch takes a significant amount of time - albeit normally less than downloading the content from the 3rd-party system. We can start storing timestamps per each document and not send the documents into Elasticsearch if their timestamp did not change.

We can combine this approach with the previous approach to save the most time possible during the sync.

This approach helps save even more time when running a sync. Now let's take a look into performance considerations for such improvements.

Measuring incremental sync performance

Now since we've taken a look into simplified code that shows how incremental syncs can work, we can try to estimate potential performance improvements.

For some connectors, incremental sync is implemented in a certain manner that optimizes the way data is fetched from a 3rd-party. For example, the Sharepoint Online connector fetches some data via delta API - only collecting documents that changed after the last sync. This improves performance in an obvious manner - less data -> less time to sync the data to latest.

For other connectors (currently all connectors except Sharepoint Online connector) incremental sync is done by framework in a generic way which was described in one of previous sections "Skip ingestion of non-modified documents into Elasticsearch".

Connectors still collect all the data from 3rd-party data source (as it does not provide a way to fetch only the changed records). However if this data contains timestamps, the connector framework compares document IDs and timestamps of already ingested documents with incoming documents. If the document exists in Elasticsearch with the same timestamp that was received from the 3rd-party data source, then this document will not be sent to Elasticsearch.

We've described abstract approach for the performance improvements with incremental syncs, but we already have these implemented in connectors, so let's dive into real numbers!

Performance tests

We will estimate the rough magnitude of improvement for incremental syncs with these performance tests, not aiming at high precision.

The two connectors chosen for this test, Google Drive and Github, were chosen because they have different IO profiles.

Google Drive acts like a file storage. It:

• Has a fast API that does not throttle too soon

• Normally stores a lot of binary content of variable size - from small to really large

• Normally stores a small number of records - tens or hundreds of thousands rather than millions

GitHub data is ingested via a more of a classic API, that:

• Throttles quite often

• Contains many records that are much smaller than those in Google Drive

• Does not send binary content at all

Due to these differences, the incremental sync performance will majorly differ.

Both tests will contain these mandatory steps:

1. Do a full sync against a 3rd-party system

2. Modify some documents on the 3rd-party system

3. Run an incremental sync and check the amount of time it takes

This setup is very bare bones but will already give a good indication of the magnitude of performance improvement. Both tests will be slightly different and I will provide results with commentary in the next section.

Setup #1 - Google Drive Connector

Initial setup will be:

• 1 folder is on Google Drive with 1553 files (100 of them are 2MB in size, 1443 are 5KB in size)

• A full sync is executed and this data gets into Elasticsearch

• More files are added into Google Drive to make it 10144 files (100 of them are 2MB in size, all the rest are 5KB in size)

• Incremental sync is executed again to pull the new data

• Then some minor changes are made to files on Google Drive (1 added, 2 deleted)

• Incremental sync is executed again

• Full sync is executed to compare the run time against incremental sync again

The following table contains the results of the described test with commentary:

Sync Description

Run time

Documents Added

Documents Deleted

Comment

Initial Full Sync

0h 4m 0s

1553

0

This is initial sync - it pulls all documents

Incremental Sync after more data was added to Google Drive

0h 20m 9s

7939

0

Run time was high as expected - a lot of documents went in

Incremental Sync after some data was slightly changed in Google Drive

0h 1m 25s

1

2

Run was very fast. It still called Google Drive API a lot, but did not have to ingest 200+MB of data into Elasticsearch

Full Sync to compare performance

0h 23m 23s

10144

0

As expected, it takes a lot of time - all the data is downloaded from Google Drive and is sent to Elasticsearch, even if it did not change. We can assume that it takes 22 minutes to download and then upload the data into Elasticsearch for the setup

In summary, incremental sync significantly improved the performance of the connector because most of the time is spent on the connector downloading the content of the files and sending this content to Elasticsearch. Full sync brings 2 * 100 + 1443 * 5 / 1024 = 207MB of content - both downloaded by connector and ingested into Elasticsearch. If only 1 large file is changed, this amount changes to only 2MB - a magnitude of 100 change. This explains the performance improvement well.

Setup #2: GitHub connector

The GitHub connector is very different since the actual volume of data it syncs is relatively small - issues, pull requests and such are reasonably small, while there are lots of them. Additionally, GitHub has strict throttling policies and throttles connector a lot.

To give a good real-world example we’ll use the Kibana Github repository with the GitHub connector and observe its performance.

Sync Description

Run time

Documents Added

Documents Deleted

Comment

Initial Full Sync

8h 40m 1s

147421

0

---

Incremental Sync ran immediately after

9h 6m 7s

59

0

This sync took even more time to run, mostly because it was constantly throttled. Connector had to fetch all the data from GitHub but sent only 59 records with a total volume of less than 1MB

Next incremental sync

9h 2m 52s

191

1

This sync was triggered immediately after previous incremental sync finished. Run time is the same due to data being almost the same and throttling being a major factor in the connector run time

Key takeaways

• As you can see, there is no performance improvement for incremental sync for the Github connector - there is barely any space for optimization as most of the time is spent by the connector querying the system and waiting for the throttling to stop.

• Documents that are extracted are reasonably small, so network throughput usage is minimal. To improve the connector run time, the incremental sync would actually have to limit the number of queries to Github, but at this point it's not implemented in the connector.

Summary

What is the primary factor that impacts the performance of incremental sync? In simplified terms, it's the raw volume of data that is ingested.

For Sharepoint Online connector there is a special logic to fetch less data via the delta API. This saves good amount of time because the delta API allows connectors to not fetch files that were not changed. Files tend to be large, thus not downloading and ingesting them will save a lot of time.

For other connectors, incremental sync is generic - it just checks document timestamps before ingesting them to Elasticsearch - if this document is already in the index and the timestamp did not change, then it is not ingested. It saves less time than the previous approach that Sharepoint Online employs but works generically for all connectors. Some connectors - ones that contain large documents - benefit from this logic a lot, while others - that get throttled by a 3rd-party system and contain relatively small documents - get no benefit from incremental syncs.

Additionally, if Elasticsearch is under heavy load, incremental sync is less likely to be throttled by Elasticsearch, thus making it more performant under load.

Let's look at the following graph:

In the graph you can see how much time each part of content extraction and ingestion takes on the timeline. In the example above the connector is spending the most time on ingesting the data, even pausing for extraction and content download. In this case incremental sync has a potential of improving the run time of the sync by 30-40%.

Let's look at another example - a system that has throttling and low throughput, but stores very little data in Elasticsearch (Sharepoint Online, GitHub, Jira, Confluence):

This system will not benefit from generic incremental syncs a lot - most of the time is spent extracting content from the 3rd-party system.

And the last example - fast and accessible system that stores huge amounts of data in Elasticsearch (Google Drive, Box, OneDrive, Network Drive):

If there aren't too many items that change in such a system between syncs, this system will benefit a lot from generic incremental syncs.

Currently connectors that potentially get the most of incremental sync are:

• Azure Blob Storage

• Box

• Dropbox

• Google Cloud Storage

• Google Drive

• Network Drive

• OneDrive

• S3

• Sharepoint Online

Other connectors will benefit less from incremental syncs, or will not benefit at all, but there's no one-size-fits-all answer here. Performance heavily depends on the profile of data ingested. The bigger each individual document is, the bigger the benefit.

Elasticsearch offers two types of index templates: legacy and composable. Composable templates introduced in Elasticsearch 7.8 that are set to replace legacy templates, both can still be used in Elasticsearch 8.

This article explores the differences between these templates and how they interact. In particular, we will focus on how you can detect which template will be used when you are creating an index. Let's get started by looking at how to create the different types of index templates.

Index templates in Elasticsearch

Legacy templates can be created using the following API:

Composable templates can be created using this API:

Component templates are a third type, which are typically used for managing multiple templates with similar structures. For example, if you need to create hundreds of templates with similar structures, you can create a component template with the common settings, mappings, and aliases, and then include it in your index templates. Component templates can be created using this API:

Important!

Determining which template an index will use when it is created

To determine which template an index will use upon creation, you can use the _simulate_index API. This API will return the template that will be used, along with any overlapping templates. However, if no composable templates are present, the API will return an empty body. In that case, you can create a dummy index and check the logs of the elected master node to determine which template will be used.

What happens if you have both legacy templates and composable templates?

As noted above, if you have both legacy and composable templates, the legacy template will be ignored as if it did not exist.

In such a case, you would get a warning message like the following when you run the command:

If a newly created composable template matches an existing legacy template with the same or includes an index pattern you will get a warning message like the following:

Use this command if you want to test it:

Notes from real life scenario

Conflicts can be annoying, and they can crash the application. Imagine that you have logstash-dev-*, logstash-prd-*, logstash-stg-* legacy templates that all working fine. If someone adds a single composable template that include index pattern like a logstash-* all legacy templates will be ignored, the fields types can be change and finally it can break the application. Because of that, it’s recommended to switch from legacy to composable templates if you are using Elasticsearch 7 and onwards.

Another good point to keep in mind is that if you run the Logstash in Elasticsearch 8 or higher, Logstash will add it's template as composable template by default. Because manage_template is set to true by default and Logstash template_api is set tocomposable for Elasticsearch 8 and onwards. It will create a Logstash composable template with logstash-* index pattern if the composable template does not exist. Yes, it will ignore all legacy templates covering logstash-* and overlap them.

Template Overlapping

1. What happens if you have two composable templates that point to the same index pattern?

As previously mentioned, if you have two composable templates that point to the same index pattern, the composable template with the highest priority will take precedence.

In this example, you have two templates—ct1 and ct2—both targeting the same index pattern test_index-. However, ct2 has a higher priority (1) than ct1 (0). Therefore, when you create an index that matches the pattern test_index-, the settings and mappings defined in ct2 will be applied before ct1. If there are the same settings in the ct1 and ct2 templates, the ct2 template will overwrite.

2. What happens if you have two legacy templates that point to the same index pattern?

As highlighted above, if you have multiple templates that point to the same index pattern, the templates with lower-order values are merged first. Templates with higher-order values are merged later, overriding templates with lower values.

If two legacy templates have the same order value, they will be sorted by name. For example, in a case with [t2, t1], t1 would be merged first, t2 would be merged later, and t2 would override t1 if there are any same mapping/settings/aliases.

Unfortunately, if you don't have composable templates, this API call responds with an empty body. So how you can check that?

The answer is to create a dummy index and check the Elasticsearch elected-master logs.

From the logs, we can see that "[test_index-1] creating index, cause [api], templates [t2, t1]".

As you can see, both legacy templates t1 and t2 have the same order; so, which one will override the other?

In this case, Elasticsearch will sort the legacy index templates according to their names and apply them. Both templates will be applied, and the first one in the list, which is t2 in this example, will override the template.

Bonus: What happens if you have two legacy templates that point to the same index pattern with same field name but inappropriate type?

Attempting to merge attributes within the legacy template, regardless of the order, is likely to fail since field definitions should remain atomic. This issue is a primary motivator for introducing the new composable templates. See the below example. We thank Philipp Krenn for adding these comments to the article.

Notes and good things to know

1. Using legacy templates in the same order can cause a lot of confusion. That’s why it's recommended to add order to the template.

2. Templates with lower-order values are merged first. Templates with higher order values are merged later, overriding templates with lower values.

3. You can't create two composable templates with the same priority.

Conclusion

In conclusion, understanding how Elasticsearch's index templates work is crucial for effective index management. By knowing how to determine which template an index will use upon creation, you can ensure that your indices are created with the correct settings, mappings, and aliases.

Resources

https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-put-template.html https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-templates-v1.html https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-simulate-index.html

TM (ESRE) enables you to add semantic search on proprietary data that can be integrated with generative AI technologies to build modern search experiences.

Snowflake is a fully managed SaaS (software as a service) that provides a single platform for data warehousing, data lakes, data engineering, data science, data application development, and secure sharing and consumption of real-time/shared data.

In this blog, we will see how to bring your snowflake data to Elasticsearch using below methods:

1. Using Logstash (periodic sync)

2. Using Snowflake Elasticsearch Python Script (one time sync)

Prerequisites

Snowflake credentials

You will have received all below credentials after signup, or you can get them from the Snowflake panel.

• Account username

• Account password

• Account Identifier

Elastic credentials

1. Visit https://cloud.elastic.co and sign up.

2. Click on Create deployment. In the pop-up, you can change the settings or keep the default settings.

3. Download or copy the deployment credentials (both username and password).

4. Also copy the Cloud ID.

5. Once you’re ready for deployment, click on Continue (or click on Open Kibana). It will redirect you to the Kibana dashboard.

Methods to ingest data from Snowflake to Elasticsearch

Method 1: Using Logstash

Logstash is an open source ETL tool where you can provide multiple sources as an input, transform (modify) it, and push to your favorite stash. One of the famous use cases of Logstash is reading logs from the file and pushing to Elasticsearch. We can also modify the data on the fly using a filter plugin, and it will push updated data to the output.

We’re going to use the JDBC input plugin to pull the data from Snowflake and push to Elasticsearch using the Elasticsearch output plugin.

1. Install Logstash by referring to the documentation.

2. Go to the Maven Central Repository and download: https://repo1.maven.org/maven2/net/snowflake/snowflake-jdbc.

3. Click on the directory for the version that you need and download the snowflake-jdbc-#.#.#.jar file. In my case, I have downloaded snowflake-jdbc-3.9.2.jar. (Refer to official documentation to learn more about the Snowflake JDBC Driver.)

4. Create a pipeline by creating file sf-es.conf. Add the below snippet and replace all credentials.

jdbc_connection_string :

Schedule: Here you can schedule to run this flow periodically using cron syntax. On every run, your data will be moved incrementally. You can check more on scheduling.

Please change according to your requirements.

JDBC Paging (Optional): This will cause a sql statement to be broken up into multiple queries. Each query will use limits and offsets to collectively retrieve the full result-set. You can use this to move all data in a single run.

Enable JDBC paging by adding below configurations:

1. Run Logstash

Method 2: Using Snowflake-Elasticsearch Python script

If Logstash is not currently in place or has not been implemented, I have written a small Python utility, which is available here on GitHub, to pull data from Snowflake and push it to Elasticsearch. This will pull all your data at one time. So if you have a small amount of data to be migrated in a non-periodic manner, you can use this utility.

Note: This is not a part of the official Elastic connectors. Elastic connectors provide support for various data sources. You can use this connector if you have a requirement to sync data from any supported data sources.

1. Installation

1. Installing dependencies

1. Change configs

• Open config/connector.yml.

• Replace credentials with the following:

1. Run connector

Verify data

1. Log in to Kibana and go to ☰ > Management > Dev Tools.

2. Copy and paste the following API GET request into the Console pane, and then click the ▶ (play) button. This queries all records in the new index.

Conclusion

We have successfully migrated the data from Snowflake to Elastic Cloud. You can achieve the same on any Elasticsearch instance, whether it is in the cloud or on prem.

Start leveraging full text and semantic search capabilities on your data set. You can also connect your data with LLMs to build Question - Answer capabilities.