Why Streams became a telemetry pipeline in front of Elasticsearch

Streams, which we released in 2025, started as a way to partition and process data once it had reached Elasticsearch. It did that job well, but its reach ended at the cluster boundary; data entered Elasticsearch first, and Streams processed it there. It landed as an Elasticsearch data stream. Meanwhile, each shipper’s parsers and field mappings stayed handwritten, as did its routing rules. There was no single place to see or change them, and they broke quietly whenever a format drifted.

So we re-built Streams to now become a self managed data pipeline capability with a new UI, and routing to any endpoint, such as Amazon S3. We also moved Streams in front of the cluster which allowed it to focus on managing the data pipeline. Processing now happens before data lands anywhere, which changes what filtering and routing mean. Anything that’s processed can be sent to S3, Elasticsearch, or both, so you have full control over where you store your telemetry. If you already use Streams with wired or classic streams, those keep working; the new model is an evolution of the same product. With the new Streams, you can: 

• Store on your own terms.

• See and manage the ingest and processing in our new UI.

• Use it without configuring anything, since it’s fully managed.

In the rest of this post, we walk through what it adds.

Open by design: Store on your own terms in Amazon S3 or Elasticsearch

Some data has to live in storage you control, whether that’s your own S3 bucket or a location that your compliance team has approved. Streams routes it there directly. A copy route sends the complete record to your bucket as newline-delimited JSON, so you keep everything. The route to Elasticsearch carries only the subset you query, and a filter on that route drops the debug noise before it reaches an index. You pay Elasticsearch storage and retention costs only for what you choose to index, and everything that you don’t index stays on infrastructure you control.

The data in your S3 bucket is meant to stay useful, too. The objects that Streams writes to S3 are designed to be queryable in place through Elasticsearch Query Language (ES|QL) Data Federation, which reads NDJSON and Parquet files straight from a bucket without ingesting them. Data Federation is now in private preview on Elastic Cloud Serverless. An investigation that needs a record you didn’t index could then reach for the bucket with the same query language that you use for Elasticsearch.

Ingest you can see and manage in Streams Map

Most pipelines live in configuration files that are hard to manage at scale and terrible to debug. We built a new UI for this, Streams Map, which puts every source, pipeline, router, and destination on one screen. When you open a pipeline to edit its processors against a sample of your real data, you see the extracted fields before you save. Open a router to change the conditions that decide where data goes.

To version control your pipelines, export the whole configuration as a YAML file, and review it in a pull request. Then, import it into another environment. The map and the file describe the same streams.

Zero ops; fully managed

Streams is a fully managed Elastic Cloud service. There’s no pipeline infrastructure for you to install or run on-premises or in your own cloud account. For now, the private preview runs on Elastic Cloud Serverless only. Other Elastic deployment options will follow.

If you run your own OTel Collectors today, four jobs are yours:

• Size them for peak volume.

• Keep them patched.

• Upgrade them without dropping data.

• Work out which instance has the broken config when something stops arriving.

With Streams, Elastic handles all four.

The processing runs in OTel Collectors that Elastic operates as part of Elastic Cloud. They sit outside Elasticsearch, so parsing and enrichment are finished before data reaches your project, as is routing, and none of that work is done on Elasticsearch compute. Indexing and search get the resources that would otherwise be spent preparing the data, while the collectors scale with your ingest volume.

What you do see is Streams Map. You edit pipelines and routers there, or import them from the YAML that you keep in version control, and the collectors pick up the change. There’s nothing for you to restart or redeploy. Your shippers point at the same Streams source throughout.

How a telemetry pipeline works in Elastic Streams

Under the hood, data comes in from a source, is shaped by a pipeline and branched by a router, and then lands in a destination. A stream is any path from a source to a destination, and the same model applies to logs, metrics, traces, and every other data type.

Component

What it is

Where it runs

Source

The endpoint your shippers send to; accepts OTLP, Prometheus Remote Write, and Elasticsearch bulk requests

Elastic Cloud Serverless

Pipeline

A sequence of processors that filter, parse, and enrich data

Elastic-managed OTel Collectors

Router

The conditions that decide which destination data goes to

Elastic-managed OTel Collectors

Destination

Where data lands: Amazon S3, Elasticsearch, or both

Your choice

Two of those pipelines overlap with things you already run, so here is how they differ:.

Stream pipeline

Elasticsearch ingest pipeline

Runs in

Elastic-managed OTel Collectors, outside the cluster

The Elasticsearch cluster

Runs when

Before data reaches any destination

At index time, on data already headed for an index

A Streams source is the endpoint that your shippers send to, which means the shippers themselves stay where they are. Streams accepts OTLP, Prometheus Remote Write, and Elasticsearch bulk requests. So Logstash, Elastic Agent, Beats, and EDOT point at a Streams source and keep sending, with no proprietary format to migrate into. With Streams, the parsing and enrichment that you would otherwise write in each shipper, as well as the routing, you now write once. In the future, Fleet-managed Elastic Agent will be able to point at a Streams source.

What we want to learn in the Elastic Streams private preview

The private preview is designed to help us understand how real environments use Streams and what they expect from it in terms of reliability and throughput, as well as latency. The answers will come from people who run Streams on their own data and tell us where it bends, which is why we’re looking for active design partners.

Design partners get early access to Streams, free of charge during the private preview, and a direct feedback loop with the product and engineering team. In return, they commit to a short weekly call with us for as long as the preview runs.

The best fit is a team on Elastic Cloud Serverless that runs telemetry from several kinds of shippers and has data that needs to land in an S3 bucket it controls. 

What you need to provide:

• An S3 bucket you control and its AWS region.

• An access key pair with write access to that bucket.

Streams stores the keys as a sealed credential, so they never appear in the configuration that you export. 

Data handling during the preview is best effort, so treat Streams as a second path beside the one you rely on; don’t route anything through Streams that you cannot afford to lose or cannot replay. 

When the private preview ends, partners move into the public technical preview, and Streams becomes available more broadly, which we expect early next year.

Sign up for a private preview

We handpick customers to preview and partner with us in achieving a better design. Tell us about the shippers you run and the processing you do, along with where your data has to live. We’ll notify the partners we select by email. Sign up today.