Meet the new Elastic Streams: Managed data pipelines for telemetry
Elastic’s new Streams gets you investigating in minutes by ingesting any data - structured or not - filtering, parsing, and enriching data with the option to even route it to your own storage end point, such as Amazon S3.
Elasticsearch turns raw logs into structured, searchable data at ingest. Follow the collect and analyze logs tutorial to see it end-to-end. Start a free cloud trial or try Elastic on your local machine now.
Elastic Streams is a managed data pipeline for telemetry on Elastic Cloud Serverless. Send it any telemetry, and it parses and enriches logs, metrics, and traces before they reach a destination. It then decides what that destination is. Send the complete record to an Amazon S3 bucket you own, and index only the subset that you actually query, or do both. Nothing has to change upstream. A Streams source accepts OpenTelemetry Protocol (OTLP) and Prometheus Remote Write, as well as Elasticsearch bulk, so Logstash, Elastic Agent, Beats, and Elastic Distribution of OpenTelemetry (EDOT) keep sending exactly as they do today. Elastic operates the OpenTelemetry (OTel) Collectors that do the processing, so you have no pipeline hosts to size or broken config to hunt down at 2:00 a.m. Private preview opens today on Elastic Cloud Serverless. Sign up for the private preview.
Why Streams became a telemetry pipeline in front of Elasticsearch
Streams, which we released in 2025, started as a way to partition and process data once it had reached Elasticsearch. It did that job well, but its reach ended at the cluster boundary; data entered Elasticsearch first, and Streams processed it there. It landed as an Elasticsearch data stream. Meanwhile, each shipper’s parsers and field mappings stayed handwritten, as did its routing rules. There was no single place to see or change them, and they broke quietly whenever a format drifted.
So we re-built Streams to now become a self managed data pipeline capability with a new UI, and routing to any endpoint, such as Amazon S3. We also moved Streams in front of the cluster which allowed it to focus on managing the data pipeline. Processing now happens before data lands anywhere, which changes what filtering and routing mean. Anything that’s processed can be sent to S3, Elasticsearch, or both, so you have full control over where you store your telemetry. If you already use Streams with wired or classic streams, those keep working; the new model is an evolution of the same product. With the new Streams, you can:
Store on your own terms.
See and manage the ingest and processing in our new UI.
Use it without configuring anything, since it’s fully managed.
In the rest of this post, we walk through what it adds.
Streams moved from inside Elasticsearch to in front of it. Processing is now managed by Elastic, visible in the new Streams Map, and can route to Amazon S3 as well as Elasticsearch.
Open by design: Store on your own terms in Amazon S3 or Elasticsearch
Some data has to live in storage you control, whether that’s your own S3 bucket or a location that your compliance team has approved. Streams routes it there directly. A copy route sends the complete record to your bucket as newline-delimited JSON, so you keep everything. The route to Elasticsearch carries only the subset you query, and a filter on that route drops the debug noise before it reaches an index. You pay Elasticsearch storage and retention costs only for what you choose to index, and everything that you don’t index stays on infrastructure you control.
The data in your S3 bucket is meant to stay useful, too. The objects that Streams writes to S3 are designed to be queryable in place through Elasticsearch Query Language (ES|QL) Data Federation, which reads NDJSON and Parquet files straight from a bucket without ingesting them. Data Federation is now in private preview on Elastic Cloud Serverless. An investigation that needs a record you didn’t index could then reach for the bucket with the same query language that you use for Elasticsearch.
Streams Map with one source and two destinations. The complete record goes to S3; the processed subset to Elasticsearch.
Ingest you can see and manage in Streams Map
Most pipelines live in configuration files that are hard to manage at scale and terrible to debug. We built a new UI for this, Streams Map, which puts every source, pipeline, router, and destination on one screen. When you open a pipeline to edit its processors against a sample of your real data, you see the extracted fields before you save. Open a router to change the conditions that decide where data goes.
To version control your pipelines, export the whole configuration as a YAML file, and review it in a pull request. Then, import it into another environment. The map and the file describe the same streams.
Streams pipeline editor, with processors on the left and the preview of the processed data on the right.
Zero ops; fully managed
Streams is a fully managed Elastic Cloud service. There’s no pipeline infrastructure for you to install or run on-premises or in your own cloud account. For now, the private preview runs on Elastic Cloud Serverless only. Other Elastic deployment options will follow.
If you run your own OTel Collectors today, four jobs are yours:
Size them for peak volume.
Keep them patched.
Upgrade them without dropping data.
Work out which instance has the broken config when something stops arriving.
With Streams, Elastic handles all four.
The processing runs in OTel Collectors that Elastic operates as part of Elastic Cloud. They sit outside Elasticsearch, so parsing and enrichment are finished before data reaches your project, as is routing, and none of that work is done on Elasticsearch compute. Indexing and search get the resources that would otherwise be spent preparing the data, while the collectors scale with your ingest volume.
What you do see is Streams Map. You edit pipelines and routers there, or import them from the YAML that you keep in version control, and the collectors pick up the change. There’s nothing for you to restart or redeploy. Your shippers point at the same Streams source throughout.
How a telemetry pipeline works in Elastic Streams
Under the hood, data comes in from a source, is shaped by a pipeline and branched by a router, and then lands in a destination. A stream is any path from a source to a destination, and the same model applies to logs, metrics, traces, and every other data type.
Component | What it is | Where it runs |
|---|---|---|
Source | The endpoint your shippers send to; accepts OTLP, Prometheus Remote Write, and Elasticsearch bulk requests | Elastic Cloud Serverless |
Pipeline | A sequence of processors that filter, parse, and enrich data | Elastic-managed OTel Collectors |
Router | The conditions that decide which destination data goes to | Elastic-managed OTel Collectors |
Destination | Where data lands: Amazon S3, Elasticsearch, or both | Your choice |
Two of those pipelines overlap with things you already run, so here is how they differ:.
Stream pipeline | Elasticsearch ingest pipeline | |
|---|---|---|
Runs in | Elastic-managed OTel Collectors, outside the cluster | The Elasticsearch cluster |
Runs when | Before data reaches any destination | At index time, on data already headed for an index |
A Streams source is the endpoint that your shippers send to, which means the shippers themselves stay where they are. Streams accepts OTLP, Prometheus Remote Write, and Elasticsearch bulk requests. So Logstash, Elastic Agent, Beats, and EDOT point at a Streams source and keep sending, with no proprietary format to migrate into. With Streams, the parsing and enrichment that you would otherwise write in each shipper, as well as the routing, you now write once. In the future, Fleet-managed Elastic Agent will be able to point at a Streams source.
One model for every entry point.
What we want to learn in the Elastic Streams private preview
The private preview is designed to help us understand how real environments use Streams and what they expect from it in terms of reliability and throughput, as well as latency. The answers will come from people who run Streams on their own data and tell us where it bends, which is why we’re looking for active design partners.
Design partners get early access to Streams, free of charge during the private preview, and a direct feedback loop with the product and engineering team. In return, they commit to a short weekly call with us for as long as the preview runs.
The best fit is a team on Elastic Cloud Serverless that runs telemetry from several kinds of shippers and has data that needs to land in an S3 bucket it controls.
What you need to provide:
An S3 bucket you control and its AWS region.
An access key pair with write access to that bucket.
Streams stores the keys as a sealed credential, so they never appear in the configuration that you export.
Data handling during the preview is best effort, so treat Streams as a second path beside the one you rely on; don’t route anything through Streams that you cannot afford to lose or cannot replay.
When the private preview ends, partners move into the public technical preview, and Streams becomes available more broadly, which we expect early next year.
Sign up for a private preview
We handpick customers to preview and partner with us in achieving a better design. Tell us about the shippers you run and the processing you do, along with where your data has to live. We’ll notify the partners we select by email. Sign up today.
How helpful was this content?
Related Content
Two rules, one outage, one alert: How change point detection cuts alert noise
Your logs already wrote the alert: Generating ES|QL detection queries in Elasticsearch
How Elastic Significant Events turns 100 log samples into detection rules
The Great Cat Escape: A hands-on guide to OpenTelemetry, Elastic APM, and Kibana alerting
