Streams, a new AI capability within Elastic Observability. Built on the Elasticsearch platform, it's designed for Site Reliability Engineers (SREs) to use logs as the primary signal for investigations, enabling faster answers and quicker issue resolution. For decades, logs have been considered too noisy, expensive, and complex to manage, and many observability vendors have treated them as a second-class citizen. Streams flips this script by transforming raw logs into your most valuable asset to immediately identify not only the root cause, but also the why behind the root cause to enable instant resolution.
Learn more from our previous article Introducing Streams
Many SREs deploy on cloud native archtiectures. Kubernetes is essentially the baseline deployment architecture of choice. Yet Kubernetes logs are messy by default. A single (data)stream often mixes access logs, JSON payloads, health checks, and internal service chatter.
Elastic Streams gives you a faster path. You can isolate subsets of logs with conditionals, use AI to generate Grok patterns from real samples, and drop documents you do not need before they add storage and query cost.
Why Kubernetes logs get messy fast
The default Kubernetes container logs stream can contain data from many services at once. In one sample, you might see:
- HTTP access logs from application pods
- Verbose worker or batch job status logs
- Platform and container lifecycle events with different formats
This is why "one global parsing rule" will fail. You need targeted processing logic per log shape or type of application. Histrocially doing this kind of custom processing has been error prone and time consuming.
What Streams Processing changes
Streams Processing (available in 9.2 and later) moves this workflow into a live, interactive experience:
- You build conditions and processors in the UI
- You validate each change against sample documents before saving
- You can use AI to generate extraction patterns from selected logs
The result is a safer way to iterate on parsing logic without guessing.
Walkthrough: parse custom application logs
We'll start from your Kubernetes stream (logs-kubernetes.containers_logs-default) and create a conditional block that scopes processing to one service.
Once the condition is saved, it will automatically filter the sample data to a subset of logs that match the condition. This is indicate by the blue highlight in the preview.
Inside that block, we'll add a Grok processor and click Generate pattern.
This agentic process will now use an LLM to generate a Grok pattern that will be used to parse the logs. By default this would be using the Elastic Inference Service, but you can configure it to use your own LLM. Review the generated pattern and accept it once the sample set validates.
Walkthrough: drop noisy postgres-loadgen documents
Not all logs are that important that we'd like to keep them around forever. For example, logs from a load testing tool like a load generator are not useful for long-term analysis, so let's drop those.
To do this we will add a second conditional block for logs you intentionally do not want to index long-term.
Add a drop processor inside this block, then validate in the Dropped tab.
Save safely with live simulation
One of the most useful parts of Streams is the preview-first workflow. You can inspect matched, parsed, skipped, failed, and dropped samples before making the change live.
YAML mode and the equivalent API request
The interactive builder works well for most edits, but advanced users can switch to YAML mode for direct control.
You can also open Equivalent API Request to copy the payload for automation and Infrastructure as Code workflows.
A note on backwards compatibility
Streams Processing builds on Elasticsearch ingest pipelines, so it works with the same ingestion model teams already use.
When you save processing changes, Streams appends logic through the stream processing pipeline model (for example via @custom conventions used by data streams). That means you can adopt conditionals, parsing, and selective dropping incrementally, without changing your Kubernetes log shippers.
What's next?
Streams Processing is consistently getting new processing capabilities. Check out the Streams documentation for the latest updates.
Over the coming months more of this will be automated and moved to the background, reducing the manual effort required to process logs.
Another miletsone we're working towards is to offer this processing at read time, rather than write time. Using ES|QL this will enable you to iterate on your parsing logic without having to worry about committing changes that are harder to revert.
Also try this out by getting a free trial on Elastic Serverless.
Happy log analytics!!!