Get data into Streams

This page covers the different ways to send data to Streams:

• Ingest new data: Use wired streams to send logs to a managed endpoint for new ingestion. Data lands in a managed hierarchy with inheritance, partitioning, and cascading configuration. Best for new deployments, custom logs, and mixed-format sources.
• Work with existing data: Use classic streams to work with data already flowing into Elasticsearch. No migration or configuration changes required.

Streams requires the following permissions:

Wired streams send your documents to a managed endpoint, from which you can route data into child streams based on partitioning rules. Child streams automatically inherit mappings, lifecycle settings, and processors from the parent, and configuration changes propagate through the hierarchy.

Two endpoints are available. Use logs.otel (recommended) when sending OTel-native data or when you want a consistent, normalized format. Streams translates ECS field names to OTel equivalents automatically. Use logs.ecs when your data already uses ECS field names and you want to preserve them without transformation.

To send data to a wired stream, configure your shipper to point to the appropriate endpoint:

processors:
  transform/logs-streams:
    log_statements:
      - context: resource
        statements:
          - set(attributes["elasticsearch.index"], "logs.otel")
service:
  pipelines:
    logs:
      receivers: [myreceiver]
      processors: [transform/logs-streams]
      exporters: [elasticsearch, otlp]
		

filebeat.inputs:
  - type: filestream
    id: my-filestream-id
    index: logs.otel
    enabled: true
    paths:
      - /var/log/*.log

# No need to install templates for wired streams
setup:
  template:
    enabled: false

output.elasticsearch:
  hosts: ["<elasticsearch-host>"]
  api_key: "<your-api-key>"
		

output {
  elasticsearch {
    hosts => ["<elasticsearch-host>"]
    api_key => "<your-api-key>"
    index => "logs.otel"
    action => "create"
  }
}
		

POST /logs.otel/_bulk
{ "create": {} }
{ "@timestamp": "2025-05-05T12:12:12", "body": { "text": "Hello world!" }, "resource": { "attributes": { "host.name": "my-host-name" } } }
{ "create": {} }
{ "@timestamp": "2025-05-05T12:12:12", "message": "Hello world!", "host.name": "my-host-name" }
		

After configuring your data source, confirm data is appearing in Discover.

For wired streams, you first need to make the index pattern available:

1. Manually create a data view for the wired streams index pattern (logs,logs.*).
2. Add the wired streams index pattern (logs,logs.*) to the observability:logSources Kibana advanced setting, which you can open from the navigation menu or by using the global search field.

Once data appears in Discover, you're ready to start organizing, parsing, and configuring retention for your streams.

Wired streams can contain fields stored in _source that are not explicitly mapped. By default, ES|QL returns an error when a query references an unmapped field. To make unmapped fields queryable, add SET unmapped_fields = "LOAD"; at the start of your ES|QL query:

SET unmapped_fields = "LOAD";
FROM logs.otel
| WHERE my_custom_field == "value"
		

When LOAD is set, unmapped fields are loaded from _source as keyword fields, or treated as null if absent from _source.

When you query a wired stream and the ES|QL editor detects an unknown column error, a Load unmapped fields quick fix action is available. Select it to apply this setting automatically.

For a conceptual overview and use cases, refer to Unmapped fields. For Kibana editor behavior, refer to Handle unmapped fields with SET unmapped_fields.

Classic streams let you use the Streams UI to extract fields and configure data retention for data that's already being ingested into Elasticsearch without additional configuration.

Classic streams:

• Are based on existing data streams, index templates, and component templates.
• Can follow the data retention policy set in the existing index template.
• Do not support hierarchical inheritance or cascading configuration updates.

Open classic streams from the following places in Kibana:

• Select Streams from the navigation menu or use the global search field, and find the data stream in the Streams table.

• Open the data stream for a specific document from Discover. To do this:

  1. From the Documents table, select the icon for a document to expand the details flyout.
  2. Under Stream, select the link to the stream (for example, logs-generic.default). The Streams UI opens filtered to the stream that contains the document.

You can also access Streams features using the Streams API. Refer to the Streams API documentation for more information.

Once your data is flowing into Streams, you can start organizing and enriching it:

• Organize your data: Use partitioning to route data subsets into dedicated child streams with independent retention and processing rules. Partitioning is only available for wired streams.
• Parse and process: Build a processing pipeline to extract structured fields from raw log messages using AI-generated or manually configured processors.