Loading

Manage your data quality with Streams

When documents fail during ingestion, you can lose data without knowing what went wrong. Streams preserves failed documents and gives you the tools to find, understand, and fix data quality issues without leaving the Streams UI.

The Streams Data quality tab provides a single place to monitor and resolve data quality issues:

  • Failed documents are preserved, not dropped: When a processing error occurs, data lands in a failure store instead of being lost, so nothing is silently discarded.
  • No separate infrastructure needed.
  • See exactly what is failing and why: The Data quality tab shows failure counts, error types, and sample messages, so you can identify problems immediately rather than searching for them.
  • Fix issues against the actual failing documents: Instead of re-ingesting data from the source, you iterate on the processor in the Processing tab using the exact documents that failed, with real-time validation before deploying the fix.

Use Streams to identify failing documents, trace the source of failure, and deploy a fix.

  1. Open the Data quality tab

    1. Open Streams from the navigation menu or use the global search field.
    2. Select your stream from the list.
    3. Go to the Data quality tab.

  2. Check the quality score

    A stream's data quality (Good, Degraded, or Poor) is displayed at the top of each stream's page so you can assess health at a glance.

    How is the quality score calculated?

    Streams uses the following ES|QL queries to calculate the quality score for a stream:

    • All documents (including failed documents): FROM <stream-name>, <stream-name>::failures | STATS doc_count = COUNT(*)
    • Failed documents only: FROM <stream-name>::failures | STATS failed_doc_count = COUNT(*)
    • Degraded documents: FROM <stream-name> METADATA _ignored | WHERE _ignored IS NOT NULL | STATS degraded_doc_count = COUNT(*)

    Streams calculates data quality as follows:

    • Good: Both the Degraded documents percentage and the Failed documents percentage are 0.
    • Degraded: Either the Degraded documents percentage or the Failed documents percentage is greater than 0 and less than or equal to 3.
    • Poor: Either the Degraded documents percentage or the Failed documents percentage is greater than 3.

  3. The Data quality tab shows a breakdown of what's wrong and when it started:

    • Failed documents: Documents rejected during ingestion because of mapping conflicts or pipeline failures. Failed documents land in the failure store rather than being dropped.
    • Degraded documents: Documents that were ingested but with fields silently ignored because of mapping issues or values that exceeded configured limits.
    • Quality score: An overall health rating based on the percentage of failed and degraded documents.
    • Trends over time: A time-series chart showing when the problem started and whether it's getting worse.

    Use the chart to understand the scope of the issue before drilling into individual documents.

    To get notified when degraded or failed document percentages exceed a threshold, create a data quality alert.

  4. Fix and validate

    Navigate to the Processing tab to edit the failing processor. The Data preview pane lets you test your changes against the actual documents that failed—so you can confirm the fix works against real data before deploying it.

    When the preview looks right, select Save changes. Streams applies the updated pipeline to all future incoming documents.

    Refer to Process your documents for detailed guidance on editing processors and using the data preview.

A failure store is a secondary set of indices inside a data stream, dedicated to storing failed documents. Instead of losing documents that are rejected during ingestion, a failure store retains them in a ::failures index, so you can review failed documents to understand what went wrong and how to fix it.

For example, for a stream called my-stream, the failure store would be accessible by querying my-stream::failures.

For more information on data quality, refer to the data set quality documentation.

To view and modify failure store in Elastic Stack, you need the following data stream level privileges:

  • read_failure_store
  • manage_failure_store

For more information, refer to Granting privileges for data streams and aliases.

When the failure store is off, the Failed documents component shows an Enable failure store link. Select the link and set a retention period for your failure store.

To get notified when the percentage of degraded documents in a stream exceeds a threshold, create an alert rule from the Data quality tab.

  1. Open the Data quality tab for the stream you want to monitor.
  2. Select Create rule .
  3. Define the conditions for your rule.