Ingest pipelines preprocess and enrich APM documents before indexing them. For example, a pipeline might define one processor that removes a field, one that transforms a field, and another that renames a field.
The default APM pipelines are defined in index templates that Fleet loads into Elasticsearch. Elasticsearch then uses the index pattern in these index templates to match pipelines to APM data streams.
Custom ingest pipelinesedit
The Elastic APM integration supports custom ingest pipelines. A custom pipeline allows you to transform data to better match your specific use case. This can be useful, for example, to ensure data security by removing or obfuscating sensitive information.
Each data stream ships with a default pipeline.
This default pipeline calls an initially non-existent and non-versioned "
@custom" ingest pipeline.
If left uncreated, this pipeline has no effect on your data. However, if utilized,
this pipeline can be used for custom data processing, adding fields, sanitizing data, and more.
In addition, ingest pipelines can also be used to direct application metrics (
metrics-apm.app.*) to a data stream with a different dataset, e.g. to combine metrics for two applications.
Sending other APM data to alternate data streams, like traces (
traces-apm.*), logs (
logs-apm.*), and internal metrics (
metrics-apm.internal*) is not currently supported.
@custom ingest pipeline naming conventionedit
@custom pipelines are specific to each data stream and follow a similar naming convention:
As a reminder, the default APM data streams are:
RUM and iOS agent application traces:
APM internal metrics:
APM transaction metrics:
APM service destination metrics:
APM service transaction metrics:
APM service summary metrics:
APM error/exception logging:
APM app logging:
To match a custom ingest pipeline with a data stream, follow the
@custom in the table above.
For example, to target application traces, you’d create a pipeline named
@custom pipeline can directly contain processors or you can use the
pipeline processor to call other pipelines that can be shared across multiple data streams or integrations.
@custom pipeline will persist across all version upgrades.
@custom ingest pipelineedit
The process for creating a custom ingest pipeline is as follows:
- Create a pipeline with processors specific to your use case
Add the newly created pipeline to an
@custompipeline that matches an APM data stream
- Roll over your data stream
If you prefer more guidance, see one of these tutorials:
Create an ingest pipeline filter — An APM-specific tutorial where you learn how to obfuscate passwords stored in the
- Transform data with custom ingest pipelines — A basic Elastic integration tutorial where you learn how to add a custom field to incoming data.
Intro to Kibana
ELK for Logs & Metrics