Known issuesedit
APM has the following known issues:
traces-apm@custom ingest pipeline applied to certain data streams unintentionally
APM Server versions: 8.12.0
If you’re using the Elastic APM Server v8.12.0,
the traces-apm@custom ingest pipeline is now additionally applied to data streams traces-apm.sampled-*
and traces-apm.rum-*, and applied twice for traces-apm-*. This bug impacts users with a non-empty traces-apm@custom ingest pipeline.
If you rely on this unintended behavior in 8.12.0, please rename your pipeline to traces-apm.integration@custom to preserve this behavior in later versions.
A fix was released in 8.12.1: elastic/kibana#175448.
Upgrading APM Server to 8.11+ might break event intake from older APM Java agents
APM Server versions: >=8.11.0
Elastic APM Java agent versions: < 1.43.0
If you are using APM Server (> v8.11.0) and the Elastic APM Java agent (< v1.43.0), the agent may be sending empty histogram metricsets.
In previous APM Server versions some data validation was not properly applied, leading the APM Server to accept empty histogram metricsets where it shouldn’t. This bug was fixed in the APM Server in 8.11.0.
The APM Java agent (< v1.43.0) was sending this kind of invalid data under certain circumstances. If you upgrade the APM Server to v8.11.0+ without upgrading the APM Java agent version, metricsets can be rejected by the APM Server and can result in additional error logs in the Java agent.
The fix is to upgrade the Elastic APM Java agent to a version >= 1.43.0. Find details in elastic/apm-data#157.
Ingesting new JVM metrics in 8.9 and 8.10 breaks upgrade to 8.11 and stops ingestion
APM Server versions: 8.11.0, 8.11.1
Elastic APM Java agent versions: 1.39.0+
If you’re using the Elastic APM Java agent v1.39.0+ to send new JVM metrics to APM Server v8.9.x and v8.10.x, upgrading to 8.11.0 or 8.11.1 will silently fail and stop ingesting APM metrics.
After upgrading, you will see the following errors:
-
APM Server error logs:
failed to index document in 'metrics-apm.internal-default' (fail_processor_exception): Document produced by APM Server v8.11.1, which is newer than the installed APM integration (v8.10.3-preview-1695284222). The APM integration must be upgraded.
-
Fleet error on integration package upgrade:
Failed installing package [apm] due to error: [ResponseError: mapper_parsing_exception Root causes: mapper_parsing_exception: Field [jvm.memory.non_heap.pool.committed] attempted to shadow a time_series_metric]
A fix was released in 8.11.2: elastic/kibana#171712.
APM integration package upgrade through Fleet causes excessive data stream rollovers
APM Server versions: <= 8.12.1 +
If you’re upgrading APM integration package to any versions <= 8.12.1, in some rare cases, the upgrade fails with a mapping conflict error. The upgrade process keeps rolling over the data stream in an unsuccessful attempt to work around the error. As a result, many empty backing indices for APM data streams are created.
During upgrade, you will see errors similar to the one below:
-
Fleet error on integration package upgrade:
Mappings update for metrics-apm.service_destination.10m-default failed due to ResponseError: illegal_argument_exception Root causes: illegal_argument_exception: Mapper for [metricset.interval] conflicts with existing mapper: Cannot update parameter [value] from [10m] to [null]
A fix was released in 8.12.2: elastic/apm-server#12219.