Fix an issue where the agent’s async task tracking could cause the user’s application to use too much memory. In cases where an application is under sustained load and is running in a memory constrained container, this looked like a memory leak.

This high memory usage could happen when application code starts async tasks (e.g. Promises, setTimeouts, async I/O) that outlive the APM Transaction (typically an HTTP request handler). The agent’s async task tracking keeps a reference to the APM Transaction (and any APM Spans) until the async task ends, thus extending the lifetime of those APM objects and the references they hold — in particular, HTTP request and response objects. This could lead to higher memory usage.

With this change, those references are removed when APM Transactions and Spans are ended, and agent memory usage is now the same as what it was before v3.24.0 when this issue was introduced. (#2528, #2489)

Fixes for ioredis instrumentation (#2460):

Fixes for redis instrumentation:

Change disableSend to no longer skip internal processing work. It now only disables communication with APM Server. Use contextPropagationOnly if your use case is to limit the APM agent’s processing to the minimum to support context propagation and log correlation.

This is listed under "Breaking changes" as a heads-up. The only possible negative result of this disableSend change is some extra CPU processing time by the agent. There is no outward functionality change.

Add contextPropagationOnly configuration option. This supports the use case of using the APM agent to propagate HTTP trace-context and to support log-correlation (adding trace.id et al fields to log records) without an APM server, and to otherwise reduce the processing time of the agent. (#2393)

This is similar to disableSend, but differs in that contextPropagationOnly: true tells the agent to skip unnecessary work.

Zero configuration support. The only required agent configuration option is serviceName. Normally the agent will attempt to infer serviceName for the "name" field in a package.json file. However, that could fail. With this version, the agent will cope with: a scoped package name (@scope/name is normalized to scope-name), a "name" that isn’t a valid serviceName, not being able to find a "package.json" file, etc. Ultimately it will fallback to "nodejs_service". (#1944)

One consequence of this change is that apm.getServiceName() will return undefined until the agent is started (check with apm.isStarted()).

A significant change was made to internal run context tracking (a.k.a. async context tracking). There are no configuration changes or API changes for custom instrumentation. (#2181)

One behavior change is that multiple spans created synchronously (in the same async task) will form parent/child relationships; before this change they would all be siblings. This fixes HTTP child spans of Elasticsearch and aws-sdk automatic spans to properly be children. (#1889)

Another behavior change is that a span B started after having ended span A in the same async task will no longer be a child of span A. (#1964)

This fixes an issue with context binding of EventEmitters, where removeListener would fail to actually remove if the same handler function was added to multiple events.

Guard against a negative value of metricsInterval, which can lead to high CPU usage as metrics are collected as fast as possible. Also ensure no metrics collection can happen if metricsInterval="0s" as intended. Before this change it was possible for some metric collection to still happen, even though none would be reported. (#2330)

This change also guards against negative and invalid values in the following configuration options: abortedErrorThreshold, apiRequestTime, and serverTimeout. If an invalid value is given, then will fallback to their default value.

Add the longFieldMaxLength integer configuration option (default 10000). Specific transaction/span/error fields (see the list below) will be truncated at this number of unicode characters. (#2193, #1921)

The errorMessageMaxLength configuration option is now deprecated, but still supported. Users should switch to using longFieldMaxLength. If errorMessageMaxLength is not specified, truncation of error messages will now use the longFieldMaxLength value.

Note that ultimately the maximum length of any tracing field is limited by the max_event_size configured for the receiving APM server.

The fields affected by longFieldMaxLength are:

Performance improvements have been made in error and stacktrace capture (#2094). This also included in two bug fixes:

The handling of sending events (transactions, spans, errors) to APM server has improved in a few ways. During temporary spikes in load and/or an APM server that is unresponsive, the agent will buffer a number of events and drop them above a certain limit (configurable via maxQueueSize). This helps ensure the agent does not overly consume memory and CPU. As well, the agent will now backoff when the APM server errors. Finally, improved error handling means it will terminate failing requests to the APM server more quickly.

Note: v1 of this agent (EOL’d 2 years ago), included a maxQueueSize config variable with a different meaning. If you have a lingering usage of that setting (also possibly as the ELASTIC_APM_MAX_QUEUE_SIZE environment variable), then it should be removed.

The APM agent’s own internal logging now uses structured JSON logging using the pino API, and formatted in ecs-logging format. The log records on stdout are now single-line JSON objects. For example:

# Before
APM Server transport error (ECONNREFUSED): connect ECONNREFUSED 127.0.0.1:8200

# After
{"log.level":"error","@timestamp":"2021-03-19T00:21:17.571Z","log":{"logger":"elastic-apm-node"},
"ecs":{"version":"1.6.0"},"message":"APM Server transport error (ECONNREFUSED): connect ECONNREFUSED 127.0.0.1:8200"}

Pretty formatting (and filtering) on the console may be done via the ecslog tool.

A custom logger is still supported as before. However, a non-pino custom logger will only receive the "message" field, and not structured log fields as they are added over time.

feat: Add automatic instrumentation of the @elastic/elasticsearch package #1870

The instrumentation of the legacy "elasticsearch" package has also changed slightly to commonalize:

feat: Add sanitize_field_names configuration option.
Allows users to configure a list of wildcard patterns to remove items from the agent’s HTTP header and application/x-www-form-urlencoded payloads. #1898

fix: Fix parsing of comma-separated strings for relevant config vars to allow whitespace around the commas. E.g.:

export ELASTIC_APM_TRANSACTION_IGNORE_URLS='/ping, /metrics*'

Config vars affected are: disableInstrumentations, transactionIgnoreUrls addPatch, and globalLabels.