Add Span Links support. (#2673)

The transaction.startSpan() and apm.startTransaction() public APIs now accept a links option for specify links. The OpenTelemetry Bridge also supports specifying links during span creation (with the limitation that span link attributes are not supported).

Add a parent option to agent.captureError(err[, options][, cb]) to allow passing in a Transaction or Span to use as the parent for the error. Before this change the current span or transaction, if any, was always used.

This option is not documented in the user docs, nor added to the TypeScript types, because it is only expected to be useful for coming OTel Bridge work.

Add support for span compression. (#2100, #2604)

By default, consecutive (sibling) exit spans of the same name, type, subtype, and destination with a duration of less than 50ms will be compressed into a single composite span. A possible case is the N+1 query problem. Traces with many consecutive matching spans will be represented — both in data and the APM UI — more efficiently.

Span compression can be disabled or matching behavior configured with the spanCompression* configuration options.

Added a new config option spanStackTraceMinDuration that replaces both captureSpanStackTraces and spanFramesMinDuration. The latter two are now deprecated, but still supported. If spanStackTraceMinDuration is specified, then any value for the deprecated two options will be ignored.

There is a significant change in default behavior of the APM agent. If none of these configuration options is specified, then the default (spanStackTraceMinDuration: -1) is that stack traces are not collected and reported for any spans. This change in default behavior was made because the CPU performance impact of collecting span stack traces was found to be too high in practice for busy and/or complex applications. This is mentioned in the "Breaking changes" section to highlight the change, but it is not considered breaking in general. The impact is that the "Stack Trace" tab in the "Span details" view in the Kibana APM app will be empty. This was already the case for some spans based on span duration. (#2565)

Implement the explicit signaling of Lambda invocation completion to the Elastic AWS Lambda Extension. This improves data flushing in a Lambda environment to ensure tracing data is only sent when the Lambda is active. This avoids possible tracing data loss while a Lambda VM is frozen. (#2485)

However, because this change triggers a bug in the extension, this version of the APM Node.js Agent must only be used with versions of the AWS Lambda Extension after v0.0.3.

Elasticsearch spans (from elasticsearch, @elastic/elasticsearch, and @elastic/elasticsearch-canary instrumentation) will no longer have an HTTP child span(s) for the underlying HTTP request. This is listed in this section to provide awareness in case some users have custom analysis of APM trace data that expects those HTTP spans.

Per the APM Agent spec for exit spans, Elasticsearch spans are now marked as exit spans and as a result, HTTP child spans are suppressed. (#2000)

As part of this change, some HTTP context has been added to Elasticsearch spans, when available: the HTTP response status_code, and the size of the response body (encoded_body_size). (#2484)

Fix an issue where the agent’s async task tracking could cause the user’s application to use too much memory. In cases where an application is under sustained load and is running in a memory constrained container, this looked like a memory leak.

This high memory usage could happen when application code starts async tasks (e.g. Promises, setTimeouts, async I/O) that outlive the APM Transaction (typically an HTTP request handler). The agent’s async task tracking keeps a reference to the APM Transaction (and any APM Spans) until the async task ends, thus extending the lifetime of those APM objects and the references they hold — in particular, HTTP request and response objects. This could lead to higher memory usage.

With this change, those references are removed when APM Transactions and Spans are ended, and agent memory usage is now the same as what it was before v3.24.0 when this issue was introduced. (#2528, #2489)

Fixes for ioredis instrumentation (#2460):

Fixes for redis instrumentation:

Change disableSend to no longer skip internal processing work. It now only disables communication with APM Server. Use contextPropagationOnly if your use case is to limit the APM agent’s processing to the minimum to support context propagation and log correlation.

This is listed under "Breaking changes" as a heads-up. The only possible negative result of this disableSend change is some extra CPU processing time by the agent. There is no outward functionality change.

Add contextPropagationOnly configuration option. This supports the use case of using the APM agent to propagate HTTP trace-context and to support log-correlation (adding trace.id et al fields to log records) without an APM server, and to otherwise reduce the processing time of the agent. (#2393)

This is similar to disableSend, but differs in that contextPropagationOnly: true tells the agent to skip unnecessary work.

Zero configuration support. The only required agent configuration option is serviceName. Normally the agent will attempt to infer serviceName for the "name" field in a package.json file. However, that could fail. With this version, the agent will cope with: a scoped package name (@scope/name is normalized to scope-name), a "name" that isn’t a valid serviceName, not being able to find a "package.json" file, etc. Ultimately it will fallback to "nodejs_service". (#1944)

One consequence of this change is that apm.getServiceName() will return undefined until the agent is started (check with apm.isStarted()).

A significant change was made to internal run context tracking (a.k.a. async context tracking). There are no configuration changes or API changes for custom instrumentation. (#2181)

One behavior change is that multiple spans created synchronously (in the same async task) will form parent/child relationships; before this change they would all be siblings. This fixes HTTP child spans of Elasticsearch and aws-sdk automatic spans to properly be children. (#1889)

Another behavior change is that a span B started after having ended span A in the same async task will no longer be a child of span A. (#1964)

This fixes an issue with context binding of EventEmitters, where removeListener would fail to actually remove if the same handler function was added to multiple events.

Guard against a negative value of metricsInterval, which can lead to high CPU usage as metrics are collected as fast as possible. Also ensure no metrics collection can happen if metricsInterval="0s" as intended. Before this change it was possible for some metric collection to still happen, even though none would be reported. (#2330)

This change also guards against negative and invalid values in the following configuration options: abortedErrorThreshold, apiRequestTime, and serverTimeout. If an invalid value is given, then will fallback to their default value.

Add the longFieldMaxLength integer configuration option (default 10000). Specific transaction/span/error fields (see the list below) will be truncated at this number of unicode characters. (#2193, #1921)

The errorMessageMaxLength configuration option is now deprecated, but still supported. Users should switch to using longFieldMaxLength. If errorMessageMaxLength is not specified, truncation of error messages will now use the longFieldMaxLength value.

Note that ultimately the maximum length of any tracing field is limited by the max_event_size configured for the receiving APM server.

The fields affected by longFieldMaxLength are:

Performance improvements have been made in error and stacktrace capture (#2094). This also included in two bug fixes:

The handling of sending events (transactions, spans, errors) to APM server has improved in a few ways. During temporary spikes in load and/or an APM server that is unresponsive, the agent will buffer a number of events and drop them above a certain limit (configurable via maxQueueSize). This helps ensure the agent does not overly consume memory and CPU. As well, the agent will now backoff when the APM server errors. Finally, improved error handling means it will terminate failing requests to the APM server more quickly.

Note: v1 of this agent (EOL’d 2 years ago), included a maxQueueSize config variable with a different meaning. If you have a lingering usage of that setting (also possibly as the ELASTIC_APM_MAX_QUEUE_SIZE environment variable), then it should be removed.

The APM agent’s own internal logging now uses structured JSON logging using the pino API, and formatted in ecs-logging format. The log records on stdout are now single-line JSON objects. For example:

# Before
APM Server transport error (ECONNREFUSED): connect ECONNREFUSED 127.0.0.1:8200

# After
{"log.level":"error","@timestamp":"2021-03-19T00:21:17.571Z","log":{"logger":"elastic-apm-node"},
"ecs":{"version":"1.6.0"},"message":"APM Server transport error (ECONNREFUSED): connect ECONNREFUSED 127.0.0.1:8200"}

Pretty formatting (and filtering) on the console may be done via the ecslog tool.

A custom logger is still supported as before. However, a non-pino custom logger will only receive the "message" field, and not structured log fields as they are added over time.

feat: Add automatic instrumentation of the @elastic/elasticsearch package #1870

The instrumentation of the legacy "elasticsearch" package has also changed slightly to commonalize:

feat: Add sanitize_field_names configuration option.
Allows users to configure a list of wildcard patterns to remove items from the agent’s HTTP header and application/x-www-form-urlencoded payloads. #1898

fix: Fix parsing of comma-separated strings for relevant config vars to allow whitespace around the commas. E.g.:

export ELASTIC_APM_TRANSACTION_IGNORE_URLS='/ping, /metrics*'

Config vars affected are: disableInstrumentations, transactionIgnoreUrls addPatch, and globalLabels.