Node.js Agent version 3.xedit

3.35.0 2022/06/01edit

Featuresedit
  • Change the instrumentation of SQS- and SNS-triggered AWS Lambda invocations: The special-casing of triggers with a single message/record has been removed. That means that instead of a possible continued distributed trace (if a single received message has a traceparent), a span link will be added to the APM transaction for each message with a traceparent. transaction.context.message. fields are no longer collected. (#2708)
  • Enable support for ioredis v5 (#2714)
  • A Docker image with the APM agent will be published for each release to docker.elastic.co/observability/apm-agent-nodejs:VERSION, for example: docker.elastic.co/observability/apm-agent-nodejs:3.35.0. (#2742)
Bug fixesedit
  • Fixes automatic Lambda handler wrapping to work with handlers that point to subfolders (ex. _HANDLER=path/to/folder.methodName) (#2709)

3.34.0 2022/05/26edit

Featuresedit
  • Add support for tedious version v10 to v14 (#2517)
  • When automatically determining serviceName and serviceVersion by looking for a "package.json", the agent will now prefer to start looking from the directory of the script being executed, rather than the current working directory. (#2420)
  • Add an experimental OpenTelemetry bridge. Briefly, the OpenTelemetry Bridge allows one to use the vendor-neutral OpenTelemetry Tracing API (@opentelemetry/api) to manually instrument your code, and have the Elastic Node.js APM agent handle those API calls. (#2641)
  • Add Span Links support. (#2673)

    The transaction.startSpan() and apm.startTransaction() public APIs now accept a links option for specify links. The OpenTelemetry Bridge also supports specifying links during span creation (with the limitation that span link attributes are not supported).

  • Add a traceContinuationStrategy configuration option to allow some control over how the APM Agent uses incoming trace-context headers for context propagation. (#2592)
  • Add span links to AWS SQS messaging spans on ReceiveMessage, one for each message (up to 1000) which has a traceparent message attribute. (#2593)
  • Add "nodejs16.x" as one of the compatible runtimes for the Node.js APM agent Lambda layers now that this runtime is available on AWS.
Bug fixesedit
  • Fixes a bug where the the agent would not serialize the database context of a span. (#2715)
  • Fix a possible crash in span compression handling on a span that was manually created without a parent span (e.g. if created with a custom childOf option). (#2701)
Choresedit
  • Add a package-lock.json file to ensure repeatable builds of the AWS Lambda layer and to assist with security issue auditing. (#2626)
  • Deprecate instrumentation for the legacy "hapi" package. While the APM agent still supports it, that instrumentation is no longer tested and support will be dropped in the next major version of the agent. Note that the "@hapi/hapi" package is still fully supported. (#2698)
  • Deprecate instrumentation for the obsolete "jade" package. "jade" was renamed to "pug" in 2015. While the APM agent still supports "jade", that instrumentation is no longer tested. (#2711)

3.33.0 2022/05/05edit

Featuresedit
  • Pulled the traceparent NPM module into a local module and replaced the random-poly-fill module with the built in require('crypto').randomFillSync function call (#2669)
  • Add a parent option to agent.captureError(err[, options][, cb]) to allow passing in a Transaction or Span to use as the parent for the error. Before this change the current span or transaction, if any, was always used.

    This option is not documented in the user docs, nor added to the TypeScript types, because it is only expected to be useful for coming OTel Bridge work.

Bug fixesedit
  • Fix a possible crash in the instrumentation of an incoming HTTP/2 request: if the underlying Http2Session was destroyed before the APM transaction was ended (on Http2Stream end). This resulted in the instrumentation using the [stream.session.socket](https://nodejs.org/api/http2.html#http2sessionsocket) proxy, which can throw ERR_HTTP2_SOCKET_UNBOUND after the session is destroyed. (#2670)
Choresedit
  • The release process is slightly changed. CI (Jenkins) now handles npm publish ... when a tag is pushed. (#2667)
  • Pulled the traceparent NPM module into a local module and replaced the random-poly-fill module with the built in require('crypto').randomFillSync function call (#2669)

3.32.0 2022/04/27edit

Featuresedit
  • Add support for node v18. (#2652)
  • Add support for span compression. (#2100, #2604)

    By default, consecutive (sibling) exit spans of the same name, type, subtype, and destination with a duration of less than 50ms will be compressed into a single composite span. A possible case is the N+1 query problem. Traces with many consecutive matching spans will be represented — both in data and the APM UI — more efficiently.

    Span compression can be disabled or matching behavior configured with the spanCompression* configuration options.

  • Marks spans as "exit spans" across all instrumentations, preventing additional child spans from being added to the exit spans. See issue for a full list of spans types that will be treated as exit spans. (#2601)
  • Allow a new span to be created/started even if its transaction has ended. This is expected to be a very rare use case. (#2653)
  • The Trace Context headers are now propagated for http2 requests. (#2656)

3.31.0 2022/03/23edit

Featuresedit
  • Add captureBody support for Hapi. (#1905)
  • If a SNS or SQS single event trigger to an instrumented Lambda function includes message attributes with the name "traceparent" (and "tracestate"), case-insensitive, then those are used to continue the trace. This was already being done for API Gateway event headers.
Bug fixesedit
  • Fix a bug with Lambda instrumentation where the APM agent would result in an otherwise working Lambda function to respond with null if the Lambda was missing the Elastic APM Lambda Extension. (#2598)
  • Fix a bug in Lambda instrumentation in the capturing of SNS and SQS event message attributes. (#2605)

3.30.0 2022/03/10edit

Breaking changesedit
  • Added a new config option spanStackTraceMinDuration that replaces both captureSpanStackTraces and spanFramesMinDuration. The latter two are now deprecated, but still supported. If spanStackTraceMinDuration is specified, then any value for the deprecated two options will be ignored.

    There is a significant change in default behavior of the APM agent. If none of these configuration options is specified, then the default (spanStackTraceMinDuration: -1) is that stack traces are not collected and reported for any spans. This change in default behavior was made because the CPU performance impact of collecting span stack traces was found to be too high in practice for busy and/or complex applications. This is mentioned in the "Breaking changes" section to highlight the change, but it is not considered breaking in general. The impact is that the "Stack Trace" tab in the "Span details" view in the Kibana APM app will be empty. This was already the case for some spans based on span duration. (#2565)

  • Implement the explicit signaling of Lambda invocation completion to the Elastic AWS Lambda Extension. This improves data flushing in a Lambda environment to ensure tracing data is only sent when the Lambda is active. This avoids possible tracing data loss while a Lambda VM is frozen. (#2485)

    However, because this change triggers a bug in the extension, this version of the APM Node.js Agent must only be used with versions of the AWS Lambda Extension after v0.0.3.

Featuresedit
  • Add faas.name and faas.version fields to Lambda transactions. (#2587)
  • Added automatic wrapping of AWS Lambda handlers (#2577)
  • Improvements to AWS Lambda instrumentation: Better transaction.name for API Gateway-triggered lambdas. Respect explicitly set serviceName, serviceVersion, and usePathAsTransactionName config settings. Default cloudProvider: none and centralConfig: false to reduce required environment variables for setting up APM instrumentation of Lambdas. (#2531)

3.29.0 2022/02/10edit

  • Fix a bug in instrumentation of @elastic/elasticsearch that caused a memory leak. (#2569)

3.28.0 2022/02/08edit

Known issue: This release includes a memory leak in instrumentation of the @elastic/elasticsearch package. If you use that package, you should not use v3.28.0 of this APM agent. (#2569)

Breaking changesedit

The following changes are not considered breaking. However, they result in a change in behavior and trace output that might impact some users, so they are highlighted here.

  • Change the redis and mysql instrumentations to not patch at all if they are listed in disableInstrumentations. This means that an application that uses one of these packages and lists that package in disableInstrumentations could see changes in the async run-context of callbacks. See #2498 and the v3.26.0 release notes which has a similar change.
  • Elasticsearch spans (from elasticsearch, @elastic/elasticsearch, and @elastic/elasticsearch-canary instrumentation) will no longer have an HTTP child span(s) for the underlying HTTP request. This is listed in this section to provide awareness in case some users have custom analysis of APM trace data that expects those HTTP spans.

    Per the APM Agent spec for exit spans, Elasticsearch spans are now marked as exit spans and as a result, HTTP child spans are suppressed. (#2000)

    As part of this change, some HTTP context has been added to Elasticsearch spans, when available: the HTTP response status_code, and the size of the response body (encoded_body_size). (#2484)

Featuresedit
  • Drop unsampled transactions when sending to APM Server v8.0+. (#2455)
  • The default serviceName string (when it is not configured and cannot be inferred from a "package.json" file) has been changed from "nodejs_service" to "unknown-nodejs-service". This is a standardized pattern used across Elastic APM agents to allow the Kibana APM app to recognize when to provide help to the user on configuring the service name. (#2491)
  • Add transaction.name to captured APM errors. This will allow the Kibana APM app to correlate error groups and transaction groups. (#2456)
  • Mark S3 spans (from aws-sdk instrumentation) as exit spans (per https://github.com/elastic/apm/blob/main/specs/agents/tracing-spans.md#exit-spans). The result is that HTTP child spans of S3 spans are no longer captured. (#2125)
Bug fixesedit
  • Fixes for run context handling for @elastic/elasticsearch instrumentation. (#2430)
  • Fixes for run context handling for cassandra-driver instrumentation. (#2430)
  • Fixes for run context handling for mongodb-core instrumentation. (#2430)
  • Fixes for run context handling for elasticsearch instrumentation. (#2430)

3.27.0 2022/01/17edit

Featuresedit
  • Add support for express-graphql 0.10.0 - 0.12.0 inclusive. (#2511)
Bug fixesedit
  • Fix an issue where the agent’s async task tracking could cause the user’s application to use too much memory. In cases where an application is under sustained load and is running in a memory constrained container, this looked like a memory leak.

    This high memory usage could happen when application code starts async tasks (e.g. Promises, setTimeouts, async I/O) that outlive the APM Transaction (typically an HTTP request handler). The agent’s async task tracking keeps a reference to the APM Transaction (and any APM Spans) until the async task ends, thus extending the lifetime of those APM objects and the references they hold — in particular, HTTP request and response objects. This could lead to higher memory usage.

    With this change, those references are removed when APM Transactions and Spans are ended, and agent memory usage is now the same as what it was before v3.24.0 when this issue was introduced. (#2528, #2489)

  • Fixes for run context handling for graphql instrumentation. (#2430)
  • Fixes for run context handling for DynamoDB instrumentation (aws-sdk package) so that a span created after an AWS client command (in the same tick, in the command callback, or promise) is not a child of the automatic AWS span. This change also ensures captured errors from failing client commands are a child of the AWS span. (#2430)
  • Fixes for run context handling for pg instrumentation. (#2430)
  • Fixes for run context handling for mongodb instrumentation. (#2512)

3.26.0 2021/12/07edit

Breaking changesedit
  • In earlier versions, the agent would propagate run context in some packages even if instrumentation for that package was disabled via disableInstrumentations. Recent versions change the semantics of disableInstrumentations to mean the agent should not touch the listed packages at all. This means that an application that uses one of these packages and lists that package in disableInstrumentations could see changes in the async run-context of callbacks. This affects: pg (v3.24.0), redis (v3.25.0), mysql (v3.25.0), ioredis (v3.26.0), mysql2 (v3.26.0). See #2498 for details.
Featuresedit
  • Add *auth* pattern to default value for sanitizeFieldNames config var, so that it is more likely to redact authentication/authorization-related HTTP headers and form fields. This pattern replaces the authorization pattern in the set of defaults. (#2427)
Bug fixesedit
  • Fix run-context handling for tedious instrumentation so that automatically created mssql spans are never the currentSpan in user code. (#2430)
  • Fix http2 instrumentation for outgoing requests to not have the created HTTP span context be active in user code. (#2430)
  • Fix run-context handling in ws instrumentation so that the span created for a ws.send(...) isn’t the "current span" in subsequent code in the same tick. (#2481)
  • Fix run-context handling for memcached instrumentation so that the automatically created Memcached span is never the currentSpan in user code. (#2430)
  • Fix a possible crash when serializing a Transaction if the incoming req.socket is null (possible if the socket has been destroyed). (#2479)
  • Fixes for run context handling for aws-sdk instrumentation (S3, SQS, SNS) so that a span created after an AWS client command (in the same tick, in the command callback, or promise) is not a child of the automatic AWS span. This change also ensures captured errors from failing client commands are a child of the AWS span. (#2430)
  • Fix http and https instrumentation for outgoing requests to not have the http span context be active in user code. (#2470)
  • Fixes for ioredis instrumentation (#2460):

    • Fix run-context so that a span created in the same tick as an ioredis client command will no longer be a child of the redis span.
    • Capture an APM error and correctly set span.outcome to "failure" when a redis client command calls back with an error.
    • Avoid a rare possible double-instrumentation of redis commands internally-queued before the RedisClient is "ready". (#2459)
    • Add destination context so Redis shows up on the Service Map.
  • Fix run-context handling for mysql2 instrumentation to avoid accidental creation of child spans of the automatic mysql spans. (#2430})

3.25.0 2021/11/24edit

Bug fixesedit
  • Fixes for redis instrumentation:

    • Fix run-context so that a span created in the same tick as a redis client command will no longer be a child of the redis span. (#2430)
    • Capture an APM error and correctly set span.outcome to "failure" when a redis client command calls back with an error.
    • Avoid a rare possible double-instrumentation of redis commands internally-queued before the RedisClient is "ready". (#2446)
  • Avoid setting the tracestate header for outgoing HTTP requests to the empty string. This can happen for non-trace-root transactions. While the HTTP spec allows empty header values, some servers do not. (#2405)
  • Deprecate transaction.subtype and transaction.action. These fields were never used by APM server. This also deprecates the apm.startTransaction(...) call signatures that take subtype and action arguments. In the next major version these two fields will be removed. (#2356)
  • Fix run-context handling for mysql instrumentation to avoid accidental creation of child spans of the automatic mysql spans. (#2430})

3.24.0 2021/11/09edit

Breaking changesedit
  • Change disableSend to no longer skip internal processing work. It now only disables communication with APM Server. Use contextPropagationOnly if your use case is to limit the APM agent’s processing to the minimum to support context propagation and log correlation.

    This is listed under "Breaking changes" as a heads-up. The only possible negative result of this disableSend change is some extra CPU processing time by the agent. There is no outward functionality change.

Featuresedit
  • Gather AWS Lambda-specific metadata on first function invocation and ensure no intake requests to APM Server are started before that metadata is available. (#2404)
  • Add contextPropagationOnly configuration option. This supports the use case of using the APM agent to propagate HTTP trace-context and to support log-correlation (adding trace.id et al fields to log records) without an APM server, and to otherwise reduce the processing time of the agent. (#2393)

    This is similar to disableSend, but differs in that contextPropagationOnly: true tells the agent to skip unnecessary work.

  • The User-Agent header used for communication with APM Server now includes the serviceName and serviceVersion. For some users this can be helpful for APM Server log analysis. (#2364)
  • In a Lambda enviornment we now collect a number of additional data fields on the Transaction object. See the spec for more information on fields collected. https://github.com/elastic/apm/blob/main/specs/agents/tracing-instrumentation-aws-lambda.md (#2156)
  • Zero configuration support. The only required agent configuration option is serviceName. Normally the agent will attempt to infer serviceName for the "name" field in a package.json file. However, that could fail. With this version, the agent will cope with: a scoped package name (@scope/name is normalized to scope-name), a "name" that isn’t a valid serviceName, not being able to find a "package.json" file, etc. Ultimately it will fallback to "nodejs_service". (#1944)

    One consequence of this change is that apm.getServiceName() will return undefined until the agent is started (check with apm.isStarted()).

Bug fixesedit
  • Stop collecting transaction breakdown metrics (transaction.breakdown.count, transaction.duration.sum.us, transaction.duration.count), as they are not being used in APM UI. (#2370)
  • Wrap fs.realpath.native when configured with asyncHooks=false. This fixes using that function (which was undefined before this fix) and a crash when importing fs-extra@10. (#2401)
  • A significant change was made to internal run context tracking (a.k.a. async context tracking). There are no configuration changes or API changes for custom instrumentation. (#2181)

    One behavior change is that multiple spans created synchronously (in the same async task) will form parent/child relationships; before this change they would all be siblings. This fixes HTTP child spans of Elasticsearch and aws-sdk automatic spans to properly be children. (#1889)

    Another behavior change is that a span B started after having ended span A in the same async task will no longer be a child of span A. (#1964)

    This fixes an issue with context binding of EventEmitters, where removeListener would fail to actually remove if the same handler function was added to multiple events.

  • Fix pino’s deprecation warning when using a custom logger with pino@6 (#2332)

3.23.0 2021/10/25edit

Breaking changesedit
  • Remove support for instrumenting versions of fastify earlier than 1.0.0. This instrumentation might still work, but is no longer supported. Fastify v1.0.0 was released in 2018. All current users should be using fastify v2 or v3 at least. See https://www.fastify.io/docs/latest/LTS/ (#2387)
Featuresedit
  • Add initial support for version 8 of @elastic/elasticsearch, which is still in pre-release. (#2385)

3.22.0 2021/10/21edit

Featuresedit
  • Add support for node v17.
  • When an error is captured, the APM agent will only immediately flush it to APM server if it is an "unhandled" error. Unhandled errors are typically those captured via the uncaughtException process event. Before this change, a captured error (e.g. for a 4xx or 5xx response from an HTTP server) was always immediately flushed. This could negatively impact performance for a service that was generating frequent errors. (#686)
Bug fixesedit
  • Improve apm.flush([cb]) to wait for inflight spans and errors before flushing data to APM server. Before this change, a recently ended span or recently captured error might not yet have completed processing (for example, stacktrace collection is asynchronous) and might not be included in the flush call. (#2294)
  • AWS Lambda changes: Disable metrics collection during the experimental phase of (re)implementing Lambda support (#2363). Some fixes for better flushing of data at the end of a Lambda invocation.
  • span.toString() and transaction.toString() have been deprecated. The exact string output may change in v4 of the agent.
  • Add Span.ids and Transaction.ids to TypeScript types. (#2347)
  • Improve span.sync determination (fixes #1996) and stop reporting transaction.sync which was never used (#2292). A minor semantic change is that span.sync is not set to a final value until span.end() is called. Before span.end() the value will always by true.
  • Guard against a negative value of metricsInterval, which can lead to high CPU usage as metrics are collected as fast as possible. Also ensure no metrics collection can happen if metricsInterval="0s" as intended. Before this change it was possible for some metric collection to still happen, even though none would be reported. (#2330)

    This change also guards against negative and invalid values in the following configuration options: abortedErrorThreshold, apiRequestTime, and serverTimeout. If an invalid value is given, then will fallback to their default value.

3.21.1 2021/09/16edit

Bug fixesedit
  • Update types to avoid imports of @types/... modules (other than @types/node), so that TypeScript users of elastic-apm-node need not manually npm install @types/connect @types/pino @types/aws-lambda to compile. (#2331)

3.21.0 2021/09/15edit

Featuresedit
  • Add the longFieldMaxLength integer configuration option (default 10000). Specific transaction/span/error fields (see the list below) will be truncated at this number of unicode characters. (#2193, #1921)

    The errorMessageMaxLength configuration option is now deprecated, but still supported. Users should switch to using longFieldMaxLength. If errorMessageMaxLength is not specified, truncation of error messages will now use the longFieldMaxLength value.

    Note that ultimately the maximum length of any tracing field is limited by the max_event_size configured for the receiving APM server.

    The fields affected by longFieldMaxLength are:

    • transaction.context.request.body, error.context.request.body - Before this change these fields were not truncated.
    • transaction.context.message.body, span.context.message.body, error.context.message.body - Before this change these fields were not truncated.
    • span.context.db.statement - Before this change this field was truncated at 10000 bytes. Truncation is now a number of unicode characters.
    • error.exception.message, error.log.message - Before this change, the default 2kB errorMessageMaxLength would apply.
  • Improve the TypeScript types by exporting more of interfaces: AgentConfigOptions, Transaction, Span, TransactionOptions, SpanOptions. (#2118)
Bug fixesedit
  • Fix a bug in apm.removePatch(module, aHandler) that would remove the last registered handler if aHandler did not match any currently registered handlers. (#2315)
  • Fix a crash in instrumentation of the old Elasticsearch client (elasticsearch) for some rarer cases of client options — for example passing multiple hosts. (#2312)
  • Ensure the internal HTTP(S) client requests made by the APM agent to APM server are not themselves traced. (#1168, #1136)
  • Fix crashing error with agent.registerMetric and active:false configuration. (#1799, #2290)

3.20.0 2021/08/12edit

Bug fixesedit
  • Fix failing tests and a possible runtime crash in @elastic/elasticsearch@7.14.0 instrumentation. (#2187)

3.19.0 2021/08/05edit

Featuresedit
  • The agent now supports the 3.x branch of apollo-server-express. (#2155)
  • Add instrumentation support for mongodb@4.x. (#2171)
Bug fixesedit
  • The agent will no longer report counting metrics with a value of zero, and will remove these metrics from the registry. (#2163)

3.18.0 2021/07/20edit

Featuresedit
  • Trace an incoming HTTP/1.1 request to an HTTP/2 server using the allowHTTP1 option. Before this change only incoming requests supporting HTTP/2 would be traced. (#2143)
  • Add instrumentation of the AWS SNS publish method when using the JavaScript AWS SDK v2 (aws-sdk). (#2157)
Bug fixesedit

3.17.0 2021/07/05edit

Featuresedit
  • Add instrumentation of all AWS S3 methods when using the JavaScript AWS SDK v2 (aws-sdk).
  • Add disableSend configuration option. This supports some use cases using the APM agent without an APM server. (#2101)
  • Add instrumentation of all DynamoDB methods when using the JavaScript AWS SDK v2 (aws-sdk).
Bug fixesedit
  • Fix inconsistencies in HTTP spans from other APM agents. span.subtype will now be "http" for HTTP, HTTPS, and HTTP/2 outgoing spans — previously it was "http", "https", "http2", respectively. As well, span.action will now be the HTTP method (e.g. "GET", "PUT", "POST"), rather than "http". (#2075)
  • Fixed error where SQS messages sent without an active transactions could crash the agent. (#2113)
  • Fixed support for proxies in destination context (#1770)

3.16.0 - 2021/06/14edit

Featuresedit
  • Added spanFramesMinDuration configuration field, allowing users to set a time threshold value that spans must reach before the agent will add a stack trace to the span. As a result, many short spans that previously included stack traces by default no longer will.
  • Prefer W3C "traceparent" header over "elastic-apm-traceparent" for incoming requests. #2079
Bug fixesedit
  • Fix a crash (TypeError: lastPrepareStackTrace) in the agent when used with React v17 and later (#1980).
  • Performance improvements have been made in error and stacktrace capture (#2094). This also included in two bug fixes:

    • Before this change, some captured errors (for example a next(new Error('boom') from an Express handler) would mark the error as "unhandled" incorrectly. "Unhandled" exceptions are those caught by an uncaughtException handler.
    • Before this change, source context lines for a stacktrace would not properly use the "sourcesContext" field of a file’s source map.

3.15.0 - 2021/05/19edit

Featuresedit
  • Add support for Node.js v16. (This also drops testing of Node.js v13 releases.) #2055
Bug fixesedit
  • Update TypeScript typings for Agent.setLabel and Agent.addLabels to include the stringify argument that was added in v3.11.0.

3.14.0 - 2021/04/19edit

Featuresedit
  • Add apm.addMetadataFilter(fn) for filtering the metadata object sent to APM server.
  • The handling of sending events (transactions, spans, errors) to APM server has improved in a few ways. During temporary spikes in load and/or an APM server that is unresponsive, the agent will buffer a number of events and drop them above a certain limit (configurable via maxQueueSize). This helps ensure the agent does not overly consume memory and CPU. As well, the agent will now backoff when the APM server errors. Finally, improved error handling means it will terminate failing requests to the APM server more quickly.

    Note: v1 of this agent (EOL’d 2 years ago), included a maxQueueSize config variable with a different meaning. If you have a lingering usage of that setting (also possibly as the ELASTIC_APM_MAX_QUEUE_SIZE environment variable), then it should be removed.

  • Adds support for Amazon SQS queues via aws-sdk instrumentation that partially implements the APM messaging spec, and adds queue.latency.min.ms, queue.latency.max.ms, and queue.latency.avg.ms metrics for SQS queues.
Bug fixesedit
  • Fixed bug where the URL property for outgoing HTTP request spans was set with the server’s IP address rather than its hostname. The Agent now sets this property with the actual URL requested by Node.js. #2035
  • Fixed bug where external services were not listed under Dependencies on the APM Service Overview page due to the trace-context propagated sample_rate value not being set on either transactions or spans.

3.13.0 - 2021/04/06edit

Featuresedit
  • The APM agent’s own internal logging now uses structured JSON logging using the pino API, and formatted in ecs-logging format. The log records on stdout are now single-line JSON objects. For example:

    # Before
    APM Server transport error (ECONNREFUSED): connect ECONNREFUSED 127.0.0.1:8200
    
    # After
    {"log.level":"error","@timestamp":"2021-03-19T00:21:17.571Z","log":{"logger":"elastic-apm-node"},
    "ecs":{"version":"1.6.0"},"message":"APM Server transport error (ECONNREFUSED): connect ECONNREFUSED 127.0.0.1:8200"}

    Pretty formatting (and filtering) on the console may be done via the ecslog tool.

    A custom logger is still supported as before. However, a non-pino custom logger will only receive the "message" field, and not structured log fields as they are added over time.

  • Add support for setting the ELASTIC_APM_LOGGER=false environment variable to disable/ignore a given custom logger. This is to support easier Debug mode for troubleshooting.
Bug fixesedit
  • Lock package dependency "elastic-apm-http-client@9.6.0" to avoid using v9.7.0 for now, because it is breaking tests. A coming release will get back on the latest of this dependency. #2032
  • Remove the "ancestors" field from a log.trace message on startup. Its info is a duplicate of info in the "startTrace" field in the same log record. #2005
  • Remove the accidental nodejs.eventloop.delay.ns metric that was always reporting a zero value. The existing nodejs.eventloop.delay.avg.ms is the intended metric. #1993

3.12.1 - 2021/02/25edit

Bug fixesedit

3.12.0 - 2021/02/21edit

Featuresedit
  • feat: Set span outcome to success or failure depending on whether an error was captured during when the span was active. #1814
  • feat: Adds public setOutcome method to span and transaction APIs, and adds a top level setTransactionOutcome and setSpanOutcome to set outcome values for the current active transaction or active span.
  • Limit the transactionSampleRate value to 4 decimal places of precision according to the shared APM spec. This ensures that propagated sampling rate in the tracestate header is short and consistent. #1979
Bug fixesedit
  • fix: It was possible for fetching central config to result in the cloudProvider config value being reset to its default. #1976
  • fix: fixes bug where tedious could crash the agent on bulk inserts #1935
    Reported via the forum. The error symptom was: Cannot read property 'statement' of undefined

3.11.0 - 2021/02/08edit

Featuresedit
  • feat: add apm.getServiceName() #1949
    This will be used by ecs-logging packages to integrate with APM.
  • feat: support numeric and boolean labels #1909
    Add an optional stringify option to apm.setLabel(name, version, stringify = true) and apm.addLabels(labels, stringify = true), which can be set false to allow numeric and boolean labels. Stringify defaults to true for backwards compatibility — stringification will be removed in a future major version.
  • feat: added support for cloud metadata fetching #1937
    Agent now collects information about its cloud environment and includes this data in the APM Server’s metadata payload. See the spec for more information.

3.10.0 - 2021/01/11edit

Featuresedit
  • feat: Improve handling of raw body parsing The agent will now report raw/Buffer encoded post bodies as <Buffer>.
  • feat: Add support for api keys #1818
    This allows the usage of API keys for authentication to the APM server
  • feat: Add automatic instrumentation of the @elastic/elasticsearch package #1870

    The instrumentation of the legacy "elasticsearch" package has also changed slightly to commonalize:

    • "span.context.destination" is set on all Elasticsearch spans, not just a subset of query-like API endpoints.
    • For query-like API endpoints (e.g. /_search), the capturing of query details on "span.context.db.statement" has changed (a) to include both the query params and the request body if both exist (separated by \n\n) and (b) to URL encode the query params, rather than JSON encoding.
  • feat: Add captureAttributes boolean option to apm.captureError() to allow disabling the automatic capture of Error object properties. This is useful for cases where those properties should not be sent to the APM Server, e.g. for performance (large string fields) or security (PII data). #1912
  • feat: Add log_level central config support. #1908
    Spec: https://github.com/elastic/apm/blob/main/specs/agents/logging.md
  • feat: Add sanitize_field_names configuration option.
    Allows users to configure a list of wildcard patterns to remove items from the agent’s HTTP header and application/x-www-form-urlencoded payloads. #1898

Bug fixesedit
  • fix: Fix parsing of comma-separated strings for relevant config vars to allow whitespace around the commas. E.g.:

    export ELASTIC_APM_TRANSACTION_IGNORE_URLS='/ping, /metrics*'

    Config vars affected are: disableInstrumentations, transactionIgnoreUrls addPatch, and globalLabels.

  • fix: Correct the environment variable for setting transactionIgnoreUrl (added in v3.9.0) from ELASTIC_TRANSACTION_IGNORE_URLS to ELASTIC_APM_TRANSACTION_IGNORE_URLS.

3.9.0 - 2020/11/30edit

Featuresedit
  • feat: support fastify 3 #1891
    Adds .default and .fastify module.exports to instrumented fastify function for 3.x line, and prefers req.routerMethod and req.routerPath for transaction name
  • feat: Set "destination" context on spans for "mongodb". #1893
    This allows Kibana APM Service Maps to show a "mongodb" node for services using the mongodb package (which includes mongoose and mongojs).
  • feat: transactionIgnoreUrl wildcard matching #1870
    Allows users to ignore URLs using simple wildcard matching patterns that behave the same across language agents. See https://github.com/elastic/apm/issues/144
Bug fixesedit
  • fix: treat set-cookie in response headers as sensitive data #1886
  • fix: Synchronous spans would never have span.sync == true. #1879

3.8.0 - 2020/11/09edit

Featuresedit
  • feat: expand k8s pod ID discovery regex #1863
  • feat: implements tracestate #1828
    Expands support for the W3C Trace Context specification by adding a tracestate header implementation, and uses this new header to track the Elastic transaction sample rate across a trace’s service boundaries.
  • feat: add span and transaction outcome #1824
    This adds an "outcome" field to HTTP(S) transactions and spans.
Bug fixesedit
  • fix(pg): prevent unhandled promise rejection #1846
  • fix: redis@2.x instrumentation was broken #1852
  • A number of fixes to the test suite.

3.7.0 - 2020/8/10edit

  • feat(knex): add support for 0.21.x #1801
  • feat(redis): add support for v3.x #1641
  • feat(graphql): add support for 15.x #1795
  • feat(koa-router): add support for 9.x #1772
  • fix(elasticsearch): ensure requests can be aborted #1566
  • fix: end span if outgoing http request ends prematurely #1583
  • fix: don’t throw on invalid URL #1771
  • fix: patch apollo-server-core > 2.14 correctly #1796
  • fix: add currentTraceIds to typings #1733

3.6.1 - 2020/5/20edit

  • fix(package): bump elastic-apm-http-client to ^9.4.0 #1756

3.6.0 - 2020/5/18edit

  • feat: add destination metadata for db spans #1687
  • feat: add support for Node.js 14 #1742
  • feat(pg): add support for pg v8.x #1743
  • feat: add metrics for external memory #1724
  • feat: enrich spans with destination info #1685
  • fix(instrumentation): add .js to module path #1711

3.5.0 - 2020/3/9edit

  • feat(error): get stack trace from Error-like objects #1613
  • fix: add logUncaughtExceptions conf option to TypeScript typings #1668

3.4.0 - 2020/2/21edit

  • feat: support W3C TraceContext traceparent header #1587
  • feat: add custom metrics API (experimental) #1571
  • feat(koa-router): add support for v8.x #1642
  • fix(cassandra): improve support for cassandra-driver v4.4.0+ #1636
  • fix: support promisifying setTimeout and friends #1636

3.3.0 - 2019/12/13edit

  • feat(config): add serverCaCertFile config #1560
  • feat(config): add central config support for transactionMaxSpans and captureBody #1555

3.2.0 - 2019/11/19edit

  • fix(metrics): only register collectors if enabled #1520
  • fix(ioredis): prevent unhandled promise rejection #1523
  • chore: add Node 13 to supported engines #1524

3.1.0 - 2019/10/16edit

Featuresedit
  • feat(mongodb): instrumentation #1423
  • fix(package): update elastic-apm-http-client to version 9.0.0 #1419
  • perf: cache ids value of transactions and spans #1434
Bug fixesedit
  • fix: always end transaction when socket is closed prematurely #1439
  • fix: change logUncaughtExceptions default to false #1432
  • fix: write stack trace of uncaught exceptions to STDERR #1429

3.0.0 - 2019/9/30edit

Breaking changesedit
  • feat: allow manual instrumentation with instrument: false #1114
  • feat: allow setting span/transaction type, subtype, and action separately (the behavior of the old type has changed) #1292
  • feat: use external as span type instead of ext #1291
  • refactor(graphql): use custom transaction type graphql for graphql requests instead of request #1245
  • feat(http): add instrumentIncomingHTTPRequests config (disableInstrumentations now behaves differently) #1298
  • chore: remove deprecated APIs #1413
  • chore: drop support for older Node.js versions #1383