Node.js Agent version 3.xedit

3.27.0 2022/01/17edit

Featuresedit
  • Add support for express-graphql 0.10.0 - 0.12.0 inclusive. (#2511)
Bug fixesedit
  • Fix an issue where the agent’s async task tracking could cause the user’s application to use too much memory. In cases where an application is under sustained load and is running in a memory constrained container, this looked like a memory leak.

    This high memory usage could happen when application code starts async tasks (e.g. Promises, setTimeouts, async I/O) that outlive the APM Transaction (typically an HTTP request handler). The agent’s async task tracking keeps a reference to the APM Transaction (and any APM Spans) until the async task ends, thus extending the lifetime of those APM objects and the references they hold — in particular, HTTP request and response objects. This could lead to higher memory usage.

    With this change, those references are removed when APM Transactions and Spans are ended, and agent memory usage is now the same as what it was before v3.24.0 when this issue was introduced. (#2528, #2489)

  • Fixes for run context handling for graphql instrumentation. (#2430)
  • Fixes for run context handling for DynamoDB instrumentation (aws-sdk package) so that a span created after an AWS client command (in the same tick, in the command callback, or promise) is not a child of the automatic AWS span. This change also ensures captured errors from failing client commands are a child of the AWS span. (#2430)
  • Fixes for run context handling for pg instrumentation. (#2430)
  • Fixes for run context handling for mongodb instrumentation. (#2512)

3.26.0 2021/12/07edit

Breaking changesedit
  • In earlier versions, the agent would propagate run context in some packages even if instrumentation for that package was disabled via disableInstrumentations. Recent versions change the semantics of disableInstrumentations to mean the agent should not touch the listed packages at all. This means that an application that uses one of these packages and lists that package in disableInstrumentations could see changes in the async run-context of callbacks. This affects: pg (v3.24.0), redis (v3.25.0), mysql (v3.25.0), ioredis (v3.26.0), mysql2 (v3.26.0). See #2498 for details.
Featuresedit
  • Add *auth* pattern to default value for sanitizeFieldNames config var, so that it is more likely to redact authentication/authorization-related HTTP headers and form fields. This pattern replaces the authorization pattern in the set of defaults. (#2427)
Bug fixesedit
  • Fix run-context handling for tedious instrumentation so that automatically created mssql spans are never the currentSpan in user code. (#2430)
  • Fix http2 instrumentation for outgoing requests to not have the created HTTP span context be active in user code. (#2430)
  • Fix run-context handling in ws instrumentation so that the span created for a ws.send(...) isn’t the "current span" in subsequent code in the same tick. (#2481)
  • Fix run-context handling for memcached instrumentation so that the automatically created Memcached span is never the currentSpan in user code. (#2430)
  • Fix a possible crash when serializing a Transaction if the incoming req.socket is null (possible if the socket has been destroyed). (#2479)
  • Fixes for run context handling for aws-sdk instrumentation (S3, SQS, SNS) so that a span created after an AWS client command (in the same tick, in the command callback, or promise) is not a child of the automatic AWS span. This change also ensures captured errors from failing client commands are a child of the AWS span. (#2430)
  • Fix http and https instrumentation for outgoing requests to not have the http span context be active in user code. (#2470)
  • Fixes for ioredis instrumentation (#2460):

    • Fix run-context so that a span created in the same tick as an ioredis client command will no longer be a child of the redis span.
    • Capture an APM error and correctly set span.outcome to "failure" when a redis client command calls back with an error.
    • Avoid a rare possible double-instrumentation of redis commands internally-queued before the RedisClient is "ready". (#2459)
    • Add destination context so Redis shows up on the Service Map.
  • Fix run-context handling for mysql2 instrumentation to avoid accidental creation of child spans of the automatic mysql spans. (#2430})

3.25.0 2021/11/24edit

Bug fixesedit
  • Fixes for redis instrumentation:

    • Fix run-context so that a span created in the same tick as a redis client command will no longer be a child of the redis span. (#2430)
    • Capture an APM error and correctly set span.outcome to "failure" when a redis client command calls back with an error.
    • Avoid a rare possible double-instrumentation of redis commands internally-queued before the RedisClient is "ready". (#2446)
  • Avoid setting the tracestate header for outgoing HTTP requests to the empty string. This can happen for non-trace-root transactions. While the HTTP spec allows empty header values, some servers do not. (#2405)
  • Deprecate transaction.subtype and transaction.action. These fields were never used by APM server. This also deprecates the apm.startTransaction(...) call signatures that take subtype and action arguments. In the next major version these two fields will be removed. (#2356)
  • Fix run-context handling for mysql instrumentation to avoid accidental creation of child spans of the automatic mysql spans. (#2430})

3.24.0 2021/11/09edit

Breaking changesedit
  • Change disableSend to no longer skip internal processing work. It now only disables communication with APM Server. Use contextPropagationOnly if your use case is to limit the APM agent’s processing to the minimum to support context propagation and log correlation.

    This is listed under "Breaking changes" as a heads-up. The only possible negative result of this disableSend change is some extra CPU processing time by the agent. There is no outward functionality change.

Featuresedit
  • Gather AWS Lambda-specific metadata on first function invocation and ensure no intake requests to APM Server are started before that metadata is available. (#2404)
  • Add contextPropagationOnly configuration option. This supports the use case of using the APM agent to propagate HTTP trace-context and to support log-correlation (adding trace.id et al fields to log records) without an APM server, and to otherwise reduce the processing time of the agent. (#2393)

    This is similar to disableSend, but differs in that contextPropagationOnly: true tells the agent to skip unnecessary work.

  • The User-Agent header used for communication with APM Server now includes the serviceName and serviceVersion. For some users this can be helpful for APM Server log analysis. (#2364)
  • In a Lambda enviornment we now collect a number of additional data fields on the Transaction object. See the spec for more information on fields collected. https://github.com/elastic/apm/blob/master/specs/agents/tracing-instrumentation-aws-lambda.md (#2156)
  • Zero configuration support. The only required agent configuration option is serviceName. Normally the agent will attempt to infer serviceName for the "name" field in a package.json file. However, that could fail. With this version, the agent will cope with: a scoped package name (@scope/name is normalized to scope-name), a "name" that isn’t a valid serviceName, not being able to find a "package.json" file, etc. Ultimately it will fallback to "nodejs_service". (#1944)

    One consequence of this change is that apm.getServiceName() will return undefined until the agent is started (check with apm.isStarted()).

Bug fixesedit
  • Stop collecting transaction breakdown metrics (transaction.breakdown.count, transaction.duration.sum.us, transaction.duration.count), as they are not being used in APM UI. (#2370)
  • Wrap fs.realpath.native when configured with asyncHooks=false. This fixes using that function (which was undefined before this fix) and a crash when importing fs-extra@10. (#2401)
  • A significant change was made to internal run context tracking (a.k.a. async context tracking). There are no configuration changes or API changes for custom instrumentation. (#2181)

    One behavior change is that multiple spans created synchronously (in the same async task) will form parent/child relationships; before this change they would all be siblings. This fixes HTTP child spans of Elasticsearch and aws-sdk automatic spans to properly be children. (#1889)

    Another behavior change is that a span B started after having ended span A in the same async task will no longer be a child of span A. (#1964)

    This fixes an issue with context binding of EventEmitters, where removeListener would fail to actually remove if the same handler function was added to multiple events.

  • Fix pino’s deprecation warning when using a custom logger with pino@6 (#2332)

3.23.0 2021/10/25edit

Breaking changesedit
  • Remove support for instrumenting versions of fastify earlier than 1.0.0. This instrumentation might still work, but is no longer supported. Fastify v1.0.0 was released in 2018. All current users should be using fastify v2 or v3 at least. See https://www.fastify.io/docs/latest/LTS/ (#2387)
Featuresedit
  • Add initial support for version 8 of @elastic/elasticsearch, which is still in pre-release. (#2385)

3.22.0 2021/10/21edit

Featuresedit
  • Add support for node v17.
  • When an error is captured, the APM agent will only immediately flush it to APM server if it is an "unhandled" error. Unhandled errors are typically those captured via the uncaughtException process event. Before this change, a captured error (e.g. for a 4xx or 5xx response from an HTTP server) was always immediately flushed. This could negatively impact performance for a service that was generating frequent errors. (#686)
Bug fixesedit
  • Improve apm.flush([cb]) to wait for inflight spans and errors before flushing data to APM server. Before this change, a recently ended span or recently captured error might not yet have completed processing (for example, stacktrace collection is asynchronous) and might not be included in the flush call. (#2294)
  • AWS Lambda changes: Disable metrics collection during the experimental phase of (re)implementing Lambda support (#2363). Some fixes for better flushing of data at the end of a Lambda invocation.
  • span.toString() and transaction.toString() have been deprecated. The exact string output may change in v4 of the agent.
  • Add Span.ids and Transaction.ids to TypeScript types. (#2347)
  • Improve span.sync determination (fixes #1996) and stop reporting transaction.sync which was never used (#2292). A minor semantic change is that span.sync is not set to a final value until span.end() is called. Before span.end() the value will always by true.
  • Guard against a negative value of metricsInterval, which can lead to high CPU usage as metrics are collected as fast as possible. Also ensure no metrics collection can happen if metricsInterval="0s" as intended. Before this change it was possible for some metric collection to still happen, even though none would be reported. (#2330)

    This change also guards against negative and invalid values in the following configuration options: abortedErrorThreshold, apiRequestTime, and serverTimeout. If an invalid value is given, then will fallback to their default value.

3.21.1 2021/09/16edit

Bug fixesedit
  • Update types to avoid imports of @types/... modules (other than @types/node), so that TypeScript users of elastic-apm-node need not manually npm install @types/connect @types/pino @types/aws-lambda to compile. (#2331)

3.21.0 2021/09/15edit

Featuresedit
  • Add the longFieldMaxLength integer configuration option (default 10000). Specific transaction/span/error fields (see the list below) will be truncated at this number of unicode characters. (#2193, #1921)

    The errorMessageMaxLength configuration option is now deprecated, but still supported. Users should switch to using longFieldMaxLength. If errorMessageMaxLength is not specified, truncation of error messages will now use the longFieldMaxLength value.

    Note that ultimately the maximum length of any tracing field is limited by the max_event_size configured for the receiving APM server.

    The fields affected by longFieldMaxLength are:

    • transaction.context.request.body, error.context.request.body - Before this change these fields were not truncated.
    • transaction.context.message.body, span.context.message.body, error.context.message.body - Before this change these fields were not truncated.
    • span.context.db.statement - Before this change this field was truncated at 10000 bytes. Truncation is now a number of unicode characters.
    • error.exception.message, error.log.message - Before this change, the default 2kB errorMessageMaxLength would apply.
  • Improve the TypeScript types by exporting more of interfaces: AgentConfigOptions, Transaction, Span, TransactionOptions, SpanOptions. (#2118)
Bug fixesedit
  • Fix a bug in apm.removePatch(module, aHandler) that would remove the last registered handler if aHandler did not match any currently registered handlers. (#2315)
  • Fix a crash in instrumentation of the old Elasticsearch client (elasticsearch) for some rarer cases of client options — for example passing multiple hosts. (#2312)
  • Ensure the internal HTTP(S) client requests made by the APM agent to APM server are not themselves traced. (#1168, #1136)
  • Fix crashing error with agent.registerMetric and active:false configuration. (#1799, #2290)

3.20.0 2021/08/12edit

Bug fixesedit
  • Fix failing tests and a possible runtime crash in @elastic/elasticsearch@7.14.0 instrumentation. (#2187)

3.19.0 2021/08/05edit

Featuresedit
  • The agent now supports the 3.x branch of apollo-server-express. (#2155)
  • Add instrumentation support for mongodb@4.x. (#2171)
Bug fixesedit
  • The agent will no longer report counting metrics with a value of zero, and will remove these metrics from the registry. (#2163)

3.18.0 2021/07/20edit

Featuresedit
  • Trace an incoming HTTP/1.1 request to an HTTP/2 server using the allowHTTP1 option. Before this change only incoming requests supporting HTTP/2 would be traced. (#2143)
  • Add instrumentation of the AWS SNS publish method when using the JavaScript AWS SDK v2 (aws-sdk). (#2157)
Bug fixesedit

3.17.0 2021/07/05edit

Featuresedit
  • Add instrumentation of all AWS S3 methods when using the JavaScript AWS SDK v2 (aws-sdk).
  • Add disableSend configuration option. This supports some use cases using the APM agent without an APM server. (#2101)
  • Add instrumentation of all DynamoDB methods when using the JavaScript AWS SDK v2 (aws-sdk).
Bug fixesedit
  • Fix inconsistencies in HTTP spans from other APM agents. span.subtype will now be "http" for HTTP, HTTPS, and HTTP/2 outgoing spans — previously it was "http", "https", "http2", respectively. As well, span.action will now be the HTTP method (e.g. "GET", "PUT", "POST"), rather than "http". (#2075)
  • Fixed error where SQS messages sent without an active transactions could crash the agent. (#2113)
  • Fixed support for proxies in destination context (#1770)

3.16.0 - 2021/06/14edit

Featuresedit
  • Added spanFramesMinDuration configuration field, allowing users to set a time threshold value that spans must reach before the agent will add a stack trace to the span. As a result, many short spans that previously included stack traces by default no longer will.
  • Prefer W3C "traceparent" header over "elastic-apm-traceparent" for incoming requests. #2079
Bug fixesedit
  • Fix a crash (TypeError: lastPrepareStackTrace) in the agent when used with React v17 and later (#1980).
  • Performance improvements have been made in error and stacktrace capture (#2094). This also included in two bug fixes:

    • Before this change, some captured errors (for example a next(new Error('boom') from an Express handler) would mark the error as "unhandled" incorrectly. "Unhandled" exceptions are those caught by an uncaughtException handler.
    • Before this change, source context lines for a stacktrace would not properly use the "sourcesContext" field of a file’s source map.

3.15.0 - 2021/05/19edit

Featuresedit
  • Add support for Node.js v16. (This also drops testing of Node.js v13 releases.) #2055
Bug fixesedit
  • Update TypeScript typings for Agent.setLabel and Agent.addLabels to include the stringify argument that was added in v3.11.0.

3.14.0 - 2021/04/19edit

Featuresedit
  • Add apm.addMetadataFilter(fn) for filtering the metadata object sent to APM server.
  • The handling of sending events (transactions, spans, errors) to APM server has improved in a few ways. During temporary spikes in load and/or an APM server that is unresponsive, the agent will buffer a number of events and drop them above a certain limit (configurable via maxQueueSize). This helps ensure the agent does not overly consume memory and CPU. As well, the agent will now backoff when the APM server errors. Finally, improved error handling means it will terminate failing requests to the APM server more quickly.

    Note: v1 of this agent (EOL’d 2 years ago), included a maxQueueSize config variable with a different meaning. If you have a lingering usage of that setting (also possibly as the ELASTIC_APM_MAX_QUEUE_SIZE environment variable), then it should be removed.

  • Adds support for Amazon SQS queues via aws-sdk instrumentation that partially implements the APM messaging spec, and adds queue.latency.min.ms, queue.latency.max.ms, and queue.latency.avg.ms metrics for SQS queues.
Bug fixesedit
  • Fixed bug where the URL property for outgoing HTTP request spans was set with the server’s IP address rather than its hostname. The Agent now sets this property with the actual URL requested by Node.js. #2035
  • Fixed bug where external services were not listed under Dependencies on the APM Service Overview page due to the trace-context propagated sample_rate value not being set on either transactions or spans.

3.13.0 - 2021/04/06edit

Featuresedit
  • The APM agent’s own internal logging now uses structured JSON logging using the pino API, and formatted in ecs-logging format. The log records on stdout are now single-line JSON objects. For example:

    # Before
    APM Server transport error (ECONNREFUSED): connect ECONNREFUSED 127.0.0.1:8200
    
    # After
    {"log.level":"error","@timestamp":"2021-03-19T00:21:17.571Z","log":{"logger":"elastic-apm-node"},
    "ecs":{"version":"1.6.0"},"message":"APM Server transport error (ECONNREFUSED): connect ECONNREFUSED 127.0.0.1:8200"}

    Pretty formatting (and filtering) on the console may be done via the ecslog tool.

    A custom logger is still supported as before. However, a non-pino custom logger will only receive the "message" field, and not structured log fields as they are added over time.

  • Add support for setting the ELASTIC_APM_LOGGER=false environment variable to disable/ignore a given custom logger. This is to support easier Debug mode for troubleshooting.
Bug fixesedit
  • Lock package dependency "elastic-apm-http-client@9.6.0" to avoid using v9.7.0 for now, because it is breaking tests. A coming release will get back on the latest of this dependency. #2032
  • Remove the "ancestors" field from a log.trace message on startup. Its info is a duplicate of info in the "startTrace" field in the same log record. #2005
  • Remove the accidental nodejs.eventloop.delay.ns metric that was always reporting a zero value. The existing nodejs.eventloop.delay.avg.ms is the intended metric. #1993

3.12.1 - 2021/02/25edit

Bug fixesedit

3.12.0 - 2021/02/21edit

Featuresedit
  • feat: Set span outcome to success or failure depending on whether an error was captured during when the span was active. #1814
  • feat: Adds public setOutcome method to span and transaction APIs, and adds a top level setTransactionOutcome and setSpanOutcome to set outcome values for the current active transaction or active span.
  • Limit the transactionSampleRate value to 4 decimal places of precision according to the shared APM spec. This ensures that propagated sampling rate in the tracestate header is short and consistent. #1979
Bug fixesedit
  • fix: It was possible for fetching central config to result in the cloudProvider config value being reset to its default. #1976
  • fix: fixes bug where tedious could crash the agent on bulk inserts #1935
    Reported via the forum. The error symptom was: Cannot read property 'statement' of undefined

3.11.0 - 2021/02/08edit

Featuresedit
  • feat: add apm.getServiceName() #1949
    This will be used by ecs-logging packages to integrate with APM.
  • feat: support numeric and boolean labels #1909
    Add an optional stringify option to apm.setLabel(name, version, stringify = true) and apm.addLabels(labels, stringify = true), which can be set false to allow numeric and boolean labels. Stringify defaults to true for backwards compatibility — stringification will be removed in a future major version.
  • feat: added support for cloud metadata fetching #1937
    Agent now collects information about its cloud environment and includes this data in the APM Server’s metadata payload. See the spec for more information.

3.10.0 - 2021/01/11edit

Featuresedit
  • feat: Improve handling of raw body parsing The agent will now report raw/Buffer encoded post bodies as <Buffer>.
  • feat: Add support for api keys #1818
    This allows the usage of API keys for authentication to the APM server
  • feat: Add automatic instrumentation of the @elastic/elasticsearch package #1870

    The instrumentation of the legacy "elasticsearch" package has also changed slightly to commonalize:

    • "span.context.destination" is set on all Elasticsearch spans, not just a subset of query-like API endpoints.
    • For query-like API endpoints (e.g. /_search), the capturing of query details on "span.context.db.statement" has changed (a) to include both the query params and the request body if both exist (separated by \n\n) and (b) to URL encode the query params, rather than JSON encoding.
  • feat: Add captureAttributes boolean option to apm.captureError() to allow disabling the automatic capture of Error object properties. This is useful for cases where those properties should not be sent to the APM Server, e.g. for performance (large string fields) or security (PII data). #1912
  • feat: Add log_level central config support. #1908
    Spec: https://github.com/elastic/apm/blob/master/specs/agents/logging.md
  • feat: Add sanitize_field_names configuration option.
    Allows users to configure a list of wildcard patterns to remove items from the agent’s HTTP header and application/x-www-form-urlencoded payloads. #1898

Bug fixesedit
  • fix: Fix parsing of comma-separated strings for relevant config vars to allow whitespace around the commas. E.g.:

    export ELASTIC_APM_TRANSACTION_IGNORE_URLS='/ping, /metrics*'

    Config vars affected are: disableInstrumentations, transactionIgnoreUrls addPatch, and globalLabels.

  • fix: Correct the environment variable for setting transactionIgnoreUrl (added in v3.9.0) from ELASTIC_TRANSACTION_IGNORE_URLS to ELASTIC_APM_TRANSACTION_IGNORE_URLS.

3.9.0 - 2020/11/30edit

Featuresedit
  • feat: support fastify 3 #1891
    Adds .default and .fastify module.exports to instrumented fastify function for 3.x line, and prefers req.routerMethod and req.routerPath for transaction name
  • feat: Set "destination" context on spans for "mongodb". #1893
    This allows Kibana APM Service Maps to show a "mongodb" node for services using the mongodb package (which includes mongoose and mongojs).
  • feat: transactionIgnoreUrl wildcard matching #1870
    Allows users to ignore URLs using simple wildcard matching patterns that behave the same across language agents. See https://github.com/elastic/apm/issues/144
Bug fixesedit
  • fix: treat set-cookie in response headers as sensitive data #1886
  • fix: Synchronous spans would never have span.sync == true. #1879

3.8.0 - 2020/11/09edit

Featuresedit
  • feat: expand k8s pod ID discovery regex #1863
  • feat: implements tracestate #1828
    Expands support for the W3C Trace Context specification by adding a tracestate header implementation, and uses this new header to track the Elastic transaction sample rate across a trace’s service boundaries.
  • feat: add span and transaction outcome #1824
    This adds an "outcome" field to HTTP(S) transactions and spans.
Bug fixesedit
  • fix(pg): prevent unhandled promise rejection #1846
  • fix: redis@2.x instrumentation was broken #1852
  • A number of fixes to the test suite.

3.7.0 - 2020/8/10edit

  • feat(knex): add support for 0.21.x #1801
  • feat(redis): add support for v3.x #1641
  • feat(graphql): add support for 15.x #1795
  • feat(koa-router): add support for 9.x #1772
  • fix(elasticsearch): ensure requests can be aborted #1566
  • fix: end span if outgoing http request ends prematurely #1583
  • fix: don’t throw on invalid URL #1771
  • fix: patch apollo-server-core > 2.14 correctly #1796
  • fix: add currentTraceIds to typings #1733

3.6.1 - 2020/5/20edit

  • fix(package): bump elastic-apm-http-client to ^9.4.0 #1756

3.6.0 - 2020/5/18edit

  • feat: add destination metadata for db spans #1687
  • feat: add support for Node.js 14 #1742
  • feat(pg): add support for pg v8.x #1743
  • feat: add metrics for external memory #1724
  • feat: enrich spans with destination info #1685
  • fix(instrumentation): add .js to module path #1711

3.5.0 - 2020/3/9edit

  • feat(error): get stack trace from Error-like objects #1613
  • fix: add logUncaughtExceptions conf option to TypeScript typings #1668

3.4.0 - 2020/2/21edit

  • feat: support W3C TraceContext traceparent header #1587
  • feat: add custom metrics API (experimental) #1571
  • feat(koa-router): add support for v8.x #1642
  • fix(cassandra): improve support for cassandra-driver v4.4.0+ #1636
  • fix: support promisifying setTimeout and friends #1636

3.3.0 - 2019/12/13edit

  • feat(config): add serverCaCertFile config #1560
  • feat(config): add central config support for transactionMaxSpans and captureBody #1555

3.2.0 - 2019/11/19edit

  • fix(metrics): only register collectors if enabled #1520
  • fix(ioredis): prevent unhandled promise rejection #1523
  • chore: add Node 13 to supported engines #1524

3.1.0 - 2019/10/16edit

Featuresedit
  • feat(mongodb): instrumentation #1423
  • fix(package): update elastic-apm-http-client to version 9.0.0 #1419
  • perf: cache ids value of transactions and spans #1434
Bug fixesedit
  • fix: always end transaction when socket is closed prematurely #1439
  • fix: change logUncaughtExceptions default to false #1432
  • fix: write stack trace of uncaught exceptions to STDERR #1429

3.0.0 - 2019/9/30edit

Breaking changesedit
  • feat: allow manual instrumentation with instrument: false #1114
  • feat: allow setting span/transaction type, subtype, and action separately (the behavior of the old type has changed) #1292
  • feat: use external as span type instead of ext #1291
  • refactor(graphql): use custom transaction type graphql for graphql requests instead of request #1245
  • feat(http): add instrumentIncomingHTTPRequests config (disableInstrumentations now behaves differently) #1298
  • chore: remove deprecated APIs #1413
  • chore: drop support for older Node.js versions #1383