Java Agent version 1.xedit

1.26.0 - 2021/09/14edit

Potentially breaking changesedit

  • If you rely on Database span subtype and use Microsoft SQL Server, the span subtype has been changed from sqlserver to mssql to align with other agents.
Breaking changesedit
  • Stop collecting the field http.request.socket.encrypted in http requests - #2136
  • Improved naming for Spring controllers - #1906
  • ECS log reformatting improvements - #1910

    • Automatically sets in all log events if set through agent configuration
    • Add log_ecs_reformatting_additional_fields option to support arbitrary fields in logs
    • Automatically serialize markers as tags where relevant (log4j2 and logback)
  • gRPC spans (client and server) can detect errors or cancellation through custom listeners - #2067
  • Add -download-agent-version to the agent attach CLI tool options, allowing the user to configure an arbitrary agent version that will be downloaded from maven and attached - #1959
  • Add extra check to detect improper agent setup - #2076
Performance improvementsedit
  • Reduce GC time overhead caused by WeakReferences - #2086, #2081
  • Reduced memory overhead by a smarter type pool caching strategy - #2102.
    The type pool cache improves the startup times by speeding up type matching (determining whether a class that’s about to be loaded should be instrumented). Generally, the more types that are cached, the faster the startup.
    The old strategy did not impose a limit to the cache but cleared it after it hasn’t been accessed in a while. However, load test have discovered that the cache may never be cleared and leave a permanent overhead of 23mb. The actual size of the cache highly depends on the application and loosely correlates with the number of loaded classes.
    The new caching strategy targets to allocate 1% of the committed heap, at least 0.5mb and max 10mb. If a particular entry hasn’t been accessed within 20s, it will be removed from the cache.
    The results based on load testing are very positive:

    • Equivalent startup times (within the margins of error of the previous strategy)
    • Equivalent allocation rate (within the margins of error of the previous strategy)
    • Reduced avg heap utilization from 10%/15mb (previous strategy) to within margins of error without the agent
    • Reduced GC time due to the additional headroom that the application can utilize.
    • Based on heap dump analysis, after warmup, the cache size is now around 59kb (down from 23mb with the previous strategy).
Bug fixesedit
  • Fix failure to parse some forms of the Implementation-Version property from jar manifest files - #1931
  • Ensure single value for context-propagation header - #1937
  • Fix gRPC non-terminated (therefore non-reported) client spans - #2067
  • Fix Webflux response status code - #1948
  • Ensure path filtering is applied when Servlet path is not available - #2099
  • Align span subtype for MS SqlServer - #2112
  • Fix potential destination host name corruption in OkHttp client spans - #2118

1.25.0 - 2021/07/22edit

Potentially breaking changesedit
  • If you rely on instrumentations that are in the experimental group, you must now set enable_experimental_instrumentations=true otherwise the experimental instrumentations will be disabled by default. Up to version 1.24.0 using an empty value for disable_instrumentations was the recommended way to override the default disable_instrumentations=experimental.
  • Support for inheritance of public API annotations - #1805
  • JDBC instrumentation sets context.db.instance - #1820
  • Add support for Vert.x web client- #1824
  • Avoid recycling of spans and transactions that are using through the public API, so to avoid reference-counting-related errors - #1859
  • Add enable_experimental_instrumentations ( [1.25.0] Added in 1.25.0. ) configuration option to enable experimental features - #1863

    • Previously, when adding an instrumentation group to disable_instrumentations, we had to make sure to not forget the default experimental value, for example when disabling jdbc instrumentation we had to set disable_instrumentations=experimental,jdbc otherwise setting disable_instrumentations=jdbc would disable jdbc and also enable experimental features, which would not be the desired effect.
    • Previously, by default disable_instrumentations contained experimental
    • Now by default disable_instrumentations is empty and enable_experimental_instrumentations=false
    • Set enable_experimental_instrumentations=true to enable experimental instrumentations
  • Eliminating concerns related to log4j2 vulnerability - We cannot upgrade to version above 2.12.1 because this is the last version of log4j that is compatible with Java 7. Instead, we exclude the SMTP appender (which is the vulnerable one) from our artifacts. Note that older versions of our agent are not vulnerable as well, as the SMTP appender was never used, this is only to further reduce our users' concerns.
  • Adding public APIs for setting destination.service.resource, destination.address and destination.port fields for exit spans - #1788
  • Only use emulated runtime attachment as fallback, remove the --without-emulated-attach option - #1865
  • Instrument javax.servlet.Filter the same way as javax.servlet.FilterChain - #1858
  • Propagate trace context headers in HTTP calls occurring from within traced exit points, for example - when using Elasticsearch’s REST client - #1883
  • Added support for naming sparkjava (not Apache Spark) transactions #1894
  • Added the ability to manually create exit spans, which will result with the auto creation of service nodes in the service map and downstream service in the dependencies table - #1898
  • Basic support for - #1854
  • Update to async-profiler 1.8.6 #1907
  • Added support for setting the framework using the public api (#1908) - #1909
Bug fixesedit
  • Fix NPE with null binary header values + properly serialize them - #1842
  • Fix ListenerExecutionFailedException when using Spring AMQP’s ReplyTo container - #1872
  • Enabling log ECS reformatting when using Logback configured with LayoutWrappingEncoder and a pattern layout - #1879
  • Fix NPE with Webflux + context propagation headers - #1871
  • Fix ClassCastException with ConnnectionMetaData and multiple classloaders - #1864
  • Fix NPE in co.elastic.apm.agent.servlet.helper.ServletTransactionCreationHelper.getClassloader - #1861
  • Fix for Jboss JMX unexpected notifications - #1895

1.24.0 - 2021/05/31edit

  • Basic support for Apache Struts 2 #1763
  • Extending the log_ecs_reformatting ( [1.22.0] Added in 1.22.0. experimental) config option to enable the overriding of logs with ECS-reformatted events. With the new OVERRIDE option, non-file logs can be ECS-reformatted automatically as well - #1793
  • Instrumentation for Vert.x Web #1697
  • Changed log level of vm arguments to debug
  • Giving precedence for the W3C tracecontext header over the elastic-apm-traceparent header - #1821
  • Add instrumentation for Webflux - #1305
  • Add instrumentation for Javalin #1822
Bug fixesedit
  • Fix another error related to instrumentation plugins loading on Windows - #1785
  • Load Spring AMQP plugin- #1784
  • Avoid IllegalStateException when multiple tracestate headers are used - #1808
  • Ensure CLI attach avoids sudo only when required and avoid blocking - #1819
  • Avoid sending metric-sets without samples, so to adhere to the intake API - #1826
  • Fixing our type-pool cache, so that it can’t cause OOM (softly-referenced), and it gets cleared when not used for a while - #1828
  • Remove single-package limitation for embedded plugins - #1780

1.23.0 - 2021/04/22edit

Breaking changesedit
  • There are breaking changes in the attacher cli. See the Features section for more information.
  • Overhaul of the attacher cli application that allows to attach the agent to running JVMs - #1667

    • The artifact of the standalone cli application is now called apm-agent-attach-cli. The attacher API is still called apm-agent-attach.
    • There is also a slim version of the cli application that does not bundle the Java agent. It requires the --agent-jar option to be set.
    • Improved logging
      The application uses Java ECS logging to emit JSON logs. The log level can be configured with the --log-level option. By default, the program is logging to the console but using the --log-file option, it can also log to a file.
    • Attach to JVMs running under a different user (unix only)
      The JVM requires the attacher to be running under the same user as the target VM (the attachee). The apm-agent-attach-standalone.jar can now be run with a user that has permissions to switch to the user that runs the target VM. On Windows, the attacher can still only attach to JVMs that are running with under the same user.
    • New include/exclude discovery rules

      • --include-all: Attach to all discovered JVMs. If no matchers are provided, it will not attach to any JVMs.
      • --include-user/--exclude-user: Attach to all JVMs of a given operating system user.
      • --include-main/--exclude-main: Attach to all JVMs that whose main class/jar name, or system properties match the provided regex.
      • --include-vmargs/--exclude-vmargs: Attach to all JVMs that whose main class/jar name, or system properties match the provided regex.
    • Removal of options

      • The deprecated --arg option has been removed.
      • The -i/--include, -e/exclude options have been removed in favor of the --<include|exclude>-<main|vmargs> options.
      • The -p/--pid options have been removed in favor of the --include-pid option.
    • Changed behavior of the -l/--list option
      The option now only lists JVMs that match the include/exclude discovery rules. Thus, it can be used to do a dry-run of the matchers without actually performing an attachment. It even works in combination with --continuous now. By default, the VM arguments are not printed, but only when the -a/--list-vmargs option is set.
    • Remove dependency on jps
      Even when matching on the main class name or on system properties,
    • Checks the Java version before attaching to avoid attachment on unsupported JVMs.
  • Cassandra instrumentation - #1712
  • Log correlation supports JBoss Logging - #1737
  • Update Byte-buddy to 1.11.0 - #1769
  • Support for user.domain #1756
  • JAX-RS supports
  • Enabling build and unit tests on Windows - #1671
Bug fixesedit
  • Fixed log correlation for log4j2 - #1720
  • Fix apm-log4j1-plugin and apm-log4j2-plugin dependency on slf4j - #1723
  • Avoid systematic MessageNotWriteableException error logging, now only visible in debug - #1715 and #1730
  • Fix rounded number format for non-english locales - #1728
  • Fix NullPointerException on legacy Apache client instrumentation when host is null - #1746
  • Apply consistent proxy class exclusion heuristic - #1738
  • Fix micrometer serialization error - #1741
  • Optimize & avoid ensureInstrumented deadlock by skipping stack-frame computation for Java7+ bytecode - #1758
  • Fix instrumentation plugins loading on Windows - #1671

1.22.0 - 2021/03/24edit

Breaking changesedit
  • Dots in metric names of Micrometer metrics get replaced with underscores to avoid mapping conflicts. De-dotting be disabled via dedot_custom_metrics. - #1700
  • Introducing a new mechanism to ease the development of community instrumentation plugins. See plugins_dir (experimental) for more details. This configuration was already added in 1.18.0, but more extensive and continuous integration testing allows us to expose it now. It is still marked as "experimental" though, meaning that future changes in the mechanism may break early contributed plugins. However, we highly encourage our community to try it out and we will do our best to assist with such efforts.
  • Deprecating ignore_user_agents in favour of transaction_ignore_user_agents, maintaining the same functionality - #1644
  • Update existing Hibernate Search 6 instrumentation to the final relase
  • The use_path_as_transaction_name option is now dynamic
  • Flushing internal and micrometer metrics before the agent shuts down - #1658
  • Support for OkHttp 4.4+ - #1672
  • Adding capability to automatically create ECS-JSON-formatted version of the original application log files, through the log_ecs_reformatting ( [1.22.0] Added in 1.22.0. experimental) config option. This allows effortless ingestion of logs to Elasticsearch without any further configuration. Supports log4j1, log4j2 and Logback. #1261
  • Add support to Spring AMQP - #1657
  • Adds the ability to automatically configure usage of the OpenTracing bridge in systems using ServiceLoader - #1708
  • Update to async-profiler 1.8.5 - includes a fix to a Java 7 crash and enhanced safe mode to better deal with corrupted stack frames.
  • Add a warning on startup when -Xverify:none or -noverify flags are set as this can lead to crashes that are very difficult to debug - #1593. In an upcoming version, the agent will not start when these flags are set, unless the system property elastic.apm.disable_bootstrap_checks is set to true.
Bug fixesedit
  • fix sample rate rounded to zero when lower than precision - #1655
  • fixed a couple of bugs with the external plugin mechanism (not documented until now) - #1660
  • Fix runtime attach conflict with multiple users - #1704

1.21.0 - 2021/02/09edit

Breaking changesedit
  • Following PR #1650, there are two slight changes with the server_url and server_urls configuration options:

    1. So far, setting server_urls with an empty string would allow the agent to work normally, apart from any action that requires communication with the APM Server, including the attempt to fetch a central configuration. Starting in this agent version, setting server_urls to empty string doesn’t have any special meaning, it is the default expected configuration, where server_url will be used instead. In order to achieve the same behaviour, use the new disable_send configuration.
    2. Up to this version, server_url was used as an alias to server_urls, meaning that one could potentially set the server_url config with a comma-separated list of multiple APM Server addresses, and that would have been a valid configuration. Starting in this agent version, server_url is a separate configuration, and it only accepts Strings that represent a single valid URL. Specifically, empty strings and commas are invalid.
  • Add cloud provider metadata to reported events, see spec for details. By default, the agent will try to automatically detect the cloud provider on startup, but this can be configured through the cloud_provider config option - #1599
  • Add span & transaction outcome field to improve error rate calculations - #1613
Bug fixesedit
  • Fixing crashes observed in Java 7 at sporadic timing by applying a few seconds delay on bootstrap - #1594
  • Fallback to using "TLS" SSLContext when "SSL" is not available - #1633
  • Fixing agent startup failure with NullPointerException thrown by Byte-buddy’s MultipleParentClassLoader - #1647
  • Fix cached type resolution triggering ClassCastException - #1649

1.20.0 - 2021/01/07edit

Breaking changesedit
  • The following public API types were public so far and became package-private: NoopScope, ScopeImpl and AbstractSpanImpl. If your code is using them, you will need to change that when upgrading to this version. Related PR: #1532
  • Add support for RabbitMQ clients - #1328
Bug fixesedit
  • Fix small memory allocation regression introduced with tracestate header #1508
  • Fix NullPointerException from WeakConcurrentMap.put through the Elasticsearch client instrumentation - #1531
  • Sending transaction_id and parent_id only for events that contain a valid trace_id as well - #1537
  • Fix ClassNotFoundError with old versions of Spring resttemplate #1524
  • Fix Micrometer-driven metrics validation errors by the APM Server when sending with illegal values - #1559
  • Serialize all stack trace frames when setting stack_trace_limit=-1 instead of none - #1571
  • Fix UnsupportedOperationException when calling ServletContext.getClassLoader() - #1576
  • Fix improper request body capturing - #1579
  • Avoid NullPointerException due to null return values instrumentation advices - #1601
  • Update async-profiler to 1.8.3 1602
  • Use null-safe data structures to avoid NullPointerException 1597
  • Fix memory leak in sampling profiler mechanism - #1592
  • Migrate some plugins to indy dispatcher #1405 #1394

1.19.0 - 2020/11/10edit

  • The agent version now includes a git hash if it’s a snapshot version. This makes it easier to differ distinct snapshot builds of the same version. Example: 1.18.1-SNAPSHOT.4655910
  • Add support for sampling weight with propagation in tracestate W3C header #1384
  • Adding two more valid options to the log_level config: WARNING (equivalent to WARN) and CRITICAL (will be treated as ERROR) - 1431
  • Add the ability to disable Servlet-related spans for INCLUDE, FORWARD and ERROR dispatches (without affecting basic Servlet capturing) by adding servlet-api-dispatch to disable_instrumentations ( [1.0.0] Added in 1.0.0. Changing this value at runtime is possible since version 1.15.0 ) - 1448
  • Add Sampling Profiler support for AArch64 architectures - 1443
  • Support proper transaction naming when using Spring’s ServletWrappingController - #1461
  • Update async-profiler to 1.8.2 1471
  • Update existing Hibernate Search 6 instrumentation to work with the latest CR1 release
  • Deprecating the addLabel public API in favor of setLabel (still supporting addLabel) - #1449
Bug fixesedit
  • Fix HttpUrlConnection instrumentation issue (affecting distributed tracing as well) when using HTTPS without using - 1447
  • Fixes class loading issue that can occur when deploying multiple applications to the same application server - #1458
  • Fix ability to disable agent on startup wasn’t working for runtime attach 1444
  • Avoid UnsupportedOperationException on some spring application startup 1464
  • Fix ignored runtime attach config_file 1469
  • Fix IllegalAccessError: Module 'java.base' no access to: package 'java.lang'... in J9 VMs of Java version >= 9 - #1468
  • Fix JVM version parsing on HP-UX #1477
  • Fix Spring-JMS transactions lifecycle management when using multiple concurrent consumers - #1496
  • Migrate some plugins to indy dispatcher 1404 1411
  • Replace System Rules with System Lambda #1434

1.18.1 - 2020/10/06edit

Bug fixesedit
  • Fix instrumentation error for HttpClient - #1402
  • Eliminate unsupported class version error messages related to loading the Java 11 HttpClient plugin in pre-Java-11 JVMs 1397
  • Fix rejected metric events by APM Server with response code 400 due to data validation error - sanitizing Micrometer metricset tag keys - 1413
  • Fix invalid micrometer metrics with non-numeric values 1419
  • Fix NoClassDefFoundError with JDBC instrumentation plugin 1409
  • Apply disable_metrics config to Micrometer metrics - 1421
  • Remove cgroup inactive_file.bytes metric according to spec 1422

1.18.0 - 2020/09/08edit

  • Deprecating ignore_urls config in favour of transaction_ignore_urls to align with other agents, while still allowing the old config name for backward compatibility - #1315
  • Enabling instrumentation of classes compiled with Java 1.4. This is reverting the restriction of instrumenting only bytecode of Java 1.5 or higher (#320), which was added due to potential VerifyError. Such errors should be avoided now by the usage of TypeConstantAdjustment - #1317
  • Enabling agent to work without attempting any communication with APM server, by allowing setting server_urls with an empty string - #1295
  • Add micrometer support - #1303
  • Add profiling_inferred_spans_lib_directory option to override the default temp directory used for exporting the async-profiler library. This is useful for server-hardened environments where /tmp is often configured with noexec, leading to java.lang.UnsatisfiedLinkError errors - #1350
  • Create spans for Servlet dispatches to FORWARD, INCLUDE and ERROR - #1212
  • Support JDK 11 HTTPClient - #1307
  • Lazily create profiler temporary files #1360
  • Convert the followings to Indy Plugins (see details in 1.18.0-rc1 relase notes): gRPC, AsyncHttpClient, Apache HttpClient
  • The agent now collects cgroup memory metrics (see details in Metrics page)
  • Update async-profiler to 1.8.1 #1382
  • Runtime attach install option is promoted to beta status (was experimental).
Bug fixesedit
  • Fixes a NoClassDefFoundError in the JMS instrumentation of MessageListener - #1287
  • Fix / by zero error message when setting server_urls with an empty string - #1295
  • Fix ClassNotFoundException or ClassCastException in some cases where special log4j configurations are used - #1322
  • Fix NumberFormatException when using early access Java version - #1325
  • Fix service_name config being ignored when set to the same auto-discovered default value - #1324
  • Fix service name error when updating a web app on a Servlet container - #1326
  • Fix remote attach jps executable not found when java binary is symlinked ot a JRE - #1352

1.18.0.RC1 - 2020/07/22edit

This release candidate adds some highly anticipated features: It’s now possible to attach the agent at runtime in more cases than before. Most notably, it enables runtime attachment on JBoss, WildFly, Glassfish/Payara, and other OSGi runtimes such as Atlassian Jira and Confluence.

To make this and other significant features, such as external plugins, possible, we have implemented major changes to the architecture of the agent. The agent now relies on the invokedynamic bytecode instruction to make plugin development easier, safer, and more efficient. As early versions of Java 7 and Java 8 have unreliable support for invokedynamic, we now require a minimum update level of 60 for Java 7 (7u60+) in addition to the existing minimum update level of 40 for Java 8 (8u40+).

We’re looking for users who would like to try this out to give feedback. If we see that the invokedynamic-based approach (indy plugins) works well, we can continue and migrate the rest of the plugins. After the migration has completed, we can move forward with external plugins and remove the experimental label from runtime attachment.

If all works like in our testing, you would not see NoClassDefFoundError s anymore when, for example, trying to attach the agent at runtime to an OSGi container or a JBoss server. Also, non-standard OSGi containers, such as Atlassian Jira and other technologies with restrictive class loading policies, such as MuleSoft ESB, will benefit from this change.

In the worst case, there might be JVM crashes due to invokedynamic-related JVM bugs. However, we already disable the agent when attached to JVM versions that are known to be problematic. Another potentially problematic area is that we now dynamically raise the bytecode version of instrumented classes to be at least bytecode version 51 (Java 7). This is needed in order to be able to use the invokedynamic instruction. This requires re-computation of stack map frames which makes instrumentation a bit slower. We don’t anticipate notable slowdowns unless you extensively (over-)use trace_methods.

Breaking changesedit
  • Early Java 7 versions, prior to update 60, are not supported anymore. When trying to attach to a non-supported version, the agent will disable itself and not apply any instrumentations.
  • Experimental support for runtime attachment now also for OSGi containers, JBoss, and WildFly
  • New mitigation of OSGi bootdelegation errors (NoClassDefFoundError). You can remove any org.osgi.framework.bootdelegation related configuration. This release also removes the configuration option boot_delegation_packages.
  • Overhaul of the ExecutorService instrumentation that avoids ClassCastException issues - #1206
  • Support for ForkJoinPool and ScheduledExecutorService (see Asynchronous frameworks)
  • Support for ExecutorService#invokeAny and ExecutorService#invokeAll
  • Added support for java.util.TimerTask - #1235
  • Add capturing of request body in Elasticsearch queries: _msearch, _count, _msearch/template, _search/template, _rollup_search - #1222
  • Add enabled flag
  • Add experimental support for Scala Futures
  • The agent now collects heap memory pools metrics - #1228
Bug fixesedit
  • Fixes error capturing for log4j2 loggers. Version 1.17.0 introduced a regression.
  • Fixes NullPointerException related to JAX-RS and Quartz instrumentation - #1249
  • Expanding k8s pod ID discovery to some formerly non-supported environments
  • When recording is set to false, the agent will not send captured errors anymore.
  • Fixes NPE in Dubbo instrumentation that occurs when the application is acting both as a provider and as a consumer - #1260
  • Adding a delay by default what attaching the agent to Tomcat using the premain route to work around the JUL deadlock issue - #1262
  • Fixes missing* MBeans on JBoss - #1257

1.17.0 - 2020/06/17edit

Bug fixesedit
  • Fixes IndexOutOfBoundsException that can occur when profiler-inferred spans are enabled. This also makes the profiler more resilient by just removing the call tree related to the exception (which might be in an invalid state) as opposed to stopping the profiler when an exception occurs.
  • Fix NumberFormatException when parsing Ingres/Actian JDBC connection strings - #1198
  • Prevent agent from overriding JVM configured truststore when not using HTTPS for communication with APM server - #1203
  • Fix java.lang.IllegalStateException with jps JVM when using continuous runtime attach - 1205
  • Fix agent trying to load log4j2 plugins from application - 1214
  • Fix memory leak in gRPC instrumentation plugin - 1196
  • Fix HTTPS connection failures when agent is configured to use HTTPS to communicate with APM server 1209

1.16.0 - 2020/05/13edit

  • The log correlation feature now adds to the MDC. See Logging frameworks for details. - #1050
  • Deprecating the incubating tag in favour of the experimental tag. This is not a breaking change, so former disable_instrumentation configuration containing the incubating tag will still be respected - #1123
  • Add a --without-emulated-attach option for runtime attachment to allow disabling this feature as a workaround.
  • Add workaround for JDK bug JDK-8236039 with TLS 1.3 #1149
  • Add log level OFF to silence agent logging
  • Adds span_min_duration option to exclude fast executing spans. When set together with one of the more specific thresholds - trace_methods_duration_threshold or profiling_inferred_spans_min_duration, the higher threshold will determine which spans will be discarded.
  • Automatically instrument quartz jobs from the quartz-jobs artifact #1170
  • Perform re-parenting of regular spans to be a child of profiler-inferred spans. Requires APM Server and Kibana 7.8.0. #1117
  • Upgrade Async Profiler version to 1.7.0
Bug fixesedit
  • When Servlet-related Exceptions are handled through exception handlers that return a 200 status code, agent shouldn’t override with 500 - #1103
  • Exclude Quartz 1 from instrumentation to avoid IncompatibleClassChangeError: Found class org.quartz.JobExecutionContext, but interface was expected - #1108
  • Fix breakdown metrics span sub-types #1113
  • Fix flaky gRPC server instrumentation #1122
  • Fix side effect of calling Statement.getUpdateCount more than once #1139
  • Stop capturing JDBC affected rows count using Statement.getUpdateCount to prevent unreliable side-effects #1147
  • Fix OpenTracing error tag handling (set transaction error result when tag value is true) #1159
  • Due to a bug in the build we didn’t include the gRPC plugin in the build so far
  • java.lang.ClassNotFoundException: Unable to load class 'jdk.internal...' is thrown when tracing specific versions of Atlassian systems #1168
  • Make sure spans are kept active during AsyncHandler methods in the AsyncHttpClient
  • CPU and memory metrics are sometimes not reported properly when using IBM J9 #1148
  • NullPointerException thrown by the agent on WebLogic #1142

1.15.0 - 2020/03/27edit

Breaking changesedit
  • Ordering of configuration sources has slightly changed, please review Configuration:

    • file now has higher priority over java system properties and environment variables,
      This change allows to change dynamic options values at runtime by editing file, previously values set in java properties or environment variables could not be overridden, even if they were dynamic.
  • Renamed some configuration options related to the experimental profiler-inferred spans feature (#1084):

    • profiling_spans_enabledprofiling_inferred_spans_enabled
    • profiling_sampling_intervalprofiling_inferred_spans_sampling_interval
    • profiling_spans_min_durationprofiling_inferred_spans_min_duration
    • profiling_included_classesprofiling_inferred_spans_included_classes
    • profiling_excluded_classesprofiling_inferred_spans_excluded_classes
    • Removed profiling_interval and profiling_duration (both are fixed to 5s now)
Bug fixesedit
  • When JAX-RS-annotated method delegates to another JAX-RS-annotated method, transaction name should include method A - #1062
  • Fixed bug that prevented an APM Error from being created when calling org.slf4j.Logger#error - #1049
  • Wrong address in JDBC spans for Oracle, MySQL and MariaDB when multiple hosts are configured - #1082
  • Document and re-order configuration priorities #1087
  • Improve heuristic for service_name when not set through config #1097

1.14.0 - 2020/03/04edit

  • Support for the official W3C traceparent and tracestate headers.
    The agent now accepts both the elastic-apm-traceparent and the official traceparent header. By default, it sends both headers on outgoing requests, unless use_elastic_traceparent_header is set to false.
  • Creating spans for slow methods with the help of the sampling profiler async-profiler. This is a low-overhead way of seeing which methods make your transactions slow and a replacement for the trace_methods configuration option. See Java method monitoring for more details
  • Adding a Circuit Breaker to pause the agent when stress is detected on the system and resume when the stress is relieved. See Circuit Breaker (experimental) and #1040 for more info.
  • Span#captureException and Transaction#captureException in public API return reported error id - #1015
Bug fixesedit
  • java.lang.IllegalStateException: Cannot resolve type description for <com.another.commercial.apm.agent.Class> - #1037
  • properly handle java.sql.SQLException for unsupported JDBC features #1035 #1025

1.13.0 - 2020/02/11edit

  • Add support for Redis Lettuce client
  • Add field for JMS message receiving spans and transactions - #970
  • Instrument log4j2 Logger#error(String, Throwable) (#919) Automatically captures exceptions when calling logger.error("message", exception)
  • Add instrumentation for external process execution through java.lang.Process and Apache commons-exec - #903
  • Add destination fields to exit span contexts - #976
  • Removed field - #993
  • Add support for Kafka clients - #981
  • Add support for binary traceparent header format (see the spec for more details) - #1009
  • Add support for log correlation for log4j and log4j2, even when not used in combination with slf4j. See Logging frameworks for details.
Bug Fixesedit
  • Fix parsing value of trace_methods configuration property #930
  • Workaround for java.util.logging deadlock #965
  • JMS should propagate traceparent header when transactions are not sampled #999
  • Spans are not closed if JDBC implementation does not support getUpdateCount #1008

1.12.0 - 2019/11/21edit

  • JMS Enhancements #911:

    • Add special handling for temporary queues/topics
    • Capture message bodies of text Messages

      • Rely on the existing ELASTIC_APM_CAPTURE_BODY agent config option (off by default).
      • Send as context.message.body
      • Limit size to 10000 characters. If longer than this size, trim to 9999 and append with ellipsis
    • Introduce the ignore_message_queues configuration to disable instrumentation (message tagging) for specific queues/topics as suggested in #710
    • Capture predefined message headers and all properties

      • Rely on the existing ELASTIC_APM_CAPTURE_HEADERS agent config option.
      • Send as context.message.headers
      • Sanitize sensitive headers/properties based on the sanitize_field_names config option
  • Added support for the MongoDB sync driver. See supported data stores.
Bug Fixesedit
  • JDBC regression- PreparedStatement#executeUpdate() and PreparedStatement#executeLargeUpdate() are not traced #918
  • When systemd cgroup driver is used, the discovered Kubernetes pod UID contains "_" instead of "-" #920
  • DB2 jcc4 driver is not traced properly #926

1.11.0 - 2019/10/31edit

  • Add the ability to configure a unique name for a JVM within a service through the service_node_name config option]
  • Add ability to ignore some exceptions to be reported as errors ignore_exceptions
  • Applying new logic for JMS javax.jms.MessageConsumer#receive so that, instead of the transaction created for the polling method itself (ie from receive start to end), the agent will create a transaction attempting to capture the code executed during actual message handling. This logic is suitable for environments where polling APIs are invoked within dedicated polling threads. This polling transaction creation strategy can be reversed through a configuration option (message_polling_transaction_strategy) that is not exposed in the properties file by default.
  • Send IP obtained through javax.servlet.ServletRequest#getRemoteAddr() in context.request.socket.remote_address instead of parsing from headers #889
  • Added ElasticApmAttacher.attach(String propertiesLocation) to specify a custom properties location
  • Logs message when transaction_max_spans has been exceeded #849
  • Report the number of affected rows by a SQL statement (UPDATE,DELETE,INSERT) in affected_rows span attribute #707
  • Add @Traced annotation which either creates a span or a transaction, depending on the context
  • Report JMS destination as a span/transaction context field #906
  • Added capture_jmx_metrics configuration option
Bug Fixesedit
  • JMS creates polling transactions even when the API invocations return without a message
  • Support registering MBeans which are added after agent startup

1.10.0 - 2019/09/30edit

  • Add ability to manually specify reported hostname
  • Add support for Redis Jedis client
  • Add support for identifying target JVM to attach apm agent to using JVM property. See also the documentation of the --include and --exclude flags
  • Added capture_jmx_metrics configuration option
  • Improve servlet error capture #812 Among others, now also takes Spring MVC `@ExceptionHandler`s into account
  • Instrument Logger#error(String, Throwable) #821 Automatically captures exceptions when calling logger.error("message", exception)
  • Easier log correlation with See docs.
  • Avoid creating a temp agent file for each attachment #859
  • Instrument View#render instead of DispatcherServlet#render #829 This makes the transaction breakdown graph more useful. Instead of dispatcher-servlet, the graph now shows a type which is based on the view name, for example, FreeMarker or Thymeleaf.
Bug Fixesedit
  • Error in log when setting server_urls to an empty string - co.elastic.apm.agent.configuration.ApmServerConfigurationSource - Expected previousException not to be null
  • Avoid terminating the TCP connection to APM Server when polling for configuration updates #823

1.9.0 - 2019/08/22edit

  • Upgrading supported OpenTracing version from 0.31 to 0.33
  • Added annotation and meta-annotation matching support for trace_methods, for example:

    • public @java.inject.* org.example.* (for annotation)
    • public @@javax.enterprise.context.NormalScope org.example.* (for meta-annotation)
  • The runtime attachment now also works when the tools.jar or the jdk.attach module is not available. This means you don’t need a full JDK installation - the JRE is sufficient. This makes the runtime attachment work in more environments such as minimal Docker containers. Note that the runtime attachment currently does not work for OSGi containers like those used in many application servers such as JBoss and WildFly. See the documentation for more information.
  • Support for Hibernate Search
Bug Fixesedit
  • A warning in logs saying APM server is not available when using 1.8 with APM server 6.x. Due to that, agent 1.8.0 will silently ignore non-string labels, even if used with APM server of versions 6.7.x or 6.8.x that support such. If APM server version is <6.7 or 7.0+, this should have no effect. Otherwise, upgrade the Java agent to 1.9.0+.
  • ApacheHttpAsyncClientInstrumentation matching increases startup time considerably
  • Log correlation feature is active when active==false
  • Tomcat’s memory leak prevention mechanism is causing a…​ memory leak. JDBC statement map is leaking in Tomcat if the application that first used it is undeployed/redeployed. See this related discussion.

Breaking Changesedit

  • The apm-agent-attach.jar is not executable anymore. Use apm-agent-attach-standalone.jar instead.

1.8.0 - 2019/07/30edit

  • Added support for tracking time spent by span type. Can be disabled by setting breakdown_metrics to false.
  • Added support for central configuration. Can be disabled by setting central_config to false.
  • Added support for Spring’s JMS flavor - instrumenting org.springframework.jms.listener.SessionAwareMessageListener
  • Added support to legacy ApacheHttpClient APIs (which adds support to Axis2 configured to use ApacheHttpClient)
  • Added support for setting server_urls dynamically via properties file #723
  • Added config_file option
  • Added option to use value as transaction name use_jaxrs_path_as_transaction_name
  • Instrument quartz jobs docs
  • SQL parsing improvements #696
  • Introduce priorities for transaction name #748. Now uses the path as transaction name if use_path_as_transaction_name is set to true rather than ServletClass#doGet. But if a name can be determined from a high level framework, like Spring MVC, that takes precedence. User-supplied names from the API always take precedence over any others.
  • Use JSP path name as transaction name as opposed to the generated servlet class name #751
Bug Fixesedit
  • Some JMS Consumers and Producers are filtered due to class name filtering in instrumentation matching
  • Jetty: When no display name is set and context path is "/" transaction service names will now correctly fall back to configured values
  • JDBC’s executeBatch is not traced
  • Drops non-String labels when connected to APM Server < 6.7 to avoid validation errors #687
  • Parsing container ID in cloud foundry garden #695
  • Automatic instrumentation should not override manual results #752
Breaking changesedit
  • The log correlation feature does not add to the MDC anymore but only and #742.

1.7.0 - 2019/06/13edit

  • Added the trace_methods_duration_threshold config option. When using the trace_methods config option with wild cards, this enables considerable reduction of overhead by limiting the number of spans captured and reported (see more details in config documentation). NOTE: Using wildcards is still not the recommended approach for the trace_methods feature.
  • Add Transaction#addCustomContext(String key, String|Number|boolean value) to public API
  • Added support for AsyncHttpClient 2.x
  • Added global_labels configuration option. This requires APM Server 7.2+.
  • Added basic support for JMS- distributed tracing for basic scenarios of send, receive, receiveNoWait and onMessage. Both Queues and Topics are supported. Async send APIs are not supported in this version. NOTE: This feature is currently marked as "experimental" and is disabled by default. In order to enable, it is required to set the disable_instrumentations configuration property to an empty string.
  • Improved OSGi support: added a configuration option for bootdelegation packages #641
  • Better span names for SQL spans. For example, SELECT FROM user instead of just SELECT #633
Bug Fixesedit
  • ClassCastException related to async instrumentation of Pilotfish Executor causing thread hang (applied workaround)
  • NullPointerException when computing Servlet transaction name with null HTTP method name
  • FileNotFoundException when trying to find implementation version of jar with encoded URL
  • NullPointerException when closing Apache AsyncHttpClient request producer
  • Fixes loading of for Spring Boot applications
  • Fix startup error on WebLogic #649
  • Disable metrics reporting and APM Server health check when active=false #653

1.6.1 - 2019/04/26edit

Bug Fixesedit
  • Fixes transaction name for non-sampled transactions #581
  • Makes log_file option work again #594
  • Async context propagation fixes

    • Fixing some async mechanisms lifecycle issues #605
    • Fixes exceptions when using WildFly managed executor services #589
    • Exclude glassfish Executor which does not permit wrapped runnables #596
    • Exclude DumbExecutor #598
  • Fixes Manifest version reading error to support jar:file protocol #601
  • Fixes transaction name for non-sampled transactions #597
  • Fixes potential classloader deadlock by preloading FileSystems.getDefault() #603

1.6.0 - 2019/04/16edit

Related Announcementsedit
  • Java APM Agent became part of the Cloud Foundry Java Buildpack as of Release v4.19
  • Support Apache HttpAsyncClient - span creation and cross-service trace context propagation
  • Added the jvm.thread.count metric, indicating the number of live threads in the JVM (daemon and non-daemon)
  • Added support for WebLogic
  • Added support for Spring @Scheduled and EJB @Schedule annotations - #569
Bug Fixesedit
  • Avoid that the agent blocks server shutdown in case the APM Server is not available - #554
  • Public API annotations improper retention prevents it from being used with Groovy - #567
  • Eliminate side effects of class loading related to Instrumentation matching mechanism

1.5.0 - 2019/03/26edit

Potentially breaking changesedit
  • If you didn’t explicitly set the service_name previously and you are dealing with a servlet-based application (including Spring Boot), your service_name will change. See the documentation for service_name and the corresponding section in Features for more information. Note: this requires APM Server 7.0+. If using previous versions, nothing will change.
  • Added property "allow_path_on_hierarchy" to JAX-RS plugin, to lookup inherited usage of @path
  • Support for number and boolean labels in the public API 497. This change also renames tag to label on the API level to be compliant with the Elastic Common Schema (ECS). The addTag(String, String) method is still supported but deprecated in favor of addLabel(String, String). As of version 7.x of the stack, labels will be stored under labels in Elasticsearch. Previously, they were stored under context.tags.
  • Support async queries made by Elasticsearch REST client
  • Added setStartTimestamp(long epochMicros) and end(long epochMicros) API methods to Span and Transaction, allowing to set custom start and end timestamps.
  • Auto-detection of the service_name based on the <display-name> element of the web.xml with a fallback to the servlet context path. If you are using a spring-based application, the agent will use the setting for for its service_name. See the documentation for service_name for more information. Note: this requires APM Server 7.0+. If using previous versions, nothing will change.
  • Previously, enabling capture_body could only capture form parameters. Now it supports all UTF-8 encoded plain-text content types. The option capture_body_content_types controls which `Content-Type`s should be captured.
  • Support async calls made by OkHttp client (Call#enqueue)
  • Added support for providing config options on agent attach.

    • CLI example: --config server_urls=http://localhost:8200,http://localhost:8201
    • API example: ElasticApmAttacher.attach(Map.of("server_urls", "http://localhost:8200,http://localhost:8201"));
Bug Fixesedit
  • Logging integration through MDC is not working properly - #499
  • ClassCastException with adoptopenjdk/openjdk11-openj9 - #505
  • Span count limitation is not working properly - reported in our forum
  • Java agent causes Exceptions in Alfresco cluster environment due to failure in the instrumentation of Hazelcast `Executor`s - reported in our forum

1.4.0 - 2019/02/14edit

  • Added support for sync calls of OkHttp client
  • Added support for context propagation for `java.util.concurrent.ExecutorService`s
  • The trace_methods configuration now allows to omit the method matcher. Example: com.example.* traces all classes and methods within the com.example package and sub-packages.
  • Added support for JSF. Tested on WildFly, WebSphere Liberty and Payara with embedded JSF implementation and on Tomcat and Jetty with MyFaces 2.2 and 2.3
  • Introduces a new configuration option disable_metrics which disables the collection of metrics via a wildcard expression.
  • Support for HttpUrlConnection
  • Adds subtype and action to spans. This replaces former typing mechanism where type, subtype and action were all set through the type in an hierarchical dotted-syntax. In order to support existing API usages, dotted types are parsed into subtype and action, however Span.createSpan and Span.setType are deprecated starting this version. Instead, type-less spans can be created using the new Span.startSpan API and typed spans can be created using the new Span.startSpan(String type, String subtype, String action) API
  • Support for JBoss EAP 6.4, 7.0, 7.1 and 7.2
  • Improved startup times
  • Support for SOAP (JAX-WS). SOAP client create spans and propagate context. Transactions are created for @WebService classes and @WebMethod methods.
Bug Fixesedit
  • Fixes a failure in BitBucket when agent deployed #349
  • Fixes increased CPU consumption #453 and #443
  • Fixed some OpenTracing bridge functionalities that were not working when auto-instrumentation is disabled
  • Fixed an error occurring when ending an OpenTracing span before deactivating
  • Sending proper null for metrics that have a NaN value
  • Fixes JVM crash with Java 7 #458
  • Fixes an application deployment failure when using EclipseLink and trace_methods configuration #474

1.3.0 - 2019/01/10edit

  • The agent now collects system and JVM metrics #360
  • Add API methods ElasticApm#startTransactionWithRemoteParent and Span#injectTraceHeaders to allow for manual context propagation #396.
  • Added trace_methods configuration option which lets you define which methods in your project or 3rd party libraries should be traced. To create spans for all public methods of classes whose name ends in Service which are in a sub-package of use this matcher: public*.*Service#* #398
  • Added span for DispatcherServlet#render #409.
  • Flush reporter on shutdown to make sure all recorded Spans are sent to the server before the program exits #397
  • Adds Kubernetes #383 and Docker metadata to, enabling correlation with the Kibana Infra UI.
  • Improved error handling of the Servlet Async API #399
  • Support async API’s used with AsyncContext.start #388
Bug Fixesedit
  • Fixing a potential memory leak when there is no connection with APM server
  • Fixes NoSuchMethodError CharBuffer.flip() which occurs when using the Elasticsearch RestClient and Java 7 or 8 #401

1.2.0 - 2018/12/19edit

  • Added capture_headers configuration option. Set to false to disable capturing request and response headers. This will reduce the allocation rate of the agent and can save you network bandwidth and disk space.
  • Makes the API methods addTag, setName, setType, setUser and setResult fluent, so that calls can be chained.
Bug Fixesedit
  • Catch all errors thrown within agent injected code
  • Enable public APIs and OpenTracing bridge to work properly in OSGi systems, fixes this WildFly issue
  • Remove to enable agent working on early Tomcat 8.5 versions
  • Fix async Servlet API issue

1.1.0 - 2018/11/28edit

  • Some memory allocation improvements
  • Enabling bootdelegation for agent classes in Atlassian OSGI systems
Bug Fixesedit
  • Update dsl-json which fixes a memory leak. See ngs-doo/dsl-json#102 for details.
  • Avoid `VerifyError`s by non instrumenting classes compiled for Java 4 or earlier
  • Enable APM Server URL configuration with path (fixes #339)
  • Reverse system.hostname and system.platform order sent to APM server

1.0.1 - 2018/11/15edit

Bug Fixesedit
  • Fixes NoSuchMethodError CharBuffer.flip() which occurs when using the Elasticsearch RestClient and Java 7 or 8 #313

1.0.0 - 2018/11/14edit

Breaking changesedit
  • Remove intake v1 support. This version requires APM Server 6.5.0+ which supports the intake api v2. Until the time the APM Server 6.5.0 is officially released, you can test with docker by pulling the APM Server image via docker pull
  • Adds @CaptureTransaction and @CaptureSpan annotations which let you declaratively add custom transactions and spans. Note that it is required to configure the application_packages for this to work. See the documentation for more information.
  • The public API now supports to activate a span on the current thread. This makes the span available via ElasticApm#currentSpan() Refer to the documentation for more details.
  • Capturing of Elasticsearch RestClient 5.0.2+ calls. Currently, the *Async methods are not supported, only their synchronous counterparts.
  • Added API methods to enable correlating the spans created from the JavaScrip Real User Monitoring agent with the Java agent transaction. More information can be found in the documentation.
  • Added Transaction.isSampled() and Span.isSampled() methods to the public API
  • Added Transaction#setResult to the public API #293
Bug Fixesedit
  • Fix for situations where status code is reported as 200, even though it actually was 500 #225
  • Capturing the username now properly works when using Spring security #183

1.0.0.RC1 - 2018/11/06edit

Breaking changesedit
  • Remove intake v1 support. This version requires APM Server 6.5.0+ which supports the intake api v2. Until the time the APM Server 6.5.0 is officially released, you can test with docker by pulling the APM Server image via docker pull
  • Wildcard patterns are case insensitive by default. Prepend (?-i) to make the matching case sensitive.
  • Support for Distributed Tracing
  • Adds @CaptureTransaction and @CaptureSpan annotations which let you declaratively add custom transactions and spans. Note that it is required to configure the application_packages for this to work. See the documentation for more information.
  • The public API now supports to activate a span on the current thread. This makes the span available via ElasticApm#currentSpan() Refer to the documentation for more details.
  • Capturing of Elasticsearch RestClient 5.0.2+ calls. Currently, the *Async methods are not supported, only their synchronous counterparts.
  • Added API methods to enable correlating the spans created from the JavaScrip Real User Monitoring agent with the Java agent transaction. More information can be found in the documentation.
  • Microsecond accurate timestamps #261
  • Support for JAX-RS annotations. Transactions are named based on your resources (ResourceClass#resourceMethod).
Bug Fixesedit
  • Fix for situations where status code is reported as 200, even though it actually was 500 #225