How we moved the OpenTelemetry Java agent to invokedynamic in 70+ pull requests

Elastic speaks OpenTelemetry natively. Send traces, logs, and metrics over OTLP straight into Elasticsearch, no proprietary agents required. See how it fits together, try it for free in the cloud, or run it locally.

The OpenTelemetry Java agent now delegates its instrumentation through invokedynamic instead of inlining bytecode into your methods. That took three years and over 70 pull requests, across more than 100 instrumentation modules. In July 2026 I closed the issue that had tracked the work since 2023.

Most users will notice nothing, which is the point, and anything you do notice is a bug. The difference is for maintainers and for anyone writing their own instrumentation. You get classloader isolation, and you can debug instrumentation in an IDE. Writing your own no longer means forking the agent or relying on its internals.

I did this migration once before, on the Elastic APM Java Agent, with a team of three. On paper, the plan the second time was simple:

1. Take the same approach we had proven in production with the Elastic APM Java Agent.

2. Convince OTel maintainers to adopt it.

3. Migrate most OTel instrumentation.

4. Declare victory and celebrate.

We wrote a blog post when we first started that work. This post is about everything between step two and step four.

What invokedynamic is and why it matters for Java agent instrumentation

If you’re interested in technical details, I recommend reading Embracing invokedynamic to tame class loaders in Java agents, the original blog post from Felix Barnsteiner and Rafael Winterhalter (Byte Buddy author).

For those who haven’t read it, here’s a short summary. Both the Elastic APM Java Agent and OTel Java instrumentation are built on Byte Buddy, a bytecode manipulation library that lets you modify application bytecode at runtime.

By default, Byte Buddy provides an inlining strategy; that is, the instrumentation injects bytecode directly into the target method. This is simple, but it trades off some flexibility. All classes referenced by the injected bytecode must be visible to the target class classloader, and we can't debug the injected code with common tools, like integrated development environments (IDEs). Plus, the initial driver was allowing end users to provide their own instrumentation without having to maintain a fork of the instrumentation agent or rely on its internal APIs.

What we internally call the "indy instrumentation" uses the invokedynamic Java Virtual Machine (JVM) bytecode instruction introduced in Java 7 to delegate instrumentation hooks without inlining the instrumentation code into the original method body.

Here is how the two instrumentation modes compare across the areas that matter in practice.

One major downside of this approach is that it requires modifying and reworking the instrumentation code. Another added challenge with instrumentation agents is that they’re deployed in thousands of production environments, so there’s a lot of pressure to avoid regressions and bugs, since they could cause dramatic crashes or performance regression in the application.

Although extensive testing is done before any release, the nature of instrumentation is that the end result is highly dependent on the target application. At best, we can validate with sample applications, but those will never cover all the edge cases that a real application and its deployment can have, such as proprietary frameworks and application servers, esoteric JVMs, OSGi, and complex class loading.

How we migrated the Elastic APM Java Agent with a team of three

At the time, there were only three engineers working on the Elastic APM Java Agent. Although that made it easier to coordinate and make decisions quickly, it also meant that we had very limited capacity to deal with support cases.

Because it's impossible and too risky to do a "big-bang" migration, we had to perform a gradual transition, moving only a few plugins at a time. One of the first lessons I learned was that even if you have limited capacity in a risk-averse environment, you can still make progress on large and complex changes; you just need to figure out how to apply changes incrementally. Also, doing so provides a natural safety net because the risk is spread across multiple releases.

A practical tool for this kind of task is what I call a "shopping list" of issues, so you can see the overall progress at a glance. We tracked our first migration here: #1337.

Although a few users systematically upgrade to the latest version whenever a new release is available, most upgrade organically over time, and this provides a natural incremental rollout of any new version. There’s a similar adoption pattern for most software and libraries. In rare cases, early adopters reported issues, but we fixed them quickly before the bulk of users upgraded.

What Log4Shell told us about the migration

However, we were in for a surprise. The Log4Shell critical vulnerability was disclosed in December 2021 (less than three weeks after the publication of our initial blog post), and it triggered a wave of emergency upgrades across the Java ecosystem. Unfortunately, the Elastic APM Java Agent was impacted, since log4j is used for logging in the agent itself, exploitable under a non-default configuration, as detailed in the security advisory.

If you were trying to upgrade log4j in your application at that time, you probably remember the chaos and the stress it caused. To me, it seemed like almost everyone was upgrading to the latest version of log4j and publishing new releases of their own software and libraries. As a consequence, the Maven central repository was under extreme load and most builds and releases failed due to timeouts.

Economists have the concept of natural experiments, where an unexpected anomaly in the real world is used to try out a hypothesis that can't be tested in a controlled environment. We don't have a direct equivalent, but this unintentional massive migration to the newer version of the Elastic APM Java Agent provided us a strong signal that the implementation was stable and without major flaws.

Retrospectively, this was a very stressful moment for the team. But when we look at the release changelog, we can see that we didn’t have any major issues that would have required more patch releases. As shown in the chart below, except for the 1.28.x releases with log4j updates, the number of patch releases remains very low and the release cadence remains stable:

Migrating the OpenTelemetry Java agent: what changed at scale

As part of Elastic’s contributions to OTel, we identified invokedynamic as something that could benefit OTel Java instrumentation.

This project includes more contributors and maintainers than the Elastic APM Java Agent did, so it’s naturally more challenging to coordinate and to convince others that it's worth the effort to migrate instrumentation for more than 100 supported technologies. Also, the OTel Java instrumentation is distributed and used by all observability vendors using OTel, so the impact of any regression is much larger.

The discussion started in July 2023, and the #8999 issue was opened to track the plan and progress of the migration. In the following weeks, my colleague Jonas Kunz wrote the initial proof-of-concept implementation and did most of the low-level work to support invokedynamic instrumentation and classloader isolation in the OTel Java instrumentation.

We faced some key challenges during this migration, some of which were similar to those with Elastic APM:

Migrating more than 100 instrumentation modules

The first is to handle a migration over a broad surface. More than 100 instrumentation modules needed to be reworked, as previously it wasn’t possible without a gradual migration and testing path.

We used multiple strategies and techniques to make this possible, including:

• Adding a runtime toggle to opt in to using invokedynamic instrumentation. This allowed local testing as well as continuous testing in continuous integration (CI) for months without impacting existing users.

• Transforming instrumentation code with Byte Buddy; for example, by setting inline = false on the @Advice annotation or making it compatible with both instrumentation modes through bytecode modification, doing all of that at runtime.

• Using large language models (LLMs) to help with mechanical code transformation in the final phases of the project.

All of this allowed testing and modifying the code over months with maximum safety. The main bottleneck remains testing and reviewing the changes.

Why classloader instrumentation stays inline

On a more technical level, some instrumentation modules are incompatible with invokedynamic and need to be kept inline; for example, with instrumentation of classloaders. The process of resolving the invokedynamic call site triggers classloading and thus can lead to a recursive resolution, which isn’t supported.

JPMS access when instrumentation moves to another module

Another interesting challenge is the compatibility with the Java Platform Module System (JPMS), introduced in Java 9. With inlined instrumentation, the instrumentation code inherits the same access as the instrumented code. When the instrumentation is instead delegated to another class in a different module and classloader, we need to grant access. Fortunately, fixing this was easy since we had already solved this in the Elastic APM Java Agent.

Injecting proxy classes into the application classloader

With a few instrumentation modules, we have to inject classes directly into the target classloader, when we need to make a class visible to the application code; for example, with the Spring framework. When the instrumentation code is no longer in the same classloader as the application code, we can't easily share a class between the two, because there’s no common classloader to load it from. eWe solved this by introducing proxy classes that are injected into the application classloader but that delegate to the instrumentation classloader, similar to what we have with Java Development Kit (JDK) proxies.

Instrumenting libraries compiled before Java 7

Last, but not least, my favorite challenge is making it work with Java versions prior to 7. Some libraries and frameworks have been compiled with older Java versions, long before the invokedynamic instruction was introduced. This is quite common in libraries that haven't been released in a long time or aren't maintained anymore. When the JVM validates such bytecode before loading it, it rejects any instruction that doesn’t match the version, even if the JVM supports it. You can solve this by upgrading the version of the application's bytecode before instrumenting it; in other words, we upgrade the bytecode to make it look like it was compiled with Java 7. 

Fortunately, the only tricky and costly operation is to recompute the stack map frames, since newer JVMs expect to know the types of everything stored on the stack. This technique is still used in the Elastic APM Java Agent, but OTel Java instrumentation has since replaced it with a different approach; when a class version doesn't support invokedynamic, it generates a helper class containing the invokedynamic instruction and invokes it via invokestatic, avoiding the need to bump the class version or recompute stack map frames at all (PR #15258).

What’s next for invokedynamic in the OpenTelemetry Java agent

OTel Java instrumentation is getting closer to the 3.0 release, in which invokedynamic instrumentation will become enabled by default for all supported modules. There are always aspects that can be improved, but the bulk of the work is done and we can consider this migration complete.

For example, in the current implementation, there are still parts of the OTel API included in the agent that are being shaded, meaning that they’re renamed at build time to prevent conflicts with another copy of the same library in the application classpath. Thanks to classloader isolation, we should be able to remove this for invokedynamic instrumentation like we did for the Elastic APM Java Agent, but doing so while preserving compatibility with inline instrumentation isn’t trivial. As usual, contributions are welcome, and I’d be happy to help anyone trying to solve this.

On a personal level, few projects that I've worked on combined this much technical depth with that amount of coordination, including classloader isolation, JPMS access rules, bytecode version tricks, and dozens of project participants, spread across two separate migrations that were three years apart. I’d like to thank the OTel maintainers and contributors, along with my colleagues at Elastic. None of this work would have been possible without them.