Forking the agent would fix this, but then you own every upstream merge. You could also use manual instrumentation for complete control, but that requires code changes and ongoing upkeep. OpenTelemetry Java extensions give you a cleaner path: a separate JAR the agent loads at startup, giving you precise control over what gets captured and exported, without touching agent or application code.

For example, the following challenges are very common:

• Health check probes are flooding your trace view.
• Span names reflect generic framework patterns rather than your domain operations.
• Some span names or attributes have high cardinality creating noise in your traces.
• Spans are missing attributes relevant to your business logic.
• Baggage headers are propagating to downstream services when they shouldn't.
• Resource attributes that describe your deployment are not automatically captured because they rely on custom environment variables.

Some of those can be solved through configuration, or by using an intermediate OpenTelemetry Collector for processing. However, this also might add complexity to the telemetry pipeline, and you might prefer to solve this at the source, where the data is captured.

What are OpenTelemetry Java extensions

An extension is a JAR file the agent loads at startup. It hooks into the agent's extension points through Java's Service Provider Interface (SPI) mechanism, the same mechanism the agent uses internally.

The extension mechanism works identically with the upstream OpenTelemetry Java agent and with Elastic's OpenTelemetry distribution. You write the extension once and it works with either.

For reference, the upstream extension documentation provides an exhaustive overview of extension points and a few examples.

This post does not aim to provide a complete reference, but focuses on simple use cases you're likely to reach for in production: renaming spans, filtering noisy traces, or propagating context that the agent doesn't cover in your environment.

Extensions also let you modify and extend the agent instrumentation itself. That goes beyond what this post covers. Here are two starting points:

• Modify instrumentation using instrumenter customizers.
• Modify or add instrumentation using the instrumentation module.

Setting up an OpenTelemetry Java extension project

An extension is a standard Java Gradle project with two requirements: the output must be a shadow JAR (a fat JAR with all extension dependencies bundled), and OpenTelemetry dependencies must be declared compileOnly so you don't bundle the SDK itself.

The shadow JAR requirement exists because the agent loads the extension in its own classloader. If you declare a dependency as implementation, it gets bundled and may conflict with the version already in the agent. Using compileOnly keeps those JARs out of the extension JAR entirely.

Here is a minimal build.gradle.kts for a simple extension that does not customize instrumentation and thus relies only on the OpenTelemetry SDK/API.

plugins {
  id("java")
  id("com.gradleup.shadow")
}

repositories {
  mavenCentral()
}

java {
  toolchain {
    languageVersion.set(JavaLanguageVersion.of(8))
  }
}

dependencies {
  // Use BOM to manage OpenTelemetry dependency versions
  compileOnly(platform("io.opentelemetry:opentelemetry-bom:1.64.0"))
  // OpenTelemetry SDK autoconfiguration SPI (provided by agent)
  compileOnly("io.opentelemetry:opentelemetry-sdk-extension-autoconfigure-spi")
  // OpenTelemetry SDK
  compileOnly("io.opentelemetry:opentelemetry-sdk")
  // Annotation processor for automatic SPI registration
  compileOnly("com.google.auto.service:auto-service:1.1.1")
  annotationProcessor("com.google.auto.service:auto-service:1.1.1")
}

tasks.assemble {
  dependsOn(tasks.shadowJar)
}

Check Maven Central for the latest version of the BOM before starting.

Extensions only depend at compile-time on the OpenTelemetry SDK and the autoconfiguration SPI. The agent provides the rest of the SDK and instrumentation implementation at runtime.

Loading OpenTelemetry Java extensions at runtime

To load an OpenTelemetry Java extension at runtime, you can use the otel.javaagent.extensions system property or OTEL_JAVAAGENT_EXTENSIONS environment variable. The value is a comma-separated list of paths to extension JARs:

java -Dotel.javaagent.extensions=/path/to/my-extension.jar -javaagent:/path/to/opentelemetry-javaagent.jar -jar myapp.jar

The upstream OpenTelemetry Java agent also lets you embed extensions directly into the agent JAR to simplify deployment.

Filtering and renaming spans with OpenTelemetry Java extensions

You can modify spans in two ways:

• Using a SpanProcessor that is called synchronously when the span starts or ends.
• Using a SpanExporter that is called asynchronously when the span is exported.

Rename spans with a SpanProcessor

SpanProcessor.onStart receives a ReadWriteSpan, which means you can call span.updateName() before the span is exported. This is the right hook for renaming based on attributes that are available at span start.

public class OperationRenamingSpanProcessor implements SpanProcessor {

  @Override
  public void onStart(Context parentContext, ReadWriteSpan span) {
    String operation = span.getAttribute(AttributeKey.stringKey("app.operation"));
    if (operation != null) {
      span.updateName(operation);
    }
  }

  @Override
  public boolean isStartRequired() { return true; }

  @Override
  public void onEnd(ReadableSpan span) {}

  @Override
  public boolean isEndRequired() { return false; }

  @Override
  public CompletableResultCode shutdown() { return CompletableResultCode.ofSuccess(); }

  @Override
  public CompletableResultCode forceFlush() { return CompletableResultCode.ofSuccess(); }
}

Register the SpanProcessor via AutoConfigurationCustomizerProvider, composing it with whatever processor you have already configured:

@AutoService(AutoConfigurationCustomizerProvider.class)
public class RenamingCustomizerProvider implements AutoConfigurationCustomizerProvider {

  @Override
  public void customize(AutoConfigurationCustomizer customizer) {
    customizer.addTracerProviderCustomizer(this::configureSdkTracerProvider);
  }

  private SdkTracerProviderBuilder configureSdkTracerProvider(
      SdkTracerProviderBuilder tracerProvider, ConfigProperties config) {
    return tracerProvider.addSpanProcessor(new OperationRenamingSpanProcessor());
  }

}

The modify-span EDOT Java extension example provides a complete implementation.

Filter spans with a SpanExporter

A SpanExporter wrapper lets you modify or drop spans before they leave the process. This works well for known noisy endpoints like health checks.

public class FilteringSpanExporter implements SpanExporter {

  private final SpanExporter delegate;

  public FilteringSpanExporter(SpanExporter delegate) {
    this.delegate = delegate;
  }

  @Override
  public CompletableResultCode export(Collection<SpanData> spans) {
    List<SpanData> filtered = new ArrayList<>();
    for (SpanData span : spans) {
      if (!"GET /health".equals(span.getName())) {
        filtered.add(span);
      }
    }
    return delegate.export(filtered);
  }

  @Override
  public CompletableResultCode flush() { return delegate.flush(); }

  @Override
  public CompletableResultCode shutdown() { return delegate.shutdown(); }
}

Register the FilteringSpanExporter via addSpanExporterCustomizer:

customizer.addSpanExporterCustomizer((existing, config) -> new FilteringSpanExporter(existing));

The modify-span EDOT Java extension example provides a complete implementation.

The approach has two limitations:

• This won't discard any child span that may have been created, for example, if the healthcheck calls the database.
• Spans filtered at the exporter have already passed through the full processor pipeline and occupied buffer space in the batch processor.

If you're dropping a large fraction of your traffic at this stage, a custom Sampler (shown below) is more efficient because it drops spans before any processing happens and also filters out child spans. Also, when using declarative configuration, the rule-based sampler lets you implement filtering on rules using only configuration.

Adding custom resource attributes with a ResourceProvider

Resource attributes describe what's running: the service name, its version, the host. A ResourceProvider lets you attach additional attributes that the agent doesn't know about, such as deployment metadata your platform injects through environment variables.

The example below uses environment variables, but it could also be a configuration file, a cloud metadata service, or any other source available to the agent at startup.

Because the SDK initialization is synchronous, when querying an external service like a metadata endpoint, this can make the agent (and thus the application) startup slower. If possible, prefer checking environment variables and local config first before calling an external service.

@AutoService(ResourceProvider.class)
public class DeploymentResourceProvider implements ResourceProvider {

  @Override
  public Resource createResource(ConfigProperties config) {
    AttributesBuilder attributes = Attributes.builder();

    String region = System.getenv("DEPLOY_REGION");
    if (region != null) {
      attributes.put(AttributeKey.stringKey("deployment.region"), region);
    }

    String buildVersion = System.getenv("BUILD_VERSION");
    if (buildVersion != null) {
      attributes.put(AttributeKey.stringKey("build.version"), buildVersion);
    }

    return Resource.create(attributes.build());
  }
}

Attributes from a ResourceProvider merge with the agent's own resource. When two providers supply the same key, the one with the higher order() value wins. The agent's built-in providers use order 0, so overriding order() to return a positive integer gives your provider priority.

The resource-attribute EDOT Java extension example provides a complete implementation.

Custom sampling in OpenTelemetry Java

When filtering at the exporter is too late or too expensive, implement a Sampler directly. The sampler runs before any span processing, so dropped spans never touch the batch buffer.

However, the sampling decision can only rely on attributes that are provided when the span starts. For example, the status code of an HTTP response can't be used as it is only available when the span ends.

The key detail: wrap the existing sampler rather than replacing it. That way, your logic composes with whatever you configured, and parent-based decisions from an upstream service are still respected.

public class HealthCheckSampler implements Sampler {

  private final Sampler delegate;

  public HealthCheckSampler(Sampler delegate) {
    this.delegate = delegate;
  }

  @Override
  public SamplingResult shouldSample(
      Context parentContext,
      String traceId,
      String name,
      SpanKind spanKind,
      Attributes attributes,
      List<LinkData> parentLinks) {
    if (spanKind == SpanKind.SERVER && name.contains("health")) {
      return SamplingResult.create(SamplingDecision.DROP);
    }
    return delegate.shouldSample(parentContext, traceId, name, spanKind, attributes, parentLinks);
  }

  @Override
  public String getDescription() {
    return "HealthCheckSampler{" + delegate.getDescription() + "}";
  }
}

Register the HealthCheckSampler via addSamplerCustomizer, which gives you both the existing sampler and the resolved config:

customizer.addSamplerCustomizer((existing, config) -> new HealthCheckSampler(existing));

Community extensions in opentelemetry-java-contrib

The opentelemetry-java-contrib repository contains several community-maintained extensions.

Some of them are already included in the OpenTelemetry Java agent (and inherited in the Elastic distribution), but are opt-in:

• azure-resources
• aws-resources
• gcp-resources
• cloudfoundry-resources
• baggage-processor

Most Elastic distribution features exist as extensions in the contrib repository, so you can use them with the upstream agent in a vendor-neutral way.

• inferred-spans
• span-stacktrace

Further reading and extension examples

The upstream extension examples cover additional extension points not shown here, including custom propagators, ID generators, and ignored-type configurers.

The Elastic baggage example shows the filtering propagator for baggage running end-to-end with a two-service application, it also demonstrates custom instrumentation to add baggage without modifying the application code.

This post covered the project setup and the patterns most likely to come up in production. Both links above go deeper: the upstream examples add extension points not covered here, and the baggage example shows a complete two-service implementation you can run locally.

When using the OpenTelemetry Java SDK or the OpenTelemetry instrumentation agent, those attributes are not automatically captured by default. In this article we will show you first how to enable them with the SDK, then using the instrumentation agent and then we will show you how using the Elastic OpenTelemetry Distribution makes it even easier.

OpenTelemetry Java SDK

The OpenTelemetry Java SDK does not capture any cloud resource attributes, however it provides a pluggable service provider interface to register resource attributes providers and application developers have to provide the implementations.

Implementations for GCP and AWS are already included in the OpenTelemetry Java Contrib repo, so if you are using one of those cloud providers then it's mostly a matter of adding those providers to your application dependencies. Thanks to autoconfiguration those should be automatically included and enabled once they are added to the application classpath. The SDK documentation provides all the details to add and configure those in your application.

If you are using a cloud provider for which no such implementation is available, then you still have the option to provide your own which is a straightforward implementation of the ResourceProvider SPI (Service Provider Interface). In order to keep things consistent, you will have to rely on the existing cloud semantic conventions.

For example here is an example of a simple cloud resource attributes provider for a fictitious cloud provider named "potatoes".

package potatoes;

import io.opentelemetry.api.common.Attributes;
import io.opentelemetry.sdk.autoconfigure.spi.ConfigProperties;
import io.opentelemetry.sdk.autoconfigure.spi.ResourceProvider;
import io.opentelemetry.sdk.resources.Resource;
import io.opentelemetry.semconv.incubating.CloudIncubatingAttributes;

public class PotatoesResourceProvider implements ResourceProvider {

@Override
public Resource createResource(ConfigProperties configProperties) {
   return Resource.create(Attributes.of(
           CloudIncubatingAttributes.CLOUD_PROVIDER, "potatoes",
           CloudIncubatingAttributes.CLOUD_PLATFORM, "french-fries",
           CloudIncubatingAttributes.CLOUD_REGION, "garden"
           ));
  }
}

OpenTelemetry Java instrumentation

The OpenTelemetry Java Instrumentation provides a java agent that instruments the application at runtime automatically for an extensive set of frameworks and libraries (see supported technologies).

Using instrumentation means that the application bytecode and the embedded libraries are modified automatically to make them behave as if explicit modifications were made in their source code to call the OpenTelemetry SDK in order to create traces, spans and metrics.

When an application is deployed with the OpenTelemetry instrumentation agent, the cloud resource attributes for GCP and AWS are included but not enabled by default since version 2.2.0. You can enable them through configuration by setting the following properties:

• For AWS: otel.resource.providers.aws.enabled=true

• For GCP: otel.resource.providers.gcp.enabled=true

Elastic OpenTelemetry Java Distribution

The Elastic OpenTelemetry Java distribution relies on the OpenTelemetry Java instrumentation which we often refer to as the Vanilla OpenTelemetry, and it thus inherits all of its features.

One major difference though is that the resource attributes providers for GCP and AWS are included and enabled by default to provide a better onboarding experience without extra configuration.

The minor cost to this is that it might make the application startup slightly slower due to having to call an HTTP(S) endpoint. This overhead is usually negligible compared to application startup but can become significant for some setups.

In order to reduce the startup overhead, or when the cloud provider is known in advance, you can selectively disable unused provider implementations through configuration:

• For AWS: otel.resource.providers.aws.enabled=false

• For GCP: otel.resource.providers.gcp.enabled=false

Conclusion

With this blogpost we have introduced what OpenTelemetry cloud resource attributes are and how they can be used and configured into application deployments using either OpenTelemetry SDK/API and Instrumentation agents.

When using the Elastic OpenTelemetry Java distribution, those resource providers are automatically provided and enabled for an easy and simple onboarding experience.

Another very interesting aspect of the cloud resource attribute providers available in the opentelemetry-java-contrib repository is that they are maintained by their respective vendors (Google and Amazon). For the end-user it means those implementations should be quite well tested and be robust to changes in the underlying infrastructure. For solution vendors like Elastic, it means we don't have to re-implement and reverse-engineer the infrastructure details of every cloud provider, hence proving that investing in those common components is a net win for the broader OpenTelemetry community.

In this blog, we’ll walk through a full example from scratch using a Java application. Apart from the Java code and using a JVM for the application, everything else works the same for the other languages supported by the Attacher.

Prerequisites

This walkthrough assumes that the following are already installed on the system: JDK 17, Docker, Kubernetes, and Helm.

The example application

While the application (shown below) is a Java application, it would be easily implemented in any language, as it is just a simple loop that every 2 seconds calls the method chain methodA->methodB->methodC->methodD, with methodC sleeping for 10 milliseconds and methodD sleeping for 200 milliseconds. The choice of application is just to be able to clearly display in the Elastic APM UI that the application is being monitored.

The Java application in full is shown here:

package test;

public class Testing implements Runnable {

  public static void main(String[] args) {
    new Thread(new Testing()).start();
  }

  public void run()
  {
    while(true) {
      try {Thread.sleep(2000);} catch (InterruptedException e) {}
      methodA();
    }
  }

  public void methodA() {methodB();}

  public void methodB() {methodC();}

  public void methodC() {
    System.out.println("methodC executed");
    try {Thread.sleep(10);} catch (InterruptedException e) {}
    methodD();
  }

  public void methodD() {
    System.out.println("methodD executed");
    try {Thread.sleep(200);} catch (InterruptedException e) {}
  }
}

We created a Docker image containing that simple Java application for you that can be pulled from the following Docker repository:

docker.elastic.co/demos/apm/k8s-webhook-test

Deploy the pod

First we need a deployment config. We’ll call the config file webhook-test.yaml, and the contents are pretty minimal — just pull the image and run that as a pod & container called webhook-test in the default namespace:

apiVersion: v1
kind: Pod
metadata:
  name: webhook-test
  labels:
    app: webhook-test
spec:
  containers:
    - image: docker.elastic.co/demos/apm/k8s-webhook-test
      imagePullPolicy: Always
      name: webhook-test

This can be deployed normally using kubectl:

kubectl apply -f webhook-test.yaml

The result is exactly as expected:

$ kubectl get pods
NAME           READY   STATUS    RESTARTS   AGE
webhook-test   1/1     Running   0          10s

$ kubectl logs webhook-test
methodC executed
methodD executed
methodC executed
methodD executed

So far, this is just setting up a standard Kubernetes application with no APM monitoring. Now we get to the interesting bit: adding in auto-instrumentation.

Install Elastic APM K8s Attacher

The first step is to install the Elastic APM K8s Attacher. This only needs to be done once for the cluster — once installed, it is always available. Before installation, we will define where the monitored data will go. As you will see later, we can decide or change this any time. For now, we’ll specify our own Elastic APM server, which is at https://myserver.somecloud:443 — we also have a secret token for authorization to that Elastic APM server, which has value MY_SECRET_TOKEN. (If you want to set up a quick test Elastic APM server, you can do so at https://cloud.elastic.co/).

There are two additional environment variables set for the application that are not generally needed but will help when we see the resulting UI content toward the end of the walkthrough (when the agent is auto-installed, these two variables tell the agent what name to give this application in the UI and what method to trace). Now we just need to define the custom yaml file to hold these. On installation, the custom yaml will be merged into the yaml for the Attacher:

apm:
  secret_token: MY_SECRET_TOKEN
  namespaces:
    - default
webhookConfig:
  agents:
    java:
      environment:
        ELASTIC_APM_SERVER_URL: "https://myserver.somecloud:443"
        ELASTIC_APM_TRACE_METHODS: "test.Testing#methodB"
        ELASTIC_APM_SERVICE_NAME: "webhook-test"

That custom.yaml file is all we need to install the attacher (note we’ve only specified the default namespace for agent auto-installation for now — this can be easily changed, as you’ll see later). Next we’ll add the Elastic charts to helm — this only needs to be done once, then all Elastic charts are available to helm. This is the usual helm add repo command, specifically:

helm repo add elastic https://helm.elastic.co

Now the Elastic charts are available for installation (helm search repo would show you all the available charts). We’re going to use “elastic-webhook” as the name to install into, resulting in the following installation command:

helm install elastic-webhook elastic/apm-attacher --namespace=elastic-apm --create-namespace --values custom.yaml

And that’s it, we now have the Elastic APM K8s Attacher installed and set to send data to the APM server defined in the custom.yaml file! (You can confirm installation with a helm list -A if you need.)

Auto-install the Java agent

The Elastic APM K8s Attacher is installed, but it doesn’t auto-install the APM application agents into every pod — that could lead to problems! Instead the Attacher is deliberately limited to auto-install agents into deployments defined a) by the namespaces listed in the custom.yaml, and b) to those deployments in those namespaces that have a specific annotation “co.elastic.apm/attach.”

So for now, restarting the webhook-test pod we created above won’t have any different effect on the pod, as it isn’t yet set to be monitored. What we need to do is add the annotation. Specifically, we need to add the annotation using the default agent configuration that was installed with the Attacher called “java” for the Java agent (we’ll see later how that agent configuration is altered — the default configuration installs the latest agent version and leaves everything else default for that version). So adding that annotation in to webhook-test yaml gives us the new yaml file contents (the additional config is shown labelled (1)):

apiVersion: v1
kind: Pod
metadata:
  name: webhook-test
  annotations: #(1)
    co.elastic.apm/attach: java #(1)
  labels:
    app: webhook-test
spec:
  containers:
    - image: docker.elastic.co/demos/apm/k8s-webhook-test
      imagePullPolicy: Always
      name: webhook-test

Applying this change gives us the application now monitored:

$ kubectl delete -f webhook-test.yaml
pod "webhook-test" deleted
$ kubectl apply -f webhook-test.yaml
pod/webhook-test created
$ kubectl logs webhook-test
… StartupInfo - Starting Elastic APM 1.45.0 …

And since the agent is now feeding data to our APM server, we can now see it in the UI:

Note that the agent identifies Testing.methodB method as a trace root because of the ELASTIC_APM_TRACE_METHODS environment variable set to test.Testing#methodB in the custom.yaml — this tells the agent to specifically trace that method. The time taken by that method will be available in the UI for each invocation, but we don’t see the sub-methods . . . currently. In the next section, we’ll see how easy it is to customize the Attacher, and in doing so we’ll see more detail about the method chain being executed in the application.

Customizing the agents

In your systems, you’ll likely have development, testing, and production environments. You’ll want to specify the version of the agent to use rather than just pull the latest version whatever that is, you’ll want to have debug on for some applications or instances, and you’ll want to have specific options set to specific values. This sounds like a lot of effort, but the attacher lets you enable these kinds of changes in a very simple way. In this section, we’ll add a configuration that specifies all these changes and we can see just how easy it is to configure and enable it.

We start at the custom.yaml file we defined above. This is the file that gets merged into the Attacher. Adding a new configuration with all the items listed in the last paragraph is easy — though first we need to decide a name for our new configuration. We’ll call it “java-interesting” here. The new custom.yaml in full is (the first part is just the same as before, the new config is simply appended):

apm:
  secret_token: MY_SECRET_TOKEN
  namespaces:
    - default
webhookConfig:
  agents:
    java:
      environment:
        ELASTIC_APM_SERVER_URL: "https://myserver.somecloud:443"
        ELASTIC_APM_TRACE_METHODS: "test.Testing#methodB"
        ELASTIC_APM_SERVICE_NAME: "webhook-test"
    java-interesting:
      image: docker.elastic.co/observability/apm-agent-java:1.55.4
      artifact: "/usr/agent/elastic-apm-agent.jar"
      environment:
        ELASTIC_APM_SERVER_URL: "https://myserver.somecloud:443"
        ELASTIC_APM_TRACE_METHODS: "test.Testing#methodB"
        ELASTIC_APM_SERVICE_NAME: "webhook-test"
        ELASTIC_APM_ENVIRONMENT: "testing"
        ELASTIC_APM_LOG_LEVEL: "debug"
        ELASTIC_APM_PROFILING_INFERRED_SPANS_ENABLED: "true"
        JAVA_TOOL_OPTIONS: "-javaagent:/elastic/apm/agent/elastic-apm-agent.jar"

Breaking the additional config down, we have:

• The name of the new config java-interesting

• The APM Java agent image docker.elastic.co/observability/apm-agent-java

• With a specific version 1.43.0 instead of latest

• We need to specify the agent jar location (the attacher puts it here)

• artifact: "/usr/agent/elastic-apm-agent.jar"

• And then the environment variables

• ELASTIC_APM_SERVER_URL as before

• ELASTIC_APM_ENVIRONMENT set to testing, useful when looking in the UI

• ELASTIC_APM_LOG_LEVEL set to debug for more detailed agent output

• ELASTIC_APM_PROFILING_INFERRED_SPANS_ENABLED turning this on (setting to true) will give us additional interesting information about the method chain being executed in the application

• And lastly we need to set JAVA_TOOL_OPTIONS to the enable starting the agent "-javaagent:/elastic/apm/agent/elastic-apm-agent.jar" — this is fundamentally how the attacher auto-attaches the Java agent

More configurations and details about configuration options are here for the Java agent, and other language agents are also available.

The application traced with the new configuration

And finally we just need to upgrade the attacher with the changed custom.yaml:

helm upgrade elastic-webhook elastic/apm-attacher --namespace=elastic-apm --create-namespace --values custom.yaml

This is the same command as the original install, but now using upgrade. That’s it — add config to the custom.yaml and upgrade the attacher, and it’s done! Simple.

Of course we still need to use the new config on an app. In this case, we’ll edit the existing webhook-test.yaml file, replacing java with java-interesting, so the annotation line is now:

co.elastic.apm/attach: java-interesting

Applying the new pod config and restarting the pod, you can see the logs now hold debug output:

$ kubectl delete -f webhook-test.yaml
pod "webhook-test" deleted
$ kubectl apply -f webhook-test.yaml
pod/webhook-test created
$ kubectl logs webhook-test
… StartupInfo - Starting Elastic APM 1.44.0 …
… DEBUG co.elastic.apm.agent. …
… DEBUG co.elastic.apm.agent. …

More interesting is the UI. Now that inferred spans is on, the full method chain is visible.

This gives the details for methodB (it takes 211 milliseconds because it calls methodC - 10ms - which calls methodD - 200ms). The times for methodC and methodD are inferred rather than recorded, (inferred rather than traced — if you needed accurate times you would instead add the methods to trace_methods and have them traced too).

Note on the ECK operator

The Elastic Cloud on Kubernetes operator allows you to install and manage a number of other Elastic components on Kubernetes. At the time of publication of this blog, the Elastic APM K8s Attacher is a separate component, and there is no conflict between these management mechanisms — they apply to different components and are independent of each other.

Try it yourself!

This walkthrough is easily repeated on your system, and you can make it more useful by replacing the example application with your own and the Docker registry with the one you use.

Learn more about real-time monitoring with Kubernetes and Elastic Observability.

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

The original instrumentation approach used in the OpenTelemetry Java agent left the maintenance and development of auto-instrumentation modules subject to some restrictions. As part of our reinforced commitment to OpenTelemetry, Elastic® helps evolve and improve OpenTelemetry projects and components. Elastic’s contribution of the Elastic Common Schema to OpenTelemetry was an important step for the open-source community. As another step in our commitment to OpenTelemetry, Elastic started contributing to the OpenTelemetry Java agent.

Elastic’s invokedynamic-based instrumentation approach

To overcome the above-mentioned limitations in developing and maintaining auto-instrumentation modules in the OpenTelemetry Java agent, Elastic started contributing its invokedynamic-based instrumentation approach to the OpenTelemetry Java agent in July 2023.

To explain the improvement, you should know that in Java, a common approach to do auto-instrumentation of applications is through utilizing Java agents that do bytecode instrumentation at runtime. Byte Buddy is a popular and widespread utility that helps with bytecode instrumentation without the need to deal with Java’s bytecode directly. Instrumentation logic that collects observability data from the target application’s code lives in so-called advice methods. Byte Buddy provides different ways of hooking these advice methods into the target application’s methods:

• Advice inlining: The advice method’s code is being copied into the instrumented target method.
• Static advice dispatching: The instrumented target method invokes static advice methods that need to be visible by the instrumented code.
• Advice dispatching with invokedynamic __:_ The instrumented target method uses the JVM’s invokedynamic bytecode instruction to call advice methods that are isolated from the instrumented code.

These different approaches are described in great detail in our related blog post on Elastic’s Java APM agent using invokedynamic. In a nutshell, both approaches, advice inlining and dispatching to static advice methods come with some limitations with respect to writing and maintaining the advice code. So far, the OpenTelemetry Java agent has used advice inlining for its bytecode instrumentation. The resulting limitations on developing instrumentations are documented in corresponding developer guidelines. Among other things, the limitation of not being able to debug advice code is a painful restriction when developing and maintaining instrumentation code.

Elastic’s APM Java agent has been using the invokedynamic approach with its benefits for years — field-proven by thousands of customers. To help improve the OpenTelemetry Java agent, Elastic started contributing the invokedynamic approach with the goal to simplify and improve the development and maintainability of auto-instrumentation modules. The contribution proposal and the implementation outline is documented in more detail in this GitHub issue.

With the new approach in place, Elastic will help migrate existing instrumentations so the OTel Java community can benefit from the invokedynamic -based instrumentation approach.

Elastic supports OTel natively, and has numerous capabilities to help you analyze your application with OTel. 

• Native OpenTelemetry support in Elastic Observability
• Best Practices for instrumenting OpenTelemetry
• Independence with OpenTelemetry on Elastic

Instrumenting with OpenTelemetry:

• Elastiflix application, a guide to instrument different languages with OpenTelemetry (this is the application the team built to highlight all the languages below)
• Python: Auto-instrumentation, Manual instrumentation
• Java: Auto-instrumentation, Manual instrumentation 
• Node.js: Auto-instrumentation, Manual instrumentation
• .NET: Auto-instrumentation, Manual instrumentation
  Go: Manual instrumentation

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.