How to change OpenTelemetry Java agent sampling at runtime

Until now, with some proprietary exceptions, configuring that telemetry had to be done at startup of the telemetry-generating component (e.g., SDK, agent, or OpenTelemetry Collector). Telemetry Policy is the emerging standard that will allow you to reconfigure some telemetry configurations without needing to restart the telemetry-generating component.

For example, let's say you want to:

• Change the trace sampling percentage to 1% during some special peak periods (eg special sales days), but keep it at 5% at all other times.
• Change the log level (e.g., to debug level or back to info) temporarily.
• Stop some instrumentations that are active.

These are all straightforward requirements. But applying any of them until now has meant changing the configuration of the component and restarting it. For a Java application (and applications in most languages) that previously meant also restarting the application, as there was no way to change the agent configuration without restarting the JVM.

Why central management needs one policy format

Central management adds another challenge. A controlling system that requests a change such as "increase log level to debug" must know how to apply that change to every component it controls. The configuration change for a collector will not look like the change for a Java SDK, even though both are applying the same decision.

Telemetry Policy addresses both problems by separating:

1. what we want to happen
2. from how a particular SDK, agent, or collector makes it happen.

Using the Elastic APM UI to set the Java agent sampling rate 0.1 (10%)

Elastic APM Java agent log statements confirming that the sampling rate value of 0.1 was received and applied

What is a Telemetry Policy in OpenTelemetry?

A Telemetry Policy is an independent rule describing the behaviour a user wants.

For example:

Sample traces at 5%

A policy that says to sample traces at 5% does not say which sampler to install, where to add a processor, or how the telemetry pipeline is laid out. Those are implementation details specific to each component. The policy only states the intended result.

The draft specification defines a policy as a typed, standalone object. Each policy has an ID, a name, and one target signal such as traces, logs, or metrics. The target determines the matching fields and actions that can be used.

Here is an example of that 5% sampling policy:

{
  "id": "sample-spans-5-percent",
  "name": "Sample traces at 5%",
  "trace": {
    "match": [
      {
        "exists": true
      }
    ],
    "keep": {
      "percentage": 5.0
    }
  }
}

The policy says to match every trace and apply 5% sampling using the policy with ID sample-spans-5-percent. Each telemetry component that understands that ID will be able to apply the policy. Any telemetry component that doesn't understand the policy will simply ignore it.

Which settings can Telemetry Policy change at runtime?

Policies are not expected to cover every configuration, or even many configurations of each component. Telemetry Policies are not a general-purpose "reconfigure anything" tool. Policies are specific to the things that users would most like to change without restarting. For application agents, our experience at Elastic suggests there are only a dozen or so important policy types that most customers need to be able to change at runtime without restarting the application, including:

• Changing the trace sampling percentage.
• Turning individual instrumentations or all instrumentations off and on.
• Turning individual signal exporters (traces, metrics, logs, profiles) off and on.
• Changing logging level.
• Changing the configuration of a select few instrumentations.
• Such as changing which HTTP routes to ignore when creating traces (eg ignore /heartbeat).
• Or changing which methods to instrument.

How a Telemetry Policy reaches an SDK, agent or collector

The overall telemetry policy pipeline has certain components, and at minimum covers the sequence:

Message -> Provider -> Policy -> Policy aggregator -> Implementer

A concrete example of changing the trace sampling percentage directed by central management using the Open Agent Management Protocol (OpAMP) helps to understand this flow:

| Stage | What it does | In the 5% sampling example | | --- | --- | --- | | Message | Carries the requested change into the component | An OpAMP message specifying a trace sampling percentage of 5%, like the JSON above | | Provider | Reads policies from an input stream | The OpAMP provider reads the message from the OpAMP data flow | | Policy | The internal representation of the change | A trace-sampling-percentage-policy with a target value of 5%, which is how the ID sample-spans-5-percent is implemented in the component* | | Policy aggregator | Combines the policy with others already applied or pending, handling source priority and merges | A pass-through, because there is only one provider | | Implementer | Applies the change to the running component | Installs a new sampler set to 5% |

Source priority decides which policy wins when two sources disagree. A message from an OpAMP source has higher priority than one from an HTTP source.

The implementer installs a new sampler unless the sampler already running can have its sampling percentage changed at runtime, in which case it updates that one.

*Note that the policy type may become a required part of the message in future. There is currently no standard mapping from a policy ID to a policy type, so the type must either be included in the message or supplied through an external mapping.

Delivering policies over OpAMP, HTTP or a local file

Telemetry Policy does not define a new transport mechanism or protocol. Policies can be delivered through existing transports:

• OpAMP.
• HTTP.
• Local files.
• Custom sources.

The same trace sampling policy could be read from a local file while testing, supplied over HTTP, or sent from a central management system using OpAMP. OpAMP is not a requirement for Telemetry Policy; it is a supported provider and is likely to be a common transport for centrally managed systems.

Telemetry Policy status in OpenTelemetry

Telemetry Policy is currently an accepted developing project. The schema, policy types, and implementation details are still being developed before they will become accepted OpenTelemetry standards.

The first experimental Java SDK implementation

The first experimental Java SDK implementation starts with dynamically changing the trace sampling probability. This is deliberately narrow, providing a simple practical way to test the complete policy flow.

Telemetry Policy separates observability decisions from the internal configuration of the component applying them. This allows a platform team to define a policy once and apply it consistently in SDKs, agents, and collectors.

In the next blog, I'll walk through the first experimental implementation in the OpenTelemetry Java contrib repo, including how it receives policies through OpAMP and changes trace sampling in a running Java application without requiring a restart.

In this second part, I show how to install any Java agent via the OpenTelemetry operator, using the Elastic Java agents as examples.

Installation and configuration recap

Part 1 of this series, Zero config OpenTelemetry auto-instrumentation for Kubernetes Java applications, details the installation and configuration of the OpenTelemetry operator and an Instrumentation resource. Here is an outline of the steps as a reminder:

1. Install cert-manager, eg kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.14.4/cert-manager.yaml
2. Install the operator, eg kubectl apply -f https://github.com/open-telemetry/opentelemetry-operator/releases/latest/download/opentelemetry-operator.yaml
3. Create an Instrumentation resource
4. Add an annotation to either the deployment or the namespace
5. Deploy the application as normal

In that first part, steps 3, 4 & 5 were implemented for the OpenTelemetry Java agent. In this blog I’ll implement them for other agents, using the Elastic APM agents as examples. I assume that steps 1 & 2 outlined above have already been done, ie that the operator is now installed. I will continue using the banana namespace for the examples, so ensure that namespace exists (kubectl create namespace banana). As per part 1, if you use any of the example instrumentation definitions below, you’ll need to substitute my.apm.server.url and my-apm-secret-token with the values appropriate for your collector.

Using the Elastic Distribution for OpenTelemetry Java

From version 0.4.0, the Elastic Distribution for OpenTelemetry Java includes the agent jar at the path /javaagent.jar in the docker image - which is essentially all that is needed for a docker image to be usable by the OpenTelemetry operator for auto-instrumentation. This means the Instrumentation resource is straightforward to define, and as it’s a distribution of the OpenTelemetry Java agent, all the OpenTelemetry environment can apply:

apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
  name: elastic-otel
  namespace: banana
spec:
  exporter:
    endpoint: https://my.apm.server.url
  propagators:
    - tracecontext
    - baggage
    - b3
  sampler:
    type: parentbased_traceidratio
    argument: "1.0"
  java:
    image: docker.elastic.co/observability/elastic-otel-javaagent:1.10.0
    env:
      - name: OTEL_EXPORTER_OTLP_HEADERS
        value: "Authorization=Bearer my-apm-secret-token"
      - name: ELASTIC_OTEL_INFERRED_SPANS_ENABLED
        value: "true"
      - name: ELASTIC_OTEL_SPAN_STACK_TRACE_MIN_DURATION
        value: "50"

I’ve included environment for switching on several features in the agent, including

1. ELASTIC_APM_PROFILING_INFERRED_SPANS_ENABLED to switch on the inferred spans implementation feature described in this blog
2. Span stack traces are automatically captured if the span takes more than ELASTIC_OTEL_SPAN_STACK_TRACE_MIN_DURATION (default would be 5ms)

Adding in the annotation …

metadata:
  annotations:
    instrumentation.opentelemetry.io/inject-java: "elastic-otel"

… to the pod yaml gets the application traced, and displayed in the Elastic APM UI, including the inferred child spans and stack traces

The additions from the features mentioned above are circled in red - inferred spans (for methodC and methodD) bottom left, and the stack trace top right. (Note that the pod included the OTEL_INSTRUMENTATION_METHODS_INCLUDE environment variable set to "test.Testing[methodB]" so that traces from methodB are shown; for pod configuration see the "Trying it" section in part 1)

Using the Elastic APM Java agent

From version 1.50.0, the Elastic APM Java agent includes the agent jar at the path /javaagent.jar in the docker image - which is essentially all that is needed for a docker image to be usable by the OpenTelemetry operator for auto-instrumentation. This means the Instrumentation resource is straightforward to define:

apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
  name: elastic-apm
  namespace: banana
spec:
  java:
    image: docker.elastic.co/observability/apm-agent-java:1.55.4
    env:
      - name: ELASTIC_APM_SERVER_URL
        value: "https://my.apm.server.url"
      - name: ELASTIC_APM_SECRET_TOKEN
        value: "my-apm-secret-token"
      - name: ELASTIC_APM_LOG_LEVEL
        value: "INFO"
      - name: ELASTIC_APM_PROFILING_INFERRED_SPANS_ENABLED
        value: "true"
      - name: ELASTIC_APM_LOG_SENDING
        value: "true"

I’ve included environment for switching on several features in the agent, including

• ELASTIC_APM_LOG_LEVEL set to the default value (INFO) which could easily be switched to DEBUG
• ELASTIC_APM_PROFILING_INFERRED_SPANS_ENABLED to switch on the inferred spans implementation equivalent to the feature described in this blog
• ELASTIC_APM_LOG_SENDING which switches on sending logs to the APM UI, the logs are automatically correlated with transactions (for all common logging frameworks)

Adding in the annotation …

metadata:
  annotations:
     instrumentation.opentelemetry.io/inject-java: "elastic-apm"

… to the pod yaml gets the application traced, and displayed in the Elastic APM UI, including the inferred child spans

(Note that the pod included the ELASTIC_APM_TRACE_METHODS environment variable set to "test.Testing#methodB" so that traces from methodB are shown; for pod configuration see the "Trying it" section in part 1)

Using an extension with the OpenTelemetry Java agent

Setting up an Instrumentation resource for the OpenTelemetry Java agent is straightforward and was done in part 1 of this two part series - and you can see from the above examples it’s just a matter of deciding on the docker image URL you want to use. However if you want to include an extension in your deployment, this is a little more complex, but also supported by the operator. Basically the extensions you want to include with the agent need to be in docker images - or you have to build an image which includes the extensions that are not already in images. Then you declare the images and the directories the extensions are in, in the Instrumentation resource. As an example, I’ll show an Instrumentation which uses version 2.5.0 of the OpenTelemetry Java agent together with the inferred spans extension from the Elastic OpenTelemetry Java distribution. The distro image includes the extension at path /extensions/elastic-otel-agentextension.jar. The Instrumentation resource allows either directories or file paths to be specified, here I’ll list the directory:

apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
  name: otel-plus-extension-instrumentation
  namespace: banana
spec:
  exporter:
    endpoint: https://my.apm.server.url
  propagators:
    - tracecontext
    - baggage
    - b3
  sampler:
    type: parentbased_traceidratio
    argument: "1.0"
  java:
    image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-java:2.5.0
    extensions:
      - image: "docker.elastic.co/observability/elastic-otel-javaagent:1.10.0"
        dir: "/extensions"
    env:
      - name: OTEL_EXPORTER_OTLP_HEADERS
        value: "Authorization=Bearer my-apm-secret-token"
      - name: ELASTIC_OTEL_INFERRED_SPANS_ENABLED
        value: "true"

Note that you can have multiple image … dir pairs, ie include multiple extensions from different images. Note also if you are testing this specific configuration that the inferred spans extension included here will be contributed to the OpenTelemetry contrib repo at some point after this blog is published, after which the extension may no longer be present in a later version of the referred image (since it will be available from the contrib repo instead).

Next steps

Here I’ve shown how to use any agent with the OpenTelemetry Operator for Kubernetes, and configure that for your system. In particular the examples have showcased how to use the Elastic Java agents to auto-instrument Java applications running in your Kubernetes clusters, along with how to enable features, using Instrumentation resources. And you can set it up for either zero config for deployments, or for just one annotation which is generally a more flexible mechanism (you can have multiple Instrumentation resource definitions, and the deployment can select the appropriate one for its application).

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

The mechanism to achieve zero-config auto-instrumentation of Java applications in Kubernetes is via the OpenTelemetry Operator for Kubernetes. This operator has many capabilities and the full documentation (and of course source) is available in the project itself. In this blog, I'll walk through installing, setting up and running zero-config auto-instrumentation of Java applications in Kubernetes using the OpenTelemetry Operator.

Installing the OpenTelemetry Operator

At the time of writing this blog, the OpenTelemetry Operator needs the certification manager to be installed, after which the operator can be installed. Installing from the web is straightforward. First install the cert-manager (the version to be installed will be specified in the OpenTelemetry Operator for Kubernetes documentation):

kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.14.4/cert-manager.yaml

Then when the cert managers are ready (kubectl get pods -n cert-manager)  …

NAMESPACE      NAME                                         READY
cert-manager   cert-manager-67c98b89c8-rnr5s                1/1
cert-manager   cert-manager-cainjector-5c5695d979-q9hxz     1/1
cert-manager   cert-manager-webhook-7f9f8648b9-8gxgs        1/1

… you can install the OpenTelemetry Operator:

kubectl apply -f https://github.com/open-telemetry/opentelemetry-operator/releases/latest/download/opentelemetry-operator.yaml

You can, of course, use a specific version of the operator instead of the latest. But here I’ve used the latest version.

An Instrumentation resource

Now you need to add just one further Kubernetes resource to enable auto-instrumentation: an Instrumentation resource. I am going to use the banana namespace for my examples, so I have first created that namespace (kubectl create namespace banana). The auto-instrumentation is specified and configured by these Instrumentation resources. Here is a basic one which will allow every Java pod in the banana namespace to be auto-instrumented with version 2.5.0 of the OpenTelemetry Java agent:

apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
  name: banana-instr
  namespace: banana
spec:
  exporter:
    endpoint: "https://my.endpoint"
  propagators:
    - tracecontext
    - baggage
    - b3
  sampler:
    type: parentbased_traceidratio
    argument: "1.0"
  java:
    image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-java:2.5.0
    env:
      - name: OTEL_EXPORTER_OTLP_HEADERS
        value: "Authorization=Bearer MyAuth"

Creating this resource (eg with kubectl apply -f banana-instr.yaml, assuming the above yaml was saved in file banana-instr.yaml) makes the banana-instr Instrumentation resource available for use. (Note you will need to change my.endpoint and MyAuth to values appropriate for your collector.) You can use this instrumentation immediately by adding an annotation to any deployment in the banana namespace:

metadata:
  annotations:
    instrumentation.opentelemetry.io/inject-java: "true"

The banana-instr Instrumentation resource is not yet set to be applied by default to all pods in the banana namespace. Currently it's zero-config as far as the application is concerned, but it requires an annotation added to a pod or deployment. To make it fully zero-config for all pods in the banana namespace, we need to add that annotation to the namespace itself, ie editing the namespace (kubectl edit namespace banana) so it would then have contents similar to

apiVersion: v1
kind: Namespace
metadata:
  name: banana
  annotations:
    instrumentation.opentelemetry.io/inject-java: "banana-instr"
...

Now we have a namespace that is going to auto-instrument every Java application deployed in the banana namespace with the 2.5.0 OpenTelemetry Java agent!

Trying it

There is a simple example Java application at docker.elastic.co/demos/apm/k8s-webhook-test which just repeatedly calls the chain main->methodA->methodB->methodC->methodD with some sleeps in the calls. Running this (kubectl apply -f banana-app.yaml) using a very basic pod definition:

apiVersion: v1
kind: Pod
metadata:
  name: banana-app
  namespace: banana
  labels:
    app: banana-app
spec:
  containers:
    - image: docker.elastic.co/demos/apm/k8s-webhook-test
      imagePullPolicy: Always
      name: banana-app
      env: 
      - name: OTEL_INSTRUMENTATION_METHODS_INCLUDE
        value: "test.Testing[methodB]"

results in the app being auto-instrumented with no configuration changes! The resulting app shows up in any APM UI, such as Elastic APM

As you can see, for this example I also added this env var to the pod yaml, OTEL_INSTRUMENTATION_METHODS_INCLUDE="test.Testing[methodB]" so that there were traces showing from methodB.

The technology behind the auto-instrumentation

To use the auto-instrumentation there is no specific need to understand the underlying mechanisms, but for those of you interested, here’s a quick outline.

1. The OpenTelemetry Operator for Kubernetes installs a mutating webhook, a standard Kubernetes component.
2. When deploying, Kubernetes first sends all definitions to the mutating webhook.
3. If the mutating webhook sees that the conditions for auto-instrumentation should be applied (ie
4. there is an Instrumentation resource for that namespace and
5. the correct annotation for that Instrumentation is applied to the definition in some way, either from the definition itself or from the namespace),
6. then the mutating webhook “mutates” the definition to include the environment defined by the Instrumentation resource.
7. The environment includes the explicit values defined in the env, as well as some implicit OpenTelemetry values (see the OpenTelemetry Operator for Kubernetes documentation for full details).
8. And most importantly, the operator
9. pulls the image defined in the Instrumentation resource,
10. extracts the file at the path /javaagent.jar from that image (using shell command cp)
11. inserts it into the pod at path /otel-auto-instrumentation-java/javaagent.jar
12. and adds the environment variable JAVA_TOOL_OPTIONS=-javaagent:/otel-auto-instrumentation-java/javaagent.jar.
13. The JVM automatically picks up that JAVA_TOOL_OPTIONS environment variable on startup and applies it to the JVM command-line.

Next steps

This walkthrough can be repeated in any Kubernetes cluster to demonstrate and experiment with auto-instrumentation (you will need to create the banana namespace first). In part 2 of this two part series, Using a custom agent with the OpenTelemetry Operator for Kubernetes, I show how to install any Java agent via the OpenTelemetry operator, using the Elastic Java agents as examples.

In this blog, we’ll walk through a full example from scratch using a Java application. Apart from the Java code and using a JVM for the application, everything else works the same for the other languages supported by the Attacher.

Prerequisites

This walkthrough assumes that the following are already installed on the system: JDK 17, Docker, Kubernetes, and Helm.

The example application

While the application (shown below) is a Java application, it would be easily implemented in any language, as it is just a simple loop that every 2 seconds calls the method chain methodA->methodB->methodC->methodD, with methodC sleeping for 10 milliseconds and methodD sleeping for 200 milliseconds. The choice of application is just to be able to clearly display in the Elastic APM UI that the application is being monitored.

The Java application in full is shown here:

package test;

public class Testing implements Runnable {

  public static void main(String[] args) {
    new Thread(new Testing()).start();
  }

  public void run()
  {
    while(true) {
      try {Thread.sleep(2000);} catch (InterruptedException e) {}
      methodA();
    }
  }

  public void methodA() {methodB();}

  public void methodB() {methodC();}

  public void methodC() {
    System.out.println("methodC executed");
    try {Thread.sleep(10);} catch (InterruptedException e) {}
    methodD();
  }

  public void methodD() {
    System.out.println("methodD executed");
    try {Thread.sleep(200);} catch (InterruptedException e) {}
  }
}

We created a Docker image containing that simple Java application for you that can be pulled from the following Docker repository:

docker.elastic.co/demos/apm/k8s-webhook-test

Deploy the pod

First we need a deployment config. We’ll call the config file webhook-test.yaml, and the contents are pretty minimal — just pull the image and run that as a pod & container called webhook-test in the default namespace:

apiVersion: v1
kind: Pod
metadata:
  name: webhook-test
  labels:
    app: webhook-test
spec:
  containers:
    - image: docker.elastic.co/demos/apm/k8s-webhook-test
      imagePullPolicy: Always
      name: webhook-test

This can be deployed normally using kubectl:

kubectl apply -f webhook-test.yaml

The result is exactly as expected:

$ kubectl get pods
NAME           READY   STATUS    RESTARTS   AGE
webhook-test   1/1     Running   0          10s

$ kubectl logs webhook-test
methodC executed
methodD executed
methodC executed
methodD executed

So far, this is just setting up a standard Kubernetes application with no APM monitoring. Now we get to the interesting bit: adding in auto-instrumentation.

Install Elastic APM K8s Attacher

The first step is to install the Elastic APM K8s Attacher. This only needs to be done once for the cluster — once installed, it is always available. Before installation, we will define where the monitored data will go. As you will see later, we can decide or change this any time. For now, we’ll specify our own Elastic APM server, which is at https://myserver.somecloud:443 — we also have a secret token for authorization to that Elastic APM server, which has value MY_SECRET_TOKEN. (If you want to set up a quick test Elastic APM server, you can do so at https://cloud.elastic.co/).

There are two additional environment variables set for the application that are not generally needed but will help when we see the resulting UI content toward the end of the walkthrough (when the agent is auto-installed, these two variables tell the agent what name to give this application in the UI and what method to trace). Now we just need to define the custom yaml file to hold these. On installation, the custom yaml will be merged into the yaml for the Attacher:

apm:
  secret_token: MY_SECRET_TOKEN
  namespaces:
    - default
webhookConfig:
  agents:
    java:
      environment:
        ELASTIC_APM_SERVER_URL: "https://myserver.somecloud:443"
        ELASTIC_APM_TRACE_METHODS: "test.Testing#methodB"
        ELASTIC_APM_SERVICE_NAME: "webhook-test"

That custom.yaml file is all we need to install the attacher (note we’ve only specified the default namespace for agent auto-installation for now — this can be easily changed, as you’ll see later). Next we’ll add the Elastic charts to helm — this only needs to be done once, then all Elastic charts are available to helm. This is the usual helm add repo command, specifically:

helm repo add elastic https://helm.elastic.co

Now the Elastic charts are available for installation (helm search repo would show you all the available charts). We’re going to use “elastic-webhook” as the name to install into, resulting in the following installation command:

helm install elastic-webhook elastic/apm-attacher --namespace=elastic-apm --create-namespace --values custom.yaml

And that’s it, we now have the Elastic APM K8s Attacher installed and set to send data to the APM server defined in the custom.yaml file! (You can confirm installation with a helm list -A if you need.)

Auto-install the Java agent

The Elastic APM K8s Attacher is installed, but it doesn’t auto-install the APM application agents into every pod — that could lead to problems! Instead the Attacher is deliberately limited to auto-install agents into deployments defined a) by the namespaces listed in the custom.yaml, and b) to those deployments in those namespaces that have a specific annotation “co.elastic.apm/attach.”

So for now, restarting the webhook-test pod we created above won’t have any different effect on the pod, as it isn’t yet set to be monitored. What we need to do is add the annotation. Specifically, we need to add the annotation using the default agent configuration that was installed with the Attacher called “java” for the Java agent (we’ll see later how that agent configuration is altered — the default configuration installs the latest agent version and leaves everything else default for that version). So adding that annotation in to webhook-test yaml gives us the new yaml file contents (the additional config is shown labelled (1)):

apiVersion: v1
kind: Pod
metadata:
  name: webhook-test
  annotations: #(1)
    co.elastic.apm/attach: java #(1)
  labels:
    app: webhook-test
spec:
  containers:
    - image: docker.elastic.co/demos/apm/k8s-webhook-test
      imagePullPolicy: Always
      name: webhook-test

Applying this change gives us the application now monitored:

$ kubectl delete -f webhook-test.yaml
pod "webhook-test" deleted
$ kubectl apply -f webhook-test.yaml
pod/webhook-test created
$ kubectl logs webhook-test
… StartupInfo - Starting Elastic APM 1.45.0 …

And since the agent is now feeding data to our APM server, we can now see it in the UI:

Note that the agent identifies Testing.methodB method as a trace root because of the ELASTIC_APM_TRACE_METHODS environment variable set to test.Testing#methodB in the custom.yaml — this tells the agent to specifically trace that method. The time taken by that method will be available in the UI for each invocation, but we don’t see the sub-methods . . . currently. In the next section, we’ll see how easy it is to customize the Attacher, and in doing so we’ll see more detail about the method chain being executed in the application.

Customizing the agents

In your systems, you’ll likely have development, testing, and production environments. You’ll want to specify the version of the agent to use rather than just pull the latest version whatever that is, you’ll want to have debug on for some applications or instances, and you’ll want to have specific options set to specific values. This sounds like a lot of effort, but the attacher lets you enable these kinds of changes in a very simple way. In this section, we’ll add a configuration that specifies all these changes and we can see just how easy it is to configure and enable it.

We start at the custom.yaml file we defined above. This is the file that gets merged into the Attacher. Adding a new configuration with all the items listed in the last paragraph is easy — though first we need to decide a name for our new configuration. We’ll call it “java-interesting” here. The new custom.yaml in full is (the first part is just the same as before, the new config is simply appended):

apm:
  secret_token: MY_SECRET_TOKEN
  namespaces:
    - default
webhookConfig:
  agents:
    java:
      environment:
        ELASTIC_APM_SERVER_URL: "https://myserver.somecloud:443"
        ELASTIC_APM_TRACE_METHODS: "test.Testing#methodB"
        ELASTIC_APM_SERVICE_NAME: "webhook-test"
    java-interesting:
      image: docker.elastic.co/observability/apm-agent-java:1.55.4
      artifact: "/usr/agent/elastic-apm-agent.jar"
      environment:
        ELASTIC_APM_SERVER_URL: "https://myserver.somecloud:443"
        ELASTIC_APM_TRACE_METHODS: "test.Testing#methodB"
        ELASTIC_APM_SERVICE_NAME: "webhook-test"
        ELASTIC_APM_ENVIRONMENT: "testing"
        ELASTIC_APM_LOG_LEVEL: "debug"
        ELASTIC_APM_PROFILING_INFERRED_SPANS_ENABLED: "true"
        JAVA_TOOL_OPTIONS: "-javaagent:/elastic/apm/agent/elastic-apm-agent.jar"

Breaking the additional config down, we have:

• The name of the new config java-interesting

• The APM Java agent image docker.elastic.co/observability/apm-agent-java

• With a specific version 1.43.0 instead of latest

• We need to specify the agent jar location (the attacher puts it here)

• artifact: "/usr/agent/elastic-apm-agent.jar"

• And then the environment variables

• ELASTIC_APM_SERVER_URL as before

• ELASTIC_APM_ENVIRONMENT set to testing, useful when looking in the UI

• ELASTIC_APM_LOG_LEVEL set to debug for more detailed agent output

• ELASTIC_APM_PROFILING_INFERRED_SPANS_ENABLED turning this on (setting to true) will give us additional interesting information about the method chain being executed in the application

• And lastly we need to set JAVA_TOOL_OPTIONS to the enable starting the agent "-javaagent:/elastic/apm/agent/elastic-apm-agent.jar" — this is fundamentally how the attacher auto-attaches the Java agent

More configurations and details about configuration options are here for the Java agent, and other language agents are also available.

The application traced with the new configuration

And finally we just need to upgrade the attacher with the changed custom.yaml:

helm upgrade elastic-webhook elastic/apm-attacher --namespace=elastic-apm --create-namespace --values custom.yaml

This is the same command as the original install, but now using upgrade. That’s it — add config to the custom.yaml and upgrade the attacher, and it’s done! Simple.

Of course we still need to use the new config on an app. In this case, we’ll edit the existing webhook-test.yaml file, replacing java with java-interesting, so the annotation line is now:

co.elastic.apm/attach: java-interesting

Applying the new pod config and restarting the pod, you can see the logs now hold debug output:

$ kubectl delete -f webhook-test.yaml
pod "webhook-test" deleted
$ kubectl apply -f webhook-test.yaml
pod/webhook-test created
$ kubectl logs webhook-test
… StartupInfo - Starting Elastic APM 1.44.0 …
… DEBUG co.elastic.apm.agent. …
… DEBUG co.elastic.apm.agent. …

More interesting is the UI. Now that inferred spans is on, the full method chain is visible.

This gives the details for methodB (it takes 211 milliseconds because it calls methodC - 10ms - which calls methodD - 200ms). The times for methodC and methodD are inferred rather than recorded, (inferred rather than traced — if you needed accurate times you would instead add the methods to trace_methods and have them traced too).

Note on the ECK operator

The Elastic Cloud on Kubernetes operator allows you to install and manage a number of other Elastic components on Kubernetes. At the time of publication of this blog, the Elastic APM K8s Attacher is a separate component, and there is no conflict between these management mechanisms — they apply to different components and are independent of each other.

Try it yourself!

This walkthrough is easily repeated on your system, and you can make it more useful by replacing the example application with your own and the Docker registry with the one you use.

Learn more about real-time monitoring with Kubernetes and Elastic Observability.

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

The original instrumentation approach used in the OpenTelemetry Java agent left the maintenance and development of auto-instrumentation modules subject to some restrictions. As part of our reinforced commitment to OpenTelemetry, Elastic® helps evolve and improve OpenTelemetry projects and components. Elastic’s contribution of the Elastic Common Schema to OpenTelemetry was an important step for the open-source community. As another step in our commitment to OpenTelemetry, Elastic started contributing to the OpenTelemetry Java agent.

Elastic’s invokedynamic-based instrumentation approach

To overcome the above-mentioned limitations in developing and maintaining auto-instrumentation modules in the OpenTelemetry Java agent, Elastic started contributing its invokedynamic-based instrumentation approach to the OpenTelemetry Java agent in July 2023.

To explain the improvement, you should know that in Java, a common approach to do auto-instrumentation of applications is through utilizing Java agents that do bytecode instrumentation at runtime. Byte Buddy is a popular and widespread utility that helps with bytecode instrumentation without the need to deal with Java’s bytecode directly. Instrumentation logic that collects observability data from the target application’s code lives in so-called advice methods. Byte Buddy provides different ways of hooking these advice methods into the target application’s methods:

• Advice inlining: The advice method’s code is being copied into the instrumented target method.
• Static advice dispatching: The instrumented target method invokes static advice methods that need to be visible by the instrumented code.
• Advice dispatching with invokedynamic __:_ The instrumented target method uses the JVM’s invokedynamic bytecode instruction to call advice methods that are isolated from the instrumented code.

These different approaches are described in great detail in our related blog post on Elastic’s Java APM agent using invokedynamic. In a nutshell, both approaches, advice inlining and dispatching to static advice methods come with some limitations with respect to writing and maintaining the advice code. So far, the OpenTelemetry Java agent has used advice inlining for its bytecode instrumentation. The resulting limitations on developing instrumentations are documented in corresponding developer guidelines. Among other things, the limitation of not being able to debug advice code is a painful restriction when developing and maintaining instrumentation code.

Elastic’s APM Java agent has been using the invokedynamic approach with its benefits for years — field-proven by thousands of customers. To help improve the OpenTelemetry Java agent, Elastic started contributing the invokedynamic approach with the goal to simplify and improve the development and maintainability of auto-instrumentation modules. The contribution proposal and the implementation outline is documented in more detail in this GitHub issue.

With the new approach in place, Elastic will help migrate existing instrumentations so the OTel Java community can benefit from the invokedynamic -based instrumentation approach.

Elastic supports OTel natively, and has numerous capabilities to help you analyze your application with OTel. 

• Native OpenTelemetry support in Elastic Observability
• Best Practices for instrumenting OpenTelemetry
• Independence with OpenTelemetry on Elastic

Instrumenting with OpenTelemetry:

• Elastiflix application, a guide to instrument different languages with OpenTelemetry (this is the application the team built to highlight all the languages below)
• Python: Auto-instrumentation, Manual instrumentation
• Java: Auto-instrumentation, Manual instrumentation 
• Node.js: Auto-instrumentation, Manual instrumentation
• .NET: Auto-instrumentation, Manual instrumentation
  Go: Manual instrumentation

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.