Zero config OpenTelemetry auto- instrumentation for Kubernetes Java applications

The OpenTelemetry Java agent has a number of ways to install the agent into a Java application. If you are running your Java applications in Kubernetes pods, there is a separate mechanism (which under the hood uses JAVA_TOOL_OPTIONS and other environment variables) to auto-instrument Java applications. This auto-instrumentation can be achieved with zero configuration of the applications and pods!

The mechanism to achieve zero-config auto-instrumentation of Java applications in Kubernetes is via the OpenTelemetry Operator for Kubernetes. This operator has many capabilities and the full documentation (and of course source) is available in the project itself. In this blog, I'll walk through installing, setting up and running zero-config auto-instrumentation of Java applications in Kubernetes using the OpenTelemetry Operator.

Installing the OpenTelemetry Operator

At the time of writing this blog, the OpenTelemetry Operator needs the certification manager to be installed, after which the operator can be installed. Installing from the web is straightforward. First install the cert-manager (the version to be installed will be specified in the OpenTelemetry Operator for Kubernetes documentation):

kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.14.4/cert-manager.yaml

Then when the cert managers are ready (kubectl get pods -n cert-manager)  ...

NAMESPACE      NAME                                         READY
cert-manager   cert-manager-67c98b89c8-rnr5s                1/1
cert-manager   cert-manager-cainjector-5c5695d979-q9hxz     1/1
cert-manager   cert-manager-webhook-7f9f8648b9-8gxgs        1/1

... you can install the OpenTelemetry Operator:

kubectl apply -f https://github.com/open-telemetry/opentelemetry-operator/releases/latest/download/opentelemetry-operator.yaml

You can, of course, use a specific version of the operator instead of the latest. But here I’ve used the latest version.

An Instrumentation resource

Now you need to add just one further Kubernetes resource to enable auto-instrumentation: an Instrumentation resource. I am going to use the banana namespace for my examples, so I have first created that namespace (kubectl create namespace banana). The auto-instrumentation is specified and configured by these Instrumentation resources. Here is a basic one which will allow every Java pod in the banana namespace to be auto-instrumented with version 2.5.0 of the OpenTelemetry Java agent:

apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
  name: banana-instr
  namespace: banana
spec:
  exporter:
    endpoint: "https://my.endpoint"
  propagators:
    - tracecontext
    - baggage
    - b3
  sampler:
    type: parentbased_traceidratio
    argument: "1.0"
  java:
    image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-java:2.5.0
    env:
      - name: OTEL_EXPORTER_OTLP_HEADERS
        value: "Authorization=Bearer MyAuth"

Creating this resource (eg with kubectl apply -f banana-instr.yaml, assuming the above yaml was saved in file banana-instr.yaml) makes the banana-instr Instrumentation resource available for use. (Note you will need to change my.endpoint and MyAuth to values appropriate for your collector.) You can use this instrumentation immediately by adding an annotation to any deployment in the banana namespace:

metadata:
  annotations:
    instrumentation.opentelemetry.io/inject-java: "true"

The banana-instr Instrumentation resource is not yet set to be applied by default to all pods in the banana namespace. Currently it's zero-config as far as the application is concerned, but it requires an annotation added to a pod or deployment. To make it fully zero-config for all pods in the banana namespace, we need to add that annotation to the namespace itself, ie editing the namespace (kubectl edit namespace banana) so it would then have contents similar to

apiVersion: v1
kind: Namespace
metadata:
  name: banana
  annotations:
    instrumentation.opentelemetry.io/inject-java: "banana-instr"
...

Now we have a namespace that is going to auto-instrument every Java application deployed in the banana namespace with the 2.5.0 OpenTelemetry Java agent!

Trying it

There is a simple example Java application at docker.elastic.co/demos/apm/k8s-webhook-test which just repeatedly calls the chain main->methodA->methodB->methodC->methodD with some sleeps in the calls. Running this (kubectl apply -f banana-app.yaml) using a very basic pod definition:

apiVersion: v1
kind: Pod
metadata:
  name: banana-app
  namespace: banana
  labels:
    app: banana-app
spec:
  containers:
    - image: docker.elastic.co/demos/apm/k8s-webhook-test
      imagePullPolicy: Always
      name: banana-app
      env: 
      - name: OTEL_INSTRUMENTATION_METHODS_INCLUDE
        value: "test.Testing[methodB]"

results in the app being auto-instrumented with no configuration changes! The resulting app shows up in any APM UI, such as Elastic APM

As you can see, for this example I also added this env var to the pod yaml, OTEL_INSTRUMENTATION_METHODS_INCLUDE="test.Testing[methodB]" so that there were traces showing from methodB.

The technology behind the auto-instrumentation

To use the auto-instrumentation there is no specific need to understand the underlying mechanisms, but for those of you interested, here’s a quick outline.

1. The OpenTelemetry Operator for Kubernetes installs a mutating webhook, a standard Kubernetes component.
2. When deploying, Kubernetes first sends all definitions to the mutating webhook.
3. If the mutating webhook sees that the conditions for auto-instrumentation should be applied (ie
   1. there is an Instrumentation resource for that namespace and
   2. the correct annotation for that Instrumentation is applied to the definition in some way, either from the definition itself or from the namespace),
4. then the mutating webhook “mutates” the definition to include the environment defined by the Instrumentation resource.
5. The environment includes the explicit values defined in the env, as well as some implicit OpenTelemetry values (see the OpenTelemetry Operator for Kubernetes documentation for full details).
6. And most importantly, the operator
   1. pulls the image defined in the Instrumentation resource,
   2. extracts the file at the path /javaagent.jar from that image (using shell command cp)
   3. inserts it into the pod at path /otel-auto-instrumentation-java/javaagent.jar
   4. and adds the environment variable JAVA_TOOL_OPTIONS=-javaagent:/otel-auto-instrumentation-java/javaagent.jar.
7. The JVM automatically picks up that JAVA_TOOL_OPTIONS environment variable on startup and applies it to the JVM command-line.

Next steps

This walkthrough can be repeated in any Kubernetes cluster to demonstrate and experiment with auto-instrumentation (you will need to create the banana namespace first). In part 2 of this two part series, Using a custom agent with the OpenTelemetry Operator for Kubernetes, I show how to install any Java agent via the OpenTelemetry operator, using the Elastic Java agents as examples.