How to combine OpenTelemetry instrumentation with Elastic APM Agent features

By

Greg Kalapos

opentelemetry_apm-blog-720x420.jpeg

Elastic APM supports OpenTelemetry on multiple levels. One easy-to understand scenario, which we previously blogged about, is the direct OpenTelemetry Protocol (OTLP) support in APM Server. This means that you can connect any OpenTelemetry agent to an Elastic APM Server and the APM Server will happily take that data, ingest it into Elasticsearch®, and you can view that OpenTelemetry data in the APM app in Kibana®.

This blog post will showcase a different use-case: within Elastic APM, we have our own APM Agents. Some of these have download numbers in the tens of millions, and some of them predate OpenTelemetry. Of course we realize OpenTelemetry is very important and it’s here to stay, so we wanted to make these agents OpenTelemetry compatible and illustrate them using OpenTelemetry visualizations in this blog.

Most of our Elastic APM Agents today are able to ship OpenTelemetry spans as part of a trace. This means that if you have any component in your application that emits an OpenTelemetry span, it’ll be part of the trace the Elastic APM Agent captures. This can be a library you use that is already instrumented by the OpenTelemetry API, or it can be any other OpenTelemetry span that an application developer added into the application’s code for manual instrumentation.

This feature of the Elastic APM Agents not only reports those spans but also properly maintains parent-child relationships between all spans, making OpenTelemetry a first-class citizen for these agents. If, for example, an Elastic APM Agent starts a span for a specific action by auto-instrumentation and then within that span the OpenTelemetry API starts another span, then the OpenTelemetry span will be the child of the outer span created by the agent. This is reflected in the parent.id field of the spans. It’s the same the other way around as well: if a span is created by the OpenTelemetry API and within that span an Elastic APM agent captures another span, then the span created by the Elastic APM Agent will be the child of the other span created by the OpenTelemetry API.

This feature is present in the following agents:

Capturing OpenTelemetry spans in the Elastic .NET APM Agent

As a first example, let’s take an ASP.NET Core application. We’ll put the .NET Elastic APM Agent into this application, and we’ll turn on the feature, which automatically bridges OpenTelemetry spans, so the Elastic APM Agent will make those spans part of the trace it reports. 

The following code snippet shows a controller:

namespace SampleAspNetCoreApp.Controllers
{
	public class HomeController : Controller
	{
		private readonly SampleDataContext _sampleDataContext;
		private ActivitySource _activitySource = new ActivitySource("HomeController");
		public HomeController(SampleDataContext sampleDataContext) => _sampleDataContext = sampleDataContext;
		public async Task<IActionResult> Index()
		{
			await ReadGitHubStars();
			return View();
		}
		public async Task ReadGitHubStars()
		{
			using var activity = _activitySource.StartActivity();
			var httpClient = new HttpClient();
			httpClient.DefaultRequestHeaders.Add("User-Agent", "APM-Sample-App");
			var responseMsg = await httpClient.GetAsync("https://api.github.com/repos/elastic/apm-agent-dotnet");
			var responseStr = await responseMsg.Content.ReadAsStringAsync();
			// …use responseStr
		}
	}
}

The Index method calls the ReadGitHubStars method and after that we simply return the corresponding view from the method.

The incoming HTTP call and the outgoing HTTP call by the HttpClient are automatically captured by the Elastic APM Agent — this is part of the auto instrumentation we had for a very long time.

The ReadGitHubStars is the one where we use the OpenTelemetry API. OpenTelemetry in .NET uses the ActivitySource and Activity APIs. The _activitySource.StartActivity() call simply creates an OpenTelemetry span that automatically takes the name of the method by using the CallerMemberNameAttribute C# language feature, and this span will end when the method runs to completion.

Additionally, within this span we call the GitHub API with the HttpClient type. For this type, the .NET Elastic APM Agent again offers auto instrumentation, so the HTTP call will be also captured as a span by the agent automatically.

And here is how the water-flow chart for this transaction looks in Kibana:

trace sample kibana

As you can see, the agent was able to capture the OpenTelemetry span as part of the trace.

Bridging OpenTelemetry spans in Python by using the Python Elastic APM Agent

Let’s see how this works in the case of Python. The idea is the same, so all the concepts introduced previously apply to this example as well.

We take a very simple Django example:

from django.http import HttpResponse
from elasticapm.contrib.opentelemetry import Tracer
import requests


def index(request):
   tracer = Tracer(__name__)
   with tracer.start_as_current_span("ReadGitHubStars"):
       url = "https://api.github.com/repos/elastic/apm-agent-python"
       response = requests.get(url)
       return HttpResponse(response)

The first step to turn on capturing OpenTelemetry spans in Python is to import the Tracer implementation from elasticapm.contrib.opentelemetry. 

And then on this Tracer you can start a new span — in this case, we manually name the span ReadGitHubStars.

Similarly to the previous example, the call to http://127.0.0.1:8000/otelsample/ is captured by the Elastic APM Python Agent, and then the next span is created by the OpenTelemetry API, which, as you can see, is captured by the agent automatically, and then finally the HTTP call to the GitHub API is captured again by the auto instrumentation of the agent.

Here is how it looks in the water-flow chart:

water-flow chart

As already mentioned, the agent maintains the parent-child relationship for all the OTel spans. Let’s take a look at the parent.id of the GET api.github.com call:

OTel span details

As you can see, the id of this span is c98401c94d40b87a.

If we look at the span.id of the ReadGitHubStars OpenTelemetry span, then we can see that the id of this span is exactly c98401c94d40b87a — so the APM Agent internally maintains parent-child relationships across OpenTelemetry and non-OpenTelemetry spans, which makes OpenTelemetry spans first-class citizens in Elastic APM Agents.

OpenTelemetry spans first-class citizens in Elastic APM Agents

Other languages

At this point, I'll stop to just replicate the exact same sample code in further languages — I think you already got the point here: in each language listed above, our Elastic APM Agents are able to bridge OpenTelemetry traces and show them in Kibana as native spans. We also blogged about using the same API in Java, and you can see examples for the rest of the languages in the corresponding agent documentation (linked above).

When to use this feature and when to use pure OpenTelemetry SDKs

This is really up to you. If you want to only have pure OpenTelemetry usage in your applications and you really want to avoid any vendor-related software, then feel free to use OpenTelemetry SDKs directly — that is a use case we clearly support. If you go that route, this feature is not so relevant to you.

However, our Elastic APM Agents already have a very big user base and they offer features that are not present in OpenTelemetry. Some of these features are span compression, central configuration, inferred spans, distributed tail based sampling with multiple APM Servers, and many more. 

If you are one of the many existing Elastic APM Agent users, or you plan to use an Elastic APM Agent because of the features mentioned above, then bridging OpenTelemetry spans enables you to still use the OpenTelemetry API and not rely on any vendor related API usage. That way your developer teams can instrument your application with OpenTelemetry, and you can also use any third-party library already instrumented by OpenTelemetry, and Elastic APM Agents will happily report those spans as part of the traces they report. With this, you can combine the vendor independent nature of OpenTelemetry and still use the feature rich Elastic APM Agents.

The OpenTelemetry bridge feature is also a good tool to use if you wish to change your telemetry library from an Elastic APM Agent to OpenTelemetry (and vice-versa), as it allows you to use both libraries together and switch them using atomic changes.

Next steps

In this blog post, we discussed how you can bridge OpenTelemetry spans with Elastic APM Agents. Of course OpenTelemetry is more than just traces. We know that, and we plan to cover further areas: currently we are working on bridging OpenTelemetry metrics in our Elastic APM Agents in a very similar fashion. You can watch the progress here.

Learn more about adding Elastic APM as part of your Elastic Observability deployment.

Sign up for Elastic Cloud free trial

Spin up a fully loaded deployment on the cloud provider you choose. As the company behind Elasticsearch, we bring our features and support to your Elastic clusters in the cloud.

Start free trial