Manual instrumentation of .NET applications with OpenTelemetry

By

David Hope

observability-launch-series-4-net-manual.jpg

In the fast-paced universe of software development, especially in the cloud-native realm, DevOps and SRE teams are increasingly emerging as essential partners in application stability and growth.

DevOps engineers continuously optimize software delivery, while SRE teams act as the stewards of application reliability, scalability, and top-tier performance. The challenge? These teams require a cutting-edge observability solution, one that encompasses full-stack insights, empowering them to rapidly manage, monitor, and rectify potential disruptions before they culminate into operational challenges.

Observability in our modern distributed software ecosystem goes beyond mere monitoring — it demands limitless data collection, precision in processing, and the correlation of this data into actionable insights. However, the road to achieving this holistic view is paved with obstacles, from navigating version incompatibilities to wrestling with restrictive proprietary code.

Enter OpenTelemetry (OTel), with the following benefits for those who adopt it:

  • Escape vendor constraints with OTel, freeing yourself from vendor lock-in and ensuring top-notch observability.
  • See the harmony of unified logs, metrics, and traces come together to provide a complete system view.
  • Improve your application oversight through richer and enhanced instrumentations.
  • Embrace the benefits of backward compatibility to protect your prior instrumentation investments.
  • Embark on the OpenTelemetry journey with an easy learning curve, simplifying onboarding and scalability.
  • Rely on a proven, future-ready standard to boost your confidence in every investment.
  • Explore manual instrumentation, enabling customized data collection to fit your unique needs.
  • Ensure monitoring consistency across layers with a standardized observability data framework.
  • Decouple development from operations, driving peak efficiency for both.

In this post, we will dive into the methodology to instrument a .NET application manually using Docker.

What's covered?

  • Instrumenting the .NET application manually
  • Creating a Docker image for a .NET application with the OpenTelemetry instrumentation baked in
  • Installing and running the OpenTelemetry .NET Profiler for automatic instrumentation

Prerequisites

  • An understanding of Docker and .NET
  • Elastic Cloud
  • Docker installed on your machine (we recommend docker desktop)

View the example source code

The full source code, including the Dockerfile used in this blog, can be found on GitHub. The repository also contains the same application without instrumentation. This allows you to compare each file and see the differences.

The following steps will show you how to instrument this application and run it on the command line or in Docker. If you are interested in a more complete OTel example, take a look at the docker-compose file here, which will bring up the full project.

Step-by-step guide

This blog assumes you have an Elastic Cloud account — if not, follow the instructions to get started on Elastic Cloud.  

Step 1. Getting started

In our demonstration, we will manually instrument a .NET Core application - Login. This application simulates a simple user login service. In this example, we are only looking at Tracing since the OpenTelemetry logging instrumentation is currently at mixed maturity, as mentioned here.

The application has the following files:

  1. Program.cs

  2. Startup.cs

  3. Telemetry.cs

  4. LoginController.cs

Step 2. Instrumenting the application

When it comes to OpenTelemetry, the .NET ecosystem presents some unique aspects. While OpenTelemetry offers its API, .NET leverages its native System.Diagnostics API to implement OpenTelemetry's Tracing API. The pre-existing constructs such as ActivitySource and Activity are aptly repurposed to comply with OpenTelemetry.

That said, understanding the OpenTelemetry API and its terminology remains crucial for .NET developers. It's pivotal in gaining full command over instrumenting your applications, and as we've seen, it also extends to understanding elements of the System.Diagnostics API.

For those who might lean toward using the original OpenTelemetry APIs over the System.Diagnostics ones, there is also a way. OpenTelemetry provides an API shim for tracing that you can use. It enables developers to switch to OpenTelemetry APIs, and you can find more details about it in the OpenTelemetry API Shim documentation. 

By integrating such practices into your .NET application, you can take full advantage of the powerful features OpenTelemetry provides, irrespective of whether you're using OpenTelemetry's API or the System.Diagnostics API.

In this blog, we are sticking to the default method and using the Activity convention which the System.Diagnostics API dictates.

To manually instrument a .NET application, you need to make changes in each of these files. Let's take a look at these changes one by one.

Program.cs

This is the entry point for our application. Here, we create an instance of IHostBuilder with default configurations. Notice how we set up a console logger with Serilog.

public static void Main(string[] args)
{
    Log.Logger = new LoggerConfiguration().WriteTo.Console().CreateLogger();
    CreateHostBuilder(args).Build().Run();
}

Startup.cs

In the Startup.cs file, we use the ConfigureServices method to add the OpenTelemetry Tracing.

public void ConfigureServices(IServiceCollection services)
{
    services.AddOpenTelemetry().WithTracing(builder => builder.AddOtlpExporter()
        .AddSource("Login")
        .AddAspNetCoreInstrumentation()
        .AddOtlpExporter()  
        .ConfigureResource(resource =>
            resource.AddService(
                serviceName: "Login"))
    );
    services.AddControllers();
}

The WithTracing method enables tracing in OpenTelemetry. We add the OTLP (OpenTelemetry Protocol) exporter, which is a general-purpose telemetry data delivery protocol. We also add the AspNetCoreInstrumentation, which will automatically collect traces from our application. This is a critically important step that is not mentioned in the OpenTelemetry docs. Without adding this method, the instrumentation was not working for me for the Login application.

Telemetry.cs

This file contains the definition of our ActivitySource. The ActivitySource represents the source of the telemetry activities. It is named after the service name for your application, and this name can come from a configuration file, constants file, etc. We can use this ActivitySource to start activities.

using System.Diagnostics;

public static class Telemetry
{
    //...

    // Name it after the service name for your app.
    // It can come from a config file, constants file, etc.
    public static readonly ActivitySource LoginActivitySource = new("Login");

    //...
}

In our case, we've created an ActivitySource named Login. In our LoginController.cs, we use this LoginActivitySource to start a new activity when we begin our operations.

using (Activity activity = Telemetry.LoginActivitySource.StartActivity("SomeWork"))
{
    // Perform operations here
}

This piece of code starts a new activity named SomeWork, performs some operations (in this case, generating a random user and logging them in), and then ends the activity. These activities are traced and can be analyzed later to understand the performance of the operations.

This ActivitySource is fundamental to OpenTelemetry's manual instrumentation. It represents the source of the activities and provides a way to start and stop activities.

LoginController.cs

In the LoginController.cs file, we are tracing the operations performed by the GET and POST methods. We start a new activity, SomeWork, before we begin our operations and dispose of it once we're done.

using (Activity activity = Telemetry.LoginActivitySource.StartActivity("SomeWork"))
{
    var user = GenerateRandomUserResponse();
    Log.Information("User logged in: {UserName}", user);
    return user;
}

This will track the time taken by these operations and send this data to any configured telemetry backend via the OTLP exporter.

Step 3. Base image setup

Now that we have our application source code created and instrumented, it’s time to create a Dockerfile to build and run our .NET Login service. 

Start with the .NET runtime image for the base layer of our Dockerfile:

FROM ${ARCH}mcr.microsoft.com/dotnet/aspnet:7.0. AS base
WORKDIR /app
EXPOSE 8000

Here, we're setting up the application's runtime environment.

Step 4. Building the .NET application

This feature of Docker is just the best. Here, we compile our .NET application. We'll use the SDK image. In the bad old days, we used to build on a different platform and then put the compiled code into the Docker container. This way, we are much more confident our build will replicate from a developers desktop and into production by using Docker all the way through.

FROM --platform=$BUILDPLATFORM mcr.microsoft.com/dotnet/sdk:8.0-preview AS build
ARG TARGETPLATFORM

WORKDIR /src
COPY ["login.csproj", "./"]
RUN dotnet restore "./login.csproj"
COPY . .
WORKDIR "/src/."
RUN dotnet build "login.csproj" -c Release -o /app/build

This section ensures that our .NET code is properly restored and compiled.

Step 5. Publishing the application

Once built, we'll publish the app:

FROM build AS publish
RUN dotnet publish "login.csproj" -c Release -o /app/publish

Step 6. Preparing the final image

Now, let's set up the final runtime image:

FROM base AS final
WORKDIR /app
COPY --from=publish /app/publish .

Step 7. Entry point setup

Lastly, set the Docker image's entry point to both source the OpenTelemetry instrumentation, which sets up the Environment variables required to bootstrap the .NET Profiler, and then we start our .NET application:

ENTRYPOINT ["/bin/bash", "-c", "dotnet login.dll"]

Step 8. Running the Docker image with environment variables

To build and run the Docker image, you'd typically follow these steps:

Build the Docker image

First, you'd want to build the Docker image from your Dockerfile. Let's assume the Dockerfile is in the current directory, and you'd like to name/tag your image dotnet-login-otel-image.

docker build -t dotnet-login-otel-image .

Run the Docker image

After building the image, you'd run it with the specified environment variables. For this, the docker run command is used with the -e flag for each environment variable.

  docker run \
       -e OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer ${ELASTIC_APM_SECRET_TOKEN}" \
       -e OTEL_EXPORTER_OTLP_ENDPOINT="${ELASTIC_APM_SERVER_URL}" \
       -e OTEL_METRICS_EXPORTER="otlp" \
       -e OTEL_RESOURCE_ATTRIBUTES="service.version=1.0,deployment.environment=production" \
       -e OTEL_SERVICE_NAME="dotnet-login-otel-manual" \
       -e OTEL_TRACES_EXPORTER="otlp" \
       dotnet-login-otel-image

Make sure that ${ELASTIC_APM_SECRET_TOKEN} and ${ELASTIC_APM_SERVER_URL} are set in your shell environment, replace them with their actual values from the cloud as shown below.

Getting Elastic Cloud variables
You can copy the endpoints and token from Kibana under the path `/app/home#/tutorial/apm`.

apm agents

You can also use an environment file with docker run --env-file to make the command less verbose if you have multiple environment variables. 

Once you have this up and running, you can ping the endpoint for your instrumented service (in our case, this is /login), and you should see the app appear in Elastic APM, as shown below:

services

It will begin by tracking throughput and latency critical metrics for SREs to pay attention to.

Digging in, we can see an overview of all our Transactions.

login

And look at specific transactions, including the “SomeWork” activity/span we created in the code above:

latency distribution graph

There is clearly an outlier here, where one transaction took over 20ms. This is likely to be due to the CLR warming up.

Wrapping up

With the code here instrumented and the Dockerfile bootstrapping the application, you've transformed your simple .NET application into one that's instrumented with OpenTelemetry. This will aid greatly in understanding application performance, tracing errors, and gaining insights into how users interact with your software.

Remember, observability is a crucial aspect of modern application development, especially in distributed systems. With tools like OpenTelemetry, understanding complex systems becomes a tad bit easier.

In this blog, we discussed the following:

  • How to manually instrument .NET with OpenTelemetry. 
  • Using standard commands in a Docker file, our instrumented application was built and started.
  • Using OpenTelemetry and its support for multiple languages, DevOps and SRE teams can instrument their applications with ease, gaining immediate insights into the health of the entire application stack and reducing mean time to resolution (MTTR).

Since Elastic can support a mix of methods for ingesting data whether it be using auto-instrumentation of open-source OpenTelemetry or manual instrumentation with its native APM agents, you can plan your migration to OTel by focusing on a few applications first and then using OpenTelemety across your applications later on in a manner that best fits your business needs.

Additional resources for OpenTelemetry with Elastic:

Developer resources:


General configuration and use case resources:

Don’t have an Elastic Cloud account yet? Sign up for Elastic Cloud and try out the instrumentation capabilities that I discussed above. I would be interested in getting your feedback about your experience in gaining visibility into your application stack with Elastic.  

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

Sign up for Elastic Cloud free trial

Spin up a fully loaded deployment on the cloud provider you choose. As the company behind Elasticsearch, we bring our features and support to your Elastic clusters in the cloud.

Start free trial