Profiler Auto instrumentationedit

This functionality is in beta and is subject to change. The design and code is less mature than official GA features and is being provided as-is with no warranties. Beta features are not subject to the support SLA of official GA features.

Quick startedit

The agent can automatically instrument .NET Framework, .NET Core, and .NET applications using the .NET CLR Profiling APIs. The Profiling APIs provide a way to instrument an application or dependency code without code changes.

The profiler requires the Microsoft C and C++ (MSVC) runtime libraries to be installed, to run on Windows. The Microsoft Visual C++ Redistributable package can be downloaded from Microsoft.

This approach works with the following

Operating system

Architecture

Windows

Linux

x64

.NET Framework 4.6.1+

.NET Core 2.1+

.NET 5

.NET Core 2.1+

.NET 5

and instruments the following assemblies

Integration name NuGet package version(s) Assembly version(s)

AdoNet

part of .NET

System.Data 4.0.0 - 4.*.*

part of .NET

System.Data.Common 4.0.0 - 5.*.*

AspNet

part of .NET Framework

System.Web 4.0.0 - 4.*.*

Kafka

Confluent.Kafka 1.4.0 - 1.*.*

Confluent.Kafka 1.4.0 - 1.*.*

MySqlCommand

MySql.Data 6.7.0 - 8.*.*

MySql.Data 6.7.0 - 8.*.*

NpgsqlCommand

Npgsql 4.0.0 - 5.*.*

Npgsql 4.0.0 - 5.*.*

OracleCommand

Oracle.ManagedDataAccess 12.2.1100 - 21.*.*

Oracle.ManagedDataAccess 4.122.0 - 4.122.*

Oracle.ManagedDataAccess.Core 2.0.0 - 2.*.*

Oracle.ManagedDataAccess 2.0.0 - 2.*.*

RabbitMQ

RabbitMQ.Client 3.6.9 - 6.*.*

RabbitMQ.Client 3.6.9 - 6.*.*

SqlCommand

part of .NET

System.Data 4.0.0 - 4.*.*

System.Data.SqlClient 4.0.0 - 4.*.*

System.Data.SqlClient 4.0.0 - 4.*.*

Microsoft.Data.SqlClient 1.0.0 - 2.*.*

Microsoft.Data.SqlClient 1.0.0 - 2.*.*

SqliteCommand

Microsoft.Data.Sqlite 2.0.0 - 5.*.*

Microsoft.Data.Sqlite 2.0.0 - 5.*.*

System.Data.SQLite 1.0.0 - 2.*.*

System.Data.SQLite 1.0.0 - 2.*.*

The .NET CLR Profiling API allows only one profiler to be attached to a .NET process. In light of this limitation, only one solution that uses the .NET CLR profiling API should be used by an application.

Auto instrumentation using the .NET CLR Profiling API can be used in conjunction with

  • the Public API to perform manual instrumentation.
  • NuGet packages that perform instrumentation using a IDiagnosticsSubscriber to subscribe to diagnostic events.

The version number of NuGet packages referenced by a project instrumented with a profiler must be the same as the version number of profiler zip file used.

General stepsedit

The general steps in configuring profiler auto instrumentation are as follows; See Instrumenting containers and services for configuration for common deployment environments.

  1. Download the elastic_apm_profiler_<version>.zip file from the Releases page of the .NET APM Agent GitHub repository, where <version> is the version number to download. You can find the file under Assets.
  2. Unzip the zip file into a folder on the host that is hosting the application to instrument.
  3. Configure the following environment variables

    .NET Framework.

    set COR_ENABLE_PROFILING = "1"
    set COR_PROFILER = "{FA65FE15-F085-4681-9B20-95E04F6C03CC}"
    set COR_PROFILER_PATH = "<unzipped directory>\elastic_apm_profiler.dll" 
    set ELASTIC_APM_PROFILER_HOME = "<unzipped directory>"
    set ELASTIC_APM_PROFILER_INTEGRATIONS = "<unzipped directory>\integrations.yml"

    <unzipped directory> is the directory to which the zip file was unzipped in step 2.

    .NET Core / .NET 5 on Windows.

    set CORECLR_ENABLE_PROFILING = "1"
    set CORECLR_PROFILER = "{FA65FE15-F085-4681-9B20-95E04F6C03CC}"
    set CORECLR_PROFILER_PATH = "<unzipped directory>\elastic_apm_profiler.dll" 
    set ELASTIC_APM_PROFILER_HOME = "<unzipped directory>"
    set ELASTIC_APM_PROFILER_INTEGRATIONS = "<unzipped directory>\integrations.yml"

    <unzipped directory> is the directory to which the zip file was unzipped in step 2.

    .NET Core / .NET 5 on Linux.

    export CORECLR_ENABLE_PROFILING=1
    export CORECLR_PROFILER={FA65FE15-F085-4681-9B20-95E04F6C03CC}
    export CORECLR_PROFILER_PATH="<unzipped directory>/libelastic_apm_profiler.so" 
    export ELASTIC_APM_PROFILER_HOME="<unzipped directory>"
    export ELASTIC_APM_PROFILER_INTEGRATIONS="<unzipped directory>/integrations.yml"

    <unzipped directory> is the directory to which the zip file was unzipped in step 2.

  4. Start your application in a context where the set environment variables are visible.

With this setup, the .NET runtime loads Elastic’s CLR profiler into the .NET process, which loads and instantiates the APM agent early in application startup. The profiler monitors methods of interest and injects code to instrument their execution.

Instrumenting containers and servicesedit

Using global environment variables causes profiler auto instrumentation to be loaded for any .NET process started on the host. Often, the environment variables should be set only for specific services or containers. The following sections demonstrate how to configure common containers and services.

Docker containersedit

A build image containing the files for profiler auto instrumentation can be used as part of a multi-stage build

ARG AGENT_VERSION=1.11.1

FROM alpine:latest AS build
ARG AGENT_VERSION
WORKDIR /source

# install unzip
RUN apt-get -y update && apt-get -yq install unzip

# pull down the zip file based on ${AGENT_VERSION} ARG and unzip
RUN curl -L -o elastic_apm_profiler_${AGENT_VERSION}.zip https://github.com/elastic/apm-agent-dotnet/releases/download/${AGENT_VERSION}/elastic_apm_profiler_${AGENT_VERSION}.zip && \
    unzip elastic_apm_profiler_${AGENT_VERSION}.zip -d /elastic_apm_profiler_${AGENT_VERSION}

The files can then be copied into a subsequent stage

COPY --from=build /elastic_apm_profiler_${AGENT_VERSION} /elastic_apm_profiler

Environment variables can be added to a Dockerfile to configure profiler auto instrumentation

ENV CORECLR_ENABLE_PROFILING=1
ENV CORECLR_PROFILER={FA65FE15-F085-4681-9B20-95E04F6C03CC}
ENV CORECLR_PROFILER_PATH=/elastic_apm_profiler/libelastic_apm_profiler.so
ENV ELASTIC_APM_PROFILER_HOME=/elastic_apm_profiler
ENV ELASTIC_APM_PROFILER_INTEGRATIONS=/elastic_apm_profiler/integrations.yml

ENTRYPOINT ["dotnet", "your-application.dll"]

Windows servicesedit

Environment variables can be added to specific Windows services by adding an entry to the Windows registry. Using PowerShell

.NET Framework service.

$environment = [string[]]@(
  "COR_ENABLE_PROFILING=1",
  "COR_PROFILER={FA65FE15-F085-4681-9B20-95E04F6C03CC}",
  "COR_PROFILER_PATH=<unzipped directory>\elastic_apm_profiler.dll",
  "ELASTIC_APM_PROFILER_HOME=<unzipped directory>",
  "ELASTIC_APM_PROFILER_INTEGRATIONS=<unzipped directory>\integrations.yml")

Set-ItemProperty HKLM:SYSTEM\CurrentControlSet\Services\<service-name> -Name Environment -Value $environment

.NET Core service.

$environment = [string[]]@(
  "CORECLR_ENABLE_PROFILING=1",
  "CORECLR_PROFILER={FA65FE15-F085-4681-9B20-95E04F6C03CC}",
  "CORECLR_PROFILER_PATH=<unzipped directory>\elastic_apm_profiler.dll", 
  "ELASTIC_APM_PROFILER_HOME=<unzipped directory>",
  "ELASTIC_APM_PROFILER_INTEGRATIONS=<unzipped directory>\integrations.yml")

Set-ItemProperty HKLM:SYSTEM\CurrentControlSet\Services\<service-name> -Name Environment -Value $environment 

<unzipped directory> is the directory to which the zip file was unzipped

<service-name> is the name of the Windows service.

The service must then be restarted for the change to take effect. With PowerShell

Restart-Service <service-name>

Internet Information Services (IIS)edit

For IIS versions before IIS 10, it is not possible to set environment variables scoped to a specific application pool, so environment variables need to set globally.

For IIS 10 onwards, environment variables can be set for an application pool using AppCmd.exe. With PowerShell

.NET Framework.

$appcmd = "$($env:systemroot)\system32\inetsrv\AppCmd.exe"
$appPool = "<application-pool>" 
$profilerHomeDir = "<unzipped directory>" 
$environment = @{
  COR_ENABLE_PROFILING = "1"
  COR_PROFILER = "{FA65FE15-F085-4681-9B20-95E04F6C03CC}"
  COR_PROFILER_PATH = "$profilerHomeDir\elastic_apm_profiler.dll"
  ELASTIC_APM_PROFILER_HOME = "$profilerHomeDir"
  ELASTIC_APM_PROFILER_INTEGRATIONS = "$profilerHomeDir\integrations.yml"
  COMPlus_LoaderOptimization = "1" 
}

$environment.Keys | ForEach-Object {
  & $appcmd set config -section:system.applicationHost/applicationPools /+"[name='$appPool'].environmentVariables.[name='$_',value='$($environment[$_])']"
}

<application-pool> is the name of the Application Pool your application uses. For example, IIS APPPOOL\DefaultAppPool

<unzipped directory> is the full path to the directory in which the zip file was unzipped

Forces assemblies not to be loaded domain-neutral. There is currently a limitation where Profiler auto-instrumentation cannot instrument assemblies when they are loaded domain-neutral. This limitation is expected to be removed in future, but for now, can be worked around by setting this environment variable. See the Microsoft documentation for further details.

.NET Core.

$appcmd = "$($env:systemroot)\system32\inetsrv\AppCmd.exe"
$appPool = "<application-pool>" 
$profilerHomeDir = "<unzipped directory>" 
$environment = @{
  CORECLR_ENABLE_PROFILING = "1"
  CORECLR_PROFILER = "{FA65FE15-F085-4681-9B20-95E04F6C03CC}"
  CORECLR_PROFILER_PATH = "$profilerHomeDir\elastic_apm_profiler.dll"
  ELASTIC_APM_PROFILER_HOME = "$profilerHomeDir"
  ELASTIC_APM_PROFILER_INTEGRATIONS = "$profilerHomeDir\integrations.yml"
}

$environment.Keys | ForEach-Object {
  & $appcmd set config -section:system.applicationHost/applicationPools /+"[name='$appPool'].environmentVariables.[name='$_',value='$($environment[$_])']"
}

<application-pool> is the name of the Application Pool your application uses. For example, IIS APPPOOL\DefaultAppPool

<unzipped directory> is the full path to the directory in which the zip file was unzipped

Ensure that the location of the <unzipped directory> is accessible and executable to the Identity account under which the Application Pool runs.

Once environment variables have been set, stop and start IIS so that applications hosted in IIS will see the new environment variables

net stop /y was
net start w3svc

systemd / systemctledit

Environment variables can be added to specific services run with systemd by creating an environment.env file containing the following

CORECLR_ENABLE_PROFILING=1
CORECLR_PROFILER={FA65FE15-F085-4681-9B20-95E04F6C03CC}
CORECLR_PROFILER_PATH=/<unzipped directory>/libelastic_apm_profiler.so 
ELASTIC_APM_PROFILER_HOME=/<unzipped directory>
ELASTIC_APM_PROFILER_INTEGRATIONS=/<unzipped directory>/integrations.yml

<unzipped directory> is the directory to which the zip file was unzipped

Then adding an EnvironmentFile entry to the service’s configuration file that references the path to the environment.env file

[Service]
EnvironmentFile=/path/to/environment.env
ExecStart=<command> 

the command that starts your service

After adding the EnvironmentFile entry, restart the service

systemctl reload-or-restart <service>

Profiler environment variablesedit

The profiler auto instrumentation has its own set of environment variables to manage the instrumentation. These are used in addition to agent configuration through environment variables.

ELASTIC_APM_PROFILER_HOME

The home directory of the profiler auto instrumentation. The home directory typically contains

  • platform specific profiler assemblies
  • a directory for each compatible target framework, where each directory contains supporting managed assemblies for auto instrumentation.
  • an integrations.yml file that determines which methods to target for auto instrumentation
ELASTIC_APM_PROFILER_INTEGRATIONS (optional)
The path to the integrations.yml file that determines which methods to target for auto instrumentation. If not specified, the profiler will assume an integrations.yml exists in the home directory specified by ELASTIC_APM_PROFILER_HOME environment variable.
ELASTIC_APM_PROFILER_EXCLUDE_INTEGRATIONS (optional)
A semi-colon separated list of integrations to exclude from auto-instrumentation. Valid values are those defined in the Integration name column in the integrations table above.
ELASTIC_APM_PROFILER_EXCLUDE_PROCESSES (optional)
A semi-colon separated list of process names to exclude from auto-instrumentation. For example, dotnet.exe;powershell.exe. Can be used in scenarios where profiler environment variables have a global scope that would end up auto-instrumenting applications that should not be.
ELASTIC_APM_PROFILER_EXCLUDE_SERVICE_NAMES (optional)
A semi-colon separated list of APM service names to exclude from auto-instrumentation. Values defined are checked against the value of ELASTIC_APM_SERVICE_NAME environment variable.
ELASTIC_APM_PROFILER_LOG (optional)

The log level at which the profiler should log. Valid values are

  • trace
  • debug
  • info
  • warn
  • error
  • none

The default value is warn. More verbose log levels like trace and debug can affect the runtime performance of profiler auto instrumentation, so are recommended only for diagnostics purposes.

ELASTIC_APM_PROFILER_LOG_DIR (optional)

The directory in which to write profiler log files. If unset, defaults to

  • %PROGRAMDATA%\elastic\apm-agent-dotnet\logs on Windows
  • /var/log/elastic/apm-agent-dotnet on Linux

If the default directory cannot be written to for some reason, the profiler will try to write log files to a logs directory in the home directory specified by ELASTIC_APM_PROFILER_HOME environment variable.

ELASTIC_APM_PROFILER_LOG_TARGETS (optional)

A semi-colon separated list of targets for profiler logs. Valid values are

  • file
  • stdout

The default value is file, which logs to the directory specified by ELASTIC_APM_PROFILER_LOG_DIR environment variable.