Troubleshootingedit

Use the information in this section to troubleshoot common problems and find answers for frequently asked questions. As a first step, ensure your stack is compatible with the Agent’s supported technologies.

Don’t worry if you can’t figure out what the problem is; we’re here to help. If you are an existing Elastic customer with a support contract, please create a ticket in the Elastic Support portal. If not, post in the APM discuss forum.

No data is sent to the APM Serveredit

If neither errors nor performance metrics are being sent to the APM Server, it’s a good idea to first check your logs and look for output just as the app starts.

If you don’t see anything suspicious in the agent logs (no warning or error), it’s recommended to turn the log level to Trace for further investigation.

Collecting agent logsedit

The way to collect logs depends on the setup of your application.

ASP.NET Coreedit

If you added the agent to your application as per the ASP.NET Core document with the UseAllElasticApm or UseElasticApm method, it will integrate with the ASP.NET Core logging infrastructure. This means the Agent will pick up the configured logging provider and log as any other component logs.

For example, the following configuration in appsettings.json limits APM agents logs to those with a log level of Warning or higher:

"Logging": {
  "LogLevel": {
    "Default": "Information",
    "Elastic.Apm": "Warning" 
  }
},

ASP.NET Classicedit

ASP.NET (classic) does not have a predefined logging system. By default, the agent is configured to emit log messages to a System.Diagnostics.TraceSource with the source name "Elastic.Apm". The TraceSource adheres to the log levels defined in the APM agent configuration. Typically, you will configure a preferred log level using an application setting in web.config.

TraceListeners can be configured to monitor log messages for the trace source, using the <system.diagnostics> section of web.config. For example, the following web.config section writes Elastic.Apm log messages to a file named my_log_file.log:

<configuration>
  <!-- other sections .... -->
  <system.diagnostics>
  <sources>
    <source name="Elastic.Apm"> 
      <listeners>
        <add name="file"
          type="System.Diagnostics.TextWriterTraceListener"
          initializeData="my_log_file.log" />
      </listeners>
    </source>
  </sources>
</system.diagnostics>
</configuration>

Other logging systemsedit

If you have a logging system in place such as NLog, Serilog, or similar, you can direct the agent logs into your logging system by creating an adapter between the agent’s internal logger and your logging system.

First implement the IApmLogger interface from the Elastic.Apm.Logging namespace:

internal class ApmLoggerAdapter : IApmLogger
{
	private readonly Lazy<Logger> _logger;
	public bool IsEnabled(ApmLogLevel level)
	{
		// Check for log level here.
		// Typically you just compare the configured log level of your logger
		// to the input parameter of this method and return if it's the same/higher or not
	}

	public void Log<TState>(ApmLogLevel apmLogLevel, TState state, Exception e, Func<TState, Exception, string> formatter)
	{
		// You can log the given log into your logging system here.
	}
}

An example implementation for NLog can be seen in our GitHub repository.

Then tell the agent to use the ApmLoggerAdapter. For ASP.NET (classic), place the following code into the Application_Start method in the HttpApplication implementation of your app which is typically in the Global.asax.cs file:

using Elastic.Apm.AspNetFullFramework;

namespace MyApp
{
	public class MyApplication : HttpApplication
	{
		protected void Application_Start()
		{
			AgentDependencies.Logger = new ApmLoggerAdapter();

			// other application setup...
		}
	}
}

During initialization, the agent checks if an additional logger was configured-- the agent only does this once, so it’s important to set it as early in the process as possible, typically in the Application_Start method.

General .NET applicationsedit

If none of the above cases apply to your application, you can still use a logger adapter and redirect agent logs into a .NET logging system like NLog, Serilog, or similar.

For this you’ll need an IApmLogger implementation (see above) which you need to pass to the Setup method during agent setup:

Agent.Setup(new AgentComponents(logger: new ApmLoggerAdapter()));

Following error appears in logs: The singleton APM agent has already been instantiated and can no longer be configured.edit

See "An InstanceAlreadyCreatedException exception is thrown".

An InstanceAlreadyCreatedException exception is thrownedit

In the early stage of a monitored process, the Agent might throw an InstanceAlreadyCreatedException exception with the following message: "The singleton APM agent has already been instantiated and can no longer be configured.", or an error log appears with the same message. This happens when you attempt to initialize the Agent multiple times, which is prohibited. Allowing multiple Agent instances per process would open up problems, like capturing events and metrics multiple times for each instance, or having multiple background threads for event serialization and transfer to the APM Server.

As an example, this issue can happen if you call the Elastic.Apm.Agent.Setup method multiple times, or if you call another method on Elastic.Apm.Agent that implicitly initializes the agent, and then you call the Elastic.Apm.Agent.Setup method on the already initialized agent.

Another example might be when you use the Public Agent API in combination with the IIS module or the ASP.NET Core NuGet package, where you enable the agent with the UseElasticApm or UseAllElasticApm methods. Both the first call to the IIS module and the UseElasticApm/UseAllElasticApm methods internally call the Elastic.Apm.Agent.Setup method to initialize the agent.

You may use the Public Agent API with the Elastic.Apm.Agent class in code that can potentially execute before the IIS module initializes or the UseElasticApm/UseAllElasticApm calls execute. If that happens, those will fail, as the Agent has been implicitly initialized already.

To prevent the InstanceAlreadyCreatedException in these scenarios, first use the Elastic.Apm.Agent.IsConfigured method to check if the agent is already initialized. After the check, you can safely use other methods in the Public Agent API. This will prevent accidental implicit agent initialization.

ASP.NET is using LegacyAspNetSynchronizationContext and might not behave well for asynchronous codeedit

If you see this warning being logged it means your classic ASP.NET Application is running under quirks mode and is using a deprecated but backwards compatible asynchronous context. This may prevent our agent from working correctly when asynchronous code introduces a thread switch since this context does not reliably restore HttpContext.Items.

To break out of quirks mode the runtime must be explicitly specified in web.config:

<httpRuntime targetFramework="4.5" />

Read more about ASP.NET quirks mode here: https://devblogs.microsoft.com/dotnet/all-about-httpruntime-targetframework

SqlEventListener Failed capturing sql statement (failed to remove from ProcessingSpans).edit

We log this warning when our SQL even listener is unable to find the active transaction. This has been only observed under IIS when the application is running under quirks mode. See "ASP.NET is using LegacyAspNetSynchronizationContext and might not behave well for asynchronous code" section for more backfround information and possible fixes.

HttpDiagnosticListenerFullFrameworkImpl No current transaction, skip creating span for outgoing HTTP requestedit

We log this trace warning when our outgoing HTTP listener is not able to get the current transaction. This has been only observed under IIS when the application is running under quirks mode. See "ASP.NET is using LegacyAspNetSynchronizationContext and might not behave well for asynchronous code" section for more backfround information and possible fixes.

Exception: System.PlatformNotSupportedException: This operation requires IIS integrated pipeline modeedit

This exception happens if the classic ASP.NET application run under an Application pool that enforces the classic pipeline mode. This prevents our agent to modify headers and thus will break distributed tracing.

The agent is only supported on IIS7 and higher where the Integrated Pipeline Mode is the default.

Startup hooks failureedit

If the startup hook integration throws an exception, additional detail can be obtained by setting the ELASTIC_APM_STARTUP_HOOKS_LOGGING environment variable before starting the application

set ELASTIC_APM_STARTUP_HOOKS_LOGGING=1

and then running the application in a context where the environment variable will be visible. In setting this value, an ElasticApmAgentStartupHook.log file is written to the directory containing the startup hook assembly, in addition to writing to standard output.

The agent causes too much overheadedit

A good place to start is All options summary. There are multiple settings with the Performance keyword which can help you tweak the agent for your needs.

The most expensive operation in the agent is typically stack trace capturing. The agent, by default, only captures stack traces for spans with a duration of 5ms or more, and with a limit of 50 stack frames. If this is too much in your environment, consider disabling stack trace capturing either partially or entirely:

The ElasticApmModule does not load or capture transactions and there are no agent logs generated on IISExpressedit

When debugging applications using Visual Studio and IISExpress, the same requirement to use the Integrated managed pipeline mode exists. Select your web application project in the solution explorer and press F4 to load the properties window. If the managed pipeline mode is set to classic, the ElasticApmModule will not load.

For example:

Should be changed to:

You may need to restart Visual Studio for these changes to fully apply.