Troubleshootingedit

Something isn’t quite working as expected? Here are some guidelines how to find out what’s going wrong.

Don’t worry if you can’t figure out what the problem is. Open a topic in the APM discuss forum and we will help you out. If you do so, please attach your debug logs so that we can analyze the problem.

Also, please check if your stack is compatible with the currently supported technologies.

Loggingedit

There are several logging related configuration options. The most important one is log_leveledit. Set the log level to DEBUG or even TRACE to get more information about the behavior of the agent. The TRACE log level is even more verbose than DEBUG and prints a stack trace every time a transaction or span is started or stopped. Also, the JSON documents which are sent to the APM server will be logged on TRACE level.

Please always post the whole content of your log files when asking for help.

When the agent starts up, you should see logs similar to these:

[main] INFO co.elastic.apm.configuration.StartupInfo - Starting Elastic APM (unknown version) on Java 10 (Oracle Corporation) Mac OS X 10.13.6
[apm-server-healthcheck] INFO co.elastic.apm.report.ApmServerHealthChecker - Elastic APM server is available: {"build_date":"2018-11-05T07:58:08Z","build_sha":"dffb98a72a262ca22adad0152f0245ea743ea904","version":"7.0.0-alpha1"}
[main] DEBUG co.elastic.apm.configuration.StartupInfo - service_name: 'elastic-apm-test' (source: Java System Properties)
[main] DEBUG co.elastic.apm.configuration.StartupInfo - log_level: 'DEBUG' (source: Java System Properties)

Make sure to execute some requests to your application before posting your log files. Each request should at least add some lines similar to these in the logs:

[http-nio-8080-exec-10] DEBUG co.elastic.apm.impl.ElasticApmTracer - startTransaction '' 00-2a82cbe3df7a0208f7be6da65be260d1-05e72d045206587a-01 {
[http-nio-8080-exec-10] DEBUG co.elastic.apm.impl.ElasticApmTracer - Activating '' 00-2a82cbe3df7a0208f7be6da65be260d1-05e72d045206587a-01 on thread 66
[http-nio-8080-exec-10] DEBUG co.elastic.apm.impl.transaction.Span - startSpan '' 00-2a82cbe3df7a0208f7be6da65be260d1-b2ffa0401105e3d8-01 {
[http-nio-8080-exec-10] DEBUG co.elastic.apm.impl.ElasticApmTracer - Activating 'APIRestController#products' 00-2a82cbe3df7a0208f7be6da65be260d1-b2ffa0401105e3d8-01 on thread 66
[http-nio-8080-exec-10] DEBUG co.elastic.apm.impl.transaction.Span - startSpan '' 00-2a82cbe3df7a0208f7be6da65be260d1-49b9d805eca42ec6-01 {
[http-nio-8080-exec-10] DEBUG co.elastic.apm.impl.ElasticApmTracer - Activating '' 00-2a82cbe3df7a0208f7be6da65be260d1-49b9d805eca42ec6-01 on thread 66
[http-nio-8080-exec-10] DEBUG co.elastic.apm.impl.ElasticApmTracer - Deactivating 'SELECT' 00-2a82cbe3df7a0208f7be6da65be260d1-49b9d805eca42ec6-01 on thread 66
[http-nio-8080-exec-10] DEBUG co.elastic.apm.impl.transaction.Span - } endSpan 'SELECT' 00-2a82cbe3df7a0208f7be6da65be260d1-49b9d805eca42ec6-01
[apm-reporter] DEBUG co.elastic.apm.report.IntakeV2ReportingEventHandler - Receiving SPAN event (sequence 23)
[apm-reporter] DEBUG co.elastic.apm.report.IntakeV2ReportingEventHandler - Starting new request to http://localhost:8200/intake/v2/events
[http-nio-8080-exec-10] DEBUG co.elastic.apm.impl.ElasticApmTracer - Deactivating 'APIRestController#products' 00-2a82cbe3df7a0208f7be6da65be260d1-b2ffa0401105e3d8-01 on thread 66
[http-nio-8080-exec-10] DEBUG co.elastic.apm.impl.transaction.Span - } endSpan 'APIRestController#products' 00-2a82cbe3df7a0208f7be6da65be260d1-b2ffa0401105e3d8-01
[apm-reporter] DEBUG co.elastic.apm.report.IntakeV2ReportingEventHandler - Scheduling request timeout in 10s
[apm-reporter] DEBUG co.elastic.apm.report.IntakeV2ReportingEventHandler - Receiving SPAN event (sequence 24)
[http-nio-8080-exec-10] DEBUG co.elastic.apm.impl.ElasticApmTracer - Deactivating 'APIRestController#products' 00-2a82cbe3df7a0208f7be6da65be260d1-05e72d045206587a-01 on thread 66
[http-nio-8080-exec-10] DEBUG co.elastic.apm.impl.ElasticApmTracer - } endTransaction 'APIRestController#products' 00-2a82cbe3df7a0208f7be6da65be260d1-05e72d045206587a-01
[apm-reporter] DEBUG co.elastic.apm.report.IntakeV2ReportingEventHandler - Receiving TRANSACTION event (sequence 25)

If you don’t see anything in your logs, the technology stack you are using is probably not supported.

After that, you should see logs indicating that the agent has successfully sent data to the APM server:

[apm-request-timeout-timer] DEBUG co.elastic.apm.report.IntakeV2ReportingEventHandler - Request flush because the request timeout occurred
[apm-reporter] DEBUG co.elastic.apm.report.IntakeV2ReportingEventHandler - Receiving FLUSH event (sequence 26)
[apm-reporter] DEBUG co.elastic.apm.report.IntakeV2ReportingEventHandler - Flushing 10912 uncompressed 2667 compressed bytes

If the APM server responds with a 400, it could indicate JSON validation errors. The log would then contain the actual documents which failed to validate:

[apm-reporter] INFO co.elastic.apm.report.IntakeV2ReportingEventHandler - Backing off for 0 seconds (±10%)
[apm-reporter] WARN co.elastic.apm.report.IntakeV2ReportingEventHandler - Server returned HTTP response code: 400 for URL: http://localhost:8200/intake/v2/events
[apm-reporter] WARN co.elastic.apm.report.IntakeV2ReportingEventHandler - {"accepted":13,"errors":[{"message":"Problem validating JSON document against schema: I[#] S[#] doesn't validate with \"span#\"\n  I[#] S[#/allOf/2] allOf failed\n    I[#] S[#/allOf/2/required] missing properties: \"transaction_id\"","document":"{\"span\":{\"name\":\"OpenTracing product span\",\"timestamp\":29352159207,\"id\":\"aeaa7e0ac95acad6\",\"trace_id\":\"d88b5cbfc4536f9a700cd114a53bfeae\",\"parent_id\":\"082fd71ce7e4089a\",\"duration\":17.992,\"context\":{\"tags\":{\"productId\":\"1\"}},\"type\":\"unknown\"}}"}]}

Debuggingedit

Sometimes reading the logs is just not enough to debug a problem. As the agent is OpenSource and released on Maven Central, debugging the agent code is really easy.

In order for your IDE to download the sources, first declare a dependency to the agent.

Note

The agent is added via the -javaagent flag. So you have to make sure that you declare the dependency in a way that the agent is not on the classpath twice. For example, when you are developing a web application, make sure that the agent is not packaged in your war in WEB-INF/lib. In the debug window of your IDE, make sure that elastic-apm-agent does not appear in the -classpath.

pom.xml. 

<dependency>
    <groupId>co.elastic.apm</groupId>
    <artifactId>elastic-apm-agent</artifactId>
    <version>${elastic-apm.version}</version>
    <scope>provided</scope>
</dependency>

Note

Even when setting the scope to provided, IntelliJ sometimes adds the agent on the classpath anyway. One workaround is to set the scope to test.

build.gradle. 

compileOnly "co.elastic.apm:elastic-apm-agent:$elasticApmVersion"

Common problemsedit

There is no data in the Kibana APM UIedit

The most common source of this problem are connection issues between the agent and the APM server.

If the APM server does not receive data from the agent, check if the agent is able to establish a connection to the server. In the agent logs, look out for logs containing Elastic APM server is available and Elastic APM server is not available.

If you see the message Elastic APM server is not available, the agent has problems connecting to the APM server. Check the setting of server_urlsedit and make sure the agent is able to connect to the server. Try to execute curl -v <apm-server-url>/healthcheck from the machine the agent is running on. The server should respond with a 200 status code.

If the APM server does not respond successfully, have a look at the APM server logs to verify that the server is actually running. Also make sure to configure your firewalls so that the host the agent runs on can open HTTP connections to the APM server.