How to easily correlate logs and APM traces for better observability

Application performance monitoring (APM) and logging both provide critical insight into your ecosystem. When paired together with context, they can provide vital clues on how to resolve problems with your applications.

As the log data you analyze becomes more complex, navigating to the relevant pieces can be tricky using traditional tools. With Elastic Observability (powered by the Elastic Stack), correlating logs with APM is as simple as a few clicks in Kibana. Easily see the logs related to a specific request the user has made for a specific transaction in your web app.

Below is a screenshot of Elastic APM, showing a nice breakdown of all operations involved in a particular transaction. If we click on Actions:

And select Show trace logs:

Et voila:

These messages are extra details related to this specific transaction. This kind of detail can really only be provided by a log as it would not fit in APM or any other type of monitoring. Now we have excellent flexibility in adding extra detail to our transactions, thanks to the Elastic Stack integrating these different facets of monitoring together. Having these different types of data in one datastore lets us quickly pivot our investigations.

You may be saying to yourself, "This is all very nice, but how can I achieve similar results in my system?"

Let's take a look at an example and walk through how to get logs and APM traces integrated. Here’s the relevant code from the Python app shown in the screenshots above. We need a way to add some information to each request a user makes to our app.

First, the basics — we need to install the Elastic APM Python Agent. Let’s add it as a dependency. This is a Flask app, which is why it says [flask] in brackets after the dependency name — details for each supported framework can be found in the Python Agent docs.

# in requirements.txt 
elastic-apm[flask]==5.3.3

First, let’s make sure we have the right imports in place:

# in app.py 
import logging 
from elasticapm.contrib.flask import ElasticAPM 
from elasticapm.handlers.logging import Formatter

Then, let’s configure the Python Agent to send information to the APM Server:

# in app.py 
app = Flask(__name__)  # initialise app as usual 
app.config['ELASTIC_APM'] = { 
  'SERVICE_NAME': 'foodbankhero', 

  # Set custom APM Server URL (default: http://localhost:8200) 
  'SERVER_URL': 'http://localhost:8200' 

  # Use if APM Server requires a token 
  # The APM Server can be configured to require a token for security - so that it can’t be flooded with random data by a third party. 
  # 'SECRET_TOKEN': '' 
} 
apm = ElasticAPM(app)  # the magic line which instruments our Flask app with APM

Now, let’s configure the logs so that they can be correlated with APM traces as shown in the screenshots above:

# in app.py 
fh = logging.FileHandler('foodbankhero.log') 

# we imported a custom Formatter from the Python Agent earlier 
formatter = Formatter("%(asctime)s - %(name)s - %(levelname)s - %(message)s") 
fh.setFormatter(formatter) 
logging.getLogger().addHandler(fh) 

# … somewhere deep within the app ... 
logging.getLogger().info(‘Foodbank {} needs {} items’.format(id, len(items))) 
logging.getLogger().info(‘Returning {} foodbank records’.format(len(results)))

Here’s a sample of what your logs will look like now:

2020-05-12 01:57:14,029 - flask.app - INFO - 172.22.0.1 - - [12/May/2020 01:57:15] "GET / HTTP/1.1" 200 - | elasticapm transaction.id=f31f884b961a6089 trace.id=02bd57be0eff9979c94951d0d53e80c0 span.id=None

With this setup, every single log line for every request will be augmented with tracing information, even requests for static assets (if your app is configured to serve them). For example:

2020-05-12 01:57:14,029 - werkzeug - INFO - 172.22.0.1 - - [12/May/2020 01:57:14] "GET /favicon.ico HTTP/1.1" 404 - | elasticapm transaction.id=f31f884b961a6089  
trace.id=02bd57be0eff9979c94951d0d53e80c0 span.id=None

Ready to break down traditional silos and solve problems faster with context? Elastic Observability makes it simple to correlate data and navigate to what you need at the speed of thought.

See how easy it is to get started with Elastic APM — just add a few lines of code to your project and start sending data to Elasticsearch. Try correlating your logs and APM with a free trial on Elastic Cloud.