Log correlation in Elasticsearchedit

In order to correlate logs from your app with transactions captured by the Elastic APM Python Agent, your logs must contain one or more of the following identifiers:

  • transaction.id
  • trace.id
  • span.id

If you’re using structured logging, either with a custom solution or with structlog (recommended), then this is fairly easy. Throw the JSONRenderer in, and use Filebeat to pull these logs into Elasticsearch.

Without structured logging the task gets a little trickier. Here we recommend first making sure your LogRecord objects have the elasticapm attributes (see logging), and then you’ll want to combine some specific formatting with a Grok pattern, either in Elasticsearch using the grok processor, or in logstash with a plugin.

Say you have a Formatter that looks like this:

import logging

fh = logging.FileHandler('spam.log')
formatter = logging.Formatter("%(asctime)s - %(name)s - %(levelname)s - %(message)s")
fh.setFormatter(formatter)

You can add the APM identifiers by simply switching out the Formatter object for the one that we provide:

import logging
from elasticapm.handlers.logging import Formatter

fh = logging.FileHandler('spam.log')
formatter = Formatter("%(asctime)s - %(name)s - %(levelname)s - %(message)s")
fh.setFormatter(formatter)

This will automatically append apm-specific fields to your format string:

formatstring = "%(asctime)s - %(name)s - %(levelname)s - %(message)s"
formatstring = formatstring + " | elasticapm " \
                              "transaction.id=%(elasticapm_transaction_id)s " \
                              "trace.id=%(elasticapm_trace_id)s " \
                              "span.id=%(elasticapm_span_id)s"

Then, you could use a grok pattern like this (for the Elasticsearch Grok Processor):

{
  "description" : "...",
  "processors": [
    {
      "grok": {
        "field": "message",
        "patterns": ["%{GREEDYDATA:msg} | elasticapm transaction.id=%{DATA:transaction.id} trace.id=%{DATA:trace.id} span.id=%{DATA:span.id}"]
      }
    }
  ]
}