IMPORTANT: No additional bug fixes or documentation updates will be released for this version. For the latest information, see the current release documentation.
Elastic Docs ›
« Introduction Structured logging with log4j2 »

Get Startededit

Step 1: Configure application loggingedit

The following logging frameworks are supported:

  • Logback (default for Spring Boot)
  • Log4j2
  • Log4j
  • java.util.logging (JUL)
  • JBoss Log Manager

Add the dependencyedit

The minimum required logback version is 1.1.

Download the latest version of Elastic logging: Maven Central

Add a dependency to your application:

<dependency>
    <groupId>co.elastic.logging</groupId>
    <artifactId>logback-ecs-encoder</artifactId>
    <version>${ecs-logging-java.version}</version>
</dependency>

If you are not using a dependency management tool, like maven, you have to manually add both logback-ecs-encoder and ecs-logging-core jars to the classpath. For example to the $CATALINA_HOME/lib directory.

Use the ECS encoder/formatter/layoutedit

Spring Boot applications

In src/main/resources/logback-spring.xml:

<?xml version="1.0" encoding="UTF-8"?>
<configuration>
    <property name="LOG_FILE" value="${LOG_FILE:-${LOG_PATH:-${LOG_TEMP:-${java.io.tmpdir:-/tmp}}}/spring.log}"/>
    <include resource="org/springframework/boot/logging/logback/defaults.xml"/>
    <include resource="org/springframework/boot/logging/logback/console-appender.xml" />
    <include resource="org/springframework/boot/logging/logback/file-appender.xml" />
    <include resource="co/elastic/logging/logback/boot/ecs-file-appender.xml" />
    <root level="INFO">
        <appender-ref ref="CONSOLE"/>
        <appender-ref ref="ECS_JSON_FILE"/>
        <appender-ref ref="FILE"/>
    </root>
</configuration>

You also need to configure the following properties to your application.properties:

spring.application.name=my-application
# for Spring Boot 2.2.x+
logging.file.name=/path/to/my-application.log
# for older Spring Boot versions
logging.file=/path/to/my-application.log

Other applications

All you have to do is to use the co.elastic.logging.logback.EcsEncoder instead of the default pattern encoder in logback.xml

<encoder class="co.elastic.logging.logback.EcsEncoder">
    <serviceName>my-application</serviceName>
</encoder>

Encoder Parameters

Parameter name Type Default Description

serviceName

String

Sets the service.name field so you can filter your logs by a particular service

eventDataset

String

${serviceName}.log

Sets the event.dataset field used by the machine learning job of the Logs app to look for anomalies in the log rate.

includeMarkers

boolean

false

Log Markers as tags

stackTraceAsArray

boolean

false

Serializes the error.stack_trace as a JSON array where each element is in a new line to improve readability. Note that this requires a slightly more complex Filebeat configuration.

includeOrigin

boolean

false

If true, adds the log.origin.file.name, log.origin.file.line and log.origin.function fields. Note that you also have to set <includeCallerData>true</includeCallerData> on your appenders if you are using the async ones.

To include any custom field in the output, use following syntax:

<additionalField>
    <key>key1</key>
    <value>value1</value>
</additionalField>
<additionalField>
    <key>key2</key>
    <value>value2</value>
</additionalField>

If you’re using the Elastic APM Java agent, log correlation is enabled by default starting in version 1.30.0. In previous versions, log correlation is off by default, but can be enabled by setting the enable_log_correlation config to true.

Step 2: Configure Filebeatedit

Configure your filebeat.inputs as follows:

filebeat.inputs:
- type: log
  paths: /path/to/logs.json
  json.keys_under_root: true
  json.overwrite_keys: true

# no further processing required, logs can directly be sent to Elastic Cloud
cloud.id: "staging:dXMtZWFzdC0xLmF3cy5mb3VuZC5pbyRjZWM2ZjI2MWE3NGJmMjRjZTMzYmI4ODExYjg0Mjk0ZiRjNmMyY2E2ZDA0MjI0OWFmMGNjN2Q3YTllOTYyNTc0Mw=="
cloud.auth: "elastic:YOUR_PASSWORD"

# Or to your local Elasticsearch cluster
#output.elasticsearch:
#  hosts: ["https://localhost:9200"]

For more information, check the Filebeat documentation.

When stackTraceAsArray is enablededit

Filebeat can normally only decode JSON if there is one JSON object per line. When stackTraceAsArray is enabled, there will be a new line for each stack trace element which improves readability. But when combining the multiline settings with a decode_json_fields we can also handle multi-line JSON:

filebeat.inputs:
  - type: log
    paths: /path/to/logs.json
    multiline.pattern: '^{'
    multiline.negate: true
    multiline.match: after
processors:
  - decode_json_fields:
      fields: message
      target: ""
      overwrite_keys: true
  # flattens the array to a single string
  - script:
      when:
        has_fields: ['error.stack_trace']
      lang: javascript
      id: my_filter
      source: >
        function process(event) {
            event.Put("error.stack_trace", event.Get("error.stack_trace").join("\n"));
        }
« Introduction Structured logging with log4j2 »