Get startededit

Step 1: Configure application loggingedit

The following logging frameworks are supported:

  • Logback (default for Spring Boot)
  • Log4j2
  • Log4j
  • java.util.logging (JUL)
  • JBoss Log Manager

Add the dependencyedit

The minimum required logback version is 1.1.

Download the latest version of Elastic logging: Maven Central

Add a dependency to your application:

<dependency>
    <groupId>co.elastic.logging</groupId>
    <artifactId>logback-ecs-encoder</artifactId>
    <version>${ecs-logging-java.version}</version>
</dependency>

If you are not using a dependency management tool, like maven, you have to manually add both logback-ecs-encoder and ecs-logging-core jars to the classpath. For example to the $CATALINA_HOME/lib directory. Other than that, there are no required dependencies.

Use the ECS encoder/formatter/layoutedit

Spring Boot applications

In src/main/resources/logback-spring.xml:

<?xml version="1.0" encoding="UTF-8"?>
<configuration>
    <property name="LOG_FILE" value="${LOG_FILE:-${LOG_PATH:-${LOG_TEMP:-${java.io.tmpdir:-/tmp}}}/spring.log}"/>
    <include resource="org/springframework/boot/logging/logback/defaults.xml"/>
    <include resource="org/springframework/boot/logging/logback/console-appender.xml" />
    <include resource="org/springframework/boot/logging/logback/file-appender.xml" />
    <include resource="co/elastic/logging/logback/boot/ecs-file-appender.xml" />
    <root level="INFO">
        <appender-ref ref="CONSOLE"/>
        <appender-ref ref="ECS_JSON_FILE"/>
        <appender-ref ref="FILE"/>
    </root>
</configuration>

You also need to configure the following properties to your application.properties:

spring.application.name=my-application
# for Spring Boot 2.2.x+
logging.file.name=/path/to/my-application.log
# for older Spring Boot versions
logging.file=/path/to/my-application.log

Other applications

All you have to do is to use the co.elastic.logging.logback.EcsEncoder instead of the default pattern encoder in logback.xml

<encoder class="co.elastic.logging.logback.EcsEncoder">
    <serviceName>my-application</serviceName>
    <serviceNodeName>my-application-cluster-node</serviceNodeName>
</encoder>

Encoder Parameters

Parameter name Type Default Description

serviceName

String

Sets the service.name field so you can filter your logs by a particular service

serviceNodeName

String

Sets the service.node.name field so you can filter your logs by a particular node of your clustered service

eventDataset

String

${serviceName}.log

Sets the event.dataset field used by the machine learning job of the Logs app to look for anomalies in the log rate.

includeMarkers

boolean

false

Log Markers as tags

stackTraceAsArray

boolean

false

Serializes the error.stack_trace as a JSON array where each element is in a new line to improve readability. Note that this requires a slightly more complex Filebeat configuration.

includeOrigin

boolean

false

If true, adds the log.origin.file.name, log.origin.file.line and log.origin.function fields. Note that you also have to set <includeCallerData>true</includeCallerData> on your appenders if you are using the async ones.

To include any custom field in the output, use following syntax:

<additionalField>
    <key>key1</key>
    <value>value1</value>
</additionalField>
<additionalField>
    <key>key2</key>
    <value>value2</value>
</additionalField>

Step 2: Enable APM log correlation (optional)edit

If you are using the Elastic APM Java agent, set enable_log_correlation to true.

Step 3: Configure Filebeatedit

  1. Follow the Filebeat quick start
  2. Add the following configuration to your filebeat.yaml file.

filebeat.yaml.

filebeat.inputs:
- type: log
  paths: /path/to/logs.json
  json.keys_under_root: true
  json.overwrite_keys: true
  json.add_error_key: true
  json.expand_keys: true

For more information, see the Filebeat reference.

When stackTraceAsArray is enablededit

Filebeat can normally only decode JSON if there is one JSON object per line. When stackTraceAsArray is enabled, there will be a new line for each stack trace element which improves readability. But when combining the multiline settings with a decode_json_fields we can also handle multi-line JSON:

filebeat.inputs:
  - type: log
    paths: /path/to/logs.json
    multiline.pattern: '^{'
    multiline.negate: true
    multiline.match: after
processors:
  - decode_json_fields:
      fields: message
      target: ""
      overwrite_keys: true
  # flattens the array to a single string
  - script:
      when:
        has_fields: ['error.stack_trace']
      lang: javascript
      id: my_filter
      source: >
        function process(event) {
            event.Put("error.stack_trace", event.Get("error.stack_trace").join("\n"));
        }