APM agent ECS reformatting with Filebeatedit

Elastic APM agents can automatically reformat application logs to Elastic Common Schema (ECS) format without needing to add an ECS logger dependency or modify the application.

Requirements

  • The Elastic APM agent for your programming language
  • Filebeat configured to monitor and capture application logs

Pros

All the benefits of using ECS logging, without having to modify the application or its configuration:

  • Simplicity: no manual parsing with Filebeat, and a configuration can be reused across applications
  • Decently human-readable JSON structure
  • APM log correlation

Cons

  • Requires an Elastic APM agent
  • Not all APM agents support this feature

Supported APM agents/languages

  • Ruby
  • Java

Step 1: Enable APM agent reformattingedit

Enable APM agent ECS reformatting in the supported APM agent:

See the Ruby or Java agent docs.

Step 2: Set up Filebeatedit

  1. Follow the Filebeat quick start to learn how to install Filebeat and connect to the Elastic Stack.
  2. Add the following configuration to your filebeat.yaml file to start collecting log data.

    filebeat.yaml.

    filebeat.inputs:
    - type: filestream 
      paths: /path/to/logs.json
      parsers:
        - ndjson:
          keys_under_root: true
          overwrite_keys: true
          add_error_key: true
          expand_keys: true
    
    processors:
      - add_host_metadata: ~
      - add_cloud_metadata: ~
      - add_docker_metadata: ~
      - add_kubernetes_metadata: ~

    Use the filestream input to read lines from active log files.