Plaintext logs with Filebeatedit

Use Filebeat to parse and ingest raw, plain-text application logs.

Requirements

  • (Optional) Elastic APM agent for your programming language (for log correlation)
  • Raw, plain-text application logs stored on the file system
  • Filebeat configured to monitor and capture application logs

Pros

  • All programming languages/frameworks are supported
  • Existing application logs can be ingested
  • Does not require modification of the application or its configuration, unless log correlation is required

Cons

  • Must parse application logs to be useful—meaning writing and maintaining Grok patterns and spending CPU cycles on parsing
  • Parsing is tied to the application log format, meaning it can differ per application and needs to be maintained over time
  • Log correlation requires modifying the application log format and inject IDs in log messages

Step 1: Use Filebeat to ingest logsedit

  1. Follow the Filebeat quick start to learn how to install Filebeat and connect to the Elastic Stack.
  2. Configure filebeat.yaml file to start collecting log data.
  3. Add the following configuration to your filebeat.yaml file to start collecting log data.

    filebeat.yaml.

    filebeat.inputs:
    - type: filestream 
      paths: /path/to/logs.log

Step 2: Parse logs at ingest timeedit

A downside of plaintext logs is that you can’t aggregate or search on the fields within the logs. To enable these features, you’ll need to parse the contents of your logs into ECS-compatible fields.

To learn how to use the Grok processor to parse application logs before indexing, see Example: Parse logs in the Common Log Format.

Learn moreedit