WARNING: Version 5.0 of Filebeat has passed its EOL date.
This documentation is no longer being maintained and may be removed. If you are running this version, we strongly advise you to upgrade. For the latest information, see the current release documentation.
When your use case requires only a subset of the data exported by Filebeat or you need to add metadata, you can use Filebeat config options to filter the data, or you can define processors.
Filebeat Config Options for Filteringedit
You can specify configuration options in the
filebeat section of the config file to define regular expressions that
match the lines you want to include and/or exclude from the output. The supported options are
For example, you can use the
include_lines option to export any lines that start with "ERR" or "WARN":
filebeat.prospectors: - input_type: log paths: - /var/log/myapp/*.log include_lines: ["^ERR", "^WARN"]
The disadvantage of this approach is that you need to implement a configuration option for each filtering criteria that you need.
See Filebeat configuration options for more information about each option.
You can define processors in your configuration to process events before they are sent to the configured output. The libbeat library provides processors for reducing the number of exported fields, and processors for enhancing events with additional metadata. Each processor receives an event, applies a defined action to the event, and returns the event. If you define a list of processors, they are executed in the order they are defined in the configuration file.
event -> processor 1 -> event1 -> processor 2 -> event2 ...
The processors are defined in the Filebeat configuration file.
For example, the following configuration drops all the DEBUG messages.
processors: - drop_event: when: regexp: message: "^DBG:"
To drop all the log messages coming from a certain log file:
processors: - drop_event: when: contains: source: "test"
See Processors Configuration for more information.
Intro to Kibana
ELK for Logs & Metrics