The files harvested by Filebeat may contain messages that span multiple
lines of text. For example, multiline messages are common in files that contain
Java stack traces. In order to correctly handle these multiline events, you need
multiline settings in the
filebeat.yml file to specify
which lines are part of a single event.
If you are sending multiline events to Logstash, use the options described here to handle multiline events before sending the event data to Logstash. Trying to implement multiline event handling in Logstash (for example, by using the Logstash multiline codec) may result in the mixing of streams and corrupted data.
Also read YAML tips and gotchas and Regular expression support to avoid common mistakes.
You can specify the following options in the
filebeat.inputs section of
filebeat.yml config file to control how Filebeat deals with messages
that span multiple lines.
The following example shows how to configure Filebeat to handle a multiline message where the first line of the message begins with a bracket (
multiline.pattern: '^\[' multiline.negate: true multiline.match: after
Filebeat takes all the lines that do not start with
[ and combines them with the previous line that does. For example, you could use this configuration to join the following lines of a multiline message into a single event:
[beat-logstash-some-name-832-2015.11.28] IndexNotFoundException[no such index] at org.elasticsearch.cluster.metadata.IndexNameExpressionResolver$WildcardExpressionResolver.resolve(IndexNameExpressionResolver.java:566) at org.elasticsearch.cluster.metadata.IndexNameExpressionResolver.concreteIndices(IndexNameExpressionResolver.java:133) at org.elasticsearch.cluster.metadata.IndexNameExpressionResolver.concreteIndices(IndexNameExpressionResolver.java:77) at org.elasticsearch.action.admin.indices.delete.TransportDeleteIndexAction.checkBlock(TransportDeleteIndexAction.java:75)
Specifies the regular expression pattern to match. Note that the regexp patterns supported by Filebeat
differ somewhat from the patterns supported by Logstash. See Regular expression support for a list of supported regexp patterns.
Depending on how you configure other multiline options, lines that match the specified regular expression are considered
either continuations of a previous line or the start of a new multiline event. You can set the
negateoption to negate the pattern.
Defines whether the pattern is negated. The default is
Specifies how Filebeat combines matching lines into an event. The settings are
before. The behavior of these settings depends on what you specify for
Consecutive lines that match the pattern are appended to the previous line that doesn’t match.
Consecutive lines that match the pattern are prepended to the next line that doesn’t match.
Consecutive lines that don’t match the pattern are appended to the previous line that does match.
Consecutive lines that don’t match the pattern are prepended to the next line that does match.
aftersetting is equivalent to
previousin Logstash, and
beforeis equivalent to
- Specifies a regular expression, in which the current multiline will be flushed from memory, ending the multiline-message.
The maximum number of lines that can be combined into one event. If
the multiline message contains more than
max_lines, any additional lines are discarded. The default is 500.
- After the specified timeout, Filebeat sends the multiline event even if no new pattern is found to start a new event. The default is 5s.
Intro to Kibana
ELK for Logs & Metrics