Manage multiline messagesedit
Forwarded content may contain messages that span multiple lines of text. For example, multiline messages are common in files that contain Java stack traces. In order to correctly handle these multiline events, you need to configure multiline settings for a specific input to specify which lines are part of a single event.
Configuration optionsedit
You can specify the following options for a specific input in the config.yaml file to control how the Elastic Serverless Forwarder deals with messages that span multiple lines.
inputs:
- type: "s3-sqs"
id: "arn:aws:sqs:%REGION%:%ACCOUNT%:%QUEUENAME%"
multiline:
type: pattern
pattern: '^\\['
negate: true
match: after
outputs:
- type: "elasticsearch"
args:
elasticsearch_url: "arn:aws:secretsmanager:eu-central-1:123456789:secret:es_url"
username: "arn:aws:secretsmanager:eu-west-1:123456789:secret:es_secrets:username"
password: "arn:aws:secretsmanager:eu-west-1:123456789:secret:es_secrets:password"
es_datastream_name: "logs-generic-default"
The forwarder takes all the lines that do not start with [ and combines them with the previous line that does. For example, you could use this configuration to join the following lines of a multiline message into a single event:
[beat-logstash-some-name-832-2015.11.28] IndexNotFoundException[no such index]
at org.elasticsearch.cluster.metadata.IndexNameExpressionResolver$WildcardExpressionResolver.resolve(IndexNameExpressionResolver.java:566)
at org.elasticsearch.cluster.metadata.IndexNameExpressionResolver.concreteIndices(IndexNameExpressionResolver.java:133)
at org.elasticsearch.cluster.metadata.IndexNameExpressionResolver.concreteIndices(IndexNameExpressionResolver.java:77)
at org.elasticsearch.action.admin.indices.delete.TransportDeleteIndexAction.checkBlock(TransportDeleteIndexAction.java:75)
Note that you should escape the opening square bracket ([) in the regular expression, because it specifies a character class i.e. a set of characters that you wish to match. You also have to escape the backslash (\) used for escaping the opening square bracket as raw strings are not used. Thus, ^\\[ will produce the required regular expression upon compiling.
inputs.[].multiline.type defines which aggregation method to use. The default is pattern. The other options are count, which enables you to aggregate a constant number of lines, and while_pattern, which aggregates lines by pattern without matching options.
inputs.[].multiline.pattern differs from the patterns supported by Logstash. See Python’s 3.9 regular expression syntax for a list of supported regexp patterns. Depending on how you configure other multiline options, lines that match the specified regular expression are considered either continuations of a previous line or the start of a new multiline event.
inputs.[].multiline.negate defines whether the pattern is negated. The default is false. This setting works only with pattern and while_pattern types.
inputs.[].multiline.match changes the grouping of multiple lines according to the schema below (works only with pattern type):
Setting for |
Setting for |
Result |
Example |
|
|
Consecutive lines that match the pattern are appended to the previous line that doesn’t match. |
|
|
|
Consecutive lines that match the pattern are prepended to the next line that doesn’t match. |
|
|
|
Consecutive lines that don’t match the pattern are appended to the previous line that does match. |
|
|
|
Consecutive lines that don’t match the pattern are prepended to the next line that does match. |
|
The after setting is equivalent to previous in Logstash, and before is equivalent to next.
inputs.[].multiline.flush_pattern specifies a regular expression, in which the current multiline will be flushed from memory, ending the multiline-message. Works only with pattern type.
inputs.[].multiline.max_lines defines the maximum number of lines that can be combined into one event. If the multiline message contains more than max_lines, any additional lines are truncated from the event. The default is 500.
inputs.[].multiline.max_bytes defines the maximum number of bytes that can be combined into one event. If the multiline message contains more than max_bytes, any additional content is truncated from the event. The default is 10485760.
inputs.[].multiline.count_lines defines the number of lines to aggregate into a single event. Works only with count type.
inputs.[].multiline.skip_newline defined whether multiline events must be concatenated, stripping the line separator. If set to true, the line separator will be stripped. The default is false.
Examples of multiline configurationedit
The examples in this section cover the following use cases:
- Combining a Java stack trace into a single event
- Combining C-style line continuations into a single event
- Combining multiple lines from time-stamped events
Java stack tracesedit
Java stack traces consist of multiple lines, with each line after the initial line beginning with whitespace, as in this example:
Exception in thread "main" java.lang.NullPointerException
at com.example.myproject.Book.getTitle(Book.java:16)
at com.example.myproject.Author.getBookTitles(Author.java:25)
at com.example.myproject.Bootstrap.main(Bootstrap.java:14)
This configuration merges any line that begins with whitespace up to the previous line:
multiline: type: pattern pattern: '^\s' negate: false match: after
This is a slightly more complex Java stack trace example:
Exception in thread "main" java.lang.IllegalStateException: A book has a null property
at com.example.myproject.Author.getBookIds(Author.java:38)
at com.example.myproject.Bootstrap.main(Bootstrap.java:14)
Caused by: java.lang.NullPointerException
at com.example.myproject.Book.getId(Book.java:22)
at com.example.myproject.Author.getBookIds(Author.java:35)
... 1 more
To consolidate these lines into a single event, use the following multiline configuration:
multiline:
type: pattern
pattern: '^\s+(at|.{3})\s+\\b|^Caused by:'
negate: false
match: after
In this example, the pattern matches and merges the following lines:
- a line that begins with spaces followed by the word at or ...
- a line that begins with the words Caused by:
In Python’s string literals, \b is the backspace character (ASCII value 8). As raw strings are not used, Python would convert the \b to a backspace. In order for our regular expression to match as expected, you need to escape the backslash \ in \b to \\b, which will produce the correct regular expression upon compiling.
Line continuationsedit
Several programming languages use the backslash (\) character at the end of a line to denote that the line continues, as in this example:
printf ("%10.10ld \t %10.10ld \t %s\
%f", w, x, y, z );
To consolidate these lines into a single event, use the following multiline configuration:
multiline: type: pattern pattern: '\\\\$' negate: false match: after
This configuration merges any line that ends with the \ character with the line that follows it.
Note that you should escape the opening backslash (\) twice in the regular expression, as raw strings are not used. Thus, \\\\$ will produce the required regular expression upon compiling.
Timestampsedit
Activity logs from services such as Elasticsearch typically begin with a timestamp, followed by information on the specific activity, as in this example:
[2015-08-24 11:49:14,389][INFO ][env ] [Letha] using [1] data paths, mounts [[/ (/dev/disk1)]], net usable_space [34.5gb], net total_space [118.9gb], types [hfs]
To consolidate these lines into a single event, use the following multiline configuration:
multiline:
type: pattern
pattern: '^\\[[0-9]{4}-[0-9]{2}-[0-9]{2}'
negate: true
match: after
This configuration uses the negate: true and match: after settings to specify that any line that does not match the specified pattern belongs to the previous line.
Note that you should escape the opening square bracket ([) in the regular expression, because it specifies a character class i.e. a set of characters that you wish to match. You also have to escape the backslash (\) used for escaping the opening square bracket as raw strings are not used. Thus, ^\\[ will produce the required regular expression upon compiling.
Application eventsedit
Sometimes your application logs contain events, that begin and end with custom markers, such as the following example:
[2015-08-24 11:49:14,389] Start new event [2015-08-24 11:49:14,395] Content of processing something [2015-08-24 11:49:14,399] End event
To consolidate these lines into a single event, use the following multiline configuration:
multiline: type: pattern pattern: 'Start new event' negate: true match: after flush_pattern: 'End event'
The flush_pattern option specifies a regex at which the current multiline will be flushed. If you think of the pattern option specifying the beginning of an event, the flush_pattern option will specify the end or last line of the event.
This example will not work correctly if start/end log blocks are mixed with non-multiline logs, or if different start/end log blocks overlap with each other. For instance, Some other log log lines in the following example will be merged into a single multiline document because they neither match inputs.[].multiline.pattern nor inputs.[].multiline.flush_pattern, and inputs.[].multiline.negate is set to true.
[2015-08-24 11:49:14,389] Start new event [2015-08-24 11:49:14,395] Content of processing something [2015-08-24 11:49:14,399] End event [2015-08-24 11:50:14,389] Some other log [2015-08-24 11:50:14,395] Some other log [2015-08-24 11:50:14,399] Some other log [2015-08-24 11:51:14,389] Start new event [2015-08-24 11:51:14,395] Content of processing something [2015-08-24 11:51:14,399] End event
Test your regexp pattern for multilineedit
To make it easier for you to test the regexp patterns in your multiline config, you can use this Multiline Regexp Test:
-
Plug in the regexp pattern at line
3, along with themultiline.negatesetting that you plan to use at line4 -
Paste a sample message between the three double quote delimiters (
""" """) at line5 -
Click
Runto see which lines in the message match your specified configuration.
