JSON content discoveryedit
The Elastic Serverless Forwarder is able to automatically discover JSON content in the payload of an input and collect the JSON objects contained in the payload.
The JSON objects can either be on a single line or spanning multiple lines. In the second case, the forwarder expects different JSON objects spanning multiple lines to be separated by a newline delimiter.
When JSON objects span multiple lines, a limit of 1000 lines is applied. Every JSON object spanning across more than 1000 lines will not be collected. Every line composing the whole JSON object will be forwarded individually instead.
If you have known payload content which includes single JSON objects that span more than 1000 lines, or if you find that relying on auto-discovery of JSON content has a big impact on performance, you can configure JSON content types within the inputs to address this. This will change the parsing logic and improve performance while overcoming the 1000 lines limit.
Where content is known to be plain text, you can improve overall performance by disabling automatic JSON content discovery completely.
To change this configuration option, set inputs.[].json_content_type to one of the following values:
- single: indicates that the content of a single item in the input payload is a single JSON object. The content can either be on a single line or spanning multiple lines. With this setting the whole content of the payload is decoded as a JSON object, with no limit on the number of lines the JSON object spans.
- ndjson: indicates that the content of a single item in the input payload is a valid NDJSON format. Multiple single JSON objects formatted on a single line should be separated by a newline delimiter. With this setting each line will be decoded as JSON object, which improves the parsing performance.
- disabled: instructs the forwarder not to attempt any automatic JSON content discovery and instead treat the content as plain text, which improves the parsing performance.
There is no need to configure the JSON content type when Expanding events from JSON object lists, unless you have single JSON objects that span more than 1000 lines.
Expanding events from JSON object listsedit
You can extract a list of events to be ingested from a specific field in the JSON file.
inputs:
- type: "s3-sqs"
id: "arn:aws:sqs:%REGION%:%ACCOUNT%:%QUEUENAME%"
expand_event_list_from_field: "Records"
outputs:
- type: "elasticsearch"
args:
elasticsearch_url: "arn:aws:secretsmanager:eu-central-1:123456789:secret:es_url"
username: "arn:aws:secretsmanager:eu-west-1:123456789:secret:es_secrets:username"
password: "arn:aws:secretsmanager:eu-west-1:123456789:secret:es_secrets:password"
es_datastream_name: "logs-generic-default"
You can define inputs.[].expand_event_list_from_field as a string with the value of a key in the JSON that contains a list of elements that must be sent as events instead of the encompassing JSON.
When routing service logs, any value set for the expand_event_list_from_field configuration parameter will be ignored, because this will be automatically handled by the Elastic Serverless Forwarder.
Exampleedit
With the following input:
{"Records":[{"key": "value #1"},{"key": "value #2"}]}
{"Records":[{"key": "value #3"},{"key": "value #4"}]}
Without setting expand_event_list_from_field, two events will be forwarded:
{"@timestamp": "2022-06-16T04:06:03.064Z", "message": "{\"Records\":[{\"key\": \"value #1\"},{\"key\": \"value #2\"}]}"}
{"@timestamp": "2022-06-16T04:06:13.888Z", "message": "{\"Records\":[{\"key\": \"value #3\"},{\"key\": \"value #4\"}]}"}
If expand_event_list_from_field is set to Records, four events will be forwarded:
{"@timestamp": "2022-06-16T04:06:21.105Z", "message": "{\"key\": \"value #1\"}"}
{"@timestamp": "2022-06-16T04:06:27.204Z", "message": "{\"key\": \"value #2\"}"}
{"@timestamp": "2022-06-16T04:06:31.154Z", "message": "{\"key\": \"value #3\"}"}
{"@timestamp": "2022-06-16T04:06:36.189Z", "message": "{\"key\": \"value #4\"}"}