Use tags and filtersedit
You can use tags and filters to tag and filter messages based on regular expressions.
Use custom tagsedit
You can add custom tags to filter and categorize items in events.
inputs:
- type: "s3-sqs"
id: "arn:aws:sqs:%REGION%:%ACCOUNT%:%QUEUENAME%"
tags:
- "tag1"
- "tag2"
- "tag3"
outputs:
- type: "elasticsearch"
args:
elasticsearch_url: "arn:aws:secretsmanager:eu-central-1:123456789:secret:es_url"
username: "arn:aws:secretsmanager:eu-west-1:123456789:secret:es_secrets:username"
password: "arn:aws:secretsmanager:eu-west-1:123456789:secret:es_secrets:password"
es_datastream_name: "logs-generic-default"
Using the above example configuration, the tags will be set in the following way:
["forwarded", "generic", "tag1", "tag2", "tag3"]
The forwarded tag is always appended and the generic tag in this example comes from the dataset.
-
Tags must be defined within
inputsin theconfig.yamlfile. - Each tag must be a string and added to the list.
Define include/exclude filtersedit
You can define multiple filters for inputs to include or exclude events from data ingestion.
inputs:
- type: "s3-sqs"
id: "arn:aws:sqs:%REGION%:%ACCOUNT%:%QUEUENAME%"
include:
- "[a-zA-Z]"
exclude:
- "skip this"
- "skip also this"
outputs:
- type: "elasticsearch"
args:
elasticsearch_url: "arn:aws:secretsmanager:eu-central-1:123456789:secret:es_url"
username: "arn:aws:secretsmanager:eu-west-1:123456789:secret:es_secrets:username"
password: "arn:aws:secretsmanager:eu-west-1:123456789:secret:es_secrets:password"
es_datastream_name: "logs-generic-default"
You can define a list of regular expressions within inputs.[].include. If this list is populated, only messages matching any of the defined regular expressions will be forwarded to the outputs.
You can define a list of regular expressions within inputs.[].exclude. If this list is populated, only messages not matching any of the defined regular expressions will be forwarded to the outputs i.e. every message will be forwarded to the outputs unless it matches any of the defined regular expressions.
Both config parameters are optional, and can be set independently of each other. In terms of rule precedence, the exclude filter is applied first and then the include filter, so exclude takes precedence if both are specified.
All regular expressions are case-sensitive and should follow Python’s 3.9 regular expression syntax.
Messages are scanned for terms that match the defined filters. Use the ^ (caret) special character to explicitly anchor the regex to the position before the first character of the string, and use $ to anchor at the end.
No flags are used when the regular expression is compiled. Please refer to inline flag documentation for alternative options for multiline, case-insensitive, and other matching behaviors.