IMPORTANT: No additional bug fixes or documentation updates
will be released for this version. For the latest information, see the
current release documentation.
Step 2: Exclude all processed files
edit
IMPORTANT: This documentation is no longer updated. Refer to Elastic's version policy and the latest documentation.
Step 2: Exclude all processed files
editFilebeat does not provide access to the state information of different inputs.
Hence, the filestream input cannot access the state information of a log input in the
Filebeat registry. You must exclude the files the log input has processed
or is processing. If you do not exclude those files, you will end up with
duplicate events in the output.
Given the file list and ingestion progress shown earlier,
you should run the log and filestream inputs simultaneously until everything
collected by the log input has made it to the output.
After the files collected by the log input are shipped and the files are deleted,
you can delete the log inputs and the exlude_files settings from filestream input.
filebeat.inputs:
- type: log
enabled: true
paths:
- /var/log/java-exceptions*.log
multiline:
pattern: '^\['
negate: true
match: after
close_removed: true
close_renamed: true
exclude_files: java-exceptions[4-5]{1}.log
- type: log
enabled: true
paths:
- /var/log/my-application*.json
scan_frequency: 1m
json.keys_under_root: true
exclude_files: my-application3.log
- type: filestream
enabled: true
id: my-java-collector
paths:
- /var/log/java-exceptions*.log
prospector.scanner.exclude_files: java-exceptions[1-3]{1}.log
- type: filestream
enabled: true
id: my-application-input
paths:
- /var/log/my-application*.json
prospector.scanner.exclude_files: my-application[1-2]{1}.log
- type: filestream
enabled: true
id: my-old-files
paths:
- /var/log/my-old-files*.log