filestream input has been generally available since 7.14 and it is highly recommended
you migrate your existing
log input configurations. The
filestream input comes with many
improvements over the old
log input, such as configurable order for parsers and more.
log input is deprecated and will eventually be removed from Filebeat. We are not fixing
new issues or adding any enhancements to the
log input. Our focus is on
This manual migration is required only if you’ve defined
log inputs manually
in your stand-alone Filebeat configuration. All the integrations or modules that are still using
log inputs under the hood will be eventually migrated automatically without any additional actions
required from the user.
In this guide, you’ll learn how to migrate an existing
log input configuration.
You must replace
log inputs with
filestream inputs, make sure you have removed
all the old
log inputs from the configuration before starting Filebeat with the new
filestream inputs. Running old
log inputs and new
filestream inputs pointed to the same files will lead to data duplication.
The following example shows three
filebeat.inputs: - type: log enabled: true paths: - /var/log/java-exceptions*.log multiline: pattern: '^\[' negate: true match: after close_removed: true close_renamed: true - type: log enabled: true paths: - /var/log/my-application*.json scan_frequency: 1m json.keys_under_root: true - type: log enabled: true paths: - /var/log/my-old-files*.log tail_files: true
For this example, let’s assume that the
log input is used to collect logs from the following files. The progress of data collection is shown for each file.
/var/log/java-exceptions1.log (100%) /var/log/java-exceptions2.log (100%) /var/log/java-exceptions3.log (75%) /var/log/java-exceptions4.log (0%) /var/log/java-exceptions5.log (0%) /var/log/my-application1.json (100%) /var/log/my-application2.json (5%) /var/log/my-application3.json (0%) /var/log/my-old-files1.json (0%)
Intro to Kibana
ELK for Logs & Metrics