WARNING: Version 5.5 of Filebeat has passed its EOL date.
This documentation is no longer being maintained and may be removed. If you are running this version, we strongly advise you to upgrade. For the latest information, see the current release documentation.
Tutorialedit
This tutorial assumes you have Elasticsearch and Kibana installed and accessible from Filebeat (see the getting started section). It also assumes that the Ingest Node GeoIP and User Agent plugins are installed. These plugins are required to capture the geographical location and browser information used by some of the visualizations available in the sample dashboards. You can install these plugins by running the following commands in the Elasticsearch home path:
sudo bin/elasticsearch-plugin install ingest-geoip sudo bin/elasticsearch-plugin install ingest-user-agent
You need to restart Elasticsearch after running these commands.
If you are using an Elastic Cloud instance, you can enable the two plugins from the configuration page.
This also assumes you have Nginx installed and writing logs in the default location and format. If you want to monitor another service for which a module exists, adjust the commands in the tutorial accordingly.
You can start Filebeat with the following command:
./filebeat -e -modules=nginx -setup
The -e flag tells Filebeat to output its logs to standard error, instead of
syslog.
The -modules=nginx flag loads the Nginx module.
The -setup flag tells Filebeat to load the associated sample Kibana
dashboards. This setup phase, in which the dashboards are loaded, doesn’t have
to be executed each time, and because it’s a relatively heavy operation, we
recommend executing it only once after installing or upgrading Filebeat. That
is why, the next commands from this tutorial are omitting the -setup flag.
Visiting the Kibana web interface now, open the Nginx dashboard and you should already see your logs parsed and visualized in several widgets.
You can also start multiple modules at once:
./filebeat -e -modules=nginx,mysql,system
Because Filebeat modules are currently in Beta, the default Filebeat
configuration may interfere with the Filebeat system module configuration. If
you plan to run the system module, edit the Filebeat configuration file,
filebeat.yml, and comment out the following lines:
#- input_type: log
#paths:
#- /var/log/*.log
For rpm and deb, you’ll find the configuration file at
/etc/filebeat/filebeat.yml. For mac and win, look in the archive that you
extracted when you installed Filebeat.
While enabling the modules from the CLI file is handy for getting started and for testing, you will probably want to use the configuration file for the production setup. The equivalent of the above in the configuration file is:
filebeat.modules: - module: nginx - module: mysql - module: system
Then you can start Filebeat simply with: ./filebeat -e.
Variable overridesedit
Each module and fileset has a set of "variables" which allow adjusting their
behaviour. To see the available variables, you can consult the
filebeat.full.yml file. For example, all filesets allow setting a custom
paths value, which is a list of Globs where the log files are searched.
These variables have default values, sometimes depending on the operating
system. You can override them either from the CLI via the -M flag, or from
the configuration file.
In the case of Nginx, for example, you can use the following if the access files are in a custom location:
./filebeat -e -modules=nginx -M "nginx.access.var.paths=[/var/log/nginx/access.log*]"
Or via the configuration file:
filebeat.modules:
- module: nginx
access:
var.paths: ["/var/log/nginx/access.log*"]
Advanced settingsedit
Behind the scenes, each module starts a Filebeat prospector. For advanced users, it’s possible to add or overwrite any of the prospector settings. For example, enabling close_eof can be done like this:
filebeat.modules:
- module: nginx
access:
prospector:
close_eof: true
Or like this:
./filebeat -e -modules=nginx -M "nginx.access.prospector.close_eof=true"
From the CLI, it’s possible to change variables or settings for multiple
modules/fileset at once. For example, the following works and will enable
close_eof for all the filesets in the nginx module:
./filebeat -e -modules=nginx -M "nginx.*.prospector.close_eof=true"
The following also works and will enable close_eof for all prospectors
created by any of the modules:
./filebeat -e -modules=nginx,mysql -M "*.*.prospector.close_eof=true"