Ingest logs with Filebeat
editIngest logs with Filebeat
editIf you haven’t already, you need to install Elasticsearch for storing and searching your data, and Kibana for visualizing and managing it. For more information, see Spin up the Elastic Stack.
Install and configure Filebeat on your servers to collect log events. Filebeat allows you ship log data from sources that come in the form of files. It monitors the log files or locations that you specify, collects log events, and forwards them to Elasticsearch. To ease the collection and parsing of log formats for common applications such as Apache, MySQL, and Kafka, a number of modules are available.
Step 1: Install Filebeat
editInstall Filebeat on all the servers you want to monitor.
To download and install Filebeat, use the commands that work with your system:
curl -L -O https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-8.7.1-amd64.deb sudo dpkg -i filebeat-8.7.1-amd64.deb
curl -L -O https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-8.7.1-x86_64.rpm sudo rpm -vi filebeat-8.7.1-x86_64.rpm
curl -L -O https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-8.7.1-darwin-x86_64.tar.gz tar xzvf filebeat-8.7.1-darwin-x86_64.tar.gz
curl -L -O https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-8.7.1-linux-x86_64.tar.gz tar xzvf filebeat-8.7.1-linux-x86_64.tar.gz
- Download the Filebeat Windows zip file from the downloads page.
-
Extract the contents of the zip file into
C:\Program Files
. -
Rename the
filebeat-<version>-windows
directory toFilebeat
. - Open a PowerShell prompt as an Administrator (right-click the PowerShell icon and select Run As Administrator).
-
From the PowerShell prompt, run the following commands to install Filebeat as a Windows service:
PS > cd 'C:\Program Files\Filebeat' PS C:\Program Files\Filebeat> .\install-service-filebeat.ps1
If script execution is disabled on your system, you need to set the
execution policy for the current session to allow the script to run. For
example:
PowerShell.exe -ExecutionPolicy UnRestricted -File .\install-service-filebeat.ps1
.
Other installation options
editStep 2: Connect to Elasticsearch and Kibana
editConnections to Elasticsearch and Kibana are required to set up Filebeat.
Set the connection information in filebeat.yml
. To locate this configuration file,
see Directory layout.
Specify the cloud.id of your Elasticsearch Service, and set cloud.auth to a user who is authorized to set up Filebeat. For example:
cloud.id: "staging:dXMtZWFzdC0xLmF3cy5mb3VuZC5pbyRjZWM2ZjI2MWE3NGJmMjRjZTMzYmI4ODExYjg0Mjk0ZiRjNmMyY2E2ZDA0MjI0OWFmMGNjN2Q3YTllOTYyNTc0Mw==" cloud.auth: "filebeat_setup:YOUR_PASSWORD"
This examples shows a hard-coded password, but you should store sensitive values in the secrets keystore. |
-
Set the host and port where Filebeat can find the Elasticsearch installation, and set the username and password of a user who is authorized to set up Filebeat. For example:
output.elasticsearch: hosts: ["https://myEShost:9200"] username: "filebeat_internal" password: "YOUR_PASSWORD" ssl: enabled: true ca_trusted_fingerprint: "b9a10bbe64ee9826abeda6546fc988c8bf798b41957c33d05db736716513dc9c"
This example shows a hard-coded password, but you should store sensitive values in the secrets keystore.
This example shows a hard-coded fingerprint, but you should store sensitive values in the secrets keystore. The fingerprint is a HEX encoded SHA-256 of a CA certificate, when you start Elasticsearch for the first time, security features such as network encryption (TLS) for Elasticsearch are enabled by default. If you are using the self-signed certificate generated by Elasticsearch when it is started for the first time, you will need to add its fingerprint here. The fingerprint is printed on Elasticsearch start up logs, or you can refer to connect clients to Elasticsearch documentation for other options on retrieving it. If you are providing your own SSL certificate to Elasticsearch refer to Filebeat documentation on how to setup SSL.
-
If you plan to use our pre-built Kibana dashboards, configure the Kibana endpoint. Skip this step if Kibana is running on the same host as Elasticsearch.
The hostname and port of the machine where Kibana is running, for example,
mykibanahost:5601
. If you specify a path after the port number, include the scheme and port:http://mykibanahost:5601/path
.The
username
andpassword
settings for Kibana are optional. If you don’t specify credentials for Kibana, Filebeat uses theusername
andpassword
specified for the Elasticsearch output.To use the pre-built Kibana dashboards, this user must be authorized to view dashboards or have the
kibana_admin
built-in role.
To learn more about required roles and privileges, see Grant users access to secured resources.
You can send data to other outputs, such as Logstash, but that requires additional configuration and setup.
Step 3: Enable and configure modules
editFilebeat uses modules to collect and parse log data.
-
Identify the modules you need to enable. To see a list of available modules, run:
filebeat modules list
filebeat modules list
./filebeat modules list
./filebeat modules list
PS > .\filebeat.exe modules list
Can’t find a module for your file type? Skip this section and configure the input manually.
-
From the installation directory, enable one or more modules. For example, the following command enables the
nginx
module config:filebeat modules enable nginx
filebeat modules enable nginx
./filebeat modules enable nginx
./filebeat modules enable nginx
PS > .\filebeat.exe modules enable nginx
-
In the module config under
modules.d
, change the module settings to match your environment. You must enable at least one fileset in the module. Filesets are disabled by default.For example, log locations are set based on the OS. If your logs aren’t in default locations, set the
paths
variable:
To see the full list of variables for a module, see the documentation under Modules.
To test your configuration file, change to the directory where the Filebeat binary
is installed, and run Filebeat in the foreground with the following options specified:
./filebeat test config -e
. Make sure your config files are in the path expected by
Filebeat (see Directory layout),
or use the -c
flag to specify the path to the config file.
For more information about configuring Filebeat, also see:
- Configure Filebeat
- Config file format
-
filebeat.reference.yml
: This reference configuration file shows all non-deprecated options. You’ll find it in the same location asfilebeat.yml
.
Step 4: Set up assets
editFilebeat comes with predefined assets for parsing, indexing, and visualizing your data. To load these assets:
-
Make sure the user specified in
filebeat.yml
is authorized to set up Filebeat. -
From the installation directory, run:
filebeat setup -e
filebeat setup -e
./filebeat setup -e
./filebeat setup -e
PS > .\filebeat.exe setup -e
-e
is optional and sends output to standard error instead of the configured log output.
This step loads the recommended index template for writing to Elasticsearch and deploys the sample dashboards for visualizing the data in Kibana.
This step does not load the ingest pipelines used to parse log lines. By default, ingest pipelines are set up automatically the first time you run the module and connect to Elasticsearch.
A connection to Elasticsearch (or Elasticsearch Service) is required to set up the initial environment. If you’re using a different output, such as Logstash, see:
Step 5: Start Filebeat
editBefore starting Filebeat, modify the user credentials in
filebeat.yml
and specify a user who is
authorized to publish events.
To start Filebeat, run:
sudo service filebeat start
If you use an init.d
script to start Filebeat, you can’t specify command
line flags (see Command reference). To specify flags, start Filebeat in
the foreground.
Also see Filebeat and systemd.
sudo service filebeat start
If you use an init.d
script to start Filebeat, you can’t specify command
line flags (see Command reference). To specify flags, start Filebeat in
the foreground.
Also see Filebeat and systemd.
You’ll be running Filebeat as root, so you need to change ownership of the
configuration file and any configurations enabled in the |
You’ll be running Filebeat as root, so you need to change ownership of the
configuration file and any configurations enabled in the |
PS C:\Program Files\filebeat> Start-Service filebeat
By default, Windows log files are stored in C:\ProgramData\filebeat\Logs
.
Filebeat should begin streaming events to Elasticsearch.
Step 6: Confirm logs are streaming
editLet’s confirm your data is correctly streaming to your cloud instance.
-
Launch Kibana:
- Log in to your Elastic Cloud account.
- Navigate to the Kibana endpoint in your deployment.
Point your browser to http://localhost:5601, replacing
localhost
with the name of the Kibana host. - Open the main menu, then click Discover.
-
Select
filebeat-*
as your data view.Each document in the index that matches the
filebeat-*
data view is displayed. By default, Discover shows data for the last 15 minutes. If you have a time-based index, and no data displays, you might need to increase the time range.You can now search your log messages, filter your search results, add or remove fields, examine the document contents in either table or JSON format, and view a document in context.
Now let’s have a look at the Logs app.