Configure Beats (without Cloud ID) | Elasticsearch Service Documentation

› › ›

« Configure Beats and Logstash with Cloud ID Configure Logstash (without Cloud ID) »

Configure Beats (without Cloud ID)edit

These instructions apply to versions of Beats before 6.0. If you are using Beats version 6.0 or later, try your Cloud ID that makes configuring Beats quick and simple. To learn more, refer to Configure Beats and Logstash with Cloud ID.

For versions of Beats before 6.0: Beats are lightweight data shippers that you install as agents on your servers to send specific types of operational data to Elasticsearch. Different Beats are available for different purposes, such as Packetbeat, Metricbeat, Winlogbeat, and Filebeat, each supported on different operating systems. All of these Beats can be configured to send events to a cluster in Elasticsearch Service, where you can analyze the data with Kibana.

Configuring your Beats agents to send events to your cluster in the Elasticsearch Service is easy. At a minimum, you need to install and configure one of the Beats on a server somewhere and have an Elasticsearch cluster in Elasticsearch Service that you can send operational data to.

To send Beats events to a secured cluster in Elasticsearch Service, you should use a user that can manage index templates, monitor the cluster, create indices, and read and write to the indices it creates. For more information on Beats users with the right privileges, check Beats and Security.

Example: Send Filebeat events to Elasticsearch Serviceedit

The following example shows how you can send operational data from Filebeat 5.6 running on a CentOS 7.1 server to a 5.6 cluster hosted on Elasticsearch Service.

If you want to learn more about how Filebeat works, check Getting started with Filebeat.

Different versions of Filebeat and different operating systems require slightly different installation instructions. Make sure you follow the right instructions if your intended setup differs from our example.

To get started with Filebeat and Elasticsearch Service:

Install Filebeat on the CentOS 7.1 server where you want to collect operational data:

curl -L -O https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-5.6.16-x86_64.rpm
sudo rpm -vi filebeat-5.6.16-x86_64.rpm

Configure Filebeat by editing the /etc/filebeat/filebeat.yml file. At a minimum, you need to point Filebeat to your Elasticsearch cluster endpoint in Elasticsearch Service. You can find the required cluster endpoint information for your deployment on the overview page in the Elasticsearch Service Console.

In our example, we’re going to use HTTPS to connect to a cluster in Elasticsearch Service that has Security (formerly Shield) enabled, so we need to specify the host with a port, the HTTPS protocol, and authentication credentials:
```
#-------------------------- Elasticsearch output ------------------------------
output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["mycluster.us-east-1.aws.found.io:9243"]

  # Optional protocol and basic auth credentials.
  protocol: "https"
  username: "filebeat_internal"
  password: "xxxxxxxxxxxxxxxxxx"
```
If you make no other change, Filebeat collects the *.log files in /var/log/. For our example, we’ll leave this default alone, but there are quite a few supported configuration options. If you want to learn more about these options, refer to Configuration Options (Reference).
Test your configuration. If configured correctly, you should receive output that includes your Elasticsearch cluster endpoint URL and a Config OK message:
```
sudo ./filebeat -configtest -e

...
2016/10/13 15:32:55.290375 client.go:107: INFO Elasticsearch url: https://mycluster.us-east-1.aws.found.io:9243
...
Config OK
```
If you receive errors during the configuration test, make sure that you issue the command from the file path where Filebeat is installed and that your templates and configuration file are in the Config path expected by Filebeat. You might need to copy these files into the right location.
Optional: Configure which index template file to load. For this example, the default filebeat.template.json installed by the Filebeat package is used. If you want to learn more about specifying a different template, check Configuring Template Loading.
Start Filebeat:
```
sudo /etc/init.d/filebeat start
```
If it isn’t enabled already, go to your deployment and then the Kibana page to enable it. The endpoint to access Kibana is available after it has been enabled.
Load the Kibana index pattern on your Elasticsearch cluster according to the instructions. You’ll use the same cluster endpoint in Elasticsearch Service that you used earlier and authenticate with the same credentials.

Beginning with Elastic Stack version 8.0, Kibana index patterns have been renamed to data views. To learn more, check the Kibana What’s new in 8.0 page.
```
./scripts/import_dashboards -only-index -es https://mycluster.us-east-1.aws.found.io:9243 -user filebeat_internal -pass xxxxxxxxxxxxxxxxxx
```
The filebeat-* index pattern you load has to match the version of Filebeat that you installed on the server where you are collecting operational data. In our example, we installed Filebeat 5.6, so we need to load the filebeat-* index pattern for 5.6 as well.
Open Kibana in Elasticsearch Service and select the filebeat-* index pattern in Kibana to explore Filebeat events:

« Configure Beats and Logstash with Cloud ID Configure Logstash (without Cloud ID) »