07 September 2017 Engineering

Monitor Your System with Metricbeat and Elastic Cloud

By Jongmin Kim (KR)Jamie Smith

Elastic Cloud is the official hosted Elasticsearch and Kibana offering, from the creators of Elasticsearch, Kibana, Beats, and Logstash. This is not the same as Amazon's AWS Elasticsearch Service. You can see details in this post.

Elastic Cloud offers a 14-day free trial. Your email address is all that you need to sign up, no credit card is required.

In this post, we will see how to:

  • Create an Elasticsearch deployment in Elastic Cloud
  • Add a new user with X-Pack Security
  • Use Metricbeat to collect system metrics, ship the metrics to the Elasticsearch deployment running in Elastic Cloud, and visualize the metrics in Kibana dashboards that automatically loaded by Metricbeat. 

1. Create an Elastic Cloud account

Go to the Elastic Cloud web page. Start an Elastic Cloud Free Trial. Note: You can skip to step 2 if you have already have an Elastic Cloud account. 

2018-05-02_07-51-53.png

Enter your email address and click the Start Free Trial button. You will receive an email verification message in your inbox. Click the Verify Email and Accept TOS button to continue. 

스크린샷 2017-08-11 21.07.57.png 

And set the password for your Elastic Cloud account.

2018-05-02_08-50-52.png

2. Start an Elasticsearch deployment and Kibana instance

Once you've created your Elastic Cloud account password, it will automatically take you to the Elastic Cloud UI. You can get back later by visiting https://cloud.elastic.co.

Click on Create deployment to create your first deployment.

2018-05-02_08-21-56.png

Choose the deployment size and region (if you are on the trial then it selects the size for you), then scroll to the bottom and click Create Deployment.  It will create and provision your Elasticsearch deployment.

2018-05-02_08-26-57.png

Once you've finished creating your Elasticsearch deployment it will show the generated password for the superuser, elasticYou will not be able to come back to this password again, so copy it somewhere safe. 

2018-05-02_08-32-48.png  

If you lose the password, you can reset it from the Security menu in the Elastic Cloud console.

2018-05-02_08-33-05.png

Navigate to the Overview menu by clicking on your deployment's name in the left sidebar. You will see Elasticsearch and Kibana access URLs.

2018-05-02_09-28-46.png

3. Create a new superuser

Click on the Kibana URL, which will take you to the Kibana login screen. Use elastic for the user, along with the automatically generated password (which you copied somewhere safe) to log in.

2018-05-02_08-34-13.png 

Elastic Cloud includes several X-Pack features, such as security and monitoring. 

If you prefer to use a more familiar username and password, you can create a new superuser using the Role Based Access Control functionality.  Visit the Management tab, and select Users in the Security section:

2018-05-02_08-34-30.png  

Click the + Create User button.

2018-05-02_08-34-39.png  

Fill in the username, password, full name, email, enter superuser for the Role, and save the new user.

2018-05-02_08-35-51.png

This will create your favorite new superuser account

2018-05-02_08-36-53.png

Try to log out and log back in with your new account to make sure everything works (if not you can still log in as elastic). 

2018-05-02_08-35-51.png

4. Configure and run Metricbeat

Now that we have Elasticsearch and Kibana configured, let's send some data its way. In this tutorial, we will use Metricbeat for this purpose.  

Kibana has a great new set of instructions to help you get different kinds of data into your Elastic Cloud deployment.  

2018-05-02_10-13-47.png

We will walk through the "Metrics" path and use our newly created user.  Click on the "Add metric data":

2018-05-02_11-29-32.png

then "System metrics" to see the steps that we will be following

2018-05-02_10-15-59.png

Metricbeat collects various kinds of metrics from your system and ships them to your Elasticsearch deployment. It runs as a Binary file, so you will have to download the correct file for your operating system. 

You can download using curl as described in the instructions, or you can visit the Metricbeat download page to download the correct binary for your OS.  Once you have it, unpack it.

2018-05-02_08-39-05.png

Once unpacked, you will find several files in the metricbeat directory:

2018-05-02_08-42-57.png 

The file metricbeat (or metricbeat.exe for Windows OS) will start your Metricbeat. The default configuration is set to send data to Elasticsearch running at http://localhost:9200 by default, so we will have to change the configuration to send it to our Elastic Cloud deployment. 

Open metricbeat.yml with your favorite code editor.

2018-05-02_10-45-25.png

You have to change configuration similar to the above.  The configuration is easy using Elastic Cloud, we can simply use the Cloud ID to connect.

Uncomment #cloud.id  and enter the Cloud ID from the deployment overview page. 

cloud.id: "My_Cluster:dXMtZWFzdC0xLmF3cy5zdGFnaW5nLmZvdW5kaXQubm8kYzQ…" 

Then uncomment #cloud.auth and enter your newly created username and password (if you skipped the user creation you can use elastic and the auto-generated password).

cloud.auth: "newuser:<supersecretpassword>" 

Be careful with yaml grammar, you have to use the right spacing on each line.  in this case, neither of the cloud fields should be indented.  

Save the metricbeat.yml file. Then run metricbeat, supplying the -setup option, which will tell the system to load the default metrics dashboards to Kibana.

metricbeat-5_5_2-darwin-x86_64_—_metricbeat_-c_metricbeat_yml_-setup_-e_—_109×20.png

TIP: if you add the -e option when running Metricbeat you can see the logs while running.

5. Check the Metricbeat dashboards

We added the -setup option when we started Metricbeat, so it automatically loaded the Metricbeat dashboards and visualizations to Kibana.  The System metrics instructions has a handy link at the bottom to take us to the system overview:

2018-05-02_10-19-21.png

Or you can go click on the "Dashboard" tab in the sidebar to see all of the different types of dashboards that ship with Metricbeat:

2018-05-02_08-53-03.png  

Verify that your systems metrics are being delivered to Elasticsearch, with a system overview:

2018-05-02_11-39-09.png

and a host overview:   

2018-05-02_11-03-47.png

Now that you have seen how it works, you can deploy Metricbeat to your other systems.  Give it a try!