07 September 2017 Engineering

Monitor Your System with Metricbeat and Elastic Cloud

By Jongmin Kim (KR)Jamie Smith

Elastic Cloud is the official hosted Elasticsearch and Kibana offering, from the creators of Elasticsearch, Kibana, Beats, and Logstash. This is not the same as Amazon's AWS Elasticsearch Service. You can see details in this post.

Elastic Cloud offers a 14-day free trial. Your email address is all you need to sign up, no credit card is required.

In this post, we will see how to:

  • Create an Elasticsearch cluster in Elastic Cloud
  • Add a new user with X-Pack Security
  • Use Metricbeat to collect system metrics, ship the metrics to the Elasticsearch cluster running in Elastic Cloud, and visualize the metrics in Kibana dashboards that automatically loaded by Metricbeat. 

1. Create an Elastic Cloud account

Go to the Elastic Cloud web page. Start an Elastic Cloud Free Trial. Note: You can skip to step 2 if you have already have an Elastic Cloud account. 

elastic_cloud_register.png

Enter your email address and click the Start Free Trial button. You will receive an email verification message in your inbox. Click the Verify Email and Accept TOS button to continue. 

스크린샷 2017-08-11 21.07.57.png 

Set the password for your Elastic Cloud account.

스크린샷 2017-08-11 21.08.19.png


2. Start an Elasticsearch cluster and Kibana instance

Once you have created your password, it will automatically take you to the Elastic Cloud UI. You can get back later by going to https://cloud.elastic.co.

Click on Create Cluster to create your first cluster.

Monitor_your_system_with_Metricbeat_on_Elastic_Cloud_-_Google_Docs.png

Choose the cluster size and region then click Create.  It will create and provision your Elasticsearch cluster.

Elastic_Cloud.png

Once you have finished creating your Elasticsearch cluster, it will pop up the generated password for the superuser, elasticYou will not be able to come back to this password again, so copy it somewhere safe. 

스크린샷 2017-08-11 21.10.25.png  

If you lose the password, you can reset it from the Security menu in the Elastic Cloud console.

스크린샷 2017-08-11 22.43.28.png

Navigate to the Overview menu. You will see Elasticsearch and Kibana access URLs.

스크린샷 2017-08-11 21.14.48.png 


3. Create a new superuser

Click on the Kibana URL, which will take you to the Kibana log in screen. Use "elastic" for the user, along with the automatically generated password (which you copied somewhere safe) to log in.

스크린샷 2017-08-11 21.17.16.png 

Elastic Cloud includes several X-Pack features, such as security and monitoring. 

If you prefer to use a more familiar username and password, you can create a new superuser using the Role Based Access Control functionality.  Visit the Management tab, and select Users in the Security section:

스크린샷 2017-08-11 21.35.51.png 

Click the + Create User button.

스크린샷 2017-08-11 21.36.03.png 

Fill in the username, password, full name, email, enter superuser for the Role, and save it.

new_user_reg.png

This will create your favorite new superuser account

new_user_registered.png

Try to logout and log back in with your new account to make sure everything works ok. 

new_login.png

4. Configure and run Metricbeat

Now that we have Elasticsearch and Kibana configured, let's send some data its way. In this tutorial, we will use Metricbeat for this purpose. 

Metricbeat collects all kinds of metrics from your system and ships them to your Elasticsearch or Logstash instance. It runs as a Binary file, so you will have to download the correct file for your operating system. Visit the to Metricbeat download page, download the correct binary for your OS, and then unpack it.

스크린샷 2017-08-11 21.43.16.png 

Once unpacked, you will find several files in the metricbeat directory.

스크린샷 2017-08-11 23.08.18.png 

The file metricbeat (or metricbeat.exe for Windows OS) will start your Metricbeat. The default configuration is set to send data to Elasticsearch running at http://localhost:9200 by default, so we will have to change the configuration to send it to our Elastic Cloud cluster. 

Open metricbeat.yml with your favorite code editor.

Monitor_your_system_with_Metricbeat_on_Elastic_Cloud_-_Google_Docs.png

Replace "localhost:9200" with our new Elasticsearch cluster URL. 

output.elasticsearch:
  #Array of hosts to connect to.
  hosts: ["localhost:9200"]

Put your newly created username and password (if you skipped the user creation you can use elastic and the auto-generated password).

  username: "<user id>" 
  password: "<password>"

Be careful with yaml grammar, you have to make right space in each line. username: & password: should be indented by 2 spaces , while the rest should start with no space.  Put your newly created username and password. If you skipped user creation, you can use elastic and the auto generated password for this step.

Save the metricbeat.yml file. Then run metricbeat, supplying the -setup option, which will tell the system to load the default metrics dashboards to Kibana.

metricbeat-5_5_2-darwin-x86_64_—_metricbeat_-c_metricbeat_yml_-setup_-e_—_109×20.png

TIP: if you add the -e option when running Metricbeat you can see the logs while running.

5. Check the Metricbeat dashboards

We added the -setup option when we started Metricbeat, so it automatically loaded the Metricbeat dashboards and visualizations to Kibana:

스크린샷 2017-08-11 21.53.03.png 

Click the Kibana Dashboard menu and you will see new Metricbeat dashboards. Verify that your systems metrics are being delivered to Elasticsearch:

스크린샷 2017-08-11 21.53.20.png  

Now that you have seen how it works, you can deploy Metricbeat to your other systems.  Give it a try!