<h3>Introduction</h3>The discovery module in Elasticsearch is responsible for finding nodes that belong to a cluster and electing a master node. The Google Compute Engine (GCE) discovery plugin uses GCE APIs to perform automatic unicast discovery. In this post, we'll work through setting up a three node Elasticsearch cluster in Google Cloud using the GCE discovery plugin, and create a custom image that can be shared among different projects.<h4>Before we start</h4>To follow along with this blog, there are two prerequisites:<ul><li><a href="https://cloud.google.com/resource-manager/docs/creating-managing-projects">Google Cloud Platform project ID</a></li><li><a href="https://cloud.google.com/sdk/downloads">Install Google Cloud SDK</a></li></ul>After setting up the project ID and installing the SDK, we have a few more administrative items to do. First, set the default project:<pre class="prettyprint">gcloud config set project sherry-test-gce </pre>Then login to Google Cloud:<pre class="prettyprint">gcloud auth login </pre>You can also set the default region and zone at this time to omit the --zone and --region flags while using the gcloud tool. This is optional. Here's how:<pre class="prettyprint">gcloud config set compute/region us-west1-b </pre>To simplify our tasks ahead, we will add the default project and region and omit them from rest of the commands. Also, all commands here that start with gcloud are executed from the local machine and not on the GCE instances.<h3>A GCE instance with Elasticsearch and GCE discovery plugin</h3><h4>Setup Firewall Rules</h4>We will begin with creating a firewall rule so that the nodes in our Elasticsearch cluster can communicate with each other. Elasticsearch uses the <a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-transport.html#_tcp_transport">TCP transport module</a> for internal communications; the default port range for the transport module is between 9300 - 9400. In addition, we will open up port 9200 to expose Elasticsearch APIs over HTTP.<pre class="prettyprint">gcloud compute firewall-rules create elasticsearch --direction=INGRESS \ --priority=1000 --network=default --action=ALLOW \ --rules=tcp:9200,tcp:9300-9400 --source-ranges=0.0.0.0/0 </pre><h4>Create an instance using gcloud SDK</h4>Now we are ready to set up a GCE instance:<pre class="prettyprint">gcloud compute instances create sherry-instance-gce-discovery \ --machine-type=n1-standard-2 --subnet=default \ --min-cpu-platform=Automatic \ --tags=elasticsearch,http-server,https-server \ --image=ubuntu-1604-xenial-v20180306 \ --image-project=ubuntu-os-cloud \ --boot-disk-size=10GB --boot-disk-type=pd-standard \ --boot-disk-device-name=sherry-gce-discovery --scopes=compute-ro </pre>The tags argument includes the Elasticsearch firewall rule along with http and https, which are optional. Once this process is complete, you should see the name of the instance, the zone it is created in, its internal and external IP, and the status of the instance.A few things to note: <ol><li>The instance we created is called sherry-instance-gce-discovery</li><li>The instance is running ubuntu-16.04.</li><li>The instance is using --scopes=compute-ro. This enables the GCE discovery module to call required APIs.</li></ol>Now, that we've created the instance, we need to connect to it:<pre class="prettyprint">gcloud compute ssh sherry-instance-gce-discovery </pre>If this is the first time you are using gcloud ssh to access a GCE instance, it will create a ssh key for you.Besides using the <a href="https://cloud.google.com/pubsub/docs/quickstart-cli">gcloud command line tool</a>, you can connect to the instance using the <a href="https://console.cloud.google.com/">Google Cloud Web console</a> that is directly accessible from the browser. Another option is to use the external IP address of the instance:<pre class="prettyprint">ssh -i ~/.ssh/google_compute_engine 12.34.56.789 </pre><h4>Install Java</h4>We will need to install JDK 8 on our new instance:<pre class="prettyprint">sudo apt-get update sudo add-apt-repository ppa:webupd8team/java sudo apt-get update sudo apt-get install oracle-java8-installer # Setting Java 8 as the default (optional) sudo apt-get install oracle-java8-set-default # Verify Java version java -version </pre><h4>Install Elasticsearch and GCE discovery plugin</h4>Installing Elasticsearch is the next task. Please see <a href="https://www.elastic.co/guide/en/elasticsearch/reference/6.2/deb.html">install Elasticsearch with Debian Package</a> and <a href="https://www.elastic.co/guide/en/elasticsearch/reference/6.2/rpm.html">install Elasticsearch with RPM</a> for installation details. In our case, we will choose an Ubuntu based image:<pre class="prettyprint">wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.2.3.deb sudo dpkg -i elasticsearch-6.2.3.deb </pre>Once Elasticsearch is ready, <a href="https://www.elastic.co/guide/en/elasticsearch/plugins/current/discovery-gce.html#discovery-gce">install the GCE discovery plugin</a>. By default, the elasticsearch-plugin command can be found at /usr/share/elasticsearch/bin for both deb and rpm distributions.<pre class="prettyprint">sudo /usr/share/elasticsearch/bin/elasticsearch-plugin install discovery-gce </pre>Now, add the following configurations to the /etc/elasticsearch/elasticsearch.yml file:<pre class="prettyprint"># By default, the cluster name is elasticsearch cluster.name: lannister # Use ${HOSTNAME} to set the node name dynamically node.name: ${HOSTNAME} # Using GCE internal hostname to bind the network services network.host: _gce:hostname_ # Set the minimum master eligible nodes to avoid the split brain issue discovery.zen.minimum_master_nodes: 2 # Specific GCE discovery plugin settings cloud.gce.project_id: sherry-test-gce cloud.gce.zone: us-west1-b discovery.zen.hosts_provider: gce </pre>All of the GCE discovery plugin settings here are required. Let's take a quick look at them:cloud.gce.project_id - The Google project ID that we are usingcloud.gce.zone - The GCE zone where the Elasticsearch nodes will livediscovery.zen.hosts_provider - We are using the GCE discovery mechanismIn the near future, we are planning to make the GCE discovery setup even easier by <a href="https://github.com/elastic/elasticsearch/issues/13618">auto-discovering some of these configurations</a>.<h4>Start Elasticsearch</h4>We are ready to start Elasticsearch and verify everything is working.<pre class="prettyprint">sudo /etc/init.d/elasticsearch start </pre>Verify the instance is working as expected by tailing the Elasticsearch cluster log to ensure Elasticsearch has started:<pre class="prettyprint">tail /var/log/elasticsearch/lannister.log </pre>Then use curl to connect to the node and verify the service is running:<pre class="prettyprint">curl ${HOSTNAME}:9200 </pre>Note, the above curl command can only run within the Google Cloud Platform. By default, nothing is opened to internet. Log off from the GCE instance as our work here is done.<h4>Create a GCE custom image</h4>We will use the instance that we built to make a custom image. We can share the image with other projects to create additional GCE instances with Elasticsearch and GCE discovery plugin installed. Although you can use the <a href="https://cloud.google.com/sdk/gcloud/reference/compute/images/export">gcloud compute images export command</a> to create a custom image from a virtual disk, we choose to use the manual method so we can clean up the files on the disk before packaging it.Using our local environment, create a storage bucket to hold the disk image from our instance:<pre class="prettyprint">gsutil mb -p sherry-test-gce -c regional -l us-west1 gs://sherry-test-bucket/ </pre>We will stop the instance sherry-instance-gce-discovery as well. This is an optional step but ensures the integrity of the disk content in the snapshot.<pre class="prettyprint">gcloud compute instances stop sherry-instance-gce-discovery </pre>Take a snapshot of the instance/disk:<pre class="prettyprint">gcloud compute disks snapshot sherry-instance-gce-discovery \ --snapshot-names=sherry-gce-discovery-snapshot </pre>Use the snapshot to build a new GCE disk that we will use to create an image later.<pre class="prettyprint">gcloud compute disks create sherry-gce-discovery-disk \ --source-snapshot sherry-gce-discovery-snapshot </pre>Make a temporary disk that is at least 50% larger than the original image disk size. We will store the raw image of the disk from the instance sherry-instance-gce-discovery and the compressed version that we will be creating on the temporary disk. Since our original disk is 10 GB, our temporary one will be 15 GB.<pre class="prettyprint">gcloud compute disks create sherry-gce-discovery-temp-disk --size 15GB </pre>We will create a temporary GCE instance with the storage-rw scope and attach both the disk from the snapshot we made earlier and the temporary disk.<pre class="prettyprint">gcloud compute instances create sherry-temp-instance --scopes storage-rw \ --disk name=sherry-gce-discovery-disk,device-name=sherry-gce-discovery-disk \ --disk name=sherry-gce-discovery-temp-disk,device-name=sherry-gce-discovery-temp-disk </pre>Connect to the new instance:<pre class="prettyprint">gcloud compute ssh sherry-temp-instance </pre>Once on the instance, list the disks and disk partitions available:<pre class="prettyprint">ls /dev/disk/by-id/ </pre>Now, format and mount the temporary disk: <pre class="prettyprint">sudo mkdir /mnt/tmp sudo mkfs.ext4 -F /dev/disk/by-id/google-sherry-gce-discovery-temp-disk sudo mount -o discard,defaults /dev/disk/by-id/google-sherry-gce-discovery-temp-disk /mnt/tmp </pre>Mount the image disk so we can modify and removed files:<pre class="prettyprint">sudo mkdir /mnt/image-disk sudo mount /dev/disk/by-id/google-sherry-gce-discovery-disk-part1 /mnt/image-disk </pre>Delete the Elasticsearch data directory. Be default, it is located at /mnt/image-disk/var/lib/elasticsearch/nodes. Each new instance will create its own Elasticsearch data directory when the Elasticsearch service starts the first time.Remove SSH key. The authorized_keys file is located in the /mnt/image-disk/home/[USER]/.ssh directory. This step is not required but highly recommended.Unmount the image disk:<pre class="prettyprint">sudo umount /mnt/image-disk/ </pre>Create a disk.raw from the image disk on the temporary disk. Please note, it must be called disk.raw.<pre class="prettyprint">sudo dd if=/dev/disk/by-id/google-sherry-gce-discovery-disk of=/mnt/tmp/disk.raw bs=4096 </pre> Create a tar file of the disk.raw file.<pre class="prettyprint">cd /mnt/tmp sudo tar czvf sherry-gce-discovery-disk.tar.gz disk.raw </pre>Copy the tar file to the bucket we created earlier from the GCE instance directly:<pre class="prettyprint">gsutil cp /mnt/tmp/sherry-gce-discovery-disk.tar.gz gs://sherry-test-bucket </pre>Log off the temporary instance after the file transfer is done.Finally, make a custom image that we can use to create new GCE instances:<pre class="prettyprint">gcloud compute images create sherry-gce-discovery-image \ --source-uri=https://storage.googleapis.com/sherry-test-bucket/sherry-gce-discovery-disk.tar.gz </pre><h4>Create instances based on our custom image</h4>We are ready to create three new instances from the images we just built.<pre class="prettyprint">gcloud compute instances create sherry-instance-1 \ --machine-type=n1-standard-2 --subnet=default \ --min-cpu-platform=Automatic --tags=elasticsearch,http-server,https-server \ --image=sherry-gce-discovery-image --image-project=sherry-test-gce \ --boot-disk-size=20GB --boot-disk-type=pd-standard \ --boot-disk-device-name=sherry-instance-1 --scopes=compute-ro gcloud compute instances create sherry-instance-2 \ --machine-type=n1-standard-2 --subnet=default \ --min-cpu-platform=Automatic --tags=elasticsearch,http-server,https-server \ --image=sherry-gce-discovery-image --image-project=sherry-test-gce \ --boot-disk-size=20GB --boot-disk-type=pd-standard \ --boot-disk-device-name=sherry-instance-2 --scopes=compute-ro gcloud compute instances create sherry-instance-3 \ --machine-type=n1-standard-2 --subnet=default \ --min-cpu-platform=Automatic --tags=elasticsearch,http-server,https-server \ --image=sherry-gce-discovery-image --image-project=sherry-test-gce \ --boot-disk-size=20GB --boot-disk-type=pd-standard \ --boot-disk-device-name=sherry-instance-3 --scopes=compute-ro </pre>Once completed, logon to each instance and start Elasticsearch. Please see <a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/install-elasticsearch.html">Elasticsearch installation instructions</a> to configure Elasticsearch to start automatically. From the instance sherry-instance-1, we can run curl sherry-instance-1:9200/_cat/nodes. We should see all three nodes in our cluster listed in the output.<pre class="prettyprint">10.138.0.8 14 29 6 0.10 0.11 0.07 mdi * sherry-instance-1 10.138.0.9 17 29 5 0.14 0.17 0.08 mdi - sherry-instance-2 10.138.0.10 13 29 5 0.13 0.22 0.10 mdi - sherry-instance-3 </pre><h4>Conclusion</h4>In a future blog, we will install and configure X-Pack to ensure our cluster is secure and production ready and the Google Cloud Storage repository plugin to provide the snapshot and restore capability. If you prefer the convenience of a managed cluster, <a href="http://cloud.elastic.co/">Elastic Cloud</a>, Elastic's official hosted solution is available on GCP. It comes with all of the X-Pack features and automatic snapshots.

blueprint-small.jpg

blueprint-circuit-board_thumb.jpg

Getting Started with the GCE Discovery Plugin on Google Cloud

if-it-fits-i-sits-scaling-the-elastic-stack-to-fit-fullbleed-three.jpg

small_medium_large.jpeg

Small, Medium, or Large - Scaling Elasticsearch and Evolving the Elastic Stack to Fit

Start free trial

Blog Footer CTA

Sign up for Elastic Cloud free trial

Principal Solutions Architect

Sherry Ger

By submitting you acknowledge that you've read and agree to our <a href="/legal/elastic-cloud-account-terms" target="_self">Terms of Service</a>, and you agree to receive the <a href="/legal/privacy-statement#how-we-use-the-information" target="_self">selected communications</a> at the contact details you provided above. See <a href="/legal/privacy-statement/" target="_self">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.

Communication Preferences - ONLY for /communication-preferences

Durch das Absenden erklären Sie, dass Sie unsere <a href="/de/legal/elastic-cloud-account-terms" target="_self">Nutzungsbedingungen</a> gelesen haben und mit ihnen einverstanden sind und dass Sie Elastic erlauben, <a href="/de/legal/privacy-statement#how-we-use-the-information" target="_self">ausgewählte Nachrichten und Informationen</a> an die oben von Ihnen bereitgestellten Kontaktangaben zu senden. Weitere Informationen, darunter auch dazu, wie Sie Ihre Einwilligung zurückziehen können, finden Sie in der <a href="/de/legal/privacy-statement/" target="_self">Datenschutzerklärung von Elastic</a>.

By submitting you acknowledge that you've read and agree to our <a href="/legal/serverless-preview-terms" target="_blank">Terms of Service</a>, and that Elastic may <a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">contact you</a> about our related products and services, using the details you provide above. See <a href="/legal/privacy-statement/" target="_blank">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.

Serverless GDPR Text

Durch das Absenden erklären Sie, dass Sie unsere <a href="/de/legal/serverless-preview-terms" target="_self">Nutzungsbedingungen</a> gelesen haben und mit ihnen einverstanden sind und dass Sie Elastic erlauben, bezüglich unserer zugehörigen Produkte und Dienstleistungen und unter Nutzung der von Ihnen oben bereitgestellten Kontaktangaben <a href="/de/legal/privacy-statement#how-we-use-the-information" target="_self">mit Ihnen in Kontakt zu treten</a>. Weitere Informationen, darunter auch dazu, wie Sie Ihre Einwilligung zurückziehen können, finden Sie in der <a href="/de/legal/privacy-statement/" target="_self">Datenschutzerklärung von Elastic</a>.

By submitting you agree to <a href="/agreements/global/cloud-services-monthly" rel="nofollow">Elastic Cloud Terms of Service</a>. Your personal data will be processed in accordance with <a href="/legal/privacy-statement">Elastic's Privacy Statement</a>.

Cloud Trial GDPR Text

Beim Absenden akzeptieren Sie die <a href="/de/agreements/global/cloud-services-monthly" rel="nofollow">Nutzungsbedingungen von Elastic Cloud</a>. Die Verarbeitung Ihrer personenbezogenen Daten unterliegt der <a href="/de/legal/privacy-statement">Datenschutzerklärung von Elastic</a>.

By submitting you acknowledge that you've read and agree to our <a href="/legal/terms-of-use" target="_blank">Terms of Service</a>, and that Elastic may <a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">contact you</a> about our related products and services, using the details you provide above. See <a href="/legal/privacy-statement/" target="_blank">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.

Newsletter GDPR Text

Durch das Absenden erklären Sie, dass Sie unsere <a href="/de/legal/terms-of-use" target="_self">Nutzungsbedingungen</a> gelesen haben und mit ihnen einverstanden sind und dass Sie Elastic erlauben, bezüglich unserer zugehörigen Produkte und Dienstleistungen und unter Nutzung der von Ihnen oben bereitgestellten Kontaktangaben <a href="/de/legal/privacy-statement#how-we-use-the-information" target="_self">mit Ihnen in Kontakt zu treten</a>. Weitere Informationen, darunter auch dazu, wie Sie Ihre Einwilligung zurückziehen können, finden Sie in der <a href="/de/legal/privacy-statement/" target="_self">Datenschutzerklärung von Elastic</a>.

Explore similar demos

Overview

Speakers

Close

See more insights

The content on this page is not available in the selected language. As Elastic grows globally, we continue to support content in multiple languages.

The content on this page is not available in the selected language.

Der Inhalt dieser Seite ist in der ausgewählten Sprache nicht verfügbar. Wir bei Elastic arbeiten daran, die bereitgestellten Inhalte in verschiedenen Sprachen anzubieten. Bis dahin bitten wir Sie um etwas Geduld und hoffen auf Ihr Verständnis!

Author

Watch now (no PT)

Watch now

See all top stories

All (no PT translation)

Contact information

Press Release

Share on Reddit

Articles by

Share this story

Share by Email

Thank you for your interest!

Contact Info

Follow us on Youtube

Load More

Share on LinkedIn

Follow us on LinkedIn

Search Integrations

All Solutions

Video for

Follow us on Twitter

See when this webinar starts in my time zone

Register to Watch

Register now

See All Posts

Can't make it? Register and we'll send you the recording. You'll also receive an email with related content

Sie können nicht dabei sein? Registrieren Sie sich, damit wir Ihnen die Aufzeichnung zukommen lassen können. Außerdem erhalten Sie eine E‑Mail mit Begleitmaterial.

Small image for

On-demand webinar

Featured webinar

Register to Attend

Share on Facebook

Reset all

Upcoming webinar

View more posts

Print

Hosted by

Sign In to Attend

View next

Follow us on Facebook

Share

Highlights

Learn More

Read less

You'll also receive an email with related content

Wir schicken Ihnen zudem relevante Informationen, die von Interesse sein könnten.

Thank you for registering. We will send you a confirmation email soon.

Vielen Dank für Ihre Registrierung. Sie erhalten in Kürze eine Bestätigungs-E‑Mail von uns.

Author

Articles by Sherry Ger

Principal Solutions Architect, Elastic

Getting Started with the GCE Discovery Plugin on Google Cloud

Small, Medium, or Large - Scaling Elasticsearch and Evolving the Elastic Stack to Fit