Streamline configuration processes with an official Elastic Stack Terraform provider

By

Omer Kushmaro

blog-thumb-hashicorp.png

We’re excited to announce that operation and SRE teams can now take advantage of an official Elastic Stack Terraform provider to configure the Elastic Stack across all types of infrastructure, whether on Elastic Cloud, or on-premises and self-managed.

This Elastic Stack Terraform provider will allow you to fully control the configuration of Elasticsearch, Kibana, Fleet, and any other component within the Elastic Stack.

The Terraform provider helps you safely manage production-related infrastructure using methodologies like infrastructure as code, which allow you to apply peer-reviewed infrastructure changes in an automated and controlled fashion.

Let’s walk through an example of how to spin up an Elastic Cloud deployment, then immediately configure it with an index lifecycle policy — using the new Elastic Stack Terraform Provider in a single Terraform script.

Getting started

To get started, include the following code block in one of your Terraform .tf configuration files:
terraform {
  required_providers {
    elasticstack = {
      source  = "elastic/elasticstack"
      version = "0.2.0"
    }
  }
}

The provider will be downloaded from the Terraform registry once you initialize it using terraform init.

Connecting to an Elastic Stack can be configured either in the provider block like so:

provider "elasticstack" {
elasticsearch {
 		username = "elastic" 
                password = "changeme" 
                endpoints = ["http://localhost:9200"] 
} 
}

Or, by using an elasticsearch_connection block with the same parameters on a per-resource level. This lets you use the provider to configure multiple Elastic Stack instances in a single configuration file and spin up a fully configured Elastic deployment on Elastic Cloud.

The Elastic Stack, as code

The Elastic Stack Terraform provider allows you to manage crucial Elasticsearch resources (and in the future, Elastic Stack resources) like:

  • Index templates
  • Snapshot repositories, snapshot policies
  • ILM policies
  • Users
  • Roles

Using the Elastic Stack provider with Elastic Cloud

The example below breaks down how to use the Elastic Stack provider to jumpstart and configure an Elastic Cloud deployment. Note that the provider works the same with both Elastic Cloud deployments, as well as on-premises or self-managed Elastic Stack clusters.

You can refer to a previous blog post on Elastic Cloud deployment resources. This post focuses on the Elastic Stack provider, its current capabilities, and partially builds on top of the previous blog’s concepts.

In order to use both the Elastic Cloud provider (to spin up an Elastic Stack on Elastic Cloud) and the Elastic Stack provider, simply use an elasticsearch_connection block alongside outputs from the ec_deployment resource.

First set up your deployment using the Elastic Cloud provider (ec), and create it with several data tier configurations:

resource "ec_deployment" "elastic-prod" {
region = "gcp-us-central1" 
name = "elastic-prod" 
version = data.ec_stack.latest.version 
deployment_template_id = "gcp-storage-optimized" 
elasticsearch {
topology {
	id=”cold”
	size=”16g”
}
topology {
	id=”hot_content”
	size=”16g”
}
topology {
	id=”warm”
	size=”32g”
}
} 
kibana {} 
}

As the configuration depicts, we’re spinning up an Elastic Cloud deployment on a Google Cloud us-central1 region, with three data tiers (one for cold, hot, and warm contents).

Next, use the Elastic Stack provider to create an index lifecycle management (ILM) policy, to match your data tiers. Create an ILM policy that will roll over hot indices after one day, move them to the warm tier after seven days, cold tier after 30 days, and finally for deletion after 60 days.

resource "elasticstack_elasticsearch_index_lifecycle" "my_ilm" {
name = "my_ilm_policy"

hot {
   min_age = "1h"
   rollover {
     max_age = "1d"
     max_size= "50g"
   }
}
warm {
   min_age="7d"
}
cold {
   min_age="30d"
}
 	delete {
   		min_age = "60d"
   		delete {}
 	}

elasticsearch_connection {
   endpoints = [ec_deployment.elastic-prod.elasticsearch[0].https_endpoint]
   username  = ec_deployment.elastic-prod.elasticsearch_username
   password  = ec_deployment.elastic-prod.elasticsearch_password
}
}

Use the elasticsearch_connection to specify the connection to your newly created (and dependent) Elastic Cloud deployment.

The final result

In the example above you’ve used the new Elastic Stack Terraform provider to configure an ILM policy for your newly spun-up Elastic Cloud deployment, all in a single Terraform HCL file.

View the Terraform documentation for more details about configuring and managing your Elastic Stack provider.

To start using the Elastic Cloud today, simply login to the Elastic Cloud console or sign up for a free 14-day trial.
The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

Sign up for Elastic Cloud free trial

Spin up a fully loaded deployment on the cloud provider you choose. As the company behind Elasticsearch, we bring our features and support to your Elastic clusters in the cloud.

Start free trial