Tech Topics

Exploring the API for Elastic Cloud Enterprise

With Elastic Cloud Enterprise, we also introduced a new RESTful API to help you get started on your own automation journey. In its current 1.0 form, the API already supports operations on many different parts of the Elastic Cloud Enterprise platform and on the Elastic Stack. In this blog post, we’ll walk you through some hands-on examples to show how you can use the API with Elasticsearch clusters and Kibana.

To reproduce the examples described here, all you need is Elastic Cloud Enterprise, installed on a host that meets the basic prerequisites. (Not sure how to get started? Try our Quick Start Guide.)

The examples in this blog post all use HTTPS over port 12443, run against localhost. Instead of localhost, you can just as easily run these examples against a remote ECE host. Using HTTPS requires that you have a TLS certificate already installed. Alternatively, you can specify the -k option to turn off certificate verification as shown in our examples or simply use HTTP over port 12400 until you get your TLS certificate sorted out. 

A first API call: What Elasticsearch clusters are there?

Let’s try a simple API call with a GET request to retrieve information about existing Elasticsearch clusters on our Elastic Cloud Enterprise installation:

curl -k -X GET -u root:password https://localhost:12443/api/v1/clusters/elasticsearch
{
  "return_count": 3,
  "elasticsearch_clusters": [{
    "cluster_name": "My First Cluster",
    "plan_info": {
      "healthy": true,
      "current": {
        "plan_attempt_id": "05e6a34d-365f-45be-bc22-22669909da1b",
        "attempt_end_time": "2017-05-09T19:22:31.706Z",
        "plan_attempt_log": [],
        "attempt_start_time": "2017-05-09T19:21:32.529Z",
        "healthy": true
      },
      "history": []
    },
    "associated_kibana_clusters": [],
    "elasticsearch": {
      "healthy": true,
      "shard_info": {
        "healthy": true,
        "available_shards": [{
          "instance_name": "instance-0000000000",
          "shard_count": 1
        }],
        "unavailable_shards": [{
          "instance_name": "instance-0000000000",
          "shard_count": 0
        }],
        "unavailable_replicas": [{
          "instance_name": "instance-0000000000",
          "replica_count": 0
        }]
      },
      "master_info": {
        "healthy": true,
        "masters": [{
          "master_node_id": "77TjkuEdT_-IHGJZpV9klg",
          "instances": ["instance-0000000000"]
        }],
        "instances_with_no_master": []
      }
    },
    "links": {
    },
    "healthy": true,
    "status": "stopped",
    "topology": {
      "healthy": true,
      "instances": [{
        "disk": {
          "disk_space_used": 0
        },
        "maintenance_mode": false,
        "service_running": true,
        "healthy": true,
        "instance_name": "instance-0000000000",
        "service_version": "5.3.2",
        "allocator_id": "192.168.40.89",
        "zone": "ece-region-1a",
        "container_started": true,
        "memory": {
          "instance_capacity": 1024,
          "memory_pressure": 12
        }
      }]
    },
    "metadata": {
      "version": 4,
      "last_modified": "2017-05-09T19:22:31.698Z",
      "endpoint": "96fc5d151c6b4b15b6ab9107aad67cef.192.168.40.89.ip.es.io"
    },
    "cluster_id": "96fc5d151c6b4b15b6ab9107aad67cef"
  }
...
  ]
}

As you can see from the JSON output returned by the API, there are three clusters, one of which is called My First Cluster. There are two other clusters whose output we omitted to keep things short. They are the logging-and-metrics and the admin-console-elasticsearch clusters that always get created with every installation of Elastic Cloud Enterprise. Take care when modifying these two clusters, as they are required for Elastic Cloud Enterprise to run smoothly. To keep your installation safe, some operations are not allowed on these clusters, but we do recommend that you enable fault tolerance and resize them.

Some of the terminology in the JSON will be unfamiliar. For example, there is a reference to various things that include the term “plan,” such as plan_info and plan_attempt_id. A plan is just a cluster configuration that gets applied to a cluster. When you change a cluster (to resize it, for example), you are simply attempting to apply a new plan. For now, you can just follow along with our examples, but if you want to learn more about what the JSON output means, our RESTful API reference content is available in OpenAPI 2.0 format with descriptions.

Create a first Elasticsearch cluster through the API

For our API exploration, we’ll be using My First Cluster. If you want to try your own hand at creating a similar cluster, the following API call includes all the necessary JSON to create a basic Elasticsearch 5.3.2 cluster:

curl -k -X POST -u root:password https://localhost:12443/api/v1/clusters/elasticsearch -H 'content-type: application/json' -d '{
  "cluster_name": "My First Cluster",
  "plan": {
    "zone_count": 1,
    "cluster_topology": [{
      "memory_per_node": 1024,
      "node_count_per_zone": 1
    }],
    "elasticsearch": {
        "version": "5.3.2"
    }
  }
}'

If you look closely, you can see that we are using the same /api/v1/clusters/elasticsearch URI as in our very first API call. The difference is that we are now performing a POST request instead of a GET to create the cluster.

If your installation of ECE does not include Elasticsearch 5.3.2, you can create a cluster for any version that you do have available. You can also add the missing versions to your installation: Starting with the 1.0 release, so-called Elastic Stack packs let you add more versions of the Elastic Stack to ECE. To learn more, see Manage Elastic Stack Versions.


When the API call has completed, Elastic Cloud Enterprise will return some information that you’re going to need later on, such as the cluster ID and the password for the elastic superuser:

{
  "elasticsearch_cluster_id": "d12ef9ad9b77447a908efa7c43da9ea6",
  "credentials": {
    "username": "elastic",
    "password": "e5PKVecVgpmlLdEAUP8ZZzWE"
  }
}

Applying a new cluster configuration: Resize

Let’s resize My First Cluster. The JSON output from our first API call tells us that the memory pressure is quite low, but the cluster is also very small with only 1024 MB of RAM. Any serious indexing activity will likely require more capacity.

Rerun the earlier API call but add d12ef9ad9b77447a908efa7c43da9ea6/plan to the GET request to make sure we get only the plan information for My First Cluster and save the output to a file, here my-first-cluster.json:

curl -k -X GET -u root:password https://localhost:12443/api/v1/clusters/elasticsearch/d12ef9ad9b77447a908efa7c43da9ea6/plan > my-first-cluster.json

Edit the my-first-cluster.json file in your favourite editor and change the value of memory_per_node to 4096:                   

{
  "zone_count": 1,
  "tiebreaker_override": false,
  "cluster_topology": [{
    "memory_per_node": 4096,
    "node_count_per_zone": 1,
    "node_configuration": "default"
  }],
  "elasticsearch": {
    "version": "5.3.2"
  }
}

Now apply the updated cluster configuration to the cluster. A simple POST request that specifies the edited JSON file should do the trick:

curl -k -X POST -u root:password https://localhost:12443/api/v1/clusters/elasticsearch/d12ef9ad9b77447a908efa7c43da9ea6/plan -H 'content-type: application/json' -d @my-first-cluster.json</p></span>

There are other changes you can make at the same time. For example, to make our cluster fault-tolerant, we could also have changed the value of zone_count to 2 or 3, which would have created cluster nodes in additional availability zones (each zone could be a rack or a server zone, for example). Note that for fault tolerance to be available, your Elastic Cloud Enterprise installation must include more than one availability zone, which is something you control with the --availability-zone parameter when you install Elastic Cloud Enterprise on a host.

Applying a new cluster configuration: Check on progress with the API

If you switched back to Cloud UI quickly enough after the last API call, you would have seen that the new configuration was being applied to your cluster:

Switching to the Cloud UI to check on progress is not feasible if you plan to use the RESTful API for automation, so the API also lets you check on the progress of configuration changes with the following call:

curl -k -X GET -u root:password https://localhost:12443/api/v1/clusters/elasticsearch/d12ef9ad9b77447a908efa7c43da9ea6/plan/pending
{
  "zone_count": 1,
  "cluster_topology": [{
    "memory_per_node": 4096,
    "node_count_per_zone": 1
  }],
  "elasticsearch": {
    "version": "5.3.2"
  }
}

The JSON with the cluster configuration that gets returned is the currently pending plan. After the configuration change has been completed, there will no longer be a pending plan, so the same call will return an error (if you also specify -v with curl the error code returned is a 404): 

curl -k -X GET -u root:password https://localhost:12443/api/v1/clusters/elasticsearch/d12ef9ad9b77447a908efa7c43da9ea6/plan/pending
{
  "errors": [{
    "code": "clusters.cluster_plan_state_error",
    "message": "This cluster ([d12ef9ad9b77447a908efa7c43da9ea6]) has no pending plan active"
  }]
}

For long-running changes that might involve large clusters, this yes or no answer to the question whether the configuration changes are done might not always be sufficiently granular. If you want more detailed information, you can also get the activity log returned back to you:

curl -k -X GET -u root:password https://localhost:12443/api/v1/clusters/elasticsearch/dc58e68c04c3466e8fa66b7dd414d466/plan/activity?show_plan_logs=true

Beware that the activity log can be quite long, as it will list every step in the cluster configuration process.

Applying a new cluster configuration: Upgrade

One of the most useful aspects of an API is that you can automate changes across a large number of clusters. Let’s say you have many clusters that are at a specific version of Elasticsearch that need to get upgraded to the next minor release, say, from version 5.2.2 to 5.3.2. You can either surreptitiously change the version of each cluster in the Cloud UI or you can automate finding all the clusters at the specific version level and upgrading them.

First, let’s get a list of all the clusters as shown in our earliest API example and then find all of the clusters that are at version 5.2.2. There is currently no way to search for clusters that are at a specific version through the RESTful API, but you can iterate over the JSON output to get the list you need. For each of these clusters, you need to obtain their current cluster configuration.

For example, let’s assume there is a cluster with the ID b743a89cff6341ac859f37a81e6c091b that returns the following plan:  

curl -k -X GET -u root:password https://localhost:12443/api/v1/clusters/elasticsearch/b743a89cff6341ac859f37a81e6c091b/plan
{
  "zone_count": 1,
  "tiebreaker_override": false,
  "cluster_topology": [{
    "memory_per_node": 1024,
    "node_count_per_zone": 1,
    "node_configuration": "default"
  }],
  "elasticsearch": {
    "version": "5.2.2"
  }
}

To get ready for the upgrade to Elasticsearch 5.3.2, you again need to modify the plan. As before when we resized our cluster, you can store the current plan information in a file and then modify the JSON. In this case, the line that specifies the version needs to be changed to read "version": "5.3.2".

After saving the file, all that’s left is to POST the new plan to effect a version upgrade:

curl -XPOST -u root:password https://localhost:12443/api/v1/clusters/elasticsearch/b743a89cff6341ac859f37a81e6c091b/plan -H 'content-type: application/json' -d @upgrade_me.json

When the upgrade operation is complete, the cluster will be at version 5.3.2 as shown in the Cloud UI:

You don’t really need to check the Cloud UI to confirm the upgrade; it’s just a convenient way to see the status of your clusters visually. You could also have issued another GET request to find out through the API whether the cluster has completed the upgrade as described in "Applying a new cluster configuration: Check on progress with the API."

Add Kibana to an Elasticsearch cluster

The RESTful API is also useful for creating new Kibana instances. In this example, we’ll show you how to create a new Kibana instance and associate it with an existing Elasticsearch cluster.

First, create a JSON file with a basic Kibana configuration, here add_kibana.json. There are different options you can specify, but the most important one is that you need to include an Elasticsearch cluster ID that the Kibana instance will be associated with. The cluster we’ll be adding Kibana to is the same My First Cluster we used before with the Cluster ID d12ef9ad9b77447a908efa7c43da9ea6.

{
  "elasticsearch_cluster_id": "d12ef9ad9b77447a908efa7c43da9ea6",
  "plan": {
    "kibana": {
    },
    "zone_count": 1
  }
}

A POST request is all that is needed to create the new Kibana instance, returning you the Kibana ID in response:

curl -k -X POST -u root:password https://localhost:12443/api/v1/clusters/kibana -H 'content-type: application/json' -d @add_kibana.json
{
  "kibana_cluster_id": "c363be6ed9c34c0f81925b1953e64003"
}

As with Elasticsearch clusters, you can find out if there are pending changes for Kibana clusters:

curl -k -X GET -u root:password https://localhost:12443/api/v1/clusters/kibana/c363be6ed9c34c0f81925b1953e64003/plan/pending
{
  "zone_count": 1,
  "cluster_topology": [{
    "memory_per_node": 1024,
    "node_count_per_zone": 1
  }],
  "kibana": {
    "version": "5.3.2"
  }
}

After the configuration changes are complete, this API call will return a message that says This cluster ([dc58e68c04c3466e8fa66b7dd414d466]) has no pending plan active.

If you switch back to the Cloud UI, you can see that My First Cluster now has a Kibana instance associated with it:

Dipping a toe into automation: Generate a roles token

One of the changes between ECE beta2 and the 1.0 release is that you now have to specify a roles token when you install ECE on additional hosts. It’s a security feature designed to prevent unauthorized hosts from joining your ECE installation. To make it easier to use roles tokens, you automatically get several tokens after installation on the first host. If you revoke these tokens or no longer have access to them, you can also use the RESTful API to generate a new token to install ECE on hosts and add the right roles at the same time. In this example, we generate a roles token for an allocator and use it to install ECE on an additional host.


To generate a roles token that can be used to assign the allocator role to hosts you install ECE on:

curl -H 'Content-Type: application/json' -u root:password http://localhost:12443/api/v1/platform/configuration/security/enrollment-tokens -d '{ "persistent": true, "roles": [ "allocator"] }'
{
  "token": "eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJzdWIiOiI3OTk4MDNmYi0wNjg1LTQ5MzYtYjEwMy00Njk5MWZiMWQ5YjEiLCJyb2xlcyI6WyJhbGxvY2F0b3IiXSwiaXNzIjoiY3VycmVudCIsInBlcnNpc3RlbnQiOnRydWV9.dpk4zaR6ePcaiojmOtSe8DCezK5wKnbRgwfxHG5q14M",
  "token_id": "799803fb-0685-4936-b103-46991fb1d9b1"
}

You are shown this token only once, so make sure you keep it somewhere safe until you no longer need it to install ECE on hosts (at which point it can be deleted).

When installing ECE on the additional host, include the token and specify the allocator role:

bash elastic-cloud-enterprise.sh install --coordinator-host 192.168.40.200 --roles-token 'eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJzdWIiOiI3OTk4MDNmYi0wNjg1LTQ5MzYtYjEwMy00Njk5MWZiMWQ5YjEiLCJyb2xlcyI6WyJhbGxvY2F0b3IiXSwiaXNzIjoiY3VycmVudCIsInBlcnNpc3RlbnQiOnRydWV9.dpk4zaR6ePcaiojmOtSe8DCezK5wKnbRgwfxHG5q14M' --roles "allocator"

After installation completes, you can use the new host's IP address, here 192.168.40.79, to verify through the RESTful API that the new allocator is available (some output was omitted for brevity):

curl -k -X GET -u root:password https://localhost:12443/api/v1/platform/infrastructure/allocators
{
  "zones": [{
    ...
    "zone_id": "ece-region-1b",
    "allocators": [{
      "public_hostname": "192.168.40.79",
      "instances": [],
      "status": {
        "connected": true
      },
      "host_ip": "192.168.40.79",
      "allocator_id": "192.168.40.79",
      "capacity": {
        "memory": {
          "total": 12398,
          "used": 0
        }
      }
    }]
  }]
}

The Cloud UI also shows that a new allocator with the same IP address is available:

Additional documentation for roles tokens is available, including information on generating ephemeral tokens that expire after a set period. To learn more, see Generate Roles Tokens.

Next Steps

These examples are intended to give you some hands-on experience with our new RESTful API, but they really only scratch the surface of what’s available to you. There are many more operations that are already possible.

With the 1.0 GA release, additional functionality is also available, such as:

  • Support for configuring the infrastructure of Elastic Cloud Enterprise itself
  • Improved support for managing Elasticsearch and Kibana, such as stopping and starting clusters, and putting clusters into maintenance mode

If you are interested in learning more about herding clusters instead of cats, take Elastic Cloud Enterprise for a spin and check out our API reference documentation.


Title image licensed under Creative Commons CC BY-SA 2.0.