Configuring Elastic Cloud

What Settings are Available?

Once you have signed into the Elastic Cloud Console, three main categories of settings are available:

Name

This setting allows you to assign a more human-friendly name to your deployment which will be used for future reference in the Elastic Cloud Console. Common choices are dev, prod, test, or something more domain specific. A consistent naming convention is helpful for multiple Elastic Cloud users.

Cloud Platform

Selects a cloud platform and a region where your Elasticsearch clusters and Kibana instances will be hosted. Elastic Cloud currently supports Amazon Web Services (AWS) and Google Cloud Platform (GCP).

Cloud platform and regions in Elastic Cloud

Regions represent availability zones in a geographic location, where your cluster will be located. When choosing a region, the general rule is to choose one as close to your application servers as possible in order to minimize network delays.

Tip

You can select your cloud platform and region only when you create a new cluster, so pick ones that works for you. They cannot be changed later. Different clusters can use different platforms and regions.

Version

Elasticsearch versions are denoted as X.Y.Z, where X is the major version, Y is the minor version, and Z is the patch level or maintenance release. The default version is the latest, stable version. At any given time, the two latest minor versions are guaranteed to be available for deployments of new clusters, such as Elasticsearch 6.2.4 and 6.1.4. Only the latest patch level within a minor version is usually available for new deployments, such as 2.4.5.

You might sometimes see additional versions listed in the user interface, beyond what we guarantee to be available, such as release candidate builds. If versions are listed, they can be deployed.

In order to be able to deliver new features and keep complexity manageable, we also need to be able to discontinue old versions. When a version nears its end-of-life point, you are typically given six months advance notice to upgrade to one of the available minor versions.

To learn more about how we support Elasticsearch versions in Elastic Cloud, see Version Policy.

You can always upgrade Elasticsearch versions without downtime, but you cannot downgrade. To learn more about upgrading versions of Elasticsearch and best practices for major version upgrades, see Version Upgrades.

Fault Tolerance

Fault Tolerance, also known as high availability, is achieved by running a deployment with replicas in multiple availability zones (availability zones), to prevent against downtime when infrastructure problems occur. We offer the options of running in one, two, or three availability zones.

Fault tolerance in Elastic Cloud

Running in two or more availability zones provides reasonably high protection against infrastructure failures and intermittent network problems. You might want three availability zones if you need even higher fault tolerance. If the cluster is mainly used for testing or development, and data loss doesn’t matter, just one zone might be sufficient.

Like many other changes, you can change the level of fault tolerance while the deployment is running. For example, when you prepare a new deployment for production use, you can first run it in a single availability zone and then add a second right before deploying to production.

While multiple availability zones increase an Elasticsearch cluster’s fault tolerance, they do not protect against problematic searches that cause nodes to run out of memory, for example. For an Elasticsearch cluster in your deployment to be highly reliable and available, it is also important to have enough memory.

The node capacity you chose is per availability zone. The reason for this is that there is no point in having two zones if the failure of one will result in a cascading error because the remaining node cannot handle the total load. Through the allocation awareness in Elasticsearch, we configure the nodes so that your Elasticsearch cluster will automatically allocate replicas between each availability zone.

Our article on Elasticsearch in Production covers availability zones and resilience against infrastructure failures in more detail.

Elasticsearch Node Size

Depending upon how much data you have and what queries you plan to run, you need to select a node size that fits your needs. There is no silver bullet for deciding how much memory you need other than simply testing it. The cluster performance metrics in the Elastic Cloud Console can tell you if your deployment is sized appropriately. Fortunately, you can change the capacity of the node later, without any downtime.

Capacity slider in Elastic Cloud

For trials, larger sizes are not available until you add a credit card.

Currently, half the memory for your Elasticsearch node is assigned to the JVM heap. For example, on a 32 GB deployment, 16 GB are allotted to heap. The disk-to-RAM ratio currently is 1:24, meaning that you get 24 GB of storage space for each 1 GB of RAM. All deployments are backed by SSD drives.

Tip

For production systems, we recommend not using less than 4 GB of RAM for your deployment, which assigns 2 GB to the JVM heap.

The CPU resources assigned to a deployment are relative to the size of your deployment, meaning that a 32 GB deployment gets twice as much CPU resources as a 16 GB deployment. All deployments are guaranteed their share of CPU resources, as we do not overcommit resources. Smaller deployments up to and including 8 GB of RAM benefit from temporary CPU boosting to improve performance when needed most.

To learn more about how much memory might be needed, see Elasticsearch in Production.

Plugins

Lists the official plugins available for your selected Elasticsearch version.

When selecting a plugin from this list you get a version that has been tested with the chosen Elasticsearch version. The reason we do not list the version chosen on this page is because we reserve the option to change it when necessary. That said, we will not force a cluster restart for a simple plugin upgrade unless there are severe issues with the current version. In most cases, plugin upgrades are applied lazily, in other words when something else forces a restart like you changing the plan or Elasticsearch runs out of memory.

If you need to control which version of the plugin gets used, you can upload the same plugin yourself as a custom plugin; however, you will need to manage updating the version yourself going forward.

Custom Plugins, Dictionaries, and Scripts

If you have uploaded any plugins or user bundles with dictionaries or scripts then this where you choose to enable them for the cluster.

Tip

Only Gold and Platinum subscriptions have access to uploading custom plugins, scripts, and dictionaries.

Scripting

The defaults for the different supported script types are generally safe to accept as is, unless you have a specific requirement. The script_fields in filters and facets are one of the features that make Elasticsearch so flexible, but they can allow arbitrary code execution, like Runtime.exec("cat /etc/passwd") and other malicious things.

We provide you with three levels of scripting control for each of the script types that are supported: You can disable the scripts completely, enable scripts to run in a sandbox or enable all scripts. Especially for inline scripts, we strongly recommend that you do not enable all inline scripts, because of the security risk that they can pose.

Note

Dynamic scripting is not available for Elasticsearch 1.x. If you need scripting, use a supported Elasticsearch version.

Sharding

You can review your Elasticsearch shard activity from Elastic Cloud. At the bottom of the Elasticsearch page, you can hover over each part of the shard visualization for specific numbers.

Shard activity

For versions before 5.0: You can select the number of shards. The default is 1. We recommend that you read Sizing Elasticsearch before your change the number of shards.

User Settings

Change how Elasticsearch runs with your own user settings. User settings are appended to the elasticsearch.yml configuration file for your Elasticsearch cluster, but not all settings are supported. To learn more, see Editing Your User Settings.

Automatic Index Creation

Determines whether an index is created automatically if you attempt to index a document into an index that does not exist.

Deletion Requires Name

Determines whether destructive actions like deleting an index require explicit index names or whether wildcards are allowed.

Kibana

For new deployments that use Elasticsearch version 5.0 and later, we automatically create a Kibana instance for you. If you use a version before 5.0 or if your deployment didn’t include a Kibana instance initially, there might not be a Kibana endpoint URL shown, yet. To enable Kibana, simply click Enable.

Enabling Kibana provides you with an endpoint URL, where you can access Kibana. It can take a short while to provision Kibana right after you click Enable, so if you get an error message when you first click the endpoint URL, try again.

For version 5.0 and later: Log into Kibana with the elastic superuser to try it out. The password was provided when you created your cluster or can be reset.

+ For versions before 5.0: and if Shield is enabled, you can log into Kibana with the admin user to try it out. The password was provided when you enabled Shield or can be reset.

In production systems, you might need to control what Elasticsearch data users can access through Kibana, so you need create credentials that can be used to access the necessary Elasticsearch resources. This means granting read access to the necessary indexes, as well as access to update the .kibana index.

Extended Maintenance Mode

In extended maintenance mode, requests to your deployment are blocked during configuration changes. You use maintenance mode to perform corrective actions that might otherwise be difficult to complete. Maintenance mode lasts for the duration of a configuration change and is turned off after the change completes.

We strongly recommend that you use maintenance mode when your deployment is overwhelmed by requests and you need to increase capacity. If your deployment is being overwhelmed because it is undersized for its workload, nodes might not respond to efforts to resize. Putting the deployment into maintenance mode as part of the configuration change can stop the deployment from becoming completely unresponsive during the configuration change, so that you can resolve the capacity issue. Without this option, configuration changes for deployments that are overwhelmed can take longer and are more likely to fail.

Deployment Management

Used with caution, there are two actions you can perform on the entire deployment:

  • Force restart - Needed only rarely, but full deployment restarts can help with a suspected operational issue before reaching out to Elastic for help. Your data will persist and when restarted, your shards will be in the same state.
  • Delete deployment - For deployments that you no longer need and don’t want to be charged for any longer. Deleting a deployment removes the deployment and all your data permanently.
Important

Use the actions here with care. Deployments are not available while they restart and deleting a cluster does really remove the cluster and all your data permanently.

Change Your Deployment Configuration

There are several reasons why you might want to change the configuration of a deployment:

  • To increase or decrease node capacity by changing the amount of reserved memory and storage.
  • To enable high availability by adjusting the number of availability zones that your deployment runs in.
  • To upgrade to new versions of Elasticsearch. You can upgrade from one major version to another, such as from 1.7.5 to 2.3.5, or from one minor version to another, such as 2.3.5 to 2.4.0. You can’t downgrade versions.
  • To change what plugins are available on your deployment.

You can change the configuration of a running deployment in the Elastic Cloud Console from several places, depending on what needs to be changed. The overview page for your deployment has several configurable options, whereas the Edit pages under Elasticsearch and Kibana allow you to make changes to those specific types of instances.

With the exception of major version upgrades, we can perform all these changes without having to interrupt your deployment. You can continue searching and indexing. The changes can also be done in bulk: in one action, you can add more memory, upgrade, adjust the number of plugins, and adjust the number of availability zones.

We perform all of these changes by making Elasticsearch nodes with the new configuration join the existing deployment in its entirety. After joining, the new nodes will recover the indexes. When they are done, they will start receiving requests. When all the new nodes are ready, we bring down the old ones.

By doing it this way, we reduce the risk of doing any changes. If the new nodes have any problems, the old ones are still there, processing requests.

Note: If you use a Platform-as-a-Service provider like Heroku, the administration console is slightly different and does not allow you to make changes that will affect the price. That must be done in the platform provider’s add-on system. You can still do things like change Elasticsearch version or plugins.

To change your deployment configuration:

  1. Sign in to the Elastic Cloud Console.
  2. Go to the appropriate page, depending on the type of change you want to make.
  3. Let the user interface guide you through the cluster configuration for your cluster.

    If you are changing an existing deployment, you can make multiple changes with a single configuration update, such as adding availability zones and changing the node capacity all in one step.

  4. Save your changes. The new configuration takes a few moments to create.