Configuring Elastic Cloud

What Settings are Available?

Once you have signed in on Elastic Cloud Console, three main categories of settings are available:

Change Your Cluster Configuration

There are several reasons why you might want to change the configuration of a cluster:

  • To increase or decrease cluster capacity by changing the amount of reserved memory and storage.
  • To enable high availability by adjusting the number of data centers that your cluster runs in.
  • To upgrade to new versions of Elasticsearch. You can upgrade from one major version to another, such as from 1.7.5 to 2.3.5, or from one minor version to another, such as 2.3.5 to 2.4.0. You can’t downgrade versions.
  • To change what plugins are available on your cluster.

You can change the configuration of a running cluster from the Configuration pane in the in the Elastic Cloud Console.

With the exception of major version upgrades, we can perform all these changes without having to interrupt your cluster. You can continue searching and indexing. The changes can also be done in bulk: in one action, you can add more memory, upgrade, adjust the number of plugins and adjust the number of availability zones.

We perform all of these changes by making the cluster with the new configuration join the existing cluster in its entirety. After joining, the new nodes will recover the indexes. When they are done, they will start receiving requests. When all the new nodes are ready, we bring down the old ones.

By doing it this way, we reduce the risk of doing any changes. If the new nodes have any problems, the old ones are still there, processing requests.

Note: If you use a Platform-as-a-Service provider like Heroku, the administration console is slightly different and does not allow you to make changes that will affect the price. That must be done in the platform provider’s add-on system. You can still do things like change Elasticsearch version or plugins.

To change your cluster configuration:

  1. Sign in to the Elastic Cloud Console.
  2. Click Configuration for an existing cluster in the sidebar or click Create New Cluster.
  3. Let the user interface guide you through the cluster configuration for your cluster.

    If you are changing an existing cluster, you can make multiple changes with a single configuration update, such as changing the capacity and upgrading to a new Elasticsearch version in one step.

  4. Save your changes. The new configuration takes a few moments to create.

Region

When choosing a region the general rule is to choose one as close to your application servers as possible in order to minimize the network delay.

Tip

You can select your region only when you create a new cluster, so pick one that works for you. Regions cannot be changed later.

Cluster Size

Depending upon how much data you have and what queries you plan to run, you need to select a cluster size that fits your needs. There is no silver bullet for deciding how much memory you need other than simply testing it. The cluster performance metrics in the Elastic Cloud Console can tell you if your cluster is sized appropriately. Fortunately, you can change the capacity of the cluster later, without any downtime.

Capacity slider in Elastic Cloud

Currently, half the memory is assigned to the JVM heap. For example, on a 32 GB cluster, 16 GB are allotted to heap. The disk-to-RAM ratio currently is 1:24, meaning that you get 24 GB of storage space for each 1 GB of RAM. All clusters are backed by SSD drives.

Tip

For production systems, we recommend not using less than 4 GB of RAM for your cluster, which assigns 2 GB to the JVM heap.

The CPU resources assigned to a cluster are relative to the size of your cluster, meaning that a 32 GB cluster gets twice as much CPU resources as a 16 GB cluster. All clusters are guaranteed their share of CPU resources, as we do not overcommit resources. Smaller clusters up to and including 4 GB of RAM benefit from temporary CPU boosting to improve performance when needed most.

To learn more about how much memory might be needed, see Elasticsearch in Production.

High Availability

High availability is achieved by running a cluster with replicas in multiple data centers (availability zones), to prevent against downtime when infrastructure problems occur. We offer the options of running in one, two, or three data centers.

Running in two data centers or availability zones is our default high availability configuration. It provides reasonably high protection against infrastructure failures and intermittent network problems. You might want three data centers if you need even higher fault tolerance. Just one zone might be sufficient, if the cluster is mainly used for testing or development.

Like many other changes, you change the level of fault tolerance while the cluster is running. For example, when you prepare a new cluster for production use, you can first run it in a single data center and then add another data center right before deploying to production.

While multiple data centers or availability zones increase a cluster’s fault tolerance, they do not protect against problematic searches that cause nodes to run out of memory, for example. For a cluster to be highly reliable and available, it is also important to have enough memory.

The cluster capacity you chose is per data center. The reason for this is that there is no point in having two data centers if the failure of one will result in a cascading error because the remaining data center cannot handle the total load. Through the allocation awareness in Elasticsearch, we configure the nodes so that your cluster will automatically allocate replicas between each data center.

Our article on Elasticsearch in Production covers availability zones and resilience against infrastructure failures in more detail.

Version

Elasticsearch versions are denoted as X.Y.Z, where X is the major version, Y is the minor version, and Z is the patch level or maintenance release. At any given time, the two latest minor versions are guaranteed to be available for deployments of new clusters, such as Elasticsearch 2.4 and 2.3.5. Only the latest patch level within a minor version is usually available for new deployments, such as 2.4.1.

You might sometimes see additional versions listed in the user interface beyond what we guarantee to be available, such as release candidate builds. If versions are listed, they can be deployed.

In order to be able to deliver new features and keep complexity manageable, we also need to be able to discontinue old versions. When a version nears its end-of-life point, you are typically given six months notice in advance to upgrade to one of the available minor versions.

To learn more about how we support Elasticsearch versions in Elastic Cloud, see Version Policy.

Upgrade

You can upgrade Elasticsearch versions for an existing cluster on the Configuration page by selecting a newer version. To learn more about upgrading versions of Elasticsearch and best practices for major version upgrades, see Version Upgrades.

Scripting

The script_fields in filters and facets are one of the features that make Elasticsearch so flexible, but they can allow arbitrary code execution, like Runtime.exec("cat /etc/passwd") and other malicious things. We provide you with three levels of scripting control for each of the script types that are supported: You can disable the scripts completely, enable scripts to run in a sandbox or enable all scripts. Especially for inline scripts, we recommend that you do not enable all inline scripts, because of the security risk that they can pose.

Custom Plugins, Dictionaries, and Scripts

If you have uploaded any plugins or user bundles with dictionaries or scripts then this where you choose to enable them for the cluster.

Tip

Only Gold and Platinum subscriptions have access to uploading custom plugins. All subscription levels, including Standard, can upload scripts and dictionaries.

Plugins

This section has the list of official plugins available for the selected Elasticsearch version. When selecting a plugin from this list you get a version that has been tested with the chosen Elasticsearch version. The main difference between selecting a plugin from this list versus uploading the same plugin as a custom plugin is in who decides the version used. The reason we do not list the version chosen on this page is because we reserve the option to change it when necessary. That said, we will not force a cluster restart for a simple plugin upgrade unless there are severe issues with the current version. In most cases, plugin upgrades are applied lazily, in other words when something else forces a restart like you changing the plan or Elasticsearch runs out of memory.

Kibana

By default, Kibana is disabled. To enable Kibana, simply click Enable.

Enabling Kibana provides you with an endpoint URL, where you can access Kibana. It can take a short while to provision Kibana right after you click Enable, so if you get an error message when you first click the endpoint URL, try again.

Tip

For version 5.0 and later, you can log into Kibana with the elastic superuser to try it out. The password was provided when you created your cluster or can be reset. For versions before 5.0 and if Shield is enabled, you can log into Kibana with the admin user to try it out. The password was provided when you enabled Shield or can be reset. In production systems, you might need to control what Elasticsearch data users can access through Kibana, so you need create credentials that can be used to access the necessary Elasticsearch resources. This means granting read access to the necessary indexes, as well as access to update the .kibana index.

User Settings

You can change how Elasticsearch runs by providing your own user settings. User settings are appended to the elasticsearch.yml configuration file for your cluster and provide custom configuration options.

Elastic Cloud supports the following user settings:

repositories.url.allowed_urls
Enables whitelisting of read-only URL repositories.
reindex.remote.whitelist
Whitelists the hosts that can be reindexed from remotely. Expects a YAML array of host:port strings. Consists of a comma-delimited list of host:port entries. Defaults to ["\*.io:*", "\*.com:*"].
script.painless.regex.enabled
Enables regular expressions for the Painless scripting language.
cluster.indices.close.enable

Enables closing indices in Elasticsearch version 2.2 and later. We strongly recommend leaving this set to false (the default). Closed indices are a data loss risk: If you close an index, it is not included in snapshots and you will not be able to restore the data. Similarly, closed indices are not included when you scale to a different cluster size or during failover operations. You might enable this setting temporarily in order to change the analyzer configuration for an existing index.

Caution

Closed indices are a data loss risk. Enable this setting only temporarily.

X-Pack (for version 5.0 and later)

The following X-Pack settings are supported:

xpack.notification.slack
Configures Slack notification settings.
xpack.notification.hipchat
Configures HipChat notification settings.
xpack.notification.pagerduty
Configures PagerDuty notification settings.
xpack.watcher.trigger.schedule.engine
Defines when the watch execution should start, based on date and time.
xpack.notification.email.html.sanitization.*
Enables email notification settings to sanitize HTML elements in emails that are sent.
xpack.monitoring.collection.interval
Controls how often data samples are collected.
xpack.monitoring.history.duration
Sets the retention duration beyond which the indices created by a monitoring exporter will be automatically deleted.
Watcher and Marvel (for versions before 5.0)

The following Watcher and Marvel settings are supported:

watcher.actions.slack.service
Configures Slack notification settings.
watcher.actions.hipchat.service
Configures HipChat notification settings.
watcher.actions.pagerduty.service
Configures Configures PagerDuty notification settings.
marvel.agent.interval
watcher.trigger.schedule.engine
Defines when the watch execution should start, based on date and time.
Tip

Remember to update user settings for alerts when performing a major version upgrade. For version 5.0 and later, the syntax is different when compared to earlier versions.

If a setting is not on this list, it cannot be set and will be rejected. Additional user settings might added in the future.

Name

This setting allows you to assign a more human-friendly name to your cluster which will be used for future reference in the Elastic Cloud Console. Common choices are dev, prod, test, or something more domain specific.