Deploy a small installationedit

The type of installation is recommended for development, test, and small-scale use cases. You need:

  • 3 hosts with 128 GB RAM
  • 3 availability zones
A small baseline installation with three hosts across three availability zones

Perform the numbered steps on more than one host, as indicated in the illustration.

Before you startedit
  • This type of installation is not recommended for high-traffic workloads.
  • You must not use spinning disks with small ECE installations, as these are not supported when you run allocators and ECE management services on the same server.
  • Note that the small-size ECE installation keeps the directors and coordinators roles (ECE management services) on the same hosts as your allocators and proxies.

Check the recommended JVM Heap sizes

Service JVM Heap Size (Xms and Xmx)

runner

1 GB

allocator

4 GB

zookeeper

4 GB

director

1 GB

constructor

4 GB

admin-console

4 GB

For production environments, you must define the memory settings for each role, except for the proxy role, as starting from ECE 2.4 the JVM proxy was replaced with a Golang-based proxy. If you don’t set any memory setting, the default values are used, which are inadequate for production environments and can lead to performance or stability issues.

Installation stepsedit
  1. Install Elastic Cloud Enterprise on the first host to start a new installation with your first availability zone. This first host holds all runner roles to help bootstrap the rest of the installation.

    bash <(curl -fsSL https://download.elastic.co/cloud/elastic-cloud-enterprise.sh) install \
       --availability-zone MY_ZONE-1 \
       --memory-settings
       '{"runner":{"xms":"1G","xmx":"1G"},
       "allocator":{"xms":"4G","xmx":"4G"},
       "zookeeper":{"xms":"4G","xmx":"4G"},
       "director":{"xms":"1G","xmx":"1G"},
       "constructor":{"xms":"4G","xmx":"4G"},
       "admin-console":{"xms":"4G","xmx":"4G"}}'

    After the installation completes, copy down the coordinator host IP address, user credentials, and roles token information. Keep this information safe.

  2. Generate a new roles token that persists for one hour on the first host, so that other hosts can join your installation with the right role permissions in the next step (referred to as MY_TOKEN). The new token needs to enable all runner roles, which none of the tokens automatically generated by the installation on the first host provide.

    curl -k -H 'Content-Type: application/json' -u admin:PASSWORD https://localhost:12443/api/v1/platform/configuration/security/enrollment-tokens -d '{ "persistent": false, "roles": ["director", "coordinator", "proxy", "allocator"] }'
  3. Install Elastic Cloud Enterprise on a second and third host, placing them into a second and a third availability zone, and assign them the same roles and memory settings as the first host. Make sure you include the coordinator host IP information from Step 1 and the new roles token from Step 2.

    bash <(curl -fsSL https://download.elastic.co/cloud/elastic-cloud-enterprise.sh) install \
      --coordinator-host HOST_IP \
      --roles-token 'MY_TOKEN' \
      --roles "director,coordinator,proxy,allocator" \
      --availability-zone MY_ZONE-2 \
      --memory-settings
      '{"runner":{"xms":"1G","xmx":"1G"},
      "allocator":{"xms":"4G","xmx":"4G"},
      "zookeeper":{"xms":"4G","xmx":"4G"},
      "director":{"xms":"1G","xmx":"1G"},
      "constructor":{"xms":"4G","xmx":"4G"},
      "admin-console":{"xms":"4G","xmx":"4G"}}'
    bash <(curl -fsSL https://download.elastic.co/cloud/elastic-cloud-enterprise.sh) install \
      --coordinator-host HOST_IP \
      --roles-token 'MY_TOKEN' \
      --roles "director,coordinator,proxy,allocator" \
      --availability-zone MY_ZONE-3 \
      --memory-settings
       '{"runner":{"xms":"1G","xmx":"1G"},
       "allocator":{"xms":"4G","xmx":"4G"},
       "zookeeper":{"xms":"4G","xmx":"4G"},
       "director":{"xms":"1G","xmx":"1G"},
       "constructor":{"xms":"4G","xmx":"4G"},
       "admin-console":{"xms":"4G","xmx":"4G"}}'
  4. Change the deployment configuration for the admin-console-elasticsearch, logging-and-metrics, and security clusters to use three availability zones and resize the nodes to use at least 4 GB of RAM. This change makes sure that the clusters used by the administration console are highly available and provisioned sufficiently.
  5. Log into the Cloud UI to provision your deployment.

If necessary, you can scale and deploy a medium installation.

==== Deploy a medium installation

This type of installation is recommended for many production setups. You need:

  • 3 hosts with at least 32 GB RAM each for directors and coordinators (ECE management services), and proxies
  • 3 hosts with 256 GB RAM each for allocators
  • 3 availability zones
A large installation with nine to twelve hosts across three availability zones

Perform the numbered steps on more than one host, as indicated in the illustration.

Before you startedit
  • Monitor the load on proxies and make sure the volume of user requests routed by the proxies does not affect the resources available to the ECE management services.
  • Note that the medium-sized Elastic Cloud Enterprise installation separates the allocator from the director and coordinator roles (ECE management services) and the proxy roles.

Check the recommended JVM Heap sizes

Service JVM Heap Size (Xms and Xmx)

runner

1 GB

allocator

4 GB

zookeeper

4 GB

director

1 GB

constructor

4 GB

admin-console

4 GB

For production environments, you must define the memory settings for each role, except for the proxy role, as starting from ECE 2.4 the JVM proxy was replaced with a Golang-based proxy. If you don’t set any memory setting, the default values are used, which are inadequate for production environments and can lead to performance or stability issues.

Installation stepsedit
  1. Install Elastic Cloud Enterprise on the first host to start a new installation with your first availability zone. This first host holds all runner roles to help bootstrap the rest of the installation, but you will remove some of its roles in a later step.

    bash <(curl -fsSL https://download.elastic.co/cloud/elastic-cloud-enterprise.sh) install \
      --availability-zone MY_ZONE-1 \
      --memory-settings
      '{"runner":{"xms":"1G","xmx":"1G"},
      "zookeeper":{"xms":"4G","xmx":"4G"},
      "director":{"xms":"1G","xmx":"1G"},
      "constructor":{"xms":"4G","xmx":"4G"},
      "admin-console":{"xms":"4G","xmx":"4G"}}'

    After the installation completes, copy down the coordinator host IP address, user credentials, and roles token information. Keep this information safe.

  2. Generate a new roles token that persists for one hour on the first host, so that other hosts can join your installation with the right role permissions in the next step (referred to as MY_TOKEN). The new token needs to enable the director, coordinator and proxy runner roles.

    curl -k -H 'Content-Type: application/json' -u admin:PASSWORD https://localhost:12443/api/v1/platform/configuration/security/enrollment-tokens -d '{ "persistent": false, "roles": ["director", "coordinator", "proxy"] }'
  3. Install Elastic Cloud Enterprise on a second and third host, placing them into a second and a third availability zone, and assign them the director, coordinator, and proxy roles. Do not assign the allocator role, as these runners should not handle any user requests. Make sure you include the coordinator host IP information from Step 1 and the new roles token from Step 2.

    bash <(curl -fsSL https://download.elastic.co/cloud/elastic-cloud-enterprise.sh) install \
      --coordinator-host HOST_IP \
      --roles-token 'MY_TOKEN' \
      --roles "director,coordinator,proxy" \
      --availability-zone MY_ZONE-2 \
      --memory-settings
       '{"runner":{"xms":"1G","xmx":"1G"},
       "zookeeper":{"xms":"4G","xmx":"4G"},
       "director":{"xms":"1G","xmx":"1G"},
       "constructor":{"xms":"4G","xmx":"4G"},
       "admin-console":{"xms":"4G","xmx":"4G"}}'
    bash <(curl -fsSL https://download.elastic.co/cloud/elastic-cloud-enterprise.sh) install \
      --coordinator-host HOST_IP \
      --roles-token 'MY_TOKEN' \
      --roles "director,coordinator,proxy" \
      --availability-zone MY_ZONE-3 \
      --memory-settings
       '{"runner":{"xms":"1G","xmx":"1G"},
       "zookeeper":{"xms":"4G","xmx":"4G"},
       "director":{"xms":"1G","xmx":"1G"},
       "constructor":{"xms":"4G","xmx":"4G"},
       "admin-console":{"xms":"4G","xmx":"4G"}}'
  4. To handle the Elasticsearch and Kibana workload, install Elastic Cloud Enterprise on a fourth, fifth, and sixth host, distributing them evenly across the existing three availability zones and assign them the allocator role. Make sure you include the coordinator host IP information and allocator roles token from Step 1.

    bash <(curl -fsSL https://download.elastic.co/cloud/elastic-cloud-enterprise.sh) install \
      --coordinator-host HOST_IP \
      --roles-token 'ALLOCATOR_TOKEN' \
      --roles "allocator" --availability-zone MY_ZONE-1 \
      --memory-settings '{"runner":{"xms":"1G","xmx":"1G"},"allocator":{"xms":"4G","xmx":"4G"}}'
    
    bash <(curl -fsSL https://download.elastic.co/cloud/elastic-cloud-enterprise.sh) install \
      --coordinator-host HOST_IP \
      --roles-token 'ALLOCATOR_TOKEN' \
      --roles "allocator" \
      --availability-zone MY_ZONE-2 \
      --memory-settings '{"runner":{"xms":"1G","xmx":"1G"},"allocator":{"xms":"4G","xmx":"4G"}}'
    
    bash <(curl -fsSL https://download.elastic.co/cloud/elastic-cloud-enterprise.sh) install \
      --coordinator-host HOST_IP \
      --roles-token 'ALLOCATOR_TOKEN' \
      --roles "allocator" \
      --availability-zone MY_ZONE-3 \
      --memory-settings '{"runner":{"xms":"1G","xmx":"1G"},"allocator":{"xms":"4G","xmx":"4G"}}'

==== Deploy a large installation

This type of installation is recommended for deployments with significant overall search and indexing throughput. You need:

  • 3 hosts with at least 32 GB RAM each for directors and coordinators (ECE management services)
  • 3 or more hosts with 256 GB RAM each for allocators
  • 3 hosts with 16 GB RAM each for proxies
  • 3 availability zones
A large installation with nine to twelve hosts across three availability zones

Perform the numbered steps on more than one host, as indicated in the illustration.

Before you startedit

Note that the large-sized Elastic Cloud Enterprise installation separates the allocator and proxy roles from the director and coordinator roles (ECE management services).

Check the recommended JVM Heap sizes

Service JVM Heap Size (Xms and Xmx)

runner

1 GB

allocator

4 GB

zookeeper

4 GB

director

1 GB

constructor

4 GB

admin-console

4 GB

For production environments, you must define the memory settings for each role, except for the proxy role, as starting from ECE 2.4 the JVM proxy was replaced with a Golang-based proxy. If you don’t set any memory setting, the default values are used, which are inadequate for production environments and can lead to performance or stability issues.

Installation stepsedit
  1. Install Elastic Cloud Enterprise on the first host to start a new installation with your first availability zone. This first host holds all runner roles to help bootstrap the rest of the installation, but you will remove some of its roles in a later step.

    bash <(curl -fsSL https://download.elastic.co/cloud/elastic-cloud-enterprise.sh) install \
      --availability-zone MY_ZONE-1 \
      --memory-settings '{"runner":{"xms":"1G","xmx":"1G"},"zookeeper":{"xms":"4G","xmx":"4G"},"director":{"xms":"1G","xmx":"1G"},"constructor":{"xms":"4G","xmx":"4G"},"admin-console":{"xms":"4G","xmx":"4G"}}'

    After the installation completes, copy down the coordinator host IP address, user credentials, and roles token information. Keep this information safe.

  2. Generate a new roles token that persists for one hour on the first host, so that other hosts can join your installation with the right role permissions in subsequent steps (referred to as MY_TOKEN). The new token needs to enable the director, coordinator, and proxy runner roles.

    curl -k -H 'Content-Type: application/json' -u admin:PASSWORD https://localhost:12443/api/v1/platform/configuration/security/enrollment-tokens -d '{ "persistent": false, "roles": ["director", "coordinator", "proxy"] }'
  3. Install Elastic Cloud Enterprise on a second and third host, placing them into a second and a third availability zone, and assign them the director and coordinator roles. Do not assign the allocator or the proxy role, as these runners should not handle or route any user requests. Make sure you include the coordinator host IP information from Step 1 and the new roles token from Step 2.

    bash <(curl -fsSL https://download.elastic.co/cloud/elastic-cloud-enterprise.sh) install \
     --coordinator-host HOST_IP \
     --roles-token 'MY_TOKEN' \
     --roles "director,coordinator" \
     --availability-zone MY_ZONE-2 \
     --memory-settings '{"runner":{"xms":"1G","xmx":"1G"},"zookeeper":{"xms":"4G","xmx":"4G"},"director":{"xms":"1G","xmx":"1G"},"constructor":{"xms":"4G","xmx":"4G"},"admin-console":{"xms":"4G","xmx":"4G"}}'
    bash <(curl -fsSL https://download.elastic.co/cloud/elastic-cloud-enterprise.sh) install \
      --coordinator-host HOST_IP \
      --roles-token 'MY_TOKEN' \
      --roles "director,coordinator" \
      --availability-zone MY_ZONE-3 \
      --memory-settings '{"runner":{"xms":"1G","xmx":"1G"},"zookeeper":{"xms":"4G","xmx":"4G"},"director":{"xms":"1G","xmx":"1G"},"constructor":{"xms":"4G","xmx":"4G"},"admin-console":{"xms":"4G","xmx":"4G"}}'
  4. To handle the Elasticsearch and Kibana workload, install Elastic Cloud Enterprise on three or more hosts, distributing them evenly across the existing three availability zones, or on however many hosts you think you need initially, and assign them the allocator role. Make sure you include the coordinator host IP information and allocator roles token from Step 1.

    bash <(curl -fsSL https://download.elastic.co/cloud/elastic-cloud-enterprise.sh) install \
      --coordinator-host HOST_IP \
      --roles-token 'ALLOCATOR_TOKEN' \
      --roles "allocator" \
      --availability-zone MY_ZONE-1 \
      --memory-settings '{"runner":{"xms":"1G","xmx":"1G"},"allocator":{"xms":"4G","xmx":"4G"}}'
    
    bash <(curl -fsSL https://download.elastic.co/cloud/elastic-cloud-enterprise.sh) install \
      --coordinator-host HOST_IP \
      --roles-token 'ALLOCATOR_TOKEN' \
      --roles "allocator" \
      --availability-zone MY_ZONE-2 \
      --memory-settings '{"runner":{"xms":"1G","xmx":"1G"},"allocator":{"xms":"4G","xmx":"4G"}}'
    
    bash <(curl -fsSL https://download.elastic.co/cloud/elastic-cloud-enterprise.sh) install \
      --coordinator-host HOST_IP \
      --roles-token 'ALLOCATOR_TOKEN' \
      --roles "allocator" \
      --availability-zone MY_ZONE-3 \
      --memory-settings '{"runner":{"xms":"1G","xmx":"1G"},"allocator":{"xms":"4G","xmx":"4G"}}'
  5. To handle the routing of user requests to Elasticsearch, install Elastic Cloud Enterprise on a three additional hosts, distributing them evenly across the existing three availability zones, and assign them the proxy role. Do not assign any other roles, as these runners should only route user requests. Make sure you include the coordinator host IP information from Step 1 and the new roles token from Step 2.

    bash <(curl -fsSL https://download.elastic.co/cloud/elastic-cloud-enterprise.sh) install \
      --coordinator-host HOST_IP \
      --roles-token 'MY_TOKEN' \
      --roles "proxy" \
      --availability-zone MY_ZONE-1 \
      --memory-settings '{"runner":{"xms":"1G","xmx":"1G"}}'
    
    bash <(curl -fsSL https://download.elastic.co/cloud/elastic-cloud-enterprise.sh) install \
      --coordinator-host HOST_IP \
      --roles-token 'MY_TOKEN' \
      --roles "proxy" \
      --availability-zone MY_ZONE-2 \
      --memory-settings '{"runner":{"xms":"1G","xmx":"1G"}}'
    
    bash <(curl -fsSL https://download.elastic.co/cloud/elastic-cloud-enterprise.sh) install \
      --coordinator-host HOST_IP \
      --roles-token 'MY_TOKEN' \
      --roles "proxy" \
      --availability-zone MY_ZONE-3 \
      --memory-settings '{"runner":{"xms":"1G","xmx":"1G"}}'
  6. Modify the first host you installed Elastic Cloud Enterprise on to prevent it from handling or routing user requests:

== Install ECE on your own premises

Before you start, make sure that your existing infrastructure meets the requirements. When you have prepared your hosts, you can decide to deploy ECE with one of the following operating systems:

and choose your preferred installation type:

=== Configure your operating system

Before installing Elastic Cloud Enterprise, you have to prepare your hosts with one of the following Linux distributions:

==== Ubuntu 16.04 LTS (Xenial Xerus) and 18.04 LTS (Bionic Beaver)

The following instructions show you how to prepare your hosts on Ubuntu 16.04 LTS (Xenial Xerus) and 18.04 LTS (Bionic Beaver).

Install a supported Linux kerneledit

Elastic Cloud Enterprise requires 3.10 or higher. The steps shown here install kernel 4.4.

  1. Refresh the package index files from their sources:

    sudo apt-get update
  2. Install the xfsprogs package available on your system. You can omit the xfsprogs package if you don’t plan to use XFS.

    sudo apt-get install -y xfsprogs
Install Dockeredit

Install Docker LTS version 18.09.9 for Ubuntu 16.04 or 19.03.13 for Ubuntu 18.04

  1. Add the Docker repository:

    curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo \
         apt-key add -
  2. Verify that you have the key with the 9DC8 5822 9FC7 DD38 854A E2D8 8D81 803C 0EBF CD88 fingerprint.

    sudo apt-key fingerprint 0EBFCD88
    ...
    pub   4096R/0EBFCD88 2017-02-22
          Key fingerprint = 9DC8 5822 9FC7 DD38 854A  E2D8 8D81 803C 0EBF CD88
    uid                  Docker Release (CE deb) <docker@docker.com>
    sub   4096R/F273FCD8 2017-02-22
    ...
  3. Add the stable docker repository:

    sudo add-apt-repository \
       "deb [arch=amd64] https://download.docker.com/linux/ubuntu \
       $(lsb_release -cs) \
       stable"
    sudo apt-get update
  4. Install the correct version of the docker-ce package, for Ubuntu 16.04 LTS (Xenial Xerus):

    sudo apt-get install docker-ce=5:18.09.9* docker-ce-cli=5:18.09.9*

    For Ubuntu 18.04 LTS (Bionic Beaver):

    sudo apt-get install docker-ce=5:19.03.13* docker-ce-cli=5:19.03.13*
Set up XFS quotasedit

XFS is required to support disk space quotas for Elasticsearch data directories. Some Linux distributions such as RHEL and CentOS already provide XFS as the default file system. On Ubuntu, we recommend that you set up an XFS file system first.

Disk space quotas set a limit on the amount of disk space an Elasticsearch cluster node can use. Currently, quotas are calculated by a static ratio of 1:32, which means that for every 1 GB of RAM a cluster is given, a cluster node is allowed to consume 32 GB of disk space.

Using LVM, mdadm, or a combination of the two for block device management is possible, but the configuration is not covered here, and it is not supported by Elastic Cloud Enterprise.

You must use XFS on all allocators.

Example: Set up XFS on a single, pre-partitioned block device named /dev/xvdg1.

  1. Format the partition:

    sudo mkfs.xfs /dev/xvdg1
  2. Create the /mnt/data/ directory as a mount point:

    sudo install -o $USER -g $USER -d -m 700 /mnt/data
  3. Add an entry to the /etc/fstab file for the new XFS volume. The default filesystem path used by Elastic Cloud Enterprise is /mnt/data.

    /dev/xvdg1	/mnt/data	xfs	defaults,nofail,x-systemd.automount,prjquota,pquota  0 2
  4. Regenerate the mount files:

    sudo systemctl daemon-reload
    sudo systemctl restart local-fs.target
Update the configurations settingsedit
  1. Stop the Docker service:

    sudo systemctl stop docker
  2. Enable cgroup accounting for memory and swap space.

    1. In the /etc/default/grub file, ensure that the GRUB_CMDLINE_LINUX= variable includes these values:

      cgroup_enable=memory swapaccount=1 cgroup.memory=nokmem
    2. Update your Grub configuration:

      sudo update-grub
  3. Configure kernel parameters

    cat <<EOF | sudo tee -a /etc/sysctl.conf
    # Required by Elasticsearch 5.0 and later
    vm.max_map_count = 262144
    # enable forwarding so the Docker networking works as expected
    net.ipv4.ip_forward = 1
    # Make sure the host doesn't swap too early
    vm.swappiness=1
    EOF
    1. Apply the settings:

      sudo sysctl -p
  4. Adjust the system limits.

    Add the following configuration values to the /etc/security/limits.conf file. These values are derived from our experience with the Elastic Cloud hosted offering and should be used for Elastic Cloud Enterprise as well.

    If you are using a user name other than elastic, adjust the configuration values accordingly.

    *                soft    nofile         1024000
    *                hard    nofile         1024000
    *                soft    memlock        unlimited
    *                hard    memlock        unlimited
    elastic          soft    nofile         1024000
    elastic          hard    nofile         1024000
    elastic          soft    memlock        unlimited
    elastic          hard    memlock        unlimited
    root             soft    nofile         1024000
    root             hard    nofile         1024000
    root             soft    memlock        unlimited
  5. If you did not create the mount point earlier (if you did not set up XFS), create the /mnt/data/ directory as a mount point:

    sudo install -o $USER -g $USER -d -m 700 /mnt/data
  6. If you set up a new device with XFS earlier:

    1. Mount the block device (change the device name if you use a different device than /dev/xvdg1):

      sudo mount /dev/xvdg1
    2. Set the permissions on the newly mounted device:

      sudo chown $USER:$USER /mnt/data
  7. Create the /mnt/data/docker directory for the Docker service storage:

    sudo install -o $USER -g $USER -d -m 700 /mnt/data/docker
Configure the Docker daemon optionsedit

Docker creates a bridge IP address that can conflict with IP addresses on your internal network. To avoid an IP address conflict, change the --bip=172.17.42.1/16 parameter in our examples to something that you know will work. If there is no conflict, you can omit the --bip parameter. The --bip parameter is internal to the host and can be set to the same IP for each host in the cluster. More information on Docker daemon options can be found in the dockerd command line reference.

  1. Update /etc/systemd/system/docker.service.d/docker.conf. If the file path and file do not exist, create them first.

    [Unit]
    Description=Docker Service
    After=multi-user.target
    
    [Service]
    Environment="DOCKER_OPTS=-H unix:///run/docker.sock --data-root /mnt/data/docker --storage-driver=overlay2 --bip=172.17.42.1/16 --raw-logs --icc=false"
    ExecStart=
    ExecStart=/usr/bin/dockerd $DOCKER_OPTS
  2. Apply the updated Docker daemon configuration:

    Reload the Docker daemon configuration:

    sudo systemctl daemon-reload

    Restart the Docker service:

    sudo systemctl restart docker

    Enable Docker to start on boot:

    sudo systemctl enable docker
  3. Enable your user to communicate with the Docker subsystem by adding it to the docker group:

    sudo usermod -aG docker $USER
  4. Recommended: Tune your network settings.

    Create a 70-cloudenterprise.conf file in the /etc/sysctl.d/ file path that includes these network settings:

    cat << SETTINGS | sudo tee /etc/sysctl.d/70-cloudenterprise.conf
    net.ipv4.tcp_max_syn_backlog=65536
    net.core.somaxconn=32768
    net.core.netdev_max_backlog=32768
    SETTINGS
  5. Pin the Docker version to ensure that the package does not get upgraded:

    echo "docker-ce hold" | sudo dpkg --set-selections
  6. Reboot your system to ensure that all configuration changes take effect:

    sudo reboot
  7. After rebooting, verify that your Docker settings persist as expected:

    sudo docker info | grep Root

    If the command returns Docker Root Dir: /mnt/data/docker, then your changes were applied successfully and persist as expected.

    If the command returns Docker Root Dir: /var/lib/docker, then you need to troubleshoot the previous configuration steps until the Docker settings are applied successfully before continuing with the installation process. For more information, see Custom Docker daemon options in the Docker documentation.

  8. Repeat these steps on other hosts that you want to use with Elastic Cloud Enterprise or follow the steps in the next section to start installing Elastic Cloud Enterprise.

==== Red Hat Enterprise Linux (RHEL) and CentOS

The following instructions show you how to prepare your hosts on Red Hat Enterprise Linux (RHEL) and on CentOS.

Install a supported Linux kerneledit

Elastic Cloud Enterprise requires 3.10 or higher.

  1. Use the following command to check your kernel version:

    uname -r

    Kernel-LT has a regression on 4.4.156. In case your OS uses this version, please install another version.

Before you proceed, update the OS, and reboot the system.

  1. Update the system:

    sudo yum update
    sudo reboot
  2. Enable the overlay2 kernel module:

    echo "overlay2" | sudo tee -a /etc/modules-load.d/overlay.conf
  3. Refresh the dynamically generated grub2 configuration and configure grub to boot the newly installed kernel 3.10 or higher:

    sudo grub2-set-default 0
    sudo grub2-mkconfig -o /etc/grub2.cfg
  4. Add the required options to the kernel boot arguments:

    sudo /sbin/grubby --update-kernel=ALL --args='cgroup_enable=memory cgroup.memory=nokmem swapaccount=1'
Install Dockeredit

Which version of Docker you install depends on whether you use Red Hat Enterprise Linux (RHEL) or CentOS.

RHEL 7edit

If you are using RHEL 7, use Docker 1.13 included with your RHEL distribution. Follow the installation instructions provided by Red Hat.

If the Docker daemon does not start correctly after the installation completes, continue with the configuration steps below.

CentOS 7edit

If you are using CentOS 7, install Docker version 18.09.9:

  1. Add the Docker repository:

    sudo tee /etc/yum.repos.d/docker.repo <<-'EOF'
    [dockerrepo]
    name=Docker Repository
    baseurl=https://download.docker.com/linux/centos/7/x86_64/stable
    enabled=1
    gpgcheck=1
    gpgkey=https://download.docker.com/linux/centos/gpg
    EOF
    
    sudo yum makecache fast
  2. Install the latest version of docker-ce 18:

    sudo yum install docker-ce-18.09.9*
RHEL 8 or CentOS 8edit

If you are using RHEL 8 or CentOS 8, install the latest version of docker-ce 19:

  1. Add the Docker repository:

    sudo yum config-manager --add-repo=https://download.docker.com/linux/centos/docker-ce.repo
    
    sudo yum makecache --timer
  2. Install containerd.io:

    sudo yum -y install https://download.docker.com/linux/centos/7/x86_64/stable/Packages/containerd.io-1.2.6-3.3.el7.x86_64.rpm
  3. Install the latest version of docker-ce 19:

    sudo yum install docker-ce-19.03.13*
Set up XFS quotasedit

XFS quotas are required to support disk space quotas for Elasticsearch data directories. Some Linux distributions such as RHEL and CentOS already provide XFS as the default file system; however, quotas might be disabled.

Disk space quotas set a limit on the amount of disk space a cluster node can use. Currently, quotas are calculated by a static ratio of 1:32, which means that for every 1 GB of RAM a cluster is given, a cluster node is allowed to consume 32 GB of disk space.

You can use use LVM, mdadm, or a combination of the two for block device management, but this configuration is not documented nor is it supported in Elastic Cloud Enterprise.

You must use XFS on all allocators.

To set up XFS with quotas on a single, pre-partitioned block device named /dev/xvdg1:

  1. Format the partition:

    sudo mkfs.xfs /dev/xvdg1
  2. Create the /mnt/data/ directory as a mount point:

    sudo install -o $USER -g $USER -d -m 700 /mnt/data
  3. Modify the entry for the XFS volume in the /etc/fstab file to add pquota,prjquota. The default filesystem path used by Elastic Cloud Enterprise is /mnt/data.

    /dev/xvdg1	/mnt/data	xfs	defaults,nofail,x-systemd.automount,prjquota,pquota  0 2
  4. Regenerate the mount files:

    sudo systemctl daemon-reload
    sudo systemctl restart local-fs.target
Update the configurations settingsedit
  1. Stop the Docker service:

    sudo systemctl stop docker
  2. Configure kernel parameters:

    cat <<EOF | sudo tee -a /etc/sysctl.conf
    # Required by Elasticsearch 5.0 and later
    vm.max_map_count = 262144
    # enable the setting in order for Docker remove the containers cleanly
    fs.may_detach_mounts = 1
    # enable forwarding so the Docker networking works as expected
    net.ipv4.ip_forward = 1
    # Make sure the host doesn't swap too early
    vm.swappiness=1
    EOF
    1. Apply the settings:

      RHEL/Centos 7:

      sudo sysctl -p
      sudo systemctl restart network

      RHEL/Centos 8:

      sudo sysctl -p
      sudo systemctl restart NetworkManager
  3. Adjust the system limits.

    Add the following configuration values to the /etc/security/limits.conf file. If you are using a user name other than elastic, adjust the configuration values accordingly.

    *                soft    nofile         1024000
    *                hard    nofile         1024000
    *                soft    memlock        unlimited
    *                hard    memlock        unlimited
    elastic          soft    nofile         1024000
    elastic          hard    nofile         1024000
    elastic          soft    memlock        unlimited
    elastic          hard    memlock        unlimited
    root             soft    nofile         1024000
    root             hard    nofile         1024000
    root             soft    memlock        unlimited

    The default limit for number of processes is too low. Remove it and rely on the kernel limit instead (for RHEL/Centos 7 only).

    sudo rm /etc/security/limits.d/20-nproc.conf
  4. If you did not create the mount point earlier (if you did not set up XFS), create the /mnt/data/ directory as a mount point:

    sudo install -o $USER -g $USER -d -m 700 /mnt/data
  5. If you set up a new device with XFS earlier:

    1. Mount the block device (change the device name if you use a different device than /dev/xvdg1):

      sudo mount /dev/xvdg1
    2. Set the permissions on the newly mounted device:

      sudo chown $USER:$USER /mnt/data
  6. Create the /mnt/data/docker directory for the Docker service storage:

    sudo install -o $USER -g $USER -d -m 700 /mnt/data/docker
  7. Disable the firewalld service. The service is not compatible with Docker and interferes with the installation of ECE. You must disable firewalld before installing or reinstalling ECE.

    sudo systemctl disable firewalld
Configure the Docker daemon optionsedit

Docker creates a bridge IP address that can conflict with IP addresses on your internal network. To avoid an IP address conflict, change the --bip=172.17.42.1/16 parameter in our examples to something that you know will work. If there is no conflict, you can omit the --bip parameter. The --bip parameter is internal to the host and can be set to the same IP for each host in the cluster. More information on Docker daemon options can be found in the dockerd command line reference.

  1. Update the /etc/systemd/system/docker.service.d/docker.conf file. If the file path and file do not exist, create them.

    [Unit]
    Description=Docker Service
    After=multi-user.target
    
    [Service]
    ExecStart=
    ExecStart=/usr/bin/dockerd --data-root /mnt/data/docker --storage-driver=overlay2 --bip=172.17.42.1/16 --raw-logs --icc=false

    With Docker version 1.13.x, Docker will not start with the --data-root option. If this is the case, try using the --graph option instead.

  2. Apply the updated Docker daemon configuration:

    1. Reload the Docker daemon configuration:

      sudo systemctl daemon-reload
    2. Restart the Docker service:

      sudo systemctl restart docker
    3. Enable Docker to start on boot:

      sudo systemctl enable docker
  3. Enable your user to communicate with the Docker subsystem by adding it to the docker group:

    sudo usermod -aG docker $USER
  4. Pin the Docker version to ensure that the docker-ce package does not get upgraded:

    echo "exclude=docker-ce" | sudo tee -a /etc/yum.conf
  5. Enable your user to communicate with the Docker subsystem by adding it to the docker group:

    sudo groupadd -f docker
    sudo usermod -aG root,docker $USER
    sudo chown root:docker /var/run/docker.sock
  6. Recommended: Tune your network settings.

    Create a 70-cloudenterprise.conf file in the /etc/sysctl.d/ file path that includes these network settings:

    cat << SETTINGS | sudo tee /etc/sysctl.d/70-cloudenterprise.conf
    net.ipv4.tcp_max_syn_backlog=65536
    net.core.somaxconn=32768
    net.core.netdev_max_backlog=32768
    SETTINGS
  7. Reboot your system to ensure that all configuration changes take effect:

    sudo reboot
  8. After rebooting, verify that your Docker settings persist as expected:

    sudo docker info | grep Root

    If the command returns Docker Root Dir: /mnt/data/docker, then your changes were applied successfully and persist as expected.

    If the command returns Docker Root Dir: /var/lib/docker, then you need to troubleshoot the previous configuration steps until the Docker settings are applied successfully before continuing with the installation process. For more information, see Custom Docker daemon options in the Docker documentation.

  9. Repeat these steps on other hosts that you want to use with Elastic Cloud Enterprise or follow the steps in the next section to start installing Elastic Cloud Enterprise.

==== SUSE Linux Enterprise Server (SLES) 12

The following instructions show you how to prepare your hosts on SLES 12.

If you want to install Elastic Cloud Enterprise on your own hosts, the steps for preparing your hosts can take a bit of time. There are two ways you can approach this:

  • Think like a minimalist: Install the correct version of Docker on hosts that meet the prerequisites for Elastic Cloud Enterprise, then skip ahead and install Elastic Cloud Enterprise. Be aware that some checks during the installation can fail with this approach, which will mean doing further host preparation work before retrying the installation.
  • Cover your bases: If you want to make absolutely sure that your installation of Elastic Cloud Enterprise can succeed on hosts that meet the prerequisites, or if any of the checks during the installation failed previously, run through the full preparation steps in this section and then and install Elastic Cloud Enterprise. You’ll do a bit more work now, but life will be simpler later on.

Regardless of which approach you take, the steps in this section need to be performed on every host that you want to use with Elastic Cloud Enterprise.

Install Docker version 18.09.9edit
  1. Add the SLES Virtualization:containers repository:

    sudo zypper ar -t rpm-md https://download.opensuse.org/repositories/Virtualization:containers/SLE_12_SP3/Virtualization:containers.repo
    sudo zypper refresh
  2. Install the correct version of the docker package:

    sudo zypper install -y docker-18.09.7_ce-98.43.1
Set up XFS on SLESedit

Elastic Cloud Enterprise can run without XFS, but XFS is required to support disk space quotas for Elasticsearch data directories. Some Linux distributions such as RHEL and CentOS already provide XFS as the default file system. On SLES 12, we recommend that you set up an XFS file system first.

Disk space quotas set a limit on the amount of disk space an Elasticsearch cluster node can use. Currently, quotas are calculated by a static ratio of 1:32, which means that for every 1 GB of RAM a cluster is given, a cluster node is allowed to consume 32 GB of disk space.

Using LVM, mdadm, or a combination of the two for block device management is possible, but the configuration is not covered here, nor is it provided as part of supporting Elastic Cloud Enterprise.

If you use XFS, you must use XFS on all allocators.

Example: Set up XFS on a single, pre-partitioned block device named /dev/xvdg1.

  1. Format the partition:

    sudo mkfs.xfs /dev/xvdg1
  2. Create the /mnt/data/ directory as a mount point:

    sudo install -o $USER -g $USER -d -m 700 /mnt/data
  3. Add an entry to the /etc/fstab file for the new XFS volume. The default filesystem path used by Elastic Cloud Enterprise is /mnt/data.

    /dev/xvdg1	/mnt/data	xfs	defaults,pquota,prjquota,x-systemd.automount  0 0
  4. Regenerate the mount files:

    sudo mount -a
Update the configurations settingsedit
  1. Stop the Docker service:

    sudo systemctl stop docker
  2. Enable cgroup accounting for memory and swap space.

    1. In the /etc/default/grub file, ensure that the GRUB_CMDLINE_LINUX= variable includes these values:

      cgroup_enable=memory swapaccount=1 cgroup.memory=nokmem
    2. Update your Grub configuration:

      sudo update-bootloader
  3. Configure kernel parameters

    cat <<EOF | sudo tee -a /etc/sysctl.conf
    # Required by Elasticsearch 5.0 and later
    vm.max_map_count = 262144
    # enable forwarding so the Docker networking works as expected
    net.ipv4.ip_forward = 1
    # Make sure the host doesn't swap too early
    vm.swappiness=1
    EOF
    1. Apply the settings:

      sudo sysctl -p
      sudo service network restart
  4. Adjust the system limits.

    Add the following configuration values to the /etc/security/limits.conf file. These values are derived from our experience with the Elastic Cloud hosted offering and should be used for Elastic Cloud Enterprise as well.

    If you are using a user name other than elastic, adjust the configuration values accordingly.

    *                soft    nofile         1024000
    *                hard    nofile         1024000
    *                soft    memlock        unlimited
    *                hard    memlock        unlimited
    elastic          soft    nofile         1024000
    elastic          hard    nofile         1024000
    elastic          soft    memlock        unlimited
    elastic          hard    memlock        unlimited
    root             soft    nofile         1024000
    root             hard    nofile         1024000
    root             soft    memlock        unlimited
  5. If you did not create the mount point earlier (if you did not set up XFS), create the /mnt/data/ directory as a mount point:

    sudo install -o $USER -g $USER -d -m 700 /mnt/data
  6. If you set up a new device with XFS earlier:

    1. Mount the block device (change the device name if you use a different device than /dev/xvdg1):

      sudo mount /dev/xvdg1
    2. Set the permissions on the newly mounted device:

      sudo chown $USER:$USER /mnt/data
  7. Create the /mnt/data/docker directory for the Docker service storage:

    sudo install -o $USER -g $USER -d -m 700 /mnt/data/docker
Configure the Docker daemon optionsedit
  1. Update /etc/sysconfig/docker. If the file path and file do not exist, create them first.

    • Docker version 18.09.9

      ## Path           : System/Management
      ## Description    : Extra cli switches for docker daemon
      ## Type           : string
      ## Default        : ""
      ## ServiceRestart : docker
      #
      DOCKER_OPTS="-H unix:///run/docker.sock -g /mnt/data/docker --storage-driver=overlay --bip=172.17.42.1/16 --raw-logs --icc=false"
  2. Apply the updated Docker daemon configuration:

    Reload the Docker daemon configuration:

    sudo systemctl daemon-reload

    Restart the Docker service:

    sudo systemctl restart docker

    Enable Docker to start on boot:

    sudo systemctl enable docker
  3. Enable your user to communicate with the Docker subsystem by adding it to the docker group:

    sudo usermod -aG docker $USER
  4. Recommended: Tune your network settings.

    Create a 70-cloudenterprise.conf file in the /etc/sysctl.d/ file path that includes these network settings:

    cat << SETTINGS | sudo tee /etc/sysctl.d/70-cloudenterprise.conf
    net.ipv4.tcp_max_syn_backlog=65536
    net.core.somaxconn=32768
    net.core.netdev_max_backlog=32768
    net.ipv4.tcp_keepalive_time=1800
    net.netfilter.nf_conntrack_tcp_timeout_established=7200
    net.netfilter.nf_conntrack_max=262140
    SETTINGS
    1. Ensure settings in /etc/sysctl.d/*.conf are applied on boot

      SCRIPT_LOCATION="/var/lib/cloud/scripts/per-boot/00-load-sysctl-settings"
      sudo sh -c "cat << EOF > ${SCRIPT_LOCATION}
      #!/bin/bash
      
      set -x
      
      lsmod | grep ip_conntrack || modprobe ip_conntrack
      
      sysctl --system
      EOF
      "
      sudo chmod +x ${SCRIPT_LOCATION}
  5. Reboot your system to ensure that all configuration changes take effect:

    sudo reboot
  6. After rebooting, verify that your Docker settings persist as expected:

    sudo docker info | grep Root

    If the command returns Docker Root Dir: /mnt/data/docker, then your changes were applied successfully and persist as expected.

    If the command returns Docker Root Dir: /var/lib/docker, then you need to troubleshoot the previous configuration steps until the Docker settings are applied successfully before continuing with the installation process. For more information, see Custom Docker daemon options in the Docker documentation.

  7. Repeat these steps on other hosts that you want to use with Elastic Cloud Enterprise or follow the steps in the next section to start installing Elastic Cloud Enterprise.

=== Install ECE

Choose the Elastic Cloud Enterprise deployment scenario that best fits your business needs:

==== Deploy a small installation

The type of installation is recommended for development, test, and small-scale use cases. You need:

  • 3 hosts with 128 GB RAM
  • 3 availability zones
A small baseline installation with three hosts across three availability zones

Perform the numbered steps on more than one host, as indicated in the illustration.

Before you startedit
  • This type of installation is not recommended for high-traffic workloads.
  • You must not use spinning disks with small ECE installations, as these are not supported when you run allocators and ECE management services on the same server.
  • Note that the small-size ECE installation keeps the directors and coordinators roles (ECE management services) on the same hosts as your allocators and proxies.

Check the recommended JVM Heap sizes

Service JVM Heap Size (Xms and Xmx)

runner

1 GB

allocator

4 GB

zookeeper

4 GB

director

1 GB

constructor

4 GB

admin-console

4 GB

For production environments, you must define the memory settings for each role, except for the proxy role, as starting from ECE 2.4 the JVM proxy was replaced with a Golang-based proxy. If you don’t set any memory setting, the default values are used, which are inadequate for production environments and can lead to performance or stability issues.

Installation stepsedit
  1. Install Elastic Cloud Enterprise on the first host to start a new installation with your first availability zone. This first host holds all runner roles to help bootstrap the rest of the installation.

    bash <(curl -fsSL https://download.elastic.co/cloud/elastic-cloud-enterprise.sh) install \
       --availability-zone MY_ZONE-1 \
       --memory-settings
       '{"runner":{"xms":"1G","xmx":"1G"},
       "allocator":{"xms":"4G","xmx":"4G"},
       "zookeeper":{"xms":"4G","xmx":"4G"},
       "director":{"xms":"1G","xmx":"1G"},
       "constructor":{"xms":"4G","xmx":"4G"},
       "admin-console":{"xms":"4G","xmx":"4G"}}'

    After the installation completes, copy down the coordinator host IP address, user credentials, and roles token information. Keep this information safe.

  2. Generate a new roles token that persists for one hour on the first host, so that other hosts can join your installation with the right role permissions in the next step (referred to as MY_TOKEN). The new token needs to enable all runner roles, which none of the tokens automatically generated by the installation on the first host provide.

    curl -k -H 'Content-Type: application/json' -u admin:PASSWORD https://localhost:12443/api/v1/platform/configuration/security/enrollment-tokens -d '{ "persistent": false, "roles": ["director", "coordinator", "proxy", "allocator"] }'
  3. Install Elastic Cloud Enterprise on a second and third host, placing them into a second and a third availability zone, and assign them the same roles and memory settings as the first host. Make sure you include the coordinator host IP information from Step 1 and the new roles token from Step 2.

    bash <(curl -fsSL https://download.elastic.co/cloud/elastic-cloud-enterprise.sh) install \
      --coordinator-host HOST_IP \
      --roles-token 'MY_TOKEN' \
      --roles "director,coordinator,proxy,allocator" \
      --availability-zone MY_ZONE-2 \
      --memory-settings
      '{"runner":{"xms":"1G","xmx":"1G"},
      "allocator":{"xms":"4G","xmx":"4G"},
      "zookeeper":{"xms":"4G","xmx":"4G"},
      "director":{"xms":"1G","xmx":"1G"},
      "constructor":{"xms":"4G","xmx":"4G"},
      "admin-console":{"xms":"4G","xmx":"4G"}}'
    bash <(curl -fsSL https://download.elastic.co/cloud/elastic-cloud-enterprise.sh) install \
      --coordinator-host HOST_IP \
      --roles-token 'MY_TOKEN' \
      --roles "director,coordinator,proxy,allocator" \
      --availability-zone MY_ZONE-3 \
      --memory-settings
       '{"runner":{"xms":"1G","xmx":"1G"},
       "allocator":{"xms":"4G","xmx":"4G"},
       "zookeeper":{"xms":"4G","xmx":"4G"},
       "director":{"xms":"1G","xmx":"1G"},
       "constructor":{"xms":"4G","xmx":"4G"},
       "admin-console":{"xms":"4G","xmx":"4G"}}'
  4. Change the deployment configuration for the admin-console-elasticsearch, logging-and-metrics, and security clusters to use three availability zones and resize the nodes to use at least 4 GB of RAM. This change makes sure that the clusters used by the administration console are highly available and provisioned sufficiently.
  5. Log into the Cloud UI to provision your deployment.

If necessary, you can scale and deploy a medium installation.