Your Own Hosts

We currently provide instructions for preparing your own hosts on the following Linux distributions:

Ubuntu 14.04 LTS (Trusty Tahr)

The following instructions show you how to prepare your hosts on Ubuntu Trusty.

If you want to install Elastic Cloud Enterprise on own hosts, the steps for preparing your hosts can take a bit of time. There are two ways you can approach this:

  • Think like a minimalist: Install the correct version of Docker on hosts that meet the prerequisites for Elastic Cloud Enterprise, then skip ahead and install Elastic Cloud Enterprise. This approach is similar to the Quick Start Guide. Be aware that some checks during the installation can fail with this approach, which will mean doing further host preparation work before retrying the installation.
  • Cover your bases: If you want to make absolutely sure that your installation of Elastic Cloud Enterprise can succeed on hosts that meet the prerequisites, or if any of the checks during the installation failed previously, run through the full preparation steps in this section and then and install Elastic Cloud Enterprise. You’ll do a bit more work now, but life will be simpler later on.

Regardless of which approach you take, the steps in this section need to be performed on every host that you want to use with Elastic Cloud Enterprise.

Install a Supported Linux Kernel

Elastic Cloud Enterprise requires 3.10 or higher. The steps shown here install kernel 4.4.

  1. Refresh the package index files from their sources:

    sudo apt-get update
  2. Install the Linux kernel and the xfsprogs package available on your system. You can omit the xfsprogs package if you don’t plan to use XFS.

    sudo apt-get install -y linux-generic-lts-xenial xfsprogs

Install Docker Version 1.11

  1. Recommended: Add the Docker repository to make installing the correct version of Docker easier:

    sudo apt-key adv --keyserver keyserver.ubuntu.com --recv 58118E89F3A912897C070ADBF76221572C52609D
    echo deb https://apt.dockerproject.org/repo ubuntu-trusty main | sudo tee /etc/apt/sources.list.d/docker.list
    sudo apt-get update
  2. Install the correct version of the docker-engine package:

    sudo apt-get install docker-engine=1.11*

Recommended: Set Up XFS on Ubuntu

Elastic Cloud Enterprise can run without XFS, but XFS is required to support disk space quotas for Elasticsearch data directories. Some Linux distributions such as RHEL and CentOS already provide XFS as the default file system. On Ubuntu, we recommend that you set up an XFS file system first.

Disk space quotas set a limit on the amount of disk space a cluster node can use. Currently, quotas are calculated by a static ratio of 1:32, which means that for every 1 GB of RAM a cluster is given, a cluster node is allowed to consume 32 GB of disk space.

Note

Using LVM, mdadm, or a combination of the two for block device management is possible, but the configuration is not covered here, nor is it provided as part of supporting Elastic Cloud Enterprise.

Example: Set up XFS on a single, pre-partitioned block device named /dev/xvdg1.

  1. Format the partition:

    sudo mkfs.xfs /dev/xvdg1
  2. Create the /mnt/data/ directory as a mount point:

    sudo install -o $USER -g $USER -d -m 700 /mnt/data
  3. Add an entry to the /etc/fstab file for the new XFS volume. The default filesystem path used by Elastic Cloud Enterprise is /mnt/data.

    /dev/xvdg1 /mnt/data xfs  defaults,nofail,pquota,prjquota  0 2

Update the Configurations Settings

  1. Stop the Docker service:

    sudo service docker stop
  2. Enable cgroup accounting for memory and swap space.

    1. In the /etc/default/grub file, ensure that the GRUB_CMDLINE_LINUX= variable includes these values:

      cgroup_enable=memory swapaccount=1
    2. Update your Grub configuration:

      sudo update-grub
  3. If you plan to run Elasticsearch 5.0 and later: Update the /etc/sysctl.conf file to set the maximum number of mapped memory areas a process can have:

    echo "vm.max_map_count=262144" | sudo tee -a /etc/sysctl.conf
  4. Verify that IPv4 forwarding is enabled:

    cat /proc/sys/net/ipv4/ip_forward

    If the output of this command is not 1, you must enable IPv4 forwarding so that Docker can work as expected:

    1. Update the /etc/sysctl.conf file to include this line:

      net.ipv4.ip_forward = 1
    2. Restart the network service:

      sudo sysctl -p
    3. Verify that the change is applied:

      cat /proc/sys/net/ipv4/ip_forward

      The output 1 indicates that IPv4 forwarding is turned on.

  5. Adjust the system limits.

    Add the following configuration values to the /etc/security/limits.conf file. These values are derived from our experience with the Elastic Cloud hosted offering and should be used for Elastic Cloud Enterprise as well.

    Tip

    If you are using a user name other than elastic, adjust the configuration values accordingly.

    *                soft    nofile         1024000
    *                hard    nofile         1024000
    *                soft    memlock        unlimited
    *                hard    memlock        unlimited
    elastic          soft    nofile         1024000
    elastic          hard    nofile         1024000
    elastic          soft    memlock        unlimited
    elastic          hard    memlock        unlimited
    root             soft    nofile         1024000
    root             hard    nofile         1024000
    root             soft    memlock        unlimited
  6. If you did not create the mount point earlier (if you did not set up XFS), create the /mnt/data/ directory as a mount point:

    sudo install -o $USER -g $USER -d -m 700 /mnt/data
  7. If you set up a new device with XFS earlier:

    1. Mount the block device (change the device name if you use a different device than /dev/xvdg1):

      sudo mount /dev/xvdg1
    2. Set the permissions on the newly mounted device:

      sudo chown $USER:$USER /mnt/data
  8. Create the /mnt/data/docker directory for the Docker service storage:

    sudo install -o $USER -g $USER -d -m 700 /mnt/data/docker
  9. Configure the Docker daemon options.

    • Set the new data path, set the storage driver, and specify the bridge IP address with the --bip parameter if there is a conflict or omit the parameter.

      Update /etc/default/docker:

      DOCKER_OPTS="-g /mnt/data/docker --storage-driver=aufs --bip=172.17.42.1/16"
  10. Apply the updated Docker daemon configuration by restarting the Docker service:

    sudo service docker restart
  11. Enable your user to communicate with the Docker subsystem by adding it to the docker group:

    sudo usermod -aG docker $USER
  12. Recommended: Tune your network settings.

    Create a 70-cloudenterprise.conf file in the /etc/sysctl.d/ file path that includes these network settings:

    cat << SETTINGS | sudo tee /etc/sysctl.d/70-cloudenterprise.conf
    net.ipv4.tcp_max_syn_backlog=65536
    net.core.somaxconn=32768
    net.core.netdev_max_backlog=32768
    SETTINGS
  13. Pin the Docker version to ensure that the docker-engine package does not get upgraded:

    echo "docker-engine hold" | sudo dpkg --set-selections
  14. Reboot your system to ensure that all configuration changes take effect:

    sudo reboot
  15. After rebooting, verify that your Docker settings persist as expected:

    sudo docker info | grep Root

    If the command returns Docker Root Dir: /mnt/data/docker, then your changes were applied successfully and persist as expected.

    If the command returns Docker Root Dir: /var/lib/docker, then you need to troubleshoot the previous configuration steps until the Docker settings are applied successfully before continuing with the installation process. For more information, see Custom Docker daemon options in the Docker documentation.

  16. Repeat these steps on other hosts that you want to use with Elastic Cloud Enterprise or follow the steps in the next section to start installing Elastic Cloud Enterprise.

Ubuntu 16.04 LTS (Xenial Xerus)

The following instructions show you how to prepare your hosts on Ubuntu Xenial.

If you want to install Elastic Cloud Enterprise on own hosts, the steps for preparing your hosts can take a bit of time. There are two ways you can approach this:

  • Think like a minimalist: Install the correct version of Docker on hosts that meet the prerequisites for Elastic Cloud Enterprise, then skip ahead and install Elastic Cloud Enterprise. This approach is similar to the Quick Start Guide. Be aware that some checks during the installation can fail with this approach, which will mean doing further host preparation work before retrying the installation.
  • Cover your bases: If you want to make absolutely sure that your installation of Elastic Cloud Enterprise can succeed on hosts that meet the prerequisites, or if any of the checks during the installation failed previously, run through the full preparation steps in this section and then and install Elastic Cloud Enterprise. You’ll do a bit more work now, but life will be simpler later on.

Regardless of which approach you take, the steps in this section need to be performed on every host that you want to use with Elastic Cloud Enterprise.

Install A Supported Linux Kernel

Elastic Cloud Enterprise requires 3.10 or higher. The steps shown here install kernel 4.4.

  1. Refresh the package index files from their sources:

    sudo apt-get update
  2. Install the Linux kernel and the xfsprogs package available on your system. You can omit the xfsprogs package if you don’t plan to use XFS.

    sudo apt-get install -y linux-generic-lts-xenial xfsprogs

Install Docker Version 1.11

  1. Recommended: Add the Docker repository to make installing the correct version of Docker easier:

    sudo apt-key adv --keyserver keyserver.ubuntu.com --recv 58118E89F3A912897C070ADBF76221572C52609D
    echo deb https://apt.dockerproject.org/repo ubuntu-xenial main | sudo tee /etc/apt/sources.list.d/docker.list
    sudo apt-get update
  2. Install the correct version of the docker-engine package:

    sudo apt-get install docker-engine=1.11*

Recommended: Set Up XFS on Ubuntu

Elastic Cloud Enterprise can run without XFS, but XFS is required to support disk space quotas for Elasticsearch data directories. Some Linux distributions such as RHEL and CentOS already provide XFS as the default file system. On Ubuntu, we recommend that you set up an XFS file system first.

Disk space quotas set a limit on the amount of disk space a cluster node can use. Currently, quotas are calculated by a static ratio of 1:32, which means that for every 1 GB of RAM a cluster is given, a cluster node is allowed to consume 32 GB of disk space.

Note

Using LVM, mdadm, or a combination of the two for block device management is possible, but the configuration is not covered here, nor is it provided as part of supporting Elastic Cloud Enterprise.

Example: Set up XFS on a single, pre-partitioned block device named /dev/xvdg1.

  1. Format the partition:

    sudo mkfs.xfs /dev/xvdg1
  2. Create the /mnt/data/ directory as a mount point:

    sudo install -o $USER -g $USER -d -m 700 /mnt/data
  3. Add an entry to the /etc/fstab file for the new XFS volume. The default filesystem path used by Elastic Cloud Enterprise is /mnt/data.

    /dev/xvdf1      /mnt/data       xfs     defaults,nofail,x-systemd.automount,prjquota,pquota  0 2
  4. Regenerate the mount files:

    sudo systemctl daemon-reload
    sudo systemctl restart local-fs.target

Update the Configurations Settings

  1. Stop the Docker service:

    sudo systemctl stop docker
  2. Enable cgroup accounting for memory and swap space.

    1. In the /etc/default/grub file, ensure that the GRUB_CMDLINE_LINUX= variable includes these values:

      cgroup_enable=memory swapaccount=1
    2. Update your Grub configuration:

      sudo update-grub
  3. If you plan to run Elasticsearch 5.0 and later: Update the /etc/sysctl.conf file to set the maximum number of mapped memory areas a process can have:

    echo "vm.max_map_count=262144" | sudo tee -a /etc/sysctl.conf
  4. Verify that IPv4 forwarding is enabled:

    cat /proc/sys/net/ipv4/ip_forward

    If the output of this command is not 1, you must enable IPv4 forwarding so that Docker can work as expected:

    1. Update the /etc/sysctl.conf file to include this line:

      net.ipv4.ip_forward = 1
    2. Restart the network service:

      sudo sysctl -p
    3. Verify that the change is applied:

      cat /proc/sys/net/ipv4/ip_forward

      The output 1 indicates that IPv4 forwarding is turned on.

  5. Adjust the system limits.

    Add the following configuration values to the /etc/security/limits.conf file. These values are derived from our experience with the Elastic Cloud hosted offering and should be used for Elastic Cloud Enterprise as well.

    Tip

    If you are using a user name other than elastic, adjust the configuration values accordingly.

    *                soft    nofile         1024000
    *                hard    nofile         1024000
    *                soft    memlock        unlimited
    *                hard    memlock        unlimited
    elastic          soft    nofile         1024000
    elastic          hard    nofile         1024000
    elastic          soft    memlock        unlimited
    elastic          hard    memlock        unlimited
    root             soft    nofile         1024000
    root             hard    nofile         1024000
    root             soft    memlock        unlimited
  6. If you did not create the mount point earlier (if you did not set up XFS), create the /mnt/data/ directory as a mount point:

    sudo install -o $USER -g $USER -d -m 700 /mnt/data
  7. If you set up a new device with XFS earlier:

    1. Mount the block device (change the device name if you use a different device than /dev/xvdg1):

      sudo mount /dev/xvdg1
    2. Set the permissions on the newly mounted device:

      sudo chown $USER:$USER /mnt/data
  8. Create the /mnt/data/docker directory for the Docker service storage:

    sudo install -o $USER -g $USER -d -m 700 /mnt/data/docker
  9. Configure the Docker daemon options.

    Update /etc/systemd/system/docker.service.d/docker.conf. If the file path and file do not exist, create them first.

    [Unit]
    Description=Docker Service
    After=multi-user.target
    
    [Service]
    Environment="DOCKER_OPTS=-H unix:///run/docker.sock -g /mnt/data/docker --storage-driver=aufs --bip=172.17.42.1/16 --raw-logs"
    ExecStart=
    ExecStart=/usr/bin/docker daemon $DOCKER_OPTS
  10. Apply the updated Docker daemon configuration:

    Reload the Docker daemon configuration:

    sudo systemctl daemon-reload

    Restart the Docker service:

    sudo systemctl restart docker

    Enable Docker to start on boot:

    sudo systemctl enable docker
  11. Enable your user to communicate with the Docker subsystem by adding it to the docker group:

    sudo usermod -aG docker $USER
  12. Recommended: Tune your network settings.

    Create a 70-cloudenterprise.conf file in the /etc/sysctl.d/ file path that includes these network settings:

    cat << SETTINGS | sudo tee /etc/sysctl.d/70-cloudenterprise.conf
    net.ipv4.tcp_max_syn_backlog=65536
    net.core.somaxconn=32768
    net.core.netdev_max_backlog=32768
    SETTINGS
  13. Pin the Docker version to ensure that the docker-engine package does not get upgraded:

    echo "docker-engine hold" | sudo dpkg --set-selections
  14. Reboot your system to ensure that all configuration changes take effect:

    sudo reboot
  15. After rebooting, verify that your Docker settings persist as expected:

    sudo docker info | grep Root

    If the command returns Docker Root Dir: /mnt/data/docker, then your changes were applied successfully and persist as expected.

    If the command returns Docker Root Dir: /var/lib/docker, then you need to troubleshoot the previous configuration steps until the Docker settings are applied successfully before continuing with the installation process. For more information, see Custom Docker daemon options in the Docker documentation.

  16. Repeat these steps on other hosts that you want to use with Elastic Cloud Enterprise or follow the steps in the next section to start installing Elastic Cloud Enterprise.

Red Hat Enterprise Linux (RHEL) 7 and CentOS 7

The following instructions show you how to prepare your hosts on Red Hat Enterprise Linux (RHEL) 7 or later and on CentOS 7 or later.

If you want to install Elastic Cloud Enterprise on own hosts, the steps for preparing your hosts can take a bit of time. There are two ways you can approach this:

  • Think like a minimalist: Install the correct version of Docker on hosts that meet the prerequisites for Elastic Cloud Enterprise, then skip ahead and install Elastic Cloud Enterprise. This approach is similar to the Quick Start Guide. Be aware that some checks during the installation can fail with this approach, which will mean doing further host preparation work before retrying the installation.
  • Cover your bases: If you want to make absolutely sure that your installation of Elastic Cloud Enterprise can succeed on hosts that meet the prerequisites, or if any of the checks during the installation failed previously, run through the full preparation steps in this section and then and install Elastic Cloud Enterprise. You’ll do a bit more work now, but life will be simpler later on.

Regardless of which approach you take, the steps in this section need to be performed on every host that you want to use with Elastic Cloud Enterprise.

Install a Supported Linux Kernel

Elastic Cloud Enterprise requires 3.10 or higher. The steps shown here install kernel 4.4.

  1. Add the repository required to obtain the kernel:

    sudo yum install -y yum-plugin-fastestmirror http://elrepo.org/linux/kernel/el7/x86_64/RPMS/elrepo-release-7.0-2.el7.elrepo.noarch.rpm
    sudo yum makecache fast
  2. Install kernel version 3.10 or higher from the repository:

    sudo yum --enablerepo=elrepo-kernel install -y kernel-lt
  3. Update the /etc/default/grub file with the required options:

    sudo sed -i -e 's|^GRUB_CMDLINE_LINUX=\(.*\)|GRUB_CMDLINE_LINUX=\"crashkernel=auto rhgb quiet console=ttyS0 cgroup_enable=memory swapaccount=1\"|g' /etc/default/grub
  4. Enable the overlay kernel module:

    echo "overlay" | sudo tee -a /etc/modules-load.d/overlay.conf
  5. Refresh the dynamically generated grub2 configuration and configure grub to boot the newly installed kernel 3.10 or higher:

    sudo grub2-mkconfig -o /etc/grub2.cfg
    sudo grub2-set-default 0

Install Docker Version 1.11

  1. Recommended: Add the Docker repository to make installing the correct version of Docker easier:

    sudo tee /etc/yum.repos.d/docker.repo <<-'EOF'
    [dockerrepo]
    name=Docker Repository
    baseurl=https://yum.dockerproject.org/repo/main/centos/7/
    enabled=1
    gpgcheck=1
    gpgkey=https://yum.dockerproject.org/gpg
    EOF
    
    sudo yum makecache fast
  2. Install the correct version of the docker-engine package:

    sudo yum install docker-engine-1.11*

Update the Configurations Settings

  1. Stop the Docker service:

    sudo systemctl stop docker
  2. If you plan to run Elasticsearch 5.0 and later: Update the /etc/sysctl.conf file to set the maximum number of mapped memory areas a process can have:

    echo "vm.max_map_count=262144" | sudo tee -a /etc/sysctl.conf
  3. Verify that IPv4 forwarding is enabled:

    cat /proc/sys/net/ipv4/ip_forward

    If the output of this command is not 1, you must enable IPv4 forwarding so that Docker can work as expected:

    1. Update the /etc/sysctl.conf file to include this line:

      net.ipv4.ip_forward = 1
    2. Restart the network service:

      sudo service network restart
    3. Verify that the change is applied:

      cat /proc/sys/net/ipv4/ip_forward

      The output 1 indicates that IPv4 forwarding is turned on.

  4. Adjust the system limits.

    Add the following configuration values to the /etc/security/limits.conf file. These values are derived from our experience with the Elastic Cloud hosted offering and should be used for Elastic Cloud Enterprise as well.

    Tip

    If you are using a user name other than elastic, adjust the configuration values accordingly.

    *                soft    nofile         1024000
    *                hard    nofile         1024000
    *                soft    memlock        unlimited
    *                hard    memlock        unlimited
    elastic          soft    nofile         1024000
    elastic          hard    nofile         1024000
    elastic          soft    memlock        unlimited
    elastic          hard    memlock        unlimited
    root             soft    nofile         1024000
    root             hard    nofile         1024000
    root             soft    memlock        unlimited
  5. Create the /mnt/data/ directory as a mount point:

    sudo install -o $USER -g $USER -d -m 700 /mnt/data
  6. Create the /mnt/data/docker directory for the Docker service storage:

    sudo install -o $USER -g $USER -d -m 700 /mnt/data/docker
  7. Disable the firewalld service. The service is not compatible with Docker and interferes with the installation of ECE. You must disable firewalld before installing or reinstalling ECE.

    sudo systemctl disable firewalld
  8. Configure the Docker daemon options.

    Update the /etc/systemd/system/docker.service.d/docker.conf file. If the file path and file do not exist, create them first.

    [Unit]
    Description=Docker Service
    After=multi-user.target
    
    [Service]
    ExecStart=
    ExecStart=/usr/bin/docker daemon -g /mnt/data/docker --storage-driver=overlay --bip=172.17.42.1/16
  9. Apply the updated Docker daemon configuration:

    Reload the Docker daemon configuration:

    sudo systemctl daemon-reload

    Restart the Docker service:

    sudo systemctl restart docker

    Enable Docker to start on boot:

    sudo systemctl enable docker
  10. Enable your user to communicate with the Docker subsystem by adding it to the docker group:

    sudo usermod -aG docker $USER
  11. Recommended: Tune your network settings.

    Create a 70-cloudenterprise.conf file in the /etc/sysctl.d/ file path that includes these network settings:

    cat << SETTINGS | sudo tee /etc/sysctl.d/70-cloudenterprise.conf
    net.ipv4.tcp_max_syn_backlog=65536
    net.core.somaxconn=32768
    net.core.netdev_max_backlog=32768
    SETTINGS
  12. Pin the Docker version to ensure that the docker-engine package does not get upgraded:

    echo "exclude=docker-engine" | sudo tee -a /etc/yum.conf
  13. Reboot your system to ensure that all configuration changes take effect:

    sudo reboot
  14. After rebooting, verify that your Docker settings persist as expected:

    sudo docker info | grep Root

    If the command returns Docker Root Dir: /mnt/data/docker, then your changes were applied successfully and persist as expected.

    If the command returns Docker Root Dir: /var/lib/docker, then you need to troubleshoot the previous configuration steps until the Docker settings are applied successfully before continuing with the installation process. For more information, see Custom Docker daemon options in the Docker documentation.

  15. Repeat these steps on other hosts that you want to use with Elastic Cloud Enterprise or follow the steps in the next section to start installing Elastic Cloud Enterprise.