Securing your Deployment

Elastic Cloud supports most of the security features that are part of the Elastic Stack. These features are designed to:

  • Prevent unauthorized access with password protection, and role-based access control.
  • Preserve the integrity of your data with message authentication and SSL/TLS encryption.

Elastic Cloud handles the installation of the security features for you, both for new deployments you create and for deployments that you upgrade. Which exact set of security features you use depends on the version of your Elasticsearch cluster.

In Elasticsearch version 5.0 and later, the security features to keep your Elasticsearch clusters safe are now part of X-Pack. If you create a cluster on Elasticsearch 5.0 or later, the X-Pack security features are always enabled, and there is no additional enablement step. With the move to X-Pack, the biggest changes to security features for the Elastic Stack include the names of the security configuration options, TLS/SSL configuration, and how roles are defined. A few privileges have also been removed. You work with users and roles in the Kibana Management app, accessible from the Security page in the Elastic Cloud Console. Two users are always created with new version 5.x clusters in Elastic Cloud, the elastic superuser and the anonymous user. If you upgrade a cluster to version 5.x, the users defined in your Shield configuration are also preserved.

For Elasticsearch versions before 5.0, the Shield plugin provides similar security features for your cluster, such as user authentication and role based access control. Shield is always installed and enabled for all newly created clusters. If your cluster did not originally enable Shield, save your Shield configuration to enable the security features. If Shield is not enabled, anyone who knows the ID of your cluster can connect to it. You work with users and roles in the Shield Editor directly in the Elastic Cloud Console. Three users are always created for clusters in Elastic Cloud: The admin user, the readwrite user, and the readonly user.

Note that when you upgrade a cluster to Elasticsearch 5.0 or later from an earlier version of Elasticsearch, your Shield configuration is migrated to X-Pack. If you used the Shield Editor before upgrading to version 5.0, you will need to switch to the Kibana Management app after upgrading.

Tip

For Elasticsearch 5.0 and later, you work with users and roles in the Kibana Management app. If you’re using a version of Elasticsearch before 5.0, you use the Security editor to work with users and roles.

Reset the elastic User Password

For versions 5.0 and later: You might need to reset the password for the elastic superuser if you cannot authenticate with the elastic user ID and are effectively locked out from an Elasticsearch cluster or Kibana. For example, if your deployment never had an active Shield configuration and you upgraded to Elasticsearch 5.0 or if you were not provided the password during the upgrade process, you will likely need to reset the password.

To reset the password:

  1. Log into the Elastic Cloud Console.
  2. On the Deployments page, select your deployment.

    Narrow the list of the deployments by name, ID, or choose from several other filters. Use a combination of them to further define the list.

  3. From your deployment menu, go to Elasticsearch and then Security.
  4. Click Reset password.
  5. Copy down the auto-generated a password for the elastic user:

    The password for the elastic user after resetting

  6. Close the window.

The password is not accessible after you close the window, so if you lose it, you need to reset the password again.

Version 5.0 and Later, Manage Users and Roles in X-Pack

You work with users and roles in the Kibana Management app, accessible from the Security page in the Elastic Cloud Console or directly from the Kibana side navigation.

The Users tab shows the users that have been configured on your deployment. You can add or delete users here and assign roles that give users specific privileges. A user can be assigned multiple roles.

The admin user in X-Pack

If you upgraded your deployment to Elasticsearch 5.0 or later from a version before 5.0, the users defined in Shield were also migrated to X-Pack and show up in this list. This includes the default admin, readwrite, and readonly users, which you can work with like any other user in the Kibana Management app.

The Roles tab shows the roles that exist on your deployment. Roles let you customize exactly which actions a user with the role can do, both on a deployment and an index level. Several users can have the same role. You can also add or delete users.

The admin role in X-Pack

Two users are always created with new version 5.x deployments:

elastic
A superuser that is somewhat similar in scope to the default admin user in Shield configurations in Elastic Cloud before version 5.0. The elastic user is not shown in the Kibana Management app and you cannot modify this user, other than to reset the password.
anonymous
A user for handling incoming requests where no authentication token can be extracted. By default, this user has no privileges. You need to assign one or more roles to this user to enable anonymous access. To learn more, see Enabling Anonymous Access.

To learn more about users and roles in X-Pack, see Setting Up User Authentication.

Example: Create a Filebeat Role and User

To send Beats data to a secured cluster in Elastic Cloud, you should use a user that has the right privileges. When we showed you a Beats example that configured Filebeat on a server to send data to Elastic Cloud, the user filebeat_internal was used, with role privileges based on the recommendations in Beats and Security.

To create a similar user and role in the Management app in Kibana:

  1. Log into the Elastic Cloud Console and select to your deployment.
  2. On the Deployments page, select your deployment.

    Narrow the list of the deployments by name, ID, or choose from several other filters. Use a combination of them to further define the list.

  3. From your deployment menu, go to Kibana and open the Kibana endpoint.
  4. From Kibana, go to the Management app.
  5. Select the Roles tab, click New Role and give the role the filebeat-writer name.
  6. Specify the right privileges. Under Cluster Privileges, select monitor and manage_index_templates. Under Indices Privileges, specify read, write, and create_index privileges for the filebeat-* indices that the Beat creates.

    The `filebeat-writer` role
  7. Click Save.
  8. Select the Users tab, click New User and give the user the filebeat_internal name.
  9. Assign the filebeat-writer role to the user under Roles.

    The `filebeat-internal` user
  10. Enter a password and confirm the password. Make a note of the password and keep it in a secure location.
  11. Click Save.

After the user and role are saved, you can use the new user to authenticate when sending data from Filebeat to your cluster.

Before Version 5.0, Manage Users and Roles in Shield

Shield is always installed, but you must save your Shield configuration to enable the security features. If you don’t enable Shield, anyone who knows the ID of your deployment can connect to it.

Enable Shield

You should enable Shield on all Elasticsearch clusters that you create. If your cluster does not have Shield enabled, anyone who knows the ID of your cluster can access it. By default, when you create a new deployment, the Shield plugin is installed on your clusters, but it is not enabled.

To enable Shield:

  1. Log into the Elastic Cloud Console.
  2. On the Deployments page, select your deployment.

    Narrow the list of the deployments by name, ID, or choose from several other filters. Use a combination of them to further define the list.

  3. From your deployment menu, go to Elasticsearch and then Security.
  4. In the Security editor, review the users and roles that will be created and copy down the passwords. Keep the passwords in a secure location.

    By default, three users are always created: The admin user, the readwrite user, and the readonly user. If you are new to Shield, you can accept these default users for now and manage your own users later.

  5. Click Save to hash the passwords, save the configuration, and enable Shield.

    When you save the configuration, the passwords are hashed and you will not be able to recover them afterwards. If you ever need to reset a password, replace the corresponding hash with your new password and then save the configuration again.

After Shield is enabled, users and applications can connect to your deployment only if they can authenticate successfully.

Tip

If you are not sure how enabling Shield will affect your deployment, you can also clone an unsecured cluster, enable Shield on the new deployment, and test there before updating or completely replacing your existing deployment.

Edit Users and Roles

You can configure users, assign their roles, and define the privileges of those roles in the Security editor. It is accessible from the Elasticsearch cluster. The Security editor works based on YAML files, so you can add comments where necessary.

Note

A valid role or user name must be at least 1 character and no longer than 30 characters. It must begin with a letter (a-z) or an underscore (_). Subsequent characters can be letters, underscores (_), digits (0-9) or any of the following symbols @, -, . or $

Users

The Users editor defines the users in your deployment. A user has a username and a password hash. You define a new user by adding a new line with the username mapping to a clear text password. When you save, the password is hashed in your browser before being saved to our servers.

You can click on New User to have a new user added to the file with a randomly generated password. Copy this password before saving, as there is no way to access the password after it has been hashed. If you lose the password, just change it to a new one.

If you run Shield in a deployment running on your own servers, you can simply copy the config/shield/users file and paste it in this editor.

This is an example of a valid users file:

# Lines starting with # are comments.
# This file expects usernames to map passwords or hashes, like the following:
admin: $2a$12$uRopB3Jz3slQQVMUTlNuPeHRhbfbFWDBrXhRk8vzK7xr0mavw2vAq
readwrite: $2a$12$Y/tCEs7RSzWrlWsjyyuB2.ARhuMis5PXj47IXvrDED/jJ8Cx1dKza
readonly: $2a$12$DQ8jiYAHiCvHWkmNM2fOgOsXgwZxsl/4PxY6zr5g5BOYRcG8dV9IW

# This password has not been hashed yet, but will be when you save.
my_user: bc3rqc0q1xwr4irq3o

To delete a user, simply remove it from this file.

Users per Role

A user can have more than one role, and a role can be assigned to multiple users.

The Users per Role editor lets you customize these. The format is one line per role, with the role name followed by a comma-delimited list of users.

# The users per role file maps roles and users like this:
# role_name: user1, user2

readonly: user1, user2
my_custom_role: user2, user3

# In this case, user2 will have the roles readonly and my_custom_role
Roles

The Roles editor lets you customize exactly which actions users with the role can do, both on a deployment and an index level.

Full documentation on configuring role-based access control is available in the Shield documentation.

You can use the default roles or adapt them. Here are two examples of roles with comments that indicate what privileges these roles grant:

# Admins can do everything
admin:
  # If you use a generic group of actions, such as "all", "read", "monitor", etc.
  # you can simply specify the group without listing every action.
  cluster: all
  indices:
    # The same is true for indices. You can also list aliases here.
    # Note that the index name '*' is quoted. This is because this is actually
    # a YAML file, and the * character can have a special meaning. It is a good
    # practice to quote all the index names.
    '*': all

# This is an example of a role with more granular access. We explicitly list
# every action possible, both on the cluster and the indicies.
# A user with this role will be able to use Kibana 4 and read the indices
# logs-* and even-more-logs-*. Dashboards/Visualizations can be modified
# but the indices being analyzed can only be read.
kibana4:
  cluster:
      - cluster:monitor/nodes/info
      - cluster:monitor/health
      # Note that we use a list here. Every action is indented and
      # starts with `- `.
  indices:
    #
    'logs-*,even-more-logs-*':
      - indices:admin/mappings/fields/get
      - indices:admin/validate/query
      - indices:data/read/search
      - indices:data/read/msearch
      - indices:admin/get
    '.kibana':
      - indices:admin/exists
      - indices:admin/mapping/put
      - indices:admin/mappings/fields/get
      - indices:admin/refresh
      - indices:admin/validate/query
      - indices:data/read/get
      - indices:data/read/mget
      - indices:data/read/search
      - indices:data/write/delete
      - indices:data/write/index
      - indices:data/write/update
      - indices:admin/create

Configure the Java Transport Client

You can connect to a deployment secured with the X-Pack security features or Shield by using the transport client. In addition to the standard transport client configuration, you’ll need to ensure two key points for Elastic Cloud deployments:

  • The X-Found-Cluster header must be included, indicating which cluster to route the connections to
  • The connecting user must have the transport_client role as well as the monitoring cluster privilege

Here is an example of how to create a transport client to connect to Elastic Cloud:

// Build the settings for our client.
String clusterId = "ac01aa2425e4a5bafdebf5100af3e9b37401055b"; // Your cluster ID here
String region = "us-east-1"; // Your region here
boolean enableSsl = true;

Settings settings = Settings.settingsBuilder()
    .put("transport.ping_schedule", "5s")
    //.put("transport.sniff", false) // Disabled by default and *must* be disabled.
    .put("cluster.name", clusterId)
    .put("action.bulk.compress", false)
    .put("shield.transport.ssl", enableSsl)
    .put("request.headers.X-Found-Cluster", clusterId)
    .put("shield.user", "username:password") // your shield username and password
    .build();

String hostname = clusterId + "." + region + ".aws.found.io";
// Instantiate a TransportClient and add the cluster to the list of addresses to connect to.
// Only port 9343 (SSL-encrypted) is currently supported.
Client client = TransportClient.builder()
        .addPlugin(ShieldPlugin.class)
        .settings(settings)
        .build()
        .addTransportAddress(new InetSocketTransportAddress(InetAddress.getByName(hostname), 9343));
Note

The example is for an ES 2.1 cluster. See GitHub for examples that work with other versions.

Some additional configuration steps might be required, such as adding the Shield JAR to your classpath. To learn more, see Configuring the Transport Client to work with a Secured Cluster (X-Pack) and Configuring the Transport Client to work with Shield.

Tip

Transport client support for Elasticsearch 6.x was added in version 6.2.4 and later ported back to make it available to newly created Elasticsearch 6.1 and 6.0 clusters. If you have an existing version 6.0 or 6.1 cluster without transport client support, you can use the transport client by upgrading your cluster to version 6.2.4 first.

Sanitize Requests

You must take care when sanitizing requests:

  • If dynamic mapping is on, arbitrary keys must not be allowed.
  • Index names can contain patterns.
  • Expensive searches can cause OutOfMemory errors.

Restrictions

  • IP filtering with X-Pack and Shield is currently not available.
  • Only the esusers realm is available. We currently do not support realms such as LDAP, Active Directory, SAML or Shield custom realms.
  • Audit Logging is disabled.
  • Client certificates, such as PKI certificates, are not supported.