Migrating from Shieldedit

With the move to X-Pack, the biggest changes to security functionality are the names of the security configuration options, TLS/SSL configuration, and how roles are defined. A few privileges have also been removed. This section describes the changes you need to make when upgrading from Shield.

Also, all endpoints have been renamed from /_shield/XYZ to /_xpack/security/XYZ.

Migrating Settingsedit

Most Shield settings have simply had their prefix changes from shield. to xpack.security.. Hare a few examples of Shield 2.x settings and their new X-Pack counterparts:

Shield 2.x Setting Name X-Pack 5.0 Setting Name

shield.authc.realms.*

xpack.security.authc.realms.*

shield.audit.outputs

xpack.security.audit.outputs

Persistent cluster state settings used for IP Filtering are not automatically migrated to the new names. You will need to manually migrate these settings.

TLS/SSL Settingsedit

X-Pack introduces a simpler way of configuring TLS/SSL within your cluster and as part of this effort, some of the settings have changed or been removed. Also, new options have been introduced, which includes the ability to more easily separate default TLS/SSL configurations from the transport configuration.

There is a hierarchy to TLS/SSL settings in X-Pack, which allows for a simple configuration for everything or customizations to be applied to individual components. The X-Pack wide configuration are the settings that being with xpack.ssl. X-Pack security adds configuration for HTTP under the settings that begin with xpack.security.http.ssl. and Transport under the settings that begin with xpack.security.transport.ssl.. Transport profiles can still have their own configurations with settings that begin with transport.profiles.$PROFILE.xpack.security.ssl. and they will fallback to the default transport configuration. Realms can also have their own configurations, which are available as settings that begin with xpack.security.authc.realms.$REALM_NAME.ssl..

There is also an important change in terms of how hostname verification is done by X-Pack security. Previously, a reverse DNS lookup would be made to determine the name of the address being connected to. This reverse DNS lookup has been removed and the name/address provided when connecting is used. Please make sure the the publish host on your nodes is one of the values in the certificate.

The following settings have been replaced with new settings:

shield.ssl

Use xpack.security.transport.ssl.enabled as a replacement to control whether ssl is enabled or disabled for node to node and transport client traffic.

shield.http.ssl

Use xpack.security.http.ssl.enabled as a replacement to control whether ssl is enabled or disabled for http traffic.

shield.ssl.ciphers

Use xpack.ssl.cipher_suites as a replacement to control the default enabled cipher suites.

shield.ssl.hostname_verification

This boolean setting has been replaced by xpack.ssl.verification_mode that supports more options as described in the ssl settings reference section.

shield.authc.realms.$REALM_NAME.hostname_verification

This boolean setting has been replaced by xpack.authc.realms.$REALM_NAME.ssl.verification_mode that supports more options as described in the ssl settings reference section.

The following settings have been removed:

shield.ssl.hostname_verification.resolve_name

This setting controlled the name resolution for SSL traffic, but was not easy to use and hard to set up. If certificates are not set with the proper DNS names, you should configure xpack.security.verification_mode as either certificate or none.

shield.ssl.session.cache_size

The value for the SSL session cache size should be configured through the use of the system property javax.net.ssl.sessionCacheSize.

shield.ssl.session.cache_timeout

The SSL session cache timeout being set incorrectly can negatively impact performance and was not widely used, so it has been removed and the default value of the JDK is used.

Migrating File-based Users and Roles to the Native Realmedit

From 5.0 onward, you should use the native realm to manage roles and local users. To migrate existing file-based users and roles to the native realm, use the migrate tool that’s included with the X-Pack plugin.

When migrating from Shield 2.x, the migrate tool should be run prior to upgrading to ensure all roles can be migrated as some may be in a deprecated format that X-Pack cannot read. The migrate tool is available in Shield 2.4.0 and higher.

The migrate tool loads the existing file-based users and roles and calls the user and roles APIs to add them to the native realm. You can migrate all users and roles, or specify the ones you want to migrate. Users and roles that already exist in the native realm are not replaced or overridden. If the names you specify with the --users and --roles options don’t exist in the file realm, they are skipped.

Run the migrate tool after you install the X-Pack plugin. For example:

$ bin/x-pack/migrate native -U http://localhost:9200 -u elastic -p changeme
-n lee,foo -r role1,role2,role3,role4,foo
starting migration of users and roles...
importing users from [/home/es/config/shield/users]...
found existing users: [test_user, joe3, joe2]
migrating user [lee]
{"user":{"created":true}}
no user [foo] found, skipping
importing roles from [/home/es/config/shield/roles.yml]...
found existing roles: [marvel_user, role_query_fields, admin_role, role3, admin,
remote_marvel_agent, power_user, role_new_format_name_array, role_run_as,
logstash, role_fields, role_run_as1, role_new_format, kibana4_server, user,
transport_client, role1.ab, role_query]
migrating role [role1]
{"role":{"created":true}}
migrating role [role2]
{"role":{"created":true}}
role [role3] already exists, skipping
no role [foo] found, skipping
users and roles imported.

The native subcommand supports the following options:

-U, --url
Endpoint URL of the Elasticsearch cluster to which you want to migrate the file-based users and roles. Required.
-u, --username
Username to use for authentication.
-p, --password
Password to use for authentication.
-n, --users
Comma-separated list of the users you want to migrate. If not specified, all users are migrated.
-r, --roles
Comma-separated list of the roles you want to migrate. If not specified, all roles are migrated.

Additionally, the -E flag can be used to specify additional settings. For example to specify a different configuration directory, the command would look like:

$ bin/x-pack/migrate native -U http://localhost:9200 -u elastic -p changeme -Epath.conf=/etc/elasticsearch

File Configuration Changesedit

In Shield, the location of the users, users_roles, roles.yml, and system_key files could be changed using settings. In X-Pack, these files must always reside in the CONFIG_DIR/x-pack directory.

esusers Realm Changesedit

In Shield, esusers was the type for the file based realm. This realm type has been renamed to file in X-Pack.

Additionally, the names of users in the file based realm are not allowed to conflict with the built-in users.

Role Format Changesedit

In X-Pack, the format for specifying roles in the roles.yml file is much stricter:

  • Role names are not allowed to conflict with the built-in roles.
  • Index names and expressions, such as .kibana or logstash-*, may no longer be used as the key for a set of indices permissions.
  • Support for comma-separated values and single values has been removed in favor of collections or arrays.
  • Syntax for field level security has changed. Instead of providing a list of allowed fields, the allowed fields are now inside an object field_security. The new syntax allows to also define exceptions. See field-level-security for more details.

The easiest way to migrate your roles, will be by converting the roles to native roles using the migrate tool prior to upgrading to X-Pack.

The following role shows how to specify multiple cluster and index permissions using collections.

Using collection notation to define roles.

get_user: 
  cluster: 
    - monitor
    - manage_security
  indices:
    - names: 
        - events_index
        - results
      privileges:
        - read
        - monitor
    - names: 
        - user_index
      privileges:
        - monitor
        - read
      query:
        term:
          department_id: 12
      field_security:
        grant:
          - name
          - email
          - contact.*
        except:
          - contact.mobile

The name of the role is get_user

The cluster privileges are defined as a collection

The first set of indices permissions contained in this role. This permission applies to the events_index and results indices and grants the read and monitor privileges.

The second set of indices permissions contained in this role. This permission applies to the user_index index, grants monitor and read privileges, limits the visible documents to those matching the query and limits the visible fields inside of the documents to name and email and all contact fields except for contact.mobile.

You can also use arrays to specify multiple cluster and index permissions. The following role uses the array notation to define the same role as in the previous example.

Using array notation to define roles.

get_user:
  cluster: [ monitor, manage_security ]
  indices:
    - names: [ events_index, results ]
      privileges: [ read, monitor ]
    - names: user_index
      privileges: [ monitor, read ]
      query:
        term:
          department_id: 12
      field_security:
        grant: [ name, email, contact.* ]
        except: [ contact.mobile ]

Values for cluster, privileges, and the entries in fields_security must be specified as a collection or array even if there’s only one value. You can specify a single value for the names field.

If you have roles that use an index name or expression as a key, you’ll need to update them. For example, the following role definitions are no longer valid.

# Unsupported role format: index name as key and privilege as value
get_user:
  indices:
    'events_index': read
# Unsupported role format: index name as the key with privileges as a value within the object
get_user:
  indices:
    'events_index':
      privileges: read

You can rewrite this role to work with X-Pack using collection notation:

# Correct role format for X-Pack
get_user:
  indices:
    - names: 
        - events_index
      privileges:
        - read

This line is the start of a privilege definition for a set of indices. Note that this line is indented further underneath indices

Similarly, if you have roles that use comma-separated values to specify multiple values, you need to rewrite them to use the collection or array notation. For example, the following role definition is no longer valid.

# Unsupported role format: comma-separated values
get_user:
  cluster: monitor, manage
  indices:
    'events_index,results':
       - privileges: read, write, create_index
       - field_security:
         - grant: fieldA, fieldB

You can rewrite this role using either collections or arrays.

Specifying multiple values with a collection.

get_user:
  cluster:
    - monitor
    - manage
  indices:
    - names :
        - 'events_index'
        - 'results'
      privileges:
        - read
        - write
        - create_index
      field_security:
        grant:
          - fieldA
          - fieldB

Specifying multiple values with an array.

get_user:
  cluster: [ 'monitor', 'manage' ]
  indices:
    - names: [ 'events_index', 'results' ]
      privileges: [ 'read', 'write', 'create_index' ]
      field_security:
        grant: [ 'fieldA', 'fieldB' ]

If you have roles that specify single values for cluster, privileges, and fields, you need to rewrite them to use the collection or array notation. For example, the following role definition is no longer valid.

# Unsupported role format: single values
get_user:
  cluster: manage
  indices:
    'events_index':
       - privileges: read
         field_security:
           grant: fieldA

You can rewrite this role using collections or arrays (or a combination of the two). Note that the names field still accepts a single value.

Specifying multiple values with collections and arrays.

get_user:
  cluster:
    - manage
  indices:
    - names: 'events_index'
      privileges:
        - read
      field_security:
        grant: [ 'fieldA' ]

Removed Privilegesedit

A few of the privileges that were available in Shield 2.x are no longer available in X-Pack. The following tables list the privileges that have been removed and their suggested replacements.

Table 1. Cluster Privileges

Removed Privilege Replacement Privilege

manage_shield

manage_security

manage_watcher

manage

monitor_watcher

monitor

Table 2. Indices Privileges

Removed Privilege Replacement Privilege

get

read

search

read

suggest

read

crud

read,write

data_access

read,write

manage_aliases

manage

Field level security and the percolatoredit

The percolator has been refactored and now requires a percolator field in the mapping. As a result, when using field level security with the percolator, access to the percolator field needs to be granted.

Auditing Changesedit

A new event type, realm_authentication_failed, has been added. These events were previously audited as authentication_failed events but in some cases produced confusing output. The realm_authentication_failed event indicates that the credentials provided by the user failed authentication for a specific realm; it does not indicate a failed login attempt as there could be other realms to try to authenticate against.

The logfile output no longer relies on the logger level to determine what events to output. The xpack.security.audit.logfile.events.include setting can be used to specify the events that should be output; you may also use the xpack.security.audit.logfile.events.exclude setting to simply exclude values from the defaults.

Some audit events would output the content of a REST request if the log level was configured at the TRACE level; control over this output has been replaced with the settings xpack.security.audit.logfile.events.emit_request_body and xpack.security.audit.index.events.emit_request_body for the logfile and index outputs. The default is false.

Finally, as part of the move to Log4j2 the logging.yml file has been replaced with the log4j2.properties file.

Run As Changesedit

In X-Pack security, the default header name for submitting requests on behalf of other users has changed to es-security-runas-user. Applications making use of the run as functionality will need to use the new header name.