How to Develop Your Own Security Extensions and Custom Realms for Elasticsearch

Elasticsearch provides authentication and authorization capabilities for the Elastic Stack, and has long had the ability to extend these capabilities through X-Pack security extensions.

The most common use for these extensions has been to write custom security realms which support authentication mechanisms that are not included natively by X-Pack. Whether it's a third party authentication provider or something that has been built in-house, a custom realm allows your own authentication scheme to operate alongside the builtin X-Pack realms such as those for Active Directory, LDAP and PKI.

A less commonly used feature of extensions is their ability to define custom roles providers. A roles provider allows you to programmatically build X-Pack security roles without needing to define them through the built-in Roles API.

Changes in Elasticsearch 6.3

As part of version 6.3 of Elasticsearch, we've made a significant change to the way these security extensions are defined. If you have an existing extension that you're upgrading to 6.3, you will need to make a few simple changes to be compatible with this new approach. If you're building a new extension, then it's even easier in 6.3 than it has been in any previous release.

In earlier versions of Elasticsearch and X-Pack, an extension was its own special thing. It was a bit like an Elasticsearch plugin, but not quite --- it was packaged and installed differently, and had less capabilities. An extension was not able to implement the full set of Elasticsearch plugin interfaces, and because it ran under a child of X-Pack's classloader, there was the potential for class name clashes and version conflicts (nicknamed "JarHell").

Starting with 6.3, security extensions are now packaged as standard Elasticsearch plugins. The ability to implement custom realms and roles providers is provided through a new service provider interface named SecurityExtension. This interface replaces the old XPackExtension class. SecurityExtension has two primary methods: getRealms and getRolesProviders. The signatures for these methods mimic those from XPackExtension which should ease the conversion of existing code.

The two biggest benefits of this change are that it moves security extensions to their own classloader that is isolated from X-Pack internal dependencies, and it allows you to also implement the many plugin interfaces Elasticsearch supports. Anyone who has wrestled with classloader problems, or "JarHell" under the old model should appreciate this improvement.

A Sample Security Extension for an eCommerce Store

For the rest of this article I'm going to walk through a sample security extension that can be deployed under Elasticsearch 6.3 with X-Pack security enabled. Because security extensions are a platinum feature of X-Pack you'll need to deploy this plugin to a cluster that is running X-Pack with either a platinum or a trial license.

This extension is designed to support a hypothetical eCommerce platform that is powered by Elasticsearch. This platform uses a single Elasticsearch cluster to operate multiple independent stores, each with their own index. All authentication is performed by the application server that sits in front of Elasticsearch, and the security extension trusts (through cryptographic signatures) the authentication decisions of the application. This is implemented as a custom security realm.

In order to provide isolation between stores, each store will connect to Elasticsearch as a different logical user, with a role that restricts access to their store's index. This means that each store has their own set of roles, but the only difference between these roles is the index name. Because this eCommerce platform regularly adds new stores, we don't want to have to define a new set of role each time a store is opened --- this would be possible through the Roles API, but it would create a multitude of roles that are almost identical. Instead, the extension defines the required roles dynamically through a roles provider.

The full source code for this security extension is available for download, including all build files, the custom realm and the roles provider. This example is intentionally simple, but provides a complete example for you to learn from. The code is not intended to be production ready --- it does not have tests, and it only handles a limited set of error conditions.

The Foundations of a Security Extension

Let's look at some of the implementation details of the plugin we're building, starting with the WebStoreSecurityPlugin class which is a standard Elasticsearch ActionPlugin. Our Security Realm needs this plugin in order to specify non-standard HTTP headers on the Elasticsearch REST Interface. The application server sends the details about the authenticated user in two custom HTTP headers - one contains information ("claims") about the user (as a JSON object), and the other contains a cryptographic signature of those claims. We'll look at how those headers are consumed when we explore the custom realm implementation.

public Collection<String> getRestHeaders() {
  return Arrays.asList(WebStoreRealm.CLAIMS_HEADER, WebStoreRealm.SIGNATURE_HEADER);
}

We also need to define that our WebStoreSecurityPlugin is the entry point for our Elasticsearch plugin, which we can do within the plugin-descriptor.properties file like so:

classname=co.elastic.blog.elasticsearch.example.WebStoreSecurityPlugin

(We need to define some other properties in that file as well, but they should be fairly self explanatory if you look at the source.)

The other key part of our plugin is the WebStoreSecurityExtension class which implements the SecurityExtension interface. This is where we register our custom security realm:

public Map<String, Realm.Factory> getRealms(ResourceWatcherService resourceService) {
  return Collections.singletonMap(WebStoreRealm.TYPE, 
    config -> new WebStoreRealm(config, resourceService));
}

As well as a custom roles provider:

public List<BiConsumer<Set<String>, ActionListener<Set<RoleDescriptor>>>>
getRolesProviders(Settings settings, ResourceWatcherService resourceWatcherService) {
  final WebStoreRolesProvider rolesProvider = new WebStoreRolesProvider();
  return Collections.singletonList(rolesProvider::lookup);
}

And because our custom realm is configurable (which we'll see later on) we also need to define its settings:

public Map<String, Set<Setting<?>>> getRealmSettings() {
  return Collections.singletonMap(WebStoreRealm.TYPE, WebStoreRealm.getSettings());
}

While our WebStoreSecurityPlugin is loaded as a standard Elasticsearch plugin, the WebStoreSecurityExtension class is loaded as a service, using the Java Service Provider Interface (SPI) mechanism. This means we need to add it to the META-INF/services/org.elasticsearch.xpack.core.security.SecurityExtension file. X-Pack security will read this file after the plugins are loaded, and then each class that is listed will be instantiated as a SecurityExtension.

With our plugin and extension now loaded, we can explore the implementation of our custom WebStoreRealm. As was mentioned earlier, this realm uses cryptographic signatures to verify the authenticity of headers coming from the application server. The public key for that signature verification is read from a certificate, the path to which is one of the realm's configuration settings.

private static final Setting<String> CERTIFICATE_PATH_SETTING
   = Setting.simpleString("certificate", Setting.Property.NodeScope);
// ...
public static Set<Setting<?>> getSettings() {
  return Sets.newHashSet(CERTIFICATE_PATH_SETTING);
}

This allows us to configure our realms as follows:

xpack.security.authc.realms:
  # We recommend always having a file realm as the highest priority realm
  file:
    type: file
    order: 0

  # Our eCommerce platform uses the native realm for administrative access
  native:
    type: native
    order: 1

  # This is our custom realm 
  webstore:
    # This "type" corresponds to WebStoreRealm.TYPE
    type: webstore
    order: 2
    # This path is relative to the "config" directory
    certificate: webstore/webstore.crt

The constructor for WebStoreRealm takes care of reading the certificate setting and converting it to a java.nio.file.Path, and it even tracks changes to that file (using ResourceWatcherService) so that we can reload the certificate if it is changed on disk. If you want to see the full implementation details, you'll need to download the source code.

public WebStoreRealm(RealmConfig config, 
                     ResourceWatcherService resourceWatcherService)
    throws IOException, CertificateException {
  super(TYPE, config);
  final Path certificatePath = getCertificatePath(config);
  this.signatureVerifier = new SignatureVerifier(certificatePath);
  watchCertificateFileForChanges(certificatePath, resourceWatcherService);
}

Handling Authentication in a Custom Realm

That covers the registration, configuration and initialization of our realm, so now we need to look at how it actually authenticates users, and for that we start with the token method. This method is responsible for looking at an incoming request, determining whether it contains some form of credentials that our realm can process, and if so returning them as an AuthenticationToken. Our implementation looks for the two headers we registered in the plugin, and then extracts them to a custom token implementation:

public AuthenticationToken token(ThreadContext threadContext) {
  final String json = threadContext.getHeader(CLAIMS_HEADER);
  final String sig = threadContext.getHeader(SIGNATURE_HEADER);
  if (Strings.hasText(json) && Strings.hasText(sig)) {
    return new WebStoreToken(json, signatureBytes(sig));
  } else {
    return null;
  }
}

Note that this token method does not perform any authentication steps --- this occurs in the authenticate method where we verify the signature header (indirectly via WebStoreToken.getClaim) and then create a new user object:

public void authenticate(AuthenticationToken authenticationToken,
                         ActionListener<AuthenticationResult> actionListener) {
  try {
    final WebStoreClaim claim = ((WebStoreToken) authenticationToken)
      .getClaim(signatureVerifier);
    actionListener.onResponse(AuthenticationResult.success(buildUser(claim)));
  } catch (GeneralSecurityException e) {
    actionListener.onResponse(AuthenticationResult.unsuccessful(
      "Cannot verify claim", e));
  } catch (Exception e) {
    actionListener.onFailure(e);
  }
}

private User buildUser(WebStoreClaim claim) {
  final String[] roles = {
    "webstore-" + claim.getStoreId() + "-" + claim.getRole(),
    "webstore"
  };
  final Map<String, Object> metadata = new HashMap<>();
  metadata.put("webstore-store", claim.getStoreId());
  metadata.put("webstore-role", claim.getRole());
  return new User(claim.getPrincipal(), roles, null, null, metadata, true);
}

Authorization and Custom Roles Providers

The buildUser method shown above constructs a new User object with the appropriate role names and metadata. Realms only ever deal with role names --- these names will be resolved to RoleDescriptors by role providers (we'll cover that in a moment). In our case, we have one fixed role "webstore" which is applied to all users who are authenticated via our realm, and a dynamic role that is constructed based on the contents of the JSON we read from the header. The JSON that is passed in the x-web-store-claims header looks something like this:

{
  "principal": "store-45-user",
  "storeId": 45,
  "role": "reader"
}

If we received that content in the header (with a valid signature) then we would assign our user two roles webstore-45-reader and webstore. The latter role would be defined in the usual way using the Elasticsearch API, or the Kibana management UI. The former role, which is dynamically generated is defined as part of our custom roles provider WebStoreRolesProvider. This class parses the role name, and extracts the store-id (45) and the level of access (reader) to construct a RoleDescriptor that grants the appropriate access to the applicable index:

private RoleDescriptor buildRole(String name, long storeId, String access) {
  final RoleDescriptor.IndicesPrivileges.Builder indexBuilder 
      = RoleDescriptor.IndicesPrivileges.builder();
  indexBuilder.indices("webstore-" + storeId);
  if (access.equals("reader")) {
    indexBuilder.privileges("read");
  } else if (access.equals("writer")) {
    indexBuilder.privileges("write");
  } else {
    throw new IllegalArgumentException("Unsupported access type " + access);
  }
  return new RoleDescriptor(name, new String[]{"monitor"}, 
      new RoleDescriptor.IndicesPrivileges[]{indexBuilder.build()}, null);
}

For our webstore-45-reader role, the above code would construct a role definition that is equivalent to the following role in the API:

{
  "cluster": [ "monitor" ],
  "indices": [
    {
      "names": [ "webstore-45" ],
      "privileges": [ "read" ]
    }
  ]
}

Building and Testing our Security Extension

Now that we've walked through the main features of this plugin, we can build and test it. The sample code ships with a mavenpom.xml that will compile and assemble a plugin zip with the appropriate dependencies and metadata files. If we run mvn package in the sample project, it will build security-spi-example-1.0.0-elasticsearch-plugin.zip in the target/ directory. You can use your preferred build tool to assemble your custom security extension as long is it produces a zip file in the correct layout. Many Elasticsearch plugins are built using gradle.

Because Security Extensions are now packaged as standard Elasticsearch plugins, we install it using the elasticsearch-plugin command:

$ bin/elasticsearch-plugin install file://${WEBSTORE_REALM_HOME}/target/security-spi-example-1.0.0-elasticsearch-plugin.zip
-> Downloading security-spi-example-1.0.0-elasticsearch-plugin.zip
[=================================================] 100%
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@     WARNING: plugin requires additional permissions     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
* java.lang.reflect.ReflectPermission suppressAccessChecks
See http://docs.oracle.com/javase/8/docs/technotes/guides/security/permissions.html
for descriptions of what these permissions allow and the associated risks.

Continue with installation? [y/N]y
-> Installed security-spi-example

We can use the elasticsearch-certutil command to generate a key-pair for signing, and then extract the PEM files from zip archive and copy webstore/webstore.crt into the elasticsearch configuration directory.

bin/elasticsearch-certutil cert -pem -out certs.zip -name webstore
unzip certs.zip -d ${ES_HOME}/config webstore/webstore.crt

With the realm configured per the example shown early, we can start up Elasticsearch and early in the startup process the logs should indicate

[INFO ][o.e.p.PluginsService     ] [test-node] loaded plugin [security-spi-example]

Now we can test our security extension. For this we will use curl and openssl to verify that we get the right behaviours based on the content of our JSON header and the associated signature.

First we put some dummy data into a couple of indices related to our eCommerce stores:

$ curl 'localhost:9200/webstore-45/_doc/1' -d '{ "name":"store-45-item-1" }' \
  -u elastic -XPUT -H "Content-Type: application/json" 
$ curl 'localhost:9200/webstore-45/_doc/2' -d '{ "name":"store-45-item-2" }' \
  -u elastic -XPUT -H "Content-Type: application/json" 
$ curl 'localhost:9200/webstore-50/_doc/1' -d '{ "name":"store-50-item-1" }' \
  -u elastic -XPUT -H "Content-Type: application/json" 
$ curl 'localhost:9200/webstore-50/_doc/2' -d '{ "name":"store-50-item-2" }' \
  -u elastic -XPUT -H "Content-Type: application/json" 

Then we create some JSON to represent a user from store number 45 with reader access, and sign it with webstore.key, the private key that we generated with elasticsearch-certutil. The openssl dgst command can generate signatures for us that match the SHA256withRSA algorithm our SignatureVerifier class expects:

$ ClaimJson='{"principal":"store-45-user","storeId":45,"role":"reader"}'
$ unzip certs.zip webstore/webstore.key
$ ClaimSig="$( printf '%s' "$ClaimJson" | \
    openssl dgst -sha256 -sign webstore/webstore.key | base64)"

Then using curl, we can authenticate to Elasticsearch using those two values as headers and see what we get:

curl "localhost:9200/_xpack/security/_authenticate" \
    -H "x-web-store-claims: $ClaimJson" -H "x-web-store-sig: $ClaimSig"
{
 "username":"store-45-user",
 "roles":["webstore-45-reader","webstore"],
 "full_name":null,
 "email":null,
 "metadata":{"webstore-store":45,"webstore-role":"reader"},
 "enabled":true
}

Success!

Elasticsearch has correctly authenticated our user based on the contents of those headers, and we have the two roles that we expect.

Now let's trying running a simple query:

$ curl "localhost:9200/_search" -H "x-web-store-claims: $ClaimJson" \
    -H "x-web-store-sig: $ClaimSig"
{
  "took":81,"timed_out":false,"_shards":{"total":5,"successful":5,"skipped":0,"failed":0},
  "hits":{"total":2,"max_score":1.0,
    "hits":[
      {"_index":"webstore-45","_type":"_doc","_id":"2","_score":1.0,"_source":{ "name":"store-45-item-2" }},
      {"_index":"webstore-45","_type":"_doc","_id":"1","_score":1.0,"_source":{ "name":"store-45-item-1" }}
    ]
  }
}

Even though our _search query didn't specify any indices, the result only included the items from store number 45. We're not permitted to read the item from store 50. Our custom realm and roles provider are working together exactly as we want them to.

What if we try and get sneaky, and change the claims in our JSON header, but don't update the signature? We'd expect that to fail because our realm verifies that the signature matches the claims:

$ ClaimJson='{"principal":"store-45-user","storeId":50,"role":"reader"}'

$ curl "localhost:9200/_search" -H "x-web-store-claims: $ClaimJson" \
    -H "x-web-store-sig: $ClaimSig"
{
 "error":{
  "root_cause":[{
   "type":"security_exception",
   "reason":"unable to authenticate user [store-45-user] for REST request [/_search]",
   "header":{"WWW-Authenticate":"Basic realm=\"security\" charset=\"UTF-8\""
 }}],
 "type":"security_exception",
 "reason":"unable to authenticate user [store-45-user] for REST request [/_search]","header":{"WWW-Authenticate":"Basic realm=\"security\" charset=\"UTF-8\""}},
 "status":401
}

Our expectation is confirmed. The signature verification failed and it prevented us from authenticating (check the elasticsearch logs to see the failure reason). There's more testing we could do, but those short tests show that our Security Extension is plugged in to Elasticsearch and is doing the job it was designed to do.

So, if you haven't already done so, download the sample code and use it as the basis for creating your own security extensions for Elasticsearch. And if you have any questions, head to our forums and ask away!