Product release

Shield 2.3.0 Released

By
Uri Boness

It is with great pleasure that we announce the 2.3 release of Shield - the security extension for the Elastic Stack.

This release, like previous releases, introduces bug fixes and enhancements to existing features (see release notes for a complete change list). In addition, in this release we also introduce two important new features: A new native API-driven realm and the role management API.

Prologue

When Shield was first released (back in January 2015), we focused on getting the basic security model right and wanted to deliver the minimum required capabilities for you to safely secure your cluster. With this in mind, we introduced the esusers realm for simple deployments, and a set of Enterprise grade realms (e.g. LDAP and Active Directory) for the more complex requirements out there.

The esusers realm was a simple authentication service that required defining the users in configuration files (very similar to unix’s users/groups or Apache’s htpasswd). While it certainly did the job when it comes to protecting the Elasticsearch cluster, when working with this realm, you never truly felt at home - it wasn’t aligned with the “Elasticsearch way”. From day one, Elasticsearch was built around its APIs - you create an index, add data, run search, monitor the cluster, all of these were exposed via a set of well defined and dedicated APIs. And here came Shield, which required you to manipulate a set of files - clearly an outlier.

esusers served all of us well, but the intention was never to stop there. From the early days of Shield we were open about it and promised our users a better way for managing users. Today we’re thrilled to deliver on that promise.

Good Morrow, Sir native

The native realm is a built-in realm in Shield that uses Elasticsearch itself to store all of its users. It’s dynamic by nature and exposes a set of APIs for user management. Lets see how you can use it.

First thing first, assuming you have Elasticsearch 2.3 installed, you’ll need to install the License and Shield plugins:

bin/plugin install license
bin/plugin install shield

Next, we’ll add an admin (super) user using the esusers tool:

bin/shield/esusers useradd elastic -p changeme -r admin

NOTE: Please promise us you’re not going to use changeme as a password in production? Pretty please?

I know what you’re thinking - “What? esusers? But…”. Yes! Remember, with Shield, all APIs in Elasticsearch are protected. In order to manage users via APIs, you will need to be and authenticate as an admin user. We use the esusers tool to enable that. Besides… having an admin (super) user defined in esusers is not such a bad idea, but we’ll get to that later.

OK, you’re all set! If you haven’t done it yet, start up Elasticsearch:

bin/elasticsearch

And add your first user:

curl -XPOST -u elastic -p changeme 'localhost:9200/_shield/user/romeo' -d '{
  "password" : "minejuliet ",
  "roles" : [ "power_user" ],
  "full_name" : "Romeo",
  "email" : "romeo@montague.it",
  "metadata" : { <5>
    "moto" : "thus with a kiss I die"
  }
}'

That is it. “romeo” is now a user in Elasticsearch. No need to run command-line tools, edit files or copy files from one node to another. In addition to that, as the example above shows, with the introduction of native we also extended the notion of a user to include its full name, email and arbitrary metadata.

Having a power_user role, grants romeo the permission to create an index and add data. Running the following command will execute on behalf of the newly added romeo user and will index a new document into the auto-created the shakespeare index. 

curl -XPUT -u romeo -p minejuliet 'localhost:9200/shakespeare/tragedy/1' -d '{
  "title" : "The Tragedy of Macbeth",
  "quote" : "All fair is foul, and foul is fair; Hover through the fog and filthy air"
}

The native realm exposes a full set of CRUD APIs to manage its users.

You can get the users:

curl -XGET -u elastic -p changeme 'localhost:9200/_shield/user/romeo'

Update users:

curl -XPUT -u elastic -p changeme 'localhost:9200/_shield/user/romeo' -d '{
  "password" : "minejuliet ",
  "roles" : [ "user" ],
  "full_name" : "Romeo",
  "email" : "romeo@montague.it",
  "metadata" : {
    "moto" : "Love is a smoke raised with the fume of sighs"
  }
}'

And delete users:

curl -XDELETE -u elastic -p changeme localhost:9200/_shield/user/romeo

As expected, once deleted, any attempt to executes requests on behalf of romeo will be rejected. Try it out:

curl -XPUT -u romeo -p minejuliet localhost:9200/shakespeare/tragedy/2 -d '{
  "title" : "The Tragedy of Hamlet, Prince of Denmark",
  "quote" : "A little more than kin, and less than kind"
}'

O esusers, esusers! Wherefore Art Thou esusers?

If you’re already using Shield, one of the first things you’ll notice in this release is that the esusers realm disappeared from our documentation. Truth is, it didn’t. It was simply renamed to file realm. We believe that the name file better describes what this realm is all about - configuring users in local files.

Yet, now with the native realm in place, do we really need to keep the file/esusers realm at all?

The answer to this question is a definite “YES!”.

There is no doubt that the native realm displaced the file realm from its throne, and indeed we expect it to become the de-facto realm to use when no other authentication services are required (e.g. LDAP, Active Directory, PKI, etc…). But the file realm is still very important and cannot be disposed.

Imagine the following scenario: Your only user was configured via the native realm and one day you forget its password. Without the file realm, you’ll be stuck in a catch 22 - You need to reset your password using the API, but you’ll need your forgotten password to do so.

That’s where the file realm proves to be essential. It enables you to login to the machines, use the bin/shield/esusers command-line tool to add an admin user and from then on, use this admin user to update/change the fogotten password.

The file realm is your only way out of situations where all users locked themselves out of the system.

NOTE: While the name of the realm changed from esusers to file, in 2.3 we kept the bin/shield/esusers tool so it won’t break any configuration scripts (e.g. puppet, ansible, chef, etc…) you may have been using to manage users. We do however plan to rename this tool in the next major release. Moreover, while its name is officially changed to file, esusers is still supported as a deprecated name, such that your current configuration won’t break when upgrading.

Roles - thou art not alone

Having a native API-driven realm is a huge step forward when it comes to managing users in your cluster. But it only brings it half way. To have a complete control over user management, you need to be able to configure both authentication and authorization.

Authorization in shield is managed via roles. A role defines a set of permissions on the cluster and its data. The permissions a user has are therefore defined by the roles it’s associated with.

Until today, roles were defined in the roles.yml configuration file. Just like the file realm, in order to change/add/remove roles you needed to edit this file, on all the nodes.

Shield 2.3 introduces a new Role Management API and just like with the native realm, we now store roles in Elasticsearch itself.

You can now add roles:

curl -XPOST -u elastic -p changeme 'localhost:9200/_shield/role/shakespeare_admin' -d '{
  "indices": [
    {
      "names": [ "shakespeare" ],
      "privileges": [ "all" ],
    }
  ]
}'

Get roles:

curl -XGET -u elastic -p changeme 'localhost:9200/_shield/role/shakespeare_admin'

Update roles:

curl -XPOST -u elastic -p changeme 'localhost:9200/_shield/role/shakespeare_admin' -d '{
  "indices": [
    {
      "names": [ "shakespeare" ],
      "privileges": [ "all" ],
    }
  ],
  "run_as": [ "romeo" ]
}'

And delete roles:

curl -XDELETE -u elastic -p changeme 'localhost:9200/_shield/role/shakespeare_admin'

The roles API, together with the new native realm complement each other in defining a complete set of user management APIs.

Epilogue

We consider the new native realm and the Role Management API to be milestone features. Not only will they make your life simpler by letting you manage users and privileges via an API, it also opens the door for tighter integration between Shield and other products in the Elastic Stack. For example, in the near future you can expect to see user and role management UIs in Kibana.

We really recommend you take Shield 2.3 for a spin, and as always, we’d love to hear your feedback.