16 August 2017 Engineering

Default Security for Elasticsearch and the Elastic Stack

By Josh Bressers

unicorn.pngVersion 6.0 of the Elastic Stack is set to be released soon. The first beta was recently released and it's loaded with new and interesting features. The changes I'm most interested in, and most involved with, are changes to default security features. Elastic's X-Pack adds a number of features to the Elastic Stack, two of those features that will see substantial changes in version 6.0 are authentication and TLS support for your cluster.

The Way it Was

In the past when you installed X-Pack it would automatically create three users: elastic, kibana, and logstash_system. All three users were created with the password of "changeme". The intent was you would change those passwords right away. There is even a setting called xpack.security.authc.accept_default_password that could be set to disallow this password once your cluster was running. Default passwords aren't ideal in a security conscious environment, but they solved the difficult problem of how to bootstrap a password.

Additionally a default cluster install would communicate between clients and nodes using unencrypted protocols over the network. Communications, including sensitive data, were not encrypted by default. A malicious user on the network could possibly intercept data as it traveled between nodes. The option existed to enable TLS encrypted communication between nodes and clients, but it was optional. This may have been acceptable behavior for a product in the past, today is different though. Security was often viewed as something you could deal with tomorrow. As history has taught us, tomorrow never comes.

stickie-notes.pngThe expectations of the industry regarding default security in products is catching up to meet the current challenges. Today many organizations have security groups and often compliance is driving their security requirements. The days of default passwords and plaintext communications are no longer acceptable. If you pay attention to any news you are constantly hearing about companies dealing with security breaches, cyber attacks, and even malicious insiders. Our primary goal at Elastic is to help you stay off the front page because of an insecure cluster. We want to be an organization that understands the current state of security as well as where it's heading next, a trusted resource you can rely on instead of a vendor that has to be dragged into the present.

Default Passwords No More

security-lock.pngI'm pleased to say there are no more default passwords for the Elastic Stack. This naturally creates an extra step when spinning up the cluster, but we've done our best to make the process simple and ensure it meets industry security expectations.

We decided to solve this by adding what we're calling a bootstrap password. A bootstrap password can be auto generated or placed in the Elasticsearch keystore before the node is started. The bootstrap password is a temporary password that is used only until a real password has been set for the elastic user. As soon as the cluster has started, you can use the setup-passwords command line tool to either set or generate strong passwords for the internal elastic, kibana, and logstash-system users. This prevents a race condition between starting the cluster and setting the administrator password. The old model suffered from this problem, before the password was changed, even if it was changed quickly, there was an amount of time where an outside user could authenticate against the node.

The details on the new password workflow can be found in the X-Pack documentation.

TLS Everywhere

castle.pngBefore we discuss the TLS changes, it's important to set the stage by explaining how we, as an industry, got to where we are today. If you've never heard of something called BeyondCorp or Zero Trust Networking I suggest you spend some time becoming familiar with these ideas as they represent the future of network design. The basic concept is that the world has changed drastically in the last few years and we've reached a point where security experts agree that your own network can no longer be trusted. That's right, the network you build with your own two hands should be considered hostile!

The thinking, even a few years ago, was that if you have a good firewall between your network and the Internet, everything would be fine. You didn't have to worry about keeping things secure inside your network because the bad guys couldn't get through your firewall. The reality is considerably different than this simplistic view. The reality is your network is probably compromised by both outside and inside attackers. The data traveling across the wires you own should be considered suspect. We have to find ways to push the trust to the endpoints now. This is exactly the sort of problem TLS was designed to solve. TLS relies on using certificates, signatures, and encryption to ensure that the communication between two endpoints will be secured, even if the network between those endpoints is hostile.

Because of this new normal, we are now making TLS communication between nodes in an elasticsearch cluster mandatory. It does add some overhead to the getting started experience—working with certificates isn't always a simple task—but we believe this is a very important next step in keeping the Elastic Stack on the right side of security innovation. Secured communications aren't the future, they're the present. Rather than waiting for the problem to become an emergency we're meeting this one head on. Using encryption by default is certainly the right move for the Elastic Stack.

If you install a single development node today you don't have to enable TLS right away. We however have something we call "bootstrap checks" that run when Elasticsearch enters production mode, such as binding to an external network interface. Verifying that TLS is enabled is now part of the bootstrap checking process. Details on how to configure TLS communications for your Elastic Stack cluster can be found in our X-Pack getting started guide.

This change also gave us an opportunity use the TLS layer to strengthen how we decide trust between nodes. The default setting was to trust any node that joined a cluster. This of course could be dangerous in an environment where you can't trust your network. We had a setting called the "system key" that could provide message integrity between nodes and prevent a rogue node from joining the cluster, but it didn't provide any encryption and was difficult to setup and manage properly. TLS proved to be a more robust and complete solution for this problem. If you were using the system key in previous versions of X-Pack this setting is no longer needed.

map-x-marks-the-spot.pngAt Elastic we will continue to track industry trends and adjust our products accordingly in the future. What is considered default security today will certainly change in the future. Just a few years ago there was little to no default security in anything, today there are certain expectations, tomorrow there will be even more expectations. As industry threats evolve, product security must evolve with it. Security isn't a destination, it's a journey that has no end. These changes are just the first steps in a long journey to adding more security to the Elastic Stack. 

Stay tuned, there's plenty more coming. In the meantime, take 6.0.0-beta1 for a spin and provide us with feedback on these changes to become an Elastic Pioneer