27 January 2015 News

Shield: Redefining What You Can Do with ELK

By Steven Schuurman, CEO

When we originally built Elasticsearch, we architected it to serve as a powerful search engine that could easily scale across a distributed environment and be used to both search and analyze any type of data — essentially serve as the backbone for a business to centralize, explore, and extract insights out of any data they want. When we released the software four years ago, however, using a search engine to also perform operational and even business analytics was a completely foreign, somewhat extreme idea — so we let our users make this discovery on their own.

Fast forward to nearly 20 million downloads later and 4x customer growth in 2014, businesses worldwide continue to implement Elasticsearch, Logstash, and Kibana for an endless variety of use cases. Adoption started with end user-facing sites like Facebook, GitHub, and Yelp implementing us for search, then moved to businesses like ING, Mozilla, and TomTom utilizing Elasticsearch for log analysis.

Over the past year, we've started to see ELK stack usage align with our initial vision of powering custom analytics use cases — many that have far exceeded our wildest dreams. The Guardian built its own analytics tool on top of Elasticsearch to make sure it publishes and presents the right news content on its site at the best time of day, not only leveraging data to make sure its content gets maximum exposure but also to guide its editorial strategy by evaluating which content its audience reads. Mayo Clinic uses our software to analyze patient data. Financial services firms analyze trade data with Elasticsearch to help advise their clients on investment decisions. Criminal investigative units put forensics data in Elasticsearch to help solve crimes. The list goes on ….

Needless to say, businesses continue to put more and more data into the ELK stack to extract insights. In order to meet customer demand to utilize ELK across even more of their business, we're extremely excited to introduce Shield, a security and administrative plugin to the ELK stack that makes it easier for our customers to do even more things with their most valuable asset: their data.

Shield's introduction today follows a successful beta with a handful of our users and customers. Some of the new capabilities Shield provides that let businesses do even more with the ELK stack include:

  • Multitenant role-based access control: With the ability to manage who can access what data, as well as the permissions around what they can do with it, businesses can feel comfortable increasing the amount of data they put in to Elasticsearch. During the Shield beta, we worked with the internal IT services team at one of the largest local governments in the U.S. They use Elasticsearch to centralize logs from 38 departments across 1,000 servers. Shield will allow them to scale their operations by giving each department access to the logs by providing role-based access multitenancy, setting permissions for each department to only have access to its own data.
  • Integration with existing enterprise infrastructure: Open source software tends to be powerful and easy-to-use, and therefore adds value right away, but where it gets dinged at times is its ability to integrate into existing enterprise infrastructure. The LDAP-based authentication support in Shield lets the ELK stack integrate with authentication systems like Active Directory, easing administrative complexity by allowing IT to continue to manage access to all their systems in a central place.
  • Compliance with security regulations: Thanks to the audit logging feature in Shield, businesses under heavy regulation can use Elasticsearch for data that's regulated or governed with a complete record of all user activity in their Elasticsearch deployment. A multi-national financial services corporation in our Shield beta started using Elasticsearch to detect credit card fraud. The audit logging provided by Shield allows them to meet compliance regulations by providing a record of who has access to regulated data in Elasticsearch and what they've done with it.
  • Secure extension to the cloud: As a lot of us know, cloud infrastructures give businesses significant cost savings when it comes to utilizing massive amounts of data at scale. Thanks to Shield's node-to-node, certificate-based SSL/TLS encryption, businesses can feel more comfortable deploying Elasticsearch in their cloud environments knowing their data is encrypted when in-flight.
  • More manageable expansion of ELK: Also thanks to role-based access control, Elasticsearch customers can set who can access and do what with the data in Elasticsearch, helping to prevent unfortunate human errors that occur from time to time (like accidentally deleting or editing data).

Shield is available starting today as part of our Development, Gold, and Platinum subscriptions (which means it's *free* for existing customers!). Those interested in checking it out can download a free 30-day trial at www.elastic.co/products/shield/.

We couldn't be more excited about the next evolution of both our software stack as well as our business, and hope our users are, too.