Shield, Watcher, and Marvel 2.0.0 GA Released

Today we're thrilled to announce the release of Shield, Watcher and Marvel 2.0. This is the first release of these extensions that is compatible with Elasticsearch 2.0.

Beyond Elasticsearch 2.0 compatibility, Shield and Watcher 2.0 introduce new and exciting features, expanding the security and alerting capabilities respectively.

Shield

• Extensible Realms - Shield 1.x focused on defining the core foundation for user authentication, and ships with 3 out-of-the-box authentication mechanisms - esusers, LDAP/AD and PKI. While these proved to be sufficient for many users and user cases, we also recognize the need for integration with additional authentication mechanisms. That's why we've opened up Shield's realm based authentication system for extensions and enabled users to plug in their own realm implementation to handle user authentication. Users with special/proprietary authentication mechanisms are no longer blocked and can now take full advantage of Shield's strong security capabilities (e.g. role-based authorization and secured communication). Read more about custom realms here.
• Field & Document ACL - Shield 2.0 introduces Field & Document level access control as first class security feature. It is now possible to define the accessible fields and documents per role. This new capability is much more than configuration change. This access control is implemented in the lowest level going all the way to the Lucene index in Elasticsearch. As a result, not only this solution is easier to maintain, but is more complete. Read more about field and document level ACLs here.
• User Impersonation - Last but not least, Shield 2.0 introduces a User Impersonation capability, where a user (with the appropriate permissions) can impersonate other users and execute requests on their behalf. This can be useful in set ups where the applications built on top of Elasticsearch already take care of user authentication, yet authorization must still be applied on the Elasticsearch side. In this scenario, one can set up a “main” user for the application, assign it impersonation rights and have that user execute all the Elasticsearch requests on behalf of the application users. Read more about user impersonation here.

Watcher

• Slack & HipChat Integration - Slack and HipChat are team/group collaboration tools that are rapidly becoming mainstream and serve as main internal communication hubs in organizations. Watcher 2.0 introduces dedicated actions to enable watch notifications to channels/rooms and users via these communication channels. Read more about the slack and hipchat actions. 
• Array Compare Condition - A new condition that simplifies spike detections over data points within time series. Following the spirit of the compare condition introduced in 1.x,  this condition is script-free, thus can be used without the need to enable dynamic scripts in Elasticsearch. Read more about the array_compare condition.
• Watch De/activation - A common request we've got from users was around disabling watches. In 1.x there is no mechanism in place to deactivate registered watches. While it was possible to work around this limitation by changing the watch trigger or by simply deleting the watch - these are nothing more than workarounds that make it difficult to manage watcher as a whole. In 2.0 it is now possible to de/activate registered watches via simple API calls, without the need to modify the watches. It is a basic capability, that should have been in Watcher from 1.0, it wasn't… we fixed it. Read more here.

Marvel

We’re super excited to introduce Marvel 2.0. With a complete UI redesign, built on top of Kibana 4, we have taken all that we’ve learned from Marvel 1.x and built an easier to use, streamlined monitoring UI. In the same spirit of Shield and Watcher, this first Marvel release lays the foundation for future growth and focuses on the key metrics required to efficiently monitor Elasticsearch 2.0.

As part of the redesign, we reduced the interface to 6 pages:

Cluster List

An increasing number of our users and customers are running multiple clusters, and Marvel now makes it easy to monitor them all from a centralized monitoring cluster. Just configure each cluster to send data to the monitoring cluster, and Marvel does the rest.

Cluster Overview

The cluster overview shows the key performance metrics for a single cluster, allowing you to quickly identify spikes or valleys. The page also shows any active shard recoveries or relocations.

Indices List

The indices list shows all indexes in the cluster, along with a host of properties. The table updates live and supports interactive filtering and sorting. Ever wonder what your biggest index is? Wonder no more.

Index Detail

The index detail page shows the key performance metrics of your index, as well as providing a clear picture of where the shards are allocated.

Nodes List

The nodes list shows the set of nodes in the cluster, along with key performance metrics. The table updates live and supports interactive filtering. Easily identify nodes with high CPU usage or low disk space.

Node Detail

The node detail page captures the key performance metrics of an individual node, as well as the set of shards on the node.

As part of being built on top of Kibana 4, there are some operational changes as well. Marvel now installs in two parts - the marvel-agent, and the Marvel user interface.

Marvel Agent

The marvel-agent installs as a plugin into your elasticsearch cluster. It captures the key performance information and stores it locally or pushes it to a separate monitoring cluster.

Marvel User Interface

The Marvel UI installs into Kibana as a plugin. This uses the new Plugin infrastructure in Kibana 4.2 to provide a separate Marvel App inside the Kibana interface, which is reachable with a new app-switcher control:

The 2.0 release is a big one for our products, and we’re eager to hear what you think. Reach out to us in the forums: https://discuss.elastic.co or by email at info@elastic.co.