Learn the basics of alerting in Elasticsearch using Watcher

<html><head></head><body>Elasticsearch has many use cases and often those use cases involve knowing when an event that is streaming into the system meets certain conditions and having additional data from the event at our disposal. For these scenarios, we developed Watcher. This component is very powerful and flexible alerting tool, however, because it is so configurable sometimes reviewing the documentation can be daunting and seem very complex.
In this article, I&#x2019;ll present the main concepts and describe the basics of Watcher to create alerts from a simple configuration for testing and a more real world implementation. By the end of these steps, you will not only have installed and configured Watcher with active alerts but you will understand the fundamentals necessary to tailor Watcher to your environment.
For this demonstration, the following software versions were used:
<ul>
	<li>Elasticsearch 2.4.0</li>
	<li>Watcher 2.4.0</li>
	<li>CentOS 6.7</li>
	<li>Kibana 4.6.1 with Sense*</li>
</ul>*not required for these examples but convenient for submitting commands.
Installing the Watcher plugin 
The installation instructions below assumes that your Elasticsearch nodes have access to the Internet. If they do not, you can manually download the .zip, tar.gz or .deb packages from the Elastic site.
*If you have not already done so, you&#x2019;ll need to install the license plugin prior to installing and configuring watcher.
Navigate to your Elasticsearch installation&#xA0;directory and run the plugin tool
<pre class="prettyprint">bin/plugin install license
</pre>*The license plugin comes with an auto-generated 30 day trial license for all Elastic plugins.
Install the Watcher plugin:
<pre class="prettyprint">bin/plugin install watcher
</pre>Repeat this procedure on every node in the cluster.
For more information on installation options: <a href="https://www.elastic.co/guide/en/watcher/current/installing-watcher.html">https://www.elastic.co/guide/en/watcher/current/installing-watcher.html</a>
Configuring Watcher
For this article, We&#x2019;ll be using a generic gmail account to send the alerts. Watcher is capable of sending the alerts to different targets such as Slack, HipChat, and PagerDuty. If you are interested in sending the alerts to different targets, you can find information in the Administering Watcher section of the documentation.
For more information see: <a href="https://www.elastic.co/guide/en/watcher/current/administering-watcher.html">https://www.elastic.co/guide/en/watcher/current/administering-watcher.html</a>
To use a gmail account as in the examples to follow, you must disable the two factor authentication to allow Watcher to send alerts from the account.&#xA0;
	 <a href="https://support.google.com/accounts/answer/6010255?hl=en">https://support.google.com/accounts/answer/6010255?hl=en</a>
To configure the account for Watcher, add the following in elasticsearch.yml substituting your account credentials. The settings are indent sensitive, use spaces not tabs:
<pre class="prettyprint">#------WATCHER CONFIG------
watcher.actions.email.service.account:
 work:
 profile: standard
 email_defaults:
 from: my.account@gmail.com
 smtp:
 auth: true
 starttls.enable: true
 host: smtp.gmail.com
 port: 587
 user: my.account@gmail.com
 password: mypassword
</pre>If you plan to use scripting in the watch, you&#x2019;ll also need to enable scripting by adding the following to the elasticsearch.yml file:
<pre class="prettyprint">#------SCRIPTING CONFIG -----
script.engine.groovy.inline.elasticsearch-watcher_watch: on
</pre>You may already have enabled scripting for other uses with the following, if you have, then the above setting is not needed.
<pre class="prettyprint">#------SCRIPTING CONFIG -----
script.inline: on
script.indexed: on
</pre>Once you have installed the plugin on all nodes and have set the configurations above, you will need to restart each node for the settings to take effect.
What is a Watch?
A watch is basically an alert configuration. It is where we define the schedule, input, conditions, and actions that control what Watcher will do. The documentation examples may appear complex, however, they can be broken down into sections that ask just four basic questions from Watcher:
<ul>
	<li>When should I check?</li>
	<li>Where should I check?</li>
	<li>What should I check for?</li>
	<li>How should I tell you?</li>
</ul>We&#x2019;ll go through how to answer these questions for Watcher and wrap up the configuration at the end so we can submit it to Elasticsearch.
When should I check?
This configuration is answered by using the following syntax. In the example below, we want Watcher to check to every 5 minutes. This is done with the trigger parameter by specifying a schedule with an interval.
<pre class="prettyprint"> &quot;trigger&quot; : { 
 &quot;schedule&quot; : { &quot;interval&quot; : &quot;5m&quot; }
 }
</pre>*There are several ways that you can specify the triggers such as by specifying certain times, days, etc. and also by specifying an interval as we have done in this example.
For more information on triggers:
	 <a href="https://www.elastic.co/guide/en/watcher/current/trigger.html">https://www.elastic.co/guide/en/watcher/current/trigger.html</a>
Where should I check?
This question is answered by using the input parameter. There are several types of inputs we can use but for our purposes, we will use two types, the simple input, which we will use for basic testing, and the search input to get data from an index in Elasticsearch. Since the inputs are loaded into the watch context, you can use the loaded information later as conditions, actions or transforms.
Example #1
In this example, we&#x2019;re using the simple input which is just basically an inline document. We&#x2019;ll use this to test our Watcher configuration and demonstrate how to pull a value, &#x201C;red&#x201D;, from our field, &#x201C;my_field&#x201D;, and inject it into the email alert in a later section.
<pre class="prettyprint"> &quot;input&quot; : {
 &quot;simple&quot; : { &quot;my_field&quot; : &quot;red&quot; }
 }
</pre>Example #2
Using this example, we&#x2019;re going to use the search input type to specify an index, &#x201C;my_monitored_index&#x201D;, and a basic query that will retrieve all the documents that have a date stamp within the last 10 minutes in the &#x201C;my_date&#x201D; field. Here we use the request portion to specify what indices we want and the body portion is where we&#x2019;ll define our query. The query section uses the same syntax as the _search body that a REST request would use. In this case, we are using a range query to get the data we want to look at.
<pre class="prettyprint"> &quot;input&quot; : { 
 &quot;search&quot; : {
 &quot;request&quot; : {
 &quot;indices&quot; : [ &quot;my_monitored_index&quot; ],
 &quot;body&quot; : {
 &quot;query&quot; : {
 &quot;range&quot; : {
 &quot;my_date&quot; : { &quot;gte&quot; : &quot;now-10m&quot; }
 }
 }
 }
 }
 }
 }
</pre>For more information on input types: 
<a href="https://www.elastic.co/guide/en/watcher/current/input.html">https://www.elastic.co/guide/en/watcher/current/input.html</a>
What should I check for?
The answer for this is optional. You can actually just have Watcher send an alert without any conditions and it will default to the always type. For instance, if all you wanted was to have a count of documents in an index sent to you every day, etc. In our example watches, we are going to specify a condition in one and in the other we are going to omit this section for testing basic alerting.&#xA0;
To specify what to look for, we use the condition parameter and denote what type of condition type we want to use (always, never, script and compare). &#xA0;It can be very simple or we can create very complex conditions using a script type. &#xA0;
In this example, we are going to look for field, &#x201C;my_field&#x201D;, to have a value of &#x201C;red&#x201D; by using the compare type with the ctx.payload variable which we can then use to reference &#x201C;my_field&#x201D;.
<pre class="prettyprint"> &quot;condition&quot; : {
 &quot;compare&quot; : {
 &quot;ctx.payload.hits.hits.0._source.my_field&quot; : { &quot;eq&quot; : &quot;red&#x201D; }
 }
 }
</pre>For more information on condition types, variables and operators:
	 <a href="https://www.elastic.co/guide/en/watcher/current/condition.html">https://www.elastic.co/guide/en/watcher/current/condition.html</a>
How should I tell you?
This is where you tell Watcher what it should do with the information it has found. Watcher can perform one or multiple actions and those actions can be injected with information about what it has found. In this example, we are going to use our previously configured gmail account in the elasticsearch.yml file to send an email with the value of &#x201C;my_field&#x201D; in the email body. This action is for our conditional example.&#xA0;
<pre class="prettyprint"> &quot;actions&quot; : {
 &quot;my_action&quot; : {
 &quot;email&quot; : {
 &quot;to&quot; : &quot;your_alert_mailbox@yourdomain.com&quot;,
 &quot;subject&quot; : &quot;Hello From Your Elasticsearch Cluster&quot;,
 &quot;body&quot; : &quot;This is the conditional watch. The value of my_field is: {{ ctx.payload.hits.hits.0._source.my_field }} &quot;
 }
 }
 }
</pre>Putting It All Together
Now that we have answered the different questions, we can now create a wrapper around the different parts and put them in Elasticsearch. The wrapper is just specifying the watcher endpoint and the name that we will give our watch. &#xA0;You can either use the Sense plugin in Kibana (which I use in the examples below) or you can use cURL through REST API.
The basic syntax is like this:
<pre class="prettyprint">PUT _watcher/watch/my_watch_name
{
 &lt;TRIGGER&gt;,
 &lt;INPUT&gt;,
 &lt;CONDITION&gt;,
 &lt;ACTIONS&gt;
}
</pre>Example Watch #1 : Testing a basic alert and injecting a variable from a simple input type.
Using the our prior work, we just need to put it together into our wrapper template and index it into Elasticsearch. Since this watch doesn&#x2019;t have any conditions, it will just fire and send an email every five minutes with the information.&#xA0;
<pre class="prettyprint">PUT _watcher/watch/my_simple_watch
{
 &quot;trigger&quot; : { 
 &quot;schedule&quot; : { &quot;interval&quot; : &quot;5m&quot; }
 },
 &quot;input&quot; : {
 &quot;simple&quot; : { &quot;my_field&quot; : &quot;red&quot; }
 },
 &quot;actions&quot; : {
 &quot;my_action&quot; : {
 &quot;email&quot; : {
 &quot;to&quot; : &quot;my_alert_mailbox@mydomain.com&quot;,
 &quot;subject&quot; : &quot;Hello From Your Elasticsearch Cluster&quot;,
 &quot;body&quot; : &quot;This is the simple watch. The value of my_field is: {{ ctx.payload.my_field }} &quot;
 }
 }
 }
}
</pre>Example Watch #2 : Testing an alert with a condition defined with data from an index.
In this watch, we will include all of the sections and index a sample document to test the condition we set, mainly looking for documents with the value of &#x201C;red&#x201D; in &#x201C;my_field&#x201D;.
<pre class="prettyprint">PUT _watcher/watch/my_conditional_watch
{
 &quot;trigger&quot; : { 
 &quot;schedule&quot; : { &quot;interval&quot; : &quot;5m&quot; }
 },
 &quot;input&quot; : { 
 &quot;search&quot; : {
 &quot;request&quot; : {
 &quot;search_type&quot; : &quot;query_and_fetch&quot;,
 &quot;indices&quot; : [ &quot;my_monitored_index&quot; ],
 &quot;body&quot; : {
 &quot;query&quot; : {
 &quot;range&quot; : {
 &quot;my_date&quot; : { &quot;gte&quot; : &quot;now-10m&quot; }
 }
 }
 }
 }
 }
 },
 &quot;condition&quot; : {
 &quot;compare&quot; : {
 &quot;ctx.payload.hits.hits.0._source.my_field&quot; : { &quot;eq&quot; : &quot;red&quot; }
 }
 },
 &quot;actions&quot; : {
 &quot;my_action&quot; : {
 &quot;email&quot; : {
 &quot;to&quot; : &quot;your_alert_mailbox@yourdomain.com&quot;,
 &quot;subject&quot; : &quot;Hello From Your Elasticsearch Cluster&quot;,
 &quot;body&quot; : &quot;This is the conditional watch. The value of my_field is: {{ ctx.payload.hits.hits.0._source.my_field }} &quot;
 }
 }
 }
}
</pre>To test this watch, we&#x2019;ll just need to index a simple document into Elasticsearch that has the value we&#x2019;re looking for but first, let&#x2019;s define the mapping for our index so that we can set the data type for the my_date field and allow us to do date math on it.
<pre class="prettyprint">PUT /my_monitored_index
{
 &quot;mappings&quot;: {
 &quot;mytype&quot;: {
 &quot;properties&quot;: {
 &quot;my_field&quot; : {
 &quot;type&quot; : &quot;string&quot;
 },
 &quot;my_date&quot;: {
 &quot;type&quot;: &quot;date&quot;,
 &quot;format&quot;: &quot;yyyy-MM-dd HH:mm:ss&quot;
 }
 }
 }
 }
}
</pre>Now we index a simple document. You will need to update the date below to match the time and date of your test in UTC. Try different tests such as changing the color value in my_field or setting the my_date field to a time outside the 10 minute window that we&#x2019;ve configured in the watch.
<pre class="prettyprint">PUT /my_monitored_index/mytype/1
{
 &quot;my_field&quot; : &quot;red&quot;,
 &quot;my_date&quot; : &quot;2016-09-14 23:00:00&quot;
}
</pre>You should receive an email alert when the condition is met. In fact, you may already be tired of receiving these alerts and you want to get rid of them. &#xA0;
You can do this easily by using the API as in the example below:
<pre class="prettyprint">DELETE _watcher/watch/my_simple_watch
</pre>Testing and Debugging Tools
You can force execution of a watch by using the _execute endpoint:
<pre class="prettyprint">POST _watcher/watch/my_simple_watch/_execute
</pre>*The _execute endpoint also allows for different debugging parameters such as modifying actions, ignoring the condition, using alternative input, etc.
For information on testing and debugging options:
	 <a href="https://www.elastic.co/guide/en/watcher/current/api-rest.html#api-rest-execute-watch">https://www.elastic.co/guide/en/watcher/current/api-rest.html#api-rest-execute-watch</a>
Summary
Watcher is a powerful and highly configurable component of the Elastic Stack. These examples are meant to demystify the steps needed to configure a watch. From here, you can use different options to catch a variety of conditions from different sources and send them to multiple targets. For example, you can tell Watcher &#x201C;Hey, I want you to check the log data coming in from my firewall every minute and see if within the last 5 minutes there were over 10 failed login attempts, if there was, send an email to our security group with the users list and notify the security chat room in Slack&#x201D;For detailed information about Watcher: 
	<a href="https://www.elastic.co/guide/en/watcher/current/introduction.html">https://www.elastic.co/guide/en/watcher/current/introduction.html</a>
For further information about the Watcher REST API: 
	<a href="https://www.elastic.co/guide/en/watcher/current/api-rest.html">https://www.elastic.co/guide/en/watcher/current/api-rest.html</a>
Using the Watcher context: 
	<a href="https://www.elastic.co/guide/en/watcher/current/scripts-templates.html#watch-execution-context">https://www.elastic.co/guide/en/watcher/current/scripts-templates.html#watch-execution-context</a>
Visualizing Watcher History in Kibana: 
	<a href="https://www.elastic.co/guide/en/watcher/current/watch-history.html#monitoring-watches">https://www.elastic.co/guide/en/watcher/current/watch-history.html#monitoring-watches</a>
 
</body></html>

blog-watch-anat-fb.jpg

blog-watch-anat-thumb.jpg

Anatomy of a Watch

Learn the basics of implementing Shield's Document Level Security in Elasticsearch and visualize results in Kibana

<html><head></head><body>Elastic Shield is capable of filtering documents using query criteria. In this blog post I&apos;ll demonstrate how to use this feature by using a simple, two document data set in Elasticsearch where documents will be filtered for the user according to the value of a field and displayed in a simple Kibana visualization. &#xA0;Users and roles will be created with the Users and Roles API.

<h2>The Scenario</h2>
There are three users; a sales manager and two account representatives. Documents are indexed using a code for one of two regions, EAST and WEST. &#xA0;Each account representative needs read access to documents in their region and the manager will need access to both regions. &#xA0;All users will be using the same Kibana visualizations to view the counts of their documents. In Kibana, users will be allowed to create/modify visualizations and dashboards to view their documents.

The following is the setup used to demonstrate DLS:

<ul>
	<li>Elasticsearch 2.3.X+&#xA0;</li>
	<li>Shield 2.3.X+</li>
	<li>Kibana 4.5.X+&#xA0;</li>
	<li>Sense 2.X+ recommended</li>
</ul>

	*commands can also be issued using cURL. 
	*For this example; Shield is required on Elasticsearch&#xA0;but not on Kibana.

For the the initial steps, you will need to log in using an account with sufficient privileges to create &#xA0;indices, roles and users. &#xA0;The following REST API examples can be used directly in Kibana through the Sense plugin.

<h2>Elasticsearch and Shield</h2>
<h3>STEP 1: Configure Authentication Realm</h3>
In this example, I&apos;ll be using the Native realm to create users so the native realm needs to be configured in Elasticsearch for Shield.

<code>elasticsearch.yml</code> Shield configuration example:

<pre class="prettyprint">#-----SHIELD CONFIG-------
shield:
 authc:
 realms:
 native:
 type: native
 order: 0
</pre>
<h3>STEP 2: Index Sample Data</h3>
First, we will index two simple documents into one index.

This creates a document for the EAST region:

<pre class="prettyprint">PUT myindex/mytype/1
{
 &quot;name&quot; : &quot;ABC Company&quot;,
 &quot;region&quot; : &quot;EAST&quot;
}
</pre>
This creates a document for the WEST region:

<pre class="prettyprint">PUT myindex/mytype/2
{
 &quot;name&quot; : &quot;DEF Company&quot;,
 &quot;region&quot; : &quot;WEST&quot;
}
</pre>
You can verify the index data using:

<pre class="prettyprint">GET myindex/mytype/_search
</pre>
<h3>STEP 3: Create Roles</h3>
Next, create the roles that will be used to allow users of each region to access their documents. We are specifying two indices: .kibana and myindex. &#xA0;The .kibana index is set up with the &quot;all&quot; privilege so that our users can log in, change settings and create visualizations. The myindex index is set up with the &quot;read&quot; privilege and we use a query to specify the documents that the users belonging to the role will be able to see.

*Note 1: The quotes inside the query have been escaped with the backslash character &quot;\&quot;. &#xA0;This is required since we want the query itself to be indexed as a string and not as an object with a &quot;match&quot; property.

*Note 2: These are the minimum options for the goals of this exercise, you can find more advanced options here:

<a href="https://www.elastic.co/guide/en/shield/current/defining-roles.html">https://www.elastic.co/guide/en/shield/current/defining-roles.html</a> 

This creates a role that will have access to only the EAST region:

<pre class="prettyprint">POST /_shield/role/myroleEAST
{
 &quot;indices&quot;: [ 
 {
 &quot;names&quot;: [ &quot;.kibana*&quot;],
 &quot;privileges&quot;: [&quot;all&quot;]
 },
 {
 &quot;names&quot;: [ &quot;myindex*&quot; ], 
 &quot;privileges&quot;: [ &quot;read&quot;, &quot;view_index_metadata&quot; ],
 &quot;query&quot;: &quot;{\&quot;match\&quot;: {\&quot;region\&quot;: \&quot;EAST\&quot;}}&quot;
 }
 ]
}
</pre>
This creates a role that will have access to only the WEST region:

<pre class="prettyprint">POST /_shield/role/myroleWEST
{
 &quot;indices&quot;: [ 
 {
 &quot;names&quot;: [ &quot;.kibana*&quot;],
 &quot;privileges&quot;: [&quot;all&quot;]
 },
 {
 &quot;names&quot;: [ &quot;myindex*&quot; ], 
 &quot;privileges&quot;: [&quot;read&quot;, &quot;view_index_metadata&quot; ],
 &quot;query&quot;: &quot;{\&quot;match\&quot;: {\&quot;region\&quot;: \&quot;WEST\&quot;}}&quot;
 }
 ]
}
</pre>
You can verify the roles by using:

<pre class="prettyprint">GET /_shield/role
</pre>
<h3>STEP 4: Create Users</h3>
Now we are ready to create the three users. One user will be assigned the role that allows access to documents marked with the EAST region, another user to the WEST region and the last account will include both roles.

*Note 1: A user assigned several roles will get the cumulative total of privileges. The permissions will get effectively OR&apos;d together and will result in &quot;most privilege&quot; access.

*Note 2: These are the minimum fields required to create the users, for more advanced options:

<a href="https://www.elastic.co/guide/en/shield/current/shield-rest.html#shield-users-rest">https://www.elastic.co/guide/en/shield/current/shield-rest.html#shield-users-rest</a> 

This will create the user myuserEAST and assign it the myroleEAST role:

<pre class="prettyprint">POST /_shield/user/myuserEAST
{
 &quot;password&quot; : &quot;mypassword&quot;, 
 &quot;roles&quot; : [ &quot;myroleEAST&quot; ]
}
</pre>
This will create the user myuserWEST and assign it the myroleWEST role:

<pre class="prettyprint">POST /_shield/user/myuserWEST
{
 &quot;password&quot; : &quot;mypassword&quot;, 
 &quot;roles&quot; : [ &quot;myroleWEST&quot; ]
}
</pre>
This will create the user myuserManager and assign it both roles:

<pre class="prettyprint">POST /_shield/user/myuserManager
{
 &quot;password&quot; : &quot;mypassword&quot;, 
 &quot;roles&quot; : [ &quot;myroleEAST&quot;,&quot;myroleWEST&quot; ]
}
</pre>
You can verify the users by:

<pre class="prettyprint">GET /_shield/user
</pre>
<h2>Kibana</h2>
In this section, we will configure Kibana to read our new index, myindex, then create a visualization. You will need to log into Kibana with an admin role user to configure the initial dashboard.

<h3>STEP 1: Set Index Pattern in Kibana</h3>
<ol>
	<li>Select the Settings Tab</li>
	<li>Uncheck the &quot;Index contains time-based events&quot; box.</li>
	<li>In the &quot;index name or pattern&quot; text box, enter &quot;myindex*&quot;</li>
	<li>Click on &quot;Create&quot;</li>
</ol>
<img src="https://api.contentstack.io/v2/assets/57740e698c9b8bce1d2359b8/download?uid=blt662010a139fd2bbd" data-sys-asset-uid="blt662010a139fd2bbd" alt="1 index pattern.png">

<h3>STEP 2: Create Visualization in Kibana</h3>
<ol>
	<li>Select Visualize tab</li>
	<li>Select the Vertical bar chart</li>
	<li>In the Step 2, click &quot;From a new search&quot;</li>
	<li>In the Select an Index Pattern drop-down box, select &quot;myindex*&quot;</li><img src="https://api.contentstack.io/v2/assets/57740ef4364563657816c921/download?uid=bltb19a6c50905a6f15" data-sys-asset-uid="bltb19a6c50905a6f15" alt="2 select a source.png">
	 
	<li>Click on X-Axis in buckets section under the &quot;select bucket type&quot;</li>
	<li>In Aggregation, Select &quot;Terms&quot;</li>
	<li>In Field, Select &quot;region&quot;</li><img src="https://api.contentstack.io/v2/assets/57740f31364563657816c930/download?uid=blta059b57dc1504889" data-sys-asset-uid="blta059b57dc1504889" alt="3 visualization config.png">
	 
	<li>Click on &quot;Apply Changes&quot; button (Green button with white arrow)</li>
	<li>Click &quot;Save Visualization&quot; button</li><img src="https://api.contentstack.io/v2/assets/57740fe46809d0d01e1fb19b/download?uid=bltb588dce9d362c42b" data-sys-asset-uid="bltb588dce9d362c42b" alt="4 save visualization.png"> 
	<li>Type name for the visualization, &quot;DLS Visualization Demo&quot; and click &quot;save&quot;</li>
</ol>
<h3>STEP 3: Test Users</h3>
The tests below demonstrate how the query clause defined in the role restricts the data for the users. &#xA0;If there are several visualizations in a dashboard on the same data, the window into the data will also be applied.

<ol>
	<li>Log into Kibana with the myuserEAST user</li>
	<li>Select the Visualizations tab</li>
	<li>Click on DLS Visualization Demo in the saved visualizations.</li><img src="https://api.contentstack.io/v2/assets/57741228712ecd8c6007d0f7/download?uid=blta9aad97a30470339" data-sys-asset-uid="blta9aad97a30470339" alt="5 open saved visualization.png"> 
</ol>
You should see the visualization with only the EAST bar:

<img src="https://api.contentstack.io/v2/assets/57741245712ecd8c6007d0fd/download?uid=blte72b2a8c43014d77" data-sys-asset-uid="blte72b2a8c43014d77" alt="6 myuserEAST visualization.png">

Repeat the test with the myuserWEST user and you should see only the WEST bar.

Repeat the test with the myuserManager and you should see both EAST and WEST bars.

<h2>Summary</h2>
This article provided a base starting point to demonstrate how Shield can be configured with Elasticsearch to restrict Kibana visualization data.

Although we used very basic and minimal configurations, the procedures can be customized in several ways to achieve security goals in your architecture and allow for more complex configurations.

For example:

<ul>
	<li>You can use additional authentication realms such as Active Directory, LDAP, etc. in your Shield configuration.</li>
	<li>You can configure a role to multiple indices.</li>
	<li>Field Level Security can also be used to restrict data in the documents.</li>
	<li>Roles can be defined using the <code>roles.yml</code> file although from Shield 2.3.0, it is recommended to use the Roles API to define them.</li>
</ul>
*Note that if the roles are defined in the <code>roles.yml</code> and in the Roles API, the file definitions will take precedence.

<ul>
	<li>A wide array of privileges can be defined for the Cluster and Index levels to meet your particular needs.</li>
	<li>Dashboards will also inherit the filtered data set coming from the visualizations.</li>
</ul>
For more information:

Shield installation and configuration

<a href="https://www.elastic.co/guide/en/shield/current/installing-shield.html">https://www.elastic.co/guide/en/shield/current/installing-shield.html</a> 

Sense installation and configuration

<a href="https://www.elastic.co/guide/en/sense/current/installing.html">https://www.elastic.co/guide/en/sense/current/installing.html</a> 

Native user authentication

<a href="https://www.elastic.co/guide/en/shield/current/native-realm.html">https://www.elastic.co/guide/en/shield/current/native-realm.html</a> 

Configuring role based access control

<a href="https://www.elastic.co/guide/en/shield/current/configuring-rbac.html">https://www.elastic.co/guide/en/shield/current/configuring-rbac.html</a> 

Setting up field and document level security

<a href="https://www.elastic.co/guide/en/shield/current/setting-up-field-and-document-level-security.html">https://www.elastic.co/guide/en/shield/current/setting-up-field-and-document-level-security.html</a> 
</body></html>

shield-security-for-elasticsearch-image.jpg

Getting Started with Shield Document Level Security in Elasticsearch

Learn the basics of securing Elasticsearch with Windows Active Directory

When learning a new system, I always find it useful to have instructions on how to install and configure a feature with minimum steps in a cookbook style format. &nbsp;It allows me to get up and running quickly without having to reference several pages with the multitude of optional settings. &nbsp;From that point, I can take a look at the reference manuals and the more advanced options to configure the components according to my end architecture requirements. 
In this post, I’ll go through installing and configuring Elastic Shield with Elasticsearch to use Windows Active Directory domain authentication for Elasticsearch Administrators. &nbsp;
This article assumes that you have already have the following prerequisites:
<ul>
	<li>Installed a 2.X &nbsp;elasticsearch cluster on Linux (I used CentOS 6.7)&nbsp;</li>
	<li>DNS has been configured on each node.</li>
	<li>Hosts have access to the Internet.&nbsp;</li>
</ul> A. Configure the Elasticsearch Yum Repository
(Repeat steps below for every node in the cluster)
*You can skip this section, if you’ve already configured it.
1. Download and install the public signing key: 
<pre class="prettyprint">rpm --import https://packages.elastic.co/GPG-KEY-elasticsearch
</pre>2. &nbsp;Create the new yum repo file for Elasticsearch:
<pre class="prettyprint">vi /etc/yum.repos.d/elasticsearch.repo
</pre>3. &nbsp;Copy and paste the following entries in the new repo file:
<pre class="prettyprint">[elasticsearch-2.x]
name=Elasticsearch repository for 2.x packages
baseurl=https://packages.elastic.co/elasticsearch/2.x/centos
gpgcheck=1
gpgkey=https://packages.elastic.co/GPG-KEY-elasticsearch
enabled=1
</pre> B. &nbsp;Configure Active Directory
1. &nbsp;Create a user who will be the Elasticsearch administrative user in Active Directory Users and Computers (ADUC). See example below for user named Elasticsearch Administrator.<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltc1cc2d4131216e0c/5fa97f2172a3526f28dba366/blog-shield-ad-1.png" data-sys-asset-uid="bltc1cc2d4131216e0c" alt="" style="display: block; margin: auto;">2. &nbsp;Create an Active Directory security group for the Elasticsearch Admins. &nbsp;This group will be used to hold our new user account.&nbsp;&nbsp;<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt20e22872493442b7/5fa97f3a4e40cf53001f505b/blog-shield-ad-2.png" style="background-color: initial; display: block; margin: auto;" data-sys-asset-uid="blt20e22872493442b7" alt="">Next, note the location of this user group, we will use this group’s Distinguished Name (DN) in the Shield role mapping in the steps to follow. &nbsp;See example below for &nbsp;the security group named ESAdmins.
 C. Installing the License and Shield Plugins
(Repeat steps below for every node in the cluster)
1. &nbsp;Install the license plugin.&nbsp;
*The license plugin is required to run commercial plugins from Elastic.&nbsp;
<pre class="prettyprint">/usr/share/elasticsearch/bin/plugin install license
</pre>*The license will be installed in a 30 day trial mode, you can update the license once a production license is purchased.
2. Install the Shield plugin.
<pre class="prettyprint">/usr/share/elasticsearch/bin/plugin install shield
</pre>*Note that starting with Elasticsearch 5.0, Elastic Shield and License plugins will be part of X-Pack as Elastic Security.
 D. Configuring Shield with an Active Directory Domain
*There are many options related to protocols, ports, multiple domains, encryption and automated security group mapping. For more information about these refer to the links at the bottom of this article.
1. &nbsp;Open the elasticsearch.yml file.
<pre class="prettyprint">vi /etc/elasticsearch/elasticsearch.yml
</pre>2. &nbsp;Add the following entries at the bottom of the file, replacing the sample domain entry with your specific domain:
<pre class="prettyprint">#-----SHIELD CONFIG------
shield:
 authc:
 realms:
 active_directory:
 type: active_directory
 order: 0
 domain_name: mydomain.local
 url: ldap://mydomain.local:389
 unmapped_groups_as_roles: false
</pre>3. &nbsp;Repeat Steps 1 and 2 for all nodes in the cluster.
4. &nbsp;Open the role_mapping.yml file.
<pre class="prettyprint">vi /etc/elasticsearch/shield/role_mapping.yml
</pre>5. &nbsp;Add the following entry at the bottom of the file, replacing the sample DN location with the location noted above in Section B, Step 2.
<pre class="prettyprint">admin:
 - "cn=ESAdmins, ou=Groups, ou=Elasticsearch, dc=mydomain, dc=local"</pre>6. &nbsp;Copy the role_mapping.yml file to all nodes in the cluster as in the examples below replacing sample node names with your node names:
<pre class="prettyprint">scp /etc/elasticsearch/shield/role_mapping.yml root@esnode2:/etc/elasticsearch/shield/
</pre>7. Restart elasticsearch service on each node.
<pre class="prettyprint">service elasticsearch restart
</pre> E. &nbsp;Test the new user and credentials
You can test by simply performing the following query replacing the sample node name with yours:
<pre class="prettyprint">curl -XGET ‘esnode1:9200’ -u esadmin
</pre>*note that only the username, esadmin, &nbsp;is specified since the shield configuration has the domain information.
Command and output should look something like this:
<pre class="prettyprint">[root@esnode1 ~]# curl -XGET 'esnode1:9200' -u esadmin
Enter host password for user 'esadmin':
{
 "name" : "esnode1",
 "cluster_name" : "marscluster1",
 "version" : {
 "number" : "2.3.2",
 "build_hash" : "b9e4a6acad4008027e4038f6abed7f7dba346f94",
 "build_timestamp" : "2016-04-21T16:03:47Z",
 "build_snapshot" : false,
 "lucene_version" : "5.5.0"
 },
 "tagline" : "You Know, for Search"
}
</pre> 
And that’s it!&nbsp;
There are many options and combinations that are possible. Once you are able to set up this basic Shield and Elasticsearch combination, you can find additional resources below to customize for your architecture requirements and needs.
For additional information on:
&nbsp;Active Directory authentication options:
<a href="https://www.elastic.co/guide/en/shield/current/active-directory-realm.html" target="_blank">https://www.elastic.co/guide/en/shield/current/active-directory-realm.html</a>
Role-based access:
<a href="https://www.elastic.co/guide/en/shield/current/configuring-rbac.html" target="_blank">https://www.elastic.co/guide/en/shield/current/configuring-rbac.html</a>
Mapping groups to roles:
<a href="https://www.elastic.co/guide/en/shield/current/mapping-roles.html" target="_blank">https://www.elastic.co/guide/en/shield/current/mapping-roles.html</a>
Encrypting Active Directory communication with SSL/TLS:
<a href="https://www.elastic.co/guide/en/shield/current/active-directory-realm.html#active-directory-ssl">https://www.elastic.co/guide/en/shield/current/active-directory-realm.html#active-directory-ssl</a>
Installing and managing Elastic licenses:
<a href="https://www.elastic.co/guide/en/shield/current/license-management.html" target="_blank">https://www.elastic.co/guide/en/shield/current/license-management.html</a>
Setting up Elasticsearch repositories:
<a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/setup-repositories.html" target="_blank">https://www.elastic.co/guide/en/elasticsearch/reference/current/setup-repositories.html</a>
<div> 
</div>

AUTHOR

Articles by Marcelo Rodriguez

Anatomy of a Watch

Getting Started with Shield Document Level Security in Elasticsearch

Quick Start Guide - Configuring Elasticsearch with Shield and Active Directory

Sign up for Elastic Cloud free trial

Products & Solutions

Company

We're hiring

Resources

Solutions

Elastic (ELK) Stack

New

Featured topics

News

AUTHOR

Articles by Marcelo Rodriguez

Anatomy of a Watch

Getting Started with Shield Document Level Security in Elasticsearch

Quick Start Guide - Configuring Elasticsearch with Shield and Active Directory

Sign up for Elastic Cloud free trial

Company

We're hiring

Resources