Get a breakdown of how authorization and authentication work in Elasticsearch. Learn more about who gets in and what data they can (and can't) see.

I’ve been with Elastic for more than 4 years — 3.5 of those years working in support and consulting and the last half year in sales. Regardless of the department I’m in, the most common questions I’ve gotten from users are about securing Elasticsearch. What type of authentication does Elasticsearch support? How do I set it up? How can I make sure users don’t see data they’re not supposed to? My goal for this blog is to answer those questions, as well as provide some guidance for resolving some common issues while configuring security.
First, some history. If you’ve been using Elasticsearch for a while, you’ll know that security was once provided by a plugin called <a href="/blog/you-know-for-security-shield-goes-ga">Shield</a> that was offered through X-Pack. Nowadays, that security functionality has been moved into the Elastic Stack (<a href="/what-is/open-x-pack">along with the rest of X-Pack</a>), with the most commonly used features available <a href="/blog/security-for-elasticsearch-is-now-free">free with the default distribution</a>. Security includes <a href="/guide/en/elasticsearch/reference/7.4/encrypting-communications.html">encrypted communication</a> (TLS/SSL), <a href="/guide/en/elasticsearch/reference/7.4/setting-up-authentication.html">authentication</a> (native, LDAP, SSO, etc), <a href="/guide/en/elasticsearch/reference/7.4/authorization.html">authorization</a> (RBAC, ABAC, etc.), <a href="/guide/en/elasticsearch/reference/7.4/ip-filtering.html">IP filtering</a>, <a href="/guide/en/elasticsearch/reference/7.4/auditing.html">audit logging</a>, and <a href="/guide/en/elasticsearch/reference/7.4/elasticsearch-security.html">much more</a>. The focus of this blog will be on the two “auths”.
<h2>Authentication in Elasticsearch</h2>Simply put, if a user or API wants to access Elasticsearch, it needs to be authenticated.
Elasticsearch supports various security methods natively, such as:
<ul>
	<li><a href="/guide/en/elasticsearch/reference/7.4/native-realm.html">Native</a> user authentication</li>
	<li><a href="/guide/en/elasticsearch/reference/7.4/active-directory-realm.html">Active Directory</a> user authentication</li>
	<li><a href="/guide/en/elasticsearch/reference/7.4/file-realm.html">File-based</a> user authentication</li>
	<li><a href="/guide/en/elasticsearch/reference/7.4/ldap-realm.html">LDAP</a> user authentication</li>
	<li><a href="/guide/en/elasticsearch/reference/7.4/pki-realm.html">PKI</a> user authentication</li>
	<li><a href="/guide/en/elasticsearch/reference/7.4/saml-realm.html">SAML</a> authentication</li>
	<li><a href="/guide/en/elasticsearch/reference/7.4/kerberos-realm.html">Kerberos</a> authentication</li>
</ul>You can even create your <a href="/guide/en/elasticsearch/reference/7.4/custom-realms.html">own integration</a> if one of these doesn’t apply to you. However, we typically recommend that you use one of the existing integrations, as they are validated and we keep developing with them to ensure proper support.
The most basic component of security within the Elasticsearch is a <a href="/guide/en/elasticsearch/reference/7.4/realms.html#realms">realm, which is what resolves and authenticates users.</a> Each of the above listed authentication methods would be considered a realm. And in order to function, Elasticsearch works with a <a href="/guide/en/elasticsearch/reference/7.4/realm-chains.html">realm chain</a>. A realm chain is a prioritized list of configured realms (from 1 to N realms) in ascending order of preference. When a user tries to access Elasticsearch, the request will step through the list sequentially until authentication succeeds or it runs out of realms to try.
Essentially, the process will try the first realm configured in the list and, if it fails, it will continue with the next until either succeeding with one of the realm items or running exhaustively through the entire list. This first step into accessing Elasticsearch is called authentication.
Once a user is authenticated, Elasticsearch will then attempt to assign one or more roles to them. Roles can be assigned to users statically or dynamically during authentication, based on some of the user properties. Depending on the realm type that authenticated the user, the user properties used to assign a role can be the group membership they have in an external system, or the suffix of their username in Elasticsearch, etc. Additionally, Elasticsearch also features <a href="/guide/en/elasticsearch/reference/7.4/run-as-privilege.html">run as functionality</a>, which allows users to submit requests on behalf of other users without requiring re-authentication.
With the authentication phase complete, the next step is authorization.You can check the entire workflow in the following chart:
<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt9b2001d36fce9b26/5d9c78fb230e52144ca9e36a/realm_chain.png" data-sys-asset-uid="blt9b2001d36fce9b26" alt="Realm chaining in Elasticsearch" width="460" "="">
<h2>Authorization in Elasticsearch</h2>Once authentication is successful, the user will be moved onto the second security checkpoint: <a href="/guide/en/elasticsearch/reference/current/authorization.html">authorization</a>. Authorization is the process of determining whether the user is allowed to execute a request, and it is done through <a href="/guide/en/elasticsearch/reference/7.4/mapping-roles.html">mapping users</a> to <a href="/guide/en/elasticsearch/reference/7.4/defining-roles.html">predefined and/or user-defined roles</a>. There are roles that come by default with Elasticsearch, but you can also create specific roles for your use case.
A role is comprised of:
<ul>
	<li>A list of users that can access</li>
	<li>Cluster privileges</li>
	<li>Global privileges</li>
	<li>Indices privileges</li>
	<li>Application privileges</li>
</ul>In this second phase, Elasticsearch will be using one of the following:<ul>
<li>The <a href="/guide/en/elasticsearch/reference/7.3/security-api-put-role-mapping.html">role_mapping API</a>, which is the preferred method because you can manage each role’s mapping in an API-driven and centralized way. This is the only way to configure role mappings in <a href="/elasticsearch/service">Elasticsearch Service</a> on Elastic Cloud.</li>
<li>The role-mapping file (role_mapping.yml) that exists in each node inside the <a href="/guide/en/elasticsearch/reference/7.4/settings.html">configuration folder</a></li>
</ul>The <code>role_mapping</code> API needs to be invoked by a user with appropriate rights to manage roles. The <code>superuser</code> role that the <code>Elastic</code> user has is an example of it — however, you can create a specific role for this, too.
Realms and roles are completely different by definition. Realms and realm chains are what we use to get authenticated, and after the authentication phase we end up using roles to map to users. While a user will be only authenticated using a single realm from the realm chain, in the second phase the user can be mapped from 1 to many roles. This is also known as role-based access control.
<h2>Troubleshooting steps for authentication and authorization issues</h2>Now that we’ve gone over the basics behind authentication and authorization, let’s take a look at some of the troubleshooting steps you can take if you run into any issues.
<h3>401 Unauthorized</h3>If you can’t be authenticated using any of the realms, you will get a response with <code>401 Unauthorized</code> status code. The easiest way to try your credentials against the list of configured realms is to use <code>cURL</code> with a flag that allows inspection (for example, <code>-v</code>). You may get a response like the following:
<pre class="prettyprint">curl https://xxxxxx:9200 -u test:test -v 
&gt; User-Agent: curl/7.54.0 
&gt; Accept: */* 
&gt; 
&lt; HTTP/1.1 401 Unauthorized 
&lt; Content-Type: application/json; charset=UTF-8 
&lt; Date: Tue, 10 Sep 2019 15:59:33 GMT 
&lt; Www-Authenticate: Bearer realm="security" 
&lt; Www-Authenticate: ApiKey 
&lt; Www-Authenticate: Basic realm="security" charset="UTF-8" 
&lt; Content-Length: 455 
&lt; Connection: keep-alive 
&lt; 
* Connection #0 to host 06618318cff64c829af1cd5a2beb91a5.us-east-1.aws.found.io left intact 
{"error":{"root_cause":[{"type":"security_exception","reason":"unable to authenticate user [test] for REST request [/]","header":{"WWW-Authenticate":["Bearer realm=\"security\"","ApiKey","Basic realm=\"security\" charset=\"UTF-8\""]}}],"type":"security_exception","reason":"unable to authenticate user [test] for REST request [/]","header":{"WWW-Authenticate":["Bearer realm=\"security\"","ApiKey","Basic realm=\"security\" charset=\"UTF-8\""]}},"status":401}%
</pre>This error message can help you get to the root cause of your issue. For example, there are specific realms such as <a href="/guide/en/elasticsearch/reference/7.4/configuring-saml-realm.html">SAML</a> that require Kibana or a custom web application to do the interaction with Elasticsearch and the Identity Provider.
<h3>Enabling error logging</h3>After setting up the realms, realm chain, roles, and role mapping, if you still have issues, you can easily configure some extra logging to get further information around the process of authentication and authorization that we covered previously.
Using the <a href="/guide/en/elasticsearch/reference/current/logging.html#configuring-logging-levels">logging configuration</a> within the cluster <code>settings</code> API, you can define any level of logging for any of the realms.
Let's take a look at an example with the <a href="/guide/en/elasticsearch/reference/7.4/configuring-ldap-realm.html">LDAP realm</a>. If the user authentication or authorization is failing, you can set up the following logging level to get more information about it:
<pre class="prettyprint">PUT /_cluster/settings 
{ 
 "transient": { 
 "logger.org.elasticsearch.xpack.security.authc.ldap":"DEBUG", 
 "logger.org.elasticsearch.xpack.security.authz":"DEBUG" 
 } 
}
</pre>This will increase the logging level for the LDAP package from default to DEBUG. You can also enable TRACE logging to get even more information such as every LDAP call done to the server and the response from it. You can find the package name for enabling on each specific REALM using our <a href="https://github.com/elastic/elasticsearch/blob/master/x-pack/plugin/security/src/main/java/org/elasticsearch/xpack/security/authc">GitHub repository</a>. For example, here is the <a href="https://github.com/elastic/elasticsearch/blob/master/x-pack/plugin/security/src/main/java/org/elasticsearch/xpack/security/authc/ldap/LdapRealm.java">code for the LDAP realm</a>. Using the first line of code you can see the package to be used for the logging level:
<pre class="prettyprint">package org.elasticsearch.xpack.security.authc.ldap;
</pre>Once this is enabled, you will get a lot of log lines, such as the following example:
<pre class="prettyprint">[2019-09-12T08:41:20,628][DEBUG][o.e.x.s.a.l.LdapRealm ] [xxxxxxx] user not found in cache, proceeding with normal authentication
[2019-09-12T08:41:21,180][TRACE][o.e.x.s.a.l.s.LdapUtils ] LDAP Search SearchRequest(baseDN='dc=xxx,dc=xxxx,dc=domain,dc=com', scope=SUB, deref=NEVER, sizeLimit=0, timeLimit=5, filter='(cn=vchatzig)', attrs={1.1}) =&gt; SearchResult(resultCode=0 (success), messageID=2, entriesReturned=1, referencesReturned=0) ([SearchResultEntry(dn='cn=xxxxxxx,ou=xxxxx,dc=intc,dc=xxxx,dc=domain,dc=com', messageID=2, attributes={}, controls={})])
</pre>Here we can see the LDAP realm trying to get the user from the cache (which could be out of date), and then doing the <code>LDAP Search</code> request to the server using the specific configurations we have provided. This is just an example of many log lines you will get.
<table style="background: #FFFFD2;">
<tbody>
<tr>
	<td>
		Note: Enabling logging is great for diagnostics, but not for performance. We recommend setting the logging level back to its default value once you know everything is working, using the following:
		
		<pre class="prettyprint">PUT /_cluster/settings 
{ 
 "transient": { 
 "logger.org.elasticsearch.xpack.security.authc.ldap": null, 
 "logger.org.elasticsearch.xpack.security.authz": null 
 } 
}
 </pre>
	</td>
</tr>
</tbody>
</table>Please note that some of these packages could change across versions, so we recommend that you double-check the documentation links for your specific Elasticsearch version you are using, along with the GitHub classes links for that specific release.
<h3>Common SAML issues</h3>The SAML realm requires an Identity Provider (such as Okta or Auth0) and a web application (Kibana is the default) that, together with Elasticsearch, acts as the Service Provider. Some of the issues we commonly see are:
<ul>
	<li>Incorrect Assertion Consumer Service URL setting in the Identity Provider (IdP) configuration. The endpoint that you usually need to configure at the Identity Provider is https://kibana.xxxx.com/api/security/v1/saml</li>
	<li>The metadata of the Identity Provider is not accessible through internet. The value of idp.metadata.path configuration in Elasticsearch either on-premise or in Elastic Cloud should be accessible through the network where Elasticsearch/Kibana are running. If that’s not the case, you need to download the metadata from a host that can access it, or ask an IdP administrator for the metadata and add it as a local file in Elasticsearch and reference it. The error shown will be similar to the following:
	<pre class="prettyprint">Caused by: net.shibboleth.utilities.java.support.resolver.ResolverException: net.shibboleth.utilities.java.support.resolver.ResolverException: Non-ok status code 404 returned from remote metadata source https://xxxx.xxxx/xxx/yyyyyyyy/sso/saml/metadata
 </pre></li>
	<li>You can get authenticated but the user is unable to open Kibana. For this we usually recommend double-checking the role mappings for the users to verify that at least one of their roles has <a href="/guide/en/kibana/7.4/xpack-security-authorization.html">access to Kibana</a>.</li>
</ul>The Identity Provider configuration is as important as the Elasticsearch/Kibana configuration, and we should always look for typos and mismatches between the expected and configured settings. Although each specific IdP has his own settings, we recommend to take a closer look at the <a href="/guide/en/elasticsearch/reference/7.3/saml-guide-idp.html">documentation</a> for each specific setting needed.
For more common issues and troubleshooting steps for SAML you can visit our <a href="/guide/en/elasticsearch/reference/current/trb-security-saml.html">troubleshooting documentation</a> that is maintained with each release.
<h2>Wrapping up</h2>Authentication in Elasticsearch is a very easy thing to set up once we understand all the concepts behind it. Also, getting an understanding around what things are working, what are not, and why can sometimes be hard but in this post we covered many options to gain a closer look. You can use our documentation to <a href="/blog/getting-started-with-elasticsearch-security">get started with security in Elasticsearch</a>, or use our <a href="/guide/en/elasticsearch/reference/current/security-troubleshooting.html">troubleshooting guide</a> for security in our documentation. Check out our Senior Product Manager for Elastic Security explaining in detail <a href="https://www.youtube.com/watch?v=nMh1HWWe6B4&feature=youtu.be">the getting started experience</a>.
We also recommend double-checking our <a href="/subscriptions">Subscriptions page</a> to learn more around features included in each level. Finally, if you have any specific questions you can reach out to your Support rep or use our <a href="https://discuss.elastic.co">Discuss forums</a>.
Happy security!

auth-blog-banner.jpg

auth-blog-thumb.jpg

Demystifying authentication and authorization in Elasticsearch

Ingest Node provides a quick and easy way to index Comma Separated Value (CSV) files in elasticsearch.

The idea of this article is to go over the capabilities of some of the features of <a href="https://www.elastic.co/guide/en/elasticsearch/reference/master/ingest.html">Ingest node</a>,&nbsp; which will be combined to parse a Comma Separated Value (CSV) file. We will go over what is an Ingest Node, what type of operations one can perform, and show a specific example starting from scratch to parse and display CSV data using the Elasticsearch and Kibana. For that we will use an <a href="https://data.ny.gov/Transportation/NYC-Transit-Subway-Entrance-And-Exit-Data/i9wp-a4ja">open catalog</a> of community issues from New York, NY. This CSV file was updated in October 2015 and It consists of <a href="https://data.ny.gov/api/assets/6D60FF9E-9143-40CF-8FC9-9D237C6AC864?download=true">32 fields</a> which draws the complete railway station entrances and exits lists. The goal will be to use the Ingest feature of Elasticsearch in a cluster on&nbsp;<a href="https://cloud.elastic.co">Elastic Cloud</a> to parse the data into a structured json, index the data, and use Kibana to build a Map of New York City that includes all this railway stations.
 
We will use the Ingest feature from Elasticsearch instead of Logstash as a way to remove the need of extra software/architecture setup for a simple problem that can be solved just with Elasticsearch.
 
With all this in place, we will be able to visualize the data and answer some questions such as “Where can we find a station with Elevators?”, “Where are most of the Stations located?”, “Which is the most dense area?” among others. Our data will come from a text file, and will turn into insights.
 
<h2 dir="ltr" style="line-height:1.38;margin-top:18pt;margin-bottom:6pt;">Cluster Setup</h2> 
To start, we are going to use a small Elastic Cloud cluster with 2GB of Memory and 48GB of Disk. We will download the CSV file with this data&nbsp;from the Export to CSV feature included in the following website: <a href="https://data.ny.gov/Transportation/NYC-Transit-Subway-Entrance-And-Exit-Data/i9wp-a4ja/data"></a><a href="https://data.ny.gov/Transportation/NYC-Transit-Subway-Entrance-And-Exit-Data/i9wp-a4ja/data">https://data.ny.gov/NYC-Transit-Subway</a>. 
 
We will use a Linux script that is composed with a simple loop to iterate through the CSV lines and send it to our cluster on Elastic Cloud. Elastic Cloud will give us an endpoint for our Elasticsearch instance. In addition to this, we need to enable Kibana to use the Developer tools and also to build the Dashboard.
 
In order to be able to parse the file, we need to replace all double quotes with single quotes and delete the first line of the file (header) before processing it. This can be done with your prefered&nbsp;tool. Each entry should look like (please note the single quotes):
 
<pre class="prettyprint">BMT,4 Avenue,25th St,40.660397,-73.998091,R,,,,,,,,,,,Stair,YES,,YES,FULL,,FALSE,,FALSE,4th Ave,25th St,SE,40.660323,-73.997952,'(40.660397, -73.998091)','(40.660323, -73.997952)'
</pre> 
<h2 dir="ltr" style="line-height:1.38;margin-top:18pt;margin-bottom:6pt;">Parsing to json</h2> 
In order to be able to search and build dashboards, we need to parse the plain text into a structured json. For this, we will send the data to elasticsearch using the following script:
 
<pre class="prettyprint">while read f1
do 
 curl -XPOST 'https://XXXXXXX.us-east-1.aws.found.io:9243/subway_info_v1/station' -H "Content-Type: application/json" -u elastic:XXXX -d "{ \"station\": \"$f1\" }"
done &lt; NYC_Transit_Subway_Entrance_And_Exit_Data.csv
</pre> 
This script will read the file (named NYC_Transit_Subway_Entrance_And_Exit_Data.csv), line by line, and send the following initial json to Elasticsearch:
 
<pre class="prettyprint">{
 "station": "BMT,4 Avenue,59th St,40.641362,-74.017881,N,R,,,,,,,,,,Stair,YES,,YES,NONE,,FALSE,,TRUE,4th Ave,60th St,SW,40.640682,-74.018857,'(40.641362, -74.017881)','(40.640682, -74.018857)'"
}
</pre> 
The json contains just one field “station”, with a single line. Once the json is sent to Elasticsearch, we need to take the information and break the station field into multiple fields, each containing a single value of the unstructured line. It is highly recommended to use the <a href="https://www.elastic.co/guide/en/elasticsearch/reference/master/simulate-pipeline-api.html">simulate API</a>&nbsp;from Elasticsearch to play and develop the pipeline before actually creating it. Initially you should just start with a document and an empty pipeline:
 
<pre class="prettyprint"> 
POST _ingest/pipeline/_simulate
{
 "pipeline": {},
 "docs": [
 {
 "station": "BMT,4 Avenue,59th St,40.641362,-74.017881,N,R,,,,,,,,,,Stair,YES,,YES,NONE,,FALSE,,TRUE,4th Ave,60th St,SW,40.640682,-74.018857,'(40.641362, -74.017881)','(40.640682, -74.018857)'"
 }
 ]
}
</pre> 
There are many processors available to process the lines, so you should consider all of them to choose which to use. In this case we will simply use the <a href="https://www.elastic.co/guide/en/elasticsearch/reference/master/grok-processor.html">Grok Processor</a> which allows us to easily define a simple pattern for our lines. The idea of the following processor is to parse using grok and finally remove the field containing the full line:
 
<pre class="prettyprint">POST _ingest/pipeline/_simulate
{
 "pipeline": {
 "description": "Parsing the NYC stations",
 "processors": [
 {
 "grok": {
 "field": "station",
 "patterns": [
 "%{WORD:division},%{DATA:line},%{DATA:station_name},%{NUMBER:location.lat},%{NUMBER:location.lon},%{DATA},%{DATA},%{DATA},%{DATA},%{DATA},%{DATA},%{DATA},%{DATA},%{DATA},%{DATA},%{DATA},%{DATA:entrance_type},%{DATA:entry},%{DATA:exit_only},%{DATA:vending}"
 ]
 }
 },
 {
 "remove": {
 "field": "station"
 }
 }
 ]
 },
 "docs": [
 {
 "_index": "subway_info",
 "_type": "station",
 "_id": "AVvJZVQEBr2flFKzrrkr",
 "_score": 1,
 "_source": {
 "station": "BMT,4 Avenue,53rd St,40.645069,-74.014034,R,,,,,,,,,,,Stair,NO,Yes,NO,NONE,,FALSE,,TRUE,4th Ave,52nd St,NW,40.645619,-74.013688,'(40.645069, -74.014034)','(40.645619, -74.013688)'"
 }
 }
 ]
}
</pre> 
This pipeline will produce the following document with structured fields and ready to be indexed into Elasticsearch:
 
<pre class="prettyprint">{
 "division": "BMT",
 "station_name": "53rd St",
 "entry": "NO",
 "exit_only": "Yes",
 "line": "4 Avenue",
 "vending": "",
 "location": {
 "lon": "-74.014034",
 "lat": "40.645069"
 },
 "entrance_type": "Stair"
}
</pre> 
<h2 dir="ltr" style="line-height:1.38;margin-top:18pt;margin-bottom:6pt;">Indexing the documents</h2> 
Before indexing the document, we need to create an index template that will match the index name that we are going to use. In order to properly be able to do document filtering and geo-location styled queries and dashboard, it is required to setup certain field types which require a specific mapping definition. This can be done by explicitly creating the index in advance, but it is better to use an index template to make it flexible. New indices following this name will be created, automatically, with all this settings and mappings. The name of the index that we will use in this case, will start with subway_info followed by the version (v1 for example). For this, we should use the following template that will match such index name: 
 
<pre class="prettyprint">PUT _template/nyc_template
{
 "template": "subway_info*",
 "settings": {
 "number_of_shards": 1
 },
 "mappings": {
 "station": {
 "properties": {
 "location": {
 "type": "geo_point"
 },
 "entry": {
 "type": "keyword"
 },
 "exit_only": {
 "type": "keyword"
 }
 }
 }
 }
}
</pre> 
After creating the template, we need to take the ingest pipeline from the simulate step and put the pipeline itself into Elasticsearch, so that we can invoke it at indexing time. The command to put the ingest pipeline should look like the following: 
 
<pre class="prettyprint">PUT _ingest/pipeline/parse_nyc_csv
{
 "description": "Parsing the NYC stations",
 "processors": [
 {
 "grok": {
 "field": "station",
 "patterns": [
 "%{WORD:division},%{DATA:line},%{DATA:station_name},%{NUMBER:location.lat},%{NUMBER:location.lon},%{DATA},%{DATA},%{DATA},%{DATA},%{DATA},%{DATA},%{DATA},%{DATA},%{DATA},%{DATA},%{DATA},%{DATA:entrance_type},%{DATA:entry},%{DATA:exit_only},%{DATA:vending}"
 ]
 }
 },
 {
 "remove": {
 "field": "station"
 }
 }
 ]
}
</pre> 
Now, we just need to read the file and sent the data to Elasticsearch using the &nbsp;pipeline we created. The pipeline definition is configured in the URL: 
 
<pre class="prettyprint">while read f1
do 
 curl -XPOST 'https://XXXXXXX.us-east-1.aws.found.io:9243/subway_info_v1/station?pipeline=parse_nyc_csv' -H "Content-Type: application/json" -u elastic:XXXX -d "{ \"station\": \"$f1\" }" 
done &lt; NYC_Transit_Subway_Entrance_And_Exit_Data.csv
</pre> 
Let it run for a while and it will create each document one at a time. Note that here we are using the index API and not the bulk API. In order to make it faster and more robust for production use cases, we recommend you use the bulk API to index these documents. At the end of the ingest process you will end up with 1868 stations in the subway_info_v1 index.
<h2 dir="ltr" style="line-height:1.38;margin-top:18pt;margin-bottom:6pt;">Building the Dashboard</h2> 
In order to build the dashboard, first we need to add the index pattern to Kibana. For that, just go to Management and add the index pattern subway_info_v1. You should unclick the “Index contains time-based events” option as it is not time series data (our data doesn’t contain a date-time field). 
 
<img src="https://api.contentstack.io/v2/uploads/590b838cc3fe4f9f070b994e/download?uid=blt8e7ce24a04bdca86" data-sys-asset-uid="blt8e7ce24a04bdca86" alt="img1.png" style="display: block; margin: auto;"> 

 
After this we can go and first create our Tile Map visualization showing all the subway stations that we have in this dataset for New York City. For that, we need to go to Visualizations and choose the Tile Map type. By choosing the geolocation field and the cardinality of the station name, we get an easy and quick view of the existing stations.
 
<img src="https://api.contentstack.io/v2/uploads/590b83bf1b2e499507278ab9/download?uid=blt0101c72c2df2ddce" data-sys-asset-uid="blt0101c72c2df2ddce" alt="img2.png" style="display: block; margin: auto;">
 
By adding some additional Visualizations, such as a Saved Search and type of entrance, we can easily build a tool to search for specific Subway stations in New York city.
 
<img src="https://api.contentstack.io/v2/uploads/590b83e5f839241308679ee9/download?uid=bltbb23847f8531a3cc" data-sys-asset-uid="bltbb23847f8531a3cc" alt="img3.png" style="display: block; margin: auto;">
 
As we can see here, downtown Manhattan is the area with the most Subway Stations. With Kibana you can select a specific rectangle and also filter on a value such as “Elevator”. For example from the 356 Stations, around 200 are located in Manhattan, and 36 stations have elevators.
 
<img src="https://api.contentstack.io/v2/uploads/590b8409e5abb99807b705ad/download?uid=bltadac7802ffc7c82e" data-sys-asset-uid="bltadac7802ffc7c82e" alt="img4.png" style="display: block; margin: auto;">
 
<h2 dir="ltr" style="line-height:1.38;margin-top:18pt;margin-bottom:6pt;">Try it out with your Dataset!</h2>As you can see, starting from scratch with a CSV file is very simple. You can run a trial on an Elasticsearch Cloud (<a href="http://cloud.elastic.co">cloud.elastic.co</a>) instance and ingest your data with just little work. Ingest Node is the tool that will allow you to convert unstructured data into structured data, and finally be able to create powerful visualizations with Kibana. Going from Zero to Hero in Elasticsearch, is as easy as spinning up a cluster. Try it for yourself!
<div> 
</div>

kibana-elasticsearch-geodata-indexing-csv-fullbleed.jpg

kibana-elasticsearch-ingest-csv-geodata-thumbnail.jpg

Indexing your CSV files with Elasticsearch Ingest Node

a-world-wihout-stopwords.jpg

analysis-process-indexing.png

Phrase Queries: a world without Stopwords

To have a better understanding of the challenges we may deal&nbsp;with when using Elasticsearch in a virtualized environment, we need&nbsp;to change the focus from&nbsp;conventional hardware problems&nbsp;to a more complex view. The purpose of this article is to uncover&nbsp;some common issues you might experience using Elasticsearch in virtual environments.<h2>A Brief History</h2>Way before Elasticsearch appeared, the concept of virtualization was taking its place as a first class citizen in computing. Virtualization refers to the act of creating a virtual (not an actual) version of something, including, among others, virtual-computer hardware platforms, operating systems, storage devices, or computer network resources. Virtualization was born in the late 1960s and early 1970s, when IBM created the CP-40/CMS (Conversational Monitor System)&nbsp;as a method of logically dividing the system resources provided by mainframe computers between different applications. Afterwards, the meaning of the term&nbsp;broadened to&nbsp;what currently is: full virtual machine (VM) implementations and control of processing, network and&nbsp;memory,&nbsp;all working&nbsp;together seamlessly&nbsp;in the cloud. <h2>Existing Platforms</h2>There are various existing platforms to handle Elasticsearch in virtual environments, all of which are different between them. Generally, the three main platforms we see used for Elasticsearch are:<ul><li dir="ltr">Cloud Solutions</li></ul><ul><ul><li dir="ltr"><a href="http://aws.amazon.com/ec2/">Amazon (Elastic Compute Cloud) EC2</a>: Amazon service with a simple web interface that allows you to create and configure virtual machine on the cloud.</li></ul></ul><ul><ul><li dir="ltr"><a href="http://azure.microsoft.com/">Microsoft Azure</a>: Microsoft's competitive alternative to Amazon EC2, which offers similar features and functionality.</li></ul></ul><ul><li dir="ltr">On-Premise&nbsp;Solution</li></ul><ul><ul><li dir="ltr"><a href="https://www.vmware.com/products/vsphere/features/esxi-hypervisor">VMware vSphere/ESX</a>: Create full virtual clusters installing on top of physical servers and making VM's partitions on it.</li></ul></ul>Finally, as a different way to handle our Elasticsearch virtualized infrastructure, <a href="http://found.no">Found by Elastic</a>&nbsp;is a hosted and fully managed Elasticsearch Software&nbsp;as a Service (SaaS). Found provides a fast, scalable, reliable and easy to operate search service hosted for you in the cloud.<h2>The Architecture</h2>As an example of how complex a virtualized architecture can be, and all the points we have to understand&nbsp;to manage Elasticsearch on a virtual environment, we can take a brief look into VMware's vSphere architecture. VMware vSphere is used to transform entire datacenters into a single&nbsp;cloud computer infrastructure, virtualizing and aggregating the main physical hardware resource across multiple systems and providing&nbsp;virtual resources to the datacenter. VMware vSphere consists of&nbsp;multiple component layers such as: <ul><li dir="ltr">Infrastructure Services - VMware vCompute, VMware vStorage and VMware vNetwork. VMware ESX and ESXi are both physical servers that abstract away from the&nbsp;processor, manage&nbsp;storage in virtual environments and simplify&nbsp;networking.</li></ul><ul><li dir="ltr">Application Services -&nbsp;Ensure&nbsp;availability, security and scalability for applications.</li></ul><ul><li dir="ltr">VMware vCenter Server - A single application&nbsp;that&nbsp;takes control of the datacenter, providing access control, performance monitoring and configurations.</li></ul><ul><li dir="ltr">Clients - Different types of clients to access VMware vSphere datacenter, where we can create and access an Elasticsearch node.</li></ul>Although the&nbsp;architecture is complex, no matter which virtualization solution we use,&nbsp;we will have tools that makes it very easy to manage entire datacenter or clusters. Those tools can help us to easily allocate storage and networking to the physical nodes, parcel out resource&nbsp;allocation (CPU, memory, disk and network bandwidth) as needed, monitor datacenter status, and more. The tools&nbsp;will allow us to configure and setup Elasticsearch in a virtual environment exactly as required&nbsp;depending on our needs. Regardless, we need to take care around&nbsp;some issues that can crop up&nbsp;with CPU, memory and disk utilization. <h2>Handling Resources</h2>There are various ways to achieve the goal of running&nbsp;Elasticsearch in a virtualized environment. Each platform and solution, whether is cloud-based or not, has his own complexity and difficulty for configuring and running. Handling resources is the key area&nbsp;for achieving&nbsp;success. <h3>CPU</h3>Every virtualization solution has limits regarding CPU usage. A physical processor core can support up to 32 virtual CPUs (vCPU) in both vSphere 6 and Azure, and 36 vCPU in Amazon EC2. As we increase CPU allocation on cloud providers, we will increase the cost for each instance. Elasticsearch uses Java, so we will need to handle a Java Virtual Machine (JVM) within our virtual environment. A good approach for JVM's is to have a minimum of two CPU's, one to handle garbage collection and JVM administration, and the other to handle the application processing. A good way to handle CPU usage is to monitor CPU utilization inside the VM using <a href="https://www.elastic.co/products/marvel">Marvel</a>. If Elasticsearch is using a lot of &nbsp;CPU resources inside the VM, it may be worth considering increasing the number of available vCPUs. <h3>Memory</h3>As well as CPU limits, there are limits for the amount of RAM we can allocate on a host depending the provider: up to 6 TB on vSphere, 244 GB on Amazon EC2, and 112 GB on Azure. As we increase memory usage, we will generally see&nbsp;increase in costs. CPU and disk usage can&nbsp;be affected by reaching memory limits. You might want to watch and monitor the Host and VM status with Marvel, to find whether you need to do something in order to decrease memory usage, such&nbsp;&nbsp;as&nbsp;refactoring Elasticsearch queries or increasing&nbsp;the amount of memory on the host. Java objects, methods, thread stacks and others, reside in Java heap. The amount of memory given to the heap will ensure us&nbsp;good — or bad — behavior of our Elasticsearch cluster. When the heap starts to fill, the Java garbage collector will start running. It is a best practice to allocate half of the total amount of memory for the heap. In addition, we have detailed information in our documentation on <a href="http://www.elastic.co/guide/en/elasticsearch/guide/current/_limiting_memory_usage.html">how to limit memory usage</a> . <h3>Disk</h3>Disk utilization is similar on a host and a VM. We need to eliminate&nbsp;disk contention as we do in any environment. If a set of disks in the host is being overused, meaning that the average I/O is close to 100%, we might see an impact in all the virtual environments that are using the same disks.&nbsp;Disk resources can also be impacted by "noisy neighbors", which are generally larger VMs running on or against the same hardware, thereby consuming resources in negative and surprising ways. <a href="http://www.elastic.co/guide/en/elasticsearch/guide/master/backing-up-your-cluster.html">Backing up</a> your Elasticsearch cluster, or <a href="http://www.elastic.co/guide/en/elasticsearch/reference/current/modules-snapshots.html">creating snapshots</a> for individual indices as well as entire clusters, is incredibly important!&nbsp;By making backups from the VM, we can ensure&nbsp;that&nbsp;we have a starting&nbsp;point to continue from in the case of failure. Creating snapshots or backups from VMs has some cost and may have an impact in the VM response time, so we may also impact&nbsp;Elasticsearch's responsiveness by doing such operations. Plus, it is just good practice&nbsp;to have a&nbsp;Backup and Snapshot policy for your clusters. Elasticsearch disk usage depends on each use case. We recommend doing stress and performance tests on the server in order to understand the amount of disk we need to allocate in order to make the cluster work well. When it comes to&nbsp;CPU and Memory, some cloud solutions can become pricey as you increase the disk allocation.<h3>Network</h3>Configuring the network is usually straight forward. There are plenty of possible configurations depending on which cloud&nbsp;provider you choose and what your needs are. You&nbsp;can share the network with the host, or create an independent network to use on your VM.&nbsp;You make consider creating a Virtualized Private Network (VPN) to isolate the cluster, as well as to secure it.<h2>The Perils of Virtualization</h2>In addition to the areas outlined above, there are a few other places where we can run into trouble running Elasticsearch in a virtualized environment. As an example, we can see one of the latest <a href="https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1317811">bugs</a> fixed on Ubuntu. A simple bug on the Ubuntu kernel (version 3.13) was causing a failure in the transport connection thread on EC2 when the&nbsp;network'&nbsp;load increased. Consequently, Elasticsearch indexing, query operations and administrative commands started to fail on EC2 instances running Ubuntu. The problem was caused by a combination of <a href="http://en.wikipedia.org/wiki/Gather-scatter_%28vector_addressing%29">Gather-scatter</a> and the maximum transmission unit limit on the network interfaces. The solution was either to update Ubuntu's kernel version and restart the EC2 instance, or disable gather-scatter. Another example is the problem that we might encounter while working in a cluster with limited resources on the VMs and losing&nbsp;one of the nodes. Shards that were allocated in that specific node will be relocated to another node, without executing any process to see if the new node contains enough resources to handle the&nbsp;new shards. In order to limit this problem, we can use <a href="http://www.elastic.co/guide/en/elasticsearch/reference/current/modules-cluster.html#forced-awareness">forced awareness</a>. Forced awareness allows us to&nbsp;force allocate new shards in specific zones that we define in the configuration. Finally, as it is too common to have more than one VM within the same hardware, to avoid the risk of losing data in a virtualized environment we can use <a href="http://www.elastic.co/guide/en/elasticsearch/reference/current/modules-cluster.html#allocation-awareness">shard allocation awareness</a>&nbsp;to prevent primary&nbsp;and replica&nbsp;shards to be located&nbsp;on the same hardware, rack or zone. We can then force each replica shard to be allocated in another VM that is not on&nbsp;the same hardware as&nbsp;the primary one.<h2>Conclusion</h2>There are many different possibilities for using&nbsp;Elasticsearch in a virtualized environment. Choosing which is the best will involve analyzing and deciding on some technical and financial tradeoffs. Consider&nbsp;the best choice for your solution: you want to&nbsp;have a configuration that allows you to use all the resources available not only effectively, but also efficiently.

Virtualization and Elasticsearch: Best Practices

Start free trial

Blog Footer CTA

Sign up for Elastic Cloud free trial

gabriel-moskovicz.jpg

Principal Solutions Architect

Gabriel Moskovicz

Register to Attend

Register to Watch

Надсилаючи цю форму ви погоджуєтеся з <a href="/legal/terms-of-use">Умовами надання послуг Elastic</a>. Ваші особисті дані будуть опрацьовані відповідно з <a href="/legal/privacy-statement">Положеннями про конфіденційність Elastic</a>.

Newsletter GDPR Text - Ukrainian

By submitting you acknowledge that you've read and agree to our <a href="/legal/elastic-cloud-account-terms" target="_self">Terms of Service</a>, and you agree to receive the <a href="/legal/privacy-statement#how-we-use-the-information" target="_self">selected communications</a> at the contact details you provided above. See <a href="/legal/privacy-statement/" target="_self">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.

Communication Preferences - ONLY for /communication-preferences

By submitting you agree to <a href="/agreements/global/cloud-services-monthly" rel="nofollow">Elastic Cloud Terms of Service</a>. Your personal data will be processed in accordance with <a href="/legal/privacy-statement">Elastic's Privacy Statement</a>.

Cloud Trial GDPR Text

Intentionally Blank

By submitting you acknowledge that you've read and agree to our <a href="/legal/serverless-preview-terms" target="_blank">Terms of Service</a>, and that Elastic may <a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">contact you</a> about our related products and services, using the details you provide above. See <a href="/legal/privacy-statement/" target="_blank">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.

Serverless GDPR Text

提交即表示您確認已詳閱並同意我們的<a href="/legal/elastic-cloud-account-terms" target="_blank">服務條款</a>，且同意 Elastic 使用您在上方提供的個人資料<a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">與您聯絡</a>，為您提供相關產品和服務的資訊。如需詳細說明或隨時想停止收到產品和服務資訊，請參閱 <a href="/legal/privacy-statement/" target="_blank">Elastic 的隱私聲明</a>。

Newsletter GDPR Text - TW

By submitting you acknowledge that you've read and agree to our <a href="/legal/elastic-cloud-account-terms" target="_blank">Terms of Service</a>, and that Elastic may <a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">contact you</a> about our related products and services, using the details you provide above. See <a href="/legal/privacy-statement/" target="_blank">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.By submitting this form, I agree to Elastic sharing my contact information with Google Cloud. I also agree to Elastic giving my information to the vendor for the purpose of shipping my food package.

GDPR Text + Google Cloud

Inviando la registrazione (1) esprimi il consenso ai <a href="/legal/terms-of-use">termini e condizioni di Elastic</a> e <a href="/legal/privacy-statement">all’informativa sulla privacy</a>, nonché (2) a ricevere saltuarie e-mail.

Newsletter GDPR Text - Italian

By submitting you acknowledge that you've read and agree to our <a href="/legal/terms-of-use" target="_blank">Terms of Service</a>, and that Elastic may <a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">contact you</a> about our related products and services, using the details you provide above. See <a href="/legal/privacy-statement/" target="_blank">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.

Newsletter GDPR Text

Load more

Overview

Can't make it? Register and we'll send you the recording. You'll also receive an email with related content

Can't make it? Register and we'll send you the recording. You'll also receive an email with related content.

You'll also receive an email with related content.

You'll also receive an email with related content

Speakers

Close

See more insights

The content on this page is not available in the selected language. As Elastic grows globally, we continue to support content in multiple languages.

Author

Articles by Gabriel Moskovicz

Principal Solutions Architect, Elastic

Demystifying authentication and authorization in Elasticsearch

Indexing your CSV files with Elasticsearch Ingest Node

Phrase Queries: a world without Stopwords

Virtualization and Elasticsearch: Best Practices