A Dive into the Elasticsearch Storage

<p><strong>UPDATE:</strong> This article refers to our hosted Elasticsearch offering by an older name, Found. Please note that Found is now known as Elastic Cloud.</p><p>When starting using Elasticsearch, it's easy to get confused about all the different ways to connect to Elasticsearch and why one of them should be preferred over the other. In this article we'll provide an overview of the different client types available and give some pointers on when one should be chosen over another.
</p><section id="introduction" class="level2">
<h2><a href="#introduction">Introduction</a></h2>
<p>Having lots of options when picking a client can be both great and confusing. It’s great because it means that there’s likely to be a client out there that meets your requirements, but it’s confusing because there’s a lot of clients to choose from and there’s a lot of overlap between them - and a decision needs to be made.
</p>
<p>Many languages have official clients written and supported by Elasticsearch, which is great as a default go-to solution, but knowing the options available and their different characteristics makes the choice a lot easier.
</p>
</section><section id="different-client-types" class="level2">
<h2><a href="#different-client-types">Different Client Types</a></h2>
<p>By default, Elasticsearch comes with support for two protocols:
</p>
<ul>
	<li><strong>HTTP</strong>: A <a href="http://en.wikipedia.org/wiki/Representational_state_transfer">RESTful</a> API</li>
	<li><strong>Native Elasticsearch binary protocol</strong>: a custom protocol developed by Elasticsearch for inter-node communiaction.</li>
</ul>
<p>It’s also possible to extend Elasticsearch with support for different protocols via plugins. There there are a few official plugins around.
</p>
<p>Using the native protocol from anything other than Java is not recommended, as it would entail implementing a lot of custom serialization.
</p>
</section><section id="transport-client" class="level2">
<h2><a href="#transport-client">Transport Client</a></h2>
<p>The Transport client is one of the native ways to connect to Elasticsearch. It is part of the official Elasticsearch distribution and thus requires your client to be written in Java (or at least run on the JVM) as well.
</p>
<p>It’s very fast and runs natively on the JVM. The serialization is effective and there’s little to no overhead in messages and operations sent to/from your Elasticsearch instances. It requires keeping the Elasticsearch server and client versions somewhat synchronized. Prior to Elasticsearch 1.0 the exact same versions would be required, but newer versions (1.0 and later) support interacting between versions. It’s also beneficial to run the same JVM update version on both the clients and the servers due to exception serializing and other potential minor differences between updates.
</p>
<p>There’s currently no support for encryption or authentication, but <a href="/what-is/elastic-stack-security">Shield</a> is announced to cover these needs. For using the Transport client on Found.no hosted clusters, you can use <a href="/guide/en/found/current/java-transport.html">our custom transport module</a> which takes care of encryption, authentication and keep-alives.
</p>
</section><section id="node-client" class="level2">
<h2><a href="#node-client">Node Client</a></h2>
<p>The Node client is very similar to the Transport client: it’s part of the official Elasticsearch distribution, requires your client run Java and so on, but there’s some significant differences as well.
</p>
<p>Where the cluster is largely indifferent about whether or not a Transport client has connected to one of the nodes in your cluster, a node client is considered part of your cluster. This means that the presence of the node client is stored in the cluster state, and that all the other nodes in the cluster will attempt to establish a <a href="/blog/found-elasticsearch-networking/#node-clients">few tcp connections</a> back to the client. This may be a significant drawback if your cluster is large or you’re using several clients.
</p>
<p>This may seem a bit exessive, but it’s currently required in order to enable the server nodes to propagate changes to the cluster state to the client. The end result of this is that the node client always has an up-to-date cluster state and a connection to every other node in the Elasticsearch cluster, which enables it to perform the routing of operations locally, be the coordinator of its own requests and so on. This skips a network jump for each and every request and results in less work for the remaining nodes in the cluster.
</p>
<p>Having the cluster state locally also costs a bit of memory, as well as CPU resources in order to perform incoming updates to the cluster state. Additionally, the Node client loads quite a few more <a href="/blog/found-elasticsearch-internals/">Modules from Elasticsearch</a>, which increases the memory footprint.
</p>
</section><section id="http-clients" class="level2">
<h2><a href="#http-clients">HTTP Clients</a></h2>
<p>HTTP is well supported in most programming languages, and this is the most common way to connect to Elasticsearch. If you’re going with HTTP, there are still one important choices to make: use an existing Elasticsearch HTTP-based library or simply just create a small wrapper for the operations you need using the HTTP client if your choice.
</p>
<p>Since HTTP is a general purpose protocol and supports a wide variety of use cases, a few important things need to be implemented by the client: connection pooling and <a href="http://en.wikipedia.org/wiki/Keepalive#HTTP_Keepalive">keep-alives</a>. Connection pooling is required in order to avoid having to pay the TCP connection establishment cost for each request. This is even more important if it uses HTTPS, which comes with an additional encryption handshake cost. Connection pooling often requires keep-alive support as well, since we would like to avoid connections being broken due to idling.
</p>
<p>While it might not be initially obvious that connection establishment actually is significant, consider that establishing a TCP connection requires a <a href="http://en.wikipedia.org/wiki/Transmission_Control_Protocol#Connection_establishment">three-way handshake</a>. Put simply, with a ping time of 50 milliseconds, it will take around 75 ms to establish a connection, in addition to the time it takes to acquire and release local resources (handling client ports, connection management, etc) and so on – this is without considering the time it takes to handle the request/response at both ends (e.g serialization). Without connection pooling, this time is added on top of each and every request. For HTTPS, which we recommend for security and privacy, the connection establishment overhead can sometimes be measured in seconds, which is even more noticeable. Considering the basic advice that end users’ response times have to be <a href="http://www.nngroup.com/articles/response-times-3-important-limits/">under 100 ms</a> in order to be observed as “instant”, even the non-encrypted overhead makes this limit almost impossible to stay within.
</p>
<p>The official (non-Java) clients written and supported by Elasticsearch all use HTTP under the hood to communicate with Elasticsearch. Our general recommendation is to use the official clients that wrap the HTTP API if possible because they take care of handling all these details.
</p>
<p>HTTP client implementations can be quite fast, and some of them even compete with the speed of the native protocol. The HTTP API of Elasticsearch is widely used and has considerable community support. The performance depends a lot on the client library however, and often needs to be configured or tweaked in order to be maximized.
</p>
</section><section id="other-protocols" class="level2">
<h2><a href="#other-protocols">Other Protocols</a></h2>
<p>As it’s possible to create a new client interface to Elasticsearch simply by writing a plugin, a lot of other protocols are supported. There are official plugins for using the <a href="http://memcached.org/">Memcached</a> and <a href="https://thrift.apache.org/">Apache Thrift</a> protocols this way. The degree of encryption supported depends on the client (for example, Thrift supports SSL, but requires some additional setup for both the client and the server).
</p>
<p>This system is quite flexible and you can conceivably use any protocol that supports the request/response pattern, but a significant drawback of using these protocols is that there are less users using them, so it’s harder to get community support. There’s also the risk that the lesser used protocols may become deprecated without any guarantee of future support, so users should only use these protocols if this risk is acceptable and the protocol and its corner cases are well known in-house. For these reasons we generally don’t recommend taking this approach.
</p>
</section><section id="conclusion" class="level2">
<h2><a href="#conclusion">Conclusion</a></h2>
<p>It’s easy to spend a lot of time figuring out the differences between the myriad of protocols and clients to use with Elasticsearch, but the choice is actually pretty simple: if possible, <em>use a high-performance HTTP client</em> that you are comfortable with or an <em>official language binding</em>.
</p>
<p>If you’re using Java, the Transport client should be chosen over the Node client unless the performance gain from using a Node client turns out to be large enough to warrant the additional network complexity. Use benchmarks to verify the performance gains.
</p>
<p>When using other non-Java JVM-based languages (e.g Scala, Clojure, Groovy, JRuby and so on), it might be worthwhile to further explore more language-native wrappers on top of either the Transport or Node clients. This may make the interaction with Elasticsearch a lot more natural in the source code. Be aware that the wrapper may not expose all the functionality of the underlying client and this could mean more poking around in order to get to it, so consider the tradeoffs compared to using the official clients here as well.
</p>
</section>

Interfacing with Elasticsearch: Picking a Client

<p><strong>UPDATE:</strong> This article refers to our hosted Elasticsearch offering by an older name, Found. Please note that Found is now known as Elastic Cloud.</p><h2>An Overview</h2><p>Raw documents from the source database or server may need some extra processing before being indexed in Elasticsearch. In this article, we consider a few different options for this processing.
</p><section id="introduction" class="level2">
<h2><a href="#introduction">Introduction</a></h2>
<p>When working with a full-text search engine such as Elasticsearch, the need to perform some kind of transformation of the incoming data before it’s indexed is likely to emerge. This transformation is often called “Document Processing”, and it is an important piece of functionality for many applications.
</p>
<p>Document processing can range from simply tagging documents with a pre-configured attribute, dynamically calculating one or more attributes based on document contents or complete document rewrites, both structurally and content-wise.
</p>
<p>Elasticsearch is open for several different ways to process documents (or document contents), and which method to use highly depends on the application and system requirements. In the following sections we’ll look at three specific methods of doing this: the mapping, as a custom plugin or using an external service.
</p>
</section><section id="what-is-document-processing" class="level2">
<h2><a href="#what-is-document-processing">What is Document Processing?</a></h2>
<p>Document processing is separate from text processing and is performed as a separate step <em>before</em> the text processing is performed. Where text processing works on <em>individual</em> fields, document processing is usually scoped around an entire document that would contain <em>multiple</em> fields.
</p>
<p>Usually, the text processing is performed on the resulting field data <em>after</em> the document has been transformed. While it’s possible to tag data as <em>already-analyzed</em>, and thus skipping the usual text processing, the details of this is outside the scope of this article.
</p>
<p>If you’re already using Elasticsearch, chances are that you’re actually doing <em>some</em> kind of document processing already, as the initial creation of the source documents on your application servers can certainly be considered a simple form of document processing. The initial fields are seeded and often this includes some <a href="http://en.wikipedia.org/wiki/Denormalization">de-normalization</a> by including some contextual information or metadata about the current document.
</p>
</section><section id="transform-field-using-script" class="level2">
<h2><a href="#transform-field-using-script">Transform Field Using Script</a></h2>
<p>A relatively unknown feature of the Elasticsearch mapping is the <a href="/guide/en/elasticsearch/reference/current/mapping-transform.html"><code>transform</code> field</a>, which runs a small script on the source document before indexing it. This ties into the <a href="/guide/en/elasticsearch/reference/current/modules-scripting.html">scripting module</a>, so scripts can either be provided inline, as part of the mapping, or pre-compiled and loaded when Elasticsearch initially bootstraps.
</p>
<p>
	<img src="https://api.contentstack.io/v2/assets/575e4c989985d58976376c37/download?uid=blt261430d41454c706?uid=blt261430d41454c706" data-sys-asset-uid="bltdd04f40b3e632509" alt="Transform field" style="max-width:100%;">
</p><figcaption>Transform field</figcaption>
<p>There are some limitations to be aware of: the transform field must be configured when the index is initially created, and it’s not possible to change or add transformed fields afterwards. This is different from most other field types in Elasticsearch, which usually allows for new fields to be added to an existing index – a part of the “schemaless” feature of Elasticsearch.
</p>
</section><section id="custom-plugin" class="level2">
<h2><a href="#custom-plugin">Custom Plugin</a></h2>
<p>If the use case calls for more elaborate processing than the <code>transform</code> field allows for, such as asynchronous processing or enriching of the data, you may want to consider writing a custom plugin.
</p>
<p>
	<img src="https://api.contentstack.io/v2/assets/575e4c983dc542cb38c08474/download?uid=bltea774a405bddecab?uid=bltea774a405bddecab" data-sys-asset-uid="blt59782facf8e74c8d" alt="Custom plugin" style="max-width:100%;">
</p><figcaption>Custom plugin</figcaption>
<p>If the processing is simple enough, you can get by with writing a plugin that hooks into the text processing of Elasticsearch by writing a custom analyzer. However, that analyzer has to work within the confines of the <a href="http://lucene.apache.org">Lucene</a> text processing system, which is limited by the environment it runs in: it is executed synchronously from an indexing thread, so it’s in a path that is likely to be highly performance critical.
</p>
<p>Not everything can be done in the context of an analyzer though. The most common example of this is asynchronous processing – or even things that <em>should</em> be performed asynchronously. Asynchronous processing may include things like using other (non-local) services or IO to provide context, for example consulting a separate database or service in order to add fields that are relevant to the document as part of the processing.
</p>
<p>For these cases, the processing has to be done before the document is actually handed off to Elasticsearch for indexing. It may still be accomplished within the Elasticsearch process by writing a custom plugin. For example, we can write a plugin that <a href="/blog/found-writing-a-plugin/#creating-the-http-handler">adds a custom endpoint</a> to Elasticsearch that pre-processes the incoming data before using the internal Elasticsearch Client interface to perform the actual indexing. At that level, doing asynchronous processing and leveraging other services is rather trivial.
</p>
<p>Doing document processing within Elasticsearch means that you tie up the scaling of the document processing with the scaling of Elasticsearch. The workloads of these two parts may differ a lot, and as such it may be beneficial to extract the document processing out into an external service. This also makes it easier to update the document processing pipeline without having to perform any changes to your Elasticsearch cluster. This is the main reason why we don’t recommend relying on using plugins such as the <a href="https://github.com/elasticsearch/elasticsearch-mapper-attachments">mapper-attachment</a> in production. While it’s great for testing and getting started – it takes almost no time to set up and enables indexing a whole host of file formats – it does not necessarily make for an efficient way to utilize the available resources. It also ties up memory on index-specific things rather than using it on speeding up searches, which is probably what you’d prefer.
</p>
</section><section id="external-systems-and-pipelines" class="level2">
<h2><a href="#external-systems-and-pipelines">External Systems and Pipelines</a></h2>
<p>So, the preferred way to go – with respect to both flexibility and scalability – is performing the document processing outside of Elasticsearch. This decouples both the scaling and upgrading of Elasticsearch from the document processing. Elasticsearch and Lucene are well known to benefit a lot from <a href="http://blog.trifork.com/2011/04/01/gimme-all-resources-you-have-i-can-use-them/">using a lot of resources</a>, and having to compete against other processes or services on the same node for these resources is counterproductive.
</p>
<p>There are a ton of external systems that can perform different transformations. One such system is the Elasticsearch’ official <a href="https://www.elastic.co/logstash">Logstash</a>, which is intended to pre-process logs, converting them from an unstructured flat log-line into a better structured JSON document that’s suitable for indexing.
</p>
<p>
	<img src="https://api.contentstack.io/v2/assets/575e4c989985d58976376c30/download?uid=blt82c79dafe7ef19b6?uid=blt82c79dafe7ef19b6" data-sys-asset-uid="bltc0a93bee3d3ccb1b" alt="External system" style="max-width:100%;">
</p><figcaption>External system</figcaption>
<p>For more complex document pre-processing, it’s possible to either use <a href="http://hadoop.apache.org">Hadoop</a>, <a href="http://spark.apache.org">Spark</a>, <a href="http://storm.apache.org">Storm</a> and similar systems, or hook up a completely custom document preprocessing system using some kind of message broker, such as <a href="http://rabbitmq.com">RabbitMQ</a>. In most of these cases, the final step of actually shipping the final documents to Elasticsearch for indexing is an exercise left for the reader. In order to do this quickly and reliably, there’s a lot of time and hairs to be saved by looking into using a message broker that already has official support from both Elasticsearch and the document processing system. One such example would be using the <a href="https://github.com/elasticsearch/elasticsearch-river-rabbitmq">RabbitMQ Elasticsearch plugin</a>.
</p>
<p>Using these external systems, it’s feasible to use tools like <a href="http://documentcloud.github.io/docsplit/">docsplit</a> on one or more worker nodes to perform OCR, generate thumbnails etc. Due to the number of system dependencies and resource usage of such tools, they’re not a good fit to install on the Elasticsearch server, but it’s <a href="https://registry.hub.docker.com/u/cymb/docsplit-dock/dockerfile/">rather easy</a> to use <a href="https://docker.com/">Docker</a> containers and integrate that with your document processing system / service of choice. This gives us a superset of the features from the mapper-attachments plugin with the flexibility and scalability of the external systems.
</p>
</section><section id="summary" class="level2">
<h2><a href="#summary">Summary</a></h2>
<p>Document processing makes it possible to enhance the documents being indexed to Elasticsearch in a variety of different ways.
</p>
<p>There’s no “ultimate” solution that handles all document processing requirements for everyone, but that doesn’t mean document processing is an unsolved problem. Instead, there are multiple solutions to pick from depending on requirements and pre-existing services and expertise within the organization.
</p>
<p>For small-scale transformations, the <code>transform</code> field type may be just fine, and in some circumstances doing the neccessary processing using a plugin to the Elasticsearch analysis-service or as a custom input endpoint may be a simple, workable solution. More elaborate document processing calls for an external system to perform the task. While it is more complex to set up and manage, it is significantly more flexible with regard to the scaling and managing of updates and changes to the processing pipeline.
</p>
<div class="references">
</div>
</section>

Document Processing and Elasticsearch

<html><head></head><body><p><strong>UPDATE:</strong> This article refers to our hosted Elasticsearch offering by an older name, Found. Please note that Found is now known as Elastic Cloud.</p><p>Scripting is an important part of the toolbox of any Elasticsearch user, and enables evaluating custom expressions that may be used to synthesize fields or provide customized scoring. In this post we take a brief look at the upcoming changes to the scripting module and the different scripting languages available to use today.
</p><p>In a recent <a href="/blog/scripting">blog post</a> about scripting, Elasticsearch outlined some coming changes to its scripting support. The two important takeaways from that post is that the sandboxing story of scripts will be improved and that the default scripting language will be changed from MVEL to Groovy.
</p><p>Let&#x2019;s take a look at the current official scripting languages.
</p><section id="the-well-known-mvel" class="level2">
<h2><a href="#the-well-known-mvel">The Well Known MVEL</a></h2>
<p><a href="https://github.com/mvel">MVEL</a> (MVFLEX Expression Language) is a simple, but powerful expression language with an intuitive syntax that most users of Elasticsearch are able to catch on to quickly. It was the default language used by scripts for Elasticsearch up and including the 1.3.x branch, but it was removed in favor of Groovy from 1.4. In 1.4 and later, the <a href="https://github.com/elasticsearch/elasticsearch-lang-mvel"><code>elasticsearch-lang-mvel</code> plugin</a> will need to be installed in order to use MVEL as a scripting language.
</p>
<p>When using MVEL, it&#x2019;s worth keeping in mind that it has no sandboxing support at all (which was part of the reason it was replaced as a default), which means any user able to execute scripts on your cluster must be trusted not to have malicious intent.
</p>
<p>It&#x2019;s still the scripting language that the most people using Elasticsearch are familiar with, but that may very well change soon. In either case, changing from MVEL to Groovy and back is a minor setting tweak (and in some cases, a small syntax change).
</p>
</section><section id="getting-groovy" class="level2">
<h2><a href="#getting-groovy">Getting Groovy</a></h2>
<p><a href="http://groovy.codehaus.org/">Groovy</a> is a well-known dynamic language for the JVM (the Java Virtual Machine, which Elasticsearch runs on). Unlike MVEL, it&#x2019;s actually a general purpose language that became known with the release of <a href="https://grails.org/">Grails</a>, which served the same purpose for <a href="http://rubyonrails.org/">Rails</a> did for <a href="https://www.ruby-lang.org/en/">Ruby</a>. Since its release, Groovy has been under constant development, and since version 2.0.0 has started supporting static compilation as well.
</p>
<p>The reasons for changing the default to Groovy were performance, development/features as well as easier sandboxing route. With the possibility of static compilation, a large community that continuously improves the language as well as the sandboxing features it comes with, it is certainly not a bad choice for a default.
</p>
<p>While Groovy supports sandboxing, we would still advise users to take extra precautions and avoid giving untrusted clients the ability to run arbitrary code, in case the sandbox ends up being possible to break out of. Even if the sandbox holds, it does not prevent simple denial of service attacks such as simple <code>while(true) {}</code> style loops which exhausts resources and prevents Elasticsearch from operating properly.
</p>
</section><section id="the-ubiquitous-javascript" class="level2">
<h2><a href="#the-ubiquitous-javascript">The Ubiquitous Javascript</a></h2>
<p>No scripting support would be complete without Javascript, and of course, Elasticsearch supports it. While it was not chosen as the default scripting language for 1.4 and forward, it&#x2019;s possible to use it by installing the official <a href="https://github.com/elasticsearch/elasticsearch-lang-javascript"><code>elasticsearch-lang-javascript</code>-plugin</a>. The reasoning behind it not being chosen as the default had to do with the performance of <a href="https://developer.mozilla.org/en-US/docs/Mozilla/Projects/Rhino">Rhino</a> and the concurrent script execution support of <a href="http://openjdk.java.net/projects/nashorn/">Nashorn</a>. Whether the differences amount to anything that matters significantly is ultimately up to you to decide for your own use case, so if you enjoy using Javascript, it still enjoys official support.
</p>
</section><section id="python" class="level2">
<h2><a href="#python">Python</a></h2>
<p>Rounding off with the officially provided scripting language support is Python, which is supported through the <a href="http://www.jython.org/">Jython</a> project. While development on that particular project has been slow due to missing resources, it&#x2019;s considered stable and works well, so it may be a good choice for anyone who is most familiar with Python.
</p>
</section><section id="custom-languages" class="level2">
<h2><a href="#custom-languages">Custom Languages</a></h2>
<p>If there&#x2019;s a programming language missing from Elasticsearch scripting, add it yourself. Any scripting language supporting <a href="https://www.jcp.org/en/jsr/detail?id=223">JSR-223</a> can be easily added using our <a href="/blog/found-extending-the-scripting-module/">Script Module primer</a> as a starting point. Other languages can be added as well, but if there&#x2019;s no implementation of it available on the JVM, it&#x2019;s certainly significantly more work doing so.
</p>
</section><section id="lucene-expressions" class="level2">
<h2><a href="#lucene-expressions">Lucene Expressions</a></h2>
<p>While <a href="https://issues.apache.org/jira/browse/LUCENE-5207">Lucene Expressions</a>, documented in the <a href="http://lucene.apache.org/core/4_9_0/expressions/index.html?org/apache/lucene/expressions/js/package-summary.html">Lucene API</a> is an experimental feature of Elasticsearch, it&#x2019;s certainly a very performant one.
</p>
<p>It&#x2019;s not really a full-blown scripting language either, but using an advanced feature of Lucene, which lies at the core of Elasticsearch, a Javascript-like expression may be used as a script. The expression is compiled once before it&#x2019;s used, and if you&#x2019;re using nested expressions, the results are even cached to avoid recomputation. Under the hood, the expressions are compiled to .class-files containing regular Java bytecode, which is about as fast it gets.
</p>
<p>Be aware of the limitations though, such as no support for stored fields and it&#x2019;s only possible to work with numeric fields.
</p>
</section><section id="suming-up" class="level2">
<h2><a href="#suming-up">sum()ing up</a></h2>
<p>Elasticsearch is very flexible when it comes to scripting, allowing its users to use their preferred scripting language to accomplish a wide variety of tasks. In a later article we&#x2019;ll take a closer look at various applications of scripts: where it makes sense to use it and where scripts should be avoided.
</p>
</section></body></html>

Scripting with Elasticsearch

<html><head></head><body><p><strong>UPDATE:</strong> This article refers to our hosted Elasticsearch offering by an older name, Found. Please note that Found is now known as Elastic Cloud.</p>
<p>In order to write a great plugin, we need to have an overview of the different plugin types and scopes Elasticsearch has made available to us. This article gives a rundown of these and also some tips on how to write a &apos;well behaved&apos; plugin.
</p><section id="introduction" class="level2">
<h2><a href="#introduction">Introduction</a></h2>
<p>Plugins make up an important part of the Elasticsearch ecosystem, and plugin authors are allowed to redefine or add - and even remove - functionality to/from Elasticsearch. In <a href="/blog/found-writing-a-plugin">Writing an Elasticsearch Plugin: Getting Started</a> we showed how to start developing a plugin from scratch; this time around we&#x2019;re going to look at the different scopes a plugin can add functionality to.
</p>
</section><section id="initialization" class="level2">
<h2><a href="#initialization">Initialization</a></h2>
<p>Plugins are discovered by looking for plugin definition files within the Java classpath of the Elasticsearch process. These provide the exact class names of the <code>Plugin</code> subclasses that Elasticsearch then loads and initializes. This process is <a href="/blog/found-writing-a-plugin/#letting-elasticsearch-know-about-our-plugin">described here</a>. After being loaded, Elasticsearch will call certain methods on the <code>Plugin</code> instances depending on the scope the plugin currently can affect.
</p>
</section><section id="plugin-component-scopes" class="level2">
<h2><a href="#plugin-component-scopes">Plugin Component Scopes</a></h2>
<p>In essence, there&#x2019;s three levels, or scopes, of components that plugins may provide. These scopes are:
</p>
<ul>
	<li>Global</li>
	<li>Index</li>
	<li>Shard</li>
</ul>
<p>The scopes determine when the components they define are used. <strong>Global</strong> scope is used whenever the associated Elasticsearch instance is initializing, <strong>index</strong> scope whenever an index is created (or opened after being closed), and <strong>shard</strong> for every shard started on the given node. This feature allows different functionality to be attached to different indexes (or even shards) within the same Elasticsearch cluster.
</p>
</section><section id="plugin-component-types" class="level2">
<h2><a href="#plugin-component-types">Plugin Component Types</a></h2>
<p>Each of the scopes has two kinds of components:
</p>
<ul>
	<li>Module</li>
	<li>Service</li>
</ul>
<p>Modules are actually <a href="https://github.com/google/guice">Guice modules</a> that usually &#x201C;bind&#x201D; classes to implementations. Often, this simply implies that one or more instances of the class will be instantiated at a later time, when it&#x2019;s required. An introduction to Guice is outside the scope of this article, but interested developers can find multiple tutorials and great introductions to the <a href="https://code.google.com/p/google-guice/wiki/Motivation">Guice framework here</a>.
</p>
<p>Services are anything that starts and may be stopped at a later time. Services in the global scope are implementations of <code>LifecycleComponent</code>, which has two important methods:<code>start</code> and<code>stop</code> that are invoked by Elasticsearch when it starts and stops respectively.
</p>
<p>Plugins also get notified when Modules are created, which enables calling public methods on the modules before they&#x2019;re used. The most common use case of this is registering additional analyzers, tokenizers or token filters on the <code>AnalysisModule</code>.
</p>
<p>For an overview over which modules are available, have a look at <a href="/blog/found-elasticsearch-internals/">Elasticsearch Internals: An Overview</a>.
</p>
<p>Additionally, plugins are allowed to provide additional settings that may be used by both Elasticsearch core and other plugins/modules.
</p>
</section><section id="creating-and-freeing-resources" class="level2">
<h2><a href="#creating-and-freeing-resources">Creating and Freeing Resources</a></h2>
<p>Classes instantiated via modules may of course contain start/stop logic as well, but they have to perform their own bookkeeping, for example by registering a <code>LifecycleListener</code> on an external <code>LifecycleComponent</code>. It&#x2019;s generally considered good practice to avoid doing heavy lifting or spawning threads during initialization, and rather use <code>LifecycleComponent</code> callbacks to actually create and - more importantly - stop and clean up resources when they&#x2019;re no longer required. Not freeing resources when a module is no longer required, for example for shard scoped modules, may lead to memory leaks.
</p>
<table>
<caption>Overview of <code>Plugin</code> methods and what kind they define.</caption>
<thead>
<tr class="header">
	<th style="text-align: center;">Method
	</th>
	<th style="text-align: center;">Kind
	</th>
</tr>
</thead>
<tbody>
<tr class="odd">
	<td style="text-align: center;">modules()
	</td>
	<td style="text-align: center;">Module
	</td>
</tr>
<tr class="even">
	<td style="text-align: center;">services()
	</td>
	<td style="text-align: center;">LifecycleComponent
	</td>
</tr>
<tr class="odd">
	<td style="text-align: center;">indexModules
	</td>
	<td style="text-align: center;">Module
	</td>
</tr>
<tr class="even">
	<td style="text-align: center;">indexServices()
	</td>
	<td style="text-align: center;">CloseableIndexComponent
	</td>
</tr>
<tr class="odd">
	<td style="text-align: center;">shardModules()
	</td>
	<td style="text-align: center;">Module
	</td>
</tr>
<tr class="even">
	<td style="text-align: center;">shardServices()
	</td>
	<td style="text-align: center;">CloseableIndexComponent
	</td>
</tr>
</tbody>
</table>
<p>In the above table, notice that index and shard services are not actually <code>LifecycleComponent</code>s or <code>ClosableIndexComponent</code>s. The difference between them is that the index and shard services are implicitly started when they are created, due to their nature (they&#x2019;re only invoked when required).
</p>
</section><section id="limitations" class="level2">
<h2><a href="#limitations">Limitations</a></h2>
<p>Not all functionality in Elasticsearch is currently intended (or even possible) to be extended or replaced. Some pieces of functionality are commonly either properly named with the <code>Internal</code> prefix, such as <code>InternalNode</code> and <code>InternalTransportClient</code>, hard-coded via Guice-bindings in a module (and not configurable) or not separated into the common interface/implementation pairs.
</p>
<p>One example of a commonly used service that is not possible to replace or extend by writing a plugin is the <code>ClusterService</code>. This service is responsible for maintaining and handling updates to the cluster state, as well as notifying cluster state listeners about any changes. The binding of this service is hard-coded into Elasticsearch and is not configurable.
</p>
<p>Another example is the <code>TransportService</code>. The <code>TransportService</code> is responsible for handling the connections to the other instances in the cluster, sending requests to the other nodes and providing the caller with the response via a callback. While it is possible to configure Elasticsearch to bind a different implementation by using the <code>transport.service.type</code>-configuration key, the <code>TransportService</code> is a class, not an interface, and is used extensively in different parts of the Elasticsearch code-base, so a significantly different implementation would be difficult since there&#x2019;s a lot of compatibility that has to be kept.
</p>
</section><section id="conclusion" class="level2">
<h2><a href="#conclusion">Conclusion</a></h2>
<p>At <a href="/found">Found</a> we see a lot of different use cases for plugins, ranging from simply providing word lists to custom <a href="/guide/en/elasticsearch/rivers/current/">rivers</a> to adding support for new and exciting analyzers to Elasticsearch.
</p>
<p>Found has a set of <a href="guide/en/found/current/official-plugins.html#_provided_plugins">officially supported plugins</a>, and the list of <a href="/guide/en/elasticsearch/reference/current/modules-plugins.html">known Elasticsearch plugins</a> keeps growing.
</p>
<p>Plugins are a source of almost endless possibilities of extensions to Elasticsearch and is a great way to provide additional functionality to Elasticsearch without having to run a custom fork. We can&#x2019;t wait to see what plugins the community comes up with next!
</p>
</section></body></html>

Elasticsearch Plugin Types

<html><head></head><body><p><strong>UPDATE:</strong> This article refers to our hosted Elasticsearch offering by an older name, Found. Please note that Found is now known as Elastic Cloud.
</p>
<p>Controlling the number of fields returned for search requests is an important aspect of maximizing Elasticsearch&apos;s performance. In this article we&apos;ll look at how we can selectively return only the fields we&apos;re interested in for each search hit in order to optimize our usage pattern.
</p><section id="introduction" class="level2">
<h2><a href="#introduction">Introduction</a></h2>
<p>When searching, using the <a href="/guide/en/elasticsearch/reference/current/search-request-fields.html"><code>fields</code>-parameter</a> allows to restrict the fields returned for each hit. This is a feature commonly used in order to optimize the amount of data transferred from Elasticsearch per hit by only selecting the relevant fields. While it may sound simple (and it is), the impact on performance may be tremendous, and is proportional both to the number of documents returned and the size of each document.
</p>
<p>Consider the case where each document has an average size of 20KB. Returning 10 hits will cause an average 200KB of data to be returned for each search request. Multiply this by 100 search requests/second and we end up needing 20 MB/s bandwidth. If we can reduce the size of the returned documents to 800 bytes, which is plausible by just returning the required fields, each request will be roughly 8KB in size, and the total bandwidth needed for 100 requests/s is only 800 KB/s. Keeping the result sets as small as possible is certainly important in order to be able to scale sensibly.
</p>
<p>By default, the <a href="/guide/en/elasticsearch/reference/current/mapping-source-field.html"><code>_source</code>-field</a> is enabled, and its contents are parsed and returned for each search hit. We can save precious server-side CPU time and disk IO by limiting the returned fields since we no longer have to load and parse the whole document source. Simply specifying <code>fields</code> when searching is enough to trigger this behavior. Should we require to also return the document source in addition to the fields, we can use <a href="/guide/en/elasticsearch/reference/current/search-request-source-filtering.html">source filtering</a>, which we&apos;ll look at shortly.
</p>
<p>One feature about <code>fields</code> that&apos;s not commonly known is the ability to select <a href="/guide/en/elasticsearch/reference/current/mapping-fields.html">metadata-fields</a> as well. Of particular note is its ability to select the <a href="/guide/en/elasticsearch/reference/current/mapping-ttl-field.html"><code>_ttl</code>-field</a>, which actually returns the number of milliseconds <em>until</em> the document expires, not the original lifespan of the document. A very handy feature indeed.
</p> </section><section id="nested-fields-and-nested-data" class="level2">
<h2><a href="#nested-fields-and-nested-data">Nested Fields and Nested Data</a></h2>
<p>Sometimes, only selecting simple fields are not enough since we need additional data from nested fields that may be dynamic. In these cases, we cannot use <code>fields</code> because we would end up with the following failure:
</p>
<pre class="sourceCode json prettyprint">{
    ...,
    &quot;_shards&quot;: {
        &quot;failures&quot;: [
            {
                &quot;index&quot;: &quot;index-name&quot;,
                &quot;shard&quot;: 2,
                &quot;status&quot;: 400,
                &quot;reason&quot;: &quot;ElasticsearchIllegalArgumentException[field [field_with_nested_data] isn&apos;t a leaf field]&quot;
            }
        ]
    },...
}
</pre>
<p>This exception is thrown because <code>fields</code> are only able to load data from leaf nodes (i.e nodes that have no children). In order to load non-leaf nodes, we have to use the <code>_source</code>-field in the search request which enables a feature called <a href="/guide/en/elasticsearch/reference/current/search-request-source-filtering.html">source filtering</a>, which we&apos;ll look at next.
</p> </section><section id="source-filtering" class="level2">
<h2><a href="#source-filtering">Source Filtering</a></h2>
<p><a href="/guide/en/elasticsearch/reference/current/search-request-source-filtering.html">Source filtering</a> allows us to control which parts of the original JSON document are returned for each hit. We can include or exclude parts of the original document based on patterns matching the field name path. It&apos;s worth keeping in mind that this only saves us on bandwidth costs between the nodes participating in the search as well as the client, not CPU or Disk, as was the case when using <code>fields</code>. This is because when using source filtering, we still have to load and parse the entire source document for each returned hit in order to match it against the inclusion and exclusion patterns. But still it&apos;s an important tool in our optimization toolbelt that is very easy to get started with.
</p>
<p>Prior to 1.0, this was known as <a href="/guide/en/elasticsearch/reference/current/search-request-fields.html#partial">partial fields</a>, which was deprecated and replaced by source filtering.
</p> </section><section id="unstored-fields" class="level2">
<h2><a href="#unstored-fields">Unstored Fields</a></h2>
<p>Returning the <a href="/guide/en/elasticsearch/reference/current/search-request-fielddata-fields.html">field data representation</a> of a field using <code>fielddata_fields</code> is a rather new addition to Elasticsearch (added in 1.0), which can be used to load fields that are not even stored (i.e fields that have <code>store: false</code> set in <a href="/guide/en/elasticsearch/reference/current/mapping-core-types.html">their mapping</a>).
</p>
<p>It&apos;s important to note that when returning data from the <a href="/guide/en/elasticsearch/reference/current/index-modules-fielddata.html">fielddata</a>, the terms for that field is loaded into memory and cached. This costs additional memory the first time around, but the same caches are used when sorting and faceting on the field in question. In other words, if you&apos;re already sorting or faceting on the field, returning it through the use of <code>fielddata_fields</code> is <em>very</em> cheap, but think twice about using it to return data from fields that are otherwise not commonly used.
</p>
<p>The field data representation of a field varies on the <a href="/guide/en/elasticsearch/reference/current/mapping-types.html">type and mapping of the field</a>, which includes the <a href="/guide/en/elasticsearch/reference/current/mapping-analyzer-field.html#mapping-analyzer-field">analyzer</a>. For example, boolean values are returned through <code>fielddata_fields</code> as the strings <code>T</code> and <code>F</code>, and most string fields are just a collection of sorted terms.
</p> </section><section id="script-fields" class="level2">
<h2><a href="#script-fields">Script Fields</a></h2>
<p>Using <a href="/guide/en/elasticsearch/reference/current/search-request-script-fields.html#search-request-script-fields">script fields</a> enables us to return the result of evaluating a script for each hit. The script may perform some computation based on one or more of the existing fields, including metadata fields, access and return parts of the source document or return basically any value, which is used as the value for the field.
</p>
<p>Here is an example that shows how we can see the remaining time and the expiration date of documents in our index, assuming we&apos;re using <a href="/guide/en/elasticsearch/reference/current/mapping-ttl-field.html">document ttls</a>.
</p>
<pre class="sourceCode json prettyprint">{
    &quot;fields&quot;: [
        &quot;_ttl&quot;
    ],
    &quot;script_fields&quot;: {
        &quot;expires_at&quot;: {
            &quot;script&quot;: &quot;new Date(doc._ttl.getValue())&quot;
        }
    }
}
</pre>
<p>Using script fields requires <a href="/guide/en/elasticsearch/reference/current/modules-scripting.html">dynamic scripts</a> to be enabled, and this has some <a href="/blog/found-elasticsearch-security/#scripting-for-fun-and-profit">important security implications</a> you need to be aware of.
</p> </section><section id="conclusion" class="level2">
<h2><a href="#conclusion">Conclusion</a></h2>
<p>After looking into our options for selecting only the relevant fields to return for search requests, we&apos;re now better suited to optimize the amount of data transferred between our Elasticsearch cluster and our search clients. We&apos;ve also seen how to return fields and information that are not normally returned by Elasticsearch searches that we still may be interested in.
</p> </section></body></html>

Managing Elasticsearch Fields When Searching

<html><head></head><body><p><strong>UPDATE:</strong> This article refers to our hosted Elasticsearch offering by an older name, Found. Please note that Found is now known as Elastic Cloud.
</p>
<p>In this article we explore a workflow for exploring new data via mapping refinements. We index an example document, look at its default mapping and iteratively improve on it to get us one step closer to our goal.
</p><section id="introduction" class="level2">
<h2><a href="#introduction">Introduction</a></h2>
<p>Elasticsearch is for many intents and purposes schema-less, which means that documents can be indexed without explicitly providing a schema. The schema is a mapping that describes the fields in the JSON documents along with their data type and how they should be indexed in the Lucene indexes that lie under the hood.
</p>
<p>In this article, I will describe a workflow for working with mappings in order to explore new data sets. I have personally used this workflow many times, and with great success. We&apos;ll also look at a new tool (Play) we&apos;ve developed that greatly speeds up this kind of experimental work.
</p>
</section><section id="the-workflow" class="level2">
<h2><a href="#the-workflow">The Workflow</a></h2>
<p>Without further ado, the workflow is a simple six-step process that may be loosely described like this:
</p>
<ol type="1">
	<li>Extract a sufficiently large set of example JSON documents from the data set.</li>
	<li>Index the documents.</li>
	<li>Run a few sample queries.</li>
	<li>Look at the automatically generated mapping.</li>
	<li>Refine the generated mapping.</li>
	<li>Repeat step 2 to 5 until I&apos;m satisfied with the result.</li>
</ol>
<p>The number of documents should be sufficiently large enough that the query results can be verified. Using too many documents wastes time, since I try to keep the iterations small. On the other hand, using too few documents makes it difficult to assess the quality of my queries and the correctness of my mapping.
</p>
<p>From this workflow, it is evident that to get the most power out of Elasticsearch, it is important to understand how this mapping works.
</p>
<section id="looking-at-the-default-dynamic-mapping" class="level3">
<h3><a href="#looking-at-the-default-dynamic-mapping">Looking at the Default Dynamic Mapping</a></h3>
<p>We start by indexing the following JSON document to a fresh Elasticsearch index:
</p>
<pre class="sourceCode json prettyprint">$ curl ...:9200/names/name -XPOST -d &apos;{
        &quot;userId&quot;: 10,
        &quot;name&quot;: {
            &quot;first&quot;: &quot;Katherine&quot;,
            &quot;last&quot;: &quot;Jones&quot;
        },
        &quot;tags&quot;: [&quot;trick manager&quot;, &quot;restaurant manager&quot;]
    }&apos;
{&quot;ok&quot;:true,&quot;_index&quot;:&quot;names&quot;,&quot;_type&quot;:&quot;name&quot;,&quot;_id&quot;:&quot;Q3Yj7mofTIW_YZTjW5Hp9w&quot;,&quot;_version&quot;:1}
</pre>
<p>By looking at the output from the indexing operation, we see that it was successfully indexed, and was assigned an automatically generated id. Let&apos;s look at the current mapping to see how it got stored:
</p>
<pre class="sourceCode json prettyprint">$ curl ...:9200/names/name/_mapping?pretty
{
  &quot;name&quot; : {
    &quot;properties&quot; : {
      &quot;name&quot; : {
        &quot;properties&quot; : {
          &quot;first&quot; : {
            &quot;type&quot; : &quot;string&quot;
          },
          &quot;last&quot; : {
            &quot;type&quot; : &quot;string&quot;
          }
        }
      },
      &quot;tags&quot; : {
        &quot;type&quot; : &quot;string&quot;
      },
      &quot;userId&quot; : {
        &quot;type&quot; : &quot;long&quot;
      }
    }
  }
}
</pre>
<p>The &quot;dynamic mapping&quot; feature of Elasticsearch has detected that our &quot;userId&quot; field was numeric and selected the <code>long</code> datatype for this field. <code>tags</code> was recognized as a list of strings, which is stored in the same Lucene field. Lastly, <code>name</code> was detected as a nested object with its own two string properties.
</p>
<p>When writing a custom mapping, I usually index a few documents before looking at the mapping instead of starting to write a mapping from scratch. The reason being that Elasticsearch is fairly good at giving me some sane defaults to base my work on, and this saves me a lot of time writing boilerplate mapping and debugging simple typos.
</p>
</section>
<section id="improving-the-mapping" class="level3">
<h3><a href="#improving-the-mapping">Improving the Mapping</a></h3>
<p>A simple query shows one important problem with this generated mapping, faceting on the tags shows the analyzed terms:
</p>
<pre class="sourceCode json prettyprint">$ curl ...:9200/names/name/_search?pretty -XPOST -d &apos;{
    &quot;query&quot;: {&quot;match_all&quot;: {}},
    &quot;size&quot;: 0,
    &quot;facets&quot;: {
        &quot;tags&quot;: {
            &quot;terms&quot;: {&quot;field&quot;: &quot;tags&quot;}
        }
    }
}&apos;
{
  &quot;took&quot; : 2,
  &quot;timed_out&quot; : false,
  &quot;_shards&quot; : {
    &quot;total&quot; : 5,
    &quot;successful&quot; : 5,
    &quot;failed&quot; : 0
  },
  &quot;hits&quot; : {
    &quot;total&quot; : 1,
    &quot;max_score&quot; : 1.0,
    &quot;hits&quot; : [ ]
  },
  &quot;facets&quot; : {
    &quot;tags&quot; : {
      &quot;_type&quot; : &quot;terms&quot;,
      &quot;missing&quot; : 0,
      &quot;total&quot; : 3,
      &quot;other&quot; : 0,
      &quot;terms&quot; : [ {
        &quot;term&quot; : &quot;trick&quot;,
        &quot;count&quot; : 1
      }, {
        &quot;term&quot; : &quot;restaurant&quot;,
        &quot;count&quot; : 1
      }, {
        &quot;term&quot; : &quot;manager&quot;,
        &quot;count&quot; : 1
      } ]
    }
  }
}
</pre>
<p>Tags are generally meant to be keywords, but the default mapping analyzes it with the default analyzer, which tokenizes and lowercases the terms. If we were just looking at the terms from the above tags facet, we could easily believe that someone in the index were tagged as <code>restaurant</code>s and <code>trick</code>s.
</p>
<p>To fix this, we can update the <a href="/guide/reference/api/admin-indices-put-mapping/">mapping</a> to store the <code>tags</code>-field as a keyword by setting its <code>index</code> setting to <code>not_analyzed</code>, which means we will not analyze it and store its contents verbatim in the index. Elasticsearch is not able to merge this setting if we update it directly:
</p>
<pre class="sourceCode json prettyprint">$ curl ...:9200/names/name/_mapping -XPOST -d &apos;{
  &quot;name&quot; : {
    &quot;properties&quot; : {
      &quot;tags&quot; : {
        &quot;type&quot; : &quot;string&quot;,
        &quot;index&quot;: &quot;not_analyzed&quot;
      }
    }
  }
}&apos;
{&quot;error&quot;:&quot;MergeMappingException[Merge failed with failures {[mapper [tags] has different index values, mapper [tags] has different tokenize values, mapper [tags] has different index_analyzer]}]&quot;,&quot;status&quot;:400}
</pre>
<p>However, changing a field by updating its type to <a href="/guide/en/elasticsearch/reference/current/_multi_fields.html">multi_field</a> is supported:
</p>
<pre class="sourceCode json prettyprint">$ curl ...:9200/names/name/_mapping -XPOST -d &apos;{
  &quot;name&quot; : {
    &quot;properties&quot; : {
      &quot;tags&quot; : {
        &quot;type&quot; : &quot;multi_field&quot;,
        &quot;fields&quot;: {
            &quot;tags&quot;: {&quot;type&quot;: &quot;string&quot;},
            &quot;keyword&quot;: {&quot;type&quot;: &quot;string&quot;, &quot;index&quot;: &quot;not_analyzed&quot;}
        }
      }
    }
  }
}&apos;
{&quot;ok&quot;:true,&quot;acknowledged&quot;:true}
</pre>
<p>We still have to re-index the data before these new fields are populated, but there&apos;s no indexing or querying downtime associated with this method. Thus, it is possible to do this operation on an index that is in use without affecting other users of the index. Note that this might be a nice strategy for staging indexes, but I still recommend avoiding experimenting with production indexes / servers, because there is currently no rollback feature available in Elasticsearch.
</p>
<pre class="sourceCode json prettyprint">$ curl ...:9200/names/name/Q3Yj7mofTIW_YZTjW5Hp9w -XPOST -d &apos;{
        &quot;userId&quot;: 10,
        &quot;name&quot;: {
            &quot;first&quot;: &quot;Katherine&quot;,
            &quot;last&quot;: &quot;Jones&quot;
        },
        &quot;tags&quot;: [&quot;trick manager&quot;, &quot;restaurant manager&quot;]
    }&apos;
{&quot;ok&quot;:true,&quot;_index&quot;:&quot;names&quot;,&quot;_type&quot;:&quot;name&quot;,&quot;_id&quot;:&quot;Q3Yj7mofTIW_YZTjW5Hp9w&quot;,&quot;_version&quot;:1}
</pre>
<p>By faceting of the &quot;virtual&quot; field <code>tags.keyword</code>, we now get our desired results:
</p>
<pre class="sourceCode json prettyprint">$ curl ...:9200/names/name/_search?pretty -XPOST -d &apos;{
    &quot;query&quot;: {&quot;match_all&quot;: {}},
    &quot;size&quot;: 0,
    &quot;facets&quot;: {
        &quot;tags&quot;: {
            &quot;terms&quot;: {&quot;field&quot;: &quot;tags.keyword&quot;}
        }
    }
}&apos;
{
  &quot;took&quot; : 2,
  &quot;timed_out&quot; : false,
  &quot;_shards&quot; : {
    &quot;total&quot; : 5,
    &quot;successful&quot; : 5,
    &quot;failed&quot; : 0
  },
  &quot;hits&quot; : {
    &quot;total&quot; : 1,
    &quot;max_score&quot; : 1.0,
    &quot;hits&quot; : [ ]
  },
  &quot;facets&quot; : {
    &quot;tags&quot; : {
      &quot;_type&quot; : &quot;terms&quot;,
      &quot;missing&quot; : 0,
      &quot;total&quot; : 2,
      &quot;other&quot; : 0,
      &quot;terms&quot; : [ {
        &quot;term&quot; : &quot;trick manager&quot;,
        &quot;count&quot; : 1
      }, {
        &quot;term&quot; : &quot;restaurant manager&quot;,
        &quot;count&quot; : 1
      } ]
    }
  }
}
</pre>
<p>Note that Elasticsearch will consider our new field <code>tags.keyword</code> to be empty until the documents in the index have been re-indexed with the updated mapping, but we do not have to re-index all the documents to test and see if the new field we introduced works as expected.
</p>
</section>
</section><section id="ending-remarks" class="level2">
<h2><a href="#ending-remarks">Ending Remarks</a></h2>
<p>Spending some time figuring out an efficient workflow for working with new or large data sets is likely to save a lot of time over the course of a project. Test mapping changes in small increments, and if possible, without indexing your complete document collection to save time both debugging and indexing.
</p>
<p>Using tools to accomplish the above can drastically decrease the time between these iterations and make it easier to experiment with new features.
</p>
</section></body></html>

A Data Exploration Workflow for Mappings

<html><head></head><body><p><strong>UPDATE:</strong> This article refers to our hosted Elasticsearch offering by an older name, Found. Please note that Found is now known as Elastic Cloud.</p><p>This article introduces the networking part of Elasticsearch. We look at the network topology of an Elasticsearch cluster, which connections are established between which nodes and how the different Java clients works. Finally, we look a bit closer on the communication channels between two nodes.
</p><section id="node-topology" class="level2">
<h2><a href="#node-topology">Node Topology</a></h2>
<p>Elasticsearch nodes in a cluster form what is known as a <a href="http://en.wikipedia.org/wiki/Network_topology#Mesh">full mesh topology</a>, which means that each Elasticsearch node maintains a connection to each of the other nodes.
</p>
<p>
	<img src="https://api.contentstack.io/v2/assets/575e4c8d9e7a83165490c152/download?uid=blt2173f39afdb94e27?uid=blt2173f39afdb94e27" data-sys-asset-uid="blt292169d40755cd62" alt="Full mesh topology with 6 cluster nodes">
</p><figcaption>Full mesh topology with 6 cluster nodes</figcaption>
<p>In order to simplify the code base, the connections are used as one-way connections, so the connection topology actually ends up looking like this:
</p>
<p>
	<img src="https://api.contentstack.io/v2/assets/575e4c8d58208ba076e28077/download?uid=blt3058d4e681869e36?uid=blt3058d4e681869e36" data-sys-asset-uid="blt561990b45246d265" alt="6 cluster nodes with connections, each line represents a connection">
</p><figcaption>6 cluster nodes with connections, each line represents a connection</figcaption>
</section><section id="adding-a-node" class="level2">
<h2><a href="#adding-a-node">Adding a Node</a></h2>
<p>When a node starts up, it reads a list of seed nodes from its configuration using <a href="/guide/en/elasticsearch/reference/current/modules-discovery-zen.html#unicast">unicast</a>, and sends <a href="/guide/en/elasticsearch/reference/current/modules-discovery-zen.html#multicast">multicast</a> messages looking for other nodes in the same cluster.
</p>
<p>As the node discovers more instances in the same cluster, it connects to them one by one, asks them for information about all the nodes they know and then attempts to connect to them all and officially join the cluster. In this way, previously running instances assist in quickly getting fresh instances up to speed on the current nodes in a cluster.
</p>
</section><section id="node-clients" class="level2">
<h2><a href="#node-clients">Node Clients</a></h2>
<p>Even <a href="/guide/en/elasticsearch/client/java-api/current/client.html#node-client">&#x201C;client&#x201D; Elasticsearch nodes</a> (i.e nodes configured with <code>node.client: true</code> or build with <code>NodeBuilder.client(true)</code>) connect to the cluster this way.
</p>
<p>This implies that the client node becomes a full participant in the full mesh network. In other words, once it starts joining the cluster, all the existing nodes in the cluster will connect back to the instance. This means that both the client and server firewalls and <a href="/guide/en/elasticsearch/reference/current/modules-network.html">publish hosts</a> must be properly configured in order to allow this. Additionally, whenever a node client joins, leaves or experiences networking issues, it causes extra work for all the other nodes in the cluster, such as opening / closing network connections and updating the cluster state with the information about the node.
</p>
<p>
	<img src="https://api.contentstack.io/v2/assets/575e4c8d12798d8e381fd677/download?uid=blt08a91ac57ff257cb?uid=blt08a91ac57ff257cb" data-sys-asset-uid="blt2da54d75934a625e" alt="Node client connecting to a cluster">
</p><figcaption>Node client connecting to a cluster</figcaption>
</section><section id="transport-clients" class="level2">
<h2><a href="#transport-clients">Transport Clients</a></h2>
<p>On the other hand, <a href="/guide/en/elasticsearch/client/java-api/current/client.html#transport-client">&#x201C;transport&#x201D; clients</a> work differently.
</p>
<p>When a Transport client connects to one or more instances in a cluster, the instances do not connect back, nor is the existence of the transport client part of the cluster state. This means that a joining / leaving transport client causes minimum extra work for the other nodes in the cluster.
</p>
<p>
	<img src="https://api.contentstack.io/v2/assets/575e4c8d9985d58976376a51/download?uid=bltb810f4f10f5305aa?uid=bltb810f4f10f5305aa" data-sys-asset-uid="bltba355f9bfb2a37a6" alt="Transport client connecting to a cluster">
</p><figcaption>Transport client connecting to a cluster</figcaption>
</section><section id="connections-and-channels" class="level2">
<h2><a href="#connections-and-channels">Connections and Channels</a></h2>
<p>What do we mean when we talk about the <code>connection</code> to a Node in Elasticsearch? In the sections above, we refrained from being specific about it and only used the term as a logical connection that allows communication between two nodes. Let&apos;s go into more detail.
</p>
<p>Usually, when we talk about connections in the context of networks, we refer to <a href="http://en.wikipedia.org/wiki/Transmission_Control_Protocol">TCP</a>-connections, which provide a reliable line of communication between two nodes.
</p>
<p>Elasticsearch uses (by default) TCP for communication between nodes, but in order to prevent important traffic such as fault-detection and cluster state changes from being affected by far less important, slower moving traffic such as query/indexing requests, it creates multiple TCP connections between each node. For each of these connections, Elasticsearch uses the term <code>channel</code>, which encapsulates the serialization / deserialization of messages over a particular connection.
</p>
<p>In earlier Elasticsearch versions there used to be three different classes of channels: <code>low</code>, <code>med</code> and <code>high</code>. After a while, <code>ping</code> was added, and a <a href="https://github.com/elasticsearch/elasticsearch/commit/35b573ff24c1462de77252f53cd60b4dcab23eda">recent change</a> renamed these channels such that they are more descriptive about their intention. At of the time of writing, the following channel classes exist:
</p>
<ul>
	<li><code>recovery</code>: 2 channels for the recovery of indexes</li>
	<li><code>bulk</code>: 3 channels for low priority bulk based operations such as bulk indexing. Previously called <code>low</code>.</li>
	<li><code>reg</code>: 6 channels for medium priority regular operations such as queries. Previous called <code>med</code>.</li>
	<li><code>state</code>: 1 channel dedicated to state based operations such as changes to the cluster state. Previously called <code>high</code>.</li>
	<li><code>ping</code>: 1 channel dedicated to pings between the instances for fault detection.</li>
</ul>
<p>The default number of channels in each of these class are configured with the configuration prefix of <code>transport.connections_per_node</code>.
</p>
<p>
	<img src="https://api.contentstack.io/v2/assets/575e4c8c0342dfd7382630f5/download?uid=blt8ac9e7f8ca891434?uid=blt8ac9e7f8ca891434" data-sys-asset-uid="blt09cfbb880e12caf9" alt="All channels in a three-node cluster. Each line represents a single channel">
</p><figcaption>All channels in a three-node cluster. Each line represents a single channel</figcaption>
<p>Elasticsearch has support for <a href="http://tldp.org/HOWTO/html_single/TCP-Keepalive-HOWTO/">TCP keepalive</a> which is not explicitly set by default. This prevents unused or idle channels from being closed, which would otherwise cascade into a complete disconnect-reconnect cycle as explained above.
</p>
<section id="adding-an-instance" class="level3">
<h3><a href="#adding-an-instance">Adding an Instance</a></h3>
<p>A consequence of the above is that adding a new instance to an existing cluster causes it to establish 13 connections to each node, and each node establishes 13 connections back to the new instance.
</p>
</section>
<section id="linked-channels" class="level3">
<h3><a href="#linked-channels">Linked Channels</a></h3>
<p>If one of the 13 channels to a particular node is closed due to intermittent network errors for example, all the other channels to the same node are closed and the connections to the node is reconnected. This is done in order to maintain some internal invariants and to ensure the internal consistency of the communication between the nodes.
</p>
</section>
</section><section id="ending-remarks" class="level2">
<h2><a href="#ending-remarks">Ending Remarks</a></h2>
<p>The fact that all the channels between two nodes in a cluster are linked makes it extra vulnerable to network issues, and this is one of the reasons why people are discouraged from trying to create a cluster between data centers that are far apart (and thus adding more sources of failure).
</p>
<p>In this article we&apos;ve shown the basic network topology of an Elasticsearch cluster and introduced the concept of <code>channels</code> that are used for communication between nodes. In a later article we&apos;ll take a closer look at what goes on inside each of these channels.
</p>
</section></body></html>

Elasticsearch Internals: Networking Introduction

<html><head></head><body><p><strong>UPDATE:</strong> This article refers to our hosted Elasticsearch offering by an older name, Found. Please note that Found is now known as Elastic Cloud.</p><p>This article gives an overview of the Elasticsearch internals. I will present a 10,000 foot view of the different modules that Elasticsearch is composed of and how we can extend or replace built-in functionality using plugins.
</p><section id="what-is-a-module" class="level2">
<h2><a href="#what-is-a-module">What is a Module?</a></h2>
<p>A module in Elasticsearch is a <a href="https://code.google.com/p/google-guice/">Guice</a> Module component that contributes configuration information and binds a specific implementation of the various interfaces of Elasticsearch. A standard Elasticsearch server without any plugins currently consists of over 100 modules:
</p>
<pre class="sourceCode bash prettyprint">$ &lt;span class=&quot;kw&quot;&gt;find&lt;/span&gt; src/main -name &lt;span class=&quot;dt&quot;&gt;\*&lt;/span&gt;Module&lt;span class=&quot;dt&quot;&gt;\*&lt;/span&gt; &lt;span class=&quot;kw&quot;&gt;|&lt;/span&gt; &lt;span class=&quot;kw&quot;&gt;grep&lt;/span&gt; -v common &lt;span class=&quot;kw&quot;&gt;|&lt;/span&gt; &lt;span class=&quot;kw&quot;&gt;wc&lt;/span&gt; -l
     &lt;span class=&quot;kw&quot;&gt;101&lt;/span&gt;
</pre>
<p>During startup, Elasticsearch collects different modules based on its configuration and runtime environment and creates what is called an <code>Injector</code>. To put it simply, an Injector is an object that can construct instances of classes without us providing the parameters required for their construction. The Injector will instead use its configured modules to locate all required dependencies and create them in a topologically sorted order for us. This both saves us a lot of time and assists us in creating a composable, modular system. And a system the size of Elasticsearch needs to be just that in order to be maintainable over time.
</p>
<p>In more technical terms, this is known as <a href="http://en.wikipedia.org/wiki/Dependency_injection">Dependency Injection</a>, which is explained shortly by <a href="http://www.jamesshore.com/Blog/Dependency-Injection-Demystified.html">James Shore</a> as:
</p>
<blockquote>
	Dependency injection means giving an object its instance variables
</blockquote>
<p>In the rest of this post we&#x2019;ll see how this makes Elasticsearch extremely pluggable and extendable. We start by taking a quick look at the HTTP side of Elasticsearch to see how Guice and dependency injection works in action.
</p>
</section><section id="http-server-modules" class="level2">
<h2><a href="#http-server-modules">HTTP Server Modules</a></h2>
<p>For example, Elasticsearch comes with a HTTP server, and to avoid coupling any component that works with HTTP operations in Elasticsearch, it has been abstracted to a <code>HttpServer</code> class, which during construction gets a <code>HttpServerTransport</code> that is by default bound to a concrete implementation called <code>NettyHttpServerTransport</code>. In other words, the <code>HttpServer</code> has no idea how the requests or responses are received and sent, it only has to deal with the application logic (i.e finding an appropriate request handler for incoming requests).
</p>
<p>If, down the line, Elasticsearch should want to switch HTTP layers due to any number of reasons such as performance, security, features etc, it&#x2019;s as simple as swapping out a configuration value (in this case <code>http.type</code>). As a matter of fact, <a href="http://www.sonian.com/">Sonian</a> has released a HTTP layer for Elasticsearch based on <a href="http://www.eclipse.org/jetty/">Jetty</a>, which does exactly this. It&#x2019;s available <a href="https://github.com/sonian/elasticsearch-jetty">here</a>. The Jetty-based implementation supports additional features like authentication, which might be a requirement in some deployments.
</p>
</section><section id="overview-with-namespaces" class="level2">
<h2><a href="#overview-with-namespaces">Overview with Namespaces</a></h2>
<p>Here&#x2019;s a bird&#x2019;s-eye view of modules and their namespaces:
</p>
<figure>
<img src="https://api.contentstack.io/v2/assets/575e4c990342dfd73826330b/download?uid=blt786bc4f193251b8e?uid=blt786bc4f193251b8e" data-sys-asset-uid="blta6b35672f304fa74" alt="Modules overview" style="max-width:100%;"><figcaption>Modules overview</figcaption>
</figure>
<p>In this overview, multiple modules are put into different namespaces according to their location in the source code tree or usage in the code. Different aspects of Elasticsearch, such as the REST interface, the plugins, rivers and transport service are separate entities with no module level dependencies between them. Modules that otherwise would have been too large or non-composable are further separated into smaller modules in nested namespaces.
</p>
</section><section id="all-modules" class="level2">
<h2><a href="#all-modules">All Modules</a></h2>
<p>The full overview of all modules is rather large. Click on the thumbnail below to see it in full size.
</p>
<p>Legend: <span style="color: orange">Namespaces</span> &#x2013; Module names &#x2013; <span style="color: green">Bound classes</span>
</p>
<p><a href="/assets/blte08c27067bffe10a/modules-all.png"><img src="https://api.contentstack.io/v2/assets/575e4c993dc542cb38c0848e/download?uid=blt764b0f4d6bad5280?uid=blt764b0f4d6bad5280" data-sys-asset-uid="blt6410aae6b17f9c7f" alt="All modules" style="max-width:100%;"></a>
</p>
</section><section id="spawning-new-modules" class="level2">
<h2><a href="#spawning-new-modules">Spawning New Modules</a></h2>
<p>Most modules only provide bindings for one or more classes or interfaces, but some modules spawn new modules when they&#x2019;re used. These modules are usually conditionally spawned depending on the current <code>Settings</code> object, which is read from the <code>elasticsearch.yml</code> configuration file during start-up. This enables plugin authors to write plugins that replace or extend built-in functionality in Elasticsearch that can be enabled and configured via the default configuration system.
</p>
<p>One concrete example of this, is the <a href="https://github.com/elastic/elasticsearch/blob/master/core/src/main/java/org/elasticsearch/discovery/DiscoveryModule.java">Discovery Module</a>, which by default uses either the <a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-discovery-zen.html">Zen</a> or <a href="https://github.com/elastic/elasticsearch/blob/master/core/src/main/java/org/elasticsearch/discovery/DiscoveryModule.java">Local</a> discovery modules, but may have its implementation swapped by setting the <code>discovery.type</code> configuration value to the canonical name of another module. This is exactly how Sonian&#x2019;s <a href="https://github.com/sonian/elasticsearch-zookeeper">ZooKeeper Discovery</a> is used.
</p>
</section><section id="extending-and-swapping-implementations" class="level2">
<h2><a href="#extending-and-swapping-implementations">Extending and Swapping Implementations</a></h2>
<p>While it&#x2019;s a rather large task to completely swap out entire implementations of components in Elasticsearch, these modules provide us with an easy way of adding functionality to Elasticsearch. Since all classes instantiated by the module system in Elasticsearch may request a reference to any other class instanted by the same system, it&#x2019;s almost trivial to extend the functionality via well defined interfaces.
</p>
<p>One of the most important modules for many developers working with Elasticsearch and Lucene is the <code>AnalysisModule</code>. The Analysis module is responsible for providing the analyzers, tokenizers, character filters and token filters used by Elasticsearch during indexing and searching. Using the API provided by the Analysis module, it&#x2019;s trivial to add any Lucene compatible analyzer, tokenizer or filter from a plugin to Elasticsearch with just a few lines of code.
</p>
</section><section id="resources" class="level2">
<h2><a href="#resources">Resources</a></h2>
<p>The overviews were created with <a href="http://freemind.sourceforge.net/wiki/index.php/Main_Page">FreeMind</a>, and the source files can be downloaded <a href="/assets/bltf3c37cafc99dbe5f/modules-all.mm">here</a>.
</p>
</section></body></html>

Elasticsearch Internals: an Overview

<p><strong>UPDATE:</strong> This article refers to our hosted Elasticsearch offering by an older name, Found. Please note that Found is now known as Elastic Cloud.</p><p>This article will give an introduction to the mapping feature of Elasticsearch. We'll define the key terms and take a closer look at what mapping is, when we specify it, how it is structured and how we can apply it to our data.</p><section id="what-is-a-schema" class="level2"><h2><a href="#what-is-a-schema">What is a Schema?</a></h2><p>A <em>schema</em> is a description of one or more fields that describes the document type and how to handle the different fields of a document.</p><p>The schema in Elasticsearch is a mapping that describes the fields in the JSON documents along with their data type, as well as how they should be indexed in the Lucene indexes that lie under the hood. Because of this, in Elasticsearch terms, we usually call this schema a “mapping”.</p><p>Conceptually, an Elasticsearch server contains zero or more indexes. An index is a container for zero or more types, which in turn has zero or more documents. To put it another way: a document has an identifier, belongs to a type, which belongs to an index. The figure below shows the documents A, B, X and Y of the type my_type, inside the index my_index.</p><p><img src="https://api.contentstack.io/v2/assets/575e4c8b0342dfd7382630ac/download?uid=bltf30eccd262213fec?uid=bltf30eccd262213fec" data-sys-asset-uid="blt4852179309023a90" style="max-width: 100%;" alt="Indexes, types and documents"/></p><figcaption>Indexes, types and documents</figcaption><p>The type called another_type and the index called another is shown in order to emphasize that Elasticsearch is multi-tenant, by which we mean that a single server can store multiple indexes and multiple types.</p><p>In the Elasticsearch documentation and related material, we often see the term “mapping type”, which is actually the name of the type inside the index, such as my_type and another_type in the figure above. When we talk about types in Elasticsearch, it is usually this definition of type. It is not to be confused with the type key inside each mapping definition that determines how the data inside the documents are handled by Elasticsearch.</p></section><section id="when-to-specify-a-custom-mapping" class="level2"><h2><a href="#when-to-specify-a-custom-mapping">When to Specify a Custom Mapping</a></h2><p>Elasticsearch has the ability to be schema-less, which means that documents can be indexed without explicitly providing a schema.</p><p>If you do not specify a mapping, Elasticsearch will by default <a href="https://www.elastic.co/guide/en/elasticsearch/reference/2.3/mapping.html">generate one dynamically</a> when detecting new fields in documents during indexing. However, this dynamic mapping generation comes with a few caveats:</p><ol type="1"><li>Detected types might not be correct.</li><li>May lead to unnecessary duplication. (The <a href="https://www.elastic.co/guide/en/elasticsearch/reference/2.3/mapping-source-field.html">_source field</a> and <a href="https://www.elastic.co/guide/en/elasticsearch/reference/2.3/mapping-all-field.html#custom-all-fields">_all field</a> especially.)</li><li>Uses default analyzers and settings for indexing and searching.</li></ol><p>For example, a timestamp is often represented in JSON as a long, but Elasticsearch will be unable to detect the field as a date field, preventing date filters and facets such as <a href="https://www.elastic.co/guide/en/elasticsearch/reference/2.3/search-aggregations-bucket-datehistogram-aggregation.html">the date histogram facet</a> from working properly.</p><p>By explicitly specifying the schema, we can avoid these problems.</p></section><section id="what-does-a-mapping-look-like" class="level2"><h2><a href="#what-does-a-mapping-look-like">What Does a Mapping Look Like?</a></h2><p>The mapping is usually provided to Elasticsearch as JSON, and is a hierarchically structured format where the root is the name of the type the mapping applies to.</p><section id="the-mapping-root" class="level3"><h3><a href="#the-mapping-root">The Mapping Root</a></h3><p>At the root level of the mapping, right under the type name, Elasticsearch supports a few “special” fields to configure how we should treat metadata that is not part of the document being posted, such as its type, id, size and its <a href="https://www.elastic.co/guide/en/elasticsearch/reference/2.3/mapping-all-field.html#custom-all-fields">fallback _all field</a>. For an extensive list of supported “special” fields, see the list of “fields” on the right-hand side of <a href="/guide/reference/mapping">Mapping Reference</a>.</p><p>The <a href="https://www.elastic.co/guide/en/elasticsearch/reference/2.3/mapping.html">root object</a> can also have a few extra attributes, which set the default indexing and search analyzers for the type, the date formats for automatically parsing dates in the type, and dynamic templates - we’ll revisit these in a future article. Apart from the fields mentioned above, there is no difference between the root level and the other mapping levels for a nested JSON document.</p></section><section id="hierarchical-levels" class="level3"><h3><a href="#hierarchical-levels">Hierarchical Levels</a></h3><p>Each level usually defines a properties setting, which maps the keys of the document at that level to its mapping. This structure is hierarchical, which means that every level down to the leaf nodes may include properties settings for its child values. To visualize it, consider the following document and its mapping:</p><p>Document:</p><pre class="sourceCode json prettyprint">{<br />    "name": {<br />        "first": "John"<br />    }<br />}<br /></pre><p>Mapping:</p><pre class="sourceCode json prettyprint">{<br />  "my_type" : {<br />    "properties" : {<br />      "name" : {<br />        "properties" : {<br />          "first" : {<br />            "type" : "string"<br />          }<br />        }<br />      }<br />    }<br />  }<br />}<br /></pre><p>While the mapping is slightly more convoluted than the document, the structure of the mapping clearly follows the structure of the document, with the added “properties”-nodes.</p></section><section id="the-type-key" class="level3"><h3><a href="#the-type-key">The type Key</a></h3><p>In the above example, we see that the document field name.first differs from the rest of the structure in that it defines a type. The type key is used on the leaf levels to tell Elasticsearch how to handle the field at the given level in the document. If the type key is omitted, as in the case of non-leaf types, Elasticsearch assumes it is of the <a href="https://www.elastic.co/guide/en/elasticsearch/reference/2.3/mapping.html">object type</a>.</p><p>The string type is one of the built-in <a href="/guide/reference/mapping/core-types">core types</a>, and Elasticsearch comes with support for many different types, such as <a href="/guide/reference/mapping/geo-point-type.html">geo_point</a> and <a href="https://www.elastic.co/guide/en/elasticsearch/reference/2.3/mapping.html">ip</a>, which can be used to effectively index and search geographical locations and IPv4 addresses respectively. Using the <a href="/guide/reference/mapping/multi-field-type.html">multi_field</a> type, we can even index a single document field into multiple virtual fields. We’ll elaborate on this in a future article.</p></section></section><section id="how-to-provide-a-mapping" class="level2"><h2><a href="#how-to-provide-a-mapping">How to Provide a Mapping</a></h2><p>There are two ways of providing a mapping to Elasticsearch. The most common way is during <a href="/guide/reference/api/admin-indices-create-index.html">index creation</a>:</p><pre class="sourceCode json prettyprint">curl -XPOST ...:9200/my_index -d '{<br />    "settings" : {<br />        # .. index settings<br />    },<br />    "mappings" : {<br />        "my_type" : {<br />            # mapping for my_type<br />        }<br />    }<br />}'<br /></pre><p>Another way of providing the mapping is using the <a href="/guide/reference/api/admin-indices-put-mapping">Put Mapping API</a>.</p><pre class="sourceCode json prettyprint">$ curl -XPUT 'http://localhost:9200/my_index/my_type/_mapping' -d '<br />{<br />    "my_type" : {<br />        # mapping for my_type<br />    }<br />}<br />'<br /></pre><p>Note that the type (my_type) is duplicated in the request path and the request body.</p><p>This API enables us to update the mapping for an already existing index, but with some limitations with regards to potential conflicts. New mapping definitions can be added to the existing mapping, and existing types may have their configuration updated, but changing the types is considered a conflict and is not accepted. It is, however, possible to pass ignore_conflicts=true as a parameter to the Mapping API, but doing so does not guarantee producing the expected result, as already indexed documents are not re-indexed automatically with the new mapping.</p><p>Because of this, specifying the mapping during creation of the indexes is recommended over using the Put Mapping API in most cases.</p></section><section id="closing-remarks" class="level2"><h2><a href="#closing-remarks">Closing Remarks</a></h2><p>I have now introduced you to the schema/mapping in Elasticsearch and demonstrated how the mapping is a hierarchical definition of data types. In a later article, I will go into more detail about <a href="/blog/found-mapping-workflow">a workflow I use when I explore new datasets with Elasticsearch</a>.</p></section>

An Introduction to Elasticsearch Mapping

<html><head></head><body><p><strong>UPDATE:</strong> This article refers to our hosted Elasticsearch offering by an older name, Found. Please note that Found is now known as Elastic Cloud.</p><h1>WARNING: This article contains outdated information. We no longer recommend taking its advice.</h1><p>Using plugins, it&apos;s possible to add new functionality to Elasticsearch without having to create a fork of Elasticsearch itself. In this article, we will go through the steps required to create a new Elasticsearch plugin from the ground up.
</p><section id="set-up-a-project-build-structure" class="level2">
<h2><a href="#set-up-a-project-build-structure">Set Up a Project Build Structure</a></h2>
<p>The only requirements to build Elasticsearch plugins are:
</p>
<ol type="1">
	<li>a working <a href="http://maven.apache.org/">Maven</a> installation</li>
	<li>a Java compiler</li>
</ol>
<p>A packaged plugin is simply a Zip file that contains one or more Java Jar files with compiled code and resources. Once a plugin is written and packaged, it can easily be added to any Elasticsearch installation in a single command.
</p>
<p>We start by creating the necessary plugin structure for our example plugin:
</p>
<pre class="sourceCode bash prettyprint">$ mkdir example-plugin
$ cd example-plugin
$ mkdir -p src/main/{java,resources,assemblies}
$ mkdir -p src/main/java/org/elasticsearch/plugin/example
</pre>
<p>Next, we need a Maven configuration file, which by convention goes into <code>pom.xml</code>:
</p>
<pre class="sourceCode xml prettyprint">&lt;modelVersion&gt;4.0.0&lt;/modelVersion&gt;
&lt;groupId&gt;org.elasticsearch.plugin.example&lt;/groupId&gt;
&lt;artifactId&gt;example-plugin&lt;/artifactId&gt;
&lt;version&gt;1.0-SNAPSHOT&lt;/version&gt;
&lt;packaging&gt;jar&lt;/packaging&gt;
&lt;properties&gt;
    &lt;elasticsearch.version&gt;0.90.3&lt;/elasticsearch.version&gt;
&lt;/properties&gt;
&lt;dependencies&gt;
    &lt;dependency&gt;
        &lt;groupId&gt;org.elasticsearch&lt;/groupId&gt;
        &lt;artifactId&gt;elasticsearch&lt;/artifactId&gt;
        &lt;version&gt;${elasticsearch.version}&lt;/version&gt;
        &lt;scope&gt;compile&lt;/scope&gt;
    &lt;/dependency&gt;
&lt;/dependencies&gt;
&lt;build&gt;
    &lt;plugins&gt;
        &lt;plugin&gt;
            &lt;artifactId&gt;maven-assembly-plugin&lt;/artifactId&gt;
            &lt;version&gt;2.3&lt;/version&gt;
            &lt;configuration&gt;
                &lt;appendAssemblyId&gt;false&lt;/appendAssemblyId&gt;
                &lt;outputDirectory&gt;${project.build.directory}/releases/&lt;/outputDirectory&gt;
                &lt;descriptors&gt;
                    &lt;descriptor&gt;${basedir}/src/main/assemblies/plugin.xml&lt;/descriptor&gt;
                &lt;/descriptors&gt;
            &lt;/configuration&gt;
            &lt;executions&gt;
                &lt;execution&gt;
                    &lt;phase&gt;package&lt;/phase&gt;
                    &lt;goals&gt;
                        &lt;goal&gt;single&lt;/goal&gt;
                    &lt;/goals&gt;
                &lt;/execution&gt;
            &lt;/executions&gt;
        &lt;/plugin&gt;
    &lt;/plugins&gt;
&lt;/build&gt;
</pre>
<p>While most parts of the Maven configuration file above is boilerplate, defining some required properties and configuring dependencies, the interesting part in this case is the reference to <code>src/main/assemblies/plugin.xml</code>, which we use to configure the packaging of the plugin:
</p>
<pre class="sourceCode xml prettyprint">&lt;?xml version=&quot;1.0&quot;?&gt;
&lt;assembly&gt;
    &lt;id&gt;plugin&lt;/id&gt;
    &lt;formats&gt;
        &lt;format&gt;zip&lt;/format&gt;
    &lt;/formats&gt;
    &lt;includeBaseDirectory&gt;false&lt;/includeBaseDirectory&gt;
    &lt;dependencySets&gt;
        &lt;dependencySet&gt;
            &lt;outputDirectory&gt;/&lt;/outputDirectory&gt;
            &lt;useProjectArtifact&gt;true&lt;/useProjectArtifact&gt;
            &lt;useTransitiveFiltering&gt;true&lt;/useTransitiveFiltering&gt;
            &lt;excludes&gt;
                &lt;exclude&gt;org.elasticsearch:elasticsearch&lt;/exclude&gt;
            &lt;/excludes&gt;
        &lt;/dependencySet&gt;
    &lt;/dependencySets&gt;
&lt;/assembly&gt;
</pre>
</section><section id="letting-elasticsearch-know-about-our-plugin" class="level2">
<h2><a href="#letting-elasticsearch-know-about-our-plugin">Letting Elasticsearch Know About Our Plugin</a></h2>
<p>While the above two files is enough to get the plugin build process working, we need to tell Elasticsearch about our plugin, which is done by adding the name of the plugin to a special <code>es-plugin.properties</code> -file on the classpath. We do this by creating <code>src/main/resourc/es-plugin.properties</code>:
</p>
<pre class="sourceCode ini prettyprint">plugin=org.elasticsearch.plugin.example.ExamplePlugin
</pre>
<p>When Elasticsearch starts up, the Elasticsearch <code>org.elasticsearch.plugins.PluginManager</code> will scan the current classpath looking for plugin configuration files and instantiate the referenced plugins. The <code>es-plugin.properties</code> file is a simple Java properties file which must contain the key <code>plugin</code>. The value should be a fully qualified class name for a compatible plugin class.
</p>
<p>A simple plugin extends <code>org.elasticsearch.plugins.AbstractPlugin</code> , which reduces boilerplate code, and may look like this:
</p>
<pre class="sourceCode java prettyprint">package org.elasticsearch.plugin.example;
import org.elasticsearch.plugins.AbstractPlugin;
public class ExamplePlugin extends AbstractPlugin { @Override public String name() { return &quot;example-plugin&quot;; }
@Override public String description() { return &quot;Example Plugin Description&quot;; }
}
</pre>
<p>The name is used in Elasticsearch to identify the plugin, for example when printing the list of loaded plugins. The description is not currently used for anything, but might be used in a potential future plugin management or information API.
</p>
</section><section id="a-working-plugin" class="level2">
<h2><a href="#a-working-plugin">A Working Plugin</a></h2>
<p>The four files we just created are actually sufficient to create an Elasticsearch plugin. If you followed the above steps, your directory tree should look something like this:
</p>
<pre class="prettyprint">&#x2502;
&#x251C;&#x2500;&#x2500; pom.xml
&#x251C;&#x2500;&#x2500; src
&#x2502;   &#x2514;&#x2500;&#x2500; main
&#x2502;       &#x251C;&#x2500;&#x2500; assemblies
&#x2502;       &#x2502;   &#x2514;&#x2500;&#x2500; plugin.xml
&#x2502;       &#x251C;&#x2500;&#x2500; java
&#x2502;       &#x2502;   &#x2514;&#x2500;&#x2500; org
&#x2502;       &#x2502;       &#x2514;&#x2500;&#x2500; elasticsearch
&#x2502;       &#x2502;           &#x2514;&#x2500;&#x2500; plugin
&#x2502;       &#x2502;               &#x2514;&#x2500;&#x2500; example
&#x2502;       &#x2502;                   &#x251C;&#x2500;&#x2500; ExamplePlugin.java
&#x2502;       &#x2514;&#x2500;&#x2500; resources
&#x2502;           &#x2514;&#x2500;&#x2500; es-plugin.properties
&#x2502;
</pre>
<p>Being Maven-based, most Java IDEs such as <a href="http://www.jetbrains.com/idea/">IDEA</a> and <a href="http://www.eclipse.org/">Eclipse</a> are able to read the project definition from the <code>pom.xml</code>-file and automatically configure a project and download the required dependencies.
</p>
<p>In the next section we will show how to build the plugin from the command line and how to get it installed.
</p>
</section><section id="building-and-installing-the-plugin" class="level2">
<h2><a href="#building-and-installing-the-plugin">Building and Installing the Plugin</a></h2>
<p>Before distributing and using the plugin, it has to be assembled, which is done via Maven:
</p>
<pre class="sourceCode bash prettyprint">$ mvn package
</pre>
<p>The above command assembles our plugin package into a single Zip file that can be installed using the Elasticsearch <code>plugin</code> command:
</p>
<pre class="sourceCode bash prettyprint">$ bin/plugin --url file:///PATH-TO-EXAMPLE-PLUGIN/target/releases/example-plugin-1.0-SNAPSHOT.zip --install example-plugin
</pre>
<p>Note that we use the <code>--url</code> option for the <code>plugin</code> command in order to inform it to get the file locally instead of trying to download it from an online repository.
</p>
<p>We can now start Elasticsearch and see that our plugin gets loaded:
</p>
<pre class="sourceCode bash prettyprint">$ bin/elasticsearch -f
[2013-09-04 17:33:27,443][INFO ][node                     ] [Andrew Chord] version[0.90.3], pid[67218], build[5c38d60/2013-08-06T13:18:31Z]
[2013-09-04 17:33:27,443][INFO ][node                     ] [Andrew Chord] initializing ...
[2013-09-04 17:33:27,455][INFO ][plugins                  ] [Andrew Chord] loaded [example-plugin], sites []
</pre>
<p>However, since our plugin doesn&#x2019;t actually do anything, there&#x2019;s very little proof that we&#x2019;ve accomplished something. The next section demonstrates how to return a simple HTTP response to the built-in HTTP server.
</p>
</section><section id="example-saying-hello" class="level2"><h2><a href="#example-saying-hello">Example: Saying Hello</a></h2><p>Plugins can add functionality to Elasticsearch in many different ways. In this section we&#x2019;ll show how to use <a href="https://code.google.com/p/google-guice/">Guice</a> to add a Module to Elasticsearch. Guice is used in Elasticsearch to wire together all the components of the server and to provide well-documented entry points into the server. Describing Guice is out of scope for this article and interested readers should have no problem finding multiple tutorials and introductions to the <a href="https://code.google.com/p/google-guice/wiki/Motivation">Guice framework</a>.
</p><section id="adding-a-new-module" class="level3">
<h3><a href="#adding-a-new-module">Adding a New Module</a></h3>
<p>To add a new Module Elasticsearch, we override the abstract <code>modules()</code> method on <code>AbstractPlugin</code>, which allows us to add multiple Modules by returning a reference to their class in a collection. We start by adding a new Module to our <code>ExamplePlugin</code>:
</p>
<pre class="sourceCode java prettyprint">package org.elasticsearch.plugin.example;
import org.elasticsearch.common.collect.Lists; import org.elasticsearch.common.inject.Module; import org.elasticsearch.plugins.AbstractPlugin;
import java.util.Collection;
public class ExamplePlugin extends AbstractPlugin { @Override public String name() { return &quot;example-plugin&quot;; }
@Override public String description() { return &quot;Example Plugin Description&quot;; } @Override public Collection&lt;Class&lt;? extends Module&gt;&gt; modules() { Collection&lt;Class&lt;? extends Module&gt;&gt; modules = Lists.newArrayList(); modules.add(ExampleRestModule.class); return modules; }
}
</pre>
<p>These modules are instantiated by Elasticsearch during initialization, and their <code>configure()</code> method is invoked. In this method, we have to make sure our <code>RestHandler</code> gets added to the list of classes to instantiate:
</p>
<pre class="sourceCode java prettyprint">package org.elasticsearch.plugin.example;
import org.elasticsearch.common.inject.AbstractModule;
public class ExampleRestModule extends AbstractModule { @Override protected void configure() { bind(HelloRestHandler.class).asEagerSingleton(); } }
</pre>
<p>By binding the class to itself as an eager singleton, we inform the framework that we would like it to instantiate just a single instance of the <code>HelloRestHandler</code> class. The class is instantiated by invoking the constructor annotated with <code>@Inject</code> once all its constructor arguments have been instantiated.
</p>
</section><section id="creating-the-http-handler" class="level3">
<h3><a href="#creating-the-http-handler">Creating the HTTP Handler</a></h3>
<p>In Elasticsearch, all <a href="http://en.wikipedia.org/wiki/Representational_state_transfer">REST</a> requests are handled by a <code>org.elasticsearch.rest.RestController</code> instance, which maintains an internal routing of paths to <code>org.elasticsearch.rest.RestHandler</code> instances, which in turn handle the requests.
</p>
<p>Adding support for a new HTTP request is as simple as obtaining a reference to the <code>RestController</code> via Guice and registering the handler with the controller.
</p>
<p>Our <code>HelloRestHandler</code> should look like this:
</p>
<pre class="sourceCode java prettyprint">package org.elasticsearch.plugin.example;
import org.elasticsearch.rest.*;
import org.elasticsearch.common.inject.Inject;
import static org.elasticsearch.rest.RestRequest.Method.GET; import static org.elasticsearch.rest.RestStatus.OK;
public class HelloRestHandler implements RestHandler { @Inject public HelloRestHandler(RestController restController) { restController.registerHandler(GET, &quot;/_hello&quot;, this); }
@Override public void handleRequest(final RestRequest request, final RestChannel channel) { String who = request.param(&quot;who&quot;); String whoSafe = (who!=null) ? who : &quot;world&quot;; channel.sendResponse(new StringRestResponse(OK, &quot;Hello, &quot; + whoSafe + &quot;!&quot;)); }
}
</pre>
</section><section id="building-and-installing-the-updated-plugin" class="level3"><h3><a href="#building-and-installing-the-updated-plugin">Building and Installing the Updated Plugin</a></h3><p>Now the folder structure should look like this:
</p><pre class="prettyprint">&#x2502;
&#x251C;&#x2500;&#x2500; pom.xml
&#x251C;&#x2500;&#x2500; src
&#x2502;   &#x2514;&#x2500;&#x2500; main
&#x2502;       &#x251C;&#x2500;&#x2500; assemblies
&#x2502;       &#x2502;   &#x2514;&#x2500;&#x2500; plugin.xml
&#x2502;       &#x251C;&#x2500;&#x2500; java
&#x2502;       &#x2502;   &#x2514;&#x2500;&#x2500; org
&#x2502;       &#x2502;       &#x2514;&#x2500;&#x2500; elasticsearch
&#x2502;       &#x2502;           &#x2514;&#x2500;&#x2500; plugin
&#x2502;       &#x2502;               &#x2514;&#x2500;&#x2500; example
&#x2502;       &#x2502;                   &#x251C;&#x2500;&#x2500; ExamplePlugin.java
&#x2502;       &#x2502;                   &#x251C;&#x2500;&#x2500; ExampleRestModule.java
&#x2502;       &#x2502;                   &#x2514;&#x2500;&#x2500; HelloRestHandler.java
&#x2502;       &#x2514;&#x2500;&#x2500; resources
&#x2502;           &#x2514;&#x2500;&#x2500; es-plugin.properties
&#x2502;
</pre><p>We can proceed to re-package our plugin, update Elasticsearch and verify that the plugin is working. Build the package by using Maven in the <code>example-plugin</code> folder:
</p><pre class="sourceCode bash prettyprint">$ mvn package
</pre><p>To update the installed plugin in Elasticsearch we have to stop Elasticsearch, uninstall the plugin, reinstall the plugin and restart Elasticsearch:
</p><pre class="sourceCode bash prettyprint">$ bin/plugin --remove example-plugin $ bin/plugin --url file:///PATH-TO-EXAMPLE-PLUGIN/target/releases/example-plugin-1.0-SNAPSHOT.zip --install example-plugin $ bin/elasticsearch -f
</pre><p>Using another terminal, we can now confirm that the plugin works as expected:
</p><pre class="sourceCode bash prettyprint">$ curl localhost:9200/hello Hello, world! $ curl localhost:9200/hello?who=Elasticsearch Hello, Elasticsearch!
</pre></section></section><section id="conclusion" class="level2">
<h2><a href="#conclusion">Conclusion</a></h2>
<p>In this article, we&#x2019;ve shown how to create a new Elasticsearch plugin from scratch. While it&#x2019;s relatively easy to get started writing plugins for Elasticsearch, writing more sophisticated plugins usually requires some insight into the current <a href="https://github.com/elasticsearch/elasticsearch">Elasticsearch code base</a>.
</p>
<p>Update: Added <code>&lt;appendAssemblyId&gt;false&lt;/appendAssemblyId&gt;</code> in pom.xml to exclude -plugin suffix in file name. This works better when publishing the plugin to a Maven repository.
</p>
</section></body></html>

Writing an Elasticsearch Plugin: Getting Started

Start free trial

Blog Footer CTA

Sign up for Elastic Cloud free trial

Version 1

Software Engineer

Njal Karevoll

<p>By submitting you acknowledge that you've read and agree to our <a href="/legal/elastic-cloud-account-terms" target="_self">Terms of Service</a>, and you agree to receive the <a href="/legal/privacy-statement#how-we-use-the-information" target="_self">selected communications</a> at the contact details you provided above. See <a href="/legal/privacy-statement/" target="_self">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.</p>

Communication Preferences - ONLY for /communication-preferences

<p>Durch das Absenden erklären Sie, dass Sie unsere <a href="/de/legal/elastic-cloud-account-terms" target="_self">Nutzungsbedingungen</a> gelesen haben und mit ihnen einverstanden sind und dass Sie Elastic erlauben, <a href="/de/legal/privacy-statement#how-we-use-the-information" target="_self">ausgewählte Nachrichten und Informationen</a> an die oben von Ihnen bereitgestellten Kontaktangaben zu senden. Weitere Informationen, darunter auch dazu, wie Sie Ihre Einwilligung zurückziehen können, finden Sie in der <a href="/de/legal/privacy-statement/" target="_self">Datenschutzerklärung von Elastic</a>.</p>

<p>By submitting you acknowledge that you've read and agree to our <a href="/legal/serverless-preview-terms" target="_blank">Terms of Service</a>, and that Elastic may <a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">contact you</a> about our related products and services, using the details you provide above. See <a href="/legal/privacy-statement/" target="_blank">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.</p>

Serverless GDPR Text

<p>Durch das Absenden erklären Sie, dass Sie unsere <a href="/de/legal/serverless-preview-terms" target="_self">Nutzungsbedingungen</a> gelesen haben und mit ihnen einverstanden sind und dass Sie Elastic erlauben, bezüglich unserer zugehörigen Produkte und Dienstleistungen und unter Nutzung der von Ihnen oben bereitgestellten Kontaktangaben <a href="/de/legal/privacy-statement#how-we-use-the-information" target="_self">mit Ihnen in Kontakt zu treten</a>. Weitere Informationen, darunter auch dazu, wie Sie Ihre Einwilligung zurückziehen können, finden Sie in der <a href="/de/legal/privacy-statement/" target="_self">Datenschutzerklärung von Elastic</a>.</p>

<p>By submitting you agree to <a href="/agreements/global/cloud-services-monthly" rel="nofollow">Elastic Cloud Terms of Service</a>. Your personal data will be processed in accordance with <a href="/legal/privacy-statement">Elastic's Privacy Statement</a>.</p>

Cloud Trial GDPR Text

<p>Beim Absenden akzeptieren Sie die <a href="/de/agreements/global/cloud-services-monthly" rel="nofollow">Nutzungsbedingungen von Elastic Cloud</a>. Die Verarbeitung Ihrer personenbezogenen Daten unterliegt der <a href="/de/legal/privacy-statement">Datenschutzerklärung von Elastic</a>.
</p>

<p>By submitting you acknowledge that you've read and agree to our <a href="/legal/terms-of-use" target="_blank">Terms of Service</a>, and that Elastic may <a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">contact you</a> about our related products and services, using the details you provide above. See <a href="/legal/privacy-statement/" target="_blank">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.</p>

Newsletter GDPR Text

<p>Durch das Absenden erklären Sie, dass Sie unsere <a href="/de/legal/terms-of-use" target="_self">Nutzungsbedingungen</a> gelesen haben und mit ihnen einverstanden sind und dass Sie Elastic erlauben, bezüglich unserer zugehörigen Produkte und Dienstleistungen und unter Nutzung der von Ihnen oben bereitgestellten Kontaktangaben <a href="/de/legal/privacy-statement#how-we-use-the-information" target="_self">mit Ihnen in Kontakt zu treten</a>. Weitere Informationen, darunter auch dazu, wie Sie Ihre Einwilligung zurückziehen können, finden Sie in der <a href="/de/legal/privacy-statement/" target="_self">Datenschutzerklärung von Elastic</a>.</p>

Explore similar demos

Overview

Speakers

Close

See more insights

The content on this page is not available in the selected language. As Elastic grows globally, we continue to support content in multiple languages.

The content on this page is not available in the selected language.

Der Inhalt dieser Seite ist in der ausgewählten Sprache nicht verfügbar. Wir bei Elastic arbeiten daran, die bereitgestellten Inhalte in verschiedenen Sprachen anzubieten. Bis dahin bitten wir Sie um etwas Geduld und hoffen auf Ihr Verständnis!

Author

Watch now (no PT)

Watch now

See all top stories

All (no PT translation)

Contact information

Press Release

Share on Reddit

Articles by

Share this story

Share by Email

Thank you for your interest!

Contact Info

Follow us on Youtube

Load More

Share on LinkedIn

Follow us on LinkedIn

Search Integrations

All Solutions

Video for

Follow us on Twitter

See when this webinar starts in my time zone

Register to Watch

Register now

See All Posts

Can't make it? Register and we'll send you the recording. You'll also receive an email with related content

Sie können nicht dabei sein? Registrieren Sie sich, damit wir Ihnen die Aufzeichnung zukommen lassen können. Außerdem erhalten Sie eine E‑Mail mit Begleitmaterial.

Small image for

On-demand webinar

Featured webinar

Register to Attend

Share on Facebook

Reset all

Upcoming webinar

View more posts

Print

Hosted by

Sign In to Attend

View next

Follow us on Facebook

Share

Highlights

Learn More

Read less

You'll also receive an email with related content

Wir schicken Ihnen zudem relevante Informationen, die von Interesse sein könnten.

Thank you for registering. We will send you a confirmation email soon.

Vielen Dank für Ihre Registrierung. Sie erhalten in Kürze eine Bestätigungs-E‑Mail von uns.

Author

Articles by Njal Karevoll

Software Engineer, Elastic

A Dive into the Elasticsearch Storage

Interfacing with Elasticsearch: Picking a Client

Document Processing and Elasticsearch

Scripting with Elasticsearch

Elasticsearch Plugin Types

Managing Elasticsearch Fields When Searching

A Data Exploration Workflow for Mappings

Elasticsearch Internals: Networking Introduction

Elasticsearch Internals: an Overview