The Elastic Stack is a far-reaching ecosystem with many interconnected parts, and the <a href="https://forge.puppet.com/elastic">official Puppet Modules</a> are no exception when it comes to the configuration management side of the house. Today, we want to highlight some developments taking place in the Elastic Puppet ecosystem to help spread awareness of new features and upcoming changes.<h3 id="modules-roadmap">Modules Roadmap</h3>First, some housekeeping items that pertain to all of our official modules.<h4 id="puppet-3">Puppet 3</h4>The most prominent upcoming change is the deprecation of Puppet 3 (if you’re skimming this post, don’t panic! You still have time). As noted in the upstream Puppet Labs page, version 3 met its end in December 2016, which means that it is not only deprecated, but end-of-life, and past the cutoff date for any future updates. As systems operators ourselves, we recognize that keeping stride with software versions is one of the challenges that comes with the discipline, particularly when it comes to large enterprises.With that in mind, we’ll be keeping version 3.8 of Puppet in our list of supported versions for a period to ensure everyone has a window of time to work in, for those cases where upgrading is a challenge. Nevertheless, the Puppet ecosystem is moving on from version 3, and we can only hold on to old versions for so long. It’s for this reason that we will be removing support for Puppet 3 with the 6.x release of the Elastic Stack, which grants many months of leeway for those operations engineers with a Puppet upgrade still on their plate. Version 6 of the&nbsp;Elastic Stack is likely to arrive in late 2017, and will be prefaced with additional announcements as its release draws closer.<h4 id="new-forge-namespace">New Forge Namespace</h4>What open source project doesn’t like changing names every so often? <a href="https://forge.puppet.com/modules?limit=25&sort_by=rank&module_groups=base%20pe_only&q=elastic">Our modules on the Puppet Forge</a> now properly live under the “elastic” namespace where all future module updates and release will land. While any references to old modules under the elasticsearch namespace will continue to work, the modules there are no longer updated. We suggest updating any automation or tooling that references released modules from the Forge to this new namespace (for example, to elastic/elasticsearch).<h3 id="official-puppet-kibana-module">Official Puppet Kibana Module</h3>One of the most common requests we hear from Puppet users is for other parts of our stack to ship official modules. At Elastic{ON} 2017 we released a beta version of <a href="https://forge.puppet.com/elastic/kibana">an official Puppet Kibana module</a>, which is now formally released at version 5.0 (major version synchronized with the rest of the Elastic Stack). The module supports everything you would expect from our Puppet modules, including Kibana plugin resources to make managing Kibana through Puppet a much easier experience.As always, we’re eager to hear from you regarding any bugs you've found or features you’d like to see and would encourage any bug reports or feature requests to be filed in the <a href="https://github.com/elastic/puppet-kibana">puppet-kibana GitHub repository</a>.<h3 id="elasticsearch-tasty-new-features">Elasticsearch: Tasty New Features</h3>The official Elasticsearch module hasn’t stood still either. In addition to the steady march of bugfixes, <a href="https://forge.puppet.com/elastic/elasticsearch">the official Elasticsearch module</a> supports a couple of useful new resource types: indices and pipelines.While the module already supported managing index templates (useful for auto-created indices from tools like Logstash or Beats), the elasticsearch::index type can help manage mappings and settings for existing indices in an Elasticsearch cluster. For example, to manage an Elasticsearch index called “logs” to retain only 1 replica per shard:<pre class="prettyprint">elasticsearch::index { 'logs': settings =&gt; { 'index' =&gt; { 'number_of_replicas' =&gt; 1 } } }</pre>Likewise, <a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/pipeline.html">pipelines</a> are now supported as well. The ability to codify processing pipelines in Puppet can help centralize your workflow for processing logs and documents in Elasticsearch (see our <a href="https://www.elastic.co/blog/new-way-to-ingest-part-1">other blog posts about ingest node pipelines for more introductory content</a>). For example, to create a pipeline that will automatically parse dates, the following Puppet resource could be used:<pre class="prettyprint">elasticsearch::pipeline { ‘parsedate’: content =&gt; { 'description' =&gt; 'Parse the date field into @timestamp', 'processors' =&gt; [{ 'date' =&gt; { 'field' =&gt; 'date', 'formats' =&gt; ['dd/MM/yyyy hh:mm:ss'], 'timezone' =&gt; 'US/Pacific' } }] } }</pre><h3 id="logstash-all-aboard-for-version-5">Logstash: All Aboard for Version 5</h3>Version 5 of Logstash brought huge improvements to the handling of the Logstash system service, adding full support for systemd, among other things. In the past, the Puppet module was responsible for much more of the work around service handling, which was a common source of problems as the module became progressively more complex in order to handle various edge-cases across distributions. With Logstash itself now handling the service, the Puppet module is much simpler and more stable when it comes to service management.It's important to note, however, that the changes in service management were so comprehensive that it became impractical for the module to support Logstash 2.x and Logstash 5.x at the same time. Thus, module version 5 only supports Logstash 5. Of course, the legacy module <a href="https://forge.puppet.com/elasticsearch/logstash">elasticsearch/logstash</a> is still available if you need to run Logstash 2.It's now possible to fully <a href="https://github.com/elastic/puppet-logstash#logstash-config-files">manage Logstash settings, startup options, and JVM options with the module</a>. Like this:<pre class="prettyprint">class { 'logstash': settings =&gt; { 'pipeline' =&gt; { 'batch' =&gt; { 'size' =&gt; 25, 'delay =&gt; 5, } } } }</pre>Since the settings are all defined as standard arrays and hashes, they work well with Hiera too.Plugin support is now comprehensive, with support for remote locations like puppet:// and http:// URLs. The latter is essential for people choosing to install the new <a href="https://www.elastic.co/guide/en/logstash/master/configuring-logstash.html" target="_self">X-Pack for Logstash</a>.Finally, a lot of progress on the module is something you probably won't see, but hopefully you get to see the results. All new features, including those above, are now developed under testing from a <a href="https://github.com/elastic/puppet-logstash/blob/master/spec/acceptance/class_logstash_spec.rb">re-written test suite</a>. The tests are there to ensure everything in the module works reliably, across every supported operating system and Puppet version.

wooden-mannequin-bleedimage.jpg

wooden-mannequin-thumb.jpg

Puppet State of the Union: May 2017

When I added rivers way back in time in the early days of Elasticsearch, the idea was somewhat novel. One of the first tasks that users do when using Elasticsearch is to get data into Elasticsearch to make it searchable, so why not allow our community to write plugins that can be installed directly on an Elasticsearch cluster&nbsp;to pull data into Elasticsearch automatically.The first few rivers implementations were quite successful and very helpful. The CouchDB river was immensely simple and popular (thanks to CouchDB changes API). Others were popular as well, for example, the RabbitMQ one. They did start to slowly show the problematic nature of rivers as well.What was the problem we were witnessing? Cluster stability. You see, by their nature, rivers deal with external systems, and those external systems require external libraries to work with. Those are great to use, but they come with an overhead. Part of it is built in overhead, things like additional memory usage, more sockets, file descriptors and so on. Others, sadly, are bugs.Part of our efforts in the past couple of years has been to improve Elasticsearch resiliency, and we kept seeing, time and time again, that rivers are a big cause for cluster instability, due to their inherent notion of working with external systems and external libraries. When Found joined us a couple of months ago, we found that they see the same thing, with rivers plugins causing most of the cluster instabilities across the thousands of clusters under management.We knew it for some time, and in the spirit of helping users build more resilient systems, we decided to deprecate rivers and ask users to focus on&nbsp;getting data to Elasticsearch from "outside" the cluster. Rivers are deprecated from 1.5 moving forward. We will probably keep the infrastructure around in 2.0, and only remove them at a later version, to ease the migration.The ease of use in getting data into Elasticsearch is still important though, so where should you go from here?For more than a year, we've had official client libraries for Elasticsearch in most programming languages. It means that hooking into your application and getting data through an existing codebase should be relatively simple. This technique also allows to easily munge the data before it gets to Elasticsearch. A common example is an application that already used an ORM to map the domain model to a database, and hooking and indexing the domain model back to Elasticsearch tends to be simple to implement. Also, Logstash, or similar tools, can be used to ship data into Elasticsearch. For example, some of the rivers Elasticsearch came with are now implemented as Logstash plugins (like the <a href="https://github.com/logstash-plugins/logstash-input-couchdb_changes">CouchDB one</a>) in the forthcoming&nbsp;Logstash 1.5. I love the community and work that has gone into a vibrant set of river plugins, and the decision to deprecate rivers was not taken lightly. What should you do if you are a river plugin author? Since rivers are relatively self-sufficient, the code can be extracted into a common library that can be used to get data into Elasticsearch. Then, this library can be used in various different places. A simple "main class" can be written to allow to execute it as a standalone process. Another option is to move the plugin to be a Logstash input.&nbsp;Logstash inputs are very simple to write, and in 1.5 the Logstash team has <a href="https://www.elastic.co/guide/en/logstash/current/input-plugins.html" target="_self">made the process</a> of writing, maintaining, and discovering plugins super simple. It is a hard decision to take something away, especially with all the effort that has gone into writing those river plugins by the wonderful authors. I deeply apologize for it, and we would love to help out with any questions and ideas on how to move forward adapting them. The issue for it is <a href="https://github.com/elastic/elasticsearch/issues/10345">#10345</a>.

Deprecating Rivers

<img src="https://api.contentstack.io/v2/assets/575e4c8f9e7a83165490c1d0/download?uid=blt0a3be37aa959b31f?uid=blt0a3be37aa959b31f" data-sys-asset-uid="blt140adad6df1b0753" style="float: right;margin: 0 0 0 20px;text-align: right;"/>Since the recently released 1.0.0-beta2 version, Packetbeat can understand the MongoDB wire protocol. This means you can now use Packetbeat and Elasticsearch to monitor the performance of your MongoDB servers. You can play with a live demo of the associated Kibana dashboard <a href="http://demo.elastic.co/packetbeat/#/dashboard/MongoDB-performance" target="_blank">here</a>.The way Packetbeat works is that it captures the network traffic, correlates the request with the response, and inserts a document into Elasticsearch describing each MongoDB query or operation seen on the wire. You can then use Kibana to discover errors or slow queries and to visualize things like “Top slowest MongoDB queries”, “Response time percentiles for a particular collection”, or the “Number of writes with unacknowledged write concern”. But more about these visualizations later, let’s start by giving credit where credit is due.<h2>Based on a community contribution</h2>The beauty of having a system like Packetbeat be open source is that anyone can add support for the protocols they need. By making it easy to add new protocols and encouraging community contributions, we’re not only extending Packetbeat’s list of supported protocols faster, but we benefit from having experts in each technology providing insights. The open source nature also means that folks can implement support for niche or proprietary protocols.While we hoped that people will start creating their own protocol modules for Packetbeat, we didn’t expect it to happen this quickly. We don’t have yet a proper developer guide for new protocols and the APIs are scarcely documented and may be inconsistent here and there. But all of&nbsp;this didn’t stop <a href="https://github.com/albanm">Alban Mouton</a> to submit the pull request for adding MongoDB support just a few days after we announced that <a href="https://www.elastic.co/blog/welcome-packetbeat-tudor-monica">Packetbeat has joined Elastic</a>. His pull request was well documented, contained unit and integration tests and even a simple <a href="https://github.com/elastic/packetbeat/blob/8f42d0fe0c20d65c8f6965fef695fd807ef5a6c1/SUPPORT_PROTOCOL.md">guide</a> to future protocol module writers.As if he didn’t do enough already, we asked Alban to contribute to this blog post as well. He sent us the following quote:"I am a enthusiastic user of Packetbeat, along with the ELK stack. The company I work for develops and integrates multiple services communicating mostly over HTTP. Tracing activity and performance at the network level is great because it separates us from the varying implementations of these services and makes system-wide analysis easy. We are at the beginning of our experience with Packetbeat but I have high hopes for the future.""Recently our main database has been MongoDB and I was looking for an opportunity to take a peek at the Go language, so here we are ! Thanks a lot to the Packetbeat team, Tudor Golubenco in particular, for playing the Open Source game so well and for bringing this rough contribution to release level."Thank you Alban for your excellent work and for leading the way to more community contributions!<h2>Mongo’s wire protocol</h2>Let’s do a quick dive in the details of the MongoDB wire protocol, so you get a better understanding of what data Packetbeat captures.It is a fairly friendly protocol to a passive decoder like Packetbeat. Messages and individual fields have lengths, so it is easy to jump over the fields that we don’t understood or we don’t need. There is little contextual state so it is easy to understand what is happening even when capturing only part of the conversion. Also, because a single serialization protocol is used in all messages (<a href="http://bsonspec.org/">BSON</a> - binary JSON), it was easy for us to leverage an existing Go library to decode them. Namely, we use the BSON implementation from the <a href="http://labix.org/mgo">mgo </a>driver, written by Gustavo Niemeyer. One thing to note is that prior to version 2.6, the write operations didn’t actually require a response from the server. By default, however, the drivers called the GetLastError method to acknowledge that the write was successful.&nbsp;<img src="https://api.contentstack.io/v2/assets/575e4c8fd8edd48f76935e07/download?uid=bltc7e6aa0b9b7cb888?uid=bltc7e6aa0b9b7cb888" data-sys-asset-uid="bltc0cf17784a88a8cc" style="max-width: 100%;"/>This was <a href="https://www.mongodb.com/docs/legacy/?site=docs" target="_self">changed in version 2.6</a>, which introduced a new wire protocol for insert, update and delete commands so that a response is always sent from the server and it doesn’t require a second command. Currently Packetbeat only supports the newer version of the protocol, so write commands from versions prior to 2.6 are not captured. We (or you!) can implement this in the future if there is demand.<img src="https://api.contentstack.io/v2/assets/575e4c8e12798d8e381fd6a9/download?uid=blt3aa2e7c3b65a7a11?uid=blt3aa2e7c3b65a7a11" data-sys-asset-uid="blt119ef9b785d07514" style="float: right;margin: 0px;text-align: right;"/>Another interesting aspect is that in a MongoDB cluster the same wire protocol is used between the three server types: mongos, config server and data shards servers. The same protocol is also used for replication between shard nodes. This means you can use Packetbeat for advanced troubleshooting of what is going on inside the cluster!To capture the traffic inside the cluster, you need to install Packetbeat not only on the routing nodes but also on the shard nodes. You likely also need to add port 27019 to the list of MongoDB ports, because this is used by the configuration nodes.<h2>Kibana dashboards</h2>As usual with the protocols we support, we also provide a sample Kibana <a href="http://demo.elastic.co/packetbeat/#/dashboard/MongoDB-performance" target="_blank">dashboard</a> to highlight some of the visualizations you can build based on the data Packetbeat collects and to give you a good starting point for creating your own dashboard. Let’s see some of the visualizations made possible by this data.<h3>MongoDB errors</h3>Packetbeat inspects automatically the response payload to check if it contains error. If it finds any, it sets the status field &nbsp;to error Error. This example shows the number of MongoDB errors evolution sliced by both the collection name and operation type.<img src="https://api.contentstack.io/v2/assets/575e4c8f9e7a83165490c1ad/download?uid=bltc9525251c2a385a3?uid=bltc9525251c2a385a3" data-sys-asset-uid="blt87269016986f5928" style="max-width: 100%;"/><h3>Total time spent per collection</h3>This example shows the total amount of time spent while querying each collection. This view adds to, for example, a 99th percentile visualization of response times because a high total count could be reached if there are slow queries but also if there are many small queries. <img src="https://api.contentstack.io/v2/assets/575e4c8f3dc542cb38c082c8/download?uid=blt5c6b1c3e14b78a0f?uid=blt5c6b1c3e14b78a0f" data-sys-asset-uid="blte488011987bacbf7" style="max-width: 100%;"/><h3>Response times and count per collection</h3>This one plots the 99th response time percentile on the Y axis and the dot size represents the number of queries in the particular time bucket. Different colors represent different collections. Just another way of visualizing performance data which can be helpful in spotting trends. In this screenshot, for example, you can notice the cyclicality of the response times.<img src="https://api.contentstack.io/v2/assets/575e4c8fd8edd48f76935deb/download?uid=blte0f1674c1eb8f622?uid=blte0f1674c1eb8f622" data-sys-asset-uid="bltc3a043c0784f87ae" style="max-width: 100%;"/><h3>Number of writes with unacknowledged write concern</h3>By inspecting the request payload and looking for the writeConcern parameter of write operations, we can plot the number of them that have an unacknowledged write concern. As always with Packetbeat and Kibana, you can drill down from this graph and find these exact transactions and who creates them. <img src="https://api.contentstack.io/v2/assets/575e4c8f0342dfd738263158/download?uid=blt008d9a2e43355a76?uid=blt008d9a2e43355a76" data-sys-asset-uid="blt78196b2628f62bd3" style="max-width: 100%;"/><h3 dir="ltr">Top slowest MongoDB queries</h3>Let’s end with a fairly obvious but very useful one: top slowest MongoDB queries by their 99th percentile.<img src="https://api.contentstack.io/v2/assets/575e4c8f43e9adc538713b6c/download?uid=blt4613a8b34c7b13d3?uid=blt4613a8b34c7b13d3" data-sys-asset-uid="blt93d87fc0dee5c910" style="max-width: 100%;"/><h3>Getting started with Packetbeat for MongoDB</h3>If you didn’t try Packetbeat before, we recommend reading and following the <a href="https://www.elastic.co/guide/en/beats/packetbeat/current/packetbeat-getting-started.html">getting started</a> guide. By default, Packetbeat doesn’t index payload information to avoid capturing sensitive data by mistake. But you can easily enable payload indexing from the configuration file. Here is an example configuration section showing the main options:<pre class="prettyprint">protocols: mongodb: ports: [27017, 27019] send_request: true # index the request payload send_response: true # index the response payload max_docs: 10 # maximum number of documents to index per request/response max_doc_length: 1024 # maximum document size to index </pre>Image credit: <a href="https://www.flickr.com/photos/kumaravel/8282786757">green leaf</a>

leaf_large.jpg

Monitoring MongoDB with Packetbeat and Elasticsearch

race_finish.jpg

race_finish_small.jpg

Benchmarking REST client and transport client

It’s been 4 months since <a href="https://github.com/elasticsearch/elasticsearch-net/releases/tag/0.12.0.0">NEST 0.12.0</a> was released and a lot has happened since then. Elasticsearch released 1.0 and even version 1.1. With this release announcement, I’m pleased to say NEST has finally caught up and brings (almost) all the awesome that Elasticsearch 1.0 and 1.1 bring to the .NET world. Even more exciting – as of today, Elasticsearch.Net and NEST are both officially supported .Net clients for Elasticsearch!(For those just learning about Elasticsearch.Net and NEST, Elasticsearch.Net is the low level client and NEST is the high level client for users of .Net and Elasticsearch.)Before diving in all the technical details of the NEST 1.0 beta release, I would first like to express a sincere thank you to all of you who’ve reached out on Twitter, GitHub issues and via email. Your kind words made all the difference these past 4 months as I busily switched jobs, refactored the client and implemented 1.0/1.1 features. I am happy to say that I have formally joined Elasticsearch Inc and will now be fully dedicated to making Elasticsearch in the .NET world awesome!So, what does ‘beta’ mean? All tests are passing, but we are still waiting for at least one new feature to land. Additionally, beta means that some new features may change before general release. Most importantly, this beta period is intended to solicit as much feedback as possible from everyone on breaking changes/oversights/bugs.<h2>Breaking Changes</h2>With this 1.0 release, we will have many breaking changes. The 0.* NEST releases have been around since 2010 and the internals were showing their age. This 1.0 release represents an almost complete refactoring of NEST.A separate section that lists the breaking changes <a href="http://nest.azurewebsites.net/breaking-changes.html">can be found here</a>. If you find any breaking changes that are not documented, we’d love to hear about them in <a href="http://www.github.com/elasticsearch/elasticsearch-net/issues">elasticsearch-net’s github issues</a>.<h2>Not one but two clients</h2>NEST 0.12.0 introduced a separate, completely generated, client interface called IRawElasticClient which allowed you to build your own requests and responses without having to worry about building endpoints. Much of the work for this 1.0 beta release has been around refactoring this functionality out of the NEST assembly and making NEST’s strongly typed IElasticClient use IRawElasticClient internally.<h3>Elasticsearch.Net</h3>Elasticsearch.Net’s client is now called IElasticsearchClient and is almost completely generated <a href="https://github.com/elasticsearch/elasticsearch/tree/v1.1.0/rest-api-spec">from the client API specification</a>, exposing all the Elasticsearch 1.1 endpoints. It brings a client that’s well aligned in spirit and architecture with the other official Elasticsearch client libraries.Elasticsearch.Net leaves all the request and response mapping up to you although it comes with support for mapping responses into a dynamic container (a slightly modified DynamicDictionary from <a href="http://www.nancyfx.org">NancyFx</a>) out of the box.Elasticsearch.Net is very much intended to be used by other high level clients.Another big new feature is built-in <a href="http://nest.azurewebsites.net/elasticsearch-net/cluster-failover.html">cluster failover/connection pooling support</a> support. This allows the client to retry failed connections on different nodes and sniff the cluster for live nodes when a node fails or at certain configurable intervals.Elasticsearch is elastic, and the client should be, too.Elasticsearch 1.0.0-beta is a prerelease package so make sure you adjust your filter inside the package manager: <img src="https://api.contentstack.io/v2/assets/575e4c8e9e7a83165490c17c/download?uid=blt3d2d801570e82738?uid=blt3d2d801570e82738" data-sys-asset-uid="bltf13ab67aca83aee6"/>or pass the -PreRelease flag to nuget from the console:<blockquote><pre class="prettyprint noescape">PM&gt; Install-Package Elasticsearch.Net -PreRelease </pre></blockquote>Please read <a href="http://nest.azurewebsites.net/elasticsearch-net/quick-start.html">the new documentation for Elasticsearch.Net</a> to find out more.<h3>NEST</h3>NEST’s client classes have completely been rewritten and internally use and still expose the Elasticsearch.Net client. It benefits from it’s cluster failover support and it no longer has complicated internal path builders.NEST is highly opinionated on how you should build and consume Elasticsearch responses, and comes with a strongly typed Query DSL. NEST has really high coverage of mapped endpoints, and the unmapped API’s are all identified and documented on the <a href="https://github.com/elasticsearch/elasticsearch-net/issues?direction=desc&milestone=5&page=1&sort=updated&state=open">roadmap</a>.<blockquote><pre class="prettyprint noescape">PM&gt; Install-Package NEST -PreRelease </pre></blockquote><h2>Changelog</h2>This release consists of 300+ commits over a 4 month timespan so I will just list the highlights:<h3>Aggregations</h3>All of Elasticsearch’s (1.0 and 1.1!) aggregations have been mapped in NEST 1.0.<blockquote><pre class="prettyprint noescape">var results = this._client.Search&lt;ElasticsearchProject&gt;(s=&gt;s .Size(0) .Aggregations(a=&gt;a .Nested("contributors", n=&gt;n .Path(p=&gt;p.Contributors) .Aggregations(t=&gt;t .Average("avg_age", m=&gt;m .Field(p=&gt;p.Contributors.First().Age) ) ) ) ) ); var bucket = results.Aggs.Nested("contributors"); var averageAge = bucket.Average("avg_age"); </pre></blockquote><h3>Snapshot and Restore</h3>You can now script your create/delete/get repositories/snapshotting/restoring with NEST.<h3>New query/filter types</h3>The <a href="https://www.elastic.co/guide/en/elasticsearch/reference/2.4/query-dsl-common-terms-query.html" target="_self">common terms query</a> and simple_query_string have all been mapped, along with other new query and filter types.<h3>Lots of new API’s are mapped</h3>Count percolations, clearing scrolls, external versioning, suggests and more are now all mapped in NEST. The goal is to get as close to a 100% API coverage as possible before the final 1.0 release goes out. Just to reiterate: Elasticsearch.Net, the low level client, already has 100% API coverage but building and handling the responses is out of its scope.<h3>Conditionless() query construct</h3>NEST comes with a powerful feature called conditionless queries which greatly simplifies writing queries with optional parts.Consider this example:<blockquote><pre class="prettyprint noescape">.Query(q=&gt;q.Term("this_term_is_conditionless", "")) </pre></blockquote>NEST will remove the term query and default to not sending a query at all in the body (equivalent to match_all)But what if you want a different query to be run in instead of match_all? This is where the Conditionless() construct comes into play:<blockquote><pre class="prettyprint noescape">.Query(q=&gt;q .Conditionless(qs=&gt;qs .Query(qcq=&gt;qcq.Term("this_term_is_conditionless", "")) .Fallback(qcf=&gt;qcf.Term("name", "do_me_instead") ) ) </pre></blockquote>Now we’ll fallback to a term query on name instead. Note that fallbacks themselves can be conditionless and Conditionless() constructs can be nested inside fallbacks as well.Also if you want to disable conditionless queries you can specify .Strict() on individual parts in the query or globally on the search descriptor.<blockquote><pre class="prettyprint noescape">.Query(q=&gt;q.Strict().Term("this_term_is_conditionless", "")) </pre></blockquote>The previous example will throw an exception because the term misses a value. To enable conditionless only for certain parts in your query, you can also do:<blockquote><pre class="prettyprint noescape">.Strict() .Query(q=&gt; q.Term("must", "exist") &amp;&amp; q.Strict(false).Term("this_term_is_conditionless", "") ) </pre></blockquote>The previous example will just send a term query on the field must, and NEST is smart enough to infer the bool is no longer needed.Finally, if you really intend to send a conditionless query to Elasticsearch you can use .Verbatim()<blockquote><pre class="prettyprint noescape">.Query(q=&gt;q.Verbatim().Term("this_term_is_conditionless", "")) </pre></blockquote>The Strict() and Verbatim() constructs are not new but worth repeating in this context.<h2>Community</h2>As with every release the level of community engagement never ceases to amaze me. Whoever proclaimed open source in .Net is dead?A huge thank you goes out to the folks who submitted the following pull requests.<ul><li><a href="https://github.com/elasticsearch/elasticsearch-net/pull/412">#412</a> Add random_score support for FunctionScore(). ty <a href="http://github.com/vovikdrg">@vovikdrg</a>!</li><li><a href="https://github.com/elasticsearch/elasticsearch-net/pull/419">#419</a> Add script score support for FunctionScore(). ty <a href="http://github.com/V1tOr">@V1tOr</a>!</li><li><a href="https://github.com/elasticsearch/elasticsearch-net/pull/419">#419</a> Added support for index status API. ty <a href="http://github.com/richclement">@richclement</a>!</li><li><a href="https://github.com/elasticsearch/elasticsearch-net/pull/430">#430</a> Allow any object to be used for a partial update. ty <a href="http://github.com/Plasma">@Plasma</a>!</li><li><a href="https://github.com/elasticsearch/elasticsearch-net/pull/431">#431</a> Add Type as constructor parameter to CustomAnalyzer. ty <a href="http://github.com/sschlesier">@sschlesier</a>!</li><li><a href="https://github.com/elasticsearch/elasticsearch-net/pull/440">#440</a> Support dictionary key evaluation in expressions. ty <a href="http://github.com/azubanov">@azubanov</a>!</li><li><a href="https://github.com/elasticsearch/elasticsearch-net/pull/444">#444</a> Renamed QueryString() to Query() MatchQueryDescriptor. ty <a href="http://github.com/gmarz">@gmarz</a>!</li><li><a href="https://github.com/elasticsearch/elasticsearch-net/pull/445">#445</a> Renamed QueryString() to Query() MultiMatchQueryDescriptor. ty <a href="http://github.com/gmarz">@gmarz</a>!</li><li><a href="https://github.com/elasticsearch/elasticsearch-net/pull/446">#446</a> Renamed QueryString() to Query() TextQueryDescriptor. ty <a href="http://github.com/gmarz">@gmarz</a>!</li><li><a href="https://github.com/elasticsearch/elasticsearch-net/pull/447">#447</a> Added Params() support to script on RangeFacetDescriptor. ty <a href="http://github.com/angielamb">@angielamb</a>!</li><li><a href="https://github.com/elasticsearch/elasticsearch-net/pull/448">#448</a> Added filter option in function_score queries. ty <a href="http://github.com/gmarz">@gmarz</a>!</li><li><a href="https://github.com/elasticsearch/elasticsearch-net/pull/479">#479</a> Added detail response mapped to _explain. ty <a href="http://github.com/andreabalducci">@andreabalducci</a>!</li><li><a href="https://github.com/elasticsearch/elasticsearch-net/pull/488">#488</a> Added nested field attribute mapped back in. ty <a href="http://github.com/lukapor">@lukapor</a>!</li><li><a href="https://github.com/elasticsearch/elasticsearch-net/pull/490">#490</a> Added size parameter to CompletionSuggestDescriptor. ty <a href="http://github.com/bgiromini">@bgiromini</a>!</li><li><a href="https://github.com/elasticsearch/elasticsearch-net/pull/540">#540</a> Added missing sort options. ty <a href="http://github.com/Grastveit">@Grastveit</a>!</li><li><a href="https://github.com/elasticsearch/elasticsearch-net/pull/536">#536</a> Worlds smallest pull request (typo). ty <a href="http://github.com/ChrisMcKee">@ChrisMcKee</a>!</li><li><a href="https://github.com/elasticsearch/elasticsearch-net/pull/541">#541</a> FunctionScoreQueryDescriptor boost mode. ty <a href="http://github.com/azubanov">@azubanov</a>!</li><li><a href="https://github.com/elasticsearch/elasticsearch-net/pull/542">#542</a> Allow fields boosting when using expressions in multimatch query. ty <a href="http://github.com/azubanov">@azubanov</a>!</li><li><a href="https://github.com/elasticsearch/elasticsearch-net/pull/545">#545</a> Show requests/response as json string in debug mode when available. ty <a href="http://github.com/azubanov">@azubanov</a>!</li><li><a href="https://github.com/elasticsearch/elasticsearch-net/pull/548">#548</a> Create cache directory for code generation if it does not exist. ty <a href="http://github.com/gmarz">@gmarz</a>!</li><li><a href="https://github.com/elasticsearch/elasticsearch-net/pull/550">#550</a> Fixed typos, formatting, verbiage, naming conventions in the documentation. ty <a href="http://github.com/paigecook">@paigecook</a>!</li><li><a href="https://github.com/elasticsearch/elasticsearch-net/pull/563">#563</a> Added support for new text query types on the multi_match query. ty <a href="http://github.com/azubanov">@azubanov</a>!</li><li><a href="https://github.com/elasticsearch/elasticsearch-net/pull/565">#565</a> Thrift connection does not clean up properly after going into faulted state. ty <a href="http://github.com/danp60">@danp60</a>!</li></ul>In addition a huge thank you goes out to all the folks who gave continuous feedback on the state of master while it underwent major refactoring.A special shout out to <a href="http://github.com/synhershko">@synhershko</a> and <a href="http://github.com/icanhasjonas">@icanhasjonas</a> for opening <a href="https://github.com/elasticsearch/elasticsearch-net/pull/523">#522</a> and <a href="https://github.com/elasticsearch/elasticsearch-net/pull/537">#537</a>Seeing <a href="https://twitter.com/luigiberrettini/status/448413008604438528">tweets</a> like <a href="https://twitter.com/tonariman/status/451354457133969408">this</a> is awesome, thank you <a href="https://twitter.com/tonariman">tonariman</a> &amp; <a href="https://twitter.com/luigiberrettini">luigiberrettini</a>!<h2>Moving forward</h2>The goal is to come to a final Elasticsearch.Net and NEST 1.0 release as soon as possible and with your help we can!Please don’t forget to post your issues and feedback in <a href="http://www.github.com/elasticsearch/elasticsearch-net/issues">elasticsearch-net’s GitHub issues</a>!

Introducing Elasticsearch.Net and NEST 1.0.0-beta1

Ever-increasing quantities and varieties of data are being created and captured. Application, system and user behavior logs, network packets, sensor metrics... you name it. The availability of these data drives a diverse and vast set of use cases ranging from intrusion detection to user behavior analysis, network monitoring, machine monitoring, and remote sensor monitoring, just to name a few.<a href="/products/beats">Beats</a> are open source, purpose-built, lightweight, and efficient agents that acquire and feed data natively to Elasticsearch. Optionally, Beats can feed data to Logstash for further refinement before being forwarded to Elasticsearch.As part of the Elastic Stack, Beats is a <a href="https://en.wikipedia.org/wiki/Telemetry" target="_blank">telemetry</a> architecture, enabling the capture and transmission of measurements and other data from remote sources to Elasticsearch for analysis, aggregation, and search. Anywhere there is change, such as production and consumption, growth and demise, acceleration and deceleration, data has significance. GPS-driven tractors capture soil depths, soil fertility, and ground temperatures, all of which influence agricultural strategy. Ocean buoys record wave amplitudes that may help to warn of impending catastrophes or inform of welcome beach conditions. Automobiles capture thousands of metrics ranging from tire pressure to average speed to fuel consumption that facilitate an understanding of a vehicle's abilities and failures, as well as the owner's driving behaviors. Network sensors capture machine communication messages that help with understanding usage patterns that drive security and infrastructure policies.In the previous article <a href="/blog/running-elasticsearch-on-aws">Running Elasticsearch on AWS</a> we configured a three-node Elasticsearch cluster on AWS EC2. It was fairly straightforward and didn't take much time or effort. We will expand on the <a href="/blog/running-elasticsearch-on-aws">Running Elasticsearch on AWS</a> example and configure Beats to feed the Elasticsearch cluster running on AWS EC2 instances. This will demonstrate how simple it is to configure a centralized machine resource monitoring solution.First, we'll provide an overview of Beats, review available Beats and the environments in which they run - Part I. Then we'll do a step-by-step example installing, configuring, and verifying Beats output - Part II.<h2>Part I - Beats Overview</h2><a href="/products/beats">Beats</a> are purpose-built lightweight data shippers, or agents, that run on remote machines and feed Elasticsearch instances. Beats make it easy to get data into Elasticsearch. Beats are available on a number of operating systems such as Debian, Redhat, Linux and Mac.There are several available <a href="/products/beats">Beats</a> including Filebeat, Metricbeat, Packetbeat, Winlogbeat and Topbeat. Each Beat has a specific purpose or multiple purposes that are logically related, allowing each Beat to focus on its specific task and do it well.<a href="/products/beats/filebeat">Filebeat</a> tails logs and can ship data to Logstash for further refinement, or directly to Elasticsearch for analysis and search. Filebeat can be installed on any machine that has applications that generate log data, such as a database or application server.<a href="/downloads/beats/metricbeat">Metricbeat</a> (Alpha) captures operating system metrics as services such as Apache web server and Redis. Metricbeat does everything Topbeat does and much more in that it captures operating system metrics such as per-process CPU, memory, and storage use as well as common application messages, such as web servers. Metricbeat will replace Topbeat, so you might consider using Metricbeat for development.<a href="/products/beats/packetbeat">Packetbeat</a> captures web, database, and other network protocols, enabling Kibana real-time analytics. Packetbeat is extensible, enabling the addition of new protocols, metrics and analytics.<a href="/downloads/beats/winlogbeat">Winlogbeat</a> captures Windows event log system, application and security data, enabling monitoring of Windows machines.<a href="/products/beats/topbeat">Topbeat</a> captures per-process memory, CPU and disk usage statistics. Topbeat can be installed across a set of machines and indexed into a central Elasticsearch cluster, enabling centralized machine resource monitoring. Topbeat is being replaced by Metricbeat, currently in Alpha, and the change is discussed in this <a href="/blog/beats-5-0-0-alpha3-released">article</a>.If you have metric data not addressed by available Beats, there's also <a href="https://www.elastic.co/guide/en/beats/libbeat/current/beats-reference.html" target="_self">Libbeat</a>, a framework for developing new Beats.The <a href="https://www.elastic.co/guide/en/beats/libbeat/current/community-beats.html">Beats Community</a> is growing and developers are contributing everything from pingbeat, for capturing ping roundtrips, to redisbeat, for Redis monitoring, and to hsbeat, for capturing Java HotSpot VM performance counters.<h2>Part II - Configuring Beats</h2>In this example we will install <a href="https://www.elastic.co/products/beats/topbeat">Topbeat</a> and <a href="https://www.elastic.co/guide/en/beats/packetbeat/current/packetbeat-overview.html">Packetbeat</a> on a development machine, in this case a Macbook Pro, and feed an Elasticsearch cluster running on AWS. Beats can be installed on any machine running a supported operating system that has network connectivity with the Elasticsearch cluster.<h3>Configure Topbeat</h3><a href="/downloads/beats/topbeat">Download Topbeat</a> to a directory on the machine that will be running the Beats.Download and extract the Topbeat archive using your preferred method, for example:<pre class="prettyprint">curl -L -O &lt;a href="https://download.elastic.co/beats/topbeat/topbeat-1.2.3-darwin.tgz"&gt;https://download.elastic.co/beats/topbeat/topbeat-...&lt;/a&gt;</pre><pre class="prettyprint">tar xzvf topbeat-1.2.3-darwin.tgz </pre><img src="https://api.contentstack.io/v2/assets/57853a1ffc8134614f684651/download?uid=blt8fbd3baf06626f6d" data-sys-asset-uid="blt8fbd3baf06626f6d" alt="get_topbeat.png"/>This will create a folder named topbeat-1.2.3-darwin that contains topbeat, topbeat.yml and topbeat.template.yml.Locate and open topbeat.yml in your preferred editor. There is a hosts property that tells Topbeat where Elasticsearch is located. Locate the hosts parameter and update it to refer to the AWS public hostname, for example:<img src="https://api.contentstack.io/v2/assets/57853a36b4a7391d0c76deed/download?uid=blt1c456161684b4a17" data-sys-asset-uid="blt1c456161684b4a17" alt="topbeat_hosts.png"/>Next, find the template property, which defines the index template that defines the field mappings required by Topbeat. Topbeat will load the index template file topbeat.template.json to Elasticsearch. Uncomment the template property along with the name and path properties that immediately follow.<img src="https://api.contentstack.io/v2/assets/57853a41fc8134614f684670/download?uid=bltedc473e2c49ece65" data-sys-asset-uid="bltedc473e2c49ece65" alt="topbeat_template.png"/>Then, run Topbeat with the command:<pre class="prettyprint">sudo ./topbeat -e -c topbeat.yml </pre>To verify documents are being indexed, we'll use curl to make a search request:<pre class="prettyprint">curl -XGET 'http://ec2-50-17-114-78.compute-1.amazonaws.com:9200/topbeat-2016.07.07/_search?pretty' </pre><img src="https://api.contentstack.io/v2/assets/57853a3bb4a7391d0c76def4/download?uid=bltaf4915db04f8e94d" data-sys-asset-uid="bltaf4915db04f8e94d" alt="topbeat_request.png"/>The curl request has several components. The first is the AWS EC2 public hostname "ec2-50-17-114-78.compute-1.amazonaws.com". This should be changed to your EC2 instance's public hostname. The second component "topbeat-2016.07.06" is the index. Lastly, is the search request "_search" with an optional "pretty" argument to format the JSON response.If you want to dive deeper into Topbeat, a comprehensive <a href="https://www.elastic.co/guide/en/beats/topbeat/current/_overview.html">Topbeat Reference</a> is available and it covers topics such as <a href="https://www.elastic.co/guide/en/beats/topbeat/current/configuring-howto-topbeat.html">configuration</a> and <a href="https://www.elastic.co/guide/en/beats/topbeat/current/securing-topbeat.html">security</a>.<h3>Configure Packetbeat</h3><a href="/downloads/beats/packetbeat">Download Packetbeat</a> to a directory on the machine that will be running the Beats. Download and extract the Packetbeat archive using your preferred method, for example:<pre class="prettyprint">curl -L -O&lt;a href="https://download.elastic.co/beats/packetbeat/packetbeat-1.2.3-darwin.tgz"&gt;https://download.elastic.co/beats/packetbeat/packe...&lt;/a&gt;</pre><pre class="prettyprint">tar xzvf packetbeat-1.2.3-darwin.tgz </pre><img src="https://api.contentstack.io/v2/assets/57853a18fc8134614f68464b/download?uid=blt4cd347d705e292c7" data-sys-asset-uid="blt4cd347d705e292c7" alt="get_packetbeat.png"/>This will create a folder named packetbeat-1.2.3-darwin that contains three files: packetbeat, packetbeat.yml, and packetbeat.template.yml.Locate and open packetbeat.yml in your preferred editor. There is a hosts property that tells Packetbeat where Elasticsearch is located. Locate the hosts parameter and update it to refer to the AWS public hostname, for example:<img src="https://api.contentstack.io/v2/assets/57853a26fc8134614f684657/download?uid=blt738187754a29771b" data-sys-asset-uid="blt738187754a29771b" alt="packetbeat_hosts.png"/>Next, find the template property, which defines the index template that defines the field mappings required by Topbeat. Topbeat will load the index template file topbeat.template.json to Elasticsearch. Uncomment the template property along with the name and path properties that immediately follow.<img src="https://api.contentstack.io/v2/assets/57853a2cfc8134614f68465e/download?uid=bltbabfd10ee4a17877" data-sys-asset-uid="bltbabfd10ee4a17877" alt="packetbeat_template.png"/><pre class="prettyprint">sudo ./packetbeat -e -c packetbeat.yml </pre>To verify documents are being indexed, we'll use curl to make a search request:<pre class="prettyprint">curl -XGET 'http://ec2-50-17-114-78.compute-1.amazonaws.com:9200/packetbeat-2016.07.07/_search?pretty' </pre><img src="https://api.contentstack.io/v2/assets/57855336b4a7391d0c76e703/download?uid=blt6665d171b20e6547" data-sys-asset-uid="blt6665d171b20e6547" alt="packetbeat_request.png"/>If you want to dive deeper into Packetbeat, a comprehensive <a href="https://www.elastic.co/guide/en/beats/packetbeat/current/packetbeat-overview.html">Packetbeat Reference</a> is available and it covers topics such as <a href="https://www.elastic.co/guide/en/beats/packetbeat/current/configuring-howto-packetbeat.html">configuration</a> and <a href="https://www.elastic.co/guide/en/beats/packetbeat/current/securing-packetbeat.html">security</a>.<h2>Part III - Summary</h2>Installing, configuring and running Beats is very easy. Beats address a broad set of use cases. They eliminate the challenge of acquiring data and feeding Elasticsearch.&nbsp;Beats can feed any network accessible Elasticsearch cluster, whether it resides on Amazon AWS EC2, Microsoft Azure, dedicated iron and even <a href="https://www.elastic.co/cloud">Elastic Cloud</a>.In the next installment of the AWS blog series, we'll cover installing Kibana on AWS EC2 and visualize the Beats indexed in this example.

aws-beats-elasticsearch.png

aws-beats-elasticsearch-720x420.png

Using Beats with Amazon AWS

security-camera-arcsight-elasticserach-full-bleed.jpg

security_camera_full bleed.jpg

Integrating Elasticsearch with ArcSight SIEM - Part 5

<blockquote>When you help people build applications with Elasticsearch every day, you run into a lot of unique requirements and scenarios. All of us here at Elasticsearch have an ever-growing “bag of tricks”, and it only seems fair to share those tricks with you. We hope you find them interesting, and perhaps useful to your application.</blockquote><hr style="width:100%">A while ago on Twitter, someone was asking if aggregations could be used to categorize data based on irregular product codes. They were combining data from several legacy systems, so the product codes were erratic and not internally consistent.There were two classes of product codes:<ul><li>AB123: Two letters followed by three digits</li><li>A99999: One letter followed by five digits</li></ul>The goal was to determine how many of each type existed, and how many were missing certain meta-data tags. Many people are stumped by situations like this, since it is initially unclear how to categorize two groups of product codes that are only related to each other by the structural format of their ID.<h3>Option 1: Pre-parse with Grok and Logstash</h3>The best solution is to deal with inconsistencies like this at the input level. For example, you can build a <a href="http://logstash.net/docs/1.4.2/filters/grok">Grok filter in Logstash</a> to identify the two different patterns and tag them appropriately. Once tagged, it is trivial to do aggregations on the structured document data. If a new pattern occurs in your data that doesn’t match the pattern, Logstash will emit a _grokparsefailure tag.This has the benefit of improving search results, since only properly tagged documents will be visible to search. You can go back and fix/reindex the “broken” data at your lesiure and iteratively improve your search results through better-tagged data.<h3>Option 2: Regex to the rescue</h3>But things are not always this simple. What if you just indexed 10TB of data and realized you forgot to include appropriate pre-parsing? Re-indexing would be a pain, or potentially impossible. We need a solution that operates on your existing data.The key is to use a regular expression and filter buckets. A <a href="/guide/en/elasticsearch/reference/current/search-aggregations-bucket-filter-aggregation.html">filter bucket</a> will hold all documents matching its filtering criteria. If we place a <a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-regexp-query.html" target="_self">regexp filter</a> inside the bucket, we can find all product IDs matching a certain pattern.Once documents have been sorted into one of the filter buckets, we can apply other bucketing and metrics to derive statistics.Let’s take a look at a very simple example. First we index some data with mixed product codes:<pre class="prettyprint lang-js">PUT /test/data/_bulk {"index":{}} {"product_code" : "AB123"} {"index":{}} {"product_code" : "XY345"} {"index":{}} {"product_code" : "AZ987"} {"index":{}} {"product_code" : "ZZ192"} {"index":{}} {"product_code" : "A99999"} {"index":{}} {"product_code" : "A12345"} {"index":{}} {"product_code" : "A98765"} {"index":{}} {"some_other_field" : "xyz"} {"index":{}} {"some_other_field" : "123"} </pre>Then we can run a very simple aggregation to sort out the various codes:<pre class="prettyprint lang-js">GET /test/data/_search?search_type=count { "aggs" :{ "total_count" : { "global" : {} }, "XX999" : { "filter" : { "regexp":{ "product_code" : "[a-z]{2}[0-9]{3}" } } }, "X99999" : { "filter" : { "regexp":{ "product_code" : "[a-z]{1}[0-9]{5}" } } }, "no_format" : { "missing" : { "field" : "product_code" } } } } </pre>This query will give a document count for each product code matching the two regex patterns. The missing bucket, which will get <a href="https://github.com/elasticsearch/elasticsearch/issues/5659">a nice performance boost in Elasticsearch v1.3.0</a>, will show all documents that don’t have any product code at all. From this base, it is easy to add extra metrics, such as the average price of each product, or the average number sold each day for the last month, etc.<h3>The takeaway tip</h3>The key to this tip is the filter bucket. This bucket accepts any Elasticsearch filter, which means you can construct the same kind of complex filtering operations which you already use in search requests. And because these are filters, they enjoy all the performance benefits inherent to filters.Starting in Elasticsearch version 1.3.0, you will also have access to the <a href="https://github.com/elasticsearch/elasticsearch/issues/6118">filters bucket</a> (note the plural). Although functionally equivalent to using multiple filter buckets, the filters simplifies the syntax for applying multiple filters at the same time.So next time you need to aggregate some statistics, but are stumped by the irregular or inconsistent nature of your data, remember the filter bucket (and the upcoming filters bucket).

Quick Tips: regex filter buckets

<div class="main">Building a highly available product requires some creative thinking. High availability is measured in two aspects when talking about distributed products. The first is the ability to survive partial cluster failure gracefully (and scale to new nodes when added), and the second is handling complete cluster failure or shutdown (and bringing it back up again).The following covers how a distributed system can handle these two problems, especially in cloud environments, and what ElasticSearch does in such cases.<h2>Partial Cluster Failure</h2>Partial cluster failure means that one or more nodes failed within the cluster. Support in a distributed system for partial cluster failure means (surprise surprise) not losing data when such events happen. It gets even more important when working in the cloud, since the control one has over the servers (for good or bad) is out of our hands. Moreover, with cloud providers innovating with cool features like Amazon’s <a href="http://aws.amazon.com/ec2/spot-instances/">spot instances</a> (the ability to bid for instances), we cannot control when our nodes operate. They basically come and go as they wish.This is a question of state, and where the state is stored. I will cover the case when the state is stored locally to the node, which means that if you lose your node, you lose the data stored with the node. It is important to understand that almost any system will work much faster with local state (local file system, â€œlocalâ€ memory) than with a shared storage solution (like NFS). Also, when more than one node needs to access shared state (like NFS), usually locking needs to take place, which hinders the scalability of the system.The most common solution to partial cluster failure is to replicate data between nodes. The ability to control the number of replicas in the cluster allows one to provide a higher degree of availability (this also relates to total cluster failure, see below). The idea of replication is to have one node that perform the operation on itself, and then to replicate the data or operation to its replicas.In the case of ElasticSearch, an index is broken down into shards, and each shard can have zero or more replicas. Both are configurable when creating an index. Here is an example of creating an index with 3 shards, each with 2 replicas (note, this is configurable on a per index basis):<pre class="dontquote prettyprint">&lt;span class="pln"&gt;$ curl &lt;/span&gt;&lt;span class="pun"&gt;-&lt;/span&gt;&lt;span class="pln"&gt;XPUT http&lt;/span&gt;&lt;span class="pun"&gt;:&lt;/span&gt;&lt;span class="com"&gt;//localhost:9200/twitter/ -d '&lt;/span&gt;&lt;span class="pln"&gt; index &lt;/span&gt;&lt;span class="pun"&gt;:&lt;/span&gt;&lt;span class="pln"&gt; number_of_shards &lt;/span&gt;&lt;span class="pun"&gt;:&lt;/span&gt;&lt;span class="pln"&gt; &lt;/span&gt;&lt;span class="lit"&gt;3&lt;/span&gt;&lt;span class="pln"&gt; number_of_replicas &lt;/span&gt;&lt;span class="pun"&gt;:&lt;/span&gt;&lt;span class="pln"&gt; &lt;/span&gt;&lt;span class="lit"&gt;2&lt;/span&gt;&lt;span class="pln"&gt; &lt;/span&gt;&lt;span class="str"&gt;'&lt;/span&gt;</pre>When an operation is presented to a node, it is routed automatically to the primary shard within its replication group, which performs the operation and then replicates it to all its backups. Replication to the backup is done in parallel using non-blocking IO, which basically means that more replicas mainly just means more traffic on the network.Replicas accept read operations as well, meaning that the more replicas you have, the more highly available your index is, but also the more scalable it is in terms of search and get operations!Once we work under the assumption that node level state is volatile, we can store the index itself in a fast medium, perhaps a local file system (even SSD), JVM heap memory, or even non-JVM (or native) memory.The choice of where to store an index for ElasticSearch is configurable on the index level. Here is an example that indicates that the â€œtwitterâ€ index be stored in main memory:<pre class="dontquote prettyprint">&lt;span class="pln"&gt;$ curl &lt;/span&gt;&lt;span class="pun"&gt;-&lt;/span&gt;&lt;span class="pln"&gt;XPUT http&lt;/span&gt;&lt;span class="pun"&gt;:&lt;/span&gt;&lt;span class="com"&gt;//localhost:9200/twitter/ -d '&lt;/span&gt;&lt;span class="pln"&gt; index &lt;/span&gt;&lt;span class="pun"&gt;:&lt;/span&gt;&lt;span class="pln"&gt; store&lt;/span&gt;&lt;span class="pun"&gt;:&lt;/span&gt;&lt;span class="pln"&gt; type &lt;/span&gt;&lt;span class="pun"&gt;:&lt;/span&gt;&lt;span class="pln"&gt; memory &lt;/span&gt;&lt;span class="str"&gt;'&lt;/span&gt;</pre>But if node storage is considered transient, what do we do regarding long term persistency of the index? The answer is the next section.<h2>Full Cluster Failure / Long Term Persistency</h2>Handling full cluster failure or shutdown means that both the state of the cluster (i.e. which indices exist, what are the their settings, what are their mapping definitions, etc.) and the content of each index must be stored persistently elsewhere.A solution for this is similar to <a href="http://www.apple.com/macosx/what-is-macosx/time-machine.html">Time Machine</a>, we need to persist the state into a long term storage. This solution is similar to what data grids have called <a href="http://www.infoq.com/articles/write-behind-caching">Write Behind</a>.The idea is that short term high availability is maintained by replication, while long term persistency is done by asynchronously writing deltas of the state into long term (persistent) storage. The benefit is that real time operations are not affected by this process.So, what happens when we restart a cluster? When it first starts up, it queries the long term storage for the cluster state (indices created, settings, mappings, etc.) and established them locally (creating indices, creating mapping). Each time a shard is first instantiated within its shard replication group, it will also recover its state from the long term persistency.ElasticSearch provides this capability with a module called <a href="/guide/reference/modules/gateway/">Gateway</a>. For example, to have the state stored in a shared file system, each node needs to start with a configuration of:<pre class="dontquote prettyprint">&lt;span class="pln"&gt;gateway &lt;/span&gt;&lt;span class="pun"&gt;:&lt;/span&gt;&lt;span class="pln"&gt; type &lt;/span&gt;&lt;span class="pun"&gt;:&lt;/span&gt;&lt;span class="pln"&gt; fs fs &lt;/span&gt;&lt;span class="pun"&gt;:&lt;/span&gt;&lt;span class="pln"&gt; location &lt;/span&gt;&lt;span class="pun"&gt;:&lt;/span&gt;&lt;span class="pln"&gt; &lt;/span&gt;&lt;span class="str"&gt;/shared/&lt;/span&gt;&lt;span class="pln"&gt;fs&lt;/span&gt;&lt;span class="pun"&gt;/&lt;/span&gt;</pre>This can also be started right from the command line using (only on unix):<pre class="dontquote prettyprint">&lt;span class="pln"&gt;$ bin&lt;/span&gt;&lt;span class="pun"&gt;/&lt;/span&gt;&lt;span class="pln"&gt;elasticsearch &lt;/span&gt;&lt;span class="pun"&gt;-&lt;/span&gt;&lt;span class="pln"&gt;f &lt;/span&gt;&lt;span class="pun"&gt;-&lt;/span&gt;&lt;span class="typ"&gt;Des&lt;/span&gt;&lt;span class="pun"&gt;.&lt;/span&gt;&lt;span class="pln"&gt;gateway&lt;/span&gt;&lt;span class="pun"&gt;.&lt;/span&gt;&lt;span class="pln"&gt;type&lt;/span&gt;&lt;span class="pun"&gt;=&lt;/span&gt;&lt;span class="pln"&gt;fs &lt;/span&gt;&lt;span class="pun"&gt;-&lt;/span&gt;&lt;span class="typ"&gt;Des&lt;/span&gt;&lt;span class="pun"&gt;.&lt;/span&gt;&lt;span class="pln"&gt;gateway&lt;/span&gt;&lt;span class="pun"&gt;.&lt;/span&gt;&lt;span class="pln"&gt;fs&lt;/span&gt;&lt;span class="pun"&gt;.&lt;/span&gt;&lt;span class="pln"&gt;location&lt;/span&gt;&lt;span class="pun"&gt;=&lt;/span&gt;&lt;span class="str"&gt;/shared/&lt;/span&gt;&lt;span class="pln"&gt;fs&lt;/span&gt;&lt;span class="pun"&gt;/&lt;/span&gt;</pre>The above means that the cluster state will be stored on a shared file system, and each index created will automatically store its state on a shared file system as well. But, since each index has its own <a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-gateway.html" target="_self">index gateway</a>, we can get clever and decide on a per index level if it should be stored using a gateway or not. For example, the following example disables long term storage of an index:<pre class="dontquote prettyprint">&lt;span class="pln"&gt;$ curl &lt;/span&gt;&lt;span class="pun"&gt;-&lt;/span&gt;&lt;span class="pln"&gt;XPUT http&lt;/span&gt;&lt;span class="pun"&gt;:&lt;/span&gt;&lt;span class="com"&gt;//localhost:9200/my_special_volatile_index/ -d '&lt;/span&gt;&lt;span class="pln"&gt; index &lt;/span&gt;&lt;span class="pun"&gt;:&lt;/span&gt;&lt;span class="pln"&gt; gateway&lt;/span&gt;&lt;span class="pun"&gt;:&lt;/span&gt;&lt;span class="pln"&gt; type &lt;/span&gt;&lt;span class="pun"&gt;:&lt;/span&gt;&lt;span class="pln"&gt; none &lt;/span&gt;&lt;span class="str"&gt;'&lt;/span&gt;</pre>As you can see, this is really powerful. More over, this is the perfect solution for cloud environments. In the cloud, storing long term state is usually done using what the cloud provider provides. For example, with Amazon, its <a href="http://aws.amazon.com/ebs/">EBS</a> or <a href="http://aws.amazon.com/s3/">S3</a>. Right out of the (current version) box, you can use Amazon EBS for long term storage in ElasticSearch (since its a mounted shared file system). In the future, there will also be a gateway module to store the state directly on Amazon S3 and others.<h2>Final Words</h2>This is only the tip of the iceberg of how ElasticSearch replication, recovery and gateway works, and the future will just increase the iceberg (right back at you, global warming!). For example, what I am playing with now is with the idea of treating indices state as a version control, being able to tag them, and recover back specific tags into new indices or current ones. I hope that at least in terms of handling cluster failures (both partial and total), ElasticSearch makes more sense now.-shay.banon</div>

Search Engine Time Machine

This is the final post in a three-part series about Instant Aggregations. Read how it all started in <a href="https://www.elastic.co/blog/the-tale-of-caching-and-why-it-matters">The Tale of Caching and Why It Matters</a> from Simon Willnauer and Shay Banon and the meaty middle detailed in <a href="https://www.elastic.co/blog/the-great-query-refactoring-thou-shalt-only-parse-once"> </a><a href="https://www.elastic.co/blog/the-great-query-refactoring-thou-shalt-only-parse-once">The Great Query Refactoring: Thou Shalt Only Parse Once</a>. Enjoy the trilogy! In 1.4.0 Elasticsearch gained a shard level ‘Request Cache’ which caches the result of the query phase on each shard keyed on the search request itself. Until 5.0 this feature was disabled by default since it wasn’t as useful as it could be for most of the computationally heavy use cases. The cache was intended to make searches faster, especially for aggregations. One problem was that ordering in JSON is not deterministic so although two requests may be logically the same, when rendered to JSON strings they may not be equal. The other main problem is that these computationally heavy use cases tend to use time windows relative to the current time so subsequent requests will tend to have slightly different time ranges. Enabling it would likely be a waste memory for most of the users &nbsp;since they would rarely ever get a cache hit. So why would we add such a feature?Here at Elastics, we try to follow the rule of “progress over perfection”. Even if we can’t utilize it’s full potential we get a step closer to our goals. Now after 2 years of engineering effort and <a href="/blog/the-great-query-refactoring-thou-shalt-only-parse-once">refactoring the entire search request</a> representation, we can finally take full advantage of it.From 5.0 the request cache will be enabled by default for all requests with `size:0`. The request cache is most useful for analytics use cases and generally analytics use cases use search requests with `size: 0`.So how did the changes made since 1.4.0 enable us to better use this feature?The search request cache gives massive improvements in search performance when running the same search multiple times on a static index. However, search requests are rarely exactly the same, especially in the time series use case.Lets look at some of the characteristics of typical search for time-series use cases:<ul><li>Queries have a &nbsp;time range component often relative to current time (e.g. from now-7d to now)</li><li>Search requests span &nbsp;across multiple indices</li><li>Indexing happens on the latest index, all other indices are static</li><li>Same search requests are run multiple times with different time ranges</li></ul>The above are common characteristics of search requests for a lot of time series use cases including common patterns of usage for Kibana. A lot of Kibana dashboards contain quite a few visualisations a lot of which are complex and can take a number of seconds for the dashboard request to be executed by Elasticsearch. Because search requests are rarely the same, the search request cache can’t be used effectively to improve search performance for these use cases.Or can it?How can we make the search request cache work when the time range is always moving? Let’s solve this by considering an example. Imagine we have the details for every residential house sale in the UK since 1995. We can index this into 21 yearly indices (e.g. house-prices-1995, house-prices-1996, …, house-prices-2016). For the purposes of this explanation let’s make each index only have 1 shard (though the same idea can easily be extended to multiple shard indices without modification) Now we can run queries against the indices by using requests like this:<pre>``` GET house-prices-*/_search { "query": { "range": { "date": { "gte": "2013-09-01", "lt": "2016-03-01" } } } } ``` </pre>The figure below shows three of these such queries overlaid on the yearly indices. These queries are typical for dashboard style use cases in that they differ only by small amounts (imagine a refreshing dashboard)<img src="https://api.contentstack.io/v2/assets/57d6e0bf39cce73c4a091058/download?uid=blt1a2b92ef4097f1ff" height="325" width="850" style="width: 850px; height: 325px;"/>A couple of things stand out from this diagram:<ul><li>All three queries match no documents in house-prices-2012 (highlighted red on the diagram)</li><li>All three queries will match all documents in house-prices-2014 and house-prices-2015 (highlighted green on the diagram)</li></ul>So the only indices which affect the matching documents between the three queries are house-prices-2013 and house-prices-2016.If we could rewrite the queries on each shard based on the range of values present in that index we could make the queries able to actually utilize the request cache. Below is the diagram with the same three queries rewritten to make them more cachable in the search request cache:<img src="https://api.contentstack.io/v2/assets/57d6e1d039cce73c4a0910ad/download?uid=blte0f8b865a8d465c8" height="325" width="850" style="width: 850px; height: 325px;"/>You can see that in the diagram above, the first query is rewritten and run as a `match_none` query on the `house-prices-2012` indexes shard and a `match_all` query on the `house-prices-2014` and `house-prices-2015` indices shards. This means that when the second and third query is run it can use the cached results of the search on the `house-prices-2012`, `house-prices-2014` and `house-prices-2015` indices shards instead of actually running a search on those shards. So for the second and third queries we only need to actually search the shards on the `house-prices-2013` and `house-prices-2016` indices.Also, even for the first query, running a `match_none` query instead of a range query will be faster since it will not need to try to lookup the date range in the index.So how does this work in practice?This feature not only relies on the search request cache mentioned at the beginning of this post, but it is also heavily based on the query refactoring that we described in a recent <a href="/blog/the-great-query-refactoring-thou-shalt-only-parse-once">blog post</a>. The reason for this is that in order to add the rewrite logic described above we need objects that represent the query and the search request that we can actually rewrite.When the search request is received by the shard, we rewrite the entire request including `query` and `post_filter` QueryBuilder objects. Each type of query has it’s own implementation of QueryBuilder which will rewrite according to it’s own rules.Compound queries, queries that themselves contain queries (such as `bool` and `constant_score`) will call rewrite on the queries they wrap. Most leaf queries, queries that do not contain other queries, currently do not contain any rewrite logic so just return themselves (to indicate they did not change).The range query however, will check the minimum and maximum value of the field (we will call this the field range) and compare this to the range it contains. There are three cases that we care about when evaluating the rewrite for the range:<ul><li>The query range and the field range do not overlap at all - In this case we can rewrite this range as a `match_none` query so we return a `MatchNoneQueryBuilder`</li><li>The query range contains the entire field range - In this case all documents which contain a value for this field will match the range so we can rewrite the range query as an unbounded range query (not a match_all query) and return it</li><li>The query range partially overlaps the field range. There’s not much we can do to help in this case so we don’t rewrite and return the original range query.</li></ul>Now that the search request has been rewritten we can check the cache with the rewritten version and either retrieve the cached result or execute the search and add the result to the cache, keyed on the rewritten search request. This has a nice side effect since the rewritten queries are “normalized” search requests that are semantically equivalent but differ in the order of keys in the json will now also produce cache hits.Why not use `match_all` ?In the case that the query range contains the entire field range you might think we could rewrite the `range` query as a `match_all` query. We can’t do this because we can’t assume that all documents have a value for the field. If we rewrite to a `match_all` query we would incorrectly match documents that have no value for the field. For this reason we instead rewrite the range to an unbounded range query (effectively [* TO *]) which still means the query is much more useful to the search request cache.Is this all worth the trouble?To answer this question we’ll go back to our yearly house price indices. This dataset contains 21 yearly indices containing 21,304,688 residential UK house sales from 1995 to 2016. I ran this fairly unscientific test on my Macbook Pro. Each index has a single shard. I created a Kibana dashboard showing 14 different visualisations relevant to the data mixing simple and complex visualisations. The top of the dashboard looks like the following:<img src="https://api.contentstack.io/v2/assets/57e02db9acc34b3254b8af0b/download?uid=bltf740e6e1aec8ca2b" height="469" width="850" style="width: 850px; height: 469px;"/>With the request cache disabled, requests to elasticsearch for the dashboard to refresh takes 12s-14s. When the request cache is enabled this drops to ~100ms. The same request is now 100x faster!But this is on completely static data so what happens when we are indexing data?To test this I deleted the data from 2010 to 2016 and loaded it in date order whilst refreshing the dashboard:<img src="https://api.contentstack.io/v2/assets/57d6e20f39cce73c4a0910cb/download?uid=blt7d49d71a4510bee5" height="502" width="850" style="width: 850px; height: 502px;"/>Now the request to Elasticsearch for the dashboard refresh takes 60-200ms. So the query is still at least 50x faster than our original un-cached query! The refresh time varies depending on how much data is present in the changing index since the search on the latest index does not hit the cache. This is because as new data is indexed, the cache on the shards of the latest index is constantly being invalidated.The Instant Aggregations feature enables much better caching of search requests by rewriting the query based on the data present on the shard. As of 5.0.0 the query is only rewritten for date range queries, but with this infrastructure in place it opens the door to the potential of rewriting other types of query to improve the request cache utilization. We even have the potential to rewrite aggregations to make them more cacheable! We have shown in this post that the potential performance improvements when we use this technique can be huge, so watch this space for more improvements in the future.

instant-aggregations-elasticsearch-colin-goodheart-smithe-request-cache-full-bleed.jpg

instant-aggregations-elasticsearch-rewriting-queries-thumbnail.jpeg

Instant Aggregations: Rewriting Queries for Fun and Profit

full-bleed-2k.jpg

thumb.jpeg

How to Develop Your Own Security Extensions and Custom Realms for Elasticsearch

<h2>First, some context</h2><a href="https://www.elastic.co/blog/hot-warm-architecture-in-elasticsearch-5-x">Hot-Warm</a> architectures are often used when we want to get the most out of our hardware. It is particularly useful when we have time-based data, like <a href="https://www.elastic.co/blog/observability-with-the-elastic-stack">logs, metrics, and APM data</a>. Most of these setups rely on the fact that this data is read-only (after ingest) and that indices can be time(or size)-based. So they can be easily deleted based on our desired retention period. In this architecture, we categorize Elasticsearch nodes into two types: 'hot' and 'warm'.Hot nodes hold the most recent data and thus handle all indexing load. Since recent data is usually the most frequently queried, these nodes will be the most powerful in our cluster: fast storage, high memory and CPU. But that extra power gets expensive, so it doesn’t make sense to store older data that isn’t queried as often on a hot node.On the other hand, warm nodes will be the ones dedicated to long-term storage in a more cost-efficient way. Data on the warm nodes is not as likely to be queried often and data within the cluster will move from hot to warm nodes based on our planned retention (achieved through <a href="https://www.elastic.co/guide/en/elasticsearch/reference/6.6/shard-allocation-filtering.html">shard allocation filtering</a>), while still being available online for queries.Starting with Elastic Stack 6.3, we've been building new features to enhance hot-warm architectures and simplify working with time-based data.<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blte8ad3502d7fba4c1/5c98d0a5c9ab24634dcd6d86/hot-warm-features-frozen-indices-transp.png" data-sys-asset-uid="blte8ad3502d7fba4c1" alt="hot-warm-features-frozen-indices-transp.png" width="550" style="margin: auto;display: block;"/><a href="https://www.elastic.co/blog/data-rollups-in-elasticsearch-you-know-for-saving-space">Data rollups</a> were first introduced in version 6.3, to save storage. In time-series data, we want fine-grained detail for the most recent data. But it is very unlikely that we will need the same for historical data, where we will typically look at datasets as a whole. And this is where rollups came in, since starting at version 6.5 we can <a href="https://www.elastic.co/blog/how-to-create-manage-and-visualize-elasticsearch-rollup-data-in-kibana">create, manage and visualize rollup data in Kibana</a>.Shortly after, we added <a href="https://www.elastic.co/guide/en/elasticsearch/reference/6.6/modules-snapshots.html#_source_only_repository">source-only snapshots</a>. This minimal snapshots will provide a significant reduction of snapshot's storage, with the tradeoff of having to reindex data if we want to restore and query. This has been available since version 6.5.In version 6.6, we released two powerful features, Index Lifecycle Management (ILM) and Frozen Indices.<a href="https://www.elastic.co/guide/en/elasticsearch/reference/6.6/index-lifecycle-management.html">ILM</a> provides the means to automate your indices management over time. It simplifies moving indices from hot to warm, allows deletion when indices are too old, or automates force merging indices down to one segment.And for the rest of this blog, we’ll talk about frozen indices.<h2>Why freeze an index?</h2>One of the biggest pain points with “old” data is that, regardless of age, indices still have a significant memory footprint. Even if we place them on cold nodes, they still use heap.A possible solution could be to close the index. If we close an index, it won’t require memory, but we will need to re-open it to run a search. Reopening indices will incur an operational cost and also require the heap it was using before being closed.On each node, there is a memory (heap) to storage ratio that will limit the amount of storage available per node. It may vary from as low as 1:8 (memory:data) for memory intensive scenarios, to something close to 1:100 for less demanding memory use cases.This is where frozen indices come in. What if we could have indices that are still open — keeping them searchable — but do not occupy heap? We could add more storage to data nodes that hold frozen indices, and break the 1:100 ratio, understanding the tradeoff that searches might be slower.When we freeze an index, it becomes read-only and its transient data structures are dropped from memory. In turn, when we run a query on frozen indices, we will have to load the data structures to memory. Searching a frozen index doesn’t have to be slow. Lucene heavily depends on the filesystem cache which might have enough capacity to retain significant portions of your index in memory. In such a case searches are comparable in speed per shard. Yet, a frozen index is still throttled such that only one frozen shard is executing per node at the same time. This aspect might slow down searches compared to unfrozen indices.<h2>How freezing works</h2>Frozen indices are searched through a dedicated, <a href="https://www.elastic.co/guide/en/elasticsearch/reference/6.6/modules-threadpool.html#search-throttled">searched-throttled</a> threadpool. This by default uses a single thread, to ensure that frozen indices are loaded into memory one at a time. If concurrent searches are happening, they will get in the queue to add additional protections to prevent nodes from running out of memory.So, in a hot-warm architecture, we will now be able to transition indices from hot to warm, and then be able to freeze them before archiving or deleting them, allowing us to reduce our hardware requirements.Before frozen indices, to reduce infrastructure cost we had to snapshot and archive our data, which adds a significant operational cost. We would have to restore data if we needed to search again. Now, we can keep our historical data available for search, without a significant memory overhead. And if we need to write again on an already froze index, we can just <a href="https://www.elastic.co/guide/en/elasticsearch/reference/6.6/unfreeze-index-api.html">unfreeze it</a>.<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blta19e364f7d3f763a/5c98d09ec9ab24634dcd6d80/index-states-transp.png" data-sys-asset-uid="blta19e364f7d3f763a" alt="index-states-transp.png" width="400" style="margin: auto;display: block;"/><h2>How to freeze an Elasticsearch index</h2>Frozen indices are easy to implement in your cluster, so let’s get started on how to use the <a href="https://www.elastic.co/guide/en/elasticsearch/reference/6.6/freeze-index-api.html">Freeze index API</a> and how to <a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/sql-index-frozen.html" target="_self">search on frozen indices</a>.First, we'll start by creating some sample data on a test index.<pre class="prettyprint">POST /sampledata/_doc { "name": "Jane", "lastname": "Doe" } POST /sampledata/_doc { "name": "John", "lastname": "Doe" } </pre>And then check that our data has been ingested. This should return two hits:<pre class="prettyprint">GET /sampledata/_search </pre>As a best practice, before you freeze an index it’s recommended to first run a <a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-forcemerge.html">force_merge</a>. This will ensure that each shard has only a single segment on disk. It will also provide much better compression and simplifies the data structures we will need when running an aggregation or a sorted search request on the frozen index. Running searches on a frozen index with multiple segments can have a significant performance overhead up to multiple orders of magnitude.<pre class="prettyprint">POST /sampledata/_forcemerge?max_num_segments=1 </pre>The next step is to just invoke a freeze on our index via de Freeze index API endpoint.<pre class="prettyprint">POST /sampledata/_freeze </pre><h2>Searching frozen indices</h2>Now that it’s frozen, you’ll see that regular searches won’t work. The reason for this is that, to limit memory consumption per node, frozen indices are&nbsp;<a href="https://www.elastic.co/guide/en/elasticsearch/reference/6.6/searching_a_frozen_index.html">throttled</a>. &nbsp;Since we could target a frozen index by mistake, we’ll prevent accidental slowdowns by specifically adding ignore_throttled=false to the request.<pre class="prettyprint">GET /sampledata/_search?ignore_throttled=false { "query": { "match": { "name": "jane" } } } </pre>Now we can check the status of our new index, by running the following request:<pre class="prettyprint">GET _cat/indices/sampledata?v&amp;h=health,status,index,pri,rep,docs.count,store.size </pre>This will return a result similar to the following, with the index status being ‘open’:<pre class="prettyprint">health status index pri rep docs.count store.size green open sampledata 5 1 2 17.8kb </pre>As mentioned above, we must protect the cluster from running out of memory, thus there is a limit in the number of frozen indices we can concurrently load for search on a node. The number of threads in the search-throttled threadpool defaults to 1, with a default queue of 100. This means that if we run more than one request, they will be queued up to a hundred. We can monitor the&nbsp;<a href="https://www.elastic.co/guide/en/elasticsearch/reference/6.6/cat-thread-pool.html#cat-thread-pool">threadpool status</a>, to check queues and rejections, with the following request:<pre class="prettyprint">GET _cat/thread_pool/search_throttled?v&amp;h=node_name,name,active,rejected,queue,completed&amp;s=node_name </pre>Which should return a response similar to:<pre class="prettyprint">node_name name active rejected queue completed instance-0000000000 search_throttled 0 0 0 25 instance-0000000001 search_throttled 0 0 0 22 instance-0000000002 search_throttled 0 0 0 0 </pre>Frozen indices might be slower, but they can be pre-filtered in a very efficient manner. It is also recommended to set the request parameter pre_filter_shard_size to 1.<pre class="prettyprint">GET /sampledata/_search?ignore_throttled=false&amp;pre_filter_shard_size=1 { "query": { "match": { "name": "jane" } } } </pre>This will not add a significant overhead to the query and will allow us to take advantage of the usual scenario. For example, when searching on a date range on time-series indices, not all shards will match.<h2>How to write to a frozen Elasticsearch index</h2>What will happen if we try to write on an already frozen index? Let’s go for it and find out.<pre class="prettyprint">POST /sampledata/_doc { "name": "Janie", "lastname": "Doe" } </pre>What happened? Frozen indices are read-only, so writing is blocked. We can check these in the index settings:<pre class="prettyprint">GET /sampledata/_settings?flat_settings=true </pre>Which will return:<pre class="prettyprint">{ "sampledata" : { "settings" : { "index.blocks.write" : "true", "index.frozen" : "true", .... } } } </pre>We have to use the&nbsp;<a href="https://www.elastic.co/guide/en/elasticsearch/reference/6.6/unfreeze-index-api.html">Unfreeze index API</a>, invoking the unfreeze endpoint on the index.<pre class="prettyprint">POST /sampledata/_unfreeze </pre>And now we'll be able to create a third document and search for it.<pre class="prettyprint">POST /sampledata/_doc { "name": "Janie", "lastname": "Doe" } GET /sampledata/_search { "query": { "match": { "name": "janie" } } } </pre>Unfreezing should be done only under exceptional situations. And remember to always run a `force_merge` before freezing the index again to ensure optimal performance.<h2>Using frozen indices in Kibana</h2>To begin with, we will need to load some sample data, like the <a href="https://www.elastic.co/guide/en/kibana/6.6/tutorial-sample-data.html">sample flight data</a>.Click on the “Add” button for Sample flight data.<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt142abc81bf2e4ac0/5c98d104632578d5055bb49f/kibana-load-data.png" alt="kibana-load-data.png" width="700" style="margin: auto;display: block;" data-sys-asset-uid="blt142abc81bf2e4ac0"/>We should now be able to see the loaded data by clicking the “View data” button. The dashboard will be similar to this one.<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt70ea061b7b398da9/5c98d0ff026179ca592ee205/flight-dashboard.png" width="100%" data-sys-asset-uid="blt70ea061b7b398da9" alt="flight-dashboard.png" style="margin: auto;display: block;"/>Now we can test freezing the index:<pre class="prettyprint">POST /kibana_sample_data_flights/_forcemerge?max_num_segments=1 POST /kibana_sample_data_flights/_freeze </pre>And if we go back to our dashboard, we’ll notice that the data has apparently “disappeared”.<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt0a68db6e72e372e7/5c98d0f9acf2956c56cb1953/empty-flight-dashboard.png" data-sys-asset-uid="blt0a68db6e72e372e7" alt="empty-flight-dashboard.png" width="100%" style="margin: auto;display: block;"/>We need to tell Kibana to allow searches on frozen indices, which is disabled by default.Go to Kibana Management, choose Advanced Settings. In the Search section, you will find that “Search in frozen indices” is disabled. Toggle to enable and save the changes.<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltf5f1d58c21e9ba65/5c98d0f249a201165fca1028/frozen-indices-kibana-settings.png" data-sys-asset-uid="bltf5f1d58c21e9ba65" alt="frozen-indices-kibana-settings.png" width="100%" style="margin: auto;display: block;"/>And the flight's dashboard will show the data again.<h2>Wrapping up</h2>Frozen Indices are a very powerful tool in hot-warm architectures. They enable a more cost-effective solution for increased retention while retaining online search. I recommend that you test your search latency with your hardware and data, to come with the right sizing and search latency for your frozen indices.Check out the <a href="https://www.elastic.co/guide/en/elasticsearch/reference/6.6/frozen-indices.html">Elasticsearch documentation</a> to learn more about the <a href="https://www.elastic.co/guide/en/elasticsearch/reference/6.6/freeze-index-api.html">Freeze index API</a>. And as always, if you have any questions, reach out on our <a href="https://discuss.elastic.co/">Discuss</a> forums. Happy freezing!

frozen-indices-freeze-index-api.jpg

frozen-indices-freeze-index-api-small.jpg

Creating frozen indices with the Elasticsearch Freeze index API

At Elastic, we are big fans of providing self-service tools to successfully deploy and maintain our software in production. One of the most popular features of Elasticsearch is the APIs used to retrieve it's operational data. While Logstash was always easy to get started with, it was a bit of a black box when it came to managing it in production. Administrators had to rely on external monitoring systems to make sure Logstash was functioning without any issues. A core theme in 5.0 release was to make it easier to manage Logstash. In this post, I'll show you how to monitor your Logstash filters using the inbuilt APIs and the <a href="https://www.elastic.co/products">Elastic Stack</a> itself.&nbsp;
<h2>What is my ingestion rate?</h2>
If you have deployed Logstash as part of a high-volume data ingestion pipeline, one of the most important metrics to monitor is “how fast is Logstash ingesting data”. Previously, to answer this question, you had to install the <a href="https://www.elastic.co/guide/en/logstash/current/plugins-filters-metrics.html">metrics filter</a> to generate the ingestion stats. Starting <a href="https://www.elastic.co/blog/logstash-5-2-0-released">with 5.2</a>, we've made it really easy to monitor Logstash using the Elastic Stack itself in addition to the <a href="https://www.elastic.co/guide/en/logstash/8.13/configuring-logstash.html" target="_self">native APIs</a>&nbsp;which were introduced in Logstash 5.0. The <a href="https://www.elastic.co/downloads/x-pack">x-pack plugin</a> for Elastic Stack bundles an agent that you can install on all your production Logstash nodes. It also ships with charts for the most important metrics information that let you visualize Logstash monitoring data out of the box. The monitoring feature in x-pack is under the <a href="https://www.elastic.co/subscriptions">Basic license</a> which means you get all this awesomeness for free. The APIs mentioned here are all part of the open source Logstash distribution.&nbsp;
Let's get started: 
<ul>
 <li>Install x-pack on your production Logstash nodes. This assumes you've already configured your production cluster for monitoring.</li>
</ul><pre class="prettyprint">cd $LS_HOME bin/logstash-plugin install x-pack </pre>
This plugin should be installed on all Logstash nodes you wish to monitor. The x-pack plugin extends your Logstash to periodically ship all the metrics it collects to your production Elasticsearch cluster. Once installed, you can configure the shipping agent using the logstash.yml file. 
<ul>
 <li>Configure your Elasticsearch end-point.</li>
</ul>
xpack.monitoring.elasticsearch.url: "http://my-es-prod-node:9200"
We strongly recommend that you enable security for your production Elasticsearch clusters. Once you've installed x-pack, you can use the extensive <a href="https://www.elastic.co/products/x-pack/security">security features</a> that ships with it. 
<ul>
 <li>Restart Logstash and navigate to http://kibana-url:5600</li>
</ul>
Pick the Logstash you want to get detailed monitoring info:
<img src="https://api.contentstack.io/v2/uploads/588febd1c387e5ea18b34892/download?uid=blt166210739a9e46e6" data-sys-asset-uid="blt166210739a9e46e6" alt="nodes_list.png" width="568" height="151" style="width: 568;height: 151;" />
<img src="https://api.contentstack.io/v2/uploads/588febf1f6ea3cf85f2c933c/download?uid=blt90f828912b55c071" data-sys-asset-uid="blt90f828912b55c071" alt="events_rate.png" width="575" height="311" style="width: 575;height: 311;" />
<h2>What are my slowest filters?</h2>
Filters are the workhorses of Logstash, which means they can be the slowest stage of your pipeline, compared to the inputs and outputs stage. The performance of a filter is also dependent on the type of data it processes. For example, the grok filter, a very popular one, uses a regular expression engine to extract fields. And certain types of event data can throw your grok filter into an expensive backtracking loop. Before we attempt to tweak the filters for performance, the first step if to actually figure out which one of them is the slowest: <pre class="prettyprint">curl localhost:9600/_node/stats/pipeline?pretty </pre><pre class="prettyprint">... "filters":[ { "id":"2178468f89ab43c6f32e934d3cb587e3c850d93a-4", "events":{ "duration_in_millis":24005, "in":49482, "out":49480 }, "name":"geoip" }, { "id":"2178468f89ab43c6f32e934d3cb587e3c850d93a-2", "events":{ "duration_in_millis":84193, "in":49483, "out":49482 }, "matches":29400, "failures":20083, "patterns_per_field":{ "message":1 }, "name":"grok" }, { "id":"2178468f89ab43c6f32e934d3cb587e3c850d93a-5", "events":{ "duration_in_millis":44353, "in":49480, "out":49477 }, "name":"useragent" } ... </pre>
For all the plugins in our pipeline, this API provides a per-pipeline break down of event stats. One particular field of interest is the duration_in_millis field which reports the total amount of time spent in a filter for all the events that passed through it. If you divide duration_in_millis by events.in, you should get the average time events have spent in this filter. From the response above, it looks like the grok filter is the slowest, followed by the useragent filter. Zooming in on the grok filters, we can also see that there are quite a bit of grok match failures. It is plausible that the regular expression engine tried parsing the pattern(s), and may have gone into a backtracking loop wasting a lot of CPU cycles, contributing to the overall slowness.
<img src="https://api.contentstack.io/v2/uploads/588fed461dd6e41e6ff76159/download?uid=blt6092a17c7ccb1cba" data-sys-asset-uid="blt6092a17c7ccb1cba" alt="tip.png" style="background-color: initial;float: left;margin: 0px;text-align: left;" />&nbsp; An important tip to our readers — if you see the response above, the ID field contains a randomly UUID which identifies each of the filters. You can also, and its highly recommended that you set the ID field in your config file to label plugins appropriately. This can help a lot when&nbsp;debugging a pipeline with multiple grok filters. For example, in your config file :<pre class="prettyprint">filter { grok { id =&gt; "apache_common_log_praser" pattern =&gt; .. } grok { id =&gt; "java_exception_logs_parser" pattern =&gt; .. } } </pre>
Now that we've identified the grok filter as the slowest, we can tweak the regular expression pattern to optimize the performance. We have <a href="https://www.elastic.co/blog/do-you-grok-grok">previously posted</a> about various ways to tame your Grok filter, which is a highly recommended read.
If you head over to the monitoring UI, the events latency graph shows you aggregated information on how long it takes on average for events to be processed in your pipeline. 
<img src="https://api.contentstack.io/v2/uploads/588fec3b647b87d73bfa45bd/download?uid=blt7676c01a7e3493a7" data-sys-asset-uid="blt7676c01a7e3493a7" alt="Screen Shot 2017-01-29 at 08.37.54.png" width="426" height="141" style="width: 426;height: 141;" />
One of my favorite parts of shipping monitoring data to Elasticsearch is the access to historical information. As you iterate over your configs, you can validate from these graphs if your changes made a difference to the overall pipeline latency. This historical information can also be used to spot performance regressions over time in your production setting. You are no longer flying in the dark! 
<h2>Slowlog</h2>
In our previous section, we showed you how to identify the slow sections of the pipeline. But, this doesn't pinpoint which events may have contributed to the slowdown. Enter the <a href="https://www.elastic.co/guide/en/logstash/current/logging.html#_slowlog">slowlog feature</a>. Slowlog adds the ability to log when a specific event takes an abnormal amount of time to make its way through the pipeline. You can define the criteria when you want the slowlog to be triggered. Just like normal application logs, you can find slow-logs in your —path.logs directory. In your logstash.yml file, add this setting: <pre class="prettyprint">slowlog.threshold.warn: 2s </pre>
When events take more than 2 seconds to be processed, you should see: <pre class="prettyprint">[2016-10-27T13:48:19,308][WARN ][slowlog.logstash.filters.grok] event processing time {:took_in_millis=&gt; 2130, :event=&gt;"{\"sequence\":1,\"@timestamp\":\"2016-10-27T20:48:18.150Z\",\"@version\":\"1\",\"host\":\"elasticbox-2.local\",\"mes sage: \"217.69.133.237 - - [11/Sep/2014:21:58:29 +0000] "GET /scripts/netcat-webserver HTTP/1.1" 304 - "-" "Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/2.0; +http://go.mail.ru/help/robots)""\"}\"} </pre>
<h2>Hot Spots</h2>
If the CPU on your Logstash instance is running red hot, you can issue a command like top -H to analyze what parts of the Logstash pipeline is consuming the most CPU. Logstash threads are labeled based on the stage and plugin name, so from the output of top, one can quickly find out which parts are CPU intensive. For example, [main]&gt;worker1 denotes a filter + output thread, while [main]&lt;generator denotes a generator input. 
<img src="https://api.contentstack.io/v2/uploads/588fed9e2886d7f63f63b5d7/download?uid=blt1f9f15aed960df49" data-sys-asset-uid="blt1f9f15aed960df49" alt="top_output.png" width="396" height="96" style="width: 396;height: 96;" />
While this is super useful, it would be nice to zoom into those threads to find the expensive parts. We have an API just for that — <a href="https://www.elastic.co/guide/en/logstash/current/hot-threads-api.html">the hot threads API</a> provides a thread dump or a snapshot of the java threads sorted by the time spent. <pre class="prettyprint">curl localhost:9600/_node/hot_threads?human ::: {} Hot threads at 2017-01-29T04:18:55+00:00, busiestThreads=3: ================================================================================-------------------------------------------------------------------------------- 64.07 % of cpu usage, state: runnable, thread name: '[main]&gt;worker0' org.jruby.RubyRegexp$SearchMatchTask.run(RubyRegexp.java:273) org.jruby.RubyThread.executeBlockingTask(RubyThread.java:1066) org.jruby.RubyRegexp.matcherSearch(RubyRegexp.java:235) org.jruby.RubyRegexp.search19(RubyRegexp.java:1780) org.jruby.RubyRegexp.matchPos(RubyRegexp.java:1720) org.jruby.RubyRegexp.match19Common(RubyRegexp.java:1701) org.jruby.RubyRegexp.match_m19(RubyRegexp.java:1680) org.jruby.RubyRegexp$INVOKER$i$match_m19.call(RubyRegexp$INVOKER$i$match_m19.gen) org.jruby.internal.runtime.methods.JavaMethod$JavaMethodOneOrNBlock.call(JavaMethod.java:350) org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:168) </pre>
In the above example response, we see again that the regex engine used by the Grok filter is the busiest thread. You could also use this API to find out if your Logstash is stuck performing I/O operations. 
<h2>Conclusion</h2>
In this post, we introduced you to our new monitoring UI feature and showed a few ways to identify slow parts of Logstash. We are not done yet — our roadmap has exciting monitoring features that will make Logstash and your pipelines even more of a breeze to manage. This post is part of a series on monitoring, so stay tuned for our next one and all our product updates.

events_rate.png

Monitoring Logstash Filters: X-ray glasses included

44281324-e7a8c880-a203-11e8-8b54-3a15c15f34a3.jpg

44281325-e7a8c880-a203-11e8-9fed-61b9ee218354.jpg

Elasticsearch Audit Trail and Log File Filter Policies Explained

This blog post outlines the upgrade path that you can use to upgrade your Elasticsearch 1.7 cluster hosted on Elasticsearch Service to a supported version. Newer versions of Elasticsearch include an incredible amount of features like <a href="/products/stack/canvas">Canvas</a>, <a href="/products/beats">Beats</a>, <a href="/solutions/apm">APM</a> and much more. Keeping a cluster updated to the latest version also provides other benefits, such as access to the latest security patches and performance improvements.
As per our <a href="/support/eol">product end-of-life policy</a>, Elasticsearch 1.7 reached end of life (EOL) 26 months ago. In late 2018, we began an email campaign informing customers that <a href="/cloud/elasticsearch-service">Elasticsearch Service</a> will cease operations for Elasticsearch 1.7 clusters on April 8, 2019. Although Elasticsearch Service makes upgrading to the next major version very easy via the console, unfortunately this upgrade path isn’t available for extremely old versions of Elasticsearch.
At a minimum, you need to use the steps in this blog post to upgrade to version 5.6.x, but we recommend that you also upgrade to Elasticsearch 6.x to avoid having to deal with an EOL version again in the near future. You should stay on version 5.6.x only if you cannot switch to a <a href="/blog/removal-of-mapping-types-elasticsearch">single mapping type</a> for indices easily and need more time before upgrading to 6.x.
<h2>How to upgrade your Elasticsearch 1.7 cluster</h2><ol>
	<li>Log into the <a href="http://cloud.elastic.co">Elasticsearch Service console</a>.</li>
	<li>Create a new 5.6.x deployment using the latest available version.</li>
	<li>Index your data into the new deployment:</li>
	<ul>
		<li style="list-style-type:disc;">If you have access to the source, index all of your data directly into the new deployment from the source application. </li>
		<li style="list-style-type:disc;">Alternatively, if you do not have access to the data source, you can use <a href="/guide/en/elasticsearch/reference/6.6/docs-reindex.html">reindex from remote</a> from your new 5.6.x deployment, where the source is set to your existing Elasticsearch 1.7 deployment. As described below, ensure that breaking changes, like mapping updates for string data types, are addressed prior to reindexing. Please also note, it would be advisable to create these indices with the appropriate mappings beforehand. 
		<pre class="prettyprint">POST _reindex
{
 "source": {
 "remote": {
 "host": "https://SOURCE_1_7_CLUSTER_URL:9243",
 "username": "SOURCE_1_7_CLUSTER_USERNAME",
 "password": "SOURCE_1_7_CLUSTER_PASSWORD"
 },
 "index": "INDEX_NAME",
 "query": {
 "match_all": {
 }
 }
 },
 "dest": {
 "index": "INDEX_NAME"
 }
}
 </pre> Important: Be sure to replace <code>SOURCE_1_7_CLUSTER_URL</code>, <code>SOURCE_1_7_CLUSTER_USERNAME</code>, <code>SOURCE_1_7_CLUSTER_PASSWORD</code>, and <code>INDEX_NAME</code> with your own information.
		</li>
	</ul>
	<li>Upgrade to 6.6.x via the Elasticsearch Service console using the Upgrade Assistant. </li>
</ol>There are some breaking changes to be aware of, though this list is not exhaustive. Please reference the breaking changes for <a href="/guide/en/elasticsearch/reference/5.6/breaking-changes-5.0.html">5.x</a> or <a href="/guide/en/elasticsearch/reference/current/breaking-changes-6.0.html">6.x</a>, depending on which version you are upgrading to:
<ul>
	<li>When upgrading from 1.7 to 5.6/6.x, the ‘string’ data type was used for all text strings, with <code>"type": "string", "index": "analyzed"</code> for full text search capable fields and <code>"type": "string", "index": "not_analyzed"</code> for fields that required an exact value. To learn more, see <a href="/blog/strings-are-dead-long-live-strings">Strings are dead, long live strings!</a> This behavior has been changed and simplified:</li>
	<ul>
		<li><code>"type": "string", "index": "analyzed"</code> now is just <a href="/guide/en/elasticsearch/reference/6.6/text.html"><code>"type": "text"</code></a></li>
		<li><code>"type": "string", "index": "not_analyzed"</code> now is just <a href="/guide/en/elasticsearch/reference/6.6/keyword.html"><code>"type": "keyword"</code></a> </li>
	</ul>
	<li>A full list of the breaking changes introduced with each major and minor update can be found in our documentation for each version: <a href="/guide/en/elasticsearch/reference/2.4/breaking-changes-2.0.html">2.x</a>, <a href="/guide/en/elasticsearch/reference/5.6/breaking-changes-5.0.html">5.x</a>, <a href="/guide/en/elasticsearch/reference/6.6/breaking-changes-6.0.html">6.x</a>. The breaking changes for minor version are listed in the documentation menu.</li>
</ul>If none of the options in this blog post work for you, you can also consider moving your data off Elasticsearch Service entirely and run Elasticsearch on your own. To get your data off Elasticsearch Service, snapshot your Elasticsearch 1.7 cluster to a <a href="/guide/en/cloud/current/ec-custom-repository.html">custom repository</a> and then restore the snapshot to a self-managed Elasticsearch 1.7 cluster. Note that this version of Elasticsearch is known to have security issues, is no longer being patched, and is not recommended.
We’ve added a huge amount of new functionality to the Elastic Stack since Elasticsearch 1.7. If you haven't seen them yet, explore the <a href="/products">new features</a> that have been released in the past few years, or even join the <a href="/v7">7.0 Elastic Pioneer Program</a> to test out features that are just around the corner.

blog-banner-release-elasticsearch.png

blog-thumb-release-elasticsearch.png

Elastic Support Alert: Upgrading Your Elasticsearch 1.7 Cluster on Elastic Cloud

Elastic Cloud and Meltdown

<html><head></head><body>Quite some time ago, with the GA release of Elasticsearch 5.0 we released an <a href="/guide/en/elasticsearch/reference/current/indices-shrink-index.html">API</a> that allows to shrink an index into a new index with fewer shards than the original index. The reasoning behind adding this functionality was to provide a tool to keep the number of shards in a cluster at bay. Indices are commonly created with a large number of shards to maximise indexing throughput but later once theses indices are rolled over into a daily or hourly index, the number of shards should be reduced again to maximise resource utilization of the cluster. While this was a step in the right direction, we were still missing the ability to go in reverse: to increase the number of shards in an index if the index has become too small for the number of documents in it.
<h2>We had a story for this, we thought. </h2>For the entire history of Elasticsearch the answer to the question of how to split an index was to just add another index and grow your shard count that way. Under the hood Elasticsearch just operates on a set of shards which allows you to add indices after the fact and search across them. This sounds like a great plan for use-cases where documents are largely immutable like metrics and logging, but not as great when &#xA0;you have to update documents, as is the case for use cases like full-text search. For this you had to go and figure out in what index the document was added and update it there or you delete the document everywhere and add it to the latest index. All of these options have downsides and can cause temporary duplicates or missing documents which are not always acceptable.
After having some deep technical conversations with some users on a trip to Munich I was convinced we have to do something about this. But some of the options scared me a fair bit. For instance splitting an index or a shard while the index is actively indexing sounds very appealing in the first place but when you start thinking about the implications and how it would work under the hood, the approach doesn&#x2019;t sound all that great anymore. Aside from all the internal changes of how we do consistent hashing, how the number of shards are immutable in an index and how we can make this all work in a backwards compatible way, the performance implications of resharding while a cluster is rebalancing or loses a node had so many edge cases that this feature would likely take many engineering iterations to get right.
Luckily we already had implemented the shrink API that was basically doing the same thing in reverse with a lot of interesting properties. Shrink works by creating a copy of the index that recovers from the original index but uses fewer shards. Wait, a copy? That sounds very expensive copying all the data. To prevent this we make use of the properties lucene provides us by never touching a file more than once. Given that we can take point-in-time snapshots of the source index and use hard-links on the filesystem level, the shrink operation is almost instant and doesn&#x2019;t require any additional space. All properties of Elasticsearch indices (such as the fixed number of shards) are maintained during this time. If any issues occur during the shrink, the shrink can be aborted and the target index deleted.
This sounds great, let&#x2019;s see if we can do the same thing for splitting an index as well. And in fact it worked like a charm. The <a href="/guide/en/elasticsearch/reference/current/indices-split-index.html">split</a> API uses the same mechanics as the shrink API. Once an index is split, Elasticsearch allocates the shards of the target index alongside of the source index and recovers from the point-in-time snapshot of the underlying Lucene index. The only significant difference is that we need to do a single pass through the index on disk to mark all documents as deleted that don&#x2019;t belong in the target shard. This means we have an additional space requirement of 1 bit per document while all other data files are reused via hard-links. Just like the shrink process this is a reasonably fast operation that typically happens within a couple of seconds and can be monitored with the <a href="/guide/en/elasticsearch/reference/current/indices-recovery.html">recovery API</a>.
Unfortunately you can&#x2019;t rename an index in Elasticsearch today. To make efficient use of this API, the usage of <a href="/guide/en/elasticsearch/reference/current/indices-aliases.html">index aliases</a> is highly recommended. If an index needs to be resized, users should split or shrink the index, flip an alias to point to the new index and delete the old index in a single atomic step, using the <a href="/guide/en/elasticsearch/reference/6.2/indices-aliases.html">update-aliases API</a>.
<h2>It&#x2019;s all about trade-offs.</h2>That is likely the story of every software engineers life. In this case we made the trade-off that indexing has to stop for a short amount of time until the index is resized since source indices must be read-only. But as mentioned above these operations are considered fast, and given that this provides transactional behavior with the least amount of surprise to the user we are following our consistent strategy to go with the safe option rather than the fancy one. For details on split factors and how to pick the number of target shards, please see the official <a href="/guide/en/elasticsearch/reference/current/indices-split-index.html">documentation</a> for the specific version of Elasticsearch you are using.
</body></html>

Resizing Elasticsearch shards for fun and profit

gears-fitting-together-2000-415.jpg

gears-fitting-together-420-220.png

Plugins Alongside the X-Pack Security Plugin for Elasticsearch

<html><head></head><body><section id="post_content">
		<article>
			TLDR: This blog post will show you how to use the percolator to enrich documents. In this particular use case we will find out the country of a certain latitude/longitude pair.
Imagine you are using Yelp or Foursquare to search for restaurants. Your current location is characterized by latitude/longitude, as is the restaurant you are looking for. This is a typical geo use case that involves finding a bunch of points within a bounded area. So what if you already know the point, but want to find the bounded area?
<h1>Making the world JSON</h1>
In order to start, we need to have access to the borders of all countries and store them in Elasticsearch. The first step is really easy, as there are a couple of JSON files available that represent country shapes. In this post, we will stick with the GeoJSON based set of Johan Sundstr&#xF6;m, available in the <a href="https://github.com/johan/world.geo.json/">world.geo.json github repo</a>. I used <code>countries.geo.json</code>, which is available <a href="https://raw.githubusercontent.com/johan/world.geo.json/master/countries.geo.json">here</a>.
The only drawback is that all the countries are in a JSON document in a single file. However ,we need a way to register a percolator per country.
Let&#x2019;s start easy and create an index with a location type, that features a <code>geo_shape</code>.
<pre class="prettyprint lang-js">PUT /countries
PUT /countries/location/_mapping
{
 &quot;location&quot; : {
 &quot;properties&quot; : {
 &quot;location&quot; : {
 &quot;type&quot; : &quot;geo_shape&quot;
 }
 }
 }
}
</pre>
<h1>Registering percolators</h1>
The first step is to modify the existing JSON. Instead of going big and using Logstash for extraction, today we will stay on the command line and use the excellent <a href="https://github.com/ddopson/underscore-cli">underscore-cli</a>. This client gives you the power of underscore on the command line, thanks to node.js.
If you want to install underscore on OSX, just use <code>brew</code> and <code>npm</code>
<pre class="prettyprint lang-shell">brew install node
npm -g install underscore-cli
</pre>
Now, you can run <code>underscore help</code> on the command line.
<pre class="prettyprint lang-bash">IFS=n&apos;
for i in $(underscore select &apos;:has(:root &gt; .id)&apos; &lt; countries.geo.json --outfmt text | grep -v &apos;&quot;id&quot;:&quot;-99&quot;&apos;) ; do
 country=$(echo &quot;$i&quot; | underscore extract id | tr -d &apos;&quot;&apos;)
 geometry=$(echo &quot;$i&quot; | underscore extract geometry)

 curl -X PUT localhost:9200/countries/.percolator/$country -d &quot;{ \&quot;query\&quot; : { \&quot;geo_shape\&quot; : { \&quot;location\&quot; : { \&quot;shape\&quot; : $geometry } } }, \&quot;type\&quot; : \&quot;location\&quot; }&quot;
done
</pre>
Quite a few bits happened here, which you might not be familiar with. First the <code>countries.geo.json</code> file is parsed and for each <code>id</code> found an entry is extracted using the <code>select</code> function. As we rely on the id to identify the country, the <code>-99</code> value is filtered out in order to not index invalid ids. The id is the three-digit ISO code (for example <code>USA</code> or <code>DEU</code>), which will be the name of the registered percolator. The last step is to extract the geo shape from the read JSON and then use the id and that JSON to register a percolation query.
In case you are wondering about the first line, it is to prevent the for loop from only treating newlines as the input field separator; by default, spaces are used as well.
After this script has run, you can verify with the count API that queries have been registered.
<pre class="prettyprint lang-js">GET /countries/.percolator/_count
</pre>
<h1>Finding countries</h1>
Now the infrastructure is set up and you can start querying. Whenever you get a document with latitude and longitude, you can easily find out the country that the location belongs to. See these example queries:
<pre class="prettyprint lang-js">GET /countries/location/_percolate
{
 &quot;doc&quot;: {
 &quot;name&quot;: &quot;Marienplatz Munich&quot;,
 &quot;location&quot;: {
 &quot;type&quot;: &quot;point&quot;,
 &quot;coordinates&quot;: [ 11.575448, 48.137393 ]
 }
 }
}

{
 ...
 &quot;total&quot;: 1,
 &quot;matches&quot;: [
 {
 &quot;_index&quot;: &quot;countries&quot;,
 &quot;_id&quot;: &quot;DEU&quot;
 }
 ]
}


GET /countries/location/_percolate
{
 &quot;doc&quot;: {
 &quot;name&quot;: &quot;Statue of Liberty&quot;,
 &quot;location&quot;: {
 &quot;type&quot;: &quot;point&quot;,
 &quot;coordinates&quot;: [ -74.0445, 40.689249 ]
 }
 }
}

{
 ...
 &quot;total&quot;: 1,
 &quot;matches&quot;: [
 {
 &quot;_index&quot;: &quot;countries&quot;,
 &quot;_id&quot;: &quot;USA&quot;
 }
 ]
}


GET /countries/location/_percolate
{
 &quot;doc&quot;: {
 &quot;name&quot;: &quot;Amsterdam&quot;,
 &quot;location&quot;: {
 &quot;type&quot;: &quot;point&quot;,
 &quot;coordinates&quot;: [ 4.8986166, 52.3747158 ]
 }
 }
}

{
 ...
 &quot;total&quot;: 1,
 &quot;matches&quot;: [
 {
 &quot;_index&quot;: &quot;countries&quot;,
 &quot;_id&quot;: &quot;NLD&quot;
 }
 ]
}
</pre>
As you can see, the <code>_id</code> field being returned now contains the three digit ISO code we used as an id to register the percolator. You could potentially use this result, extract the id and add it to the document in order to run aggregations per country.
In fact, this is just one example of how to use the percolator to enrich documents. You can consider this a pre-processing step. Even before you index your data, you can execute a percolate query and enrich your document with the data returned, and then index it. You could use this for categorization (e.g. if your document description matches a <code>pizza</code> query it might be a restaurant and you could tag it like that) or enrich documents in order to aggregate on them (how many check-ins happened in Germany in the last month?). Or you could just track movements of rental cars and make sure they never leave the country the were rented in.
You could try to get more accurate data (by state or by city or another region) and enrich based on that.
If you have more awesome (or rare!) use cases about the percolator, we would love to know about it! Share them on <a href="http://twitter.com/Elasticsearch">Twitter</a> and who knows, they may even be featured on our site!
			<aside id="post_tags">
							</aside>
		</article>
			</section></body></html>

Using the Percolator for Geo Tagging

<html><head></head><body>Welcome to the next step in our story of bringing you great new features!&#xA0;Where previous versions of Elastic Cloud Enterprise&#xA0;either required you to reinstall or included only new Elastic Stack packs, you can now live upgrade a production ECE installation to a new version with little or no downtime.
What is &#x201C;little or no downtime?&#x201D; If your ECE installation includes more than one proxy and if you are using a load balancer as recommended, there is no downtime. (Installations with a single proxy or no load balancer will experience a short outage as each proxy is upgraded.)&#xA0;You simply initiate the upgrade process and let ECE upgrade itself:&#xA0;ECE will replace the containers with its own software and dependencies one by one, on all hosts. Upgrading ECE does not touch any of the containers that run your Elasticsearch clusters and Kibana instances. Your applications continue to perform as expected and your users won&#x2019;t know that you are replacing their spaceship piece by piece in mid-flight. We like to call this operational happiness. There are two ideas behind our new live- upgrade functionality: to make the upgrade process safe and to make it simple. It&#x2019;s all designed to just work.
<ul>
	<li>To make upgrading safe, the entire process is designed to be failsafe. Containers get upgraded sequentially and the upgrade status of each container is tracked. The upgrade process also monitors that each new container is viable. If there is an issue with any part of the upgrade, the entire process is rolled back.</li>
	<li>To make upgrading simple, a single command performs the upgrade across your entire installation. There&#x2019;s no need to connect to individual hosts to upgrade them one by one and there are no additional manual steps to get your ECE installation going again after the upgrade. Best of all, there&#x2019;s no need to stop ECE.</li>
</ul>Before you start upgrading, we recommend that you <a href="/guide/en/cloud-enterprise/1.0/ece-upgrade.html#ece-upgrade-prereqs">check the prerequisites</a>. There is some small print you should know around free disk space and things you shouldn&#x2019;t be doing whilst you upgrade. (Don&#x2019;t add or remove hosts during the upgrade process, for example.)
Ready? To upgrade your installation:
1. Download and run the latest installation script for Elastic Cloud Enterprise 1.0.2 with the <code>upgrade</code> action from a host that holds the director role:
<pre class="prettyprint" style="margin-left: 20px;">bash &lt;(curl -fsSL https://download.elastic.co/cloud/elastic-cloud-enterprise.sh) upgrade
</pre><ol>
</ol>You can follow along whilst each container for ECE is upgraded on the hosts that are part of your installation. After the upgrade process completes, <a href="/guide/en/cloud-enterprise/1.0/ece-manage-elastic-stack.html">add the Elastic Stack packs</a> that ship with ECE 1.0.2 to get every last little bit of the new version.
To learn more about how the upgrade process works, see <a href="/guide/en/cloud-enterprise/1.0/ece-upgrade.html">Upgrade Your Installation</a>. And if you are curious to know what else is in version 1.0.2, take a look at our <a href="/guide/en/cloud-enterprise/1.0/ece-release-notes-1.0.2.html">Release Notes</a>.
</body></html>

Rebuilding-Rocket-2000px.png

Rebuilding-Space-Station-Thumnail.jpg

Live Upgrading Elastic Cloud Enterprise

<html><head></head><body>Elasticsearch 5.x has ten new <a href="https://www.elastic.co/guide/en/elasticsearch/reference/5.0/bootstrap-checks.html">&quot;bootstrap checks&quot;</a> which run when Elasticsearch starts up to check for configuration problems that might cause exciting failures after a node sees serious use. If any of these check fail, the node will abort during startup if it is bound to a non-local IP address. We earnestly hope that these checks are just a minor annoyance and that they prevent major heartache.<h2>Serial Killer</h2>Concrete example time! We recently added a <a href="https://github.com/elastic/elasticsearch/blob/7eb6260131aeabf25ff8301b2e3b9ada6297e692/core/src/main/java/org/elasticsearch/bootstrap/BootstrapCheck.java#L502-L532">check</a> that fails if we detect that the JVM that Elasticsearch is running in is using the &quot;serial&quot; collector. The serial collector is designed for single-threaded applications or applications running with extremely small heaps. Elasticsearch is very much a multi-threaded application and will not fare well with an extremely small heap so the collector isn&#x2019;t a good fit for it. This collector can cause nodes to time out of the cluster because this collector stops the world while it is running and those pauses can be quite substantial for large heaps because the collection process is single-threaded. If these pauses run longer than ninety seconds the master node might time out its periodic pings to the node and remove it from the cluster. When the node finishes its long GC cycle it will rejoin the cluster, causing the node to &quot;bounce&quot; in and out of the cluster.We hope the <a href="https://www.elastic.co/guide/en/elasticsearch/reference/5.0/_use_serial_collector_check.html">serial collector check</a> saves someone a lot of trouble down the line. In fact we&#x2019;re fairly sure it will because we&#x2019;ve seen the serial collector in action. One thing buying a subscription from Elastic entitles you to is support and &quot;one of my nodes is periodically leaving the cluster and coming back&quot; is a thing that comes up from time to time. Sometimes we track the failures back to the serial garbage collector. Users won&#x2019;t see those failures in production on 5.x because Elasticsearch will refuse to start with the serial garbage collector at all.

Since the startup scripts explicitly set up a different collector we expect this check to trigger very rarely but given how insidious the failure mode is we thought it was worth checking.<h2>The File Limits are Too Damn Low</h2>Another issue that we see very frequently is &quot;too many open files&quot;. This happens when a user (often unknowingly) has their file descriptor limits set far too conservatively, like 4096. Elasticsearch needs lots of file descriptors to open many indices composed of many segments (and, on Unix-based operating systems, things like network sockets are file descriptors too). If this limit is set too low and Elasticsearch&#xA0;bumps up against the limit, things like shard allocation can fail and disaster can ensue.So, we added the <a href="https://www.elastic.co/guide/en/elasticsearch/reference/5.0/_file_descriptor_check.html">file descriptor bootstrap check</a>. It&#x2019;s the original bootstrap check, inspired by <a href="https://github.com/elastic/elasticsearch/issues?utf8=&#x2713;&amp;q=is%3Aissue%20is%3Aclosed%20%22too%20many%20open%20files%22%20">issues</a> opened on our <a href="https://github.com/elastic/elasticsearch">GitHub repository</a>, and <a href="https://discuss.elastic.co/search?q=%22too%20many%20open%20files%22%20category%3A6">posts</a> on our <a href="https://discuss.elastic.co/c/elasticsearch">discussion forum</a>. It simply checks at startup the current file descriptor limit and screams loudly if it&#x2019;s below 65536. It can take a few minutes to track down exactly the right incantation to increase the limit on your operating system and this is super annoying but it sure beats losing data in production because you never knew the limit should be higher.<h2>These Warnings go to Eleven</h2>A core value of the Elasticsearch team is &quot;it should be easy to start exploring Elasticsearch&quot;. In the past few years this <a href="https://www.elastic.co/blog/elasticsearch-the-server">has to compete</a> with values like &quot;Elasticsearch should be secure&quot; and &quot;Elasticsearch should be stable&quot; but it is still important. In that vein, we still want Elasticsearch to start for folks that just download a distribution and start it without configuring anything. Every failing bootstrap check will simply print a <code>WARN</code>.<pre class="prettyprint">[2016-09-23T12:19:25,588][WARN ][o.e.b.BootstrapCheck ] [DOteo05] max file descriptors [4096] for elasticsearch process likely too low, increase to at least [65536]</pre>and Elasticsearch will start as though nothing was wrong. We hope these warnings makes a first time user think &quot;Elasticsearch cares enough about stability to warn me when something is poorly configured.&quot;The trouble is that people ignore warnings. We really want first time users to scan the list of warnings, realize that they should go fix those things before going to production, and keep playing with Elasticsearch. We hope they play long and hard and fall in love with Elasticsearch like we all did. After all that, we expect folks that&#xA0;saw the warnings to forget that there are things to clean up. And, of course, there are folks who start Elasticsearch in the background and don&#x2019;t read the logs at all, missing the warnings entirely.So for these bootstrap checks to really be effective they have to change from warnings to actual errors at some point. They have to stop Elasticsearch from starting and make you fix them. We want to be sure that any &quot;real&quot; installation of Elasticsearch passes all of these checks so we don&#x2019;t have to help anyone recover from a disaster caused by something like running out of file descriptors. The best measure we could find for what makes a &quot;real&quot; installation is binding to a non-local IP address. We briefly toyed with the idea of an explicit &quot;production mode&quot; flag but decided against it because it would be too easy for folks to turn it off to &quot;work around&quot; one of the bootstrap checks.Binding to a non-local IP address is a fairly good way to detect a &quot;real&quot; installation because you can&#x2019;t form a multi-node cluster without doing so. And you can&#x2019;t serve requests from servers other than the one running Elasticsearch either. Well, you could play tricks with tunnels but that is fairly arcane. Anyway, we think that 99% of folks that have &quot;real&quot; &quot;production&quot; installation of Elasticsearch will bind to a non-local IP address. We also think that most folks that bind to a non-local IP address are either setting up a &quot;real&quot; installation of Elasticsearch or are running Elasticsearch in a container or VM, presumably set up using a configuration management system like Puppet/Chef/Ansible/Salt/bash-scripts-and-duct-tape.So that is the bootstrap checks, annoying you now so you don&#x2019;t get paged on a Saturday morning at 3 AM because a bulk import caused a long GC which caused a node to drop out of the cluster, all because the wrong garbage collector is configured.</body></html>

hand-smaller.jpg

bootstrap-checks-thumbnail.jpg

Bootstrap checks: Annoying you now instead of devastating you later!

<html><head></head><body>If you have no idea what questions you will want to ask your data when you start ingesting it, columnar storage is probably a good option for you: it helps in two areas that are often close to the heart of users who deal with large amounts of data:
<ul>
	<li>Storage efficiency: Since data that belongs to the same field is stored together, it can be expected to be quite homogeneous, which is something that can be leveraged to make <a href="https://en.wikipedia.org/wiki/Column-oriented_DBMS#Compression">compression</a> more efficient.</li>
	<li>&quot;Fast&quot; queries/aggregations: The fact that data for the same field is stored together also helps make <a href="https://en.wikipedia.org/wiki/Column-oriented_DBMS#Benefits">better use of the file-system cache</a> since you will not have to load data for fields that are not needed by the query. In addition, by splitting the data into blocks and adding some meta-data about the range or set of unique values in the header of the block, some queries may be able to skip some blocks entirely at query time. There are also some techniques that allow to <a href="https://en.wikipedia.org/wiki/Succinct_data_structure">query the data without decompressing it first</a>.</li>
</ul>You might have noticed the quotes around Fast in the above paragraph. The reason is that however fast a linear scan is, it is still a linear scan and performs in linear time with the amount of queried data. Elasticsearch takes a different approach that consists in indexing all fields by default and only exposing queries that can leverage these indices so that identifying the matching documents does not need to visit all the data. Well, almost only, there is one query that might run a linear scan in order to identify matching documents, the <a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-script-query.html">script query</a>. The purpose of this query is to not require you to reindex for once-in-a-while questions that you had not thought you would need to ask your data when you indexed it, such as finding all documents whose value for field X is less than the value of field Y. But other than this one, all queries make use of some form of index in order to quickly identify matching documents.
While Elasticsearch does not use a column-oriented view of the data for searching, it still needs one for workloads that work best with columnar data such as sorting and aggregations. In the next sections, we will do a quick survey of the history of columnar data in Lucene and Elasticsearch.
<h2>First, there was fielddata</h2>Lucene was originally designed as a search library, allowing users to get the most relevant documents for a particular query. But soon users wanted to do more: they wanted to be able to sort by arbitrary fields, aggregate information about all matching documents, etc. To cover these needs, Lucene added a feature called <a href="https://lucene.apache.org/core/3_0_3/api/core/org/apache/lucene/search/FieldCache.html">FieldCache</a>, which would &quot;uninvert&quot; the inverted index in order to build a column-oriented view of the data in memory. The initial release of Elasticsearch back in 2010 used the same mechanism with what it called fielddata, which was similar to <code>FieldCache</code>, but with more flexible caching management.
<h2>Then doc values</h2>The growing use of <code>FieldCache</code> was embarassing: it required an amount of memory that was linear with the amount of indexed data, and made reopening the index slower since <code>FieldCache</code> entries had to be reloaded on all new segments, some of which being potentially very large due to merging. So on the one hand you had an efficient inverted index structure that allowed to find matching documents, but most collectors then had to rely on this inefficient memory-intensive data-structure in order to compute interesting things about the data. This is what lead Lucene to introduce <a href="http://blog.trifork.com/2011/10/27/introducing-lucene-index-doc-values/">doc values</a> in Lucene 4.0, which was released in the end of 2012. Just like <code>FieldCache</code>, doc values provide a column-oriented view of the data, except that they are computed at index time and stored in the index. Their memory footprint is very low and getting doc values ready to use on a new segment is a matter of opening a file.
The fact that doc values are computed at index time also gives more opportunities for compression. The longer it takes to uninvert fielddata, the longer users have to wait before changes to the index becomes visible, which is undesirable. On the other hand doc values are either computed asynchronously at merge or on small datasets at flush time, so it is fine to spend more time doing interesting compression. Here is a non exhaustive list of some compressions techniques that doc values use that fielddata doesn&apos;t:
<ul>
	<li>Since doc values can afford to perform two passes on the data, they do a first pass on numeric fields to compute the required number of bits per value and a second one to do the encoding. This way they can use a fine-grained list of numbers of bits per value for the encoding. On the other hand fielddata only used 8, 16, 32 or 64 since changing the number of bits on the fly would incur a costly resize.</li>
	<li>The (sorted) terms dictionary for string fields is split into blocks and each block is compressed based on shared prefixes between consecutive terms.</li>
	<li>Numeric doc values compute the greatest common divisor between all values and only encode the non-common part. This is typically useful for dates that only have a second or day granularity.</li>
</ul><h2>Elasticsearch integration</h2>Doc values became available in Elasticsearch in version <a href="https://www.elastic.co/blog/1-0-0-released">1.0</a> (February 2014) as an opt-in. However at that time, the performance did not quite match that of fielddata yet: Elasticsearch was hiding doc values behind its existing fielddata API, introducing overhead, and it took Lucene some time before introducing a <a href="https://issues.apache.org/jira/browse/LUCENE-5748">dedicated type for multi-valued numerics</a> and a <a href="https://issues.apache.org/jira/browse/LUCENE-5731">random-access API for index inputs</a> that helped performance significantly. Both these concerns got fixed in <a href="https://www.elastic.co/blog/elasticsearch-1-4-0-released">Elasticsearch 1.4</a>, which was the first release to have matching performance of fielddata and doc values. We then enabled doc values by default in Elasticsearch 2.0 and the next major release of Elasticsearch (5.0) will not support fielddata anymore, leaving doc values as the only option for having a columnar representation of the data.
<h2>Specifics of Elasticsearch&apos;s column store</h2>Does it make Elasticsearch a good general-purpose replacement for column-stores? No. As usual, it is better to <a href="https://en.wikipedia.org/wiki/Unix_philosophy#Do_One_Thing_and_Do_It_Well">do one thing and do it well</a>. But the Elasticsearch column store has one key characteristic that makes it interesting: values are indexed by the doc id of the document that they belong to. What does it mean? Doc ids are transient identifiers of documents that live in a segment. A Lucene index is made of several components: an inverted index, a <a href="https://www.elastic.co/blog/lucene-points-6.0">bkd tree</a>, a column store (doc values), a document store (stored fields) and term vectors, and these components can communicate thanks to these doc ids. For instance, the inverted index is able to return an iterator over matching doc ids, and these doc ids can then be used to look up the value of a field thanks to doc values: this is how aggregations work.
This makes Elasticsearch particularly good at running analytics on small subsets of an index, since you will only pay the price for documents that match the query. This is how user interfaces to Elasticsearch like Kibana make it easy to slice and dice the data, by recursively filtering subsets that seem to have some interesting properties and running analytics on them.
Moreover, Lucene is geared towards making search operations fast. Thus doc values do not store the raw bytes for each document in case of a string field. Instead it writes separately a terms dictionary containing all unique values in sorted order and writes the indexes of string values in the column store. This helps since small integers make better keys than strings and allow to run eg. terms aggregations more efficiently by keying on the term index rather than the term bytes and using an array rather than a hash table as a data structure for storing per-term counts.<h2>What&apos;s next?</h2>Given that doc values need to be indexed by the doc id of the document they belong to, the current approach that Lucene takes is to reserve a fixed amount of space per document. While this makes doc values lookups very efficient, this has the undesired side-effect of not being space-efficient for sparse fields, since documents that do not have a value would still require the same amount of storage as documents that have values. This is why Elasticsearch works best when all documents in an index have a very similar set of fields.
However, there are ongoing developments that are exploring <a href="https://issues.apache.org/jira/browse/LUCENE-7407">switching doc values to an iterator API</a> rather than a random-access API so that these sparse cases could be handled more efficiently and that compression could be more efficient using techniques like <a href="https://fr.wikipedia.org/wiki/Run-length_encoding">run-length encoding</a>.</body></html>

blog-column-store-fb.jpg

blog-column-store-thumb.jpg

Elasticsearch as a column store

<html><head></head><body>A year and a few months ago, <a href="/blog/index-vs-type">we blogged about the differences of types and indices</a>, including &quot;when to pick which.&quot; &#xA0;If you don&apos;t wish to read the full history, I&apos;ll give you the TL;DR:&#xA0;the conclusion was something like: 
<blockquote>Multiple types in the same index really shouldn&apos;t be used all that often and one of the few use cases for types is&#xA0;parent child relationships.
</blockquote>Sadly, not everyone has stumbled on all of our <a href="https://www.elastic.co/blog">blogs</a> (shameless plug!), which means a lot of our users have gone on using types for what they were never really intended for. &#xA0;This raised&#xA0;a very good engineering question: should we continue this confusing &quot;type&quot; construct? &#xA0;Should we just (continue)&#xA0;recommending against it? &#xA0;Should we give the real use case of parent-child&#xA0;a proper first-class citizen in the Elasticsearch hierarchy? &#xA0;The good news is we have a much less confusing future that we&apos;re moving forward on.&#xA0; The bad news is eventually this will be a completely breaking change which means you will need to make&#xA0;application-side modifications as soon as you can, though we&apos;re really, really trying extra hard to make it as easy&#xA0;for you as possible. &#xA0;First, let&apos;s talk about why the change has come about.
<h2>Why</h2>Elastic (the company behind Elasticsearch) has seen&#xA0;thousands of support cases opened by our users and customers about the problems they&apos;ve run into. &#xA0;Because of this, it puts us in a unique situation to see what sort of common problems people run into with Elasticsearch. &#xA0;With respect to types, this has come down to 3 real key elements:
<ul>
	<li>User expectations: This is probably the most prominent issue and Elastic is unfortunately at least partially to blame: once a bad analogy is out there, it is nearly impossible to kill. &#xA0;At some points in the past, we have stated&#xA0;that Elasticsearch indices were like traditional RDBMS databases while Elasticsearch types were similar to tables. &#xA0;However this was really an oversimplification of reality and as a result many people have the mental model that types are equivalent&#xA0;to tables in the relational world. &#xA0;In reality, in Elasticsearch, the underlying data structures are the same for the entire index, not per type. &#xA0;Due to the misconceptions, one of the&#xA0;most common pitfalls we&apos;ve seen is that users expect fields to be independent across types. However, they must be of the same field type. So&#xA0;/my-index/type-a/my-field&#xA0;must use the same data type as&#xA0;/my-index/type-b/my-field.</li>
	<li>Sparsity:&#xA0;<a href="/guide/en/elasticsearch/reference/5.3/general-recommendations.html#sparsity">Sparsity should be avoided</a>! While Lucene 7 (which has recently <a href="https://github.com/elastic/elasticsearch/pull/24089">merged into master</a>)&#xA0;improves the handling of sparsity, it should still be&#xA0;avoided where possible. Types almost always&#xA0;increase the sparsity of your data because different types have different fields. &#xA0;So by removing types there is one less pitfall out there waiting to get you at some point.</li>
	<li>Scoring: Documents are scored by index and not by type so storing different entities in the same index can interfere with the relevance calculation for each entity type. Again, this is a bit counter-intuitive and many users miss this, which is&#xA0;another potential pitfall.</li>
</ul><h2>What</h2>As the one main use case derived from types vs indices is parent-child relationships, we have decided to supersede the &quot;type&quot;&#xA0;with a special field that stores the relationship between documents. &#xA0;We feel this represents a much better feel of the data. &#xA0;However, doing so is complicated, which gets us to the &quot;when&quot; element...
<h2>When</h2>We want the transition away from &quot;types&quot; to be as smooth as can be, so we&apos;re targeting long, multi-phase&#xA0;deprecation process to get us there. &#xA0;At the time of this writing, the current engineering targets for these are something like the following:
<ul>
	<li>In 5.x, add a new feature (index.mapping.single_type set to true/false), which will allow you to preview what the type removal will start to look like. &#xA0;This will be great for anybody that has a separate test environment and/or wants to start testing early.</li>
	<li>We plan to introduce a new breaking change currently targeted for 6.0 to make it so new indices will only allow a single type&#xA0;to be created to help you get better prepared for 7.x. &#xA0;Don&apos;t worry though -- the multi-type indices you created in 5.x will continue to work as before in 6.x. &#xA0;This phased roll-in is intended to&#xA0;give&#xA0;you some phase-in time as you upgrade without client-side breaks. &#xA0;In addition, we plan to:
	<ul>
		<li>Make _uid consistent with this change by removing the type from it.</li>
		<li>Add a new feature for &quot;<a href="https://github.com/elastic/elasticsearch/issues/20257">typeless parent/child fields</a>&quot; called &quot;join fields&quot;.</li>
		<li>Provide typeless URLs in preparation for the migration to 7.0.</li>
	</ul></li>
	<li>We currently plan to make the final breaking change related to this in 7.0, by removing types entirely from the Elasticsearch APIs</li>
</ul><h2>What does this mean for me?</h2>The answer to this question depends on what use case you&apos;re using the Elastic Stack for:
<ul>
	<li>Most logging &amp; security analytics users will find the transition completely seamless: Beats and Logstash don&apos;t generally use types and where they do, there&apos;s common alignment on the teams to try make the transition work without you thinking about it.</li>
	<li>If you&apos;re using Elasticsearch as a search and/or document datastore/database, you&apos;ll want to review your type usage, especially your parent/child usage.</li>
</ul>Of course, we recommend people get ahead of whatever they can by adopting things like this early. &#xA0;If you&apos;re looking to do so, you can use Kibana&apos;s Console / dev-tools to reindex any particular data moving data with a &quot;_type&quot; field into a &quot;type&quot; field.
<pre style="white-space: pre-wrap; box-sizing: border-box; font-family: sfmono-regular, consolas, &quot;liberation mono&quot;, menlo, courier, monospace; font-size: 11.9px; margin-bottom: 16px; font-stretch: normal; line-height: 1.45; word-wrap: normal; padding: 16px; background-color: rgb(246, 248, 250); color: rgb(36, 41, 46);">POST _reindex
{
 &quot;source&quot;: {
 &quot;index&quot;: &quot;old&quot;
 },
 &quot;dest&quot;: {
 &quot;index&quot;: &quot;new&quot;
 },
 &quot;script&quot;: {
 &quot;inline&quot;: &quot;&quot;&quot;
 ctx._id = ctx._type + &quot;-&quot; + ctx._id;
 ctx._source.type = ctx._type;
 ctx._type = &quot;doc&quot;;
&quot;&quot;&quot;
 }
}
</pre>This moves the special &quot;_type&quot; field over to &quot;type&quot; which you can then use in subsequent filtering, aggregations, etc.
<h2>I have more questions! &#xA0;Tell me more! </h2>Step 1: don&apos;t panic :) Step 2: please let us know on our <a href="https://discuss.elastic.co/">forums</a>
We&apos;re actively looking for any particular problems this may cause with your use case, so if you think you may have some, please talk to us!
</body></html>

Indices, types, and parent / child: current status and upcoming changes in Elasticsearch

Screen Shot 2016-11-08 at 10.16.01 AM.png

I have but one .condition(). Timeseries if-then-else with Timelion

<html><head></head><body><section id="post_content">
		<article>
			<blockquote>When you help people build applications with Elasticsearch every day, you run into a lot of unique requirements and scenarios. All of us here at Elasticsearch have an ever-growing &#x201C;bag of tricks&#x201D;, and it only seems fair to share those tricks with you. We hope you find them interesting, and perhaps useful to your application. </blockquote>
<hr style="width:100%">
For a recent project, I was working with short documents that represented movie reviews. The goal was a fairly common machine learning task: sentiment analysis and classification. If you&#x2019;ve tried to build a sentiment analyzer before, you know that it can be tricky. The sentiment of a document depends on not just the words used, but the context in which they are used.
For example, <code>&quot;He is happy&quot;</code> is clearly a &#x201C;positive&#x201D; sentiment. But if you add a single word it becomes entirely different: <code>&quot;He is not happy&quot;</code>
There are many ways to attack this problem, such as shingles and phrase matching. For some classifiers, however, providing additional explicit information about the &#x201C;context&#x201D; of a word can be very useful. Ideally, we&#x2019;d like to tag a word and say &#x201C;this word was in a negative context&#x201D;. 
With the help of <a href="https://twitter.com/clintongormley">@clintongormley</a>&#x2018;s super-human regular expression experience, I built a very simple &#x201C;negative connotation&#x201D; analyzer for Elasticsearch. Let&#x2019;s take a look first, then I&#x2019;ll explain the key points:
<pre class="prettyprint noescape">curl -XPUT localhost:9200/my_data/ -d &apos;
{
 &quot;settings&quot; : {
 &quot;analysis&quot; : {
 &quot;char_filter&quot; : {
 &quot;pre_negs&quot; : {
 &quot;type&quot; : &quot;pattern_replace&quot;,
 &quot;pattern&quot; : &quot;(\w+)\s+((?i:never|no|nothing|nowhere|noone|none|not|havent|hasnt|hadnt|cant|couldnt|shouldnt|wont|wouldnt|dont|doesnt|didnt|isnt|arent|aint))\b&quot;,
 &quot;replacement&quot; : &quot;~$1 $2&quot;
 },
 &quot;post_negs&quot; : {
 &quot;type&quot; : &quot;pattern_replace&quot;,
 &quot;pattern&quot; : &quot;\b((?i:never|no|nothing|nowhere|noone|none|not|havent|hasnt|hadnt|cant|couldnt|shouldnt|wont|wouldnt|dont|doesnt|didnt|isnt|arent|aint))\s+(\w+)&quot;,
 &quot;replacement&quot; : &quot;$1 ~$2&quot;
 }
 },
 &quot;analyzer&quot; : {
 &quot;negcon_tagger&quot; : {
 &quot;type&quot; : &quot;custom&quot;,
 &quot;tokenizer&quot; : &quot;whitespace&quot;,
 &quot;filter&quot; : [&quot;lowercase&quot;, &quot;kstem&quot;],
 &quot;char_filter&quot; : [&quot;pre_negs&quot;, &quot;post_negs&quot;]
 }
 }
 }
 }
}&apos;
</pre>
The heart of this trick is the pair of <code>char_filter</code>. Character filters are under-appreciated and extremely useful in certain circumstances. They provide a way to modify your input before any analysis takes place.
In this example, we are building a custom analyzer which applies two <code>char_filters</code>. The character filters are of the <code>pattern_replace</code> variety, which performs a standard regex-based replacement.
The regex patterns in this example finds a &#x201C;negative&#x201D; word (like &#x201C;not&#x201D;, &#x201C;haven&#x2019;t&#x201D;, etc) and tags the words immediately preceeding and following it with a tilde. An important note is that since <code>char_filters</code> are pre-analysis, you should make your patterns case-insensitive using <code>(?i:)</code>
Essentially, this converts <code>&quot;The boy was not happy&quot;</code> into <code>&quot;The boy ~was not ~happy&quot;</code>. By explicitly tagging &#x201C;happy&#x201D; with a tilde, it can only be matched with a search for &#x201C;~happy&#x201D;, which distinguishes it from the regular &#x201C;happy&#x201D; token. This was important to the classifier I was using (a naive bayes) and helped improve classification accuracy.
Just to verify, you can check the tokenization output using the Analyze API:
<pre class="prettyprint noescape">$curl -XGET &apos;localhost:9200/my_data/_analyze?analyzer=negcon_tagger&amp;pretty&apos; -d &quot;The boy was not happy&quot;

{
 &quot;tokens&quot; : [ {
 &quot;token&quot; : &quot;the&quot;,
 &quot;start_offset&quot; : 0,
 &quot;end_offset&quot; : 3,
 &quot;type&quot; : &quot;word&quot;,
 &quot;position&quot; : 1
 }, {
 &quot;token&quot; : &quot;boy&quot;,
 &quot;start_offset&quot; : 4,
 &quot;end_offset&quot; : 7,
 &quot;type&quot; : &quot;word&quot;,
 &quot;position&quot; : 2
 }, {
 &quot;token&quot; : &quot;~was&quot;,
 &quot;start_offset&quot; : 8,
 &quot;end_offset&quot; : 12,
 &quot;type&quot; : &quot;word&quot;,
 &quot;position&quot; : 3
 }, {
 &quot;token&quot; : &quot;not&quot;,
 &quot;start_offset&quot; : 13,
 &quot;end_offset&quot; : 15,
 &quot;type&quot; : &quot;word&quot;,
 &quot;position&quot; : 4
 }, {
 &quot;token&quot; : &quot;~happy&quot;,
 &quot;start_offset&quot; : 16,
 &quot;end_offset&quot; : 21,
 &quot;type&quot; : &quot;word&quot;,
 &quot;position&quot; : 5
 } ]
}
</pre>
Of course, this is just a very simple implementation. For example, we may not want to tag stopwords like &#x201C;was&#x201D; with negative connotation flags. Further, this method will hinder stopwording since it modifies the token in such a way that stop-lists are ineffective. In a future post, I&#x2019;ll show a native token-filter which produces the same output but is more robust.
I hope this post was interesting and potentially useful to your application. We&#x2019;ll endeavor to keep publishing little tricks like this, since Elasticsearch has many such little nooks and crannies which can build very powerful tools if you know where to look.
			<aside id="post_tags">
							</aside>
		</article>
			</section></body></html>

Quick Tips: Negative Connotation Filter

<html><head></head><body>Last week we <a href="https://www.elastic.co/blog/elastic-cloud-enterprise-1-1-0-released">released ECE 1.1.0</a> with new features and bug fixes. Since then, we&apos;ve discovered a critical bug that leads to Kibana being unavailable post-upgrade. This bug only affects users who upgraded from 1.0.x versions of ECE to 1.1.0.&#xA0;

	Note: NEW installs of ECE 1.1.0 are not affected
<h3>Issue</h3>If you have upgraded to ECE 1.1.0 and any of the following are true:
<ul>
	<li>You have tried to restart allocators</li>
	<li>You have tried to increase capacity by adding&#xA0;hosts.,</li>
</ul>If you have done those things, you will see Kibana authentication issues and unavailability. Monitoring will also be affected.
<h3>Immediate Workaround</h3>As an immediate workaround, we&apos;ve created a script that can fix this issue on your deployment, which is <a href="https://download.elastic.co/cloud/fix-ece-1.1.0-upgrade.sh">available for download</a>. You can run this script anywhere that has access to the coordinator node on port <code>12400</code>.
 The script requires two pieces of information:
<ol>
	<li><code>NODE_IP</code> which is the coordinator&apos;s IP address</li>
	<li><code>AUTH</code> which is the password for the admin console root user</li></ol>		Note: The software package <a href="https://stedolan.github.io/jq/download/">jq</a> is required on the host running this script for it to run properly. Here&apos;s what it looks like:<pre>[user@localhost ~]$ curl -O https://download.elastic.co/cloud/fix-ece-1.1.0-upgrade.sh
[user@localhost ~]$ chmod a+x ./fix-ece-1.1.0-upgrade.sh
[user@localhost ~]$ ./fix-ece-1.1.0-upgrade.sh
Please enter IP address of the coordinator:&lt;your ip here&gt;
Please enter root password to admin console:&lt;your password here&gt;</pre>The script will test connectivity, and if successful, make the changes. If not, it will let you know. After that has happened, it will prompt you to remove <code>frc-services-forwarders-services-forwarder</code> from every host.&#xA0;This is a manual step, which you can do with this command on every host running ECE: <pre>docker rm -f frc-services-forwarders-services-forwarder </pre>		Once this is removed, ECE will automatically restart the above service&#xA0;with the patched configuration.<h3>Official patched version</h3>		We are working on a 1.1.1 release that addresses this situation. We are targeting a 1.1.1 release early next week which will allow you to upgrade from 1.1.0 or any older versions.If you are planning to upgrade your existing deployment to 1.1.0, we strongly recommend to hold off until we release 1.1.1.We apologize for the inconvenience this may have caused and look forward to releasing 1.1.1 ASAP. Please contact us if you have any questions about this.</body></html>

blog-es-banner.jpg

blog-es-thumb.jpg

Elastic Cloud Enterprise 1.1.0 upgrade issues and workaround

<html><head></head><body><article>
			<h1>introduction</h1>
In the 1.2.1 release we fixed a routing bug that had been introduced in the 1.2.0 release. This was described in the&#xA0;<a href="http://www.elasticsearch.org/blog/elasticsearch-1-2-1-released/">1.2.1 blog post</a>. The contents of this post apply only to users that ran 1.2.0.
Before we can discuss the tool we developed to help fix damage from the bug, we have to understand the problem the bug created. When Elasticsearch stores a document it has to decide which shard to put the document in. It does this by computing a hash of the document&#x2019;s &#x201C;UID&#x201D;, which is by default the type#id tuple of the document. Elasticsearch then uses that hash modulo the number of shards to pick a shard. We have to make sure that this hash function works the same across different versions of Elasticsearch. If it doesn&#x2019;t, version X could expect to find a given document in shard 1 and version X+1 could expect the same document in shard 2. In this case, version X+1 could not directly get the document if it were indexed while running version X.
The bug in 1.2.0 broke this consistent, cross-version hashing. 1.2.1 restored the same hash operation as used in previous versions (except 1.2.0). Once you&#x2019;ve upgraded to 1.2.1 there are a few potential problems:
<ul>
<li>documents indexed in 1.2.0 are now in the &#x201C;wrong&#x201D; shard. You may not be able to retrieve (get) documents by ID when they were indexed in the wrong shard.</li>
<li>documents that were updated in 1.2.0 may now exist in two places: the correct shard (as in 1.2.1) and the wrong shard (as in 1.2.0).</li>
</ul>
The percentage of documents in the wrong shard varies by the number of shards in the index. With 2 shards, all documents are routed correctly. With 4 shards, 50% of the documents will be routed incorrectly. With 5 shards, 80% of documents will be routed incorrectly. 
Since you have probably continued to update documents, Elasticsearch cannot tell which of two documents with the same UID is the correct (newer) one. To be clear, it cannot tell if the document in the correct shard was put there recently and is the newer version, or if it was put there some time ago and is the older version. We&#x2019;ll discuss this more later in this post.
<h1>how the tool helps</h1>
We&#x2019;ve developed the&#xA0;<a href="http://download.elasticsearch.org/elasticsearch/elasticsearch-fix-routing/elasticsearch-fix-routing-1.0.zip">fix-routing tool</a>&#xA0;to give you as much help as we can in addressing the issue. We&#x2019;ll walk through a few specific use cases to illustrate the use of the tool.
<h2>re-indexing the data may be the simplest solution</h2>
We have provided the tool as a way to help you. After you read the options below you may conclude that re-indexing the data is simpler. That is a valid option, which will work fine. Just be sure that you have upgraded to 1.2.1 before re-indexing.
You will also either want to delete the index you are replacing or index into a new index. If you re-index into the existing index you will not fix the problem; the re-index will not change the location of the documents in the wrong shard.
There is no need to use this tool for any index that you choose to re-index.
<h2>tool prerequisites and preparation</h2>
The tool has a number of requirements to run correctly:
<ul>
<li>JDK 1.7 installed on the host running the tool</li>
<li>_source must have been stored</li>
<li>Your cluster is running version 1.2.1</li>
<li>You are using the default hashing algorithm (most users are)</li>
<li>You are not using the _type in the computation of the routing key (very few users do)</li>
</ul>
If you do not store the _source in Elasticsearch you can still run the count and delete operations described below. But, the tool will not be able to help with any other parts of conflict resolution.
You can run the tool on a running cluster. You do not have to block client access while running it. If you continue to allow updates to the cluster while the tool runs, you will have to repeat the procedure, with the second run of the procedure cleaning up duplicates created while the first instance ran.
Some operations of the tool will delete documents from an index. As a result we recommend taking a backup of each index before running the tool on that index. See&#xA0;<a href="http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/setup-upgrade.html">the upgrade instructions</a>&#xA0;for a backup procedure.
The tool uses a MVEL script as part of its execution. This script, called bad_routes.mvel, is included in the Zip file download. You need to place it in the&#xA0;<code>config/scripts</code>&#xA0;directory of all data nodes. There is no need to restart the nodes after putting this script in the directory. By default, it will take one minute for Elasticsearch to load the script and make it available via the API.
Note that the tool will work even if dynamic scripting is disabled.
<h1>using the tool</h1>
Be sure you have&#xA0;<a href="http://download.elasticsearch.org/elasticsearch/elasticsearch-fix-routing/elasticsearch-fix-routing-1.0.zip">downloaded the tool</a>. After you unzip the download, you will have a jar file and a MVEL script.
<h2>understanding the scope of the problem</h2>
The tool can give you a count on a per-index basis of the number of documents in the wrong shard:
<pre class="prettyprint noescape">&gt; java -jar elasticsearch-fix-routing-1.0.jar localhost 9300 index1 count</pre>
You can run this for each index that may have had writes while running 1.2.0 to understand how many documents were impacted.
You can also get an Elasticsearch query that will find the misplaced documents. This can be helpful if you want to write your own conflict resolution program.
<pre class="prettyprint noescape">&gt; java -jar elasticsearch-fix-routing-1.0.jar localhost 9300 index1 query</pre>
<h2>if you do not perform updates</h2>
If you do not update existing documents in an index, you can be sure there are no duplicates. Usually, people that are using Elasticsearch for the storage of logs and other machine generated data will want to follow these steps.
In this case the tool can be used to move misplaced documents from the wrong shard to the correct shard. Moving documents with the tool requires first copying them and then deleting them from the original (incorrect) shard.
<pre class="prettyprint noescape">&gt; java -jar elasticsearch-fix-routing-1.0.jar localhost 9300 index1 copy_if_missing</pre>
At this point you should be able to get &#x2013;&#xA0;not search for&#xA0;&#x2013; one of the misplaced documents. You should check a few to make sure the copy worked. Then delete the docs in the wrong shard.
<pre class="prettyprint noescape">&gt; java -jar elasticsearch-fix-routing-1.0.jar localhost 9300 index1 delete</pre>
Searches may return duplicate documents if you choose to not delete the documents in the wrong shard (as above).
<h2>if you have done updates and use external versioning</h2>
In general, Elasticsearch cannot tell which version of two duplicate documents is the correct one. However, if you use external versioning, it can. The newer document will have the higher external version number. If this is the case for your system, you can run:
<pre class="prettyprint noescape">&gt; java -jar elasticsearch-fix-routing-1.0.jar localhost 9300 index1 copy_version</pre>
This command will make sure that the higher version document is in the correct shard. Then, after you check that you now have two copies of the documents that were in the wrong shard, delete the one copy of the document that was in the wrong shard:
<pre class="prettyprint noescape">&gt; java -jar elasticsearch-fix-routing-1.0.jar localhost 9300 index1 delete</pre>
<h2>if you have done updates and do not use external versioning</h2>
This is the most difficult case. As mentioned earlier, the fix-routing program cannot tell which copy of the document should be preserved. In this case, you will have to use your domain knowledge of your documents to effect a fix. Start by getting the list of documents that have duplicates and then get the source of the documents that are in the wrong shard. The&#xA0;<code>get</code>&#xA0;command below will produce one JSON file per shard with the contents of misplaced documents for that shard.
<pre class="prettyprint noescape">&gt; java -jar elasticsearch-fix-routing-1.0.jar localhost 9300 index1 get_uids &gt; wrong_shard_uids.txt &gt; java -jar elasticsearch-fix-routing-1.0.jar localhost 9300 index1 get</pre>
Then iterate over each document returned and record the UID of those documents for which you want to keep the copy that was in the wrong shard. Call this list wrong_shard_restore. You can delete all the documents in the wrong shard since you have extracted them from the index. To delete all the documents that are in the wrong shard:
<pre class="prettyprint noescape">&gt; java -jar elasticsearch-fix-routing-1.0.jar localhost 9300 index1 delete</pre>
Then, for each document in wrong_shard_restore, re-index it using the document in one of the files output by the&#xA0;<code>get</code>&#xA0;command above. This will overwrite the version that was in the correct shard with the version that was in the wrong shard.
The documents that were in the correct shard and that you want to preserve will not be impacted by these operations. Your ability to correctly put only the documents to restore in wrong_shard_restore insures this.
<h1>other comments</h1>
You can run a count operation after you delete documents. This will help you verify that the delete did what was expected. Be careful here, as if you are still adding documents to the index the counts from the beginning of the procedure may be different from those at the end.
We&#x2019;re investigating our testing procedures to understand how this bug escaped our notice. We hope to publish a postmortem with the results of our analysis.
You should think about the backups you have made with 1.2.0 and 1.2.1 prior to performing this procedure. You may want to take a snapshot after performing this procedure, then delete the backups that had documents in the wrong shard.
			<aside id="post_tags">
							</aside>
		</article></body></html>

A tool to help with routing issues from Elasticsearch 1.2.0

<html><head></head><body>In Kibana 5.2 we introduced the internal concept of clusters which, as a Kibana contributor or plugin author, will affect how you communicate with Elasticsearch. This change is transparent to the end-user and establishes the foundation for more features to come, beginning with Tribe support.

<h2>Prior to 5.2</h2>
Before 5.2, the Elasticsearch Kibana plugin provided two ways to communicate with your Elasticsearch cluster: you could either use the <a href="https://www.elastic.co/guide/en/elasticsearch/client/javascript-api/current/index.html">elasticsearch-js</a> client directly or use callWithRequest.

<h3>elasticsearch-js client</h3>
The elasticsearch-js client provides&#xA0;an&#xA0;interface which is&#xA0;well-documented in&#xA0;the <a href="https://www.elastic.co/guide/en/elasticsearch/client/javascript-api/current/quick-start.html">Quick Start</a> section. It takes care of configuration and creation automatically, so using it is&#xA0;just a matter of calling a method which correlates to the <a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/index.html">Elasticsearch API</a>.

Let&#x2019;s see how that looks using <a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-health.html">Cluster Health</a> as an example:

<pre class="prettyprint">const { client } = server.plugins.elasticsearch;
client.cluster.health().then(response =&gt; {
 console.log(`cluster status is: #{response.status}`);
})
</pre>
<h3>callWithRequest</h3>
The&#xA0;callWithRequest&#xA0;method&#xA0;ensures&#xA0;that the request is made in the context of the currently authenticated user. It uses the same underlying elasticsearch-js client, but requires a bit more&#xA0;configuration. You&apos;ll need to provide the request object, which can be obtained from the route handler, and you&apos;ll need to pass&#xA0;the API path as an argument instead of calling it directly.

Let&#x2019;s see how the previous example looks now:

<pre class="prettyprint">const { callWithRequest } = server.plugins.elasticsearch;
callWithRequest(request, &apos;cluster.health&apos;).then(response =&gt; {
 console.log(`cluster status is: #{response.status}`);
})
</pre>
<h2>5.2 and beyond</h2>
In 5.2 we now have multiple clusters, and we need to specify one before making requests to Elasticsearch. Currently, there are two primary clusters; admin and data. The admin cluster contains the <code>.kibana</code> index and anything else used to persist state in Kibana. The data cluster contains all of the documents that you have indexed and would like to visualize.

To access one of the clusters, the Elasticsearch plugin provides getCluster. Let&#x2019;s see how callWithRequest looks now:

<pre class="prettyprint">const { callWithRequest } = server.plugins.elasticsearch.getCluster(&apos;data&apos;);
callWithRequest(request, &apos;cluster.health&apos;).then(response =&gt; {
 console.log(`cluster status is: #{response.status}`);
})
</pre>
<h3>callWithInternalUser</h3>
In lieu of exposing the client directly, we now offer a method callWithInternalUser, and other than omitting the request argument it has the same interface as callWithRequest. It&#x2019;s worth noting that in most situations you will want to use callWithRequest, because it&#xA0;ensures that the user has privileges to perform the action. callWithInternalUser is more appropriate&#xA0;for startup tasks and migrations, where&#xA0;the credentials specified within the <code>kibana.yml</code> are required. Should you still require&#xA0;the underlying client, you can obtain it by calling getClient on the cluster.

Let&#x2019;s see how this looks:

<pre class="prettyprint">const { callWithInternalUser } = server.plugins.elasticsearch.getCluster(&apos;data&apos;);
callWithInternalUser(&apos;cluster.health&apos;).then(response =&gt; {
 console.log(`cluster status is: #{response.status}`);
})
</pre>
<h2>Writing client-side Javascript?</h2>
If you&apos;re injecting the es service into your controllers or services and expect the data cluster, there are no changes necessary. If you&apos;re expecting the admin cluster, you&apos;ll want to use the&#xA0;esAdmin service.

<h2>What&#x2019;s Next?</h2>
If you&#x2019;re a plugin developer, you will likely need to make some changes for 5.2 compatibility. These changes should be transparent to the user and pave the way for improvements and features to come. If you have questions, reach out to us in <a href="https://webchat.freenode.net/#kibana">#kibana</a> or our&#xA0;<a href="https://discuss.elastic.co/c/kibana">forums</a>. If you run into a bug, please file a <a href="https://github.com/elastic/kibana/issues">GitHub Issue</a>.
</body></html>

train-track-sunset-full-bleed.jpg

train-track-sunset.jpg

Kibana Plugin Developers meet Elasticsearch Clusters

<html><head></head><body>
	Today, we are happy to announce the bug fix release of&#xA0;
	Elasticsearch 1.4.2, based on Lucene 4.10.2, and the bug fix release of Elasticsearch 1.3.7. You can download them and read the full changes list here: 
<div class="itemizedlist">
	<ul class="itemizedlist" type="disc">
		<li class="listitem">
		Latest stable release: <a class="ulink" href="http://www.elasticsearch.org/downloads/1-4-2" target="_top">Elasticsearch 1.4.2</a>.
		</li>
		<li class="listitem">
		Bug fixes for 1.3.x: <a class="ulink" href="http://www.elasticsearch.org/downloads/1-3-7" target="_top">Elasticsearch 1.3.7</a>
		</li>
	</ul>
</div>
	For blog posts about past releases see:
<div class="itemizedlist">
	<ul class="itemizedlist" type="disc">
		<li class="listitem">
		1.4: <a class="ulink" href="http://www.elasticsearch.org/blog/elasticsearch-1-4-1-released/" target="_top">1.4.1</a>, <a class="ulink" href="http://www.elasticsearch.org/blog/elasticsearch-1-4-0-released/" target="_top">1.4.0</a>, <a class="ulink" href="http://www.elasticsearch.org/blog/elasticsearch-1-4-0-beta-released/" target="_top">1.4.0.Beta1</a>
		</li>
		<li class="listitem">
		1.3: <a class="ulink" href="http://www.elasticsearch.org/blog/elasticsearch-1-4-1-released/" target="_top">1.3.6</a>, <a class="ulink" href="http://www.elasticsearch.org/blog/elasticsearch-1-4-0-released/" target="_top">1.3.5</a>, <a class="ulink" href="http://www.elasticsearch.org/blog/elasticsearch-1-3-4-released/" target="_top">1.3.4</a>, <a class="ulink" href="http://www.elasticsearch.org/blog/elasticsearch-1-3-3-released/" target="_top">1.3.3</a>, <a class="ulink" href="http://www.elasticsearch.org/blog/elasticsearch-1-3-2-released/" target="_top">1.3.2</a>, <a class="ulink" href="http://www.elasticsearch.org/blog/elasticsearch-1-3-1-released/" target="_top">1.3.1</a>, <a class="ulink" href="http://www.elasticsearch.org/blog/elasticsearch-1-3-0-released/" target="_top">1.3.0</a>.
		</li>
	</ul>
</div>
	You can read about all of the changes that have been made in the 
	<a class="ulink" href="http://www.elasticsearch.org/downloads/1-4-2" target="_top">1.4.2 release notes</a> and the <a class="ulink" href="http://www.elasticsearch.org/downloads/1-3-7" target="_top">1.3.7 release notes</a>, but we highlight some of the more important changes below:
<div class="section">
	<div class="titlepage">
		<div>
			<div>
				<h3 class="title"><a id="_bug_fixes" href="#_bug_fixes"></a>Bug fixes</h3>
			</div>
		</div>
	</div>
	
		We use randomized testing extensively in Elasticsearch, and it has helped us to find and fix these issues:
	
	<div class="itemizedlist">
		<ul class="itemizedlist" type="disc">
			<li class="listitem">
			If a node holding a primary shard was restarted while a replica shard was still recovering from the primary, it could delete the transaction log on the primary, resulting in data loss. (<a class="ulink" href="https://github.com/elasticsearch/elasticsearch/pull/8917" target="_top">#8917</a>)
			</li>
			<li class="listitem">
			The ScriptService can deadlock entire nodes if the script index is recovering. (<a class="ulink" href="https://github.com/elasticsearch/elasticsearch/pull/8901" target="_top">#8901</a>)
			</li>
			<li class="listitem">
			Forcefully releasing the Index Writer lock can cause shard corruption. (<a class="ulink" href="https://github.com/elasticsearch/elasticsearch/pull/8892" target="_top">#8892</a>)
			</li>
		</ul>
	</div>
</div><div class="section">
	<div class="titlepage">
		<div>
			<div>
				<h3 class="title"><a id="_performance_improvements" href="#_performance_improvements"></a>Performance improvements</h3>
			</div>
		</div>
	</div>
	
		Our bigger users with complex setups sometimes see performance bottlenecks which are not obvious at smaller scale. Their reports have helped us to make these improvements:
	
	<div class="itemizedlist">
		<ul class="itemizedlist" type="disc">
			<li class="listitem">
			The disk allocation decider, which makes decisions about shard allocation based on free disk space, is now significantly faster and improves recovery speed after a cluster restart. (<a class="ulink" href="https://github.com/elasticsearch/elasticsearch/pull/8803" target="_top">#8803</a>)
			</li>
			<li class="listitem">
			Snapshots made to a shared file system are much faster than before. (<a class="ulink" href="https://github.com/elasticsearch/elasticsearch/pull/8749" target="_top">#8749</a>)
			</li>
			<li class="listitem">
			Unnecessary cluster state updates have been reduced, which reduces network traffic and speeds up recovery. (<a class="ulink" href="https://github.com/elasticsearch/elasticsearch/pull/8933" target="_top">#8933</a>, <a class="ulink" href="https://github.com/elasticsearch/elasticsearch/pull/8413" target="_top">#8413</a>)
			</li>
			<li class="listitem">
			The index stats API shouldn&apos;t be blocked by shard recovery. (<a class="ulink" href="https://github.com/elasticsearch/elasticsearch/pull/8910" target="_top">#8910</a>)
			</li>
		</ul>
	</div>
</div><div class="section">
	<div class="titlepage">
		<div>
			<div>
				<h3 class="title"><a id="_try_it_out" href="#_try_it_out"></a>Try it out</h3>
			</div>
		</div>
	</div>
	
		Please 
		<a class="ulink" href="http://www.elasticsearch.org/downloads/1-4-2" target="_top">download Elasticsearch 1.4.2</a>, try it out, and let us know what you think on Twitter (<a class="ulink" href="https://twitter.com/elasticsearch" target="_top">@elasticsearch</a>). You can report any problems on the <a class="ulink" href="https://github.com/elasticsearch/elasticsearch/issues" target="_top">GitHub issues page</a>.
	
</div></body></html>

Elasticsearch 1.4.2 and 1.3.7 Released - December 16, 2014

<html><head></head><body>We&apos;re pleased to announce Watcher 1.0.0-Beta2!This release of Watcher includes bug fixes and enhancements to features introduced in Beta1.But we didn&#x2019;t stop there. Since the release of Beta1, we&#x2019;ve received wonderful feedback from all of you trying out Watcher. We&#x2019;re delighted with the activity in the <a href="https://discuss.elastic.co/c/watcher">forum</a>&#xA0;and I can honestly say that your questions have helped us improve Watcher in every aspect - functionality, usability and documentation.And yes, we are talking the talk and walking the walk. We took some of your wonderful suggestions and included them in this release. For more details, please read the <a href="https://www.elastic.co/guide/en/watcher/current/release-notes.html">release notes</a>.Installing Beta2 is as easy as it was installing Beta1. You can use the same beta key you received in the Beta1 registration email, but this time appending &#xA0;`-Beta2` to your key:bin/plugin -i elasticsearch/watcher/&lt;watcher-beta-key&gt;-Beta2Haven&#x2019;t registered yet? <a href="https://www.elastic.co/downloads/watcher">Do it now </a>and try Watcher out!</body></html>

Watcher Beta 2 - Fixes, Features, Feedback

<html><head></head><body>It&#x2019;s time to forget everything you knew about queries and filters: Elasticsearch 2.0 will make much better decisions by itself instead of relying on users to formulate an optimized query. This change will be almost invisible at the API level but let&#x2019;s dive into the internal changes that made it possible. Most changes that are mentioned in this blog post have been done in Lucene 5.0, 5.1 and 5.2 and will be integrated in Elasticsearch 2.0.
<h2>The query/filter merge</h2>Previous versions of Elasticsearch used to handle queries and filters as different citizens: you had queries that could match and score, and filters that could only match and were cacheable. In Elasticsearch 2.0, queries and filters are the same internal object, and these objects can be configured to score documents, or to skip scoring. Then the query DSL makes sure to propagate the information correctly: for instance the <code>must</code> clause of a
	<code>bool</code> query would need to produce scores if the
	<code>bool</code> query needs to produce scores too, while a
	<code>must_not</code> clause would never need to produce scores since it may only be used for filtering documents out.
In order to make the query DSL more consistent with this change, we have deprecated the
	<code>filtered</code> query in favour of a new
	<code>filter</code> clause on the
	<code>bool</code> query:
	<code>filter</code> clauses are like
	<code>must</code> clauses except that they do not contribute to the score. This means that the following query:
<pre class="programlisting prettyprint lang-js prettyprinted">{
 &#x201C;filtered&#x201D; : {
 &#x201C;query&#x201D;: { query definition },
 &#x201C;filter&#x201D;: { filter definition }
 }
}
</pre>should now be replaced with
<pre class="programlisting prettyprint lang-js prettyprinted">{
 &#x201C;bool&#x201D; : {
 &#x201C;must&#x201D;: { query definition },
 &#x201C;filter&#x201D;: { filter definition }
 }
}
</pre>Note that the query DSL is still backward compatible in spite of this change: if you try to run a <code>filtered</code> query, it will parse as a
	<code>bool</code> query internally. However, we would encourage you to migrate to the new syntax as the 
	<code>filtered</code> query will be removed in a future release.
While this might look like a minor change, this is actually very useful for us. For instance, we used to have 3 queries/filters that could perform conjunctions: the
	<code>bool</code> query, the
	<code>bool</code> filter and the
	<code>filtered</code> query, and it happens that they all computed conjunctions in a slightly different way. The fact that we only have the
	<code>bool</code> query now allows us to perform optimizations in a more robust way, and in particular to leverage two-phase execution efficiently.
<h2>Two-phase execution</h2>The previous filter API could be consumed in two ways: either using iterators over matching documents, or using an optional random-access API that allowed to check if a particular document matched the filter or not. Everything is good so far, except that the best way to consume a filter depended on which kind of filter you had: for instance the
	<code>script</code> filter was more efficient when using the random-access API while the 
	<code>bool</code> filter was more efficient using the iterator API. This was quite a nightmare to optimize and was the root cause why the
	<code>bool</code> filter on the one hand and the
	<code>and</code> and
	<code>or</code> filters on the other hand performed differently.
To fix this issue, Lucene introduced a new two-phase iteration API, which consists of an approximation phase that can quickly iterate over a superset of the matching documents, and a verification phase that can be used to check if a document in this superset actually matches the query. A good example of use of this API is the phrase query: if you search for &#x201C;quick fox&#x201D;, a good approximation would be to search for documents that contain both &#x201C;quick&#x201D; and &#x201C;fox&#x201D;, and then as a verification step we could read positions to see if we can find these terms at consecutive positions.
Why is it useful? Imagine that you are searching for &#x201C;quick fox&#x201D; and applying a filter as well. The fact that we can dissociate the approximation from the verification phase allows us to first intersect the approximation with the filter, so that we will verify positions on a smaller set of documents. This two-phase iteration pattern also applies to other queries: for instance geo-distance queries can use a bounding box as an approximation and a distance computation as a verification, and filters that only make sense in a random-access fashion such as the
	<code>script</code> filter can return all documents in the index as an approximation and run the script as a verification.
A nice side-effect of this change is that the <code>bool</code>,
	<code>and</code> and
	<code>or</code> filters now behave the same, there is no need to pick one depending on the wrapped filters. By the way, we have deprecated the
	<code>and</code> and
	<code>or</code> filters and now encourage you to use
	<code>bool</code> instead.
<h2>Smarter filter caching</h2>One issue that we often saw in previous Elasticsearch releases is that some queries were slow due to over-caching of filters. Take the term filter as an example: it can directly return iterators that are backed by disk-based postings lists, and these postings lists include skip lists that allow for efficient skipping, which is typically useful for conjunctions. However, if you want to cache the results, you need to consume the postings list entirely and load its content into memory, which means you can&#x2019;t skip over unnecessary documents quickly anymore. In summary, filters should never be cached if they are not going to be reused, otherwise caching them is wasteful.
Elasticsearch changed filter caching to be entirely automatic. The way it works is that Elasticsearch keeps track of the 256 most recently used filters, and only caches those that appear 5 times or more in this history. The assumption is that over caching is more dangerous than under caching, given that uncached filters are already fast. So we would rather make sure they are reused before caching them. Also Elasticsearch does not cache anymore on segments which have less than 10000 documents or 3% of the documents of the index, since such segments are no bottleneck for query execution, on the contrary to the larger segments, and are also likely to be picked for a merge quickly since the merge policy favors small merges.
As a consequence of this change, 
	the
	<code>_cache</code> and
	<code>_cache_key</code> options are now deprecated and will be ignored if provided.
<h2>More efficient filter cache</h2>In Elasticsearch 1.x, filters are cached using actual bit sets. While bit sets have a nice worst-case of 1 bit per existing document, this is quite wasteful when you have only a couple of bits that are sets out of several millions of existing bits. To improve this situation, Elasticsearch switched to a sparse bit set implementation for its filter cache in 2.0 called <a href="https://www.elastic.co/blog/frame-of-reference-and-roaring-bitmaps">roaring bitmaps</a>.
Also large terms filters are now prefix-compressed in order to take less memory in the filter cache.
These two changes mean that you will now be able to cache more filters with the same amount of memory that you were giving to previous releases of Elasticsearch.
<h2>Better execution of multi-term queries</h2>Multi-term queries are a family of queries that match several terms depending on some condition. For instance
	<code>prefix</code> queries match terms that start with a given sequence of bytes,
	<code>regexp</code> queries match terms that match a given regular expression, etc. All these queries are executed the same way: we create a bit set, then visit all matching terms and read their postings lists into that bit set. This approach scales well when you have many terms, unlike the
	<code>bool</code> query. One drawback however, is that it cannot leverage skipping and needs to visit all matching documents for all terms. In order to improve this, we changed multi-term queries to rewrite to a
	<code>bool</code> query when they only&#xA0;match a&#xA0;few terms in order to be more efficiently intersected with other queries.
<h2>Improved spans</h2>Finally, span queries got some interesting improvements, and in particular now fully leverage the new two-phase iteration infrastructure. Additionally, two new span queries are exposed: <a href="https://www.elastic.co/guide/en/elasticsearch/reference/master/query-dsl-span-within-query.html">span-within</a> and <a href="https://www.elastic.co/guide/en/elasticsearch/reference/master/query-dsl-span-containing-query.html">span-containing</a>.
All of these changes work together to allow Elasticsearch to make better decisions about how to execute your queries optimally. You no longer need to think about whether to use bool vs and, the order of your query clauses, or whether to cache or not. Elasticsearch now does it all for you.</body></html>

Better query execution coming to Elasticsearch 2.0

<html><head></head><body>Today we&apos;re pleased to bring you the story of the creation of SeeMeSpeak, a Ruby application that allows users to record gestures for those learning sign language. <a href="https://twitter.com/Argorak">Florian Gilcher</a>, one of the organizers of the <a href="http://www.meetup.com/ElasticSearch-UG-Berlin/">Berlin Elasticsearch User Group</a> participated in a hackathon last weekend with three friends, resulting in this brand new open source project using Elasticsearch on the back end.

Last weekend we participated in <a href="http://railsrumble.com/">RailsRumble 2013</a>, a 48 hour hacking contest to build something fancy on top of Ruby (nowadays, Rails is not required). Beyond fancy, we wanted to build something that solves a real problem that we could continue developing after the hackathon concluded. It turns out that we found a common scratch to itch: our entire group was interested in <a href="http://en.wikipedia.org/wiki/Sign_language">sign language</a>.

<img src="https://api.contentstack.io/v2/assets/575e4c9612798d8e381fd7f9/download?uid=blt6c9453d718003122?uid=blt6c9453d718003122" data-sys-asset-uid="bltff92b5092da048cd" style="max-width:100%;">

<h2>The Project</h2>
Sadly, there are almost no good learning resources for sign language on the internet. If material is available, licensing is a hassle or both the licensing and the material is poorly documented. Documenting sign language yourself is also hard, because producing and collecting videos is difficult. You need third-party recording tools, video conversion and manual categorization. That&apos;s a sad state in a world where every notebook has a usable camera built in!

Our idea was to leverage modern browser technologies to provide an easy recording function and a quick interface to categorize the recorded words. The result is <a href="http://rumble.seemespeak.org">SeeMeSpeak</a>.

<h2>We Have a Hundred Problems...</h2>
Handling UserMedia (video and audio) in the browser is still in its infancy. Especially with recording, which mostly works through elaborate hacks like <a href="https://github.com/antimatter15/whammy">whammy.js</a> that need to be set up very carefully. Given only about 24 hours of full work - our team valued sleep and rest - we expected recording and proper video conversion to take up most of our time.

<h2>... The Data Store Shouldn&apos;t Be One of Them</h2>
Thus, we were looking for a data store that:

<ul>
	<li>is easy to set up for all developers</li>
	<li>is easy to handle on the server</li>
	<li>solves our immediate problems</li>
	<li>helps us to iterate quickly</li>
	<li>allows for future refinements in our problem space</li>
	<li>is resource-friendly</li>
</ul>
The last point stems from the contest rules: all applications have to run on a Linode instance with 1GB of memory. Considering that we have to convert videos on the same machine, we cannot be wasteful.

Our problem is language and searching, so Apache Lucene is an obvious pick. Three of our team members had already worked with Elasticsearch and the decision was made quickly; we used Elasticsearch and couldn&apos;t be happier with it.

<h2>Elasticsearch for Tiny Data</h2>
Elasticsearch supported the goals of our fast prototype very well:

Setting up a reasonably configured elasticsearch instance as a developer is as easy as:

<ul>
	<li>download and unzip</li>
	<li><code>cd elasticsearch; bin/elasticsearch -f</code></li>
</ul>
With packages available for many platforms, Elasticsearch is quickly configured on the server as well. Also, contrary to popular belief, Elasticsearch can be run in a small memory space by setting <code>ES_HEAP_SIZE</code> to a small value. Given that we expected less than 5000 documents for our initial prototype, we were able to easily fit Elasticsearch in less around 256MB of memory.

Setting up the new <a href="https://github.com/elasticsearch/elasticsearch-ruby">offical Ruby client</a> was also a breeze, it integrates well with the connection-less data models provided by <a href="https://github.com/solnic/virtus">Virtus</a>.

On the content side, we had 3 main problems: tagging, flagging and transcription. The transcriptions to a video should be searchable. At the same time, we want to flag videos by a fixed set of criteria (if a word is vulgar or an insult, we want to know). Tagging is free-form and allows users to categorize words. All this is possible using Elasticsearch&apos;s dynamic mapping. Our main datatype ended up looking roughly like this:

<pre class="prettyprint lang-js">{
 &quot;transcription&quot;: &quot;hello from seemespeak&quot;,
 &quot;tags&quot;: [&quot;funny&quot;],
 &quot;flags&quot;: [&quot;casual&quot;],
 &quot;reviewed&quot;: false,
 &quot;language&quot;: &quot;ASL&quot;
}
</pre>
This is a very workable application-side data format that Elasticsearch&apos;s dynamic mapping indexes very well. More specifically, it gives you all you need to search it as expected:

<ul>
	<li>The <code>match</code> query makes implementing a search box over the transcription field very easy.</li>
	<li>tags, flags and review status can be easily handled using the <code>term</code> and <code>terms</code> filters. A <code>bool</code> filter helps you combining those.</li>
	<li>For the random video on the frontpage and the full list, we used the newly introduced <a href="http://www.elasticsearch.org/blog/0-90-4-released/">function_score query</a> with a randomly changing seed, giving every user different, but stable results.</li>
</ul>
We found that the Elasticsearch&apos;s query language makes is very easy to build queries iteratively. We started with a plain <code>match_all</code> query to give frontend developers enough content to start building and added filters and randomizations over the first day. By the end of the first day, we were done with all search scenarios.

The current SeeMeSpeak is a prototype. We didn&apos;t venture into fine-tuning by implementing custom analyzers, e.g. for proper stemming and synonym searches. We were confident that we could do this on the second day, but wanted to invest our time in other things, like translating our German corpus to English for our English-speaking visitors and translating the whole site into two languages.

All in all, Elasticsearch allowed us not to worry too much about data storage and push out a minimum viable version of our product with usable features over the course of a weekend.

<h2>Wrapping Up</h2>
Elasticsearch is not only a great database for &quot;big data,&quot; it is also a great fit for rapid application development in early stages. We were able to validate all our scenarios and use-cases very quickly. As always, investing time and effort into your data store will improve results in the long run, but we were impressed by the number of features we got out of Elasticsearch so quickly without starting with fine tuning.

<h2>How You Can Help</h2>
SeeMeSpeak is open source and solves a real issue: building an openly licenced and well tagged corpus for sign languages. You can contribute <a href="http://github.com/seemespeak">on GitHub</a>. Also, you can help us by voting or leaving feedback on the <a href="http://railsrumble.com/entries/39-see-me-speak">RailsRumble</a> site. Feel free to try out our recording function. Just try to say <a href="http://rumble.seemespeak.org/videos?transcription=hallo">hello</a>.

Thank you!

<a href="http://twitter.com/argorak">Florian Gilcher</a>, Elasticsearch Berlin User Group Co-Organizer, <a href="http://twitter.com/kaeff">Klaus Fl</a>, <a href="http://twitter.com/jannilpferd">Jan Nietfeld</a>, <a href="http://twitter.com/bitboxer">Bodo Tasche</a>

Many thanks to Florian, Klaus, Jan and Bodo for sharing their experiences with us. We are always excited to share stories from the community about how they are using Elasticsearch, so please <a href="http://twitter.com/lhawthorn">reach out to me</a> any time you&apos;d like to share you story.
</body></html>

Tiny Data: Rapid Development with Elasticsearch

<html><head></head><body>I am happy to announce that the second milestone of <a href="http://www.elasticsearch.org/hadoop">Elasticsearch for Apache Hadoop</a> 1.3 (also known as es-hadoop) has been released. M2 brings several new major features to the table, including:

<h2>Support for Elasticsearch 1.0 (RC1+)</h2>
es-hadoop supports Elasticsearch 1.0 RC1 or higher while preserving compatibility with the Elasticsearch 0.90.x line. The code base automatically detects the target Elasticsearch version and uses the appropriate <a href="http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/breaking-changes.html">features</a>.

<h2>id awareness</h2>
es-hadoop 1.3 M1 enabled new indexes to be created in Elasticsearch directly from Hadoop jobs. M2 takes this feature several steps forward, enabling both index creation and updating. Furthermore, one can specify all the meta-data options if needed: document <tt>parent</tt>, <tt>time-to-live</tt>, <tt>routing</tt> and <tt>timestamp</tt>.

<h2>Field aliasing</h2>
Higher level abstractions on Map/Reduce, like Cascading, Apache Hive and Apache Pig provide mapping and data types on top of raw data for easier manipulation. With es-hadoop, one can also define field aliasing, decoupling the Hadoop libraries structural declaration from the underlying document fields for cleaner syntax and improved readability.

<h2>Field extraction</h2>
Performance is a top priority for es-hadoop. As such, M2 only loads the data that it must from Elasticsearch; rather than loading the entire <tt>source</tt> document, es-hadoop retrieves just the required fields, also known as projection. This implementation of projection results in less network, CPU and memory usage.

<h2>JSON support</h2>
Besides supporting the Map/Reduce, Cascading, Hive and Pig data types, M2 also allows JSON data to be indexed. Don&#x2019;t forget that es-hadoop can now extract the document metadata (such as <tt>id</tt>) if instructed to do so.

<h2>Parallel writes and ingest mitigation</h2>
Users with high ingestion volumes will be happy to hear that M2 provides significant updates on the write front. Now, all writes to Elasticsearch are parallelized on the target shards; this implementation prevents network bottlenecks as the load is spread across multiple nodes. In case of excessive loads on Elasticsearch, regardless of the reason, 1.3 M2 automatically retrieves only the rejected payload before ingesting further data. This temporary throttling prevents data spillage.

<h2>Improved I/O</h2>
In a similar vein, es-hadoop 1.3 M2 provides automatic cluster discovery and, in case of network errors, automatic fall back to the rest of the available nodes.

<h2>Snapshot/Restore for HDFS</h2>
Last but not least, 1.3 M2 introduces <a href="http://www.elasticsearch.org/blog/introducing-snapshot-restore/">snapshot and restore</a> support for <a href="https://github.com/elasticsearch/elasticsearch-hadoop/tree/master/repository-hdfs">HDFS</a> through a separate, umbrella project (<tt>elasticsearch-repository-hdfs</tt>). Once installed in the Elasticsearch cluster, this standalone plugin enables any <tt>DFS</tt> storage (from the omnipresent HDFS to pluggable implementations such as <a href="http://aws.amazon.com/s3/">Amazon S3</a> or <a href="https://cloud.google.com/products/cloud-storage/">Google Cloud Storage</a>) to be used for backing up and restoring data within running Elasticsearch clusters.

Of course, this release also includes the usual bug fixes, compatibility improvements and refinements. As always, we welcome feedback so <a href="http://www.elasticsearch.org/hadoop">take 1.3 M2 for a spin</a> and <a href="http://www.elasticsearch.org/guide/en/elasticsearch/hadoop/current/resources.html">let us know</a> what you think.

P.S. If you are migrating from M1, you might want to read up on the <a href="http://www.elasticsearch.org/guide/en/elasticsearch/hadoop/current/changes-13m2.html">configuration changes in M2</a>.
</body></html>

Elasticsearch for Apache Hadoop 1.3 M2 released

Enriching Your Searches With Open Geo Data

<html><head></head><body>
	The People behind the Projects is a new series of first-person blog posts from the individuals who contribute to the fantastic, elastic world of Elasticsearch. We&apos;re a close-knit family at Elasticsearch. The goal of this series is to erode the boundaries between the so-called personal and the professional. We want to share our stories with you. So in this series, Elasticsearchers talk about where they&apos;re from, how they first started engaging with Elasticsearch, and what motivates them.


	<a href="http://www.twitter.com/mcmesser">Marty Messer</a> is the Vice President of Customer Care at Elasticsearch. 
	&#x2014;
	 
	<img src="https://api.contentstack.io/v2/assets/575e4c7343e9adc53871368c/download?uid=blt84d610e3fb51e5d1?uid=blt84d610e3fb51e5d1" data-sys-asset-uid="blt20c0fd2ddb02ea4d" width="315" height="300" style="float:right; margin:4px 5px;">When I join a company, it&apos;s typically at the stage of growth where the vision is so focused, the work pressure is so intense, and the product release schedule is so tight that the welcome note from the dev team has always been one of relief: &#x201C;Thank god, you&apos;re here!&quot; Developers typically don&apos;t like dealing with support or the operational issues of integration. They like product development.


	However, at Elasticsearch, the situation was different. The developers loved customer interaction, and enjoyed dealing with support, and I had to gently hug it away from them.


	Enterprise software is a complicated and multidisciplinary field. In the Customer Care department at Elasticsearch, we have teams of support-focused engineers working alongside the developers of the product, and, boy, will our developers do anything for a customer. The precedents for this value system have been role-modeled and emulated since the founding of this company. Being involved in Customer Care also makes developers more aware of the operational aspects of building the platform. This customer-centricity brings us closer, and this fact is apparent in our customer surveys. Our customer survey response rates are 30 percent, three times the industry standards. People want to tell us how they are feeling. And this is a testament not only to the effectiveness of our Customer Care process, but also to the excellence of our recruiting. It&apos;s about the people we are bringing in who make all the difference.


	<img src="https://api.contentstack.io/v2/assets/575e4c739e7a83165490bcde/download?uid=bltc5e1bb3044947789?uid=bltc5e1bb3044947789" data-sys-asset-uid="blt4f80f556b98b5487" width="392" height="250" style="float:left; margin:4px 5px;">Earlier this year, we started working with an EdTech company with a 100 million users. They use Elasticsearch to let their users search across more than 2 billion terms to find the most relevant learning material. They were having some operational performance issues, and so we started having a conversation with them. As we started peeling off the layers, we realized they were using an operating system that no other customer was using, and this selection was causing the operational performance issues (and this is with Elasticsearch, which has been downloaded over 10 million times!). When you start calling people&apos;s choice of operating systems into question, they can get prickly. Selecting an operating system is like a religious choice. It&apos;s like saying, &#x201C;Your baby&apos;s ugly.&quot; But we knew we needed to have a conversation, and so we laid out a list of scientific reasons for why the operating system was a reason for concern. But instead of a formal presentation that could have come across as condescending, we ended up having a simple conversation. And the tech lead at the company told me, &#x201C;You know, we have never really liked this system, and we have sort of been looking for a reason to migrate. This is going to be it.&quot; So then we were able to offer a more optimal set of operating systems as options, and have a more consultative conversation about the next steps regarding the choice of architecture and analytics.


	At the end of that conversation, I felt good that I had succeeded in setting his expectations higher than what he had himself thought possible. We talked about building standalone clusters. We then went on to build a parallel infrastructure to AB test and to demonstrate the effectiveness of our recommended operating system. The conversation went beyond what our client team expected of us. For me to confidently set expectations higher than what the customer anticipated is so rare in the world of support. It&apos;s way more than just dealing with a bug here.


	The act of seeking support poses a burden to one&apos;s self-image. Complicated technical issues aren&apos;t always fun. People come to us because something isn&apos;t working. But our team practices a form of intellectual honesty, of being aware that we may know Elasticsearch best, but the customer team knows their business and their technical domain inside out. At the end, we share the same goal, and the goal is to have the best, the most stable Elasticsearch installation possible in every use case.


	Psychologist Daniel Kahneman argues in Thinking, Fast and Slow that people remember the end of the experience, not the experience itself. I think that&apos;s very relevant in the context of support. It really matters how we leave a customer feeling when we are done.


	<img src="https://api.contentstack.io/v2/assets/575e4c739e7a83165490bcdd/download?uid=blt4cdc54f4e99108d3?uid=blt4cdc54f4e99108d3" data-sys-asset-uid="blt0f2be6d81dab4437" style="max-width:100%;">
</body></html>

The People Behind the Projects: More than a Bug

<html><head></head><body>
	Today, we are pleased to announce the bugfix release of Elasticsearch 1.5.1, based on Lucene 4.10.4. This is the latest stable version of Elasticsearch.


	You can <a class="ulink" href="/downloads/past-releases/elasticsearch-1-5-1">read the full changes list and download Elasticsearch 1.5.1 here</a>.


	This release contains an important bug fix which affects the speed of allocating a shard to a new node. The first phase of shard recovery copies all the segment files from the source node to the target node. During this phase, any index, update, or delete requests are recorded in the transaction log, to be replayed on the target node at the end of recovery. If the shard is very large, this transaction log can accumulate many events which need to be replayed.&#xA0;


	 Previously, merges of new segments were disabled on the target node during recovery. A large transaction log would result in an explosion of small new segments, severely impacting the speed of recovery. Issue <a class="ulink" href="https://github.com/elastic/elasticsearch/pull/10463" target="_top">#10463</a> changes the behaviour to enable merges on the target shard during recovery.


	Other notable bug fixes include:

<div class="itemizedlist">
	<ul class="itemizedlist" type="disc">
		<li class="listitem">
		Forcing a refresh if too many deletes fill up the version map (<a class="ulink" href="https://github.com/elastic/elasticsearch/pull/10312" target="_top">#10312</a>)
		</li>
		<li class="listitem">
		Better handling of repositories containing many snapshots (<a class="ulink" href="https://github.com/elastic/elasticsearch/pull/10366" target="_top">#10366</a>)
		</li>
		<li class="listitem">
		Several fixes for the experimental <a class="ulink" href="/guide/en/elasticsearch/reference/current/search-request-inner-hits.html" target="_top">inner hits</a> feature (<a class="ulink" href="https://github.com/elastic/elasticsearch/pull/10388" target="_top">#10388</a>, <a class="ulink" href="https://github.com/elastic/elasticsearch/pull/10353" target="_top">#10353</a>, <a class="ulink" href="https://github.com/elastic/elasticsearch/pull/10309" target="_top">#10309</a>, <a class="ulink" href="https://github.com/elastic/elasticsearch/pull/10235" target="_top">#10235</a>)
		</li>
	</ul>
</div>

	Finally, in case you haven&apos;t seen it, <a class="ulink" href="/blog/deprecating_rivers" target="_top">rivers have been deprecated</a>.


	Please download <a class="ulink" href="/downloads/past-releases/elasticsearch-1-5-1" target="_top">Elasticsearch 1.5.1</a>, try it out, and let us know what you think on Twitter (<a class="ulink" href="https://twitter.com/elastic" target="_top">@elastic</a>). You can report any problems on the <a class="ulink" href="https://github.com/elastic/elasticsearch/issues" target="_top">GitHub issues page</a>.
</body></html>

Elasticsearch 1.5.1 Released

<html><head></head><body>
	We have a new release candidate (<a href="https://www.elastic.co/downloads/logstash">RC3</a>) for 1.5.0. After RC2 was&#xA0;released last month, we got a lot of input from the community. Among them were reports of performance regressions, which has been the central focus of this release. We also fixed important bugs so we decided to provide an update before making 1.5.0 generally available. Check out the <a href="https://github.com/elastic/logstash/blob/master/CHANGELOG" target="_blank">changelog</a> for details.
	The highlights:
<h2>
Performance regression</h2>
	Users reported a drop in throughput performance when compared to 1.4.2. This was especially evident in configurations&#xA0;using conditionals in filters and outputs. We tweaked the code generation in these pipeline stages and made it more efficient. <a href="https://github.com/elastic/logstash/issues/2870" target="_blank">#2870</a> has the details including numbers.<h2>
Shutdown issues</h2>Fixed a bug which manifested in Logstash not shutting down gracefully. The root cause for this issue was pipeline not sending the right shutdown signal to the plugins. (<a href="https://github.com/elastic/logstash/issues/2796" target="_blank">#2796</a>, &#xA0;<a href="https://github.com/elastic/logstash/issues/2871" target="_blank">#2871</a>)
<h2>	Java Options</h2>
	Added the ability to append extra JVM options while running Logstash. Setting the&#xA0;LS_JAVA_OPTS environment variable will add to the default options. This enhancement&#xA0;also provides&#xA0;the ability to completely replace the JVM options. For example, users may change the GC algorithm if they wish and use their own instead of the out of the box settings. (#<a href="https://github.com/elastic/logstash/pull/2942" target="_blank">2942)</a>&#xA0;
	 
<h2>Grok Filter regression</h2>
	Fixed an issue where coercion of int and floats in Grok pattern was causing empty fields to be created (in some cases). This was a side effect of the performance fixes made previously&#xA0;for the Grok library. (<a href="https://github.com/elastic/logstash/issues/2897" target="_blank">#2897</a>)
<h2>File Input</h2>
	Users mentioned permission issues (with respect to sinceDB) during Logstash shutdown. Fixed this and other corruption issues while writing file position to sinceDB in the Filewatch gem. (<a href="https://github.com/elastic/logstash/issues/2935" target="_blank">#2935</a>, <a href="https://github.com/elastic/logstash/issues/2882" target="_blank">#2822</a>, <a href="https://github.com/elastic/logstash/issues/2964" target="_blank">#2964</a>)
	 
	 
	As always, thanks to all our users for the valuable feedback and contributions. We are in the home stretch for releasing 1.5.0. Please download <a href="https://www.elastic.co/downloads/logstash">RC3</a>, try it out, and&#xA0;send your feedback by opening an <a href="https://github.com/elastic/logstash/issues" target="_blank">issue</a>&#xA0;on GitHub&#xA0;or <a href="https://twitter.com/elastic" target="_blank">tweet</a> to us.&#xA0;
	 
</body></html>

Logstash 1.5.0 RC3 released

<html><head></head><body>One of the&#xA0;new&#xA0;features that was added to Kibana in version&#xA0;4.1 is something we call Pinned Filters. Pinned Filters allows you to craft filters in one part of Kibana and bring those filters with you to all the other parts. It started as an enhancement idea from a user in the IRC channel, and now we can&apos;t imagine using Kibana without it.&#xA0; Creating Pinned Filters is a simple, 1-click operation. You start by creating some filters, for example, by <a href="https://www.elastic.co/guide/en/kibana/current/discover.html#discover-filters">clicking on document values in Discover</a>. <img src="https://api.contentstack.io/v2/assets/575e4c949e7a83165490c28c/download?uid=blte40c4b415af3d4ca?uid=blte40c4b415af3d4ca" data-sys-asset-uid="blt0670ee9516b61742">Next, you need to decide what filters you&apos;d like to take with you. Once you&apos;ve decided, simply hover over the filter and click the pin icon, and you&apos;ll see a pin&#xA0;icon&#xA0;show up on the filter. This icon&#xA0;lets&#xA0;you quickly see&#xA0;which filters are pinned. You can click on the pin icon again to un-pin the filter.&#xA0;<img src="https://api.contentstack.io/v2/assets/575e4c95d8edd48f76935efa/download?uid=blt6ffb3fdc9b6e9502?uid=blt6ffb3fdc9b6e9502" data-sys-asset-uid="blt3e43cf38254db3ae"> If you&apos;ve got a lot of filters and you simply want to pin them all, you could spend your time clicking on each one, but there&apos;s a faster way. If you click on the Actions&#xA0;menu in the filter bar, it will open a menu that allows you manipulate all&#xA0;of the filters, simply click on Pin.<img src="https://api.contentstack.io/v2/assets/575e4c953dc542cb38c083dc/download?uid=blt3deca09ddb42205a?uid=blt3deca09ddb42205a" data-sys-asset-uid="blte21b2ad2c4e3c0b7">Now that you&apos;ve got one or&#xA0;more pinned filters, when you go to another app, like Visualize, the filters will follow along. It&apos;s that simple! <img src="https://api.contentstack.io/v2/assets/575e4c959e7a83165490c29c/download?uid=blt2186714d7ff4a4bb?uid=blt2186714d7ff4a4bb" data-sys-asset-uid="bltc4b7be77407d6df9">Generally, users use filters to tease out interesting records in their data set, and pinned filters let them get more out of that effort. For example, after&#xA0;filtering&#xA0;down data in a visualization, they can take those filters over to Discover and see the records responsible for that visualization.&#xA0;That concludes our crash course in Pinned Filters, now go forth and pin!</body></html>

pinned-filters-thumbtack.jpg

pinned-filters-thumbtack-thumb.jpg

Stick With It - Pinned Filters in Kibana

<html><head></head><body>Hello, fellow Logstashers!
We&#x2019;ve been pleased with the community response to our recent 2.0 release. &#xA0;As this was a major release, there were some breaking changes, especially in conjunction with the Elasticsearch 2.0 release. &#xA0;One of these changes was that <a href="https://www.elastic.co/guide/en/elasticsearch/reference/2.0/breaking_20_mapping_changes.html#_field_names_may_not_contain_dots">Elasticsearch 2.0 does not support field names with a .&#xA0;(or dot character) in them</a>. &#xA0;A few Logstash plugins had to be changed to not use dotted fields to be compatible, like the metrics and elapsed filters. &#xA0;Unfortunately, many users have no control over the sources of their fields. &#xA0;This has resulted in a poor user experience where dotted fields existed. &#xA0;To address this issue, the <a href="https://github.com/logstash-plugins/logstash-filter-de_dot" target="_blank">de_dot</a>&#xA0;filter has been created. &#xA0;You can find the plugin documentation for it <a href="https://www.elastic.co/guide/en/logstash/master/plugins-filters-de_dot.html" target="_blank">here</a>.
You can install the de_dot filter using the plugin command:
<pre class="prettyprint">bin/plugin install logstash-filter-de_dot</pre>The de_dot filter will replace dots with underscores by default. &#xA0;It&#x2019;s as simple as that! &#xA0;If I had a dotted field called <code>baked.apple.pie</code>, the de_dot filter would change the field name to <code>baked_apple_pie</code>. &#xA0;You can also choose the separator with the <code>separator</code>&#xA0;configuration option. &#xA0;A separator does not have to be a single character, either. &#xA0;You can make a separator anything you want, except a dot. &#xA0;Please remember to change your queries and filters in Kibana to match whatever you change in the de_dot filter!
We also added an option to translate dots into nested fields. &#xA0;If you set <code>nested =&gt; true</code>&#xA0;in your configuration, Logstash will ignore the <code>separator</code>&#xA0;and attempt to convert dotted fields into nested fields. &#xA0;For example, if I had a dotted field called <code>top.level1.level2</code>, it would become <code>[top][level1][level2]</code>. &#xA0;The flexibility is yours to choose.
We&#x2019;ve provided a <code>fields</code>&#xA0;configuration option to allow you to specify which fields need to be processed. &#xA0;If no fields are provided, Logstash will check all&#xA0;of your fields for dots. &#xA0;The de-dot process can be a performance hit to your pipeline, so pre-specifying fields will prevent Logstash from having to check each for dots. &#xA0;Please note that you won&#x2019;t need to use a conditional to check for dots in field names, as the plugin does this for you.&#xA0; The idea behind specifying <code>fields</code>&#xA0;manually is to prevent Logstash from having to check every field, which can result in improved performance. &#xA0;If you are in a situation where you do not know all possible field names, it may be helpful to tag events which may&#xA0;have dotted fields, and put the de_dot filter within a conditional that checks for that tag.
The <code>fields</code>&#xA0;option also allows you to de_dot nested fields. &#xA0;The current release of the de_dot filter will only check the top level of an event for dotted fields. &#xA0;If you have a dotted field in a nested object, the de_dot filter cannot find it without help. &#xA0;In this case you must use the <code>fields</code>&#xA0;directive to specify a nested field. &#xA0;This can be done by using the field reference syntax: <code>fields =&gt; [ &quot;[top][level.1]&quot;, &quot;[top][level.2]&quot; ]</code>&#xA0;will find the nested dotted fields <code>level.1</code>&#xA0;and <code>level.2</code>. &#xA0;You can use either the <code>separator</code>&#xA0;or <code>nested</code>&#xA0;options to act on these fields.
We hope that the de_dot filter will help ease your transition to Logstash 2.0 and Elasticsearch 2.0. &#xA0;If you need help configuring the de_dot filter, please visit our discussion forum at <a href="https://discuss.elastic.co/c/logstash">https://discuss.elastic.co/c/logstash</a>. &#xA0;If you have suggestions or find an issue, please submit a ticket at <a href="https://github.com/logstash-plugins/logstash-filter-de_dot">https://github.com/logstash-plugins/logstash-filter-de_dot</a>.
Happy Logstashing!
</body></html>

filter.jpg

Introducing the de_dot filter

<html><head></head><body><section id="post_content">
		<article>
			We are pleased to announce the release of Logstash 1.5.0 Beta 1! You can download it <a href="http://www.elasticsearch.org/overview/logstash/download/">here</a>.
Note: This is a Beta release &#x2013; please do not use in production.
<h2><a href="https://gist.github.com/suyograo/4b13850ad26c2f0d2957#whats-in-it" rel="noreferrer" name="user-content-whats-in-it"></a>What&apos;s in it?</h2>
The main themes of 1.5.0 are plugin management, performance improvements, and <a href="http://kafka.apache.org/">Apache Kafka</a> integration. One of the core strengths of Logstash is the availability of plugins and ease of adding them to your pipeline to extend behavior. With this release we are making it easier to develop, manage, and publish plugins. We also made Logstash faster, so you can process more data in less time. Intrigued? Let&apos;s dive into the details.
<h2><a href="https://gist.github.com/suyograo/4b13850ad26c2f0d2957#plugin-ecosystem-changes" rel="noreferrer" name="user-content-plugin-ecosystem-changes"></a>Plugin Ecosystem Changes</h2>
Logstash has a rich collection of over 165 plugins (inputs, filters, outputs, and codecs) which are both developed by Elasticsearch and contributed to by the community. Managing that many plugins poses unique trade-offs between usability and agility. On one hand, bundling all the plugins with Logstash makes it easier to use, but forces the community to wait for new releases of Logstash to incorporate plugin updates. On the other hand, separating the distribution of plugins from Logstash makes updates easier, but impacts ease of use (especially for new users).
As we move the project forward, we are trying to balance both of these aspects. Previously, we divided all available plugins into &#x2018;core&#x2019; and &#x2018;contrib&#x2019; packages. The most commonly used plugins in &#x2018;core&#x2019; were shipped by default with Logstash. The community-contributed plugins shipped separately as the &apos;contrib&apos; package. With the 1.5.0 release, we are taking a step closer to making plugin management even better for our users. We have moved all the plugins into their own self-contained packages. Using rubygems as a packaging framework, we&apos;re publishing and distributing these plugins via <a href="http://rubygems.org">rubygems.org</a>. We have also added infrastructure to easily install, update, and remove plugins on top of Logstash.
For example, to install the S3 output plugin, you would have to:
<pre>&lt;code&gt;$LS_HOME/bin/plugin install logstash-output-s3
</pre>
and that&apos;s it! Logstash will download the gem and its dependencies from rubygems.org and install them so you can start sending your data to S3 buckets.
Downloadable Logstash releases will still include lots of plugins out of the box, but now you can upgrade individual plugins and install new ones, at any time! Watch out for a more detailed blog post on our plugin ecosystem changes soon.
<h2><a href="https://gist.github.com/suyograo/4b13850ad26c2f0d2957#performance-improvements" rel="noreferrer" name="user-content-performance-improvements"></a>Performance Improvements</h2>
Logstash 1.5.0 is much faster. Let&apos;s highlight two areas where performance has gone way up.
<h3><a href="https://gist.github.com/suyograo/4b13850ad26c2f0d2957#grok-filter" rel="noreferrer" name="user-content-grok-filter"></a>Grok Filter</h3>
Grok filter is used to describe patterns for extracting structured data in Logstash. In this release, we have increased the throughput of the popular grok filter in some patterns by 100%. Put another way, you can process more data through Logstash when using the grok filter.
In our benchmark testing, we compared throughput in 1.5.0 and 1.4.2 by processing 6.9 million entries of Apache Web access log lines using the <code>COMBINEDAPACHELOG</code> grok pattern. Throughput in 1.5.0 increased from 34K events per second (eps) to 50K eps. Both tests were run on an eight-core machine with eight worker threads in Logstash. These tests we run with a single grok filter and measured throughput of events processed in the pipeline using a <code>stdin</code> input and <code>stdout</code> output. Please note that overall performance will vary with hardware and Logstash configuration used.
<h3><a href="https://gist.github.com/suyograo/4b13850ad26c2f0d2957#json-serializationdeserialization" rel="noreferrer" name="user-content-json-serializationdeserialization"></a>JSON Serialization/Deserialization</h3>
We implemented JSON serialization and deserialization using the <a href="https://github.com/guyboertje/jrjackson">JrJackson</a> library, which improved the throughput by over 100%. In our previously mentioned performance tests, we sent 500,000 JSON events 1.3KB in size and measured a throughput increase from 16K eps to 30K eps. With events 45KB in size, throughput increased from 850 eps to 3.5K eps. Yeah, we thought that was pretty good, too.
<h2><a href="https://gist.github.com/suyograo/4b13850ad26c2f0d2957#apache-kafka-integration" rel="noreferrer" name="user-content-apache-kafka-integration"></a>Apache Kafka Integration</h2>
Today, Apache Kafka is ubiquitous in large-scale data processing systems. In scaling Logstash deployments, Kafka can also be used as an intermediate message buffer to store data between the shipping instances and indexing instances.
In 1.5.0, we have added built-in support for the Logstash Kafka input and output plugin that was originally developed by <a href="https://github.com/joekiller/logstash-kafka">Joseph Lawson</a>. We have enhanced these plugins by adding integration tests and documentation, and we will continue to develop new Kafka features. We have also added an <a href="http://avro.apache.org/">Apache Avro</a> codec so you can easily consume events stored in Kafka, enrich them, and analyze them using the ELK stack.
Adding Kafka input is as simple as:
<pre>$LS_HOME/bin/plugin install logstash-input-kafka
</pre>
Kafka output:
<pre>$LS_HOME/bin/plugin install logstash-output-kafka
</pre>
<h2><a href="https://gist.github.com/suyograo/4b13850ad26c2f0d2957#improved-security" rel="noreferrer" name="user-content-improved-security"></a>Improved Security</h2>
We have improved the security of the Elasticsearch output, input, and filter by adding authentication and transport encryption support. For instance, with the HTTP protocol you can configure SSL/TLS to enable encryption and HTTP basic authentication to provide a username and password while making requests. These capabilities will enable Logstash to natively integrate with the forthcoming <a href="http://www.elasticsearch.org/overview/shield/">Elasticsearch Shield</a> security product.
<h2><a href="https://gist.github.com/suyograo/4b13850ad26c2f0d2957#documentation" rel="noreferrer" name="user-content-documentation"></a>Documentation</h2>
Previously, Logstash documentation was hosted on <a href="http://logstash.net">logstash.net</a>, which made it cumbersome to find information when working with the rest of the ELK stack. We are now moving documentation for 1.5.0 and all future releases to the elasticsearch.org website under the <a href="http://www.elasticsearch.org/guide/en/logstash/current/index.html">Logstash Guide</a>. With this migration, <a href="http://elasticsearch.org/guide">elasticsearch.org/guide</a> becomes the single location for all reference documentation to learn and use the ELK stack. As we iterate on this beta release, we are actively working on improving the presentation and documentation quality. (Note: All of the old Logstash documentation will continue to be available in the current <a href="http://logstash.net/docs/1.4.2/">logstash.net</a> location.)
<h2><a href="https://gist.github.com/suyograo/4b13850ad26c2f0d2957#bug-fixes-and-enhancements" rel="noreferrer" name="user-content-bug-fixes-and-enhancements"></a>Bug Fixes and Enhancements</h2>
In addition to all these new features, Logstash 1.5.0 has fixed a number of bugs and enhanced the functionality of many features. We would like to highlight few of them here:
<ul>
<li>Allow storing &#x2018;metadata&#x2019; to an event which is not sent/encoded on output. This eliminates the need for intermediate fields for example, while using date filter.(<a href="https://github.com/elasticsearch/logstash/issues/1834">#1834</a>, <a href="https://logstash.jira.com/browse/LOGSTASH-1798">#LOGSTASH-1798</a>)</li>
<li>Fixed file descriptor leaks when using HTTP. The fix prevents Logstash from stalling, and in some cases crashing from out-of-memory errors (<a href="https://github.com/elasticsearch/logstash/issues/1604">#1604</a>)</li>
<li>Twitter input: added improvements, robustness, fixes. <code>full_tweet</code> option now works, and we handle Twitter rate limiting errors (<a href="https://github.com/elasticsearch/logstash/issues/1471">#1471</a>)</li>
<li>Filters that generated events (multiline, clone, split, metrics) now propagate those events correctly to future conditionals (<a href="https://github.com/elasticsearch/logstash/issues/1431">#1431</a>)</li>
<li>Elasticsearch output: Logstash will not create a <code>message.raw</code> field by default now. Message field is <code>not_analyzed</code> by Elasticsearch and adding a multi-field was essentially doubling the disk space required, with no benefit</li>
<li>Remove ability to run multiple subcommands from bin/logstash like <code>bin/logstash agent -f sample.conf -- web</code> (<a href="https://github.com/elasticsearch/logstash/issues/1797">#1797</a>)</li>
</ul>
You can read about the features, enhancements, and bug fixes in this release in the Logstash 1.5.0.Beta1 <a href="https://github.com/elasticsearch/logstash/blob/master/CHANGELOG">changelog</a>
<h2><a href="https://gist.github.com/suyograo/4b13850ad26c2f0d2957#give-it-a-spin" rel="noreferrer" name="user-content-give-it-a-spin"></a>Give it a spin!</h2>
Please download Logstash 1.5.0 Beta 1, try it out, and let us know what you think on Twitter (<a href="http://twitter.com/elasticsearch">@elasticsearch</a>). You can also report any problems on the Github <a href="https://github.com/elasticsearch/logstash/issues">issues</a> page.
			<aside id="post_tags">
							</aside>
		</article>
			</section></body></html>

Logstash 1.5.0.Beta1 Released

<html><head></head><body>
	Eeeeeeeee! It&apos;s here! Kibana 4 Beta 2 is ready to be poked, prodded, and otherwise fiddled with by you, your friends, and your exceptionally intelligent pets. We look forward to your feedback.

	If you just can&apos;t wait to get started, <a href="/downloads/kibana" title="/downloads/kibana">grab Kibana 4 Beta 2 here</a>, otherwise read on for the good stuff.

	Along with dozens of small tweaks and fixes, here are some of the fun new things to look out for in this release of Kibana:
<h2>Map Support</h2>
	Maps are back and they are more powerful than ever! The new tile map visualization makes use of Elasticsearch&apos;s awesomely powerful <code>geohash_grid</code> to display geographic data, such as visualizing relative response times:

	<img src="https://api.contentstack.io/v2/assets/575e4d9a12798d8e3820033c/download?uid=blt9c1aa62c92bb348e?uid=blt9c1aa62c92bb348e" style="max-width:100%;">
<h2>Visualization Options</h2>
	In Beta 1, bar charts were always stacked. In Kibana 4 Beta 2, we&apos;ve added options that allow you to modify how visualizations display their data. For example, by grouping bars:

	<img src="https://api.contentstack.io/v2/assets/575e4d9b0342dfd738266086/download?uid=blt31706af8071f9e61?uid=blt31706af8071f9e61" style="max-width:100%;">

	Or displaying bars as percentages:

	<img src="https://api.contentstack.io/v2/assets/575e4d9b3dc542cb38c0b12a/download?uid=blt87d48888a6af3973?uid=blt87d48888a6af3973" style="max-width:100%;">
<h2>Area Charts</h2>
	Beta 2 brings back area charts, both stacked and unstacked:

	<img src="https://api.contentstack.io/v2/assets/575e4d9b9985d5897637976b/download?uid=blt2b54dad859c69f0a?uid=blt2b54dad859c69f0a" style="max-width:100%;">
<h2>Advanced Parameters</h2>
	We aim to support as many Elasticsearch features as possible, but sometimes we just haven&apos;t gotten to that one aggregation option you really need right now. For that option, we&apos;ve introduced a JSON input which allows you to specify additional aggregation parameters to send with your request. For example, maybe you want to pass <code>shard_size</code> in a <code>terms</code> agg, or increase the <code>precision_threshold</code> in a cardinality agg. In this case, we pass a script as an advanced parameter, taking the log of the <code>_value</code> of the <code>bytes</code> field and placing it on the X-axis:

	<img src="https://api.contentstack.io/v2/assets/575e4d9b12798d8e3820035a/download?uid=bltc8a358993404ed63?uid=bltc8a358993404ed63" style="max-width:100%;">
<h2>Data Tables</h2>
	Sometimes you want a flashy chart, and sometimes you just want the numbers. The data table visualization makes that desire a reality:

	<img src="https://api.contentstack.io/v2/assets/575e4d9b58208ba076e2aeae/download?uid=blt25fa82f0a07abbaf?uid=blt25fa82f0a07abbaf" style="max-width:100%;">
<h2>Hey! Where Did My Dashboards Go?</h2>
	The internal Kibana index changed from <code>kibana-int</code> to <code>.kibana</code>. We recommend you move your documents (e.g., dashboards, settings, visualizations) from the old index to the new one. However you can also simply set <code>kibanaIndex: &quot;kibana-int&quot;</code> in kibana.yml.
<h2>What Are We Working on Now?</h2>
	Check out our path to Kibana 4 GA on our <a href="https://github.com/elasticsearch/kibana/labels/roadmap" title="https://github.com/elasticsearch/kibana/labels/roadmap" target="_blank">roadmap</a>. As always, we welcome feedback/bugs/fixes on <a href="https://github.com/elasticsearch/kibana/issues" title="https://github.com/elasticsearch/kibana/issues" target="_blank">GitHub</a>
</body></html>

kibana4-bubble-map.jpg

Kibana 4 Beta 2: Get It Now

<html><head></head><body>Here on the Kibana team, we&apos;re in a unique position; we consume the products that other teams are building as part of what we are building. At a minimum, we have to stand up an Elasticsearch instance and we have to index some data. Short of that, we can&apos;t run Kibana, and we can&apos;t do our jobs.&#xA0;
This may seem simple enough, but it starts to get complicated pretty quickly when we need to test against many different versions of Elasticsearch. We also often need to test different plugin configurations, different datasets, and even run&#xA0;development versions of Elasticsearch. With all these varying requirements, things can get complicated and time consuming quickly. 
In order to make our lives easier, it&apos;s in our best interest to automate as much of that process as possible. Naturally, we&apos;ve built some tools to help with that. 
<h2>Automating Data</h2>Let&apos;s work backwards and start with indexing data first. To help us with that task, we use a tool called <a href="https://www.npmjs.com/package/makelogs" target="_blank">makelogs</a>. As the project states, it&#xA0;pushes fake HTTP traffic logs into Elasticsearch. There&apos;s a little bit of edge-case data that it generates, but it&apos;s basically the kind of logs you expect from any Apache or nginx server, with a bell shaped traffic pattern and everything. 
<img src="https://api.contentstack.io/v2/assets/575e4d0343e9adc538714e63/download?uid=bltb86257faeb11df87" data-sys-asset-uid="bltb86257faeb11df87" alt="makelogs_data.png" style="max-width:100%;">
It&apos;s a quick and dirty -&#xA0;and very convenient -&#xA0;way for us to get data indexed so that we can start creating visualizations. It&apos;s really handy for smoke-testing things, particularly when we are spinning up new clusters, which we do a lot. It&apos;s also a great way to get people new to the team up and running, before they find something real that they want to index. 
<h2>Automating Elasticsearch</h2>Makelogs is great, but its scope is tiny, and its utility outside of Kibana development is limited. Perhaps more interesting is the tool we use to automate Elasticsearch, a tool named <a href="https://www.npmjs.com/package/esvm" target="_blank">esvm</a>.
Short for Elasticsearch Version Manager, esvm was born out of our need to maintain not just multiple versions of Elasticsearch, but also multiple clusters, each with its own unique configuration and&#xA0;data. It will download the specified version of Elasticsearch, start it up with the default configuration, use your computer&apos;s hostname for the cluster name to prevent external auto-joining, and even wrap its output to help make it a little easier to read.
<img src="https://api.contentstack.io/v2/assets/575e4d0343e9adc538714e65/download?uid=blt4399748b74abede2" data-sys-asset-uid="blt4399748b74abede2" alt="esvm-no-options.png" style="max-width:100%;">
If you want to run the latest version of Elasticsearch, no arguments are required, just run <code>esvm</code>. If you want a different version, just pass it as an argument, like <code>esvm 2.0.1</code>. If, instead of a version, you need to run a build from a specific branch, something like <code>esvm -b 2.x</code> will do just that.
All of that is handy, but esvm gets really interesting, and really powerful, when you use it with a JSON config file.&#xA0; 
<h3>Automating Cluster Configuration</h3>The esvm config takes all the settings from the standard elasticsearch.yml and passes them as runtime settings. Common settings like enabling CORS, turning on mlock, and turning off multicast make great default settings. Any clusters you define will inherit the default settings, but also allow you to override them as needed. Plus, you can define the version or branch to use, and any plugins to install. For example, if you wanted to run a 3 node cluster built from the latest commit on the master&#xA0;branch, this is all it takes: 
<pre class="prettyprint">&quot;clusters&quot;: {
 &quot;latest&quot;: {
 &quot;branch&quot;: &quot;master&quot;,
 &quot;nodes&quot;: 3
 },
}
</pre>Then run <code>esvm -c esvm.json latest</code> and you&apos;re up and running.
From there, adding plugins is&#xA0;easy,&#xA0;just add a &quot;plugins&quot; section&#xA0;to the&#xA0;cluster configuration and a list of the plugins you&apos;d like to install.
<pre class="prettyprint">&quot;plugins&quot;: [
 &quot;license&quot;,
 &quot;shield&quot;
]
</pre>Relaunch the cluster and now you&apos;ve got Shield installed with an evaluation license ready to go. And if you&apos;d like to&#xA0;pre-define some users and their roles, just add this to the configuration: 
<pre class="prettyprint">&quot;shield&quot;: {
 &quot;users&quot;: [
 {
 &quot;username&quot;: &quot;kibana&quot;,
 &quot;password&quot;: &quot;notsecure&quot;,
 &quot;roles&quot;: [&quot;kibana4_server&quot;]
 },
 {
 &quot;username&quot;: &quot;user&quot;,
 &quot;password&quot;: &quot;notsecure&quot;,
 &quot;roles&quot;: [&quot;kibana4&quot;]
 }
 ]
}
</pre>And because you can define&#xA0;as many cluster configurations as you&apos;d like, a single JSON file will allow you to easily spin up whatever configuration&#xA0;you need. Of course, once you have&#xA0;a few clusters defined you may find yourself forgetting what&apos;s available. Enter the <code>--list</code>&#xA0;flag.
<img src="https://api.contentstack.io/v2/assets/575e4d039985d58976377e11/download?uid=blt3b358d2224175199" data-sys-asset-uid="blt3b358d2224175199" alt="esvm-config-list.png" style="max-width:100%;">
So that&apos;s makelogs and esvm, two tools that make our jobs easier and help us focus on building Kibana, instead of setting up infrastructure.&#xA0;
We&apos;re also working on building tools that will make our users&apos; lives easier too, the first of which is our <a href="https://github.com/elastic/generator-kibana-plugin" target="_blank">Kibana plugin generator</a>, which helps you get started writing your own Kibana plugins.&#xA0;But that&apos;s a topic for another post. Stay tuned!
</body></html>

chicago-elasticon-tour-11-19-2015.jpg

development-tooling-is-magic.jpg

Development Tooling Behind Kibana

<html><head></head><body><article>
			With the release of Elasticsearch 1.3.8 and 1.4.3, we have, by default, disabled the ability for Groovy scripts to be sent dynamically with a request or an indexed script, but Groovy remains the default scripting language. This blog post is here to show you how to continue to use scripts for unsandboxed languages, albeit a little less dynamically.
This blog post exists to help you to understand what that means and, more importantly, how to continue using scripts to get important tasks done safely.
<div class="section">
<div class="titlepage">
<div>
<div>
<h2 class="title"><a id="_what_is_a_dynamic_script" href="#_what_is_a_dynamic_script"></a>What is a Dynamic Script?</h2>
</div>
</div>
</div>
For anyone not familiar with Elasticsearch, you can submit scripts as a part of many different requests: searches, aggregations, updates, upserts, and deletes by query. In short, you can effectively add scripting behavior to extend beyond the normal behavior for your own use cases.
For example, this is a dynamic script that returns documents that have the same value for <code class="literal">field1</code> and <code class="literal">field2 + shift</code>:
<pre class="programlisting prettyprint lang-json">GET /_search { &#xA0; &quot;query&quot;:{ &#xA0; &#xA0; &quot;filtered&quot;:{ &#xA0; &#xA0; &#xA0; &quot;filter&quot;:{ &#xA0; &#xA0; &#xA0; &#xA0; &quot;script&quot;:{ &#xA0; &#xA0; &#xA0; &#xA0; &#xA0; &quot;script&quot;:&quot;doc[&apos;field1&apos;].value == (doc[&apos;field2&apos;].value + shift)&quot;, &#xA0; &#xA0; &#xA0; &#xA0; &#xA0; &quot;lang&quot;:&quot;groovy&quot;, &#xA0; &#xA0; &#xA0; &#xA0; &#xA0; &quot;params&quot;:{ &#xA0; &#xA0; &#xA0; &#xA0; &#xA0; &#xA0; &quot;shift&quot;:3 &#xA0; &#xA0; &#xA0; &#xA0; &#xA0; } &#xA0; &#xA0; &#xA0; &#xA0; } &#xA0; &#xA0; &#xA0; } &#xA0; &#xA0; } &#xA0; } }</pre>
You can change the language, which may naturally require the syntax to change and add limitations (e.g., <a class="ulink" href="http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-scripting.html#_lucene_expressions_scripts" target="_top">Lucene Expressions</a> instead of Groovy scripts), by changing the <code class="literal">lang</code> parameter to the chosen language.
<div class="section">
<div class="titlepage">
<div>
<div>
<h3 class="title"><a id="_why_is_that_dynamic" href="#_why_is_that_dynamic"></a>Why is that Dynamic?</h3>
</div>
</div>
</div>
The above is an example of a dynamic script because the actual script portion needs to be interpreted and compiled on the server side dynamically. Dynamic scripts are any script that is sent to the data nodes via an Elasticsearch API, which includes <a class="ulink" href="http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-scripting.html#_indexed_scripts" target="_top">indexed scripts</a>.
In other words, if the script is not stored with all of the data nodes, then it is treated as a dynamic script.
</div>
</div>
<div class="section">
<div class="titlepage">
<div>
<div>
<h2 class="title"><a id="_what_to_expect_with_dynamic_scripting_disabled" href="#_what_to_expect_with_dynamic_scripting_disabled"></a>What to Expect with Dynamic Scripting Disabled</h2>
</div>
</div>
</div>
Due to the changes in the latest release, dynamic scripting is now disabled by default for Groovy. With that and given the earlier example script, this is what you should expect to see if you submitted that request (reduced for brevity and emphasis):
<pre class="programlisting prettyprint lang-json">{ &#xA0; &#xA0;&quot;error&quot;:&quot;SearchPhaseExecutionException[Failed to execute phase [query], all shards failed; shardFailures {[8FJ02MofSnqVvOQ10BXxhQ][test][0]: SearchParseException[[test][0]: from[-1],size[-1]: Parse Failure [Failed to parse source [{...}]]]; nested: ScriptException[dynamic scripting for [groovy] disabled]...&quot;, &#xA0; &#xA0;&quot;status&quot;:400 }</pre>
The key part of the error message is &#x201C;ScriptException[dynamic scripting for [groovy] disabled]&#x201D;.
</div>
<div class="section">
<div class="titlepage">
<div>
<div>
<h2 class="title"><a id="_how_to_continue_to_use_scripting" href="#_how_to_continue_to_use_scripting"></a>How to Continue to use Scripting</h2>
</div>
</div>
</div>
There are three ways to submit a script to Elasticsearch. The two dynamic ways are per-request scripting (shown) or using an indexed script. Using an indexed script works by storing Groovy scripts in Elasticsearch itself, and using them on demand (this works really well, but it still allows untrusted users to add their own scripts given open access to your cluster!). These work in much the same way as a stored procedure works in a relational database: you prewrite your script and invoke it by name later as part of a request.
<pre class="programlisting prettyprint lang-json">GET /_search { &#xA0; &quot;query&quot;:{ &#xA0; &#xA0; &quot;filtered&quot;:{ &#xA0; &#xA0; &#xA0; &quot;filter&quot;:{ &#xA0; &#xA0; &#xA0; &#xA0; &quot;script&quot;:{ &#xA0; &#xA0; &#xA0; &#xA0; &#xA0; &quot;script_id&quot;:&quot;your_custom_script&quot;, &#xA0; &#xA0; &#xA0; &#xA0; &#xA0; &quot;lang&quot;:&quot;groovy&quot;, &#xA0; &#xA0; &#xA0; &#xA0; &#xA0; &quot;params&quot;:{ &#xA0; &#xA0; &#xA0; &#xA0; &#xA0; &#xA0; &quot;shift&quot;:3 &#xA0; &#xA0; &#xA0; &#xA0; &#xA0; } &#xA0; &#xA0; &#xA0; &#xA0; } &#xA0; &#xA0; &#xA0; } &#xA0; &#xA0; } &#xA0; } }</pre>
Notice that not much changed, except that the inner <code class="literal">script</code> became <code class="literal">script_id</code> and its value changed to the name of the prewritten script.
The non-dynamic way to provide scripts to Elasticsearch is to write your script to a file on disk, instead of indexing them, and then store it with the configuration for each script that you plan to use. This is how you can continue to use unsandboxed scripts with dynamic scripting disabled, for any scripting language.
In the first example, the Groovy script was literally <code class="literal">doc[&apos;field1&apos;].value == doc[&apos;field2&apos;].value + shift</code>. You can write that script into your file with a &#x201C;.groovy&#x201D; extension:
<pre class="programlisting prettyprint lang-json">doc[&apos;field1&apos;].value == (doc[&apos;field2&apos;].value + shift)</pre>
If this file were named <code class="literal">your_custom_script.groovy</code> and placed in the <code class="literal">config/scripts</code> directory of all of your Elasticsearch data nodes, then Elasticsearch will pick up the script within 60 seconds (<a class="ulink" href="http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-scripting.html#_automatic_script_reloading" target="_top">configurable by changing <code class="literal">watcher.interval</code></a> in your elasticsearch.yml) and pre-compile it for use with future requests. You do need to ensure that the file is readable by the user running Elasticsearch! After writing this to disk, your configuration directory should look something like this:
<pre class="literallayout">config/
 elasticsearch.yml
 logging.yml
 scripts/
 your_custom_script.groovy</pre>
This is intentionally not as dynamic as submitting the script with each request or indexing the script, but it does still allow for dynamically adding scripts in a trusted environment.
</div>
<div class="chapter">
<div class="titlepage">
<div>
<div>
<h1 class="title"><a id="_using_a_script_written_to_disk" href="#_using_a_script_written_to_disk"></a>Using a script written to disk</h1>
</div>
</div>
</div>
Your script will not be usable until the script is loaded, at which point you will see something like this in your log file(s):
<pre class="literallayout">[2015-02-11 11:14:47,066][INFO ][script ] [Sergei Kravinoff] compiling script file [/path/to/elasticsearch-1.4.3/config/scripts/your_custom_script.groovy]</pre>
Once your script has been loaded by all of your Elasticsearch data nodes, then you can begin to use it. To use it, you supply the name of your script as the <code class="literal">file</code> (not <code class="literal">script_id</code>!):
<pre class="programlisting prettyprint lang-json">GET /_search { &#xA0; &quot;query&quot;:{ &#xA0; &#xA0; &quot;filtered&quot;:{ &#xA0; &#xA0; &#xA0; &quot;filter&quot;:{ &#xA0; &#xA0; &#xA0; &#xA0; &quot;script&quot;:{ &#xA0; &#xA0; &#xA0; &#xA0; &#xA0; &quot;file&quot;:&quot;your_custom_script&quot;, &#xA0; &#xA0; &#xA0; &#xA0; &#xA0; &quot;lang&quot;:&quot;groovy&quot;, &#xA0; &#xA0; &#xA0; &#xA0; &#xA0; &quot;params&quot;:{ &#xA0; &#xA0; &#xA0; &#xA0; &#xA0; &#xA0; &quot;shift&quot;:3 &#xA0; &#xA0; &#xA0; &#xA0; &#xA0; } &#xA0; &#xA0; &#xA0; &#xA0; } &#xA0; &#xA0; &#xA0; } &#xA0; &#xA0; } &#xA0; } }</pre>
Note: the language is optional because Groovy is the default language. If you use a different scripting language or if you change the default scripting language (e.g., to Lucene Expressions), then the language must be supplied to help it to find the right script.&#xA0;As a best practice, we recommend that applications include the <code class="literal">lang</code> parameter. This will insulate you from any future changes in the default scripting language.
</div>
<div class="chapter">
<div class="titlepage">
<div>
<div>
<h1 class="title"><a id="_questions" href="#_questions"></a>Questions?</h1>
</div>
</div>
</div>
If you have any questions, then please do not hesitate to reach out to us on Twitter (<a class="ulink" href="https://twitter.com/elasticsearch" target="_top">@elasticsearch</a>). You can report any problems on the <a class="ulink" href="https://github.com/elasticsearch/elasticsearch/issues" target="_top">GitHub issues page</a>.
</div>
			<aside id="post_tags">
							</aside>
		</article></body></html>

Running Groovy Scripts without Dynamic Scripting

<html><head></head><body>
	Curator 3.0 is a terrific update! It may not look like much has changed if you have been using a 2.x version, but there have been many improvements!
	 
	 
	I am pleased to present these changes to you!
<h2>Why change it?</h2>Is&#xA0;Curator 2&#xA0;working just fine for you? I&apos;m so glad to hear that! You can, of course, continue to use Curator 2. Because I was dissatisfied with the performance of some operations, I decided to fix it. For example, the Elasticsearch API does not require you to delete indices one at a time, but Curator 2 did. I was dissatisfied with the index selection parameters. While I did have some unit testing, Curator was sorely lacking command-line integration testing. Some bugs might have been easily detected with command-line-level integration testing. All of these, and more still, led me to write Curator 3.
<h2>Installation and Upgrading</h2>
	Install the latest version of Curator with:
	 
	<code>pip install elasticsearch-curator</code>
	 
	 
	Curator can be upgraded to the latest version by running:
	 
	<code>pip install -U elasticsearch-curator</code>

	Note: This is a major release update. Command-lines from Curator version 2 will not work with Curator 3. With some small modifications, though, they should be good to go.

	Learn more about alternative methods of installation at 
	<a href="https://github.com/elastic/curator/wiki/Installation">https://github.com/elastic/curator/wiki/Installation</a>. 
	 
<h2>What&apos;s new?</h2>
	I am so glad you asked!
<ul>
	<li>Updated API</li>
	<li>Command-line based on <a href="http://click.pocoo.org/3/">Click</a></li>
	<li>Pipelined index &amp; snapshot selection</li>
	<li>Simultaneous operations (where allowed)</li>
	<li>Improved Python package</li>
	<li>New commands</li>
	<li>Improved testing</li>
	<li>Python3 support restored</li>
</ul>
	 
	Let&apos;s take a brief look at each of these.
<h2>Updated API</h2>
	In Curator 2, the Curator API did index (or snapshot) selection in many of the methods. This required a staggering amount of command-line options. It also required fairly complex methods. Now with Curator 3, index selection and filtering is handled before any command method--close, delete, optimize,&#xA0;etc.&#xA0;--is called. The API methods expect to have the list of indices to act on sent to them as a parameter. As of Curator 3.0.1, a build_filter method has been added to assist in building index filters, as with the command-line.
	Learn more at 
	<a href="http://curator.readthedocs.org">http://curator.readthedocs.org</a>.
<h2>Command-line based on Click</h2>
	<a href="http://click.pocoo.org/3/">Click</a> is an amazing tool to help build sophisticated command-line interfaces. Using Click also allows me to do full integration testing of the software as though I were issuing commands at a command-line.

	The big change for the command-line in Curator 3 is the additional sub-commands 
	indices and snapshots, for index and snapshot selection, respectively.

	To see how this works, you can use the <code>--help</code> flag at each sub-command:
<pre class="prettyprint lang-js"> curator --help 
 curator show --help 
 curator show indices --help </pre>
	Learn more in the wiki, in 
	<a href="https://github.com/elastic/curator/wiki/index-selection">index selection</a> and&#xA0;<a href="https://github.com/elastic/curator/wiki/snapshot-selection">snapshot selection</a>.
<h2>Pipelined index and snapshot selection</h2>
	To continue where the last section left off, the new index selection parameters can work together in ways that Curator 2 could not touch.

	You can now specify a line like:<pre class="prettyprint lang-js"> curator delete indices --older-than 30 --newer-than 60 --time-unit days \ 
 --timestring &apos;%Y.%m.%d&apos; --prefix logs --suffix prod \ 
 --exclude logs-2015.02.01-prod --exclude 2015.01.31 \ 
 --index logs-2015.02.01-dev </pre>
	This line will delete all indices older than 30 days, but newer than 60 days, with logs as a prefix and prod&#xA0;as a suffix, exclude the logs indicated by the patterns given, and force-include <code>logs-2015.02.01-dev</code>.

	Simply put, Curator 2 could not achieve this level of customizability.

	Curator 3 also comes with an <code>--all-indices</code> option to allow you to select all indices. However, if you select the delete command with <code>--all-indices</code>, the Kibana indices of .kibana, kibana-int, and .kibana-marvel will be auto-pruned. If you wish for Curator to delete one or more of these indices,&#xA0;you would have to manually include them with the <code style="line-height: 1.6em;">--index</code> flag. The <code style="line-height: 1.6em;">--index</code> flag bypasses all other filtering to add the specified index to the list to be operated on.
	In fact, the <code>--index</code> flag, if used by itself, will allow you to act on specified indices directly:
<pre class="prettyprint lang-js"> curator delete indices --index index1 --index index2 --index index3 </pre>
	This allows for operations on individual indices, which previously required the use of curl or direct API calls.

	The best part of this change is that you can use Curator on non-time-series indices now! Curator is no longer restricted to operating on time-series indices!

	Learn more in the wiki, in 
	<a href="https://github.com/elastic/curator/wiki/index-selection">index selection</a> and&#xA0;<a href="https://github.com/elastic/curator/wiki/snapshot-selection">snapshot selection</a>.
<h2>Simultaneous operations (where allowed)</h2>
	Curator 2 would only do one action at a time. This could make some operations take far longer than was needful. Curator 3 now only acts one-at-a-time when:
<ul>
	<li>Optimizing indices</li>
	<li>Adding or removing indices from aliases</li>
	<li>Deleting snapshots</li>
	<li>Disabling bloom filters with a <code>--delay</code> specified (Only in Elasticsearch versions older than 1.4)</li>
</ul>
	All other operations are done simultaneously, or in batches. Elasticsearch has a 4K limit for incoming requests (a changeable default). If the list of indices is too big, this could result in an error. To prevent this, Curator will try to automatically compensate for extremely long lists of indices by slicing them&#xA0;into smaller batches. For example, 365 indices named logstash-YYYY.MM.dd takes 4 or 5 batches to complete.<h2>Improved Python package</h2>
	Because everything is segmented into separate modules in the API or CLI, the package is much cleaner. This is makes adding or reviewing code much, much simpler.
<h2>New commands</h2>
	Introducing the 
	<a href="https://github.com/elastic/curator/wiki/Open">open</a> and <a href="https://github.com/elastic/curator/wiki/Replicas">replicas</a> commands. With the improved index selection, you can now re-open closed indices in a batch! You can also change the replica count of indices, even when closed!
<h2>Improved testing</h2>
	While the current tests do not necessarily catch every conceivable test or condition, a lot of effort was made to get complete code coverage--that is to say, every line of code is evaluated at least once. This has already caught several potential bugs before they could make it into a released version! The&#xA0;code coverage is currently at 99%! This includes command-line simulating integration testing! For the first time in Curator&apos;s history, the CLI is being tested, and this is a great beginning.
	A separate blog post will be following to discuss this in further detail.
<h2>Python3 support restored</h2>
	Somewhere around version 2.1, Curator stopped working with Python3. I apologize for this, if it affected you. I made it a top priority to make Curator 3 work with Python3. All tests pass with 99% code coverage in both Python2 and Python3!
<h2>Conclusion</h2>
	Curator 3 is a leap forward in performance, usability, and reliability. I would love to have your feedback, positive or negative. As always, if you come across any issues, do not hesitate to raise an issue on 
	<a href="https://github.com/elastic/curator/issues">GitHub</a>.

	Happy Curating!
</body></html>

Curator 3.0 Released

<html><head></head><body>Today, just in time for the holidays, we are happy to announce the release of Elasticsearch 0.90.9, which is based on Lucene 4.6. This is the current stable release in the 0.90 series. You can download it <a href="http://www.elasticsearch.org/downloads/0-90-9/">here</a>.Unfortunately, 0.90.8 had a bug in the <a href="http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/cluster-health.html"><code>cluster-health</code></a> API which might report too few shards if one of your indices is status `red`. While this doesn&apos;t put your data at risk, it might cause sysadmins to suffer a moment of extreme panic.While we were about it, we found a bug in the <a href="http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-terms-filter.html#_terms_lookup_mechanism">terms lookup filter</a> which could, in rare cases, cause search requests to hang. And lastly, a bug was introduced in 0.90.8 which prevented index templates from being loaded when placed under the config directory.Apologies for the quick re-release, but we hope it makes your holiday more festive and relaxed. Thanks to those who reported these bugs so promptly. Please <a href="http://www.elasticsearch.org/downloads/0-90-9/">download 0.90.9</a> and let us know what you think.</body></html>

Elasticsearch 0.90.9 released

<html><head></head><body>The Elasticsearch dev team is pleased to announce the release of&#xA0;Elasticsearch 0.90.3, which is based on Lucene 4.4. You can download it&#xA0;<a title="0.90.3" href="http://www.elasticsearch.org/downloads/0-90-3/">here</a>.Noticeable changes in 0.90.3 include performance improvements to the has_child query, and improved cluster stability and handling of large cluster states (many indices/shards/aliases). A full list of changes can be found <a href="https://github.com/elasticsearch/elasticsearch/issues?labels=v0.90.3&amp;sort=created&amp;direction=desc&amp;state=closed&amp;page=1">here</a>.An exciting new experimental feature that we back ported to 0.90.3 is a new type of suggester called <a href="https://github.com/elasticsearch/elasticsearch/issues/3376">completion</a>. This is a real-time, prefix oriented suggestion that supports updates in real time while also maintaining an efficient (speed and resource) data structure to provide extreme low latency suggestions cases, specifically as-you-type ones. A blog post describing this exciting new features with actual use cases will be posted shortly.Please <a title="0.90.3" href="http://www.elasticsearch.org/downloads/0-90-3/">download and use 0.90.3</a>, and let us know what you think.Note: If you use the analysis, mapper attachments or cloud plugins, you&apos;ll need to update to the newest version of the plugin to be compatible with the new 0.90.3 release.</body></html>

0.90.3 Released

<html><head></head><body>Today we make available the first update to Kibana 4 incorporating a number of important usability and stability fixes.Grab the new release here: <a href="http://www.elasticsearch.org/overview/kibana/installation/" title="Kibana 4 Download">Download Kibana 4.0.1</a> 
For more information on Kibana 4 see the original release blog post: <a href="http://www.elasticsearch.org/blog/kibana-4-literally/" title="Kibana 4 release blog post">Kibana 4. Literally.</a><h3>Upgrades and Polling for Elasticsearch</h3>Kibana 4.0.1 improves on the upgrade process in 2 ways in an attempt to make both upgrades to Kibana and to Elasticsearch more seamless. <ul>
<li>Upgrades from 4.0.0-RC1 and above to 4.0.1 apply cleanly and automatically, without the need to run a migration script. We expect this to be the case into the future</li>
<li>Kibana 4 performs the upgrade on startup, thus Elasticsearch must be available. In Kibana 4.0.0 the server startup would fail if Elasticsearch had not fully started. In Kibana 4.0.1 the server will poll for Elasticsearch until it is ready.</li>
</ul><h3>bug fixes</h3>In addition to the two big items above we also fixed a variety of other issues, including: <ul>
<li>Fixed the momentary flash of the title bar in iframed charts and dashboards</li>
<li>Fixed the Windows-specific kibana.bat to work with paths that have spaces</li>
<li>Fixed a bug in which pie chart slice color could become disconnected from the legend</li>
<li>Removed the doc_value warning when dealing with analyzed strings as analyzed strings can not have doc_values.</li>
<li>Removed the undocumented and unsupported HOST and PORT environmental variables that were causing issues when telling Kibana to bind to some other interface or port.</li>
<li>Fixed links in Discover to documents with a _id containing slashes.</li>
<li>Controller initialization will now properly abort when dependancies are not met.</li>
</ul></body></html>

Kibana 4.0.1 Released

Kibana 4 & Civic Hacking: Investigating Campaign Contributions

<html><head></head><body>It&apos;s been a while since we released the <a href="http://www.elasticsearch.org/blog/nest-elasticsearch-1-0-ga/">1.0 GA version</a> of the NEST &amp; .NET clients for Elasticsearch. In the meantime, we&apos;ve done two <a href="https://github.com/elasticsearch/elasticsearch-net/releases/tag/1.0.1">minor</a> <a href="https://github.com/elasticsearch/elasticsearch-net/releases/tag/1.0.2">bugfix</a> releases, as most of our efforts have focused on getting <code>1.1</code> out the door.And, here you have it - all that&apos;s new in NEST and Elasticsearch.NET.<h2>Map all the APIs!</h2>We are very pleased to announce that we now support all the API&apos;s that are exposed in Elasticsearch 1.3.2!<ul>
<li><a href="http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-percolate.html#_multi_percolate_api">multi percolate api</a></li>
<li><a href="http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/indices-aliases.html">alias exist API</a></li>
<li><a href="http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/indices-types-exists.html#indices-types-exists">type exist API</a></li>
<li><a href="http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-explain.html#search-explain">explain API</a></li>
<li><a href="http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-snapshots.html#_snapshot_status">snapshot status API</a></li>
<li><a href="http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/indices-recovery.html#indices-recovery">indices recovery API</a></li>
<li><a href="http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/cluster-pending.html#cluster-pending">cluster pending tasks API</a></li>
<li><a href="http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/cluster-stats.html#cluster-stats">cluster stats API</a></li>
<li><a href="http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/cluster-nodes-shutdown.html#cluster-nodes-shutdown">node shutdown API</a></li>
<li><a href="http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/indices-templates.html">template exist API</a></li>
<li><a href="http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/cluster-nodes-hot-threads.html#cluster-nodes-hot-threads">nodes hot_threads API</a></li>
<li><a href="http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-shards.html#search-shards">search shards API</a></li>
<li><a href="http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-snapshots.html#_repositories">get repository API</a></li>
<li>ping API</li>
<li><a href="https://github.com/elasticsearch/elasticsearch-net/pull/933">put alias API and delete alias API endpoints </a>&#xA0;(technically both of these were available through the <code>_aliases</code> API already but we added them in for completeness&apos; sake)</li>
</ul>And NEST now also supports all of the <a href="http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/cat.html">_cat API&apos;s</a>.The only APIs still missing from NEST are the Search Template related APIs. While we are pretty far in spiking a nice way to support mustache templates using C#, we decided to let it bake a while longer instead of rushing it into this 1.1 release.We&apos;ve also exposed the internal <code>DoRequest()</code> on the NEST and Elasticsearch.NET clients so that you can hit any URL while still taking advantage of the cluster failover and built-in response handling. This new feature will come in handy when calling Elasticsearch plugin APIs.<h2>Map all the aggregations!</h2>We&apos;ve added support for the following aggregations:<ul>
<li><a href="http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-aggregations-metrics-percentile-rank-aggregation.html#search-aggregations-metrics-percentile-rank-aggregation">percentile ranks aggregation</a></li>
<li><a href="http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-aggregations-metrics-geobounds-aggregation.html">geo bounds aggregation</a></li>
<li><a href="http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-aggregations-metrics-top-hits-aggregation.html#search-aggregations-metrics-top-hits-aggregation">top hits aggregation</a></li>
<li><a href="http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-aggregations-bucket-reverse-nested-aggregation.html">reverse nested aggregation</a></li>
</ul><h1>Loads of miscellaneous improvements</h1>We&apos;ve also improved our API coverage for existing APIs in tons of places and fixed several bugs. Please have a look at our complete changelog <a href="https://github.com/elasticsearch/elasticsearch-net/releases/tag/1.1.0">here</a>.We have to give a special shoutout to <a href="https://twitter.com/PForsgard">Patric Forsgard</a> who sent a total of 8 pull requests and improved our multifield mapping API in a backwards compatible fashion. Thanks to Patric&apos;s work, the multifield mapping API more closely resembles the new <code>Elasticsearch 1.0</code> way of mapping <code>multi_field</code> properties.<h1>Future plans</h1>While we are very pleased to release 1.1, we can not sit still while our colleagues working on Elasticsearch are moving at such an incredible pace. The list of <a href="https://github.com/elasticsearch/elasticsearch/issues?q=label%3Av1.4.0.Beta1+">1.4 features</a> will be our main focus for the 1.2 release. We&apos;ll also be going over all the existing APIs in a very rigorous manner to see if there are features that have been updated that we&apos;ve missed.As always please don&apos;t hesitate to submit your questions/suggestions/issues to either <a href="https://github.com/elasticsearch/elasticsearch-net/issues">GitHub</a> or <a href="http://stackoverflow.com/questions/tagged/nest">stackoverflow under the &apos;nest&apos; tag</a>.</body></html>

NEST and Elasticsearch.NET 1.1 released!

curly-braces-elasticsearch-elastic-json-thumbnail.png

Elasticsearch for Apache Hadoop 2.2.0-m1 and 2.1.1

Elastic 8.11 includes the GA of the Elastic Learned Sparse EncodeR (ELSER), tech preview of ES|QL, Elastic AI Assistant support for Amazon Bedrock, and Elasticsearch Relevance Engine (ESRE) improvements in vector search.

What's new?

Start today

Generative AI

8.11-header.png

Elastic 8.11: Preview of ES|QL, GA of Elastic Learned Sparse EncodeR

Elastic is the best developer platform for building AI search and generative AI apps with expanded ecosystem partners, full document vector search, a proprietary ML model for semantic search now in GA, and a simple API-first developer experience. 

All in on Apache Lucene

Second to none in semantic search and RAG

Choice and flexibility across the ecosystem

A simple developer experience

Elastic’s innovative investments to support an open ecosystem and a simpler developer experience

Search

elastic-de-135742-blogheader-pav_V1.png

Paving the way for modern search workflows and generative AI apps

Explore the integration of Elasticsearch's search relevance capability with ChatGPT's question-answering capability to enhance your domain-specific knowledge base. Learn how to harness ChatGPT to enrich your information repository like never before!

elasticdocs gpt list the steps free trial

blog-elastic-elasticdocs-gpt-list-the-steps-free-trial.png

What is ChatGPT?

Elasticsearch — you know, for search!

Build Generative AI Search Engines and Applications

A search toolkit for the AI era

How to use ChatGPT with Elasticsearch

image_(7).png

Technical setup

Elastic Cloud setup

blog-elastic-start-free-trial.png

blog-elastic-create-first-deployment.png

blog-elastic-manage-this-deployment.png

blog-elastic-deployments-edit-monitoring.png

blog-elastic-machine-learning-instances.png

blog-elastic-save-equivalent-api-request.png

blog-elastic-cancel-confirm.png

blog-elastic-change-summary.png

blog-elastic-save-deployment-credentials.png

blog-elastic-applications-hardware-profile.png

Eland

blog-elastic-open-in-colab.png

blog-elastic-hugging-face.png

Elasticsearch Index and Web Crawler

blog-elastic-create-an-elasticsearch-index.png

blog-elastic-select-an-ingestion-method.png

blog-elastic-machine-learning-inference-pipelines.png

blog-elastic-configure-add-new-pipeline.png

blog-elastic-available-indices.png

blog-elastic-crawl-rules.png

blog-elastic-search-engines-crawl.png

blog-elastic-crawling.png

Connecting with OpenAI API

blog-elastic-api-key-generated.png

Python backend setup

Clone or download the python program

Sample chat responses

blog-elastic-elasticdocs-gpt-show-api-call.png

blog-elastic-elasticdocs-gpt-how-add-new-integration.png

blog-elastic-elasticdocs-gpt-show-build-boat.png

Elasticsearch’s robust retrieval + the power of ChatGPT

Stack

Cloud

illustration-elevate-our-process-1680x980.png

ChatGPT and Elasticsearch: OpenAI meets private data

Elastic Search 8.13 delivers speed improvements for vector search with Apache Lucene 9.10. LTR brings the power of reranking. Bringing in data through connectors is now simpler with improved synchronization options and an expanded connector catalog.

Elasticsearch Platform

ELK Stack

Elastic Cloud

Observability

Security

Search

By industry

By solution

Customer spotlight

Developers

Connect

Learn

Help

See what's happening at Elastic

Thank you!

Follow us

About us

Join us

Press

Partners

Trust & Security

Investor relations

EXCELLENCE AWARDS