Heartbeat has a new high-performance scheduler written in Go. In this article you'll learn patterns and techniques for writing high performance Go code.

<p>Curious about how to write more idiomatic concurrent code in Go? It’s not always easy or intuitive, even if you’ve done lots of concurrent programming in other languages. I’ve been lucky to have worked in a <a href="https://github.com/elastic/beats">well-written code base</a>, and had the expert advice of Beats core area lead <a href="https://github.com/urso">Steffen Siering</a>&nbsp;along the way.&nbsp;
</p>
<p>In this post I’ll walk you through how we implemented a <a href="https://github.com/elastic/beats/pull/14890">new scheduler</a>&nbsp;for <a href="https://discuss.elastic.co/c/beats/heartbeat">Heartbeat</a>&nbsp;that is part of our upcoming 7.6.0 release. I’ll talk about the high level concurrency strategy for implementing this scheduler, and some of the challenges faced therein. This post assumes a general familiarity with <a href="https://en.wikipedia.org/wiki/Concurrent_computing">concurrent programming</a>, and focuses on the high level approach to concurrency within our scheduler.
</p>
<p>If you don’t already know, Heartbeat is the agent behind the <a href="https://www.elastic.co/products/uptime-monitoring">Uptime app</a>. It regularly pings a site via ICMP, TCP, or HTTP on a schedule to determine whether it’s up or down, and also reports some stats on its performance. The Heartbeat scheduler is in charge of dispatching those tasks at a predetermined interval. So our scheduler performs a job much like cron or <code>java.util.Timer</code>.
</p>
<p>For Heartbeat’s purposes we need a scheduler that:
</p>
<ul>
	<li>Is reliable (our old one had some issues that surfaced in high concurrency)</li>
	<li>Is fast and memory efficient</li>
	<li>Supports both the addition and removal of jobs without stopping the scheduler</li>
	<li>Manages spawned subtasks (essentially continuations) from multi-part jobs</li>
	<li>Supports limiting the maximum number of running sub-tasks</li>
</ul>
<h2>A naive initial attempt
</h2>
<p>My initial attempt to refactor the scheduler was going to be so easy to write. I had the bright idea to lean on the lightweightness of goroutines and take an approach that while only seemingly naive, would be&nbsp;beautifully minimal and leverage the efficiency of Golang’s goroutine scheduler and <code>time.Timer</code>. I was very wrong here.
</p>
<p>In this scheme, for each scheduled task we’d launch a goroutine that immediately blocked on a <code>time.Timer</code>, looping after each task invocation. This meant that for N defined jobs, we’d always have N goroutines, mostly in a blocked state waiting for their timers to trigger. My thinking was that goroutines are super-lightweight, and <code>time.Timer</code> is efficient. So this should just work, would be easy to implement, would check for correctness, and would be the sort of thing that you could get working in some sort of rough shape in an afternoon. Well, it turns out only the last part was correct. After building a PoC of this it turned out that goroutines are not as light as I’d hoped, and neither is the <code>time.Timer</code> type.
</p>
<p>This naive approach used much more memory (at least 20% more) than the old timer, ran slower, and used more CPU in intensive use cases. I didn’t keep exact stats because the results were clearly terrible, and it was obvious that we’d need to take a more robust approach. My hope that lightweight Go internals would let me write dead simple code was completely off.
</p>
<h2>A more robust approach
</h2>
<p>After trying, and failing to find a silver bullet to fix the previous approach, I realized I needed to start over. It became apparent that we had to manage the resources more explicitly, and try to minimize <code>time.Timer</code> instances and goroutines. The standard approach to this is to use a priority queue to keep track of which timer is executing next. This is a classic approach to this problem used in other timer libraries. Implementing it in a thread-safe way was quite a bit more effort. After a lot more work than my naive approach, we wound up with something that performed significantly better than the old scheduler in terms of both CPU and memory utilization.
</p>
<p>The new implementation combines:
</p>
<ul>
	<li>A priority queue implemented via a heap, storing of functions for the dispatch of tasks.&nbsp;</li>
	<li>Go’s <a href="https://godoc.org/golang.org/x/sync/semaphore">WeightedSemaphore</a>&nbsp;for limiting concurrent task execution.</li>
	<li>Go’s <a href="https://golang.org/pkg/sync/#WaitGroup">WaitGroup</a>&nbsp;to wait for all of a job’s sub-tasks to return.</li>
	<li>Go’s atomic <a href="https://golang.org/pkg/sync/atomic/#CompareAndSwapInt32">CompareAndSwap</a>&nbsp;functions to transition between states.</li>
	<li>Go’s <a href="https://golang.org/pkg/context/">Context</a>&nbsp;to provide an easy method of terminating and ongoing tasks from the root of the code.</li>
</ul>
<h2>General structure
</h2>
<p>Our <a href="https://github.com/elastic/beats/tree/master/heartbeat/scheduler">scheduler code</a>&nbsp;is split into several packages:
</p>
<ul>
	<li>scheduler contains the high level scheduler which accepts new jobs, and handles their removal. It also contains the high level logic around requeuing jobs, executing their sub-tasks, and limiting concurrency.</li>
	<li>scheduler/schedule represents the definition of a schedule, and offers a <code>Next(now) time.Time</code> method, that returns the next scheduled execution based on a given now value.</li>
	<li>scheduler/timerqueue is a thread-safe queue that lets you schedule tasks for a single execution making efficient use of timers.</li>
</ul>
<h2>Minimizing timer instances with a priority queue
</h2>
<p>A key part of keeping things efficient is minimizing the number of running <code>time.Timer</code> instances. We only really need one timer, activating when the next scheduled task is ready, and then a check to see how many tasks are due to be executed. We only need to reset this <code>time.Timer</code> if a new task is scheduled that’s due to be executed before any other tasks. So, what we need is a way of tracking our tasks in order of execution.
</p>
<p>The perfect data structure for this is a <em>priority queue</em>, or heap. You can find our implementation in Heartbeat’s <a href="https://github.com/elastic/beats/blob/master/heartbeat/scheduler/timerqueue/heap.go">timerqueue/heap.go</a>. With a heap, we get O(log n) insertion and removal of elements. O(1) ‘peek’ing at the highest priority item, and O(log n) ‘pop’ing of the highest priority removal (peek + delete).
</p>
<p>This means every time a new item is scheduled, adding it is a matter of:
</p>
<ol>
	<li>Peeking to see if it is scheduled to run before the current highest priority item O(1)</li>
	<li>Pushing it onto the queue O(log n)</li>
</ol>
<p>Go comes with a <a href="https://golang.org/pkg/container/heap/">heap implementation</a>, but Go’s heap implementation is not thread-safe. The <a href="https://github.com/elastic/beats/blob/master/heartbeat/scheduler/timerqueue/queue.go">TimerQueue</a>&nbsp;type we created provides thread-safe methods built on top of Go’s heap operations. We spawn a background goroutine that listens for events that mutate the heap, thus serializing access to the underlying data structure. If you’re familiar with the <em>active object</em> pattern, this should ring a bell.
</p>
<h2>Running recursive tasks using weighted semaphores and wait groups
</h2>
<p>When Heartbeat executes a job, there may be more than one sub-task invoked as a result. For instance, if <code>mode:all</code> is set, Heartbeat will ping each IP returned from the DNS lookup as a separate sub-task. Heartbeat supports limiting the number of running sub-tasks (not jobs) via the <code>scheduler.limit</code> option. The core logic around this can be found in <a href="https://github.com/elastic/beats/blob/master/heartbeat/scheduler/scheduler.go#L242">scheduler.runRecursiveTask</a>.
</p>
<p>Let’s examine the way Heartbeat limits task execution here. You may be wondering why we limit tasks, not jobs. The reason is that we want users to be able to limit the number of file descriptors in use at one time. One task will use at most one file descriptor. This lets users plan around OS file descriptor limits with more predictability. Golang’s <a href="https://godoc.org/golang.org/x/sync/semaphore">WeightedSemaphore</a>&nbsp;is a great fit for our needs here. It lets you define a quantity of some resource, then reserve some portion of that capacity. If a goroutine attempts to grab more resources than are available it blocks, waiting for that resource to be released. In our case we grab one unit per sub-task.
</p>
<p>We use Go’s <a href="https://golang.org/pkg/sync/#WaitGroup">WaitGroup</a>&nbsp;to determine when all of a job’s sub-tasks are done so we can do some post-processing. <code>WaitGroup</code> lets you block a goroutine until a counter tracking some number of inflight tasks reaches zero. One nice property of <code>WaitGroup</code> is that if you’re kicking off N tasks, you can just increment it by N, instead of invoking <code>add</code> N times.
</p>
<h2>Using atomic operations to handle state transitions
</h2>
<p>The scheduler has a simple life cycle defined by a simple state machine. It can be in either a <code>preRunning</code>, <code>running</code>, or <code>stopped state</code>, with no backwards transitions. To manage these transitions, we use atomic compare and set (CAS) operations, which are wrapped in Beats’ utility atomic&nbsp;package. We use this utility package to ease some of the awkwardness of using the Go <a href="https://golang.org/pkg/sync/atomic/">atomic</a>&nbsp;package, which is a bit cumbersome.
</p>
<p>The nice thing about using CAS for a state machine is that you can ensure that a transition happens exactly once. See <a href="https://github.com/elastic/beats/blob/master/heartbeat/scheduler/scheduler.go#L120">Scheduler.Start</a>&nbsp;for an example of how we use it to start the scheduler at most once, no matter how many times <code>Start()</code> is invoked.
</p>
<h2>Using context to shut down spawned tasks
</h2>
<p>The scheduler controls the lifecycle of most Heartbeat operations. As a result, it’s responsible for shutting down all user defined monitors when asked. There’s no way to say ‘shut down any goroutines you may have spawned’ in Go. You have to let those goroutines gently know it’s time to shut down.
</p>
<p>An idiomatic way to do this is by passing Go’s <a href="https://golang.org/pkg/context/">context.Context</a> throughout your code. By passing this type throughout the scheduling code, we can easily abort the entire tree of downstream processes by cancelling the top level context. The next time a spawned Go process does a select, it can attempt a channel read off <code>(Context).Done()</code> and stop processing immediately. This does require you to always check <code>(Context).Done()</code> any time you may block in a select, though. It’s a little cumbersome, and can potentially be tricky to integrate with non-channel based concurrency like mutexes and weighted semaphores, but it’s a great way to write code that can respond to shutdown requests correctly. Additionally, network/io libraries usually take a context, meaning that our shutdown can be correctly handled outside of code we’ve directly written.
</p>
<h2>Wrapping things up
</h2>
<p>I hope this has been a useful post in terms of looking at the high level concurrency strategy for go code. If you have any questions, comments, or feedback drop us a line on the <a href="https://discuss.elastic.co/c/beats/heartbeat">Heartbeat topic on Discuss</a>. And if you’d like to see Heartbeat in action, check it out on <a href="https://demo.elastic.co/app/uptime">demo.elastic.co</a>.
</p>

blog-train-track-junction-banner.jpg

blog-train-track-junction-thumb.jpg

A tour of Go concurrency patterns via the new Heartbeat scheduler

Introducing Lookslike — a new open source Go testing/schema library to help you with all your data validation needs.

testing-shapes-blog-banner.jpg

testing-shapes-blog-thumb.jpg

Testing data shapes with go-lookslike

We're excited to share the newest Logstash released with you. Meet version 6.3.

<html><head></head><body><p>Today is a big day -- we&#x2019;re releasing the first version of Logstash since we&#x2019;ve opened our X-Pack code. This is meaningful for a number of reasons which we&#x2019;ve covered in detail in <a href="/products/x-pack/open">our opening X-Pack announcement</a>.
</p><p>This release also brings a number of significant new features and performance improvements. We&#x2019;ve improved our experimental Java execution engine, boosted performance through code refactors in various parts of Logstash, and made communicating across pipelines easier and more efficient. On the plugins&apos; side, we&apos;ve extended the S3 and SQS plugins to support custom endpoints and regions. Read on for the a discussion of the highlights in this release. You can, of course, find the full list of changes in the <a href="/guide/en/logstash/current/logstash-6-3-0.html">release notes</a>.
</p><h3>Read This if You Use Persistent Queues</h3><p>Logstash 6.3.0 contains significant bug fixes for our Persistent Queue (PQ) feature. Unfortunately, we were forced to change the on-disk serialization format in 6.3.0. This means that if you currently use the PQ, you must drain or delete the queue on your current version before upgrading. We have detailed instructions on this process on our <a href="/guide/en/logstash/current/upgrading-logstash-pqs.html">Upgrading Persistent Queues documentation</a>.
</p><h3>Inter-pipeline Communication Just got a LOT Easier (Beta)</h3><p>Many people today use the multiple pipelines feature of Logstash along with an external queuing technology such as Redis to create a multi-staged processing pipeline. While this is an appropriate situation in some scenarios, we&#x2019;ve made this easier and more efficient with our Pipeline-to-Pipeline communications feature. This lets you connect pipelines within a Logstash process simply and with maximum efficiency since everything stays within a single process.
</p><p>It&#x2019;s easy to use since we implemented this through two new built-in inputs and outputs, both called <code>pipeline</code>. Look below for an example of this feature:
</p><pre class="prettyprint"># config/pipelines.yml
- pipeline.id: upstream
  config.string: input { stdin {} } output { pipeline { send_to =&gt; [myVirtualAddress] } }
- pipeline.id: downstream
  config.string: input { pipeline { address =&gt; myVirtualAddress } }
</pre><p>You can read about this feature in detail on our<a href="/guide/en/logstash/6.3/pipeline-to-pipeline.html">&#xA0;Pipeline to Pipeline documentation page</a>. This feature is currently in Beta.
</p><h3>Help Logstash by Test Driving Our Experimental Java Execution Feature!</h3><p>We&#x2019;ve made huge strides with our Java pipeline execution engine. With 6.3.0 we&#x2019;ve been able to improve pipeline compilation times significantly for larger configs and improve the runtime performance of many configurations. This represents a major refactoring of a core piece of Logstash and we&#x2019;re not ready to turn it on by default just yet. We&#x2019;d love more feedback on it and, if you find any issues, <a href="https://github.com/elastic/logstash/issues/new">bug reports</a>. Our current plan is to make this new engine the default in 6.4.0 so long as no significant outstanding issues are found. You can test this feature by enabling the <code>--experimental-java-execution</code> flag in 6.3.0. We&#x2019;ve written up <a href="/blog/meet-the-new-logstash-java-execution-engine">a detailed breakdown of performance differences in 6.3.0</a> too.
</p><h1></h1><h3></h3><h2></h2><h3>Support for custom endpoints and regions in S3 and SQS plugins</h3><p>For a long time, Logstash has had the ability to read from and send data to both S3 and SQS AWS Services. With only AWS in mind, these plugins connect implicitly to AWS and only allow selecting a region from a hard-coded list.&#xA0;Now, to support on premise deployments of the AWS cloud services and other compatible services, we&apos;ve added the ability to customize both <a href="https://www.elastic.co/guide/en/logstash/6.3/plugins-inputs-s3.html#plugins-inputs-s3-endpoint">the endpoint</a> and <a href="https://www.elastic.co/guide/en/logstash/6.3/plugins-inputs-s3.html#plugins-inputs-s3-region">the region</a> for the S3 and SQS input and output plugins.</p><h4>Other plugin improvements</h4><ul><li><strong>Exec Input:&#xA0;</strong>support periodic executing through <a href="https://www.elastic.co/guide/en/logstash/6.3/plugins-inputs-exec.html#plugins-inputs-exec-schedule">the &apos;schedule&apos; option</a></li><li><span style="font-weight: 700;">Redis Input:</span>&#xA0;basic support for <a href="https://www.elastic.co/guide/en/logstash/6.3/plugins-inputs-redis.html#plugins-inputs-redis-ssl">SSL/TLS</a> and <a href="https://www.elastic.co/guide/en/logstash/6.3/plugins-inputs-redis.html#plugins-inputs-redis-path">Unix sockets</a><span></span></li></ul></body></html>

blog-logstash-banner.jpg

Logstash Blog Thumb.jpg

Logstash 6.3.0 Released

<html><head></head><body><p>Hello fellow Logstashers! Here&apos;s what&apos;s new in Logstash for the past week.&#xA0;</p><p><strong>WIP of Inter-Pipeline Communications</strong></p><p>Now that multiple pipelines exist, it&apos;s logical that users would want to connect them. <a href="https://github.com/elastic/logstash/pull/9225">#9225</a> describes the use cases in depth and is a (mostly) functional implementation of this concept. One tricky thing here is dealing with immutability of events in an efficient way. The PR contains a naive copy on write event cloning facility which will need to be improved before we can take this out of WIP. If this is an interesting feature to you, however, your feedback would be welcome!</p><p><strong>Java Plugin API Design Discussions</strong></p><p>We&apos;ve been hard at work on the Java plugin API. There are a couple interesting design discussions and PoCs for those interested. You can track all these discussions, and progress in general, on <a href="https://github.com/elastic/logstash/issues/9215">#9215</a>, which is the meta issue for this project.</p><p><strong>Logstash Centralized Management now featured on Kibana home page</strong></p><p>If you have a Gold or higher license, Logstash&apos;s Centralized Management feature will now show up on the Kibana home page. x-pack-kibana#4753 kibana#17027</p><p><strong>Other stuff</strong></p><ul><li><span>Logstash-filter-dns - 3.0.8:&#xA0;</span>&#xA0;Fix bug where forward lookups would not cache timeout errors.&#xA0;<a href="https://github.com/logstash-plugins/logstash-filter-dns/pull/41">logstash-filter-dns#41</a></li><li><a href="https://github.com/logstash-plugins/logstash-filter-dns/pull/41"></a><span>logstash-input-tcp - 5.0.4:</span>&#xA0;Update Netty/tc-native versions to match those in beats input&#xA0;<a href="https://github.com/logstash-plugins/logstash-input-tcp/pull/113">logstash-input-beats#113</a></li><li>Logstash Monitoring charts have been patched in a recent merge to use&#xA0;the &quot;max&quot; aggregation for each timeseries bucket, instead of &quot;avg&quot;. Using &quot;avg&quot; was trappy because it could mask major deviations, which might be interesting in a Monitoring context.&#xA0;</li><li>The Pipeline Viewer is preparing to get a &quot;config file&quot; layout in addition to the current &quot;graph&quot; layout. To this end, a converter was written to convert the data structure behind the graph layout to one that will enable the config file layout.&#xA0;</li><li>The Pipeline Viewer icons have been updated to use the latest Logstash icons defined in EUI. We were also able to reference the icon assets directly in EUI instead of copying them into monitoring; this helps with an initiative to get SVG assets out of the monitoring plugin.</li></ul><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltb03963a297f0902a/5c306d6ee5b78fdd0bed3657/new-svg-icons.png" data-sys-asset-uid="bltb03963a297f0902a" alt="new-svg-icons.png"></p></body></html>

Logstash Lines for 2018-03-13

<html><head></head><body><p>Hi all! With Elasticon wrapped up we&apos;re back with our weekly updates! Read on for what&apos;s new here!</p><p><strong>New Chart to Visualize PQ Size and Capacity</strong></p><p>We merged in a patch that will provide a chart that shows the total size and capacity across all persistent queues on a Logstash node. Once this is released, you&apos;ll be able to access the chart in the Advanced tab of the Logstash Node Monitoring UI if you have x-pack installed.</p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt13463538f0ab7d25/5c57ee3e22d96be10bcca3b6/lsmonnapqs.png" data-sys-asset-uid="blt13463538f0ab7d25" alt="lsmonnapqs.png"></p><p><strong>Inter-pipeline Comms are Closer</strong></p><p>As announced at Elasticon, inter-pipeline communications are a big deal for the future of Logstash. This feature lets users easily connect pipelines running on the same Logstash creating arbitrary internal topologies. There are two main patterns where this is quite useful:&#xA0;</p><ol><li>&quot;Distributor&quot;: One pipeline runs a beats input , classifies the data, then sends it to a specific Logstash pipeline for that business purpose. This massively DRYs up the configs, while still letting users use a single port for beats.</li><li>&quot;Output Decoupler&quot;: Users often complain that if one output temporarily blocks so does the other. This feature lets users insert a PQ between pipelines, allowing for temporary buffering if an output does disappear.</li></ol><p>With the core PR (<a href="https://github.com/elastic/logstash/pull/9206">logstash#9206</a>)&#xA0; open we now have all the code up we&apos;d need to support this. With that PR and <a href="https://github.com/logstash-plugins/logstash-integration-internal/pull/1">logstash-integration-internal#1</a> inter-pipeline communications work. We do need to put in work to merge this code and ensure that the quality the code is up to par, but we are close.</p><p><strong>Exec Filter can Now Be Run on a Schedule</strong></p><p>With logstash-input-exec#14 the Logstash exec filter can now be run on a schedule! This means Logstash can repeatedly execute a command a-la-cron (with cron syntax) and ingest its output. This really opens up a lot of use cases for Logstash, and is a welcome improvement!</p><p><strong>Fixed Memory Leak in Logstash 5.x (JRuby 1.7)</strong></p><p>In Jruby 1.7 some ruby constructs can leak memory. This affected the grok filter, which was patched here: <a href="https://github.com/logstash-plugins/logstash-filter-grok/pull/136">logstash-filter-grok#136</a> .</p><p><strong>Reduce Memory Usage in Beats Input</strong></p><p>The Beats input can use excessive memory when there are many clients, esp. when pipelining values in the beats client are set to larger values. <a href="https://github.com/logstash-plugins/logstash-input-beats/pull/301">logstash-input-beats#301</a> reduces memory usage mitigating this issue.</p><p><strong>Java API Proposal, Early Stages</strong></p><p>We&apos;re hard at work on our new Java API in <a href="https://github.com/elastic/logstash/pull/9137">logstash#9137</a>. We&apos;re scoping that PR quite small, bypassing some more complex features and dependency management, but it&apos;s an important first step.&#xA0;</p><p><strong>Other Patches and Improvements</strong></p><ul><li>Make &apos;-v&apos; fast on windows: <a href="https://github.com/elastic/logstash/pull/8508">logstash#8508</a></li><li>Eliminate Ruby calls in worker loop: <a href="https://github.com/elastic/logstash/pull/9178">logstash#9178</a></li><li>Config validation fixes for AWS based plugins: <a href="https://github.com/logstash-plugins/logstash-mixin-aws/pull/34">logstash-mixin-aws#34</a></li><li>Fix millisecond handling in JDBC input: <a href="https://github.com/logstash-plugins/logstash-input-jdbc/pull/260">logstash-input-jdbc#260</a></li><li>Fix interpolation when &apos;update&apos; used in ES output: <a href="https://github.com/logstash-plugins/logstash-output-elasticsearch/pull/385">logstash-output-elasticsearch#385</a></li><li>Github input stability fixes: <a href="https://github.com/logstash-plugins/logstash-input-github/pull/1">logstash-input-github#1</a></li><li>Fix missing logs in beats input: <a href="https://github.com/logstash-plugins/logstash-input-beats/pull/247">logstash-input-beats#247</a></li><li>Add sophos UTM def to netflow codec: <a href="https://github.com/logstash-plugins/logstash-codec-netflow/pull/105">logstash-codec-netflow#105</a></li></ul></body></html>

Logstash Lines for 2018-03-06

<html><head></head><body><p>Hello&#xA0;Logstashers! Here&apos;s what&apos;s new in the land of Logstash this past week :)</p><p><strong>Improved KV Filter Delimiter / Value Matching</strong></p><p>The KV Filter now can use regexps to define values and field delimiters. This is especially useful where you might have one or more values separating data, like &apos;k=v:k=v:::k=v&apos; etc. There&apos;s some very impressive regexp-fu to refactor this plugin to support this ability. The two PRs (<a href="https://github.com/logstash-plugins/logstash-filter-kv/pull/55">#55</a>,<a href="https://github.com/logstash-plugins/logstash-filter-kv/pull/54">#54</a>) that implement this make for some very interesting reading!</p><p><strong>Java API Proposal Discussion</strong></p><p>We&apos;re finally making some serious headway on a Java API. We have a proposal (<a href="https://github.com/elastic/logstash/pull/9137">#9137</a>) you can read if you&apos;re interested!&#xA0;This is somewhat early stage, we&apos;re just discussing, but our goal is to define a minimal portion of the LS API, enough to convert some plugins to Java.</p><p><strong>The Elasticsearch Output will now Only Sniff Non-Master Nodes</strong></p><p>We just merged a patch that will prevent master nodes from being sniffed by the Elasticsearch output. This patch brings this output inline with our best practice of not sending data to master nodes. (<a href="https://github.com/logstash-plugins/logstash-output-elasticsearch/pull/694">#694</a>) You can try this out by upgrading your Elasticsearch output plugin today to version 9.0.3.</p><p><strong>Other Fixes</strong></p><ul><li>Fix broken queue stats with PQ: <a href="https://github.com/elastic/logstash/pull/9116">#9116</a></li><li>Meetup Input: Search by group name instead of id:&#xA0;<a href="https://github.com/logstash-plugins/logstash-input-meetup/pull/13">#13</a></li><li>Fix issue where `sql_last_value` could lose precision:&#xA0;<a href="https://github.com/logstash-plugins/logstash-input-jdbc/pull/257">#257</a></li><li>Input SQS: Fix sample IAM Policy:&#xA0;<a href="https://github.com/logstash-plugins/logstash-input-sqs/pull/44">#44</a></li></ul><p><a href="https://github.com/logstash-plugins/logstash-input-sqs/pull/44"></a></p><ul>
</ul></body></html>

Logstash Lines for 2018-02-14

<html><head></head><body><p>The big news this week is of course that 6.2.0 is now live! Check out the <a href="https://www.elastic.co/blog/logstash-6-2-0-released">blog post</a> for details.</p><p>Let&apos;s now move on to non-release news...</p><p><strong>Logstash Versioned Plugin Reference Now Live!</strong></p><p>I&apos;m very excited to announce that we now host the docs for every published version of every Logstash plugin in our new <a href="https://www.elastic.co/guide/en/logstash-versioned-plugins/current/index.html">Versioned Plugin Reference</a>!&#xA0;This is a key feature for users of older Logstashes, since they can now reference older docs.&#xA0;One thing we&apos;re still thinking through is how to link these to specific Logstash versions, since the plugin versions don&apos;t correlate. Currently the best way is to download that version and see which plugin version comes with it by looking in the Gemfile.lock.</p><p><strong>Streaming JSON</strong></p><p>One area of inefficiency and frustration is dealing with JSON. Logstash has a notion of both json_lines and json when it comes to decoding data. We had the idea of seeing if we could switch to Jackson&apos;s streaming mode to deal with JSON instead. This would, for most people, mean they wouldn&apos;t have to worry about that distinction when reading JSON, since a streaming parser would handle either situation.</p><p>This is in an early experimental phase, we&apos;re not entirely sure how this will look for the product, but it&apos;s something we&apos;re excited about, and a <a href="https://github.com/elastic/logstash/pull/9077">POC PR</a> is open.</p><p><strong>Other fixes</strong></p><p><strong>Syslog Input</strong></p><p>Better syslog input compatibility with codecs that change the syslog field. <a href="https://github.com/logstash-plugins/logstash-input-syslog/pull/49">#49</a></p><p>Allow syslog input to use custom groks to be more flexible: <a href="https://github.com/logstash-plugins/logstash-input-syslog/pull/47">#47</a></p><p><strong>Fluentd Codec</strong></p><p>Improve fluent codec to encode message tags in a more standard way: <a href="https://github.com/logstash-plugins/logstash-codec-fluent/pull/21">#21</a></p><p><strong>Mutate Filter</strong></p><p>Handle EU comma style decimals: <a href="https://github.com/logstash-plugins/logstash-filter-mutate/pull/115">#115</a></p></body></html>

Logstash Lines for 2018-02-07

<html><head></head><body><p><strong>Logstash 5.6.7 and 6.1.3 Released</strong></p><p>They&apos;re hot off the press and ready for <a href="https://www.elastic.co/downloads/logstash">downloading</a>.&#xA0;The notable new things here are:</p><p>6.1.3 fixes a bug where single shot pipelines, like the ES input or stdin, might not fully process all incoming data.</p><p>Both of these releases include support for scheduled queries. The plugin docs aren&apos;t yet updated on the site, but you can view them&#xA0;<a href="https://github.com/logstash-plugins/logstash-input-elasticsearch/blob/master/docs/index.asciidoc">in the git repo for the moment</a>.</p><p>There are more items in the <a href="https://www.elastic.co/guide/en/logstash/6.1/logstash-6-1-3.html">release notes for 6.1.3</a> and the <a href="https://www.elastic.co/guide/en/logstash/5.6/logstash-5-6-7.html">release notes for 5.6.7</a>, but I&apos;ll omit them here for brevity</p><p><strong>Filebeat pipelining may cause extra memory usage</strong></p><p>Filebeat now defaults to enabling the pipelining option. This can cause Logstash to buffer more data than normal. There has been one case in the field of this causing an issue, but that involved a very high ratio of Filebeats to Logstashes. We&apos;re currently <a href="https://github.com/logstash-plugins/logstash-input-beats/issues/286">working</a> on improving efficiency on the beats input side.</p><p>A short term workaround is to set pipelining in the Filebeat Logstash output to a lower number than the new default of 5.</p><p><strong>Pipeline Viewer gets timeseries charts in detail drawer</strong></p><p>In 6.2.0 we&apos;re targeting the release of&#xA0;a detail drawer for the Pipeline Viewer. Users could click on a vertex in the pipeline and view the vertex&apos;s details in the drawer. This included scalar metrics about that vertex (e.g. a grok filter vertex).</p><p>In 6.3.0 we are targeting an&#xA0;enhancement of&#xA0;these detail drawer metrics to be timeseries charts. Users can now get a historical picture of how a particular plugin was performing in a pipeline over time.</p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltca0c0b0fbc4fc087/5c30745dad674d236e764986/detail-graph-ss.png" data-sys-asset-uid="bltca0c0b0fbc4fc087" alt="detail-graph-ss.png"><br></p><p><strong>Javafication and Core Cleanups Continue!</strong></p><p>We&apos;ve been able to do a ton of work moving various bits of Ruby code in Logstash core to Java, as well as removing some redundant classes. By moving to Java we&apos;ve: 1.) Improved maintainability 2.) Improved performance.&#xA0;</p><p><strong>Notable Plugin Fixes/Improvements</strong></p><p><a href="https://github.com/logstash-plugins/logstash-filter-grok/pull/133">Fix for thread leak in Grok filter.</a></p><p><a href="https://github.com/logstash-plugins/logstash-filter-mutate/pull/115">EU Decimal comma support for mutate</a>.</p></body></html>

Logstash Lines: Update for January 30, 2018

<html><head></head><body><p>A lot has been going on since our last update! We&apos;re glad to bring a number of new features.
</p><p><strong>X-Pack Pipeline Viewer gets a basic detail drawer (Unreleased)</strong>
</p><p><span></span>This is not yet in the currently released version of X-Pack, but we&apos;re hoping to launch this soon. Users will be able to&#xA0;click on vertices in the Pipeline Viewer to open a detail drawer.&#xA0;
</p><p>The drawer shows (for now) basic details about the vertex. In the future the drawer will show timeseries charts in place of scalar metrics as well as custom metrics for certain plugins (e.g. grok and elasticsearch).
</p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt0bab0e48395d37c2/5bbc71a363ed239936a7d98e/ss1.png" data-sys-asset-uid="blt0bab0e48395d37c2" alt="ss1.png" "=""></p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltfc49171f641dbf02/5bbc719c20bcc78f36235b4b/ss2.png" style="background-color: initial;" data-sys-asset-uid="bltfc49171f641dbf02" alt="ss2.png">&#xA0;<br></p><p><strong>A Variety of Improvements to Core Internals</strong>
</p><p>A variety improvements have been made to core pipeline/queue/execution code. We&apos;ve had a number of PRs to improve the code quality and java-ify the core execution logic in Logstash. We&apos;ve also found some performance improvements in the process. &#xA0;I&apos;d link to the PRs, but there&apos;s really just too many. We&apos;ve made amazing progress on this front. Thanks once again for this effort!
</p><div>
	<p><strong>The Secret Store will Most Likely Land in 6.2.0</strong>
	</p>
	<p>We&apos;ve successfully landed the Secret store in the branch targeting&#xA0;6.2.0<span class="redactor-invisible-space"></span>. This feature lets users store secrets on disk in an encrypted format.&#xA0;<a href="https://github.com/elastic/logstash/pull/8935">[PR]</a>&#xA0;. Docs to come soon.<strong>Logstash on Windows Handles Paths with Spaces</strong>
	</p></div><p>For a while now Logstash hasn&apos;t worked correctly on Windows boxes when inside of a folder containing spaces. This PR fixes that.&#xA0;<a href="https://github.com/elastic/logstash/pull/8931">[PR]</a>&#xA0;
	</p><p><strong>The Elasticsearch Input Now Executes Query on a Schedule</strong>
	</p><p>This has been a long asked for feature. Up till now the ES input could only be used to do a one-time import of data in an ES index. With the new scheduled execution feature it can run on a schedule, periodically pulling in new data. <a href="https://github.com/logstash-plugins/logstash-input-elasticsearch/pull/81">[PR]</a>
	</p><p><strong>Plugin Releases in the past week:</strong>
	</p><p><strong>logstash-output-http v5.2.0:&#xA0;</strong>Added json_batch format. Sends multiple events in a single request. <a href="https://github.com/logstash-plugins/logstash-output-http/pull/78">#78</a>
</p><p><strong>logstash-input-elasticsearch v4.2.0:&#xA0;</strong>Add support for scheduling periodic execution of the query <a href="https://github.com/logstash-plugins/logstash-input-elasticsearch/pull/81">#81</a>
</p><p><strong>logstash-input-gelf v3.1.0:&#xA0;</strong>Add support for listening on TCP socket (before only UDP was available) <a href="https://github.com/logstash-plugins/logstash-input-gelf/pull/59">#59</a>
</p><p><strong>logstash-codec-multiline v3.0.9:&#xA0;</strong>Fixed concurrency issue causing random failures when multiline codec was used together with a multi-threaded input plugin <a href="https://github.com/logstash-plugins/logstash-codec-multiline/pull/60">#60</a>
</p><p>logstash-output-email v4.1.0:&#xA0;
</p><p>Add bcc suport <a href="https://github.com/logstash-plugins/logstash-output-email/pull/55">#55</a> *community contributed by @jstoja*
</p><p>Add mustache templating:<a href="https://github.com/logstash-plugins/logstash-output-email/pull/51">#51</a> *community contributed by @jstoja*
</p></body></html>

Logstash Lines: Update for January 22, 2018

<html><head></head><body><p>In this edition of Logstash Lines we have some great news and new features!</p><p><strong>JDBC Static Filter<br></strong></p><p>We have an awesome new filter, the <a href="https://github.com/logstash-plugins/logstash-filter-jdbc_static/">JDBC static filter</a> . This filter enriches a events with a fast in memory SQL database. That database is seeded with data from a remote database via a JDBC connection. This is useful if there&apos;s a relatively static set of data on a remote DB you&apos;d like to enrich local data with. The local DB can be reloaded periodically via a scheduler.</p><p>Due to some technical issues we haven&apos;t updated the docs yet, but you can find them <a href="https://github.com/logstash-plugins/logstash-filter-jdbc_static/blob/master/docs/index.asciidoc">here</a> for now.</p><p><strong>The Secret Store Moves one Step Closer</strong></p><p>The secret store is a big change to LS, one that we&apos;re targeting for 6.2.0. We&apos;ve almost got it ready to merge into master sans some docs and a couple more tests. It looks like it&apos;ll be close to the feature freeze date in 4 days, but should make it. The secret store will let users encrypt credentials on disk thus helping customers who have a requirement for no plaintext passwords on disk.&#xA0;</p><p><strong>Looking Forward on Core</strong></p><p>We&apos;re also making significant progress in our effort to bring a framework for distributed processing to Logstash. We don&apos;t have much we want to discuss now since a lot of it is ideas stage, but we&apos;ve got exciting stuff cooking that we can&apos;t wait to show you once we&apos;ve gotten a bit further along.&#xA0;</p><p><strong>RabbitMQ Output now Parallel</strong></p><p>We received a note that the RabbitMQ output was quite slow. We then discovered that it was completely serial, no matter how many workers you had. This has now been <a href="https://github.com/logstash-plugins/logstash-output-rabbitmq/pull/71">patched</a> . Update your RabbitMQ output to the latest version to get this performance boost.</p><p><strong>Beats Input NPE Fix</strong></p><p>The beats input could crash with an NPE under load before a fix we released in 5.0.6. We recommend users to update to the latest version, 5.0.7 whenever possible.&#xA0;</p><p><strong>Other LS Core Changes:</strong></p><p><a href="https://github.com/elastic/logstash/pull/8897">Perf improvements</a>&#xA0;to java exec.</p><p>We now use the multiple pipelines feature by default in RPM/DEB packages&#xA0;<a href="https://github.com/elastic/logstash/pull/8894">via this PR.</a></p><p><strong>Plugin Releases in the past two weeks:</strong></p><p><strong>logstash-output-http v5.1.2</strong></p><p>Add check to avoid hanging pipeline if an empty event array is passed in. #80</p><p><strong>logstash-input-s3 v3.2.0</strong></p><p>Add support for auto-detecting gzip files with .gzip extension, in addition to existing support for *.gz</p><p>Improve performance of gzip decoding by 10x by using Java&apos;s Zlib</p><p><strong>logstash-input-beats v5.0.6</strong></p><p>Re-order Netty pipeline to avoid NullPointerExceptions in KeepAliveHandler when Logstash is under load</p><p>Improve exception logging</p><p>Upgrade to Netty 4.1.18 with tcnative 2.0.7</p><p><strong>logstash-output-rabbitmq v5.1.0</strong></p><p>Use shared concurrency / multiple channels for performance</p><p><strong>logstash-filter-csv v3.0.8</strong></p><p>Added support for tagging empty rows which users can reference to conditionally drop events</p><p>Notable Pull Requests in the past two weeks:</p><p><strong>logstash-input-beats</strong></p><p>Re-order pipeline to avoid NPE in KeepAliveHandler #283</p><p><strong>logstash-input-s3</strong></p><p>use Java gzip libraries for buffered decoding of gzip files #127</p><p><strong>logstash-filter-csv</strong></p><p>Added feature to skip empty rows preventing broken event warnings #60&#xA0; *contribution by Abdul Hussain (@AbdulHaseebHussain)*</p><p><strong>logstash-filter-elasticsearch</strong></p><p>add ability to copy meta/docinfo fields and aggregations into Logstash event #83&#xA0; *contribution by Fabien Baligand (@fbaligand)*</p><p>support copying nested fields #84 *contribution by Sam Woods (@samwoods1) + improvement by our Ry*</p><p><strong>logstash-filter-kubernetes_metadata</strong></p><p>Authorization header fix #5 *contribution by @antalov*</p><p><strong>logstash-filter-mutate</strong></p><p>Capitalize Feature Addition #106&#xA0;</p><p><strong>logstash-output-email</strong></p><p>Add mustache templating for email body #51&#xA0; *contribution by Julien Bordellier (@jstoja)*</p><p>Add bcc parameter #55 *contribution by Julien Bordellier (@jstoja)*</p><p><strong>logstash-output-rabbitmq</strong></p><p>Use shared concurrency for performance / simplicity #71</p></body></html>

Logstash Lines: Update for January 12, 2018

<html><head></head><body><p>We&apos;re back from the holidays with some great new features!</p><p><strong>New Pipelines Overview Page</strong></p><p>In 6.0 we released the Logstash Pipeline Viewer. To get to this view, users would first have to go through the Pipelines view. This view listed the user&apos;s pipelines as cards with each card listing out the pipeline&apos;s versions.<br></p><p>We are targeting 6.2 for a&#xA0;redesigned tPipelines view to not only serve as a navigational tool to the Pipeline Viewer but also to provide users with a summary view of their pipelines&apos; health.</p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltb794386d8243df17/5c3075acb2b2be79676e84b2/pipeline-index-new.png" data-sys-asset-uid="bltb794386d8243df17" alt="pipeline-index-new.png"></p><p>Many thanks to several Elasticians who volunteered their time to help usability test designs for the new Pipelines view.</p><p><strong>Users will be able to specify the pipeline ID from the CLI</strong></p><p>With <a href="https://github.com/elastic/logstash/pull/8868">this PR</a>&#xA0;users will be able to specify the pipeline ID from the CLI from 6.2.0 onward with the new --pipeline.id option. This is useful for those not using the pipelines.yml file.&#xA0;</p><p><strong>Stuff Happens</strong></p><p>More consistent handling of IP addr lookups (Targetting LS 7.0) in TCP input as of&#xA0;<a href="https://github.com/logstash-plugins/logstash-input-tcp/pull/79">logstash-input-tcp#79</a></p><p>Better handling of empty source fields in geoip plugin as of&#xA0;<a href="https://github.com/logstash-plugins/logstash-filter-geoip/pull/134/files">logstash-filter-geoip#134</a></p><p>We <a href="https://github.com/elastic/logstash/pull/8881">now</a> have a tool that can generate a list of LS deps + their licenses for Legal compliance reasons.<br></p></body></html>

Logstash Lines: Update for January 2, 2018

<html><head></head><body><p>Hello Logstashers! We&apos;re glad to present you all the past couple week&apos;s news in convenient digest form! This past week has been mostly about fixes. We have some big picture things we&apos;re working on, but nothing in enough shape to share yet.
</p>
<ul>
	<li><a href="https://github.com/elastic/logstash/issues/8823">6.1.0/6.1.1 regression: JSON with empty keys can cause LS to crash</a>We already have a fix merged in. It didn&apos;t make 6.1.1, we are now targeting 6.1.2 for this fix.</li>
	<li>
	<div><a href="https://github.com/logstash-plugins/logstash-input-beats/pull/278">Fix connection/FD leak in Beats input</a><br>
	</div>
	<div>We found a bug in the Beats input (thanks PH/Rob Bavey!) that caused it to leak file descriptors / TCP connections when under back pressure. This is fixed in LS 6.1.1 . Users of older Logstashes can upgrade the beats input plugin.
	</div></li>
	<li><a href="https://github.com/logstash-plugins/logstash-filter-grok/pull/131">Fixed a bug where LS could crash due to bad interrupt code in grok</a>
	<div>The Logstash grok filter uses thread interrupts. We found a bug where in some cases we might not properly clear interrupts between invocations, causing subsequent code that throws interrupted exceptions to trigger in places where it might not be caught. This is fixed in 6.1.1 . Users of older Logstashes may upgrade their grok filter to the latest version.
	</div></li>
	<li><a href="https://github.com/logstash-plugins/logstash-input-udp/pull/32">Fixed stability / threadsafety problem in the UDP input</a>
	<div>The UDP input shared a codec among threads. Codecs are intentionally not threadsafe, and are supposed to be thread local. The fix in this case was to use separate codec instances for each UDP input worker thread. This has been patched and included in 6.1.1
	</div></li>
	<li>
	<div><a href="https://github.com/elastic/logstash/pull/8836">Fixed inconsistent hashing of config files in some esoteric situations</a>
	</div>This caused a test failure, but almost certainly not any real world ones. Up until this change set we used to traverse our graphs and hash them based on the contents of the graph representation. This is somewhat tricky and inherently fragile since any changes to the graph need to be incorporated into the hash. Since our IR includes the source code that generated that graph we just moved to hashing that, which is faster and more reliable.</li>
	<li>
	<div><a href="https://github.com/logstash-plugins/logstash-output-elasticsearch/pull/714">Correctly log major version in all scenarios in ES output</a>
	</div>
	<div>One of our error messages for the ES output always showed the ES version as null, this fixes that.
	</div></li>
</ul>
<div>
</div>
<div>
</div>
<div style="color: rgb(34, 34, 34); font-family: arial, sans-serif; font-size: 12.8px; background-color: rgb(255, 255, 255);">
</div></body></html>

Logstash Lines: Update for December 19, 2017

Introducing the Logstash Pipeline Viewer — a GUI to get a birds-eye view of your Logstash pipelines and understand their performance.

main_image.png

You know, for visualizing your Logstash pipelines

We're incredibly proud to announce the best Logstash ever with Logstash 6.1.0!

<html><head></head><body><p>Logstash 6.1.0 has launched! We&apos;ve got some great new features to talk about! Read about it here, or just head straight over&#xA0;to our <a href="https://www.elastic.co/downloads/logstash">downloads page</a>&#xA0;and give it a shot! However, you may want to take a minute to read about <a href="https://www.elastic.co/guide/en/logstash/6.0/breaking-changes.html">breaking changes</a>&#xA0;and <a href="https://www.elastic.co/guide/en/logstash/6.1/logstash-6-1-0.html">release notes</a>&#xA0;first. Read on for what&apos;s new in Logstash 6.1.0.
</p><p><strong>File Based Ruby Scripting Support</strong>
</p><p>We&#x2019;re proud to announce a great new way to extend Logstash functionality in 6.1.0. Complex modification of events in Logstash is now much easier due to our new feature, file based Ruby scripting via the Logstash Ruby filter. While the Ruby filter already lets you use custom ruby code to modify events, that code must live inside the Logstash configuration file itself, which doesn&#x2019;t work well for longer pieces of code, and is hard to debug. Additionally, there&#x2019;s never been a good way to reuse or test that code. File based Ruby scripts can take arguments, letting you reuse code within your Logstash configs easily.
</p><p>This new feature lets you write Ruby code in an external file, with tests inline in that same file, and reuse that anywhere you&#x2019;d like. Another nice feature here is that we can generate accurate line numbers in stack traces for exceptions in file based Ruby scripts, making them much easier to debug. The full details for configuring this are available in the <a href="https://www.elastic.co/guide/en/logstash/current/plugins-filters-ruby.html">Ruby filter docs</a>. For a short example see below:
</p><p>To configure the ruby filter with a file use:
</p><pre>filter {
  ruby {
    # Cancel 90% of events
    path =&gt; &quot;/etc/logstash/drop_percentage.rb&quot;
    script_params =&gt; { &quot;percentage&quot; =&gt; 0.9 }
  }
}

</pre><p>The file &apos;drop_percentage.rb&apos; would look like:
</p><pre>def register(params)
  @should_reject = params[&quot;reject&quot;]
end

def filter(event)
  return [] if event.get(&quot;message&quot;) == @should_reject
  event.set(&quot;field_test&quot;, rand(10))
  extra_processing(event)
  [event]
end

def extra_processing(event)
  # ..
end

test &quot;non rejected events&quot; do
  parameters do
    { &quot;reject&quot; =&gt; &quot;hello&quot; }
  end

  in_event { { &quot;message&quot; =&gt; &quot;not hello&quot; } }

  expect(&quot;events to flow through&quot;) do |events|
    events.size == 1
  end

  expect(&quot;events to mutate&quot;) do |events|
    events.first.get(&quot;field_test&quot;).kind_of?(Numeric)
  end
end</pre><p><strong>Experimental New Execution Engine</strong></p><p>Logstash 6.1.0 brings some exciting new changes to the Logstash internals. We&#x2019;ve been working on a full rewrite of the internal execution engine in Logstash. This rewrite moves the core execution logic from JRuby to Java/JVM Bytecode. With this approach we&#x2019;ll be able to pave the way to more performance improvements in the future, as well as the ability to write optimized Logstash plugins in any JVM language.</p><p>This feature is currently disabled by default, and users should note that it is experimental and not yet ready for production. To enable this feature, you&#x2019;ll need to use the &apos;--experimental-java-execution&apos;&#xA0;flag. We encourage users to try this flag out in test and staging environments and report any bugs found. Our hope is to make this the default execution method sometime in the 6.x timeframe.</p></body></html>

blog-logstash-thumb.jpg

Logstash 6.1.0 Released

<html><head></head><body><p>Hello Logstashers! We&apos;re glad to present you all the past couple week&apos;s news in convenient digest form!</p><p><strong>Added Support for Resetting Logging Back to Default<br></strong></p><p>We&apos;ve now <a href="https://github.com/elastic/logstash/pull/8786">added support</a> for resetting logging settings changed via the Logstash web API back to their defaults.&#xA0;</p><p><strong>Size caps for log files + gzip on rollover</strong></p><p>We&apos;ve added some much improvements fixes for Logstash&apos;s log4j logging. We will cap log files at 100MB per file, and gzip files as they&apos;re rolled over automatically. We&apos;re targeting 6.2 with this change.&#xA0;</p><p><strong>HTTP Input Now Returns 429s Sometimes/ Runs faster/ No Longer Uses Excessive File Descriptors</strong></p><p>The Logstash HTTP input previously exhibited poor behavior when the queue was blocked. If a client connected and timed out, LS would not release the connection, but rather, block indefinitely, causing the client to potentially time out. The HTTP protocol doesn&#x2019;t deal well with long running requests. This plugin will now either return a 429 (busy) error when Logstash is backlogged, or it will time out the request.</p><p>If a 429 error is encountered clients should sleep, backing off exponentially with some random jitter, then retry their request.</p><p>This plugin will block if the Logstash queue is blocked and there are available HTTP input threads. This will cause most HTTP clients to time out. Sent events will still be processed in this case. This behavior is not optimal and will be changed in a future release. In the future, this plugin will always return a 429 if the queue is busy, and will not time out in the event of a busy queue.</p><p>While doing this work, we discovered that the HTTP input had some synchronization bottlenecks which were unnecessary. Users might see a nice perf boost on multicore systems.</p><p><strong>LIR Fixes&#xA0; / Experimental Execution Fixes</strong></p><p>One of the big changes in 6.0 is our work on a new execution backend as well as a new IR for our compiler. We&apos;ve seen a number of small discrepancies pop up in the 6.x series that we&apos;ve been ironing out with both the new IR (used by all users) and the new execution engine (will be optional with 6.1). We&apos;ve done this work in a <a href="https://github.com/elastic/logstash/pull/8831">variety</a> <a href="https://github.com/elastic/logstash/pull/8824">of</a> <a href="https://github.com/elastic/logstash/pull/8774">commits</a>.</p><p><strong>Other Fixes</strong></p><ul><li><a href="https://github.com/logstash-plugins/logstash-codec-line/pull/13/files">Fix thread safety of line codec with UDP / other multithreaded codecs:</a></li><li><a href="https://github.com/logstash-plugins/logstash-input-jdbc/pull/252">Fix Connection / Memory leak in JDBC Input</a></li><li><a href="https://github.com/logstash-plugins/logstash-input-irc/pull/24">Multiple bug fixes for the IRC input</a></li><li><a href="https://github.com/logstash-plugins/logstash-input-syslog/pull/43">Fix Syslog shutdown behavior</a></li></ul></body></html>

Logstash Lines: Update for December 12, 2017

We're incredibly proud to announce the best Logstash ever with Logstash 6.0.0!

<html><head></head><body><p>We&#x2019;re&#xA0;glad to announce that Logstash 6.0.0 has launched! Today marks the first day of 6.0&#x2019;s inter-planetary mission of making life easier for systems administrators and engineers. Can&#x2019;t wait another minute to try it? Head over to our <a href="https://www.elastic.co/downloads/logstash">downloads page</a>&#xA0;and give it a shot! However, you may want to take a minute to read about <a href="https://www.elastic.co/guide/en/logstash/6.0/breaking-changes.html">breaking changes</a>&#xA0;and <a href="https://www.elastic.co/guide/en/logstash/6.0/logstash-6-0-0.html">release notes</a>&#xA0;first. Read on for what&apos;s new in Logstash 6.0!</p><h3><span></span>Streamline Processing with Multiple Pipelines</h3><p>One pipeline no longers rules them all. Logstash 6.0 introduces the ability to run multiple pipelines concurrently for different use cases. The pipelines run together in the same instance, but with independent inputs, filters, and outputs to enable users to isolate processing logic per data source. This keeps your pipeline logic focused and concise. Historically, many Logstash users combine multiple use cases into a single pipeline which requires adding complex conditional logic. With multiple pipelines that&#x2019;s no longer necessary; now, you can organize your config more cleanly, and execute your pipeline more efficiently by using dedicated pipelines per use case.</p><p><span id="docs-internal-guid-6fb5cd00-9bd5-5f01-6af3-fe5ada7ab7d6"><img src="https://lh5.googleusercontent.com/KZ-SFO9bj48u2RAc0Q9unQEHO1ILxNxKHUFxueGa4qD1z-MTkIm3QqPIZk3hwfKTZWHuvgH5gqIAI4gExD4Ps3x5F59zdDzQb_LU6Tj7HqRg_0llDkaGctnAwaHeLT9kCPFgcYKD" width="271" height="237"></span><br></p><p>To add to the fun, each pipeline also has its own independent settings and lifecycles which you can tune to match the respective workload profile desired for that data source. For instance, you may want to allocate more pipeline worker threads for a high volume logging pipeline, and throttle back resources for a lower intensity local metrics pipeline. For more details on multiple pipelines, take a peek at the <a href="https://www.elastic.co/guide/en/logstash/6.0/multiple-pipelines.html">documentation</a>&#xA0;and <a href="https://www.elastic.co/blog/logstash-multiple-pipelines">blog post</a>!</p><h3>Manage Pipelines Centrally with the Elastic Stack</h3><p>In the past, managing pipeline configurations was either a manual task, or you would use config management tools like Puppet or Chef to assist with operational automation. With Logstash 6.0, the centralized pipeline management feature now enables you to manage and automatically orchestrate your Logstash deployments directly with the Elastic Stack through the Kibana single pane of glass.&#xA0;</p><p><span id="docs-internal-guid-6fb5cd00-9bd5-a75d-14b5-bbc290fab71f"><img src="https://lh3.googleusercontent.com/EYcpyojtY6NyN0KYc1O0iaxxjUhN-lIBITnjbqHE-abEerFSbp1LdChRnwcbPV7UHRdADWolhjSS-JdXoXBKqEUd75v5S547rICgwA6jtIOc5VCFireOyCYoAhGYMMlFlesH5deO" width="385" height="310"></span><br></p><p>This feature brings a Pipeline Management UI to Kibana, which you can use to create, edit, and delete pipelines. Underneath this UI we use Elasticsearch to store your pipeline configurations. With a few simple settings, your Logstash nodes can be configured to watch for changes on these pipelines, letting you seamlessly push out pipeline changes without additional operational infrastructure beyond the Elastic Stack.</p><p>Centralized pipeline management is available as an X-Pack feature. To learn more, take a look at the <a href="https://www.elastic.co/guide/en/logstash/6.0/logstash-centralized-pipeline-management.html">documentation</a>&#xA0;and the <a href="https://www.elastic.co/blog/logstash-centralized-pipeline-management">blog post</a>!</p><h3>Visualize&#xA0;Pipeline Logic and Performance</h3><p>We&#x2019;re also proud to announce the long awaited <a href="https://www.elastic.co/guide/en/logstash/6.0/logstash-pipeline-viewer.html">Pipeline Viewer</a> in this 6.0 release, which is a welcome addition to the Logstash Monitoring UI. With this new tool you can both visualize your pipeline configuration and troubleshoot performance bottlenecks at the plugin level. The pipeline viewer displays your Logstash pipelines as a DAG (Directed Acyclic Graph). Overlaid on this DAG are relevant performance metrics for individual inputs, filters, and outputs. Potential performance problems are highlighted to let you quickly determine which portions of your pipeline may be bottlenecks.</p><p>Users should note that this is a beta feature and may be subject to change. One known issue is the lack of ability to cleanly render very large pipelines. This is an area of active work, so expect significant improvements to this feature as we improve it further and move it into general availability.</p><p><span id="docs-internal-guid-6fb5cd00-9bd5-e64a-a40c-8d933b4c0e0d"><span style="font-size: 11pt; font-family: Arial; color: rgb(0, 0, 0); background-color: transparent; vertical-align: baseline; white-space: pre-wrap;"><img src="https://lh5.googleusercontent.com/bUi05zoXkD9Izjtkews24YdMhBSSxq8fYihHRbs5TaSW9pnWJdFNho-zJfwaW74oGqGVFUbHE0vJ1Hps0BhlNA-ny5H9nDOdkc2kjjyNrSomx7n_cdLKGb303PwmKzuc7RF-I5Bd" width="624" height="483" style="border-width: initial; border-style: none; transform: rotate(0rad);"></span></span><br></p><h3>A Smooth Path from Ingest Node to Logstash</h3><p>Some users enjoy the convenience of Elasticsearch Ingest Nodes when getting started with the Elastic Stack. However, at a certain level of complexity, Ingest Nodes may&#xA0;not have all the features required to solve your problem&#xA0;and a migration to Logstash may&#xA0;be in order. To ease these transitions we&#x2019;ve created an <a href="https://www.elastic.co/guide/en/logstash/6.0/ingest-converter.html">Ingest pipeline conversion&#xA0;tool</a>&#xA0;that ships with Logstash 6.0. The converter takes an Ingest Node pipeline as an input and spits out the&#xA0;respective Logstash pipeline. See the <a href="https://www.elastic.co/blog/ingest-node-to-logstash-configuration-converter">blog post</a> for more insight!</p><h3>Now With&#xA0;JRuby 9k</h3><p>We&#x2019;ve spent a lot of time moving the Logstash project over to the latest major version of JRuby: JRuby 9000, which has support for modern Ruby syntax and enhanced internals more amenable to optimization. For plugin developers, this means that your old plugins will continue to work in all but the most rare of cases. This upgrade also means that you can use new Ruby features going forward, as well as using Ruby libraries only compatible with JRuby 9k.</p></body></html>

Logstash 6.0.0 GA Released

A new execution engine brings the potential for future performance!

<html><head></head><body><p>Welcome back to The Logstash Lines! In these weekly posts, we&apos;ll share the latest happenings in the world of Logstash and its ecosystem.
</p>
<p><strong>Runtime Pipeline Code Generation via Janino</strong>
</p>
<p>I&apos;m really excited to announce a new&#xA0;patch that just landed in master and will most likely be released in&#xA0;Logstash 6.1<span class="redactor-invisible-space"><span class="redactor-invisible-space"><span class="redactor-invisible-space"><span class="redactor-invisible-space">:</span></span></span></span>&#xA0;native java execution with runtime <a href="https://github.com/elastic/logstash/pull/8500">bytecode generation via Janino</a>. This is all thanks to the amazing work of <a href="https://github.com/original-brownbear">Armin Braun</a>, who&apos;s leveraged his significant expertise to deliver us this patch. You can test it out by using the special <code>--experimental-java-execution</code> flag on the master branch of Logstash.&#xA0;
</p>
<p>Logstash will have two ways of executing in releases with this patch:</p>
<ol>
	<li>The current execution, which is very Jruby specific, crufty, and troublesome in terms of supporting pure java plugins efficiently</li>
	<li>The new pure java implementation which doesn&apos;t have a performance advantage for ruby plugins but should have some nice gains for pure java ones.</li>
</ol>
<p>Our goal is to continue to improve the feature flagged code generation and&#xA0;get feedback from users willing to try it out as an experimental feature, then make it the default execution sometime in 6.x once it&apos;s been tested more in the field. If you&apos;re interested in giving it a shot it&apos;s probably best to wait till it&apos;s out of master and in a proper release behind the aforementioned feature flag&#xA0;as this feature is still under active development.
</p>
<p><strong>Notable Patches in master:</strong>
</p>
<ul>
	<li>Buttoned some bugs for our cloud ID support for 6.0, ensuring a flawless and easy setup for cloud users in 6.0</li>
	<li>Added the Logstash version info to startup.</li>
	<li>Cleaned up a lot of internals related to rubygems</li>
	<li>Added the ability to reference metadata fields to the ES Input</li>
</ul></body></html>

Logstash Lines: Experimenting With Bytecode Generation

<html><head></head><body><p>Welcome back to The Logstash Lines! In these weekly posts, we&apos;ll share the latest happenings in the world of Logstash and its ecosystem.</p><p><strong>Goodbye Travis, Hello Jenkins! (for core)<br></strong></p><p>We&apos;ve finished transitioning logstash core away from Travis to our own Jenkins infrastructure. Being on fast, stable hardware has really improved the reliability of our tests. Additionally, this move has reduced our PR test time from 45 minutes to often under 10 minutes per commit. We now split up some of the work among multiple jobs.</p><p>Another part of this work has been stabilizing tests. The tests for Logstash are green more often now than they have been in a long time; all while the amount and coverage of testing in Logstash has grown astronomically.&#xA0;</p><p><strong>More Bundled Plugins than Before 6.0</strong></p><p>In Logstash 6+ all commercially supported plugins will now be part of the package. This has been a problem historically for users in air-gapped environments, who would need to go through the cumbersome process of downloading plugins that are part of our supported list. (#8318). &#xA0;The added plugins are: the aggregate filter, anonymize filter, de-dot filter, elasticsearch filter, jdbc_streaming filter, truncate filter, email output, and lumberjack output.</p><p><strong>Improvements to Windows Platform Support</strong></p><p>We&apos;ve continued to make strides improving our windows support. We&apos;ve been steadily making our Windows experience more consistent and correct, as well as improving test support in a variety of PRs.</p><p><strong>Fixes in master:</strong></p><ul><li>Upgraded Jackson library to correctly serialize large integers&#xA0;</li><li>Fixed a variety of layout and other bugs for our Kibana interfaces (thanks Shaunak!)</li><li>Fixed rspec.bat on windows to use the correct JVM options: <a href="https://github.com/elastic/logstash/pull/8359">https://github.com/elastic/logstash/pull/8359</a></li><li><a href="https://github.com/elastic/logstash/pull/8359"></a>Fixed a bug in the Elasticsearch output that would cause exceptions when throwing some exceptions: <a href="https://github.com/logstash-plugins/logstash-output-elasticsearch/pull/680">https://github.com/logstash-plugins/logstash-outpu...</a></li></ul></body></html>

Logstash Lines: More Bundled Plugins and Testing Improvements

This blog post covers the history of how outputs and output workers work in the pipeline and how to use them most effectively.

<html><head></head><body><h2 style="box-sizing: border-box; margin-bottom: 16px; font-size: 1.5em; line-height: 1.25; padding-bottom: 0.3em; border-bottom-width: 1px; border-bottom-style: solid; border-bottom-color: rgb(238, 238, 238); color: rgb(51, 51, 51); font-family: -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Roboto, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;; background-color: rgb(255, 255, 255);">Summary</h2><ul><li>If you&#x2019;re having performance problems read the new&#xA0;<a href="https://www.elastic.co/guide/en/logstash/2.3/performance-troubleshooting.html">Performance Tuning Guide</a>&#xA0;Logstash output workers are confusing.</li><li>Logstash &lt;= 2.1 had slow and safe defaults</li><li>Logstash 2.2 had fast and unsafe defaults, this was mostly fine, but sometimes not</li><li>Logstash 2.3 went back to slow and safe defaults</li><li>Logstash 2.4 will let us start making plugins fast and safe as a default again, but we need to update individual plugins to do this using the new &#x2018;shared&#x2019; output type.</li><li>If your output is the bottleneck today manually set the output workers setting higher. This may cause increased resource usage and may cause some issues on some plugins, but it generally works for common plugins.</li></ul><h2 style="box-sizing: border-box; margin-top: 24px; margin-bottom: 16px; font-size: 1.5em; line-height: 1.25; padding-bottom: 0.3em; border-bottom-width: 1px; border-bottom-style: solid; border-bottom-color: rgb(238, 238, 238); color: rgb(51, 51, 51); font-family: -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Roboto, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;; background-color: rgb(255, 255, 255);">A Brief Overview</h2><p style="box-sizing: border-box; margin-bottom: 16px; color: rgb(51, 51, 51); font-family: -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Roboto, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;; line-height: 21px; background-color: rgb(255, 255, 255);">This blog post is meant to clarify the output worker changes in Logstash 2.2+, and to discuss how we&#x2019;ll be improving output workers in the future. Logstash 2.2 introduced the&#xA0;<a href="https://www.elastic.co/guide/en/logstash/current/pipeline.html">NG Pipeline</a>&#xA0;which provided a sizable performance improvement. It has created some confusion however. Prior to 2.2 there was a notion of Filter Workers (FWs) and Output Workers (OWs). Filter workers executed everything in the&#xA0;<code style="box-sizing: border-box; font-family: Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace; font-size: 11.9px; padding: 0.2em 0px; border-radius: 3px; background-color: rgba(0, 0, 0, 0.0392157);">filter {}</code>&#xA0;portion of the config, and each plugin in the&#xA0;<code style="box-sizing: border-box; font-family: Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace; font-size: 11.9px; padding: 0.2em 0px; border-radius: 3px; background-color: rgba(0, 0, 0, 0.0392157);">output {}</code>&#xA0;portion could use a user definable number of OW threads. In 2.2 these were both replaced with one concept of Pipeline Workers (PWs). Confusingly, you can still set OWs, and they still do something very important. Let&#x2019;s compare and contrast.&#xA0;</p><p style="box-sizing: border-box; margin-bottom: 16px; color: rgb(51, 51, 51); font-family: -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Roboto, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;; line-height: 21px; background-color: rgb(255, 255, 255);">The animation below contrasts the threading model between 2.1 and 2.2.<span class="redactor-invisible-space"><span class="redactor-invisible-space"><span class="redactor-invisible-space"><span class="redactor-invisible-space"><span class="redactor-invisible-space"><span class="redactor-invisible-space"><span class="redactor-invisible-space"><span class="redactor-invisible-space"><span class="redactor-invisible-space"></span></span></span></span></span></span></span></span></span><br></p><p style="box-sizing: border-box; margin-bottom: 16px; color: rgb(51, 51, 51); font-family: -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Roboto, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;; line-height: 21px; background-color: rgb(255, 255, 255);" rel="box-sizing: border-box; margin-bottom: 16px; color: rgb(51, 51, 51); font-family: -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Roboto, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;; line-height: 21px; background-color: rgb(255, 255, 255);"><span class="redactor-invisible-space"><span class="redactor-invisible-space"><span class="redactor-invisible-space"><span class="redactor-invisible-space"><span class="redactor-invisible-space"><span class="redactor-invisible-space"><span class="redactor-invisible-space"><span class="redactor-invisible-space"><span class="redactor-invisible-space"><img src="https://api.contentstack.io/v2/assets/57927100b252bbf533797ba3/download?uid=blt044416b614ff9178" data-sys-asset-uid="blt044416b614ff9178" alt="thread_changes.gif"><br></span></span></span></span></span></span></span></span></span></p><p style="box-sizing: border-box; margin-bottom: 16px; color: rgb(51, 51, 51); font-family: -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Roboto, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;; line-height: 21px; background-color: rgb(255, 255, 255);">To put it a bit more formally:</p><ul><li>Logstash &lt; 2.2<ul><li>Filters / Outputs handle one event at a time</li><li>Queue between filter workers and output workers</li><li>Dedicated Thread per FW</li><li>Dedicated Thread per OW</li><li>Independently controllable filter/output workers</li></ul></li><li>Logstash &gt;= 2.2<ul><li>Filters and outputs deal in batches rather than single events.</li><li>No queue between filter worker and output worker as they now process batches as one unit</li><li>One Pipeline Worker thread pool shared by both filters and outputs</li><li>OWs are no longer threads, they are shared objects that can only be used by a single Pipeline Worker at a time.</li></ul></li></ul><p style="box-sizing: border-box; margin-bottom: 16px; color: rgb(51, 51, 51); font-family: -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Roboto, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;; line-height: 21px; background-color: rgb(255, 255, 255);">The upshot of this change is that in 2.2+&#xA0;<code style="box-sizing: border-box; font-family: Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace; font-size: 11.9px; padding: 0.2em 0px; border-radius: 3px; background-color: rgba(0, 0, 0, 0.0392157);">-w</code>, which now controls the pipeline worker setting, controls how many concurrent threads can exist for the filter and output stage combined (PWs). The output worker setting controls how many of those threads can simultaneously work on that output. The takeaways here are:</p><p style="box-sizing: border-box; margin-bottom: 16px; color: rgb(51, 51, 51); font-family: -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Roboto, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;; line-height: 21px; background-color: rgb(255, 255, 255);">Output workers should never be set to a number &gt; the number of pipeline workers, since that is the maximum number that can be simultaneously used. You may need more PWs than cores on your system since multiple PWs can be blocked on I/O in a single OW object. Increasing the number of OWs often provides a tangible benefit but be careful. For some outputs adding more can either slow them down or cause them to break in subtle ways.</p><p style="box-sizing: border-box; margin-bottom: 16px; color: rgb(51, 51, 51); font-family: -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Roboto, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;; line-height: 21px; background-color: rgb(255, 255, 255);">A more in-depth exploration of the history and the evolution of pipeline architecture is available in this video by Logstash creator Jordan Sissel. We encourage you to watch this!<br></p><p style="box-sizing: border-box; margin-bottom: 16px; color: rgb(51, 51, 51); font-family: -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Roboto, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;; line-height: 21px; background-color: rgb(255, 255, 255);"><br></p> <iframe style="width: 500px; height: 281px;" src="//www.youtube.com/embed/FPLHS9Pmgk0" allowfullscreen frameborder="0"></iframe><h2 style="box-sizing: border-box; margin-top: 24px; margin-bottom: 16px; font-size: 1.5em; line-height: 1.25; padding-bottom: 0.3em; border-bottom-width: 1px; border-bottom-style: solid; border-bottom-color: rgb(238, 238, 238); color: rgb(51, 51, 51); font-family: -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Roboto, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;; background-color: rgb(255, 255, 255);">Why Autoscaling Output Workers in Logstash 2.2 was a Mistake</h2><p style="box-sizing: border-box; margin-bottom: 16px; color: rgb(51, 51, 51); font-family: -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Roboto, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;; line-height: 21px; background-color: rgb(255, 255, 255);">The NG pipeline was released in 2.2 a few months after 2.0 went out. In Logstash 2.0 we had switched the default protocol for the Elasticsearch output to HTTP. One concern here was that while the plugin was nearly identical in performance / resource utilization to the old default of the Elasticsearch Transport protocol it needed more OWs to reach parity. There was a concern that users upgrading from 1.x to 2.x might feel this slowdown due to the defaults and not tune OW settings. We thought it would also be generally good to, by default, set the number of OWs for every plugin to be equal to the number of PWs. We code reviewed each output we maintain for concurrency issues and set the required options on ones that didn&#x2019;t support &gt; 1 OW, which meant that the pipeline would leave these at a single OW. This turned out to have some...err&#x2026; adverse effects.</p><p style="box-sizing: border-box; margin-bottom: 16px; color: rgb(51, 51, 51); font-family: -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Roboto, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;; line-height: 21px; background-color: rgb(255, 255, 255);">The main problem with this approach is that doing post-hoc code reviews for concurrency is hard. We have a truly massive amount of plugins which made the task even harder. We started getting bug reports in the 2.2.x series related to plugins we&#x2019;d missed that may have been threadsafe in terms of data structures, but not in terms of logic.</p><p style="box-sizing: border-box; margin-bottom: 16px; color: rgb(51, 51, 51); font-family: -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Roboto, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;; line-height: 21px; background-color: rgb(255, 255, 255);">We made the decision that it was premature to assume that all outputs would behave well in a concurrent environment and to let those bugs shake out in a minor release, so, we reverted it, setting the default back to one OW. So, users who saw speed gains in 2.2.x might have seen speed losses in 2.3.x out of the box. Improving performance is a top priority for us, but correctness of outputs is of even greater importance. Luckily we can achieve both with the new model we&#x2019;re incorporating into the upcoming Logstash 2.4 and 5.0 series.</p><p style="box-sizing: border-box; margin-bottom: 16px; color: rgb(51, 51, 51); font-family: -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Roboto, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;; line-height: 21px; background-color: rgb(255, 255, 255);">All this being said, if you&#x2019;re having performance problems please read the new&#xA0;<a href="https://www.elastic.co/guide/en/logstash/2.3/performance-troubleshooting.html%5BPerformance">Performance Tuning Guide</a>. Its an easy to follow set of diagnostic and remedial procedures for managing Logstash performance.</p><h2 style="box-sizing: border-box; margin-top: 24px; margin-bottom: 16px; font-size: 1.5em; line-height: 1.25; padding-bottom: 0.3em; border-bottom-width: 1px; border-bottom-style: solid; border-bottom-color: rgb(238, 238, 238); color: rgb(51, 51, 51); font-family: -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Roboto, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;; background-color: rgb(255, 255, 255);">The Future</h2><p style="box-sizing: border-box; margin-bottom: 16px; color: rgb(51, 51, 51); font-family: -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Roboto, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;; line-height: 21px; background-color: rgb(255, 255, 255);">The future is in just&#xA0;<a href="https://github.com/elastic/logstash/issues/5653">getting rid of OWs</a>. We call these &#x2018;shared&#x2019; outputs. Outputs that can&#x2019;t be parallelized will be the default, and will only support a single instance with synchronized access. The latest version of the ES output (currently only available in 5.x alphas, but most likely coming to ES 2.4.x) is the first shared Logstash output. These outputs manage all their own concurrency internally as a single shared object. The new ES output will run in parallel for as many PWs as you have with no additional configuration. There are some concurrency controls but these must be custom defined by the plugin in a way that makes sense for its own IO. For instance, on the Elasticsearch output plugin the maximum # of connections will be tunable (it defaulted to a very high number by default, at 100), but aside from that there aren&#x2019;t any.</p><p style="box-sizing: border-box; color: rgb(51, 51, 51); font-family: -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Roboto, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;; line-height: 21px; margin-bottom: 0px !important; background-color: rgb(255, 255, 255);">In the future we hope to refactor as many Logstash output plugins as possible to use the new&#xA0;<code style="box-sizing: border-box; font-family: Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace; font-size: 11.9px; padding: 0.2em 0px; border-radius: 3px; background-color: rgba(0, 0, 0, 0.0392157);">concurrency :shared</code>&#xA0;option, per&#xA0;<a href="https://github.com/elastic/logstash/issues/5661">Issue #5661</a>&#xA0;and use finer grain locking than what the OW system provided. We&#x2019;ll provide an update when this is done. In the meanwhile, we encourage our users to discuss these issues on our <a href="https://discuss.elastic.co/c/logstash">forum</a>, and open GitHub issues. We are also available on <a href="https://twitter.com/elastic">twitter</a> and <a href="https://webchat.freenode.net/#logstash">IRC</a> to help you out!</p></body></html>

log-grains-logstash-output-workers-full-bleed.jpg

bg_ss2.jpg

A History of Logstash Output Workers

We’ve released Logstash 2.3.1 and Logstash 2.2.4 which contain important compatibility and security updates. We highly recommend that all users upgrade.

<html><head></head><body><p>Hot on the heels of 2.3.0 and 2.2.3, we&#x2019;ve released Logstash 2.3.1 and Logstash 2.2.4 which contain important compatibility and security updates. We highly recommend that all users upgrade to either version immediately.&#xA0;You can read the detailed release notes <a href="https://www.elastic.co/guide/en/logstash/2.3/releasenotes.html">here</a>, or jump directly to our <a href="https://www.elastic.co/downloads/logstash">downloads page</a> if you can&apos;t wait to test drive this release! Additionally, we&apos;ve released an update to the 2.2.x series, with a bunch of important bugs packaged in 2.2.4. You can read the changelog for 2.2.4&#xA0;<a href="https://github.com/elastic/logstash/blob/2.2/CHANGELOG.md">here</a>.</p><h2>Regression in Regex handling in JRuby</h2><p>The upgraded version of JRuby we included in 2.3.0 contained an important fix for Windows users, but introduced a dangerous thread safety bug for regular expressions used within Logstash.
</p><h2><a id="Environment_Variable_Support_Now_Deactivated_By_Default_6"></a>Environment Variable Support Now Deactivated By Default</h2><p>We are very excited to have introduced support for environment variables in configuration files in Logstash 2.3.0! However, we didn&#x2019;t anticipate that some existing configs would be incompatible with this change. The environment variable support we added treats <code>$</code> or <code>${</code> characters as part of our new variable syntax. This creates problems for people using the characters for other purposes, such as in password values. As such, we have disabled environment variable interpolation by default in 2.3.1 to make the upgrade path easier. You can enable it in 2.3.1 with the <code>--allow-env</code> flag on the CLI. In addition, we have removed support for <code>$VAR</code> environment variables completely. Only the <code>${VAR}</code> syntax will be supported going forward as it minimizes the potential for conflicts. This change was completed in <a href="https://github.com/elastic/logstash/pull/4963">#4963</a>
</p><h2><a id="Fixed_Broken_Conditionals_10"></a>Fixed Broken Conditionals</h2><p>Logstash 2.3.0 includes our new dynamic reloading feature. A change to the pipeline internals required by this feature caused conditionals (anywhere an <code>if</code> was used in a config) to not work correctly. This was fixed in <a href="https://github.com/elastic/logstash/pull/4970">#4970</a>. Logstash versions prior to 2.3.0 were not affected by this bug.
</p><h2><a id="Reverting_the_new_Java_Event_In_Logstash_231_14"></a>Reverting the new Java Event In Logstash 2.3.1</h2><p>The new pure Java implementation of the Event class Logstash 2.3.1 is lightning fast, but unfortunately not as compatible as we&#x2019;d have liked for a minor release. In particular, it could cause problems with some custom Ruby filter scripts and custom plugins from the community.  We take our commitment to compatibility, and versioning semantics, seriously. Though we have reverted to the prior Ruby Event implementation, the Java version remains the correct technical direction and we will most likely be reintroducing it in Logstash 5.0 if not sooner. If any breaking changes need to be made, we will ensure these changes are communicated as clearly and broadly as possible.
</p><h2><a id="Passwords_Printed_in_Log_Files_under_Some_Conditions_18"></a>Passwords Printed in Log Files under Some Conditions</h2><p>It was discovered that, in Logstash 2.1.0+, log messages generated by a stalled pipeline during shutdown will print plaintext contents of password fields. While investigating this issue we also discovered that debug logging has included this data for quite some time. Our latest releases fix both leaks. You will want to scrub old log files if this is of particular concern to you. This was fixed in issue <a href="https://github.com/elastic/logstash/pull/4965">#4965</a>
</p><h2><a id="Fixed_Config_Test_Flag_in_Logstash_230_22"></a>Fixed Config Test Flag in Logstash 2.3.0</h2><p>The Logstash 2.3.0 release inadvertently broke the <code>--config-test</code> option, this has been fixed in Logstash 2.3.1. This was fixed in issue <a href="https://github.com/elastic/logstash/issues/4933">#4933</a>
</p><h2><a id="Other_Issues_26"></a>Other Issues</h2><p>The accounting of issues fixed in the Logstash 2.3.1 / 2.2.4 release is tracked on issue <a href="https://github.com/elastic/logstash/issues/4981">#4981</a>.
</p><h2><a id="Feedback_30"></a>Feedback</h2><p>We are super excited for this release of Logstash and look forward to your feedback. You can reach us at our <a href="https://discuss.elastic.co/c/logstash">forum</a>, open issues in our <a href="https://github.com/elastic/logstash/issues">GitHub repo</a>, or tweet at us <a href="https://twitter.com/elastic">(@elastic)</a>.
</p><p>Happy &apos;stashing!
</p></body></html>

logstash-2-3-1-full-bleed.jpg

Logstash 2.3.1 and 2.2.4 Released

Our guide to upgrading your Logstash cluster to Logstash 2.2+. Do you know how to tune our new pipeline?

NG Pipeline.png

Upgrade Guide for Logstash 2.2

Logstash 2.2.0 and 2.1.2 is available for download as part of the release bonanza! Logstash 2.2.0 includes changes to the internal architecture of the pipeline.

Untitled design (4).jpg

Logstash 2.2.0 and 2.1.2 released

Using the new Logstash Nmap codec we show how you can gain new insight into the internals of your network.

<html><head></head><body><p>In this post we&#x2019;ll look at a brand new logstash codec plugin: <a href="https://www.elastic.co/guide/en/logstash/2.2/plugins-codecs-nmap.html">logstash-codec-nmap</a>. This plugin lets you directly import <a href="https://nmap.org/">Nmap</a> scan results into Elasticsearch where you can then visualize them with Kibana. Nmap is somewhat hard to describe because its a sort of swiss army knife of network tools. It crams many different features into a single small executable. I&#x2019;ve put together a small list of things you can do with Nmap below, though it is by no means complete!
</p><ul>
	<li>Ping one or more hosts and discover the RTT for the ping</li>
	<li>Issue traceroutes to one or more hosts</li>
	<li>Check one or more hosts for one or more open ports</li>
	<li>Scan a network for all open hosts and ports</li>
	<li>Attempt to detect the OS of a host and the application running on a port</li>
	<li>Check for complex states, the like <a href="https://nmap.org/nsedoc/scripts/ssl-heartbleed.html">OpenSSL Heartbleed</a> vulnerability via custom <a href="https://nmap.org/book/nse.html">NSE</a> scripts</li>
</ul><p>Using Logstash, Elasticsearch, and Kibana you can create neat dashboards, like the one I have for my home LAN below:
</p><p><img src="https://api.contentstack.io/v2/assets/575e4d2143e9adc538715393/download?uid=bltc8b4cb72881c66d6?uid=bltc8b4cb72881c66d6" alt="Logstash Nmap Dashboard Home LAN Kibana" style="max-width:100%;">
</p><h2><a id="Monitoring_Host_Availability_with_Nmap_13"></a>Monitoring Host Availability with Nmap</h2><p>Let&#x2019;s start by just poking around with some Nmap basics. Let&#x2019;s say we simply want to check if a host is up or not with an ICMP ping. We can do this by simply running <code>sudo nmap -sP example.net</code>. You should see the output below:
</p><pre class="prettyprint">Starting Nmap 7.01 ( https://nmap.org ) at 2016-01-26 12:28 CST
 Nmap scan report for example.net (93.184.216.34)
 Host is up (0.020s latency).
 Other addresses for example.net (not scanned): 2606:2800:220:1:248:1893:25c8:1946
 Nmap done: 1 IP address (1 host up) scanned in 0.21 seconds
</pre><p>You may be wondering why we need to run our ping command with <code>sudo</code>. The reason is that ICMP ping packets <em>must</em> be sent as root, otherwise the <a href="https://nmap.org/bennieston-tutorial/">connect</a> method will be used. This is <a href="http://unix.stackexchange.com/questions/51874/root-owned-program-with-setuid-bit-on">why the ping command is setuid with root as  owner</a> on most platforms. While the output here is nice and human readable, it is not something Logstash can parse. To get machine readable XML output you&#x2019;ll need to use the <code>-oX filename</code> option, which if <code>-</code> is used for the filename redirects to stdout. Let&#x2019;s try running <code>sudo nmap -sP example.net -oX -</code>. You should see the same output, but with more verbosity and in XML format.
</p><p>Now that we&#x2019;ve got our bearings let&#x2019;s setup a Logstash server to receive this data. To understand this setup let&#x2019;s quickly recap what a Logstash codec is. Logstash codecs simply provide a way to specify how raw data should be decoded, regardless of source. This means that we can use the Nmap codec to read Nmap XML from a variety of inputs. We could read it off a message queue or via syslog for instance, before passing the data on to the Nmap codec.
</p><p>A very flexible solution for a lot of people is to use the <a href="https://www.elastic.co/guide/en/logstash/current/plugins-inputs-http.html">Logstash HTTP input</a>. This input sets up a webserver inside the logstash process which listens for requests and turns the request body into a Logstash event&#x2013;in our case using the nmap codec. For now we&#x2019;ll use a the <code>rubydebug</code> output on <code>stdout</code> to let us see the parsed nmap data. Try out the config below:
</p><pre class="prettyprint">input {
  http {
    host =&gt; &quot;127.0.0.1&quot;
    port =&gt; 8000
    codec =&gt; nmap
  }
}
output {
  stdout {
    codec =&gt; rubydebug
  }
}
</pre><p>Then, in your Logstash folder, run <code>bin/plugin install logstash-codec-nmap</code>. Once you have the Nmap codec installed you can start logstash with <code>bin/logstash -f my_config</code>.
Logstash is now ready to watch for Nmap XML on port 8000.
</p><p>You can send a simple ping by running the following in your shell. We&#x2019;ll use cURL to transport the Nmap XML to Logstash.
</p><p><code>nmap -sP example.net -oX - | curl -H &quot;x-nmap-target: example.net&quot; http://localhost:8000 -d @-</code>
</p><p>Note that we&#x2019;re using <code>-oX -</code> with Nmap to send XML to stdout, and <code>-d @-</code> to set stdin as the request body for our request. We&#x2019;re also setting a custom header: <code>x-nmap-target</code>.The Logstash HTTP input will make this header available to us as part of our events.
</p><p>After sending this request you should see a bunch of output in your terminal from logstash. There should be two Logstash events in your terminal now, one with a <code>type</code> of <code>nmap_host</code>, the other <code>nmap_scan_metadata</code> (note that this &#x2018;type&#x2019; field is distinct from the Logstash convention of &#x2018;@type&#x2019;). You should also see a bunch of HTTP metadata from the HTTP input, including our custom <code>x-nmap-target</code> header.
</p><h2><a id="Nmap_Codec_Event_Types_55"></a>Nmap Codec Event Types</h2><p>OK! So, now we know how to get some basic info out of nmap. Let&#x2019;s take a deeper look at the data coming out of the Nmap codec, which does some restructuring and denormalization of the Nmap XML.
</p><p>Nmap scans create multiple events of multiple types. The types are listed below:
</p><ul>
	<li><code>nmap_scan_metadata</code>: An object containing top level information about the scan, including how many hosts were up, and how many were down. Useful for the case where you need to check if a DNS based hostname does not resolve, where both those numbers will be zero.</li>
	<li><code>nmap_host</code>: One event is created per host. This is the full data covering an individual host, including open ports and traceroute information as a nested structure.</li>
	<li><code>nmap_port</code>: One event is created per host/port. This duplicates data already in <code>nmap_host</code>: This was put in for the case where you want to model ports as separate documents in Elasticsearch (which Kibana prefers).</li>
	<li><code>nmap_traceroute_link</code>: One of these is created per traceroute &#x2018;connection&#x2019;, with a <code>from</code> and a <code>to</code> object describing each hop. Note that traceroute hop data is not always correct due to the fact that each tracing ICMP packet may take a different route. Also very useful for Kibana visualizations.</li>
</ul><h2><a id="A_Small_Network_Monitor_66"></a>A Small Network Monitor</h2><p>Using the Elasticsearch output and Kibana we can setup a more fully featured example. This is something I run on my own home network to check a few different things. Here we&#x2019;ll use some of Nmap&#x2019;s more powerful features, the ability to target an entire subnet at once. My home network runs on the subnet 192.168.1.0/24, for instance. We can turn pretty much all the useful options on with the <code>-A</code> flag, which will per the <a href="https://nmap.org/book/man-briefoptions.html">Nmap documentation</a> <em>&#x201C;Enable OS detection, version detection, script scanning, and traceroute&#x201D;</em> giving us the command below.
</p><p><code>nmap -A 192.168.1.0/24 -oX - | curl -H &quot;x-nmap-target: local-subnet&quot; http://localhost:8000 -d @-</code>.
</p><p>In the line above we&#x2019;ve used a different descriptive HTTP header in this example. We could use this in our Logstash configuration to help divide and filter the output of different cURL commands, though in this case we will not.
</p><p>Next, to put this into Elasticsearch in a sane manner we&#x2019;ll need an <a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping.html">Elasticsearch mapping</a>. Luckily, I&#x2019;ve created one that handles the necessary document types for this demo, which you can find <a href="https://raw.githubusercontent.com/logstash-plugins/logstash-codec-nmap/master/examples/elasticsearch/elasticsearch_nmap_template.json">here</a>. You&#x2019;ll want to download this file somewhere locally, then point to it in <a href="https://raw.githubusercontent.com/logstash-plugins/logstash-codec-nmap/master/examples/elasticsearch/logstash_nmap.conf">this config</a>, where <code>&quot;./elasticsearch_nmap_template.json&quot;</code> is specified. Since this mapping template is so radically different than the default Logstash template we&#x2019;ve configured the Logstash Elasticsearch output to send data to timestamped indexes prefixed with <code>nmap-logstash-*</code>, instead of the usual <code>logstash-*</code>, to prevent template collisions.
</p><p>If you put that in place with a simple cronjob and let it run for a while you&#x2019;ll see some interesting results in your network over time. After we have some data we can run some tests and see what we get! Loading the results up in Kibana I can use this data to break down OSes on my network over time, like so:
</p><p><img src="https://api.contentstack.io/v2/assets/575e4d229e7a83165490d9ae/download?uid=blt35e5e16396c63c98?uid=blt35e5e16396c63c98" alt="OSes over time in logstash using nmap" style="max-width:100%;">
</p><p>You could also use this data to chart your typical outbound network routes by aggregating based on the <code>from.ttl</code> in the produced <code>nmap_traceroute_link</code> documents, as well as correlate that with those link&#x2019;s RTT. There&#x2019;s a rich set of data in these documents, too rich to go into detail here. I highly recommend browsing the details of the <code>rubydebug</code> output to see what&#x2019;s possible.
</p><h2><a id="Visualizing_Outbound_Routes_82"></a>Visualizing Outbound Routes</h2><p>We might also be interested in our outbound connectivity, not just what&#x2019;s live on the network. We can use Nmap to both help us discern our outbound routes via traceroutes, as well as determine if we&#x2019;ve lost connectivity to the outbound world. To do that we&#x2019;ll need to, as you might imagine, hit some targets outside our network. We can do this by running the following Nmap command:
</p><p><code>sudo nmap --traceroute -sP example.net -oX - | curl -H &#x201C;x-nmap-target: remote-check&#x201D; http://localhost:8000 -d @-</code>
</p><p>This will test if we can ping the outside world, and provide a traceroute that could be helpful in diagnosing network problems as well. If you put this in cron along with the previous command you can check for host uptime, ping time, as well as the latency for all hosts along the path.
</p><h2><a id="Next_Steps_90"></a>Next Steps</h2><p>This module is in its early stages! Look for more from <a href="https://twitter.com/andrewvc">me</a> regarding this codec in coming blog posts. You can always find us on IRC, talk to us on <a href="https://discuss.elastic.co/c/logstash">discuss.elastic.co</a> and <a href="https://github.com/elastic/logstash">GitHub</a>. We are looking forward to your feedback!
</p></body></html>

1_home_lan_kibana_nmap_dashboard.jpg

Using Nmap + Logstash to Gain Insight Into Your Network

The release of Logstash 2.0 is near! It's primarily about 2 things: better versioned releases and out of the box compatibility with Elasticsearch 2.0.

Logstash 2.0.0 beta1 released

<p>I’m pleased to announce the release of a brand new Logstash input: <a href="https://www.elastic.co/guide/en/logstash/current/plugins-inputs-http_poller.html">HTTP Poller</a>. With this new input you’ll be able to repeatedly poll one or more HTTP endpoints and turn the response into Logstash events. There are a number of practical uses for this plugin, like:
</p><ul>
	<li>Monitoring a daemon such as HAProxy or Apache’s HTTP stats end points for metrics such as total open connections, or the number of busy workers</li>
	<li>Checking that your website is up and responding in a timely manner</li>
	<li>Hitting a custom metrics endpoint in a webapp to gain deep insight into some process not exposed through logs</li>
</ul><p>The syntax for this plugin is dead simple to boot, as seen in the example below:
</p><pre class="prettyprint">input {
  http_poller {
    # List of urls to hit
    # URLs can either have a simple format for a get request
    # Or use more complex HTTP features
    urls =&gt; {
      some_service =&gt; "http://localhost:8000"
      some_other_service =&gt; {
        method =&gt; "POST"
        url =&gt; "http://localhost:8000/foo"
      }
    }
    # Maximum amount of time to wait for a request to complete
    request_timeout =&gt; 30
    # How far apart requests should be
    interval =&gt; 60
    # Decode the results as JSON
    codec =&gt; "json"
    # Store metadata about the request in this key
    metadata_target =&gt; "http_poller_metadata"
  }
}
output {
  stdout {
    codec =&gt; rubydebug
  }
}
</pre><p>This request will put the HTTP responses of the polled endpoints into the <code>message</code> field and provide metadata like response timing and HTTP response headers in the <code>http_poller_metadata</code> field.
</p><h2><a id="Using_HTTP_Poller_to_monitor_website_status_43"></a>Using HTTP Poller to monitor website status</h2><p>We’ll start with a simple example; using HTTP Poller to monitor whether a given URL is up, down, or responding slowly. The following Logstash config will hit a webserver on <code>http://localhost:8000</code>. If you don’t have one up, you can start one that takes a variable length of time to return a JSON response by running <code>ruby -rsinatra -e 'set :port, 8000; get("/") { n= rand(10) / 10.0; sleep n; "{\"t\": #{n}}" }'</code> in your console (assuming you have ruby installed and have installed the sinatra gem with <code>gem install sinatra</code>). After that, try running Logstash with the sample config below. The sample config has been heavily annotated to make reading it easy, even for a complete logstash novice.
</p><p>Using the config below, you can generate Kibana charts like the one just underneath this paragraph, showing the ratio of slow to fast requests to your service over time. If you have Elastic's <a href="https://www.elastic.co/what-is/elasticsearch-alerting">Watcher</a> set&nbsp;up, you can use that to automatically send you alerts when you receive slow requests as well
</p><p><img src="https://api.contentstack.io/v2/assets/575e4c930342dfd738263220/download?uid=blt1258163f65d4cf37?uid=blt1258163f65d4cf37" data-sys-asset-uid="blt8c4282f417a5443e" style="max-width:100%;">
</p><pre class="prettyprint">input {
  http_poller {
    urls =&gt; {
      "localhost" =&gt; "http://localhost:8000"
    }
    automatic_retries =&gt; 0
    # Check the site every 10s
    interval =&gt; 10
    # Wait no longer than 8 seconds for the request to complete
    request_timeout =&gt; 8
    # Store metadata about the request in this field
    metadata_target =&gt; http_poller_metadata
    # Tag this request so that we can throttle it in a filter
    tags =&gt; website_healthcheck
  }
}
filter {
  # The poller doesn't set an '@host' field because it may or may not have meaning
  # In this case we can set it to the 'name' of the host which will be 'localhost'
  # The name is the key used in the poller's 'url' config
  if [http_poller_metadata] {
    mutate {
      add_field =&gt; {
        "@host" =&gt; "%{http_poller_metadata[name]}"
      }
    }
  }
  # Classify slow requests
  if [http_poller_metadata][runtime_seconds] and [http_poller_metadata][runtime_seconds] &gt; 0.5 {
    mutate {
      add_tag =&gt; "slow_request"
    }
  }
  # Classify requests that can't connect or have an unexpected response code
  if [http_request_failure] or
     [http_poller_metadata][code] != 200 {
     # Tag all these events as being bad
     mutate {
       add_tag =&gt; "bad_request"
     }
 }
 if "bad_request" in [tags] {
    # Tag all but the first message every 10m as "_throttled_poller_alert"
    # We will later drop messages tagged as such.
    throttle {
      key =&gt; "%{@host}-RequestFailure"
      period =&gt; 600
      before_count =&gt; -1
      after_count =&gt; 1
      add_tag =&gt; "throttled_poller_alert"
    }
    # Drop all throttled events
    if "throttled_poller_alert" in [tags] {
      drop {}
    }
    # The SNS output plugin requires special fields to send its messages
    # This should be fixed soon, but for now we need to set them here
    # For a more robust  and flexible solution (tolerant of logstash restarts)
    # Logging to elasticsearch and using the Watcher plugin is advised
    mutate {
      add_field =&gt; {
        sns_subject =&gt; "%{@host} is not so healthy! %{@tags}"
        sns_message =&gt; '%{http_request_failure}'
        codec =&gt; json
      }
    }
  }
}
output {
  # Catch throttled messages for request failures
  # If we hit one of these, send the output to stdout
  # as well as an AWS SNS Topic
  # UNCOMMENT THIS TO ENABLE SNS SUPPORT
  #if "http_request_failure" in [tags] {
  #  sns {
  #    codec =&gt; json
  #    access_key_id =&gt; "YOURKEY"
  #    secret_access_key =&gt; "YOURSECRET"
  #    arn =&gt; "arn:aws:sns:us-east-1:773216979769:logstash-test-topic"
  #  }
  #}
  elasticsearch {
    protocol =&gt; http
  }
  stdout {
    codec =&gt; rubydebug
  }
}
</pre><h2><a id="Using_HTTP_Poller_to_monitor_HAProxy_stats_and_Apache_serverstatus_pages_154"></a>Using HTTP Poller to monitor HAProxy stats and Apache server-status pages</h2><p>Both <a href="http://www.haproxy.org/">HAProxy</a> and <a href="http://httpd.apache.org/">Apache HTTPD</a> support stats API endpoints for to get information like the number of open connections. In the following example I’ll show how to setup Logstash to record this information to elasticsearch.
</p><p>The key takeaway here is that you can use the HTTP Poller to monitor the health of HAProxy and Apache with greater insight than you’d get with logs alone. Additionally, you can use it to trigger alerts via AWS SNS topics when those thresholds are passed. Those SNS topics can be configured to send texts or emails to alert an operator.
</p><p>Implementing this requires you to <a href="http://serverfault.com/a/198271">enable the stats port on HAProxy</a> as well as <a href="http://httpd.apache.org/docs/2.2/mod/mod_status.html">enable mod_status on apache</a>. To make this easier to try out I’ve prepared a script that will launch a set of docker machines with this stuff all setup. To run it you’ll just need bash, docker, and <a href="https://docs.docker.com/machine/">docker-machine</a>. Try checking out all the code in <a href="https://github.com/logstash-plugins/logstash-input-http_poller/tree/master/examples/ha-apache">this directory</a>. After you have the code run <code>buildit.sh</code>, which will launch the docker machines and write out a sample <code>logstash.conf</code> file. After you’ve done that, just run <code>logstash -f logstash.conf</code> to see it with action. If you hit the haproxy server (whose address will be printed out by buildit) with traffic and load the the <code>kibana.elasticdump</code> file into <code>.kibana</code> with <a href="https://github.com/taskrabbit/elasticsearch-dump">elasticdump</a>,  you should see something like the kibana dashboard below.
</p><p><img src="https://api.contentstack.io/v2/assets/575e4c933dc542cb38c083a3/download?uid=bltfebb17b70ea68e55?uid=bltfebb17b70ea68e55" data-sys-asset-uid="bltc9e6da078bab0eae" style="max-width:100%;"><br>
</p><p>Notice that we can graph such things as HAProxy sessions, the response times of polling requests (which rise as the server is more and more saturated, and which HAProxy services are active. All things that cannot be exposed via plain log data, but can be reached via HTTP polling.
</p><p>If you’d rather not run the examples to see the config used, I’ve reproduced a well commented version of it below:
</p><pre class="prettyprint">input {
  # Setup one poller for httpd, we keep these separate to tag them differently
  http_poller {
    urls =&gt; {
      "custom_httpd_t1" =&gt; { url =&gt; "http://192.168.99.100:8001/server-status?auto"}
      "custom_httpd_t2" =&gt; { url =&gt; "http://192.168.99.100:8002/server-status?auto"}
      "custom_httpd_t3" =&gt; { url =&gt; "http://192.168.99.100:8003/server-status?auto"}
     }
     tags =&gt; apache_stats
     codec =&gt; plain
     metadata_target =&gt; http_poller_metadata
     interval =&gt; 1
  }
  # Another poller, this time for haproxy
  http_poller {
    urls =&gt; {
      ha_proxy_stats =&gt; "http://statsguy:statspass@192.168.99.100:1936/;csv"
    }
    tags =&gt; haproxy_stats
    codec =&gt; plain
    metadata_target =&gt; http_poller_metadata
    interval =&gt; 1
   }
    # Pull the regular Apache/HAProxy logs via docker commands
    # This is a hack for the purposes of this example
  pipe {
    command =&gt; "docker logs -f custom_httpd_t1"
    tags =&gt; [ "apache" ]
    add_field =&gt; { "@host" =&gt; "custom_httpd_t1" }
  }
  pipe {
    command =&gt; "docker logs -f custom_httpd_t2"
    tags =&gt; [ "apache" ]
    add_field =&gt; { "@host" =&gt; "custom_httpd_t2" }
  }
  pipe {
    command =&gt; "docker logs -f custom_httpd_t3"
    tags =&gt; [ "apache" ]
    add_field =&gt; { "@host" =&gt; "custom_httpd_t3" }
  }
    pipe {
      command =&gt; "docker logs -f custom_haproxy"
      tags =&gt; [ "haproxy" ]
      add_field =&gt; { "@host" =&gt; "custom_haproxy" }
    }
}
filter {
  if [http_poller_metadata] {
    # Properly set the '@host' field based on the poller's metadat
    mutate {
      add_field =&gt; {
        "@host" =&gt; "%{http_poller_metadata[name]}"
      }
    }
  }
  # Processed polled apache data
  if "apache_stats" in [tags] {
    # Apache stats uses inconsistent key names. Make sure all fields are camel cased, no spaces
    mutate {
      gsub =&gt; ["message", "^Total ", "Total"]
    }
    # Parse the keys/values in the apache stats, they're separated by ": '
    kv {
      source =&gt; message
      target =&gt; apache_stats
      field_split =&gt; "\n"
      value_split =&gt; ":\ "
      trim =&gt; " "
    }
    # We can make educated guesses that strings with mixes of numbers and dots
    # are numbers, cast them for better behavior in Elasticsearch/Kibana
    ruby {
      code =&gt; "h=event['apache_stats']; h.each {|k,v| h[k] = v.to_f if v =~ /\A-?[0-9\.]+\Z/}"
    }
  }
  # Process polled HAProxy data
  if "haproxy_stats" in [tags] {
    split {}
    # We can't read the haproxy csv header, so we define it statically
    # This is because we're working line by line, and so have no header context
    csv {
       target =&gt; "haproxy_stats"
       columns =&gt; [ pxname,svname,qcur,qmax,scur,smax,slim,stot,bin,bout,dreq,dresp,ereq,econ,eresp,wretr,wredis,status,weight,act,bck,chkfail,chkdown,lastchg,downtime,qlimit,pid,iid,sid,throttle,lbtot,tracked,type,rate,rate_lim,rate_max,check_status,check_code,check_duration,hrsp_1xx,hrsp_2xx,hrsp_3xx,hrsp_4xx,hrsp_5xx,hrsp_other,hanafail,req_rate,req_rate_max,req_tot,cli_abrt,srv_abrt,comp_in,comp_out,comp_byp,comp_rsp,lastsess,last_chk,last_agt,qtime,ctime,rtime,ttime ]
    }
    # Drop the haproxy CSV header, which always has this special value
    if [haproxy_stats][pxname] == "# pxname" {
      drop{}
    }
    # We no longer need the message field as the CSV filter has created separate
    # fields for data.
    mutate {
      remove_field =&gt; message
    }
    # Same as the cast we did for apache
    ruby {
      code =&gt; "h=event['haproxy_stats']; h.each {|k,v| h[k] = v.to_f if v =~ /\A-?[0-9\.]+\Z/}"
    }
  }
  # Process the regular apache logs we captured from the docker pipes
  if "apache" in [tags] {
    grok {
      match =&gt; [ "message", "%{COMMONAPACHELOG:apache}" ]
    }
  }
  # We're going to email ourselves on error, but we want to throttle the emails
  # so we don't get so many. This says only send one every 5 minutes
  if "_http_request_failure" in [tags] {
    throttle {
      key =&gt; "%{@host}-RequestFailure"
      period =&gt; 600
      before_count =&gt; -1
      after_count =&gt; 1
      add_tag =&gt; "_throttled_poller_alert"
    }
    # Drop all throttled events
    if "_throttled_poller_alert" in [tags] {
      drop {}
    }
    # The SNS output plugin requires special fields to send its messages
    # This should be fixed soon, but for now we need to set them here
    mutate {
      add_field =&gt; {
        sns_subject =&gt; "%{@host} unreachable via HTTP"
        sns_message =&gt; "%{http_request_failure}"
      }
    }
  }
}
output {
  # Store everything in the local elasticsearch
  elasticsearch {
    protocol =&gt; http
  }
  # Catch throttled messages for request failures
  # If we hit one of these, send the output to stdout
  # as well as an AWS SNS Topic
  # UNCOMMENT TO ENABLE SNS
  #if "_http_request_failure" in [tags] {
  #  sns {
  #    codec =&gt; json
  #    access_key_id =&gt; "YOURKEY"
  #    secret_access_key =&gt; "YOURSECRET"
  #    arn =&gt; "arn:aws:sns:us-east-1:773216979769:logstash-test-topic"
  #  }
    stdout {
      codec =&gt; rubydebug
    }
  }
}
</pre><h2><a id="Using_the_HTTP_Client_Mixin_in_Your_Own_Plugin_342"></a>Using the HTTP Client Mixin in Your Own Plugin</h2><p>HTTP Poller is the first plugin to use <a href="https://github.com/logstash-plugins/logstash-mixin-http_client">logstash-mixin-http_client</a>. If you need to add an HTTP client to a plugin you’re writing consider using the <code>HttpClient</code> mixin. This mixin will add a bunch of well validated configuration options and sane defaults to your plugin for free. Using it is as simple as adding <code>include LogStash::PluginMixins::HttpClient</code> to the body of your plugin. This will expose a new <code>client</code> method in your plugin class, which is an instance of the <a href="https://github.com/cheald/manticore">Manticore</a> http client. Manticore is a well written and performant client is based on Apache Commons HTTP. Of note is Manticore’s ability to execute requests asynchronously using thread pools with a simple API
</p><h2><a id="Wrapping_Up_346"></a>Wrapping Up</h2><p>I hope these examples have been useful! If you find any other uses for the http input poller, <a href="https://discuss.elastic.co/">let us know</a>! If you think you’ve found a bug in it, please <a href="https://github.com/logstash-plugins/logstash-input-http_poller">submit an issue</a>.
</p>

HTTP Poller, Opening up a New World for Logstash

<html><head></head><body><p><strong>UPDATE:</strong> This article refers to our hosted Elasticsearch offering by an older name, Found. Please note that Found is now known as Elastic Cloud.
</p>
<p>We&apos;ll cover the basics of scoring using functions while taking a look at some use cases where functional scoring techniques are highly useful and effective.
</p><section id="introduction" class="level2">
<h2><a href="#introduction">Introduction</a></h2>
<p>At the heart of any search engine, Elasticsearch included, is the concept of scoring. Scoring can be loosely defined as finding data meeting some set of criteria and returning it sorted in order of relevance. Relevance is often implemented via an algorithm like TF-IDF, which tries to find out which documents are most textually similar to the submitted query. While TF-IDF and its cousins such as BM25 are fantastic, there are times where the question of relevance must be answered via other algorithms, or augmented with other scoring heuristics. It is here that Elasticsearch&#x2019;s <code>script_score</code> and <code>function_score</code> features become useful. This article will cover the usage of these tools.
</p>
<p>One example of a domain where text similarity is not the most important factor is geo search. If one is looking to find a good coffee shop near a given point, ranking coffee shops by how textually similar they are to the query won&#x2019;t be very useful to the user, but ranking them in order of how geographically close they are may be.
</p>
<p>Another example might be videos on a video sharing site, where search results should probably factor in the relative popularity of a video. If a pop star uploads a video with a given title, thus garnering millions of views, that video should probably outrank an unpopular video with similar textual relevance.
</p>
</section><section id="diving-in" class="level2">
<h2><a href="#diving-in">Diving In</a></h2>
<p>Let&#x2019;s dive in and see how we might approach the problem we just mentioned, ranking videos on a website using a combination of textual relevance and the videos popularity on the site. The best way to approach these problems is to think of the objectives in plain English, then convert them into an actual mathematical formula. In the case of our video site, our objectives can be listed as below.
</p>
<ul>
	<li>Factor in the standard TF-IDF textual relevance of the video metadata (title, description, etc.)</li>
	<li>Factor in the number of likes a given video has received</li>
	<li>Factor in the number of views a given video has received</li>
</ul>
<p>Given these criteria, we can come up with a naive formula, something like: <code>score*(likes+views+1)</code>. Note that we have specified the sum of <code>likes</code> and <code>views</code> as a coefficient (adding a 1 in case both are 0). Using multiplication here is the right approach, since the absolute values of <code>score</code> should not be counted upon. The specific numbers returned as a <code>score</code> may vary across Elasticsearch versions, queries, and even across shards. There are a number of obvious problems with this naive approach, primarily that very poor matches with large numbers of likes and/or views will rank very high, despite their poor textual relevance. We need to tame the impact of large <code>likes</code> and <code>views</code> values. It is obvious that a good text match with 1 view will score lower than a poor match with 1mm views, this is a situation we must fix post-haste!
</p>
<p>A simple way to dampen the effects of more popular videos, thus capping the influence of any one coefficient, is by using using its logarithm instead, so the above formula might be changed to:
</p>
<p><code>score*log(likes+views+1)</code>
</p>
<p>This formula will make the algorithm quite a bit more fair, not letting ultra-popular videos dominate all results. The algorithm at this point is an OK conceptual model for our scoring algorithm, but it most likely will not work perfectly out of the box. Let&#x2019;s try and set up some test data and give it a shot. In the example below we&#x2019;ll create a schema for our video site, setup some test data, then run some queries against it.
</p>
<pre class="sourceCode bash prettyprint"># Create the index
curl -XPOST http://localhost:9200/searchtube

# Create the mappings
curl -XPUT http://localhost:9200/searchtube/video/_mapping -d &apos;{
  &quot;video&quot;: {
    &quot;properties&quot;: {
      &quot;title&quot;: {
        &quot;type&quot;: &quot;string&quot;,
        &quot;analyzer&quot;: &quot;snowball&quot;
      },
      &quot;description&quot;: {
        &quot;type&quot;: &quot;string&quot;,
        &quot;analyzer&quot;: &quot;snowball&quot;
      },
      &quot;views&quot;: {
        &quot;type&quot;: &quot;integer&quot;
      },
      &quot;likes&quot;: {
        &quot;type&quot;: &quot;integer&quot;
      },
      &quot;created_at&quot;: {
        &quot;type&quot;: &quot;date&quot;
      }
    }
  }
}&apos;

# Create some docs
curl -XPUT http://localhost:9200/searchtube/video/1 -d &apos;{
  title: &quot;Sick Sad World: Cold Breeze on the Interstate&quot;,
  description: &quot;Is your toll collector wearing pants, a skirt, or nothing but a smile? Cold Breeze on the Interstate, next on Sick, Sad World.&quot;,
  views: 500,
  likes:2,
  created_at: &quot;2014-04-22T08:00:00&quot;
}&apos;

curl -XPUT http://localhost:9200/searchtube/video/2 -d &apos;{
  title: &quot;Sick Sad World: The Severed Pianist&quot;,
  description: &quot;When he turned up his nose at accordion lessons, they cut off his inheritance molto allegro. The Severed Pianist, next on Sick, Sad World.&quot;,
  views: 6000,
  likes: 100,
  created_at: &quot;2014-04-22T12:00:00&quot;
}&apos;

curl -XPUT http://localhost:9200/searchtube/video/3 -d &apos;{
  title: &quot;Sick Sad World: Avant Garde Obstetrician&quot;,
  description: &quot;Meet the avant-garde obstetrician who has turned his cast offs into art work. Severed Umbilical cord sculpture next, on Sick, Sad World.&quot;,
  views: 100,
  likes: 130,
  created_at: &quot;2014-04-22T23:00:00&quot;
}&apos;
</pre>
<p>With this data set we can test out our formula with the query below.
</p>
<pre class="sourceCode bash prettyprint">curl -XPOST http://localhost:9200/searchtube/_search -d &apos;
{
  &quot;query&quot;: {
    &quot;function_score&quot;: {
      &quot;query&quot;: {&quot;match&quot;: {&quot;_all&quot;: &quot;severed&quot;}},
      &quot;script_score&quot;: {
        &quot;script&quot;: &quot;_score * log(doc[&apos;likes&apos;].value + doc[&apos;views&apos;].value + 1)&quot;
      }
    }
  }
}&apos;
</pre>
<p>By using the <code>script_score</code> option to a <code>function_score</code> query we were able to directly translate our mathematical formula into an Elasticsearch formula! It is possible to refine this score further, by weighting <code>likes</code> more than <code>views</code> perhaps, or by further altering the scale. In the next section, we&#x2019;ll cover decay functions, and there, we&#x2019;ll see how we can turn this <code>script_score</code> query into a cleaner array of decay functions.
</p>
</section><section id="decay-functions-in-elasticsearch" class="level2">
<h2><a href="#decay-functions-in-elasticsearch">Decay Functions in Elasticsearch</a></h2>
<p>Functional scoring techniques are useful for more than just modifying the default Elasticsearch scoring algorithm, they can be used to completely replace it. A great example of this is a &#x2018;trending&#x2019; search, showing items within a topic that are rapidly gaining in popularity.
</p>
<p>Such scores cannot be based on a simple metric such as &#x2018;likes&#x2019; or &#x2018;views&#x2019;, but must be continually adjusted based on the current time. A video receiving 1000 views in 1 hour should generally be considered &#x2018;hotter&#x2019; than a video receiving 10000 views in 24 hours. Elasticsearch ships several decay functions that make solving this sort of problem a breeze.
</p>
<p>For our video database, what we&#x2019;d like is to show our most popular queries within the previous day, weighting our more recent videos higher. Elasticsearch has three types of &#x2018;curves&#x2019; for dealing with this, <code>gauss</code>, <code>exp</code>, and <code>linear</code>. They let you map a value to a point on a curve. The linear distribution is perhaps simplest. Linear decay means that all points in time decay evenly, that the farther you get from the origin of the more the item is decayed. An exponential decay looks like a parabola, the farther away you get from the origin the more severely the values are decayed. Finally, the gaussian distribution has more of an &#x2018;s&#x2019; on its side shape, dropping off very little for values near the origin, then falling sharply in an almost linear manner, then tapering off once again at a very low value. Graphs of these functions can be seen in the image below, and even snazzier 3d versions of them can be found <a href="/guide/en/elasticsearch/reference/current/query-dsl-function-score-query.html">in this Elasticsearch blog post</a> on the topic, along with detailed formulas.
</p>
<p>
	<img src="https://api.contentstack.io/v2/assets/575e4c9a0342dfd738263326/download?uid=blte48018d5b70491cf?uid=blte48018d5b70491cf" data-sys-asset-uid="blt0e87fbb15f0308a4" alt="Linear, Gaussian, and Exponential Curves" style="max-width:100%;">
</p><figcaption>Linear, Gaussian, and Exponential Curves</figcaption>
<p>Of these various formulas, the <code>gauss</code> formula is the one I most often find to be useful. The gentle taper at the start of the curve means that, in our video example for instance, more recent items get clustered together at the top, and that there isn&#x2019;t a big gap between an item posted 1 hour ago, and one posted 1.5 hours ago. At some point however we want it to start severely penalizing videos, which is where the more linear portion of the curve becomes useful, that&#x2019;s where videos start to quickly disappear down the results page. Finally, we want it to &#x2018;bottom out&#x2019; at some point, which is what the bottom of the curves does. This separates our results into three neat compartments with a very natural transition between all three parts. In plain English this means that very recent videos get a score penalty, moderately recent videos get a larger score penalty, increasing more and more with age, and old videos get a severe penalty.
</p>
<p>The shape of the curve may be controlled with the <code>origin</code>, <code>scale</code>, <code>offset</code>, and <code>decay</code>. These three variables are your main tools for controlling the shape of the curve. The <code>origin</code> and <code>scale</code> parameters can be thought of as your <code>min</code> and <code>max</code>, defining a bounding box within which the curve will be defined. If we want our trending videos list to cover a full day it would be best to define the <code>origin</code> as the current timestamp, and the <code>scale</code> as <code>24h</code>. The offset can be used to fully flatten the curve at the start, setting it to <code>1h</code> for instance, removes all penalties for recent videos. Finally, the <code>decay</code> option alters how severely the document is demoted based on its position. The default <code>decay</code> value is <code>0.5</code>, larger values will make the curve much steeper, and its effect much more pronounced.
</p>
<p>
	<img src="https://api.contentstack.io/v2/assets/575e4c9a3dc542cb38c084b3/download?uid=blt7248cd4fa246ebc6?uid=blt7248cd4fa246ebc6" data-sys-asset-uid="bltdcf0258560f9a47e" alt="Origin and Scale visualized" style="max-width:100%;">
</p><figcaption>Origin and Scale visualized</figcaption>
<p>We can now craft a &#x2018;trending&#x2019; query, that will factor in recency in scoring results. Examine the query below.
</p>
<pre class="sourceCode bash prettyprint">{
  &quot;query&quot;: {
    &quot;function_score&quot;: {
        &quot;functions&quot;: [
          {
            &quot;gauss&quot;: {
                &quot;created_at&quot;: {
                    &quot;origin&quot;: &quot;2014-04-22T23:50:00&quot;,
                    &quot;scale&quot;: &quot;12h&quot;,
                    &quot;offset&quot;: &quot;1h&quot;,
                    &quot;decay&quot;: 0.3
                }
            }
          },
          {
            &quot;gauss&quot;: {
              &quot;likes&quot;: {
                  &quot;origin&quot;: 20000,
                  &quot;scale&quot;: 20000
              }
            }
          },
          {
            &quot;gauss&quot;: {
              &quot;views&quot;: {
                  &quot;origin&quot;: 20000,
                  &quot;scale&quot;: 20000
              }
            }
          }
        ]
    }
  }
}&apos;
</pre>
<p>You&#x2019;ll notice that this query ranks the &#x201C;Sick Sad World: Avant Garde Obstetrician&#x201D; episode as the first video, even though it only has 100 views, and 130 likes, while the next result has thousands of views and likes. This is due to recency being factored in. You&#x2019;ll also note that we no longer have <code>script_score</code> query, and have refactored both <code>likes</code> and <code>views</code> into our query as additional functions using gaussian curves. By using the built-in guassian decay functions our query should execute a bit faster than a script query, and we&#x2019;ll be able to tune the shape of those adjusting curves a little easier as well.
</p>
<p>These sorts of queries can be customized further. By specifying additional functions you can change the shape of the curve any way you wish, and integrate many more additional variables. Additionally, these queries can be used for geographic queries, helping to cluster results more tightly around geo points.
</p>
<p>For a more complete rundown of the options function score queries make available, check out <a href="/guide/en/elasticsearch/reference/current/query-dsl-function-score-query.html">this documentation</a> on the topic!
</p>
</section></body></html>

A Gentle Intro to Function Scoring

All About Analyzers, Part Two

<html><head></head><body><p><strong>UPDATE:</strong> This article refers to our hosted Elasticsearch offering by an older name, Found. Please note that Found is now known as Elastic Cloud.
</p>
<p>In this article we&apos;ll survey various analyzers, each of which showcases a very different approach to parsing text.
</p><section id="introduction" class="level2">
<h2><a href="#introduction">Introduction</a></h2>
<p>Choosing the right analyzer for an Elasticsearch query can be as much art as science. Analyzers are the special algorithms that determine how a string field in a document is transformed into terms in an inverted index.  In this article we&apos;ll survey various analyzers, each of which showcases a very different approach to parsing text.
</p>
<p>Ten tokenizers, thirty-one token filters, and three character filters ship with the Elasticsearch distribution; a truly overwhelming number of options. This number can be increased further still through plugins, making the choices even harder to wrap one&apos;s head around. Combinations of these tokenizers, token filters, and character filters create what&apos;s called an analyzer. There are eight standard analyzers defined, but really, they are simply convenient shortcuts for arranging tokenizers, token filters, and character filters yourself. While reaching an understanding of this multitude of options may sound difficult, becoming reasonably competent in the use of analyzers is merely a matter of time and practice. Once the basic mechanisms behind analysis are understood, these tools are relatively easy to reason about and compose.
</p>
</section><section id="the-components-of-an-analyzer" class="level2">
<h2><a href="#the-components-of-an-analyzer">The Components of an Analyzer</a></h2>
<p>Inside an analyzer is a small processing pipeline consisting of the following phases: 1) Character filtering, 2) Tokenization, and 3) Token Filtering. The ultimate goal of analyzer is, of course, to convert a string into a series of tokens. An example analyzer is illustrated in the diagram below. Try and follow along with it as you read the remainder of this paragraph. The execution flow starts with a string entering into the analyzer. This string is first run through optional character filters, each of which transforms the string in a specific way, say lowercasing the text or substituting words, and outputs a transformed string. The string output of the character filters is then passed into a tokenizer, the only required component in an analyzer, which emits a list of tokens. Each token contains both a string value and a position number indicating where in the token stream it is located. Finally, these tokens are optionally passed through token filters which can further alter the tokens.
</p>
<p>
	<img src="https://api.contentstack.io/v2/assets/575e4c8c3dc542cb38c08267/download?uid=blt51e787daed39eae9?uid=blt51e787daed39eae9" data-sys-asset-uid="blt030836913b9beeae" style="max-width:100%;" alt="Analyzer Pipeline">
</p><figcaption>Analyzer Pipeline</figcaption>
<p>Elasticsearch ships with a handful of default analyzers. Custom analyzers can be configured via the settings API, at either the index or cluster level. The configuration for an example custom analyzer can be seen in the code sample below.
</p>
<pre class="prettyprint">PUT /my-index/_settings
{
  &quot;index&quot;: {
    &quot;analysis&quot;: {
      &quot;analyzer&quot;: {
        &quot;customHTMLSnowball&quot;: {
         &quot;type&quot;: &quot;custom&quot;, 
          &quot;char_filter&quot;: [
            &quot;html_strip&quot;
          ], 
          &quot;tokenizer&quot;: &quot;standard&quot;,
          &quot;filter&quot;: [
            &quot;lowercase&quot;, 
            &quot;stop&quot;, 
            &quot;snowball&quot;
          ]  
        }}}}}
</pre>
<p>The intent of the analyzer above is as follows:
</p>
<ul>
	<li>Remove all HTML tags from the source text via the <code>html_strip</code> char filter.</li>
	<li>Break up the text along word boundaries and remove punctuation with the <code>standard</code> tokenizer.</li>
	<li>Lowercase all the tokens.</li>
	<li>Remove tokens that are stopwords, such as &apos;the&apos;, &apos;and&apos; and other irrelevant common words.</li>
	<li>Stem all the tokens using the <code>snowball</code> token filter.</li>
</ul>
<p>Let&apos;s follow the data-flow of this example analyzer given the source text <code>The two &lt;em&gt;lazy&lt;/em&gt; dogs, were slower than the less lazy &lt;em&gt;dog&lt;/em&gt;.</code>. If we pipe our source text through this analyzer, it will take the path shown in the diagram that follows.
</p>
<figure>
<img src="https://api.contentstack.io/v2/assets/575e4c8c9985d58976376a3c/download?uid=bltee4e0b427d8fdad4?uid=bltee4e0b427d8fdad4" data-sys-asset-uid="blt847943826fe6865e" style="max-width:100%;" alt="A Custom Analyzer"><figcaption>A Custom Analyzer</figcaption>
</figure>
</section><section id="picking-an-analyzer" class="level2">
<h2><a href="#picking-an-analyzer">Picking an Analyzer</a></h2>
<p>The hardest part of building an analyzer is determining the correct components to use. In the following sections we&apos;ll build our familiarity with analyzers by looking at some common use cases.
</p>
<section id="searching-natural-language" class="level3">
<h3><a href="#searching-natural-language">Searching Natural Language</a></h3>
<section id="stemming" class="level4">
<h4><a href="#stemming">Stemming</a></h4>
<p>When searching bodies of natural language, things like blog posts, magazine articles, legal documents, etc., it is often good to use a <em>stemming</em> token filter, such as the included <code>stemmer</code>, <code>snowball</code>, <code>kstem</code>, or <code>porter_stem</code> token filters. These sorts of analyzers help normalize the spelling of words, for instance translating the tokens &apos;sing&apos;, &apos;sings&apos;, and &apos;singing&apos; all into the single stem &apos;sing&apos; as with the <code>snowball</code> analyzer. You can see this in action by executing the following request to test out the analyzer. Also notice the stop words &apos;they&apos; and &apos;are&apos; are dropped by the default snowball analyzer.
</p>
<pre class="prettyprint">GET http://localhost:9200/_analyze?text=I%20sing%20he%20sings%20they%20are%20singing&amp;analyzer=snowball
// Output (abbreviated)
{
  &quot;tokens&quot;: [
    {&quot;token&quot;: &quot;i&quot;, &quot;position&quot;: 1, ...},
    {&quot;token&quot;: &quot;sing&quot;, &quot;position&quot;: 2, ...},
    {&quot;token&quot;: &quot;he&quot;, &quot;position&quot;: 3, ...},
    {&quot;token&quot;: &quot;sing&quot;, &quot;position&quot;: 4, ...},
    {&quot;token&quot;: &quot;sing&quot;, &quot;position&quot;: 7, ...},
  ]
}
</pre>
<p>Stemming algorithms exploit the fact that it is usually rare that searches need to distinguish between a search for &quot;singing&quot; and &quot;sings&quot;. It may, however be the case that such differences <em>do</em> matter. As we&apos;ll see, for searches involving content like human names, product names, etc, stemming can make little sense. As a simple example, the phrases &apos;fly fishing&apos; and &apos;flying fish&apos; both stem to &apos;fli fish&apos; with a snowball stemmer; a terrible result. In a travel guide this could be a real problem. Stemming is not the be all end all of tokenization strategies. In cases where exact spelling matters it may be better to do without a stemmer and use a simpler strategy, only filtering the tokens through a <code>lowercase</code> filter. Simpler strategies also lend themselves to some of the more exotic query types, like <a href="/blog/found-fuzzy-search">fuzzy queries</a>, which can provide unexpected results when coupled with complex analyzers. Finally, remember that you can combine multiple queries with tools like the <code>bool</code> and <code>dis_max</code> queries.
</p>
</section>
<section id="internationalization" class="level4">
<h4><a href="#internationalization">Internationalization</a></h4>
<p>Some languages, like German, Finnish and Korean use compounded words which can be problematic for search engines. Where in English one might use the phrase &quot;Ballet Dancer&quot;, in German the words &quot;Ballett&quot; and &quot;T&#xE4;nzerin&quot; would be combined into a single word &quot;Ballettt&#xE4;nzerin&quot;. A search for &quot;Balett&quot; would not match the the compound word unless it is split on the proper boundary. To solve this issue the words must be <em>decompounded</em> via a plugin, like the <a href="https://github.com/jprante/elasticsearch-analysis-decompound">Elasticsearch Word Decompounder Analysis Plugin</a>. This will split such words up along those boundaries.
</p>
</section>
<section id="optimizing-phrase-searches-with-shingles" class="level4">
<h4><a href="#optimizing-phrase-searches-with-shingles">Optimizing Phrase Searches with Shingles</a></h4>
<p>In an application where phrases are frequently used in queries, the use of a <code>shingle</code> analyzer can both boost performance and improve result quality. Whereas the analyzers we&apos;ve seen up to now break up words on either letter or word boundaries, the shingle analyzer takes a piece of source text and creates groups of phrases. It is, essentially, an nGram token filter that works at the word level instead of the character level. So the sentence &quot;Beware the Ides of March&quot;, processed using a shingle token filter with <code>max_shingle_size: 3</code> and <code>min_shingle_size: 1</code>, would yield the tokens &quot;Beware&quot;, &quot;Beware the&quot;, &quot;Beware the Ides&quot;, &quot;the&quot;, &quot;the Ides&quot;, &quot;the Ides of&quot;, and so on and so forth.
</p>
<p>Generating shingles increases the amount of storage and memory needed for indexes, but comes with two benefits. First, performance for phrase queries can be significantly improved since the phrase can be matched against a single multi-word term, rather than having to be the result of multiple AND-ed terms. Second, scoring can be calculated more accurately since the <a href="/blog/found-similarity-in-elasticsearch">TF/IDF similarity</a> of the actual phrase, rather than the individual terms will yield a more accurate result. For more on searching with shingles, check out this post titled <a href="/blog/searching-with-shingles"><em>search with shingles</em></a> from the Elasticsearch Blog, <a href="http://www.hathitrust.org/blogs/large-scale-search/slow-queries-and-common-words-part-2"><em>Slow Queries and Common Words</em></a> from the Large Scale Search blog, and <a href="http://stackoverflow.com/a/8590700/11105">this stack overflow response from Lucene developer Robert Muir</a>.
</p>
<p>While shingles can provide a massive performance boost, they can also cause some performance problems, primarily related to the fact that there are more unique combinations of words than individual words. Setting <code>omit_term_freq_and_positions</code> to <code>true</code> for a shingle field can greatly reduce the size of an index and speed up searches. Since a phrase search usually matches only one shingle, there isn&apos;t a huge amount of value in maintaining position info. That being said, longer phrase searches will become impossible without this data, so be sure that the appropriate shingle size is set.
</p>
</section>
</section>
</section><section id="searching-textual-tokens" class="level2">
<h2><a href="#searching-textual-tokens">Searching Textual Tokens</a></h2>
<p>Some types of text are harder to classify. Usernames on a social site, category names on a shopping site, tags for blog posts, and other such items require different strategies from the approaches to natural language we previously discussed. Techniques such as stemming can lead to false positives, and even merge terms that should be kept separate. In these cases it&apos;s often important to preserve the uniqueness of the tokens.
</p>
<section id="searching-tokens-exactly" class="level3">
<h3><a href="#searching-tokens-exactly">Searching Tokens Exactly</a></h3>
<p>The simplest strategy for dealing with such tokens is to disable analysis completely by specifying <code>&quot;index&quot;: &quot;not_analyzed&quot;</code>, in the mapping for that field. The data will be as exact matches for the text as stored in the document. As an example of using a <code>not_analyzed</code> field, let&apos;s assume we have a CMS with various roles for various users: &quot;writer&quot;, &quot;publisher&quot;, and &quot;admin&quot;. We may want to be able to search users by role. In this case the user interface will probably have a dropdown select box that can be used to pick an exact user type, eliminating the possibility of typos, or much variation in input. This is an ideal situation to specify the field as <code>not_analyzed</code>.
</p>
</section>
<section id="searching-tokens-imprecisely" class="level3">
<h3><a href="#searching-tokens-imprecisely">Searching Tokens Imprecisely</a></h3>
<p>In the previous example the search use case was perfectly suited to the constrained input imposed by the GUI. Such a system would fare poorly when faced with more freeform user input. Usernames are a common use case that serves well as an illustration of this point. In a username search, once again stemming algorithms make little sense. We plainly do need some analysis however. At the very least, a query for &quot;CmdrTaco&quot; should match the user &quot;cmdrtaco&quot;, in other words a <code>lowercase</code> token filter seems to be in order at the very minimum. We might also want to show the user a list of best possible matches, it&apos;d be nice if a search for &apos;cmrdtaco&apos; still matched, handling misspellings would be a great feature here.
</p>
<p>For a username search such as we&apos;ve described there are two approaches that are good starting points. A trivial analyzer composed of a <code>keyword</code> tokenizer, which emits the entire input stream as a token, combined with only a <code>lowercase</code> token filter, will provide the required output. This output will handle <code>match</code>, <code>prefix</code>, and <code>match</code> queries with a <code>fuzziness</code> value provided well. For more information on fuzzy searches specifically, see <a href="/blog/found-fuzzy-search">How to Use Fuzzy Searches in Elasticsearch</a>.
</p>
<section id="using-ngrams-for-advanced-token-searches" class="level4">
<h4><a href="#using-ngrams-for-advanced-token-searches">Using NGrams for Advanced Token Searches</a></h4>
<p>An additional strategy might be to index usernames as with an <code>nGram</code> or <code>edgeNGram</code> tokenizer. These tokenizers break up text into configurable sized tuples of letters. For instance, the word &quot;news&quot;, run through a <code>min_gram:1, max_gram:2</code> <code>nGram</code> tokenizer would be broken up into the tokens &quot;n&quot;, &quot;e&quot;, &quot;w&quot;, &quot;s&quot;, &quot;ne&quot;, &quot;ew&quot;, and &quot;ws&quot;. This sort of analysis does really well when it comes to imprecise matching. Remember that by default a <code>match</code> query will analyze the input query, then get a list of terms back out, and finally run a giant &quot;OR&quot; query of the terms involved. This means that the items with the largest number of matching terms will be returned. While this does mean a large number of false positives will be returned, it also means that the most relevant results will be at the top.
</p>
</section>
<section id="using-phonetic-analyzers" class="level4">
<h4><a href="#using-phonetic-analyzers">Using Phonetic Analyzers</a></h4>
<p>Phonetic analyzers are a powerful tool for dealing with things like real names and usernames. These can be used by installing the <a href="https://github.com/elasticsearch/elasticsearch-analysis-phonetic"><code>elasticsearch-analysis-phonetic</code></a> plugin. The goal of a phonetic analyzer, like <code>metaphone</code> or <code>soundex</code> is to convert the source text into a series of tokens that represent syllabic sounds. This lets users find things that sound <em>like</em> the query text. For instance, when searching a list of names, one might want a query for the user with the name &apos;schmidt&apos; to be matched if a user searches for the term &apos;terminates&apos;. The spelling difference doesn&apos;t matter because both words sound the same.
</p>
<p>An example of a search with a metaphone component can be seen in 
	
	this example on Play
	
	. Notice that the search for &apos;schmit&apos; matches both the names &apos;schmidt&apos; and &apos;schmitt&apos;. The use of an exact matching &apos;not_analyzed&apos; field is critical for ensuring that exact matches always come first. In the second query in the example, where there is an exact match for &apos;schmidt&apos;, the exact match winds up being scored much higher to ensure its primacy in the results. For a more complete treatment of this sort of search see this 
	<a href="http://elasticsearch-users.115913.n3.nabble.com/help-needed-with-the-query-td3177477.html#a3178856">forum thread on name searches</a> by Elasticsearch developer Clinton Gormley.
</p>
</section>
</section>
</section><section id="combining-multiple-approaches" class="level2">
<h2><a href="#combining-multiple-approaches">Combining Multiple Approaches</a></h2>
<p>There&apos;s no reason why only a single approach need be taken for a given search. Using Elasticsearch&apos;s <code>multi_field</code> option one may index a single <code>_source</code> field multiple times, as shown in the example below, taken from the 
	
	metaphone Play
	
	 mentioned above.
</p>
<pre class="sourceCode json prettyprint">// Document mapping
{
  &quot;properties&quot;: {
    &quot;name&quot;: {
      &quot;fields&quot;: {
        &quot;name_metaphone&quot;: {
          &quot;type&quot;: &quot;string&quot;, 
          &quot;analyzer&quot;: &quot;mf_analyzer&quot;
        }, 
        &quot;name_exact&quot;: {
          &quot;index&quot;: &quot;not_analyzed&quot;, 
          &quot;type&quot;: &quot;string&quot;
        }
      }, 
      &quot;type&quot;: &quot;multi_field&quot;
    }
  }
}
</pre>
<p>In this example we can search either the <code>name_metaphone</code> or <code>name_exact</code> fields, to get different results for the same incoming data. There&apos;s no reason why one must restrict oneself to a single approach, and by using <code>bool</code> and <code>dis_max</code> queries, multiple query types may be combined for your specific dataset.
</p>
</section><section id="going-forward" class="level2">
<h2><a href="#going-forward">Going Forward</a></h2>
<p>There&apos;s much more to cover on this topic! If you&apos;d like to learn more, check out the second article in this series: <a href="/blog/found-text-analysis-part-2">All About Analyzers, Part 2</a>. Becoming proficient with analyzers requires a lot of experimentation, and as far as that goes there&apos;s no quicker way to play around with different analyzers than Play, our elasticsearch &apos;fiddle&apos;. I, in fact, used it to test out the examples in this article. 
	
</p>
</section></body></html>

All About Analyzers, Part One

<html><head></head><body><p><strong>UPDATE:</strong> This article refers to our hosted Elasticsearch offering by an older name, Found. Please note that Found is now known as Elastic Cloud.
</p>
<p>Elasticsearch&apos;s Fuzzy query is a powerful tool for a multitude of situations. Username searches, misspellings, and other funky problems can oftentimes be solved with this unconventional query. In this article we clarify the sometimes confusing options for fuzzy searches, as well as dive into the internals of Lucene&apos;s FuzzyQuery.
</p><section id="introduction" class="level2">
<h2><a href="#introduction">Introduction</a></h2>
<p>Searching natural language is inherently imprecise. Since computers can&apos;t comprehend natural language, there are a plethora of approaches to search, each with its own advantages and drawbacks. Lucene, the technology underlying Elasticsearch, is a swiss-army knife composed of many text processing tools. Each tool within it is a heuristic, an algorithmic shortcut in lieu of true linguistic comprehension. Some of these tools, like the Snowball stemmer and the Metaphone phonetic analyzer, are quite sophisticated. These tools mimic grammatical and phonetic aspects of language comprehension respectively. Other tools are very basic, like the prefix query type, which simply matches the beginning letters of words. Fuzzy queries sit somewhere in the middle of this toolchest in terms of sophistication; they find words that need at most a certain number of character modifications, known as &apos;edits&apos;, to match the query. For instance, a fuzzy search for &apos;ax&apos; would match the word &apos;axe&apos;, since only a single deletion, removing the &apos;e&apos;, is required to match the two words.
</p>
</section><section id="a-simple-fuzzy-match" class="level2">
<h2><a href="#a-simple-fuzzy-match">A Simple Fuzzy Match</a></h2>
<p>Fuzzy queries can most easily be performed through additional arguments to the <code>match</code> query type, as seen in the example below this paragraph. In the example, the final request, a search for &apos;vaacuum&apos;, which has an extra A, should still turn up the &apos;Vacuum&apos; product. The <code>fuzziness</code> argument specifies that the results match with a maximum edit distance of 2. It should be noted that <code>fuzziness</code> should only be used with values of 1 and 2, meaning a maximum of 2 edits between the query and a term in a document is allowed. Larger differences are far more expensive to compute efficiently and are not processed by Lucene. The official documentation still refers to setting fuzziness to float values, like 0.5, but these values are in-fact deprecated, and are harder to reason about.
</p>
<pre class="prettyprint"># Create the index
PUT /fuzzy_products
# Create the product mapping
PUT /fuzzy_products/product/_mapping
{
  &quot;product&quot;: {
    &quot;properties&quot;: {
      &quot;name&quot;: {
        &quot;type&quot;: &quot;string&quot;,
        &quot;analyzer&quot;: &quot;simple&quot;
      }
    }
  }
}
# Upload some documents
PUT /fuzzy_products/product/1
{&quot;name&quot;: &quot;Vacuum Cleaner&quot;}
PUT /fuzzy_products/product/2
{&quot;name&quot;: &quot;Turkey Baster&quot;}
# Perform a fuzzy search!
POST /fuzzy_products/product/_search
{
  &quot;query&quot;: {
    &quot;match&quot;: {
      &quot;name&quot;: {
        &quot;query&quot;: &quot;Vacuummm&quot;,
        &quot;fuzziness&quot;: 2,
        &quot;prefix_length&quot;: 1
      }
    }
  }
}
</pre>
</section><section id="determining-edit-distance" class="level2">
<h2><a href="#determining-edit-distance">Determining Edit Distance</a></h2>
<p>The metric used by fuzzy queries to determine a match is the Damerau-Levenshtein distance formula. Simply put, the Damerau-Levenshtein distance between two pieces of text is the number of insertions, deletions, substitutions, and transpositions needed to make one string match the other. As an example, the Levenshtein distance between the words &#x201C;ax&#x201D; and &#x201C;axe&#x201D; is 1 due to the single deletion required.
</p>
<p>The Damerau-Levenshtein distance formula is a modification of the classic Levenshtein distance formula, altering it by adding transposition as a valid operation. Both formulas are supported, with Damerau-Levenshtein being the default, and classic Levenshtein being selectable by setting <code>transpositions</code> to false in the query. The utility of transpositions can be seen in the case of comparing the strings &apos;aex&apos; and &apos;axe&apos;. When using the classic Levenshtein distance formula, &apos;aex&apos; is not one, but two edits away; the &apos;e&apos; must be deleted, after which a new &apos;e&apos; is inserted in the proper place, while in Damerau-Levenshtein, a single operation, swapping the &apos;e&apos; and the &apos;x&apos;, suffices. Extrapolating out from this, using classic Levenshtein would mean that &apos;aex&apos; is as far away from &apos;axe&apos; as &apos;faxes&apos; is; an example showing why Damerau-Levenshtein makes greater intuitive sense in most cases.
</p>
<p>When dealing with fuzzy searches in particular, it is vital to understand that in elasticsearch text is first run through an <a href="http://exploringelasticsearch.com/book/searching-data/analysis.html">analyzer</a> before being made available for search. When data is indexed it is processed into what are known as &apos;terms&apos;, the actual searchable units in the database. It is the analyzed terms (what terms are is covered in <a href="/blog/found-elasticsearch-from-the-bottom-up">Elasticsearch from the Bottom Up</a>), not the actual stored documents that are searched. That means that when performing fuzzy queries, the query text may be compared to an unanticipated term value as a result of analysis, leading to sometimes confusing results. This also means that if <a href="/guide/en/elasticsearch/reference/current/analysis-synonym-tokenfilter.html">synonyms</a> are enabled on a field the synonyms may be matched, even if that word does not appear at all in the source text. For instance, if one were to use a fuzzy query over an ngram analyzed field, the results would likely be bizarre, as ngrams break words up into many small letter combinations, many of which are only an edit or two away, though the actual words involved are quite dissimilar. This also means that if using say, a snowball analyzer, a fuzzy search for &apos;running&apos;, will be stemmed to &apos;run&apos;, but will not match the misspelled word &apos;runninga&apos;, which stems to &apos;runninga&apos;, because &apos;run&apos; is more than 2 edits away from &apos;runninga&apos;. This can cause quite a bit of confusion, and for this reason, it often makes sense only to use the <code>simple</code> analyzer on text intended for use with fuzzy queries, possibly disabling synonyms as well. To clarify this, a diagram showing a fuzzy query running against a snowball analyzed document can be seen below.
</p>
<p>
	<img src="https://api.contentstack.io/v2/assets/575e4c8b43e9adc538713ab9/download?uid=blt19ff87a32a904536?uid=blt19ff87a32a904536" data-sys-asset-uid="blt9f4c5c9fe47cabdb" style="max-width:100%;" alt="A fuzzy query using a snowball analyzer">
</p><figcaption>A fuzzy query using a snowball analyzer</figcaption>
</section><section id="the-different-types-of-fuzzy-searches" class="level2">
<h2><a href="#the-different-types-of-fuzzy-searches">The Different Types of Fuzzy Searches</a></h2>
<p>Multiple types of fuzzy search are supported by elasticsearch and the differences can be confusing. The list below attempts to disambiguate these various types.
</p>
<ul>
	<li><strong><a href="/guide/en/elasticsearch/reference/current/query-dsl-match-query.html"><code>match</code></a> query + <code>fuzziness</code> option:</strong> Adding the <code>fuzziness</code> parameter to a <code>match</code> query turns a plain match query into a fuzzy one. Analyzes the query text before performing the search.</li>
	<li><strong><a href="/guide/en/elasticsearch/reference/current/query-dsl-fuzzy-query.html"><code>fuzzy</code></a> query:</strong> The elasticsearch <code>fuzzy</code> query type should generally be avoided. Acts much like a <code>term</code> query. Does <em>not</em> analyze the query text first.</li>
	<li><strong><a href="/guide/en/elasticsearch/reference/current/query-dsl-flt-query.html"><code>fuzzy_like_this</code></a>/<a href="/guide/en/elasticsearch/reference/current/query-dsl-flt-field-query.html"><code>fuzzy_like_this_field</code></a>:</strong> A <a href="/guide/en/elasticsearch/reference/current/search-more-like-this.html"><code>more_like_this</code></a> query, but supports fuzziness, and has a tuned scoring algorithm that better handles the characteristics of fuzzy matched results.*</li>
	<li><strong><a href="/guide/en/elasticsearch/reference/current/search-suggesters.html"><code>suggesters</code></a>:</strong> Suggesters are not an actual query type, but rather a separate type of operation (internally built on top of fuzzy queries) that can be run either alongside a query, or independently. Suggesters are great for &apos;did you mean&apos; style functionality.</li>
</ul>
<p>A <code>match</code> query with the <code>fuzziness</code> parameter set is perhaps the most versatile of the fuzzy queries. The <code>fuzzy</code> query type supports the exact same behavior, except it does not allow for any analysis for the query text. Additionally, the <code>fuzzy</code> query type is a subset of the functionality of a <code>match</code> query makes it more confusing than useful.
</p>
<p>The <code>fuzzy_like_this</code>, or FLT, queries are useful for providing recommendations based on a large piece of source text that may contain misspellings or other inaccuracies separated by edit distance. There are two different queries in this category <code>fuzzy_like_this</code> and <code>fuzzy_like_this_field</code>, which both provide the same functionality, the latter query simplifying the syntax for the case where only a single field is used. These queries take a parameter, <code>like_text</code>, consisting of a large piece of text, say the body of an article, and try to find documents &apos;like&apos; that one. The text in the article is checked, and query terms are weighted based their frequency in the <code>like_text</code> with special tweaks (disabling the coord factor, source text based IDF) for combining the various terms from the source text. FLT queries work best for cases where the corpus contains a large number of misspellings, otherwise a standard <code>more_like_this</code> query will have better performance. More detail can be found on the <a href="/guide/en/elasticsearch/reference/current/query-dsl-flt-query.html">official documentation for Fuzzy More Like This</a>.
</p>
<p>Suggesters are fantastic in the case where a user types in &apos;New Yrok&apos;, for a search, having intended to type in &apos;New York&apos;. Applications can suggesters to present a &apos;did you mean&apos; box in the UI recommending a potential correction. Beware that for the specific case of type-ahead search the <a href="/guide/en/elasticsearch/reference/current/search-suggesters-completion.html">completion API</a> offers better performance given the latency sensitive nature of type-ahead searches. More information on suggesters can be found on the <a href="/guide/en/elasticsearch/reference/current/search-suggesters.html"><code>search suggesters</code></a> page.
</p>
</section><section id="performance-considerations" class="level2">
<h2><a href="#performance-considerations">Performance Considerations</a></h2>
<p>Even though Lucene&apos;s Levenshtein distance implementation is state of the art, and quite fast, it is still much slower than a plain <code>match</code> query. The runtime of the query grows with the number of unique terms in the index. That is to say, when performing a fuzzy search the main criteria is not how many documents will be returned, but how many unique terms across the cluster there are for the fields being searched. If there are 100 documents with 10,000 unique words apiece, searching that index will be slower than searching 10,000 documents where the field being searched only has 100 unique words.
</p>
<p>The primary reason for this slowness is that a standard match query can quickly check the terms index, an internal datastructure used by Lucene, for a match and find documents extremely quickly using binary search. This process is fast even for large dictionaries since binary searches scale well. Fuzzy queries, on the other hand, use a more advanced algorithm involving a <a href="http://en.wikipedia.org/wiki/Deterministic_finite_automaton">DFA</a> which must process a large number of terms. Processing the much larger number of terms required for a fuzzy search is always slower than a simple binary search. As an example, running a fuzzy search on the English language Wikipedia dataset takes approximately 320ms for the term &#x201C;history&#x201D; given a <code>min_similarity</code> of 2, while a plain <code>match</code> search for the same term runs in a mere 35ms, an order of magnitude difference. Given a <code>min_similarity</code> of 1, the search time is only 150ms. It should be noted that these tests were not conducted in a statistically sound manner, and these times can be affected significantly by factors particular to individual applications. The Wikipedia dataset in particluar, has many more terms than a large number of common use cases.
</p>
<p>Performance can be significantly improved by requiring that matches have exact prefix matches with the query. This dramatically shortens the search space at the expense of not finding words with a misspelling toward the front. The longer the prefix length, the faster the speedup. Specifying a <code>prefix_length</code> of 1 cuts the 320ms query mentioned above down to a mere 104ms. Increasing this number yielded further speed gains. Generally, it is advisable to require a prefix for datasets with large numbers of terms or performance will be poor, oftentimes in the 100s of milliseconds.
</p>
<p>The <code>max_expansions</code> setting, which defines the maximum number of terms the fuzzy query will match before halting the search, can also have dramatic effects on the performance of a fuzzy query. Cutting down the query terms has a negative effect, however, in that some valid results may not be found due to early termination of the query. It is important to understand that the <code>max_expansions</code> query limit works at the shard level, meaning that even if set to 1, multiple terms may match, all coming from different shards. This behavior can make it seem as if <code>max_expansions</code> is not in effect, so beware that counting unique terms that come are returned is not a valid way to determine if <code>max_expansions</code> is working.
</p>
</section><section id="alternatives-to-fuzzy-matching" class="level2">
<h2><a href="#alternatives-to-fuzzy-matching">Alternatives to Fuzzy Matching</a></h2>
<p>Fuzzy matching isn&apos;t always the right tool for the job, oftentimes imprecise matches can be found through other techniques. The <a href="https://github.com/elasticsearch/elasticsearch-analysis-phonetic">phonetic analysis plugin</a> contains a number of fascinating tools for approximating matches, such as the <a href="http://en.wikipedia.org/wiki/Metaphone">metaphone</a> analyzer, which finds words that sound similar to other words. For instance, if what is required is making sure words like <code>run</code> and <code>ran</code> are both considered equivalent, a smart analyzer like a Snowball analyzer is preferable. Alternatively, for checking misspellings, N-gram analysis <a href="https://web.archive.org/web/20140209084956/http://exploringelasticsearch.com/book/searching-natural-language/searching-non-word-text.html">as described in this short tutorial</a> can run quite a bit faster at query time, depending on the dataset. N-Grams do, however, come at the cost of additional storage / memory usage, slightly more index-time processing, and a long tail of false positives after good matches.
</p>
</section><section id="more-reading" class="level2">
<h2><a href="#more-reading">More Reading</a></h2>
<ul>
	<li>For a more in-depth look at the performance of Fuzzy Queries in elasticsearch, Lucene core contributor Mike McCandless has written a fascinating <a href="http://blog.mikemccandless.com/2011/03/lucenes-fuzzyquery-is-100-times-faster.html">blog post</a> on the topic. There is also a video, <a href="http://vimeo.com/26517310">Finite State Automata in Lucene and Solr</a>, of a presentation by Dawid Weiss that covers Levenshtein and other automata in Lucene.</li>
	<li>If you would like to learn more about Levenshtein Automata, Nick Johnson has a great post <a href="http://blog.notdot.net/2010/07/Damn-Cool-Algorithms-Levenshtein-Automata">Damn Cool Algorithms: Levenshtein Automata</a>.</li>
</ul>
</section></body></html>

Articles by Andrew Cholakian

A tour of Go concurrency patterns via the new Heartbeat scheduler

Testing data shapes with go-lookslike

Logstash 6.3.0 Released

Logstash Lines for 2018-03-13

Logstash Lines for 2018-03-06

Logstash Lines for 2018-02-14

Logstash Lines for 2018-02-07

Logstash Lines: Update for January 30, 2018

Logstash Lines: Update for January 22, 2018

Products & Solutions

Company

Resources

Featured topics

News

Articles by Andrew Cholakian

A tour of Go concurrency patterns via the new Heartbeat scheduler

Testing data shapes with go-lookslike

Logstash 6.3.0 Released

Logstash Lines for 2018-03-13

Logstash Lines for 2018-03-06

Logstash Lines for 2018-02-14

Logstash Lines for 2018-02-07

Logstash Lines: Update for January 30, 2018

Logstash Lines: Update for January 22, 2018