18 February 2015

This Week in Elasticsearch - February 18, 2015

By Alexander Reelsen

Welcome to This Week in Elasticsearch. In this roundup, we try to inform you about the latest and greatest changes in Elasticsearch. We cover what happened in the GitHub repositories, as well as many Elasticsearch events happening worldwide, and give you a small peek into the future of the project.

Elasticsearch core

  • Core: Remove ability to run optimize and upgrade APIs async (#9640, 2.0.0, 1.5.0)
  • Cluster task: Ensure that we don’t pass negative timeInQueue to writeVLong() (#9662, 1.5.0, 1.4.4, 1.3.9)
  • Testing: Introduce basic validation of the REST spec (#9629, 2.0.0, 1.5.0, 1.4.4)
  • Internal: Introduce TimedPrioritizedRunnable base class to all commands that call InternalClusterService.updateTasksExecutor (#9671, 2.0.0, 1.5.0, 1.4.4)
  • Testing: Ensure RepeatOnException rule is available in test-jar (#9675, 2.0.0, 1.5.0, 1.4.4)
  • Internal: only ack index store deletion on data nodes (#9672, 2.0.0, 1.5.0)
  • REST: expose master_timeout flag on GET _template & HEAD _template (#9688, 2.0.0, 1.5.0)
  • Recovery: Handle corruptions during recovery finalization (#9619, 2.0.0, 1.5.0)
  • Store: Use Directory#fileLength() less during calculating checksums (#9689, 2.0.0, 1.5.0)
  • Aggregations: Prevent negative intervals in date_histogram (#9690, 2.0.0, 1.5.0, 1.4.4)
  • Core: Move as much as possible into abstract Engine (#9678, 2.0.0, 1.5.0)
  • Build: Add marvel and license to the release script (#9697, 2.0.0, 1.5.0, 1.4.4)
  • Internal: Removed Master from names.txt (#9698, 2.0.0, 1.5.0)
  • Internal: Remove XCollector (#9677, 2.0.0)
  • Aggregations: Clean up date_histogram (#9637, 2.0.0)
  • Engine: Move more methods into abstract Engine (#9717, 2.0.0, 1.5.0)
  • Gateway: Don’t rename recovery translog in gateway (#9719, 2.0.0)
  • Query API: Added nested scope to parse context that keeps track the current nested level during search request parsing (#9692, 2.0.0, 1.5.0)
  • Core: move InternalEngine.segmentStats() into abstract Engine (#9728, 2.0.0, 1.5.0)
  • Build: Rename Maven repository id (#9733, 2.0.0, 1.5.0)
  • Packaging: Add antlr and asm dependencies (9696, 2.0.0, 1.5.0, 1.4.4)
  • Mapping: Require the full path (without type) for field lookups (#8872, 2.0.0)
  • FileSystemUtils: Only create backup copies if files differ (#9592, 2.0.0)
  • CliTool: Add command to warn on permission/owner change (#9508, 2.0.0)
  • Aggs: Refactor aggregations to use lucene5-style collectors (#9544, 2.0.0)
  • Cat API: Change behaviour of indices segments api to allow no indices (#9219, 2.0.0, 1.5.0)
  • Dependencies: Update joda-time to v2.7 (#9610, 2.0.0, 1.5.0)
  • Allow configuration of the GC log file via an environment variable (#8479, 2.0.0)
  • Term Vectors: Requests times are now measured (#9583, 2.0.0, 1.5.0)

In Apache Lucene this Past Week

Elasticsearch Ecosystem

Here’s some more information about what is happening in the ecosystem we are maintaining around the ELK stack – that’s Elasticsearch plus Logstash and Kibana – including plugin and driver releases.

  • Kibana 4 RC1 has been released, please test and give feedback!
  • Shield 1.0.1 has been released in order to work with Elasticsearch 1.4.3.
  • Adrien Grand authored a nice blog post about roaring bitmaps.
  • From the Docker Files: Raúl Cumplido shares his tips for getting the ELK stack up and running, and how he created a Docker container for the 3 services to ensure a repeatable process for future deployments.
  • If you’re using Apache Hadoop and Spark, you won’t want to miss this post from Avi Levi on using the ELK stack for setting up a central logging infrastructure.
  • Dan Kingshott’s excellent post on open source log aggregation gives an overview of how to set up Kibana in an OpenStack cloud or in VMs, and shares how to use Chef for configuration management in this environment as well.
  • Interested in alerting for your website? Elasticsearch’s Percolator feature is your friend! Fabio Ponciroli wrote up how he and his colleagues added this functionality to their infrastructure at their company’s recent hack day.
  • An Elasticsearch full-text search integration for the dovecot IMAP server has been published.

Slides & Videos

Joe Jasinski shows how he uses Django, Elasticsearch, and Haystack for searching the contents of his website to the Chicago Python User Group.

The MongoDB meetup group in Delhi recently heard from Bharvi Dixit about some use cases for MongoDB with Elasticsearch.

The Elasticsearch meetup in London had some fantastic presentations at their last gathering.

Where to Find Us

We'd love to feature all the great Elasticsearch, Logstash, and Kibana presentations and meetups happening worldwide in this section. If you're speaking or hosting a meetup, let our Director of Developer Relations, Leslie Hawthorn, know!

Austria

The Vienna Elasticsearch Users Group will get together on March 4 to talk Elasticsearch at Cloud Foundry and more. Register now to let the organizers know you’re attending.

Germany

India

The Configuration Management Magic Meetup will convene on February 21 in Bangalore. Among the many talks on offer, you can hear all about Log Analysis using Elasticsearch, Kibana and Fluentd. Register now to save your seat.

Israel

The Tel Aviv-Yafo ELK meetup group will be talking How to Use ELK to Analyze Logs from a Large Production AWS Environment on February 24. You can register now to attend.

South Africa

The inaugural Capetown Elasticsearch Meetup will convene on March 5 to talk shop and plan for the future of the group. Register now to let the organizers know you plan to attend.

Taiwan

The Agile Code Camp team are convening a developer and designer hack day on February 23, and attendees will be developing with Elasticsearch. Register now to attend the full day event.

United Kingdom

Elasticsearch will be out in force at QCon London, which returns to the Queen Elizabeth II Conference Center this year. You can visit us at our booth on the show floor, plus we’ll be having one of our engineers take the stage for the main program. Those details are in the works, but in the meantime you can take a look at Kristoffer Dyrkorn’s talk information. He’ll be sharing the story of how Elasticsearch and other technologies are powering the Norwegian Roads Authority’s brand new system to provide real-time traffic information to travelers throughout Norway.

United States

If you’re at Strata in San Jose this week, don’t forget to join Costin Leau, creator of Elasticsearch for Apache Hadoop, at the conference. He’ll present on Search Evolved: Unraveling Your Data on Friday, Feb 20 at 11:30 AM.

Coming up in San Francisco next month, you can join the SF MySQL Meetup to hear all about using MySQL and the ELK stack for audit logging. The user group will get together on March 11, but the event is filling up quickly. Don’t miss out! Register now to save your seat.

And for our friends beyond Silicon Valley:

Where to Find You

PSST! If you’re a regular reader of This Week in Elasticsearch, a.k.a TWIES, you’re thinking of skipping this section. You may even be thinking to yourself, yes of course I will drop a note on Twitter when I am giving a talk on all things ELK. That’s awesome, because we’d like to showcase every meetup, conference presentation and workshop on Elasticsearch, Logstash, and Kibana happening worldwide. And now, we’ve made it even easier for you to get support for your meetup!

Head on over to our meetups page! (And we’ll still totally send you swag if you’re giving a talk on anything ELKy at a conference.)

Oh yeah, we're also hiring. If you'd like us to find you for employment purposes, just drop us a note. We care more about your skill set and passion for Elasticsearch, Kibana, and Logstash than where you rest your head.

Trainings

If you are interested in Elasticsearch training we have courses taught by our core developers coming up in:

  • Amsterdam – February 19, 2015 (Core Elasticsearch Training)
  • London – February 25, 2015 (Hands on Workshop)
  • London – February 26, 2015 (Core elasticsearch)
  • Bangalore – February 25, 2015 (Hands on Workshop)
  • Bangalore – February 26, 2015 (Core Elasticsearch Training)
  • New Delhi – March 17, 2015 (Core Elasticsearch Training)
  • San Francisco – March 17, 2015 (Hands on Workshop)
  • Mountain View – March 18, 2015 (Core Elasticsearch Training)
  • Northern Virginia – March 24, 2015 (Core Elasticsearch Training)
  • Munich – March 24, 2015 (Core Elasticsearch Training)
  • Munich – March 26, 2015 (Hands on Workshop)
  • Stockholm – March 25, 2015 (Core Elasticsearch Training)
  • Los Angeles – March 25, 2015 (Core Elasticsearch Training)
  • Paris – March 25, 2015 (Hands on Workshop)
  • Paris – March 26, 2015 (Core Elasticsearch Training)
  • San Francisco – April 7, 2015 (Core Elasticsearch Training)
  • Seoul – April 8, 2015 (Core Elasticsearch Training)
  • Tokyo – April 13, 2015 (Core Elasticsearch Training)
  • Amsterdam – April 14, 2015 (Hands on Workshop)
  • Austin – April 23, 2015 (Core Elasticsearch Training)
  • New York – April 28, 2015 (Hands on Workshop)
  • New York – April 29, 2015 (Core Elasticsearch Training)