14 January 2015

This Week in Elasticsearch - January 14, 2015

By Alexander Reelsen

Welcome to This Week in Elasticsearch. In this roundup, we try to inform you about the latest and greatest changes in Elasticsearch. We cover what happened in the GitHub repositories, as well as many Elasticsearch events happening worldwide, and give you a small peek into the future of the project.

Elasticsearch core

  • Update API: Removed parent parameter from update request (#9163, 2.0.0)
  • Internal: Assert transport threads do not call blocking code from (#9164, 2.0.0, 1.5.0)
  • Indices API: Ensure GET Index API is consistent with documentation (#9178, 2.0.0, 1.5.0, 1.4.3)
  • Core: Verify the index state of concrete indices after alias resolution (#9057, 2.0.0, 1.5.0, 1.4.3)
  • Core: Fold engine into IndexShard (#9181, 2.0.0, 1.5.0)
  • Mapping API: Remove includeExisting flag from adding <code>ObjectMapper and FieldMapper listeners (#9184, 2.0.0, 1.5.0)
  • Core: Add _shards header to all write responses (#7994, 2.0.0)
  • Mappings: Remove allow_type_wrapper setting (#9185, 2.0.0)
  • Core: Simplify Engine construction and ref counting (#9211, 2.0.0, 1.5.0)
  • Aggregations: Numeric metric can now be formatted (#9032, 2.0.0, 1.5.0)
  • Dependencies: Upgrade to Lucene 5 r1650327 (#9206, 2.0.0)
  • Recovery: Be more resilient to partial network partitions (#8720, 2.0.0, 1.5.0)
  • Core: Fix Store.checkIntegrity() for Lucene 3.x index files. (#9234, 1.5.0, 1.4.3, 1.3.8)
  • Windows: makes elasticsearch.bat more friendly to automated processes (#9160, 2.0.0, 1.5.0, 1.4.3)
  • REST API: Remove status code from main ´/´ action (#8865, 2.0.0)
  • Index lifecycles: Pass index settings to IndicesLifecycle#beforeIndexCreated and <code>#afterIndexShardClosed (#9245, 2.0.0, 1.5.0)
  • Upgrade API: Log when upgrade starts and stops (#9229, 2.0.0, 1.5.0, 1.4.3)
  • Mappings: Remove fieldSearchAnalyzer and <code>fieldSearchQuoteAnalyzer from MapperService (#9262, 2.0.0)
  • Mapping API: Using default=null for <code>_timestamp field creates a index loss on restart (#9233, 2.0.0, 1.5.0, 1.4.3)
  • Java API: Make package private getters public if there have corresponding public setters (#9273, 2.0.0, 1.5.0)
  • Aggregations: Add aimplifications (#9097, 2.0.0)
  • Internal: Change snapshot state for unreleased versions and add validation tests for constants (#9228, 2.0.0, 1.5.0, 1.4.3)

In Apache Lucene this past week

  • Lucene's ConcurrentMergeScheduler now auto-throttles the merge IO MB/sec by default so merges won't steal all available IO. The throttle adapts up and down according to the incoming indexing rate.
  • Sortcomparator APIs now work per-segment, just like the Collector API. This is an API change only, laying the ground work for future optimizations depending on each segment.
  • A new boolean spins(Path) attempts to determine whether a path resides on a non-spinning solid-state disk for better dynamic defaults.
  • Performance of resolving deleted terms to their Lucene doc ids has improved substantially due to a silly now-fixed performance bug.
  • Cutover addIndexes(IndexReader...) to a <a href="https://issues.apache.org/jira/browse/LUCENE-6165">new <code>CodecReader API, laying the ground work for future merging / upgrading optimizations, and also making the index splitters much faster.
  • In the latest Java 9 build, Lucene lost the Java API to fsync a directory on Unix, which breaks the durability of commit. Lucene committers are working with Oracle to try to find a solution.

Elasticsearch Ecosystem

Here's some more information about what is happening in the ecosystem we are maintaining around the ELK stack - that's Elasticsearch plus Logstash and Kibana - including plugin and driver releases.

  • Peter Kim shares his tips on figuring out the amount of disk required for an Elasticsearch deployment.
  • If you're a fan of Google Compute Engine, you'll be thrilled to hear that Google has announced the release of Click to Deploy for Elasticsearch on GCE. It's a great way to set up an Elasticsearch cluster, configured as you'd like, in just a few clicks.
  • You love us... you really love us! And you love open source, too. Check out this post from Wired talking about the fastest growing databases of 2014.
  • Yuriy Gerasimov has a guide for centralizing logs with Logstash -- and doing so using pre-built Docker images, great for developers wanting to get started quickly.
  • Users of the Apache Camel Elasticsearch component will love this post on simplifying index management from Umberto Nicoletti.
  • This great post from Njal Karevoll at Found is perfect for those of you getting started with Elasticsearch -- giving an overview of the different client types available, and how to choose which to pick.
  • Or, if you'd like a different way to get started with Elasticsearch, oliver006 (an international person of mystery!) has not one, but TWO ways to do so: using Elasticsearch to index and search Hacker News, and using Elasticsearch to index your Gmail inbox.

Slides & Videos

Gaspar Muñoz and Santiago Mola spoke at the Madrid Elasticsearch meetup on December 15. The video from their talk -- On-the-fly ETL con EFK: ElasticSearch, Flume y Kibana -- is now available, en español!

At one the December gathering of the Elasticsearch London MeetUp group, Ian Plosker - CTO & Co-founder at Orchestrate.io - shared some great real-world lessons in his talk "Schemalessness Gone Wrong: Improving Elasticsearch with 'Tuplewise Transform'"

Where to find Us

We'd love to feature all the great Elasticsearch, Logstash, and Kibana presentations and meetups happening worldwide in this section. If you're speaking or hosting a meetup, let our Director of Developer Relations, Leslie Hawthorn, know!


The Vienna Elasticsearch user group will be doing a joint meetup with the Vienna DB meetup group on Tuesday, January 20 (in Vienna, of course!). Karel Minarik and Martijn van Groningen of Elasticsearch will both be presenting. Save your spot and get signed up!


If you're headed out to FOSDEM (Europe's largest gathering of open source developers!) in Brussels at the end of January, be sure to squeeze time into your schedule for the BeLux (Belgium / Luxembourg) Elasticsearch user group meetup on Friday, January 30. Can't make it? Catch us at the Elasticsearch booth, or visit the Open Source Search devroom track at FOSDEM.


Our partner Intrafind is organizing an "Elasticsearch Expert Talk" on January 28 in Munich starting at 1pm. You can register using this link.
The Search Meetup Karlsruhe group will be gathering on January 29 for a presentation from Nico Heid on the ELK stack. Check out their meetup page for more information and to register for the event.

New Zealand

Leslie Hawthorn and Mark Walkom will both be speaking at linux.conf.au (also known as LCA) in Auckland. This fantastic event is happening now, and runs through January 16, and is definitely worth attending!

The Netherlands

Come to the CRI Service Kennissessie in Houten on January 15 to learn about enterprise logging with Elasticsearch, rsyslog, Docker, and Kibana. Save your place by registering for the meetup today.

United Arab Emirates

The Hadoop User Group is gathering in Dubai on January 27 for a meetup. Elasticsearch's own David Pilato will be joining via video conference to give a presentation titled "Make Sense of your (BIG) data!" Get registered today to save your spot.

United Kingdom

If you couldn't decide between the Bath Ruby Battlebot event and the upcoming Elasticsearch meetup, we've got great news: The Elasticsearch meetup has changed dates, so now you can attend both! Get signed up today for this meetup in Bath, now on January 22. There is still a remaining speaking slot, too, so be sure to let the organizer know if you're interested.

United States

The South Shore .NET user group in Plymouth, Massachusetts, is getting together on January 29 for a presentation on Elasticsearch for data mining. This meetup will be covering Elasticsearch basics and a few other areas, and have you ready to add Elasticsearch to your data analysis toolkit. Sign up to save your spot!

Where to Find You

PSST! If you're a regular reader of This Week in Elasticsearch, a.k.a TWIES, you're thinking of skipping this section. You may even be thinking to yourself, yes of course I will drop a note on Twitter when I am giving a talk on all things ELK. That's awesome, because we'd like to showcase every meetup, conference presentation and workshop on Elasticsearch, Logstash, and Kibana happening worldwide. And now, we've made it even easier for you to get support for your meetup!

Head on over to our meetups page! (And we'll still totally send you swag if you're giving a talk on anything ELKy at a conference.)

Oh yeah, we're also hiring. If you'd like us to find you for employment purposes, just drop us a note.  We care more about your skill set and passion for Elasticsearch, Kibana, and Logstash than where you rest your head.


If you are interested in Elasticsearch training we have courses taught by our core developers coming up in:

  • Bangalore - January 19, 2015 (Getting Started Workshop)
  • Bangalore - January 20, 2015 (Core Elasticsearch Training)
  • Paris - January 21, 2015 (Getting Started Workshop)
  • Paris - January 22, 2015 (Core Elasticsearch Training)
  • London - January 29, 2015 (Core Elasticsearch Training)
  • Sydney - February 4, 2014 (Core Elasticsearch Training)
  • Sydney - February 5, 2014 (Hands-on Workshop with Elasticsearch, Logstash & Kibana)
  • San Francisco - February 4, 2014 (Core Elasticsearch Training)
  • San Francisco - February 5, 2014 (Hands-on Workshop with Elasticsearch, Logstash & Kibana)
  • Melbourne - February 9, 2014 (Core Elasticsearch Training)
  • New York - February 11, 2014 (Core Elasticsearch Training)
  • New York - February 12, 2014 (Hands-on Workshop with Elasticsearch, Logstash & Kibana)
  • Amsterdam - February 18, 2014 (Core Elasticsearch Training)
  • Amsterdam - February 19, 2014 (Hands-on Workshop with Elasticsearch, Logstash & Kibana)
  • Los Angeles - February 19, 2014 (Hands-on Workshop with Elasticsearch, Logstash & Kibana)
  • London - February 25, 2014 (Hands-on Workshop with Elasticsearch, Logstash & Kibana)