Welcome to This Week in Elasticsearch. In this roundup, we try to inform you about the latest and greatest changes in Elasticsearch. We cover what happened in the GitHub repositories, as well as many Elasticsearch events happening worldwide, and give you a small peek into the future of the project.
- Internal: clean up memory reuse a bit. (#9272, 2.0.0)
- Dependencies: Upgrade to lucene-5.1.0-snapshot-1652032 (#9318, 2.0.0)
- Upgrade API: Fix version check in bytes to upgrade that spans major versions (#9340, 2.0.0, 1.5.0, 1.4.3)
- Aggregations: Post collection the children agg should also invoke that phase on its wrapped child aggs (#9291, 2.0.0, 1.5.0, 1.4.3)
- Internal: Pass through all exceptions in
IndicesLifecycleListeners(#9330, 2.0.0, 1.5.0)
- Tribe node: remove closed indices from cluster state (#9334, 2.0.0, 1.5.0, 1.4.3, 1.3.8)
- Core: Increase default translog flush size from 200mb to 512 mb (#9341, 2.0.0)
- Query cache: Queries are never cached when date math expressions are used (including exact dates) (#9269, 2.0.0, 1.5.0, 1.4.3)
- Snapshot/Restore: Add support for changing index settings during restore process (#9285, 2.0.0, 1.5.0)
- Lucene: Expose auto-IO-throttle from Lucene's
- Snapshot/Restore: Add validation of restored persistent settings (#9051, 2.0.0, 1.5.0)
- Cleanup: Remove
gateway.type: local(#9128, 2.0.0)
- Dependencies: Upgrade netty to 3.10.0.Final (#9132, 2.0.0, 1.5.0)
- Tests: Make sure snapshots created with old version of elasticsearch can be restored (#8968, 2.0.0, 1.5.0)
- Mapping API: Add
ignore_missingoption to <code>timestamp (#9104, 2.0.0, 1.5.0)
In Apache Lucene this Past Week
- The release branch for Lucene 5.0 was recently cut, and the committer team is busy preparing an RC and release. If you're interested in all the new features upcoming in the 5.0 release, our very own Mike McCandless has you covered or simply check out the changes list.
- There were a number of performance improvements committed this week, including speeding up MUST_NOT clauses for Boolean Queries, resolution of deleted docIDs and bulk copying of raw compressed bytes on field merge.
- Lucene's ease of use improved with the removal of out-of-order scoring and the addition of the new CustomAnalyzer class. The CustomAnalyzer class makes it easy to build up an arbitrary analyzer by appending tokenizer and token filters using the analysis factory APIs.
- The committer team is working closely with Oracle and the OpenJDK development team to address breakage to fsyncing directory file descriptors introduced with a recent revision of Java 9.
Here's some more information about what is happening in the ecosystem we are maintaining around the ELK stack - that's Elasticsearch plus Logstash and Kibana - including plugin and driver releases.
"Cloud years are functionally equivalent to dog years": ICYMI How we built ElasticSearch cluster on #AWS http://t.co/JDtZ8Y5mOi #opensource
— Mingle (@thatsmingle) January 20, 2015
- Zachary Tong authored a tutorial to help you get the most out of Elasticsearch as an analytics engine: Introduction to Aggregations. A must read for anyone looking to get the most out of their Elasticsearch deployments beyond full-text search, and the first in a series of post on Elasticsearch and Analytics. Stay tuned to this blog for more!
- We love sharing our user's success stories. Check out this post from the folks at TrackJS on how the scaled up their Elasticsearch infrastructure, 'on a budget'. You might also want to register for our upcoming webcast where the TrackJS will folks will give you even more detail on scaling Elasticsearch, fast!, for fun and profit.
- Kent Anderson shared a handy how to, including configurations for Ubuntu servers, in his post Monitoring Web Service Performance with the ELK Stack.
- Nikos Fertakis shared an article on how Skroutz - a Greek price comparison website serving up details on more than 7.5M products - uses ELK to make their customers' lives better. The entire deep dive is quite useful, but the infrastructure architecture bits and their use of scripted aggregations is particularly interesting.
I have to say @PaketManager does a lot of things right! Moving NEST over was a pretty smooth experience https://t.co/FsXI2j31QM
— Martijn Laarman (@Mpdreamz) January 20, 2015
Slides & Videos
Mark Warkolm introduces the ELK stack at the recent linux.conf.au conference
Phil Wills from TheGuardian.com on how they've scaled to 1M unique browser per month using Elasticsearch & Scala
Levi Reich from Thomson Reuters on their Elasticsearch use case, and our very own Peter Kim introduces the ELK stack
Where to find Us
We'd love to feature all the great Elasticsearch, Logstash, and Kibana presentations and meetups happening worldwide in this section. If you're speaking or hosting a meetup, let our Director of Developer Relations, Leslie Hawthorn, know!
Q&A with speakers and @elasticsearch team at @SeattleNode meetup! pic.twitter.com/rZHGEYux6R
— Ryan Roemer (@ryan_roemer) January 21, 2015
Leslie Hawthorn and Mark Warkolm will be visiting the Melbourne Search Users Group tomorrow. You'll hear from them on an Introduction to the ELK stack and how to contribute to the ELK community. Mark Wallis from Lexer will also be presenting, discussing Search Interfaces and User Experience. You can still register to attend for this Jan 21 meetup.
- If you're headed out to FOSDEM (Europe's largest gathering of open source developers!) in Brussels at the end of January, be sure to squeeze time into your schedule for the BeLux (Belgium / Luxembourg) Elasticsearch user group meetup on Friday, January 30.
- Can't make it the meetup? Catch us at the conference at the Elasticsearch booth, or visit us in the Open Source Search Dev Room.
- Right after FOSDEM, we're heading to Ghent for Configuration Management Camp. Leslie Hawthorn & Robyn Bergeron will see you in the hallway track between talks!
The 12th Elasticsearch Paris Meetup is on in Paris tomorrow night, but we're currently sold out. You can still add yourself to the wait list. We'll let folks know if spaces open up tomorrow.
- Britta Weber will present on the ELK stack at the Berlin User Group Meeting on January 27. You can register here to save your spot.
- In Cologne instead of Berlin on January 27? No worries! The Trefpunkt Semicolon User Group will be talking Applications for Elasticsearch. RSVP here.
- Our partner Intrafind is organising an "Elasticsearch Expert Talk" on January 28 in Munich starting at 1pm. You can register using this link.
- The Search Meetup Karlsruhe group will be gathering on January 29 for a presentation from Nico Heid on the ELK stack. Check out their meetup page for more information and to register for the event.
- Alexander Reelsen will talk about ELK stack use cases at the OOP Konferenz in Munich next Thursday, Jan 29. Ping him if you'd like to meet up!
- The Search Meetup Munich group will get together on Feb 5 for a special Elasticsearch edition: Oliver Eilhard on his Elasticsearch Go Client, Alexander Reelsen on the Percolator, plus Q&A with our CTO, Shay Banon. Register now to save your seat.
United Arab Emirates
The Hadoop User Group is gathering in Dubai on January 27 for a meetup. Elasticsearch's own David Pilato will be joining via video conference to give a presentation titled "Make Sense of your (BIG) data!" Get registered today to save your spot.
- If you couldn't decide between the Bath Ruby Battlebot event and the upcoming Elasticsearch meetup, we've got great news: The Elasticsearch meetup has changed dates, so now you can attend both! Get signed up today for this meetup in Bath, now on January 22. Our very own Colin Goodheart-Smithe will be joining for Q&A.
- Heading to Monki Gras before FOSDEM? So are we! Alan Hardy will be attending talks and would love to talk all things Elasticsearch & the ELK stack with you in the hallway track. May sure to say hello!
- The January London Elasticsearch Meetup is already full up, but you can still join the waitlist. The group convenes on January 29 to talk the ELK stack on AWS, Elasticsearch for Lead Generation and Building UIs with Search Templates.
- If you're a star in the Ansible Galaxy, you're no doubt attending AnsibleFest London on Feb 5. Stop by and say hello to Alan Hardy and Samir Bennacer at our table in the exhibits area!
- The South Shore .NET user group in Plymouth, Massachusetts, is getting together on January 29 for a presentation on Elasticsearch for data mining. This meetup will be covering Elasticsearch basics and a few other areas, and have you ready to add Elasticsearch to your data analysis toolkit. Sign up to save your spot!
- Heading to Strata in San Jose? Join Costin Leau, creator of Elasticsearch for Apache Hadoop, at the conference. We'll update you soon with details of Costin's presentation. Plus, we're working on scheduling an Elasticsearch Silicon Valley Meetup to coincide with Costin's visit. Sty tuned!
- Robyn Bergeron will be speaking at SCALE 13x on DevOps + Open Source == BFF Practices! Join her to learn more about DevOps (practice, theory, and otherwise!), shared habits of successful open source communities and DevOps practitioners, and tips for how you or your organization can start applying these habits today.
@karmiq on #elasticsearch and #ruby right now at @metalab pic.twitter.com/2ukrmwX1Bv
— elasticsearch Vienna (@elasticvienna) January 20, 2015
Where to Find You
PSST! If you're a regular reader of This Week in Elasticsearch, a.k.a TWIES, you're thinking of skipping this section. You may even be thinking to yourself, yes of course I will drop a note on Twitter when I am giving a talk on all things ELK. That's awesome, because we'd like to showcase every meetup, conference presentation and workshop on Elasticsearch, Logstash, and Kibana happening worldwide. And now, we've made it even easier for you to get support for your meetup!
Head on over to our meetups page! (And we'll still totally send you swag if you're giving a talk on anything ELKy at a conference.)
Oh yeah, we're also hiring. If you'd like us to find you for employment purposes, just drop us a note. We care more about your skill set and passion for Elasticsearch, Kibana, and Logstash than where you rest your head.
If you are interested in Elasticsearch training we have courses taught by our core developers coming up in:
- Paris - January 22, 2015 (Core Elasticsearch Training)
- London - January 29, 2015 (Core Elasticsearch Training)
- San Francisco - February 4, 2014 (Hands-on Workshop with Elasticsearch, Logstash & Kibana)
- San Francisco - February 5, 2014 (Core Elasticsearch Training)
- Melbourne - February 9, 2014 (Core Elasticsearch Training)
- New York - February 11, 2014 (Core Elasticsearch Training)
- New York - February 12, 2014 (Hands-on Workshop with Elasticsearch, Logstash & Kibana)
- Amsterdam - February 18, 2014 (Core Elasticsearch Training)
- Amsterdam - February 19, 2014 (Hands-on Workshop with Elasticsearch, Logstash & Kibana)
- Los Angeles - February 19, 2014 (Hands-on Workshop with Elasticsearch, Logstash & Kibana)
- London - February 25, 2014 (Hands-on Workshop with Elasticsearch, Logstash & Kibana)