Welcome to This Week in Elasticsearch. In this roundup, we try to inform you about the latest and greatest changes in Elasticsearch. We cover what happened in the GitHub repositories, as well as many Elasticsearch events happening worldwide, and give you a small peek into the future of the project.
- Scripting: Prevent possible deadlock in
TransportGetIndexedScriptAction(#8266, master, 1.x, 1.4 and 1.3)
- MLT Field Query: Remove from master (#8238, master)
- Recovery: change check for finished to a ref count check (#8271, master and 1.x)
- Core: Fix location information for loggers (#8052, master, 1.x and 1.4)
- Plugins: Support usage of
ES_JAVA_OPTSin plugin commands (#8288, master, 1.x and 1.4)
- Store: Cut over
MetaDataStateFormatto <code>NIO Path API (#8297, master and 1.x)
- Parent/Child: Cut over the child filter in nested filter to use fixed bitset cache (#8303, master, 1.x and 1.4)
- Disk-based allocation: Reroute shards automatically when high disk watermark is exceeded (#8270, master and 1.x)
- Filter caches: Use a 1024 byte minimum weight for filter cache entries (#8304, master, 1.x, 1.4 and 1.3)
- Netty: Add HTTP pipelining support (#8299, master, 1.x, 1.4 and 1.3)
- Search: Reduce memory usage during fetch source sub phase (#8138, master, 1.x and 1.4)
- Core: Allow to configure custom thread pools (#8247, master, 1.x and 1.4)
- Packaging: Export
JAVA_HOMEin RPM init script (#5434, master, 1.x and 1.4)
- Geo: Fix
IndexedGeoBoundingBoxFilterto not modify the bits of other filters (#8325, master, 1.x, 1.4 and 1.3)
- Discovery: Don't accept a dynamic update to
min_master_nodeswhich is larger then current master node count (#8321, master and 1.x)
RandomScoreFunctionnow accepts long, as well a strings (#8311, master, 1.x and 1.4)
- Stats: Return
0instead of <code>-1 for unknown/non-exposed
ramBytesUsed()(#8291, master and 1.x)
- Internal: when corruption strikes, don't create exceptions with circular references
(#8331, master, 1.x and 1.4)
- Aggregations: Meta data support (#8279, master)
- Discovery: Improve the lifecycle management of the join control thread in zen discovery (#8327, master, 1.x and 1.4)
- Store: Remove special file handling from
DistributorDirectory(#8276, master and 1.x)
- Geo sort: remove unneeded code from
GeoDistanceBuilder(#8338, master and 1.x)
- Routing: Switch to murmurhash3 to route documents to shards (#7954, master)
- Mappings: Generate dynamic mappings for empty strings (#8329, master and 1.x)
- Gateway: Prefer recovering the state file that uses the latest format (#8343, master and 1.x)
Here's some more information about what is happening in the ecosystem we are maintaining around the ELK stack - that's Elasticsearch plus Logstash and Kibana - including plugin and driver releases.
- In case you missed it, we released Elasticsearch 1.4 GA today, along with version 1.3.5. Get it while it's hot!
- The elasticsearch-net client 1.2.2 has been released and is now more compatible with 1.4.0 Beta1.
- Loren Siebert at DigitalGov - an arm of the United States General Services Administration - wrote up two in-depth blog posts on their use of Elasticsearch. Read on to learn all about using Elasticsearch for Image Search in A Picture is Worth a Thousand Tokens, part one and part two.
- It's good to be loved. Read on to hear from GigaOm on how Microsoft's Azure cloud platform leverages Elasticsearch.
- Fred de Villamil shared his experience migrating his Elasticsearch cluster from Canada to France with zero downtime. Lots of useful, juicy details in there for anyone about to perform a cluster migration.
- Thierry Delprat treated us to a close up view of scaling up and stress testing Elasticsearch at Nuxeo - up to 1B documents.
- Mark Litwintschik authored a detailed tutorial on monitoring Django applications using the ELK stack.
- Sloan Ahrens published the next post in his series on Elasticsearch and Apache Spark. Check out his latest work on Deploying Elasticsearch and Spark to the Cloud.
- And in another yet another excellent article series, Robin Moffatt will teach you how to perform analytics across your data stored in Apache Hadoop using Elasticsearch and Kibana. Read on for part one, part two and part three, with each post explaining how to set up a different part of the data pipeline.
- Benjamin Pollack shared a post on Having Fun with Elasticsearch and Python. Perfect for the Pythonista who is newer to Elasticsearch.
- Elijah Paul published a step by step tutorial on Analyzing Microsoft Exchange 2013 Message Tracking Logs using NXLog & ELK.
Slides & Videos
Where to find Us
We'd love to feature all the great Elasticsearch, Logstash, and Kibana presentations and meetups happening worldwide in this section. If you're speaking or hosting a meetup, let our Director of Developer Relations, Leslie Hawthorn, know!
David Pilato will host a Hands-on Elasticsearch and Kibana workshop at Devoxx Belgium 2014. David's workshop kicks off on Monday, November 10th at 9:30 AM, and the conference runs from November 10-14 in Antwerp.
I'll be talking about #Logstash, #Elasticsearch and #Kibana at @BSidesTO -- ELK != Plain 'Ol Syslog - More info: http://t.co/iCaXCvDQbY
— Jason Kendall (@coolacid) October 29, 2014
Simon Willnauer will keynote the DrivingIT Conference in Copenhagen. Stay tuned for more schedule details, but Simon will cover Randomized Testing during his presentation, a.k.a. "evil tests." The conference takes place on November 14, 2014.
- The 10th Elasticsearch Paris Meetup is on for November 13. RSVPs are now open, so register ASAP. We fill all of our seats very quickly for the Paris meetups.
- You can also hear from David Pilato at the Drupagora conference on all matters Elasticsearch & Drupal. Join David on November 14 at 2:00 PM for his talk; the conference runs the full day in Paris.
Brand new event on tonight in Berlin: Casual meetup with the Elasticsearch Berlin User Group and some Elasticsearchers in town for GOTO Berlin. Go register now - refreshments are on us!
For folks in Berlin, we'll have a booth at the GOTO Berlin conference, plus two awesome talks from Elasticsearchers:
- Britta Weber: Make sense of your logs
- Leslie Hawthorn: The Human Element in Development: What Your Tools Say about Your Company Culture
Join us at the Kosmos Event Center on November 6th and 7th for GOTO Berlin 2014!
And, we have a few of other great events coming up in Germany:
- Open Source Monitoring Conference, Nuremburg: Alexander Reelsen on Using the ELK stack in Systems Administration.
- data 2 day, Karlsruhe: Patrick Peschlow on Elasticsearch successfully at scale.
- Mimacom Days, Stuttgart: Dmitri Marx on Making Sense of your data to give you actionable insights.
Isabel Drost-Fromm will be giving two talks at ApacheCon Europe:
- Patches Welcome: Contributing to Apache Projects in a Nutshell - Nov. 17th at 2:40 PM
- Your search doesn't work: how to find out whether the search box you offer your users is helping - Nov. 19th at 10:40 AM
ApacheCon Europe is on November 17-21, 2014 in Budapest.
Our CEO, Steven Schuurman, will be speaking at the Web Summit conference. You can hear from Steven on Friday, November 6th on raising capital outside of Silicon Valley. Web Summit is on in Dublin November 4-6th.
Shay Banon and Boaz Leskes will both take the stage at DevOps Days Tel Aviv, which takes place November 23-24th. Shay will hold a fireside chat on Dev, Ops and Culture, and Boaz will lead a workshop on the ELK stack.
Not heading to DevOps Days? No worries! We've got a meetup planned for November 23rd. You can register now to attend.
Join David Pilato at CodeMotion Milan, November 26-29th. David will teach you all about How to Make Sense of Your Big Data using Elasticsearch.
- The Japan Java Users Group Community Conference is coming up on November 15th in Tokyo. You can hear from our Developer Advocate, Jun Ohtani, on all things Elasticsearch and Kibana. Elasticsearch is proud to sponsor this great community event!
- The next Elasticsearch Study Session has been scheduled for November 18th at 7:30 PM. Register now to save your place.
Next Elasticsearch NL meetup is scheduled. Mark the date: 17//11. Talks by ING and RIPE. See you there! RSVP: http://t.co/CLtdU5ySMf
— Boaz Leskes (@bleskes) October 29, 2014
New meetup on *tonight*! Bucharest Big Data user group will talk #OpenSource Analytics using Hadoop & #Elasticsearch http://t.co/xfHKkB9BhO
— Leslie Hawthorn (@lhawthorn) November 5, 2014
Heading to Strata Barcelona? Great, so are we! You can hear from Shay Banon - How Search Can Save Your Hadoop Investment - and Costin Leau - Identifying Outliers at Scale Using Real-Time Search Engines - plus visit us on the show floor. We are going to have special gifts on hand too, so make sure to stop by to say hello.
Even if you're not planning to attend Strata, we have other great talks on offer in Barcelona on November 20th. You can register now for our Special Strata Meetup.
And, just because we can't have enough awesome things happening in Barcelona in one week, you can also hear from Clinton Gormley at the NoSQL Matters Conference on November 22nd. Clinton will deep dive into The Elasticsearch Query DSL: Not Just for Wizards!
Call Madrid home instead of Barca? Awesome, we've got you covered!
David Pilato will teach you all about How to Make Sense of Your Big Data at Codemotion Madrid on Friday, November 21st at 12:15 PM. Not attending Codemotion Madrid? There's also a meetup scheduled for November 24th, where you can hear all about your fellow users' experiences with Elasticsearch. Register now to save your place.
Honza Kral will be speaking at the Øredev conference in Malmö on Analyzing & Search Data Using Elastisearch. The conference runs Nov. 4-7th, and Honza takes the stage on Nov. 6th at 5:40 PM.
While Honza is visiting Sweden, we figured it would be a great time to host a meetup in Malmö. Please join us Friday, November 7th to hear from Honza on How to Make Sense of Your Big Data using Elasticsearch. Register now to save your place.
The Elasticsearch team will be at the Big Data Analytics 2014 conference in London on November 13th and we are honored to have the UK's own Daily Mail present how Elasticsearch is helping to put newspapers ahead of the digital fold.
DC friends, go see @skearns64 at Enterprise Search & Discovery conf tomorrow Lots of #Elasticsearch goodness on offer http://t.co/R6tsqZH5VV
— Leslie Hawthorn (@lhawthorn) November 5, 2014
- On tonight in San Francisco: Kurt Hurtado will speak at the San Francisco Cloud Mafia meetup on November 5th, where he'll talk the ELK stack in a DevOps environment. This meetup is incredibly popular, so get registered now if you'd like to attend.
- The Elasticsearch Seattle Meetup will convene on November 6th and you can hear from our Director of Solutions Architecture, Steve Mayzak, on Elasticsearch in the Real World. Register now to save your place.
- Heading to Colorado for Defrag? Nifty - we'll be there! Come visit us in the exhibits area, plus make sure to catch Peter Kim's talk, Getting Insights from Uncommonly Common Occurrences in your Data. See you in Broomfield November 18th & 19th!
Meetup November in Philadelphia: Building an educational content search application using #Elasticsearch @Wharton http://t.co/VVVU7P5H6h
— jamestyack (@jamestyack) October 28, 2014
PDX friends, @smayzak will demo #Kibana 4 at the Nov. 13th #Elasticsearch meetup + what's new in the ELK stack! http://t.co/3CE0TPmEYZ
— Leslie Hawthorn (@lhawthorn) November 5, 2014
Where to Find You
Our Director of Developer Relations, Leslie Hawthorn, is hard at work to help folks create more Elasticsearch meetup groups and to help meetup organizers find more speakers. If you are interested in either effort, let her know.
Oh yeah, we're also hiring. If you'd like us to find you for employment purposes, just drop us a note. We care more about your skill set and passion for Elasticsearch, Kibana, and Logstash than where you rest your head.
If you are interested in Elasticsearch training we have courses taught by our core developers coming up in:
- Paris - November 17, 2014 (Getting started workshop)
- Bangalore - November 17, 2014 (Getting started workshop)
- Bangalore - November 18, 2014 (Core Elasticsearch Training)
- London - November 25, 2014 (Getting Started Workshop)
- London - November 26, 2014 (Core Elasticsearch Training)
- Amsterdam - December 9, 2014 (Getting Started Workshop)
- San Francisco - December 9, 2014 (Getting Started Workshop)
- Amsterdam - December 10, 2014 (Core Elasticsearch Training)
- San Francisco - December 11, 2014 (Core Elasticsearch Training)
- Bangalore - December 15, 2014 (Getting Started Workshop)
- Bangalore - December 16, 2014 (Core Elasticsearch Training)