Welcome to This Week in Elasticsearch. In this roundup, we try to inform you about the latest and greatest changes in Elasticsearch. We cover what happened in the GitHub repositories, as well as many Elasticsearch events happening worldwide, and give you a small peek into the future of the project.
- Docs: java add static imports for query and filter builders (#6920, master and 1.5)
- Recovery: Mark last file chunk to fail fast if payload is truncated (#7830, master and 1.4)
- Docs: Document the Java BulkProcessor (#7638, master and 1.5)
- Docs: IndexResponse.matches() does not exist anymore (#7548, master and 1.5)
- Docs: Update java source example (#7701, master and 1.5)
- Recovery: mapping check during phase2 should be done in cluster state update task (#7744, master and 1.4)
- Discovery: Give a unique id to each ping response (#7769, master and 1.4)
- Core: Check if from + size don't cause overflow and fail with a better error. (#7778, master, 1.4 and 1.3)
- Internal: remove ForceSyncDirectory (#7804, master and 1.4)
- Add option to take currently relocating shards' sizes into account (#7753, master and 1.5)
- Indexed Scripts/Templates: Cleanup (#7568,#7559,#7647,#7567, master and 1.5)
- Bulk API: Do not fail whole request on closed index (#6410, master and 1.4)
- Core: Add AbstractRunnable support to ThreadPool to simplify async operation on bounded threadpools (#7765, master and 1.5)
- Internal: add indices setter to IndicesRequest interface (#7734, master and 1.4)
- Internal: make sure that update internal requests share the same original headers and request context (#7766, master and 1.4)
- Internal: make sure that all delete mapping internal requests share the same original headers and context (#7736, master and 1.4)
Here's some more information about what is happening in the ecosystem we are maintaining around the ELK stack - that's Elasticsearch plus Logstash and Kibana - including plugin and driver releases.
RT @elasticsearch: Big news: #PuppetApproved is here & we're honored to be part of the inaugural module class http://t.co/FTvm3Cw6ao
— Puppet Labs (@puppetlabs) September 23, 2014
- Want to learn more about how aggregations? Look no further than our latest walk through: the Top Hits Aggregation as explained by Martijn van Groningen.
- James Turbull shared a post on making Logstash and Syslog play nicely together.
- Rızacan Tufan authored a deep dive on Port Scan Detection using Netflow + Elasticsearch and Kibana. By the time you're done reading, you'll be able to create your own NetSA dashboard.
- deviantony shared a through tutorial on setting up the ELK stack on Ubuntu 12.04. Bonus: there are some Elasticsearch Marvel tips on offer, too!
- Reindexing Elasticsearch and looking for zero downtime? Christian Uhl offered some tips, tricks and tradeoffs on the blog of our partner firm, codecentric.
- Vincent Prouillet shared a how to on finding trending things using Elasticsearch, Python and Redis.
- Florian Hopf published another post in his excellent series on Elasticsearch use cases. The latest: Index and Search Log Files.
- The engineering team at COOKPAD shared a blog post about using Elasticsearch to power their Indonesian recipes site. (日本語で)
- New talk: At GOTO Copenhagen? Jettro Coenradie from our partner firm Trifork will treat you to a live demo and talk on Visualizing Traffic Data with Elasticsearch & c3js. Jettro will be on stage tomorrow, Thursday, September 25th at 3:50 PM.
- Yann Cluchey, the awesome human who organizes the Elasticsearch London User Group, will take the stage at GOTO Aarhus on Tuesday, September 30th to share his company's Elasticsearch use case. Join Yann at 1:20 PM to hear all about Cogenta - Making Enterprise Data Available in Real Time with Elasticsearch. GOTO Aarhus runs September 29-30th.
- David Pilato will be heading to BDX.io to tell you all about how to Make Sense of your Big Data. BDX.io takes place on October 17th in Bordeaux.
- David will run a hands on Elasticsearch and Kibana workshop at Forum PHP Paris 2014. You can join David on October 23rd from 9:30 AM to 1:30 PM, though separate registration is required. Forum PHP Paris runs October 23-24th at the Belfry Montrouge.
- Patrick Peschlow from our partner firm codecentric AG will walk you through Elasticsearch: Lessons Learned. (Thursday, Oct. 9th at 10 AM)
- Isabel will tell you all about how to properly architect search for your website. (Friday, Oct. 10th at 10 AM)
- Alexander will give you an introduction to the ELK stack, including using it to build real-time analytics dashboards. (Friday, Oct. 10th at 2:00 PM)
- The DevOps Amsterdam Meetup group is hosting a special session welcoming attendees from DrupalCon. You can join us on Tuesday, September 30th at 6 PM at Old School Amsterdam, steps away from the RAI conference center. Bastian Widmer will be sharing Amazee Labs' ELK stack use case. You can register now to attend.
- Steven Merill of Phase2 Technology will teach you all about using open source logging and monitoring tools. The best part? He'll share all about how the ELK stack powers log management for Drupal.org (think billions of logs). Steven speaks on Wednesday at 2:15 PM.
- Nikolay Ignatov and Welin Welchev from Propeople will present on Elasticsearch, the New School Search Engine for Drupal. Join them on Thursday at 10:45 AM to learn all about the Elasticsearch Connector module!
- New conference: We'll have a booth at DevOps Days Chicago! Please plan to stop by and visit the Elasticsearch team in between sessions. DevOps Days Chicago runs October 7-8th at the Willis Tower.
- New conference: We'll be in the exhibits hall at Strata and Hadoop World NYC on October 13-15th. Stay tuned for news on a special Strata Meetup and more on where to find us at the conference!
- New conference: Leslie Hawthorn will be speaking at the All Things Open Conference on What Does Big Data Really Mean for Your Business? All Things Open takes place October 22-23 in Raleigh, North Carolina, and Leslie will speak at 2:15 PM on Wednesday.
- PuppetConf is on this week! If you find yourself in the San Francisco area join us in listening to Jordan Sissel's talk on Got Logs? Get answers with Elasticsearch ELK, Wednesday 24th, 5:10 PM. The conference runs September 20-24th.
- In or around Bellevue, WA on October 2nd? Cool, so are we! Please join Steve Mayzak to learn all about the ELK stack: what it is, what you can do with it and how it can provide actionable insights into your data. You can register now to attend the CloudTalk Meetup, and many thanks to these fine folks for hosting us!
- Paris - September 25, 2014 (Getting started with Elasticsearch & the ELK stack)
- Madrid - October 7, 2014 (core Elasticsearch training)
- Amsterdam - October 16, 2014 (Getting started with Elasticsearch & the ELK stack)
- New York - October 23, 2014 (core Elasticsearch training)
- New York - October 23, 2014 (Getting started with Elasticsearch & the ELK stack)
- London - October 28, 2014 (core Elasticsearch training)
- London - October 30, 2014 Getting started with Elasticsearch & the ELK stack)
Elasticsearch Meetup (at @Bol_com HQ in Utrecht) https://t.co/SrFmkQuxjf pic.twitter.com/s43FlQN1Nu
— Ralph Meijer (@ralphm) September 22, 2014
Slides & VideosWe celebrated our largest ever Elasticsearch Netherlands Meetup with Bol.com on Monday. Many thanks to them for hosting, and to Renzo for sharing their very detailed use case!
Prepping for @DrupalConEur next week with @becw's slides from #DrupalCampFV http://t.co/Oefe08rawj
— Leslie Hawthorn (@lhawthorn) September 24, 2014
Where to Find Us
We'd love to feature all the great Elasticsearch, Logstash, and Kibana presentations and meetups happening worldwide in this section. If you're speaking or hosting a meetup, let our Director of Developer Relations, Leslie Hawthorn, know!
The Vienna Elasticsearch User Group will convene on October 2nd at 7 PM. This next meetup will focus on Elasticsearch use cases. You can register now to attend.
demain au @toulousejug mon lightning talk "comment transformer vos logs en insights avec #Logstash #Kibana #ES" cc @dadoonet ;-)
— Leonardo Noleto (@leonardo_noleto) September 24, 2014
7/10 : Séminaire avec @elasticsearch - Valorisez toutes vos données à l'heure du Big Data ! Inscrivez-vous ! http://t.co/bLyUY1XnTK
— Alter Way (@alterway) September 22, 2014
The code.talks conference is back in Hamburg this year on October 9th and 10th, though with a new name. (We had a great time there last year when this conference was known simply as Developer Conference EU.) Alexander Reelsen will be speaking once again, along with Isabel Drost-Fromm.
Here's all the Elasticsearch and ELK stack information on offer at code.talks:
The week after code.talks, Leslie Hawthorn will be heading LinuxCon Europe in Dusseldorf, where she'll speak on the panel Empowering Your Corporate Open Source Software Developers. The panel takes place on Wednesday, October 15th at 2:30 PM, and the conference runs October 13-15th at Congress Centre Dusseldorf.
You can join Suyog this Saturday for our first ever Elasticsearch Meetup in Bangalore!
Super excited to be talking at the first #Elasticsearch meetup in my hometown, Bangalore with @vedang. Join us! http://t.co/ysCRm3nM42
— Suyog Rao (@suyograo) September 22, 2014
We're very excited that this year's DrupalCon EU will be in our hometown of Amsterdam on Sept. 29th - Oct. 3rd. Leslie Hawthorn will be out and about in the hallway track, so make sure to say hello to her! There are also some great talks on all things Elasticsearch and the ELK stack on offer from these Drupalistas:
Can you tell the success of a patch deployed to production?We @amazeelabs can. #Elasticsearch #devops pic.twitter.com/mrazKsmfLT
— Bastian Widmer (@dasrecht) September 18, 2014
The next Elasticsearch Oslo Meetup is on for October 7th at 6 PM. Register now to save your seat. We'll get back to you with more details as soon as the meeting agenda is finalized.
Honza Kral will be speaking at PyCon Poland 2014. The conference runs from October 16-19th in Orle Gniazdo. We'll be back to you with more details on Honza's presentation soon!
Patrick Peschlow from our partner firm codecentric AG will take the stage at the 2nd Annual Coding Serbia Conference on September 25th. Join him at 8:00 PM to learn how you can go from Elasticsearch Novice to Expert in just 90 minutes.
The first ever Elasticsearch Meetup in Madrid has been scheduled! The meetup will take place on October 7th from 7:00 - 8:30 PM. Our very own Clinton Gormley will cover all things Elasticsearch. You can register now to save your place.
The fifth Elasticsearch Stockholm Meetup is on for October 1st at 5:30 PM. You'll hear from EBay/Tradera about how they perform cluster migrations in real-time, plus all about SaltStack and Elasticsearch from Marcus Granstrom, the Stockholm meetup organizer. Register now to save your place. If there's a wait list, no worries. We have even more talks coming up soon in Stockholm - stay tuned to this space!
David Pilato will return to Soft-Shake in Geneva once again this year. He will speak on Advanced Search for Your Legacy Application in the Big Data Track. Soft-Shake takes place on October 23-24th, and David will speak in Slot 5 on Thursday.
Please join the folks from the Enterprise Search Cambridge group for a full day Elasticsearch hackathon on September 26th! You can register now to attend.
Where to Find You
Our Director of Developer Relations, Leslie Hawthorn, is hard at work to help folks create more Elasticsearch meetup groups and to help meetup organizers find more speakers. If you are interested in either effort, let her know.
Oh yeah, we're also hiring. If you'd like us to find you for employment purposes, just drop us a note. We care more about your skill set and passion for Elasticsearch, Kibana, and Logstash than where you rest your head.
If you are interested in Elasticsearch training we have courses taught by our core developers coming up in: