This Week in Elasticsearch - March 04, 2015

By
Alexander Reelsen

Welcome to This Week in Elasticsearch. In this roundup, we try to inform you about the latest and greatest changes in Elasticsearch. We cover what happened in the GitHub repositories, as well as many Elasticsearch events happening worldwide, and give you a small peek into the future of the project.

Elasticsearch core

  • Internal: stop passing default cluster name to cluster state read operation (#9888, 2.0.0)
  • Internal: Upgrade 3.x segments on engine startup (#9899, 1.5.0, 1.4.4)
  • Testing: NPE on ElasticsearchIntegrationTest (#9907, 2.0.0, 1.5.0)
  • Mappings: Add enabled flag for _field_names to replace disabling through index=no (#9893, 2.0.0, 1.5.0)
  • Mappings: Lock down _routing field (#9895, 2.0.0)
  • Mappings: Lock down _field_names field (#9912, 2.0.0)
  • Core: Remove component settings from AbstractComponent (#9919, 2.0.0)
  • Core: Remove Restart API and remove Node#stop() (#9921, 2.0.0)
  • Dependencies: Upgrade to lucene r1662607 (#9915, 2.0.0)
  • Tests: Add back tests.cluster (#9916, 2.0.0, 1.5.0)
  • Recovery: unify RecoveryState management to IndexShard and clean up semantics (#9902, 2.0.0, 1.5.0)
  • Native: Use direct mapping call in Kernel32Library (#9923, 2.0.0)
  • Recovery: Synchronize RecoveryState.timer methods (#9943, 2.0.0, 1.5.0)
  • Engine: back port #9648 – Fix deadlock problems when API flush and finish recovery happens concurrently (#9942, 1.4.5)
  • Store: renaming temp files should log errors of delete in trace (#9933, 1.5.0)
  • Testing: Make sure test end with ..Tests (#9945, 2.0.0, 1.5.0)
  • Testing: Work around URI encode limitations in RestClient (#9946, 2.0.0, 1.5.0, 1.4.5)
  • Dependencies: Upgrade to Lucene 4.10.4 bugfix release (#9960, 1.5.0, 1.4.5)
  • Tests: Use all found index files instead of static list for static bwc tests (#9854, 2.0.0, 1.5.0, 1.4.5)
  • Store: Improve safety when deleting files from the store (#9801, 2.0.0, 1.5.0)
  • Mappings: Lock down _id field (#9842, 2.0.0)
  • Mappings: Lock down _type field (#9869, 2.0.0)
  • Indices: Schedule pending delete if index store delete fails (#9856, 2.0.0, 1.5.0)
  • Core: Allow primary promotion on shadow replica without failing the shard (#9786, 2.0.0, 1.5.0)
  • Cluster health: Add pending tasks count (#9877, 2.0.0, 1.5.0)
  • Testing: Reduce tests.policy a bit more (#9868, 2.0.0)
  • Recovery: RecoveryState clean up (#9811, 2.0.0, 1.5.0)
  • DateMath: Use time zone when rounding (#9885, 2.0.0, 1.5.0, 1.4.5)
  • Query API: Add support for minimum_should_match to simple_query_string (#9864, 2.0.0, 1.5.0)
  • Mappings: Lock down _index field (#9870, 2.0.0)
  • Logging: Fix logging a RoutingNode object, log an object with a good .toString instead (#9863, 2.0.0, 1.5.0)
  • Tests: check node ports availability when using unicast discovery (#9886, 2.0.0, 1.5.0)
  • Core: Handle truncated translog gracefully (#9797, 2.0.0, 1.5.0, 1.4.5)
  • Internal: refactor settings filtering (#9748, 2.0.0)
  • Snapshot/Restore: Allow deletion of snapshots with corrupted snapshot files (#9569, 2.0.0, 1.5.0)
  • Snapshot/Restore: add ability to retrieve currently running snapshots (#9400, 2.0.0)
  • Transport: added a simple request tracer, logging incoming and outgoing requests (#9286, 2.0.0, 1.5.0)
  • Mappings: Remove _boost field (#9897, 2.0.0)
  • Engine: close snapshots before recovery counter (#9760, 1.5.0, 1.4.5)
  • Netty: Change access modifiers to protected in NettyHttpTransport (#9724, 2.0.0, 1.5.0)
  • Testing: Make sure we restart the suite cluster after each test failure (#9015, 2.0.0, 1.5.0, 1.4.5)
  • Mappings: Lock down _ttl field (#9914, 2.0.0)
  • Mappings: Lock down _size field (#9913, 2.0.0)

In apache lucene this past week

Elasticsearch Ecosystem

Here's some more information about what is happening in the ecosystem we are maintaining around the ELK stack – that's Elasticsearch plus Logstash and Kibana – including plugin and driver releases.

  • Kibana 4.0.1 was just released this morning. Rashid Khan has all the details. But if you want to jump right in, you can get the newest release here.
  • For our lovely users of Kibana, we've prepared a set of video tutorials to get up to speed with all the new features in Kibana 4. These short videos will teach you more about navigation and creating pie charts. For folks more familiar with Kibana 3, we've got how-tos on migrating your Hits and Terms panels to K4.
  • Looking for more in-depth information on K4? You may enjoy the video from our recent webcast, What's New in Kibana 4. (registration required)
  • From the even more Kibana 4 love department: Alexandre Eleuterio Santos Lourenco concludes a three-part article series on the ELK Stack for your centralized logging needs with a how-to on setting up Kibana 4 and its feature set, including building visualizations. Awesome walk through with lots of screenshots within for the visually-oriented learner.
  • Looking to use Elasticsearch on AWS? As a follow on from our recent webcast, we've published a how-to guide that dives into the details of sizing your clusters, from RAM through to disk and network considerations. Check it out and let us know what you think.
  • Heading to Elastic{ON} next week? No worries if you're not able to attend our first ever user conference – we're planning to video most of the sessions. To get a taste of what we'll be discussing next week and a peek into our talk selection process, take a look at Behind the Antlers: Life Lessons from the Elastic{ON} CFP.
  • SoCal Linux Expo, a.k.a. SCALE, just celebrated its 13th year. You can check out all the Elasticsearch and ELK Stack highlights from the conference in our conference recap post.
  • New to building queries for Elasticsearch? Greg Kempe of Code for South Africa shared an excellent introduction to creating queries for full-text search.
  • Looking for a how-to on setting up Elasticsearch on EC2 using Vagrant? Look no further than this article from Ricardo Vegas. Great step-by-step instructions for folks looking to kick the tires quickly.
  • Haven't yet discovered the power of Elasticsearch Aggregations? Tony Messias published an excellent introduction, including plenty of code snippets so you can see what's going on under the hood.

Slides & Videos

An introduction to Elasticsearch and the ELK Stack. Lots of material, but well worth a gander for users who are just getting started.

For lovers of Django and Elasticsearch, this recent presentation from the Berlin Django User Group is for you!

Excellent introduction to application logging using Logstash

Better Decisions Through Data Using the ELK Stack (auf Deutsch)

Where to find Us

We’d love to feature all the great Elasticsearch, Logstash, and Kibana presentations and meetups happening worldwide in this section. If you’re speaking or hosting a meetup, let our Director of Developer Relations, Leslie Hawthorn, know!

France

  • The second Elasticsearch France Meetup in Lyon will take place on March 24. Among other topics, you'll hear about performance monitoring from the Linux cluster administrator at the Computing Centre of the National Institute of Nuclear Physics and Particle Physics (CC-IN2P3). Register now to save your spot.
  • The Elasticsearch France Meetup returns to Paris on March 26. Mark your calendar for March 6, too, as that's when registration will open for the meetup.
  • David Pilato will speak at NoSQL Matters Paris on March 27 at 12 noon. He'll treat you to Advanced Search for your Legacy Application.

Germany

  • The Munich AWS Users Group will meet on March 9 to talk setting up the ELK stack on AWS, amongst other topics. Register now to save your seat.
  • The Search Technology Meetup Hamburg group will get together on March 17 to talk Elasticsearch in Practice. You'll learn about performance and scalability essentials, plus a real world use case. Register now to attend.
  • The Elasticsearch Frankfurt Meetup group will convene on March 25 to talk Kibana 4 and Shield. Register now to save your seat.
  • The Elasticsearch Berlin User Group will get together on March 31. As usual, there will be an introduction to Elasticsearch for beginners. These lovely folks are looking for a speaker, so please do get in touch with them if you're interested in sharing your story. You can contact the organizers and RSVP for the meetup here.

India

The next Delhi Elasticsearch Meetup will take place on March 14, covering what's new in the ELK stack. Register now to save your place.

Italy

David Pilato will take the stage at Code Motion Rome to discuss Advanced Search for Your Legacy Application. David speaks at 3 PM on March 28, and the conference runs the 25-28.

The Netherlands

For folks in or near Utrecht, you can register now for the GOTO Night on Elasticsearch. Attendees will hear from Anne Veling and Jettro Coenradie, including a recap of the Elastic{ON} conference. The GOTO Night is scheduled for March 25 at bol.com's offices.

South Africa

The inaugural Capetown Elasticsearch Meetup will convene on March 5 to talk shop and plan for the future of the group. Register now to let the organizers know you plan to attend.

Sweden

The 8th Elasticsearch Stockholm Meetup is coming up on March 25. Mark Harwood will cover data analysis using Kibana 4. We're still searching for a second speaker, so please do get in touch if you're interested in presenting. You can also register now to save your seat.

United Kingdom

Elasticsearch will be out in force at QCon London, which returns to the Queen Elizabeth II Conference Center this year. You can visit us at Booth 12 on the show floor. QCon runs March 2-6.

And for folks looking for ELK stack goodness in Bristol, you can join the local Java Meetup on March 10 to hear about using Elasticsearch & Kibana alongside Apache Storm. Register now to save your seat.

Not planning to come to QCon London, but hoping to hear more about the ELK stack? The March London Elasticsearch Meetup is on for March 24, featuring speakers from Couchbase, OpenTable, and Postcode Anywhere. Register now to save your seat.

United States

  • Featured Meetup: We're holding the next Elasticsearch NYC meetup on March 18 to coincide with the Spark Summit East conference. Sign up now — you don't want to miss the discussion on how iHeartRadio does real-time analytics with es-hadoop, and how they use Elasticsearch.
  • And, if you're heading to Spark Summit East 2015, make sure to catch Costin Leau's talk on Using Spark and Elasticsearch for real-time data analysis. Costin takes the stage at 5 PM on March 18, and the conference runs from March 18-19 at the Sheraton New York Times Square Hotel.
  • The League of Professional Systems Administrators New Jersey will get together in Lawrence Township on March 5 to talk Systems Log Aggregation with the ELK Stack. Register now to save your place.
  • The DevOps West Michigan User Group will meet on March 9 to talk about how the ELK stack can provide you sanity in a sea of logs. Register now to join the fun in Grand Rapids.
  • The San Francisco MySQL User Group will convene on March 11 to discuss MySQL log auditing using the ELK stack. Register now to save your place.
  • The South Shore .NET Users Group in Plymouth, MA will convene on March 26, covering a recap from the Elastic{ON} conference. Register now to save your seat.

Where to Find You

PSST! If you're a regular reader of This Week in Elasticsearch, a.k.a TWIES, you're thinking of skipping this section. You may even be thinking to yourself, yes of course I will drop a note on Twitter when I am giving a talk on all things ELK. That's awesome, because we'd like to showcase every meetup, conference presentation and workshop on Elasticsearch, Logstash, and Kibana happening worldwide. And now, we've made it even easier for you to get support for your meetup!

Head on over to our meetups page! (And we'll still totally send you swag if you're giving a talk on anything ELKy at a conference.)

Oh yeah, we're also hiring. If you'd like us to find you for employment purposes, just drop us a note. We care more about your skill set and passion for Elasticsearch, Kibana, and Logstash than where you rest your head.

Trainings

If you are interested in Elasticsearch training we have courses taught by our core developers coming up in:

  • Hyderabad – March 11, 2015 (Hands on Workshop)
  • Hyderabad – March 12, 2015 (Core Elasticsearch Training)
  • New Delhi – March 16, 2015 (Hands on Workshop)
  • New Delhi – March 17, 2015 (Core Elasticsearch Training)
  • San Francisco – March 17, 2015 (Hands on Workshop)
  • Mountain View – March 18, 2015 (Core Elasticsearch Training)
  • Johannesburg – March 23, 2015 (Core Elasticsearch Training)
  • Munich – March 24, 2015 (Core Elasticsearch Training)
  • Stockholm – March 25, 2015 (Core Elasticsearch Training)
  • Los Angeles – March 25, 2015 (Core Elasticsearch Training)
  • Paris – March 25, 2015 (Hands on Workshop)
  • Munich – March 26, 2015 (Hands on Workshop)
  • Paris – March 26, 2015 (Core Elasticsearch Training)
  • San Francisco – April 7, 2015 (Core Elasticsearch Training)
  • Amsterdam – April 7, 2015 (Hands on Workshop)
  • Amsterdam – April 8, 2015 (Core Elasticsearch Training)
  • Seoul – April 8, 2015 (Core Elasticsearch Training)
  • Tokyo – April 13, 2015 (Core Elasticsearch Training)
  • Austin – April 23, 2015 (Core Elasticsearch Training)
  • New York – April 28, 2015 (Hands on Workshop)
  • New York – April 29, 2015 (Core Elasticsearch Training)
  • London – April 29, 2015 (Hands on Workshop)