This Week in Elasticsearch - February 25, 2015

By
Luca Cavanna

Welcome to This Week in Elasticsearch. In this roundup, we try to inform you about the latest and greatest changes in Elasticsearch. We cover what happened in the GitHub repositories, as well as many Elasticsearch events happening worldwide, and give you a small peek into the future of the project.

Elasticsearch core

  • Aggregations: Formatted bucket key_as_string in date_histogram according to time_zone (#9744, 2.0.0)
  • Test: Added REST tests to check if expression scripting works (#9750, 2.0.0, 1.5.0, 1.4.4)
  • Build: Check that all packages contain the same jar files (#9757, 2.0.0, 1.5.0, 1.4.4, 1.3.9)
  • Recovery: Only iterate the files that we recovered from the commit (#9761, 2.0.0, 1.5.0)
  • Test: Minimized tests.policy (#9765, 2.0.0)
  • Core: Hot threads to include timestamp and params (#9773, 2.0.0)
  • Test: Removed use of globalTempDir() and forbid it from future use. (#9777, 2.0.0, 1.5.0)
  • Snapshot/Restore: Refactored how restore cleans up files after snapshot was restored (#9770, 2.0.0, 1.5.0)
  • Indices: Retry if shard deletes fail due to IOExceptions (#9784, 2.0.0, 1.5.0)
  • Internal: Upgrade to Lucene r1660560 (#9746, 2.0.0)
  • Indices: NodeEnv to lock all shards for an index (#9799, 2.0.0, 1.5.0)
  • Internal: Removed redundant file-system metadata operations (#9807, 2.0.0)
  • Tests: Removed global shared cluster (#9781, 2.0.0, 1.5.0)
  • Internal: Fixed field mappers to always pass through index settings (#9780, 2.0.0)
  • Aggs: Fixed rounding issues when using date_histogram and time zones (#9790, 1.5.0, 1.4.5)
  • Scripting: Removed deprecated script parameter names (#9815, 2.0.0)
  • Aggs: Fixed rounding issue using date_histogram with pre_zone_adjust_large_interval (#9828, 1.5.0, 1.4.5)
  • Internal: Cleaned up file-system operations (#9827, 2.0.0, 1.5.0)
  • Test: Don’t reuse single node tests node instance across tests (#9813, 2.0.0, 1.5.0)
  • Internal: Added missing @Override annotations (#9832, 2.0.0)
  • Mappings: Locked down _uid field (#9836, 2.0.0)
  • Scripting: Add String to the default whitelisted receivers (#9837, 2.0.0, 1.5.0)
  • Internal: Remove InternalNode class and use Node directly (#9844, 2.0.0)
  • Internal: refactor settings filtering (#9748, 2.0.0)
  • Test: Use all found index files instead of static list for static bwc tests (#9854, 2.0.0, 1.5.0, 1.4.5)
  • Store: Improve safety when deleting files from the store (#9801, 2.0.0, 1.5.0)
  • Mappings: Lock down _type field (#9869, 2.0.0)
  • Tests: Removed thread leak filter (#9843, 2.0.0, 1.5.0)
  • Mappings: Lock down _id field (#9842, 2.0.0)
  • Tests: Added tests for TimezoneRounding (#9796, 1.5.0, 1.4.5)
  • Build: Renamed Maven repository id (#9733, 2.0.0, 1.5.0)
  • Store: Added simple cache for StoreStats (#9709, 2.0.0, 1.5.0)
  • Internal: Flush IndexWriter to disk on close and shutdown (#7563, 2.0.0, 1.5.0)
  • Aggs: Use time_zone in date_histogram to format bucket keys (#9710, 2.0.0)
  • Codecs: Removed the ability to have custom per-field postings and doc values formats (#9741, 2.0.0)
  • Inner hits: Don’t fail if an object is specified as a nested value instead of an array (#9743, 2.0.0, 1.5.0)
  • Filters: Don’t use the fixed bitset filter cache for child nested level filters, but the regular filter cache instead (#9740, 2.0.0)
  • Replication: Shadow replicas on shared filesystems (#9727, 2.0.0, 1.5.0)
  • Aggs: Significant terms should provide a simple percentage scoring heuristic (#9720, 2.0.0, 1.5.0)
  • Test: Adds randomization to geo test suite (#9658, 2.0.0)
  • Internal: Perform a single `listAll` from `FileSwitchDir` (#9666, 2.0.0, 1.5.0, 1.4.5)
  • Store: Cache fileLength for fully written files (#9683, 2.0.0, 1.5.0)
  • Discovery: Prevented stale master nodes from sharing dated cluster states to nodes that have moved to a different master node (#9632, 2.0.0, 1.5.0)
  • Snapshot/Restore: Allow deletion of snapshots with corrupted snapshot files (#9569, 2.0.0, 1.5.0)
  • Mappings: Update cluster state with type mapping also for failed indexing request (#8692, 2.0.0, 1.5.0)

In apache lucene this past week

Elasticsearch Ecosystem

Here’s some more information about what is happening in the ecosystem we are maintaining around the ELK stack – that’s Elasticsearch plus Logstash and Kibana – including plugin and driver releases.

  • Kibana 4 GA was released last week! Check out the full details, including teasers for building your own Kibana visualizations.
  • Need a little more Kibana 4 love? Rashid Khan shared a post on the order of aggregations execution – think of it as how Elasticsearch goes about analyzing your data, and how Kibana displays the results.
  • And, we’ve got even more K4 goodness – Peter Kim published an article on using Elasticsearch and Kibana 4 for analysis of open government data sets. Yay civic hacking!
  • Also last week, we released Elasticsearch 1.4.4 and 1.3.9, a small bug fix release, mainly including a packaging fix for our RPM and DEB packages if you are using Lucene expression scripts.
  • As part of our ongoing guest blogger series, Oliver Eilhard shared the story of how Elasticsearch made his auto racing team faster. (We’re also always on the look out for amazing posts like Oliver’s, so if you enjoyed the piece and are inspired to share your story, please email us at stories@elasticsearch.com.)
  • Alex Brasetvik authored a blog post on common use cases for Elasticsearch. Excellent resource if you’re just starting out using it.
  • Javier Ray from Tryolabs published a deep dive on Elasticsearch Analyzers as part of the lead up to Elastic{ON}. (And many thanks to Tryolabs for sponsoring our first ever User Conference!)
  • The Luminis Engineering team authored a lovely how to on using the ELK stack to detect potential attacks on your WordPress blogs. Great step by step how to, including all needed configs for Logstash (v 1.5).
  • Ed King authored a great in-depth tutorial on integrating the ELK stack with Cloud Foundry.
  • And who doesn’t want to know how to run Elasticsearch on a Raspberry Pi? Richard Hayler has all the news you can use.

    • Slides & Videos

    Excellent introduction to Elasticsearch: what it does, fundamental concepts, and how to make it do useful things for you out of the box

    An introduction to Logstash and its ecosystem. Pablo Figue’s presentation from last week’s Elasticsearch Berlin User Group meeting From the January London Elasticsearch Meetup

    Where to find Us

    We'd love to feature all the great Elasticsearch, Logstash, and Kibana presentations and meetups happening worldwide in this section. If you're speaking or hosting a meetup, let our Director of Developer Relations, Leslie Hawthorn, know!

    Austria

    The Vienna Elasticsearch Users Group will get together on March 4 to talk Elasticsearch at Cloud Foundry and more. Register now to let the organizers know you’re attending.

    France

    • The second Elasticsearch France Meetup in Lyon will take place on March 24. Among other topics, you’ll hear about performance monitoring from the Linux cluster administrator at the Computing Centre of the National Institute of Nuclear Physics and Particle Physics (CC-IN2P3). Register now to save your spot.
    • The Elasticsearch France Meetup returns to Paris on March 26. Mark your calendar for March 6, too, as that’s when registration will open for the meetup.
    • David Pilato will speak at NoSQL Matters Paris on March 27 at 12 noon. He’ll treat you to Advanced Search for your Legacy Application.

    India

    The next Delhi Elasticsearch Meetup will take place on March 14, covering what’s new in the ELK stack. Register now to save your place.

    Italy

    David Pilato will take the stage at Code Motion Rome to discuss Advanced Search for Your Legacy Application. David speaks at 3 PM on March 28, and the conference runs the 25-28.

    The Netherlands

    For folks in or around Utrecht, you can register now to attend the GOTO Night: Elasticsearch on March 25. Among the discussion topics, you’ll get a recap of our first ever user conference, Elastic{ON}.

    Portugal

    The KWAN SmallTalks User Group will get together on March 3 to hear from João Duarte on the ELK Stack. Register now to save your spot.

    South Africa

    The inaugural Capetown Elasticsearch Meetup will convene on March 5 to talk shop and plan for the future of the group. Register now to let the organizers know you plan to attend.

    Sweden

    The 8th Elasticsearch Stockholm Meetup is coming up on March 24. Mark Harwood will cover data analysis using Kibana 4. We’re still searching for a second speaker, so please do get in touch if you’re interested in presenting. You can also register now to save your seat.

    United Kingdom

    Elasticsearch will be out in force at QCon London, which returns to the Queen Elizabeth II Conference Center this year. You can visit us at our booth on the show floor, plus we’ll be having one of our engineers take the stage for the main program. Those details are in the works, but in the meantime you can take a look at Kristoffer Dyrkorn’s talk information. He’ll be sharing the story of how Elasticsearch and other technologies are powering the Norwegian Roads Authority’s brand new system to provide real-time traffic information to travelers throughout Norway. QCon runs March 2-6.

    And for folks looking for ELK stack goodness in Bristol, you can join the local Java Meetup on March 10 to hear about using Elasticsearch & Kibana alongside Apache Storm. Register now to save your seat.

    Not planning to come to QCon London, but hoping to hear more about the ELK stack? The March London Elasticsearch Meetup is on for March 24, featuring speakers from Couchbase, OpenTable, and Postcode Anywhere. Register now to save your seat.

    United States

    • If you’re in Boston, be sure to join the Boston Elasticsearch meetup group on March 2 to hear about how to build a scalable ELK stack on AWS. Be sure to save yourself a spot — we hope you can dig out of the snow by then!
    • The League of Professional Systems Administrators New Jersey will get together in Lawrence Township on March 5 to talk Systems Log Aggregation with the ELK Stack. Register now to save your place.
    • We’re holding the next Elasticsearch NYC meetup on March 18 to coincide with the Spark Summit East conference. Sign up now — you don’t want to miss the discussion on how iHeartRadio does real-time analytics with es-hadoop, and how they use Elasticsearch.
    • And, if you’re heading to Spark Summit East 2015, make sure to catch Costin Leau’s talk on Using Spark and Elasticsearch for real-time data analysis. Costin takes the stage at 5 PM on March 18, and the conference runs from March 18-19 at the Sheraton New York Times Square Hotel.
    • The San Francisco MySQL User Group will convene on March 11 to discuss MySQL log auditing using the ELK stack. Register now to save your place.
    • The South Shore .NET Users Group in Plymouth, MA will convene on March 26, covering a recap from the Elastic{ON} conference. Register now to save your seat.

    Where to Find You

    PSST! If you’re a regular reader of This Week in Elasticsearch, a.k.a TWIES, you’re thinking of skipping this section. You may even be thinking to yourself, yes of course I will drop a note on Twitter when I am giving a talk on all things ELK. That’s awesome, because we’d like to showcase every meetup, conference presentation and workshop on Elasticsearch, Logstash, and Kibana happening worldwide. And now, we’ve made it even easier for you to get support for your meetup!

    Head on over to our meetups page! (And we’ll still totally send you swag if you’re giving a talk on anything ELKy at a conference.)

    Oh yeah, we're also hiring. If you'd like us to find you for employment purposes, just drop us a note. We care more about your skill set and passion for Elasticsearch, Kibana, and Logstash than where you rest your head.

    Trainings

    If you are interested in Elasticsearch training we have courses taught by our core developers coming up in:

    • Bangalore – February 25, 2015 (Hands on Workshop)
    • London – February 26, 2015 (Core elasticsearch)
    • Bangalore – February 26, 2015 (Core Elasticsearch Training)
    • Hyderabad – March 11, 2015 (Hands on Workshop)
    • Hyderabad – March 12, 2015 (Core Elasticsearch Training)
    • New Delhi – March 16, 2015 (Hands on Workshop)
    • New Delhi – March 17, 2015 (Core Elasticsearch Training)
    • San Francisco – March 17, 2015 (Hands on Workshop)
    • Mountain View – March 18, 2015 (Core Elasticsearch Training)
    • Johannesburg – March 23, 2015 (Core Elasticsearch Training)
    • Northern Virginia – March 24, 2015 (Core Elasticsearch Training)
    • Munich – March 24, 2015 (Core Elasticsearch Training)
    • Munich – March 26, 2015 (Hands on Workshop)
    • Stockholm – March 25, 2015 (Core Elasticsearch Training)
    • Los Angeles – March 25, 2015 (Core Elasticsearch Training)
    • Paris – March 25, 2015 (Hands on Workshop)
    • Paris – March 26, 2015 (Core Elasticsearch Training)
    • San Francisco – April 7, 2015 (Core Elasticsearch Training)
    • Amsterdam – April 8, 2015 (Core Elasticsearch Training)
    • Amsterdam – April 7, 2015 (Hands on Workshop)
    • Seoul – April 8, 2015 (Core Elasticsearch Training)
    • Tokyo – April 13, 2015 (Core Elasticsearch Training)
    • Austin – April 23, 2015 (Core Elasticsearch Training)
    • New York – April 28, 2015 (Hands on Workshop)
    • New York – April 29, 2015 (Core Elasticsearch Training)
    • London – April 29, 2015 (Hands on Workshop)