This Week in Elasticsearch - September 03, 2014

Welcome to This Week in Elasticsearch. In this roundup, we try to inform you about the latest and greatest changes in Elasticsearch. We cover what happened in the GitHub repositories, as well as many Elasticsearch events happening worldwide, and give you a small peek into the future of the project.

Elasticsearch Core

  • Mappings: Added support for empty field arrays (#7271, master and 1.4)
  • Index templates: Made template filtering generic and extensible via plugins (#7454, master and 1.4)
  • Explain API: fix explanation streaming (#7257, master and 1.4)
  • Query API: Return match_all in case there are no clauses in boolean query (#7240, master, 1.4 and 1.3)
  • Fielddata: Remove soft/resident caches (#7443, master and 1.4)
  • Transaction log: Add checksums (#7232, master and 1.4)
  • Internal: Do not execute cluster state changes if current node is no longer master (#6230, master and 1.4)
  • Discovery: Do not use versions to optimize cluster state copying for a first update from a new master (#6466, master and 1.4)
  • Discovery: Immediately start Master|Node fault detection pinging (#6706, master and 1.4)
  • FunctionScore: Refactor RandomScoreFunction to be consistent, and return values in range [0.0, 1.0] (#7446, master and 1.4)
  • Filters: Introduced FixedBitSetFilterCache that guarantees to produce a FixedBitSet and does evict based on size or time (#7037, master and 1.4)
  • Index create API: Validate create index requests' number of primary/replica shards (#7496, master and 1.4)
  • Test: write heap dump to log folder (#7452, master and 1.4)
  • Aggregations: Encapsulate AggregationBuilder name and make getter public (#7425, master and 1.4)
  • Scan: Use ConcurrentHashMap instead of HashMap, because the readerStates is accessed by multiple threads during the entire scroll session (#7478, master, 1.4, 1.3 and 1.2)
  • Client: Make the query builder nullable in filteredQuery (#7398, master and 1.4)
  • Suggest API: Fix encoding of multiple levels for geo precision (#7368, master and 1.4)
  • Core: Don't update indexShard if it has been removed before (#7509, master and 1.4)
  • Aggregations: The nested aggregator should iterate over the child doc ids in ascending order (#7514, master, 1.4 and 1.3)
  • Suggest API: Improved java client API (#7435, master and 1.4)
  • Rest API: Add the cluster name to the / endpoint (#7524, master and 1.4)
  • Mappings: keep parameters in mapping for _timestamp, _index and _size even if disabled (#7475, master, 1.4, 1.3 and 1.2)
  • Function Scoring: Added optional weight parameter per function (#6955, master and 1.4)
  • Internal: Extract a common base class for (Master|Nodes)FaultDetection (#7512, master and 1.4)
  • Cluster: Refactored ClusterStateUpdateTask protection against execution on a non master (#7511, master and 1.4)
  • Discovery: Accumulated improvements to ZenDiscovery (#7496, master and 1.4)
  • Indexing: Use Flake IDs instead of random UUIDs when auto-generating id field (#5941, master and 1.4)
  • Engine: Verify checksums on merge (#7360, master and 1.4)
  • Internal: Do not use a background thread to disconnect node which are removed from the ClusterState (#7543, master and 1.4)
  • Aggregations: Forbid usage of aggregations in conjunction with search_type=SCAN (#7429, master and 1.4)

Elasticsearch Ecosystem

Here's some more information about what is happening in the ecosystem we are maintaining around the ELK stack - that's Elasticsearch plus Logstash and Kibana - including plugin and driver releases.

Slides & Videos

Great overview of the ELK stack for logging & data visualization

For all you PaaS lovers out there, check out how to use the ELK stack on OpenShift Origin

Great exploration of using the ELK stack and other tools to monitor your infrastructure

Where to Find Us

We'd love to feature all the great Elasticsearch, Logstash, and Kibana presentations and meetups happening worldwide in this section. If you're speaking or hosting a meetup, let our Community Manager, Leslie Hawthorn, know!

Austria

The Vienna Elasticsearch User Group will convene on October 2nd at 7 PM. This next meetup will focus on Elasticsearch use cases and the organizers are looking for more folks to share their story! Interested in presenting? Just want to attend, network and learn? You can register now for the meetup or volunteer to present via meetup.com.

Denmark

  • Dmitri Marx will be presenting at next week's Findability Day in Copenhagen! Next Thursday, Sept. 11th, Dmitri will teach you all about how Elasticsearch can provide you with actionable insights into your data. Check out all the details on Findability Day, an event organized by our partner firm, Findwise.
  • Yann Cluchey, the awesome human who organizes the Elasticsearch London User Group, will take the stage at GOTO Aarhus on Tuesday, September 30th to share his company's Elasticsearch use case. Join Yann at 1:20 PM too hear all about Cogenta - Making Enterprise Data Available in Real Time with Elasticsearch. GOTO Aarhus runs September 29-30th.

Ireland

We've got two great talks on tomorrow, September 4th in Dublin! Boaz Leskes will present at NoSQL Matters Dublin on Elasticsearch: Deep Dive into Analytics Using Aggregations. For those not attending the conference or in need of even more Elasticsearch goodness, you can catch Boaz on the full ELK stack at the DevOps Ireland Meetup that same evening. Many thanks to the DevOps Ireland folks for hosting us!

France

Heading to JUG SummerCamp 2014? Great! So is our wonderful Developer Advocate, David Pilato. David will cover Advanced Search for Your Legacy Application, and JUG SummerCamp is on September 19th.

Japan

The Netherlands

We're also super excited that this year's DrupalCon EU will be in our hometown of Amsterdam on Sept. 29th - Oct. 3rd. Leslie Hawthorn will be out and about in the hallway track, so make sure to say hello to her! There are also some great talks on all things Elasticsearch and the ELK stack on offer from these Drupalistas:

  • Steven Merill of Phase2 Technology will teach you all about using open source logging and monitoring tools. The best part? He'll share all about how the ELK stack powers log management for Drupal.org (think billions of logs). Steve speaks on Wednesday at 2:15 PM.
  • Nikolay Ignatov and Welin Welchev from Propeople will present on Elasticsearch, the New School Search Engine for Drupal. Join them on Thursday at 10:45 AM to learn all about the Elasticsearch Connector module!

Norway

Heading to JavaZone in Oslo? David Pilato will host a workshop showing you how to get Hands on with Elasticsearch and Kibana. David's workshop takes place on September 9th at 1:30 PM, and the conference runs September 9-11th.

United Kingdom

  • Please join the folks from the Enterprise Search Cambridge group for a full day Elasticsearch hackathon on September 26th! You can register now to attend.

United States

  • New Meetup: Ancestry.com will be hosting the next San Francisco Elasticsearch Meetup! You can join us on September 9th to hear about Elasticsearch Core and the ELK Stack from members of our developer team. Full details coming soon, but in the meantime get registered now!(This meetup always fills up quickly, so please do save a spot now or release your place on the RSVP list if you know you cannot attend.)
  • If you're attending the Heartland Developers Conference in Omaha, make sure to check out Andrew Oschner's talk on the ELK stack. Andrew will teach you all about troubleshooting logging in production, and he'll have some cool Elasticsearch swag to hand out to session participants. Andrew speaks on September 4th, and the conference runs the 3rd-5th.
  • Attending the first ever FOSSETCON? Join our very own Jordan Sissel for a Keynote Address on Open Source is More than Code. You can also see another of our stalwart Logstash developers, Aaron Mildenstein, on an introduction to the ELK stack. FOSSETCON runs September 11-13th in Orlando.
  • Join several of our core developers for the latest Denver Meetup on September 15th. We're still finalizing the agenda, but our hosts, Sendgrid, will be discussing their Elasticsearch use case. Register now for updates on timing and location.
  • PuppetConf is around the corner! If you find yourself in the San Francisco area join us in listening to Jordan Sissel's talk on Got Logs? Get answers with Elasticsearch ELK, Wednesday 24th, 5:10 PM. The conference runs September 20-24th.
  • In or around Bellvue, WA on October 2nd? Cool, so are we! Please join Steve Mayzak to learn all about the ELK stack: what it is, what you can do with it and how it can provide actionable insights into your data. You can register now to attend the CloudTalk Meetup, and many thanks to these fine folks for hosting us!

Where to Find You

Our Community Manager, Leslie Hawthorn, is hard at work to help folks create more Elasticsearch meetup groups and to help meetup organizers find more speakers. If you are interested in either effort, take a moment to let her know.

Oh yeah, we're also hiring. If you'd like us to find you for employment purposes, just drop us a note.  We care more about your skill set and passion for Elasticsearch, Kibana, and Logstash than where you rest your head.

Trainings

If you are interested in Elasticsearch training we have courses taught by our core developers coming up in:

  • Paris - September 9, 2014 (core Elasticsearch training)
  • San Francisco - September 11, 2014 (ELK workshop)
  • Austin - September 18, 2014 (ELK workshop)
  • Paris - September 25, 2014 (ELK workshop)
  • Stockholm - October 7, 2014 (core Elasticsearch training)
  • Madrid - October 7, 2014 (core Elasticsearch training)
  • Amsterdam - October 14, 2014 (core Elasticsearch training)
  • Chicago - October 14, 2014 (core Elasticsearch training)
  • Los Angeles - October 21, 2014 (core Elasticsearch training)