This Week in Elasticsearch - August 27, 2014
Welcome to This Week in Elasticsearch. In this roundup, we try to inform you about the latest and greatest changes in Elasticsearch. We cover what happened in the GitHub repositories, as well as many Elasticsearch events happening worldwide, and give you a small peek into the future of the project.
- Geo: fixes circle radius calculation (#7301, master and 1.4)
- Warmer API: Add support for query cache (#7326, master and 1.4)
- Aggregations: Scriptable Metrics Aggregation (#5923, master and 1.4)
- Internal: make sure that all shard level requests hold the original indices (#7319, master and 1.4)
- Stats: Prevent
ShardStats(#7356, master and 1.4)
- Query Cache: Add hit and miss count (7355#, master and 1.4)
- Term vector API: return
found: falsefor docs between index and refresh (#7121, master and 1.4)
- Aggregations: Remove the logic to optionally sort/dedup values on the fly (#7276, master and 1.4)
- Facets: Removal from master (#7337, master)
- Internal: Remove
- Parser: Clarify XContentParser/Builder interface for binary vs. utf8 values (#7367, master and 1.4)
- More Like This Query: Switch to using the multi-termvectors API (#7014, master and 1.4)
- Internal: make sure that multi_percolate request hands over its context and headers to its corresponding shard requests (#7371, master and 1.4)
- Internal: make sure that multi_search request hands over its context and headers to its corresponding search requests (#7374, master and 1.4)
- Aggregations: Fixes pre and post offset serialization for histogram aggs (#7312, master and 1.4)
- Store: Allow to get metadata from arbitrary commit points (#7376, master, 1.4 and 1.3)
- _ttl: Report conflict when trying to disable _ttl (#7316, master and 1.4)
- Store: Improve recovery / snapshot restoring file identity handling (#7351, master and 1.4)
- Java API: Add back string op type to
IndexRequest(#7387, master and 1.4)
- Dependencies: Upgrade to Apache HttpComponents client 4.3.5 (#7342, master and 1.4)
- Internal: move index templates api back to indices category and make put template and create index implement
IndicesRequest(#7378, master and 1.4)
- More Like This Query: defaults to all possible fields for items (#7382, master and 1.4)
- Engine: Force optimize was not passed to shard request (#7404, master, 1.4, 1.3 and 1.2)
- Internal: made original indices optional for broadcast delete and delete by query shard requests (#7406, master and 1.4)
- Context suggester: fix infinite loop in
GeolocationContextMapping(#7433, master, 1.4 and 1.3)
- Internal: discovery.id.seed is ignored (#7437, master and 1.4)
- Stats: add
segments.index_writer_max_memoryto see index writer's max RAM usage before buffered documents must be written to a new segment (#7440, master and 1.4)
- Core: use Java's built-in
ConcurrentHashMap(#7296, master and 1.4)
- Mapping API: report conflict on merge and throw exception on
_allfield (#7377, master and 1.4)
- Engine: Wait until engine is started up when acquiring searcher
(#7455, master, 1.4 and 1.3)
- Snapshot/Restore API: Add BWC layer to .si / segments_N hashing (#7434, master and 1.4)
- Internal: Add LZF safe encoder in
LZFCompressor(#7466, master and 1.4)
- Internal: Add all unsafe variants of LZF compress library functions to forbidden APIs (#7468, master and 1.4)
- Parent/Child: If
_parentfield points to a non existing parent type, then skip the has_parent query/filter (#7362, master, 1.4 and 1.3)
- Parent/child: Add missing support for the field data loading option to the
_parentfield (#7402, master, 1.4 and 1.3)
In case you've missed the #Hadoop #Elasticsearch webinar, the recording is available here: http://t.co/pkdu9wxn5H Enjoy!
— Costin Leau (@costinl) August 21, 2014
Here's some more information about what is happening in the ecosystem we are maintaining around the ELK stack - that's Elasticsearch plus Logstash and Kibana - including plugin and driver releases.
#elasticsearch Definitive Guide goes to the printers Sept 1. Get 50% off now at http://t.co/st3k0vFeoN
— Clinton Gormley (@clintongormley) August 26, 2014
- Alexander Reelsen shared a feature deep dive on how to use the percolator to enrich documents. If you love using Elasticsearch for geo search, this post is for you. (And we want to know all about *your* percolator use cases, so read on and talk to us about it!
- The team at the Smithsonian's Cooper Hewitt Museum in New York City is hard at work preparing for their reopening. They'll now help you find all the beautiful things in their collection with Elasticsearch.
- Vic Hargrave shares all about how to use the ELK stack for managing OSSEC alert logs.
- Using Elasticsearch on AWS? Awesome. Pavel Polyakov wrote up a great two part tutorial to help you get started. Check out preparing the environment & configuring Elasticsearch.
- The latest use case for Elasticsearch at Mozilla, this time from their stalwart intern Marcell Vazquez-Chanlatte.
- Our partner firm, Exensio, published the latest installment in their series on using Elasticsearch's aggregations along with D3.js to visualize data from IoT devices. (auf Deutsch)
Slides & Videos
New to Elasticsearch? Rick Winfrey introduces you to its features, with great getting started code examples in Ruby. Thanks Rick!
Isabel Drost-Fromm treats us to walk through of the options of determining search quality & how Elasticsearch can help you in your quest
Rafael Lopes on all things Elasticsearch, AWS & High Availability at the recent Elasticsearch Brasil Meetup
An rogue elk stares from an office window ... photo highlights of the day http://t.co/GmGmu0Lkpo pic.twitter.com/WDAw1YHYyq
— The Guardian (@guardian) August 25, 2014
Where to find Us
We'd love to feature all the great Elasticsearch, Logstash, and Kibana presentations and meetups happening worldwide in this section. If you're speaking or hosting a meetup, let our Community Manager, Leslie Hawthorn, know!
Yann Cluchey, the awesome human who organizes the Elasticsearch London User Group, will take the stage at GOTO Aarhus on Tuesday, September 30th to share his company's Elasticsearch use case. Join Yann at 1:20 PM too hear all about Cogenta - Making Enterprise Data Available in Real Time with Elasticsearch. GOTO Aarhus runs September 29-30th.
We've got two great talks coming up on September 4th! Boaz Leskes will present at NoSQL Matters Dublin on Elasticsearch: Deep Dive into Analytics Using Aggregations. For those not attending the conference or in need of even more Elasticsearch goodness, you can catch Boaz on the full ELK stack at the DevOps Ireland Meetup that same evening. Many thanks to the DevOps Ireland folks for hosting us!
The next Elasticsearch NL meetup will be hosted at Utrecht HQ of @bol_com, talking about their ELK stack.C u Sep 22! http://t.co/c66D6rfIlm
— Boaz Leskes (@bleskes) August 25, 2014
We're also super excited that this year's DrupalCon EU will be in our hometown of Amsterdam on Sept. 29th - Oct. 3rd. Leslie Hawthorn will be out and about in the hallway track, so make sure to say hello to her! There are also some great talks on all things Elasticsearch and the ELK stack on offer from these Drupalistas:
- Steven Merill of Phase2 Technology will teach you all about using open source logging and monitoring tools. The best part? He'll share all about how the ELK stack powers log management for Drupal.org (think billions of logs). Steve speaks on Wednesday at 2:15 PM.
- Nikolay Ignatov and Welin Welchev from Propeople will present on Elasticsearch, the New School Search Engine for Drupal. Join them on Thursday at 10:45 AM to learn all about the Elasticsearch Connector module!
Heading to JavaZone in Oslo? David Pilato will host a workshop showing you how to get Hands on with Elasticsearch and Kibana. David's workshop takes place on September 9th at 1:30 PM, and the conference runs September 9-11th.
- New Meetup: The Elasticsearch London User Group will convene on September 3rd, and registration is now open. Don't let that we're looking for speaker notice fool you - our stalwart community organizer, Yann Cluchey of Cogenta, has a full line up set. We'll announce full agenda and location details soon!
- Please join the folks from the Enterprise Search Cambridge group for a full day Elasticsearch hackathon on September 26th! You can register now to attend.
- New Meetup: The LOPSA San Diego user group will get together tomorrow at 6 PM to talk Docker basics plus using Docker to deploy and manage a live multi-node cluster of Elasticsearch, Redis, and Logstash. Registration is still open!
- If you're attending the Heartland Developers Conference in Omaha, make sure to check out Andrew Oschner's talk on the ELK stack. Andrew will teach you all about troubleshooting logging in production, and he'll have some cool Elasticsearch swag to hand out to session participants. Andrew speaks on September 4th, and the conference runs the 3rd-5th.
- Attending the first ever FOSSETCON? Join our very own Jordan Sissel for a Keynote Address on Open Source is More than Code. You can also see another of our stalwart Logstash developers, Aaron Mildenstein, on an introduction to the ELK stack. FOSSETCON runs September 11-13th in Orlando.
- Join several of our core developers for the latest Denver Meetup on September 15th. We're still finalizing the agenda, but our hosts, Sendgrid, will be discussing their Elasticsearch use case. Register now for updates on timing and location.
- PuppetConf is around the corner! If you find yourself in the San Francisco area join us in listening to Jordan Sissel's talk on Got Logs? Get answers with Elasticsearch ELK, Wednesday 24th, 5:10 PM. The conference runs September 20-24th.
Excellent turn out for tonight's #Elasticsearch Chicago Meetup. Many thanks to @kCura for speaking and hosting us! pic.twitter.com/5WOb6C66no
— Leslie Hawthorn (@lhawthorn) August 20, 2014
Where to Find You
Our Community Manager, Leslie Hawthorn, is hard at work to help folks create more Elasticsearch meetup groups and to help meetup organizers find more speakers. If you are interested in either effort, take a moment to let her know.
Oh yeah, we're also hiring. If you'd like us to find you for employment purposes, just drop us a note. We care more about your skill set and passion for Elasticsearch, Kibana, and Logstash than where you rest your head.
If you are interested in Elasticsearch training we have courses taught by our core developers coming up in:
- New York - August 27, 2014 (ELK workshop)
- London - September 2, 2014 (core Elasticsearch training)
- London - September 4, 2014 (ELK workshop)
- Paris - September 9, 2014 (core Elasticsearch training)
- San Francisco - September 11, 2014 (ELK workshop)
- Austin - September 18, 2014 (ELK workshop)
- Bangalore - September 24, 2014 (core Elasticsearch training)
- Paris - September 25, 2014 (ELK workshop)
- Stockholm - October 7, 2014 (core Elasticsearch training)
- Madrid - October 7, 2014 (core Elasticsearch training)
- Amsterdam - October 14, 2014 (core Elasticsearch training)
- Chicago - October 14, 2014 (core Elasticsearch training)
- San Francisco - October 14, 2014 (core Elasticsearch training)
- Los Angeles - October 21, 2014 (core Elasticsearch training)