This Week in Elasticsearch - September 03, 2014
Welcome to This Week in Elasticsearch. In this roundup, we try to inform you about the latest and greatest changes in Elasticsearch. We cover what happened in the GitHub repositories, as well as many Elasticsearch events happening worldwide, and give you a small peek into the future of the project.
- Mappings: Added support for empty field arrays (#7271, master and 1.4)
- Index templates: Made template filtering generic and extensible via plugins (#7454, master and 1.4)
- Explain API: fix explanation streaming (#7257, master and 1.4)
- Query API: Return
match_allin case there are no clauses in boolean query (#7240, master, 1.4 and 1.3)
- Fielddata: Remove soft/resident caches (#7443, master and 1.4)
- Transaction log: Add checksums (#7232, master and 1.4)
- Internal: Do not execute cluster state changes if current node is no longer master (#6230, master and 1.4)
- Discovery: Do not use versions to optimize cluster state copying for a first update from a new master (#6466, master and 1.4)
- Discovery: Immediately start Master|Node fault detection pinging (#6706, master and 1.4)
- FunctionScore: Refactor RandomScoreFunction to be consistent, and return values in range
[0.0, 1.0](#7446, master and 1.4)
- Filters: Introduced
FixedBitSetFilterCachethat guarantees to produce a
FixedBitSetand does evict based on size or time (#7037, master and 1.4)
- Index create API: Validate create index requests' number of primary/replica shards (#7496, master and 1.4)
- Test: write heap dump to log folder (#7452, master and 1.4)
- Aggregations: Encapsulate
AggregationBuildername and make getter public (#7425, master and 1.4)
- Scan: Use
HashMap, because the readerStates is accessed by multiple threads during the entire scroll session (#7478, master, 1.4, 1.3 and 1.2)
- Client: Make the query builder nullable in
filteredQuery(#7398, master and 1.4)
- Suggest API: Fix encoding of multiple levels for geo precision (#7368, master and 1.4)
- Core: Don't update indexShard if it has been removed before (#7509, master and 1.4)
- Aggregations: The nested aggregator should iterate over the child doc ids in ascending order (#7514, master, 1.4 and 1.3)
- Suggest API: Improved java client API (#7435, master and 1.4)
- Rest API: Add the cluster name to the
/endpoint (#7524, master and 1.4)
- Mappings: keep parameters in mapping for
_sizeeven if disabled (#7475, master, 1.4, 1.3 and 1.2)
- Function Scoring: Added optional weight parameter per function (#6955, master and 1.4)
- Internal: Extract a common base class for (Master|Nodes)FaultDetection (#7512, master and 1.4)
- Cluster: Refactored
ClusterStateUpdateTaskprotection against execution on a non master (#7511, master and 1.4)
- Discovery: Accumulated improvements to
ZenDiscovery(#7496, master and 1.4)
- Indexing: Use Flake IDs instead of random UUIDs when auto-generating id field (#5941, master and 1.4)
- Engine: Verify checksums on merge (#7360, master and 1.4)
- Internal: Do not use a background thread to disconnect node which are removed from the
ClusterState(#7543, master and 1.4)
- Aggregations: Forbid usage of aggregations in conjunction with
search_type=SCAN(#7429, master and 1.4)
Here's some more information about what is happening in the ecosystem we are maintaining around the ELK stack - that's Elasticsearch plus Logstash and Kibana - including plugin and driver releases.
If you use Elasticsearch with Laravel, I've got a package (beta) that integrates ES types with Laravel models: https://t.co/s4Kk5C6cWB
— Adam Fairholm (@adamfairholm) August 29, 2014
- We've released the latest versions of Mapper Attachments for Elasticsearch and Spring Factories for Elasticsearch.
- Florian Hopf published the latest in his series on Elasticsearch use cases. This week, you can enjoy a deep dive on Geospatial Search.
- Jeff Potts penned a tutorial on using Apache JMeter to test Elasticsearch. We're looking forward to your post on visualizing the results of your testing with the ELK stack, Jeff!
- Fabien Wernli shared an incredibly in-depth tutorial, including demo, on modern monitoring and alerting using Elasticsearch, Riemann, and more.
- Kunihiko Kido wrote an excellent article on how to use the Elasticsearch top hits aggregation. (日本語で)
L'usine logicielle de Voyages-SNCF. Avec du #elasticsearch dedans :) http://t.co/yB6f5v4j8l
— David Pilato (@dadoonet) September 1, 2014
Slides & Videos
Great overview of the ELK stack for logging & data visualization
For all you PaaS lovers out there, check out how to use the ELK stack on OpenShift Origin
Great exploration of using the ELK stack and other tools to monitor your infrastructure
Where to Find Us
We'd love to feature all the great Elasticsearch, Logstash, and Kibana presentations and meetups happening worldwide in this section. If you're speaking or hosting a meetup, let our Community Manager, Leslie Hawthorn, know!
The Vienna Elasticsearch User Group will convene on October 2nd at 7 PM. This next meetup will focus on Elasticsearch use cases and the organizers are looking for more folks to share their story! Interested in presenting? Just want to attend, network and learn? You can register now for the meetup or volunteer to present via meetup.com.
- Dmitri Marx will be presenting at next week's Findability Day in Copenhagen! Next Thursday, Sept. 11th, Dmitri will teach you all about how Elasticsearch can provide you with actionable insights into your data. Check out all the details on Findability Day, an event organized by our partner firm, Findwise.
- Yann Cluchey, the awesome human who organizes the Elasticsearch London User Group, will take the stage at GOTO Aarhus on Tuesday, September 30th to share his company's Elasticsearch use case. Join Yann at 1:20 PM too hear all about Cogenta - Making Enterprise Data Available in Real Time with Elasticsearch. GOTO Aarhus runs September 29-30th.
We've got two great talks on tomorrow, September 4th in Dublin! Boaz Leskes will present at NoSQL Matters Dublin on Elasticsearch: Deep Dive into Analytics Using Aggregations. For those not attending the conference or in need of even more Elasticsearch goodness, you can catch Boaz on the full ELK stack at the DevOps Ireland Meetup that same evening. Many thanks to the DevOps Ireland folks for hosting us!
まだまだ募集中です！ / 『サーバ/インフラエンジニア養成読本 ログ収集〜可視化編』 出版記念！執筆者が語る大講演会！ - dots. - http://t.co/9JsXb8XLMg
— Jun Ohtani (@johtani) September 1, 2014
参加募集を開始しました。/ 第6回elasticsearch勉強会 #elasticsearch #elasticsearchjp - elasticsearch勉強会 | Doorkeeper - http://t.co/vJk89DnXQ2
— Jun Ohtani (@johtani) August 29, 2014
The next Elasticsearch NL meetup will be hosted at Utrecht HQ of @bol_com, talking about their ELK stack.C u Sep 22! http://t.co/c66D6rfIlm
— Boaz Leskes (@bleskes) August 25, 2014
We're also super excited that this year's DrupalCon EU will be in our hometown of Amsterdam on Sept. 29th - Oct. 3rd. Leslie Hawthorn will be out and about in the hallway track, so make sure to say hello to her! There are also some great talks on all things Elasticsearch and the ELK stack on offer from these Drupalistas:
- Steven Merill of Phase2 Technology will teach you all about using open source logging and monitoring tools. The best part? He'll share all about how the ELK stack powers log management for Drupal.org (think billions of logs). Steve speaks on Wednesday at 2:15 PM.
- Nikolay Ignatov and Welin Welchev from Propeople will present on Elasticsearch, the New School Search Engine for Drupal. Join them on Thursday at 10:45 AM to learn all about the Elasticsearch Connector module!
Save the date & watch this space: next #Elasticsearch NL meetup will be livestreamed at http://t.co/Ily9dsBFSo (22 Sept, starting @ 19 CEST)
— Leslie Hawthorn (@lhawthorn) September 1, 2014
Heading to JavaZone in Oslo? David Pilato will host a workshop showing you how to get Hands on with Elasticsearch and Kibana. David's workshop takes place on September 9th at 1:30 PM, and the conference runs September 9-11th.
- Please join the folks from the Enterprise Search Cambridge group for a full day Elasticsearch hackathon on September 26th! You can register now to attend.
- New Meetup: Ancestry.com will be hosting the next San Francisco Elasticsearch Meetup! You can join us on September 9th to hear about Elasticsearch Core and the ELK Stack from members of our developer team. Full details coming soon, but in the meantime get registered now!(This meetup always fills up quickly, so please do save a spot now or release your place on the RSVP list if you know you cannot attend.)
- If you're attending the Heartland Developers Conference in Omaha, make sure to check out Andrew Oschner's talk on the ELK stack. Andrew will teach you all about troubleshooting logging in production, and he'll have some cool Elasticsearch swag to hand out to session participants. Andrew speaks on September 4th, and the conference runs the 3rd-5th.
- Attending the first ever FOSSETCON? Join our very own Jordan Sissel for a Keynote Address on Open Source is More than Code. You can also see another of our stalwart Logstash developers, Aaron Mildenstein, on an introduction to the ELK stack. FOSSETCON runs September 11-13th in Orlando.
- Join several of our core developers for the latest Denver Meetup on September 15th. We're still finalizing the agenda, but our hosts, Sendgrid, will be discussing their Elasticsearch use case. Register now for updates on timing and location.
- PuppetConf is around the corner! If you find yourself in the San Francisco area join us in listening to Jordan Sissel's talk on Got Logs? Get answers with Elasticsearch ELK, Wednesday 24th, 5:10 PM. The conference runs September 20-24th.
- In or around Bellvue, WA on October 2nd? Cool, so are we! Please join Steve Mayzak to learn all about the ELK stack: what it is, what you can do with it and how it can provide actionable insights into your data. You can register now to attend the CloudTalk Meetup, and many thanks to these fine folks for hosting us!
Where to Find You
Our Community Manager, Leslie Hawthorn, is hard at work to help folks create more Elasticsearch meetup groups and to help meetup organizers find more speakers. If you are interested in either effort, take a moment to let her know.
Oh yeah, we're also hiring. If you'd like us to find you for employment purposes, just drop us a note. We care more about your skill set and passion for Elasticsearch, Kibana, and Logstash than where you rest your head.
If you are interested in Elasticsearch training we have courses taught by our core developers coming up in:
- Paris - September 9, 2014 (core Elasticsearch training)
- San Francisco - September 11, 2014 (ELK workshop)
- Austin - September 18, 2014 (ELK workshop)
- Paris - September 25, 2014 (ELK workshop)
- Stockholm - October 7, 2014 (core Elasticsearch training)
- Madrid - October 7, 2014 (core Elasticsearch training)
- Amsterdam - October 14, 2014 (core Elasticsearch training)
- Chicago - October 14, 2014 (core Elasticsearch training)
- Los Angeles - October 21, 2014 (core Elasticsearch training)