This week in Elasticsearch - July 02, 2014
Welcome to This Week in Elasticsearch. In this roundup, we try to inform you about the latest and greatest changes in Elasticsearch. We cover what happened in the GitHub repositories, as well as many Elasticsearch events happening worldwide, and give you a small peek into the future of the project.
- JVM Options: Disable explicit GC by default (#6637, master and 1.3)
- Recovery API: Report ongoing relocation recoveries (#6585, master and 1.3)
- Scripting: Wrap groovy script exceptions in a serializable Exception object (#6598, master and 1.3)
- Mapping API: Update mapping on master in async manner (#6648, master and 1.3)
- Recovery: from local gateway should re-introduce new mappings (#6659, master and 1.3)
- Snapshot/Restore API: Add ability to restore partial snapshots (#5742, master and 1.3)
- Snapshot/Restore API: Improve deletion of corrupted snapshots (#6383, master, 1.3 and 1.2)
- Core: The
ignore_unavailable=truesetting also ignores indices that are closed (#6471, master, 1.3 and 1.2)
- Core: Prevent non segment readers from entering the filter cache and the field data caches (#6553, master, 1.3 and 1.2)
- Core: cancel entire recovery if shard closes on target node during the recovery operations (#6645, master and 1.3)
- Recovery: Cancel recovery if shard on the target node closes during recovery operation (#6645, master and 1.3)
- Translog: Better support for partial buffer reads/writes in translog infrastructure (#6576, master, 1.3 and 1.2)
- Scripting: Add sandboxing for GString-based method invocation (#6596, master and 1.3)
- Scripting: Switch to Groovy as the default scripting language (#6571, master)
- Query API: Support Match query with operator and, cutoff_frequency and stacked tokens (#6573, master, 1.3 and 1.2)
- Scripting: Remove MVEL as a built-in scripting language (#6610, master)
- Scripting: Add a transformer to translate constant BigDecimal to double for groovy (#6609, master and 1.3)
- Dependencies: Upgrade to Lucene 4.9 (#6623, master and 1.3)
- Internal: Make sure we don't reuse arrays when sending and error back in netty transport channel (#6631, master, 1.3 and 1.2)
- Thread Pools: Rejection status code should be 429 to tell client to back off (#6629, master and 1.3)
- Store: Wrap
RateLimiterrather than copy
RateLimitedIndexOutput(#6625, master and 1.3)
- Recovery: Wait for mapping updates during local recovery (#6666, master and 1.3)
Full house at last week's ELK Stack open space session at DevOps Days Silicon Valley
Photo credit: Ilan Rabinovitch
Here's some more information about what is happening in the ecosystem we are maintaining around the ELK stack - that's Elasticsearch plus Logstash and Kibana - including plugin and driver releases.
OpenTable <3s Elasticsearch
LOL our logging ElasticSearch cluster has just reached 1 Billion documents :) pic.twitter.com/XPdCdxkhry
— Paul Stack (@stack72) June 30, 2014
- We have released the official Smart Chinese Analysis Plugin, version 2.2.0, and the official memcached Transport Plugin, version 2.2.0.
- In case you missed it, Elasticsearch is now "Certified on Spark." Read all about our Spark integration from Costin Leau, creator of Elasticsearch for Apache Hadoop.
- Jon Gifford from Loggly treated us to a deep dive on why they chose Elasticsearch for Reliable, Scalable Log Management.
- From the cool apps department: Photon Maps, an open source geocoder built for OpenStreetMap data and based on Elasticsearch. Take a look at their awesome demo, or head over to GitHub to get the code.
- Ravi Bhatt shared his experiences with Elasticsearch Query Performance Optimization. The article is quite detailed, but read all the way through to get ideas on how to do proper performance tuning and minimize latency.
- Florian Hopf treated us to a deep dive on some of the options for integrating Elasticsearch with a Java application.
- Pedro Alonzo authored an excellent how to deploying an Elasticsearch based app on Microsoft Azure.
- Yuriy Bondaruk wrote up an introduction to Using the Power of Real-Time Distributed Search with Elasticsearch. It's the first article in a series, so stay tuned for more!
- For Windows users, there's a new guide to installing Logstash on a Windows server with Kibana in IIS.
- If you love Logstash and Zabbix, you'll enjoy this how to on sending comments to Logstash from within the Zabbix frontend.
- Tomas Jansson authored an extensive post on making CQRS simple by using eventstore and Elasticsearch.
- Our partner firm Exensio has penned an article on using Elasticsearch for the Internet of Things. Read on for a taste of using it to store sensor data. (auf Deutsch)
- Jun Ohtani has joined Elasticsearch as our first employee in Tokyo! He asked us to share this news with our users in Japan: はてなブックマークの新機能がElasticsearchにより実現されました。
Slides & Videos
Costin Leau's Presentation from the recent Hadoop Summit North America 2014
Britta Weber explains the significant terms aggregation, a feature that allows to users to identify terms that are relevant to a particular set of documents
- Ryan Clark treats us to a look at how Ninefold uses the ELK stack to present a multitude of logged information to their hosting customers and our support engineers. If you enjoy the slides, you'll love the accompanying blog post.
- Damien Alexandre's great presentation on using Elasticsearch to Search Like a Boss. (en français)
Jean Baptiste Favre shares details on Blablacar's system architecture and how they use the ELK stack (en français)
Where to find Us
We'd love to feature all the great Elasticsearch, Logstash, and Kibana presentations and meetups happening worldwide in this section. If you're speaking or hosting a meetup, let our Community Manager, Leslie Hawthorn, know!
The 8th Elasticsearch France Meetup will take place in Paris on July 7th at Coworkshop. Space is very much limited, so take a look and register now to save your place.
- New meetup: If you're in or around Cologne on July 9th, please join Alexander Reelsen and Britta Weber for the first ever Elasticsearch Meetup in Cologne! Alex will cover What's New in Elasticsearch, and Britta will present on the Significant Terms Aggregation. Doors open at 6:30 PM, and registration is open.
- Our brand new employee in Germany, Dmitri Marx, will be attending Java Forum Stuttgart along with Livia Froelicher. Make sure to say hello to them between sessions! The conference takes place on July 17th at the Culture & Convention Centre Liederhalle.
- EuroPython comes to Berlin this year, with the conference running from July 21-27th at the Berlin Congress Center. Please stop by the Elasticsearch table to learn all about how the ELK stack can make life better for Pythonistas. Make sure to catch Honza Kral, Django and Elasticsearch core developer, on lessons learned from building the Python Elasticsearch client. Honza will take the stage at 2:00 PM on July 24th. Bonus: we're hosting a meetup during the conference - details to follow soon!
- Following on from EuroPython, Honza Kral will be speaking at the PyData Conference on Making Sense of your big data with Elasticsearch. Honza presents on July 27th at 2:10 PM, and the PyData Conference will also be held at the Berlin Congress Center.
- The IGT Cloud Meetup Group is hosting a workshop on Applied Cloud Computing with Google Cloud Platform at Google Tel Aviv. Topics will include BigQuery with Logstash as Application Log Analysis Platform. The workshop runs from 9:30-13:30 on July 7th.
- The OpsTalk Meetup Group will get together on
- The Second Elasticsearch Tel Aviv Meetup will convene on July 28th, focusing on real world use cases. The agenda is now finalized and you can hear from three different companies on how they use Elasticsearch. You can register now.
The very first Elasticsearch Italy Meetup has been scheduled for July 16th! Please join us to hear from Luca Cavanna on What's New in Elasticsearch. You'll also be treated to a case study talk on how Lutech uses Elasticsearch in its Threat Management System for Breach Detection, Intelligence & Response. Doors open at 6:00 PM, and registration is now open.
The Elasticsearch Study Session will convene on July 14th at 6:30 PM. Register now to get a chance to hear from our core developers Honza Kral and Igor Motov. Even cooler, our CTO Shay Banon will be attending this meetup, so register now to save your place!
The Auckland JVM Users Group will rebooting their meetup series, with their newest offering focused on Elasticsearch. You can join them on Tuesday, July 22nd at 6:00 PM, and registration is open.
Clinton Gormley will be speaking on Scaling Real-Time Search and Analytics with Elasticsearch at Mimacom Days Barcelona on July 10th. Clint takes the stage at 9:15 AM.
- If you're in or around Bath on July 14th, the South-West Elasticsearch Community will convene to talk about how you're using Elasticsearch. You can register for the inaugural meeting, which will kick off at 7 PM.
- The London VoiP User Group will get together on July 22nd at 6:30 PM. These folks are in search of a venue, so if you're excited to hear about Eye-candy from CDRs with the ELK stack, register now and help these good folks find a place to meetup.
- The annual O'Reilly Open Source Conference is back again in Portland, Oregon! The Elasticsearch team will be out in force, with two talks, an ELK BoF and a lovely table in the exhibit hall where we can answer all of your questions. More details to follow, but we hope to see many of our friends there! OSCON runs from July 20-24th at the Oregon Convention Center.
- The TechLife Cincinnati Meetup Group will get together on July 16th at 6 PM to talk the ELK stack and other open source monitoring tools. The hosts are welcoming the Cincinnati Blacks in Technology group to present during this meetup.
- The Philadelphia Elasticsearch Meetup group will be getting together for their second meetup on June 26th. During this meetup, you'll learn all about how Wildbit uses Elasticsearch to build their popular applications, Postmark & Beanstalk. Doors open at 6:00 PM.
- The SYN/(st)ACK NYC User Group will convene on June 30th at 6:30 PM. Squarespace is hosting the festivities, and Franklin Angulo, one of their Software Engineers, will be presenting. Franklin will cover building a scalable monitoring and alerting system using Graphite, Grafana, Collectd, Nagios, Logstash, Elasticsearch, and Kibana, among others.
Where to Find You
Our Community Manager, Leslie Hawthorn, is hard at work to help folks create more Elasticsearch meetup groups and to help meetup organizers find more speakers. If you are interested in either effort, take a moment to let her know.
Oh yeah, we're also hiring. If you'd like us to find you for employment purposes, just drop us a note. We care more about your skill set and passion for Elasticearch, Kibana, and Logstash than where you rest your head.
If you are interested in Elasticsearch training we have courses taught by our core developers coming up in:
- Seoul - July 10, 2014 (core Elasticsearch training)
- London - July 14, 2014 (core Elasticsearch training)
- Tokyo - July 14, 2014 (core Elasticsearch training)
- Berlin - July 15, 2014 (ELK workshop)
- Milan - July 16, 2014 (core Elasticsearch training)
- Tel Aviv - July 23, 2014 (core Elasticsearch training)
- San Francisco - August 5, 2014 (core Elasticsearch training)
- Sao Paulo - August 14, 2014 (core Elasticsearch training)
- Rio de Janeiro - August 18, 2014 (core Elasticsearch training)