This Week in Elasticsearch - February 18, 2015
Welcome to This Week in Elasticsearch. In this roundup, we try to inform you about the latest and greatest changes in Elasticsearch. We cover what happened in the GitHub repositories, as well as many Elasticsearch events happening worldwide, and give you a small peek into the future of the project.
It's ready! #Kibana 4 RC1 is freshly baked. Get the skinny on all the features or just jump right in. http://t.co/AvnxrOZWSA #HappyAnalyzing
— elasticsearch (@elasticsearch) February 12, 2015
Elasticsearch core
- Core: Remove ability to run optimize and upgrade APIs async (#9640, 2.0.0, 1.5.0)
- Cluster task: Ensure that we don’t pass negative
timeInQueue
towriteVLong()
(#9662, 1.5.0, 1.4.4, 1.3.9) - Testing: Introduce basic validation of the REST spec (#9629, 2.0.0, 1.5.0, 1.4.4)
- Internal: Introduce
TimedPrioritizedRunnable
base class to all commands that callInternalClusterService.updateTasksExecutor
(#9671, 2.0.0, 1.5.0, 1.4.4) - Testing: Ensure
RepeatOnException
rule is available in test-jar (#9675, 2.0.0, 1.5.0, 1.4.4) - Internal: only ack index store deletion on data nodes (#9672, 2.0.0, 1.5.0)
- REST: expose
master_timeout
flag onGET _template
&HEAD _template
(#9688, 2.0.0, 1.5.0) - Recovery: Handle corruptions during recovery finalization (#9619, 2.0.0, 1.5.0)
- Store: Use
Directory#fileLength()
less during calculating checksums (#9689, 2.0.0, 1.5.0) - Aggregations: Prevent negative intervals in
date_histogram
(#9690, 2.0.0, 1.5.0, 1.4.4) - Core: Move as much as possible into abstract Engine (#9678, 2.0.0, 1.5.0)
- Build: Add marvel and license to the release script (#9697, 2.0.0, 1.5.0, 1.4.4)
- Internal: Removed
Master
from names.txt (#9698, 2.0.0, 1.5.0) - Internal: Remove XCollector (#9677, 2.0.0)
- Aggregations: Clean up
date_histogram
(#9637, 2.0.0) - Engine: Move more methods into abstract
Engine
(#9717, 2.0.0, 1.5.0) - Gateway: Don’t rename recovery translog in gateway (#9719, 2.0.0)
- Query API: Added nested scope to parse context that keeps track the current nested level during search request parsing (#9692, 2.0.0, 1.5.0)
- Core: move
InternalEngine.segmentStats()
into abstract Engine (#9728, 2.0.0, 1.5.0) - Build: Rename Maven repository id (#9733, 2.0.0, 1.5.0)
- Packaging: Add antlr and asm dependencies (9696, 2.0.0, 1.5.0, 1.4.4)
- Mapping: Require the full path (without type) for field lookups (#8872, 2.0.0)
- FileSystemUtils: Only create backup copies if files differ (#9592, 2.0.0)
- CliTool: Add command to warn on permission/owner change (#9508, 2.0.0)
- Aggs: Refactor aggregations to use lucene5-style collectors (#9544, 2.0.0)
- Cat API: Change behaviour of indices segments api to allow no indices (#9219, 2.0.0, 1.5.0)
- Dependencies: Update joda-time to v2.7 (#9610, 2.0.0, 1.5.0)
- Allow configuration of the GC log file via an environment variable (#8479, 2.0.0)
- Term Vectors: Requests times are now measured (#9583, 2.0.0, 1.5.0)
In Apache Lucene this Past Week
- Many silly javadocs problems are fixed, and future problems are now quickly detected using javac’s doclint options, including a workaround for a Java 8 javadoc regression, plus better IDE support.
- Awesome progress on the marriage of filters and queries: BooleanQuery now also accepts BooleanClause.FILTER, applying the filter using its existing high-performance conjunction logic for MUST clauses, and (finally!), Filter now extends Query.
- Two-phase intersection breaks costly queries like PhraseQuery into a cheap (conjunction of the terms) and expensive part (checking all positions) so that BooleanQuery can first get global agreement on all cheap parts across all clauses before consulting the expensive parts. For example a distance filter could be a cheap bounding box plus an expensive true distance calculation.
- FSDirectory.listAll() no longer does a costly fstat operation on each file.
- On trunk, Lucene’s core now compiles with Java8′s compact1 profile (all other modules are compact2), making it clear that we only require a limited portion of the JDK’s APIs.
- Lucene no longer uses Java’s Unsafe APIs.
- A new diversifying collector, in Lucene’s misc module, lets you collect diverse yet still high-scoring hits.
- CheckIndex is much faster when the index has term vectors.
- SparseFixedBigSet.ramBytesUsed now works on JVMs whose memory allocations are not 8-byte aligned.
- Lucene’s spatial module can now do 2D spatial faceting, e.g. to generate geographic heatmaps.
- The useful index debugging tool, Luke, may become a Lucene module, making it more first-class so it’s less likely to break with each Lucene release.
- We were horrified to discover that test randomness had been eliminated in some tests due to accidentally committed @Seed annoations. Those tests are fixed and we have now banned @Seed entirely.
#ODI log analysis in #Elasticsearch and #Kibana.
30 minutes from concept to dashboard; great tools to work with. pic.twitter.com/23t4vuxnon
— Robin Moffatt (@rmoff) February 10, 2015
Elasticsearch Ecosystem
Here’s some more information about what is happening in the ecosystem we are maintaining around the ELK stack – that’s Elasticsearch plus Logstash and Kibana – including plugin and driver releases.
- Kibana 4 RC1 has been released, please test and give feedback!
- Shield 1.0.1 has been released in order to work with Elasticsearch 1.4.3.
- Adrien Grand authored a nice blog post about roaring bitmaps.
- From the Docker Files: Raúl Cumplido shares his tips for getting the ELK stack up and running, and how he created a Docker container for the 3 services to ensure a repeatable process for future deployments.
- If you’re using Apache Hadoop and Spark, you won’t want to miss this post from Avi Levi on using the ELK stack for setting up a central logging infrastructure.
- Dan Kingshott’s excellent post on open source log aggregation gives an overview of how to set up Kibana in an OpenStack cloud or in VMs, and shares how to use Chef for configuration management in this environment as well.
- Interested in alerting for your website? Elasticsearch’s Percolator feature is your friend! Fabio Ponciroli wrote up how he and his colleagues added this functionality to their infrastructure at their company’s recent hack day.
- An Elasticsearch full-text search integration for the dovecot IMAP server has been published.
Slides & Videos
Joe Jasinski shows how he uses Django, Elasticsearch, and Haystack for searching the contents of his website to the Chicago Python User Group.
The MongoDB meetup group in Delhi recently heard from Bharvi Dixit about some use cases for MongoDB with Elasticsearch.
The Elasticsearch meetup in London had some fantastic presentations at their last gathering.
Where to Find Us
We'd love to feature all the great Elasticsearch, Logstash, and Kibana presentations and meetups happening worldwide in this section. If you're speaking or hosting a meetup, let our Director of Developer Relations, Leslie Hawthorn, know!
Austria
The Vienna Elasticsearch Users Group will get together on March 4 to talk Elasticsearch at Cloud Foundry and more. Register now to let the organizers know you’re attending.
Germany
Want to know more @logstash and happen to be in Berlin next 24/02, join me @elasticberlin (http://t.co/Kh6SyckgnW) for an all night on logs!
— Pere Urbón-Bayes (@purbon) February 11, 2015
India
The Configuration Management Magic Meetup will convene on February 21 in Bangalore. Among the many talks on offer, you can hear all about Log Analysis using Elasticsearch, Kibana and Fluentd. Register now to save your seat.
Israel
The Tel Aviv-Yafo ELK meetup group will be talking How to Use ELK to Analyze Logs from a Large Production AWS Environment on February 24. You can register now to attend.
Speaker Update: Peter Vulgaris from @facebook speaks in the #Elasticsearch track at #GOTOams: http://t.co/TcA6fFyHjp
— GOTO Amsterdam (@GOTOamst) February 11, 2015
South Africa
The inaugural Capetown Elasticsearch Meetup will convene on March 5 to talk shop and plan for the future of the group. Register now to let the organizers know you plan to attend.
Taiwan
The Agile Code Camp team are convening a developer and designer hack day on February 23, and attendees will be developing with Elasticsearch. Register now to attend the full day event.
United Kingdom
Elasticsearch will be out in force at QCon London, which returns to the Queen Elizabeth II Conference Center this year. You can visit us at our booth on the show floor, plus we’ll be having one of our engineers take the stage for the main program. Those details are in the works, but in the meantime you can take a look at Kristoffer Dyrkorn’s talk information. He’ll be sharing the story of how Elasticsearch and other technologies are powering the Norwegian Roads Authority’s brand new system to provide real-time traffic information to travelers throughout Norway.
United States
If you’re at Strata in San Jose this week, don’t forget to join Costin Leau, creator of Elasticsearch for Apache Hadoop, at the conference. He’ll present on Search Evolved: Unraveling Your Data on Friday, Feb 20 at 11:30 AM.
Coming up in San Francisco next month, you can join the SF MySQL Meetup to hear all about using MySQL and the ELK stack for audit logging. The user group will get together on March 11, but the event is filling up quickly. Don’t miss out! Register now to save your seat.
And for our friends beyond Silicon Valley:
- Robyn Bergeron will be speaking this Saturday at SCALE 13x on DevOps + Open Source == BFF Practices! Join her to learn more about DevOps (practice, theory, and otherwise!), shared habits of successful open source communities and DevOps practitioners, and tips for how you or your organization can start applying these habits today.
- If you’re attending the Linux Foundation Collaboration Summit, make sure to say hello to Leslie Hawthorn. She’ll be there to answer any of your questions about the ELK stack in the hallway track. Collab Summit is on Feb 18-20 in Santa Rosa, California.
- If you’re in Boston, be sure to join the Boston Elasticsearch meetup group on March 2 to hear about how to build a scalable ELK stack on AWS. Be sure to save yourself a spot — we hope you can dig out of the snow by then!
- The next Elasticsearch NYC meetup is coming up on March 18. Sign up now
- The Chicago MySQL Users Group will be getting together on Feb 19 to talk MySQL Audit Logging and the ELK stack. Register now to save your seat.
- For folks in Atlanta who love Elasticsearch and OpenStack, the meetup on Feb 19 will be a great place to be. Sign up now to hear about Managing Your OpenStack logs with ELK.
It's ready! #Kibana 4 RC1 is freshly baked. Get the skinny on all the features or just jump right in. http://t.co/AvnxrOZWSA #HappyAnalyzing
— elasticsearch (@elasticsearch) February 12, 2015
Where to Find You
PSST! If you’re a regular reader of This Week in Elasticsearch, a.k.a TWIES, you’re thinking of skipping this section. You may even be thinking to yourself, yes of course I will drop a note on Twitter when I am giving a talk on all things ELK. That’s awesome, because we’d like to showcase every meetup, conference presentation and workshop on Elasticsearch, Logstash, and Kibana happening worldwide. And now, we’ve made it even easier for you to get support for your meetup!
Head on over to our meetups page! (And we’ll still totally send you swag if you’re giving a talk on anything ELKy at a conference.)
Oh yeah, we're also hiring. If you'd like us to find you for employment purposes, just drop us a note. We care more about your skill set and passion for Elasticsearch, Kibana, and Logstash than where you rest your head.
Trainings
If you are interested in Elasticsearch training we have courses taught by our core developers coming up in:
- Amsterdam – February 19, 2015 (Core Elasticsearch Training)
- London – February 25, 2015 (Hands on Workshop)
- London – February 26, 2015 (Core elasticsearch)
- Bangalore – February 25, 2015 (Hands on Workshop)
- Bangalore – February 26, 2015 (Core Elasticsearch Training)
- New Delhi – March 17, 2015 (Core Elasticsearch Training)
- San Francisco – March 17, 2015 (Hands on Workshop)
- Mountain View – March 18, 2015 (Core Elasticsearch Training)
- Northern Virginia – March 24, 2015 (Core Elasticsearch Training)
- Munich – March 24, 2015 (Core Elasticsearch Training)
- Munich – March 26, 2015 (Hands on Workshop)
- Stockholm – March 25, 2015 (Core Elasticsearch Training)
- Los Angeles – March 25, 2015 (Core Elasticsearch Training)
- Paris – March 25, 2015 (Hands on Workshop)
- Paris – March 26, 2015 (Core Elasticsearch Training)
- San Francisco – April 7, 2015 (Core Elasticsearch Training)
- Seoul – April 8, 2015 (Core Elasticsearch Training)
- Tokyo – April 13, 2015 (Core Elasticsearch Training)
- Amsterdam – April 14, 2015 (Hands on Workshop)
- Austin – April 23, 2015 (Core Elasticsearch Training)
- New York – April 28, 2015 (Hands on Workshop)
- New York – April 29, 2015 (Core Elasticsearch Training)