Welcome to This Week in Elasticsearch. In this roundup, we try to inform you about the latest and greatest changes in Elasticsearch. We cover what happened in the GitHub repositories, as well as many Elasticsearch events happening worldwide, and give you a small peek into the future of the project.
Lots of love from the Mozilla InfoSec Team
Wait, a realtime patch and vulnerability dashboard using elasticsearch and kibana? Why yes! (note: still beta) pic.twitter.com/ab14kwBDnA
— Michael Henry (@neoCrimeLabs) May 30, 2014
- Recycling: change the default type of the page recycler to
SOFT_CONCURRENT(#6320, master and 1.3)
- Dependencies: Upgrade to netty 3.9.1 (#6331, master and 1.3)
- Aggregations: Fixed bug in
top_hitsaggregation to not fail with NPE when shard results are empty (#6346, master and 1.3)
- Aggregations: added pagination support to
top_hitsaggregation by adding
fromoption (#6299, master and 1.3)
- Zen Discovery: Add dedicated join timeout that is higher than
ping.timeoutfor join (#6342, master and 1.3)
- Aggregations: Take the 'from' into account when getting a fetched hit
(commit, master and 1.3)
- Parent/Child: Added min_children/max_children to
has_childquery/filter (#6019, master and 1.3)
- Lucene: Reuse Lucene's TermsEnum for faster _uid/version lookup (#6212, master and 1.3)
- Geo: Valid Polygon crossing dateline failed to parse (#6179, master and 1.3)
- Mapping API: Fix
include_in_allin type (#6301, master, 1.3 and 1.2)
- Memory management: do not enforce the BigArrays limit on the network layer and the translog (#6332, master and 1.3)
- More Like This Query: values of a multi-value fields are compared at the same level (#6130, master and 1.3)
- Routing: Restore shard routing (#6391, master, 1.3 and 1.2)
- Translog: Revert unlimited flush_treshold_ops for translog (#6377, master, 1.3 and 1.2)
- Aggregations: Adds GeoBounds Aggregation (#5634, master and 1.3)
- Indexing: Clear
versionMapon refresh (not flush) to reduce heap usage (#6379, master and 1.3)
Here's some more information about what is happening in the ecosystem we are maintaining around the ELK stack - that's Elasticsearch plus Logstash and Kibana - including plugin and driver releases.
- We released Elasticsearch 1.2.1 yesterday. We recommend that anyone using version 1.2.0 upgrade immediately to version 1.2.1. This blog post has full details.
- We released Marvel 1.2 just a few hours ago. Get it while it's hot!
- Karel Minařík has released the elasticsearch-persistence Rubygem, which allows storage Ruby objects/models in Elasticsearch via the repository pattern.
- We've added two new chapters to The Definitive Guide to Elasticsearch: Aggregations and Dealing with Human Language.
- In case you missed it, we welcomed Elizabeth K. Joseph from HP as our latest guest blogger last week. Elizabeth shared an extensive guide to how the ELK stack makes life easier for OpenStack developers.
- SELKS, a Suricata and ELK stack based IDS/IPS, has been released.
- You may be interested in hearing how Crate, a zero-administration data store company, leverages Elasticsearch to build its product. (We love to hear that folks are inspired by Elasticsearch's simplicity!)
- Pivotal Cloud Foundry now ships with Elasticsearch as of their 1.2 release.
- The US Food and Drug Administration (FDA) has released Open FDA, an Elasticsearch-based API that serves up FDA publically accessible data about prescription medications. Stay tuned - they're planning to add access to their medical device and food data sets.
- François Gravelaine penned an article on using Elasticsearch and Symfony to hydrate objects via Transformers.
- James Robert Somers from Pivotal Labs authored an article on using our Ruby test cluster extension to launch a separate Elasticsearch cluster for isolating tests in Ruby.
- Henrik Walker Moe shared a write up of Bouvet's experiences moving to Elasticsearch on Azure. Lots of juicy details and diagrams!
- Jérémie Jourdin shared his experiences implementing access controls for Elasticsearch indices. (en français)
- Ricardo Martins wrote a detailed how to on deploying the ELK stack on CentOS. (Em Português Brasileiro)
MT @SandyBoydCA: Standing room only for @costinl's #hadoopsummit talk on #elasticsearch for real-time analytics pic.twitter.com/PmqUiL2xg0
— Paige Roberts (@RobertsPaige) June 3, 2014
Slides & Videos
One of the best talks @berlinbuzzwords #bbuzz Scoring for human beings by @a2tirb http://t.co/vu2l2yEl9V. Get the video when it's out !
— Lucian Precup (@lucianprecup) May 27, 2014
You got it, Lucian!
Britta Weber on Scoring for Human Beings at Berlin Buzzwords 2014
Adrien Grand on Elasticsearch Aggregations at Berlin Buzzwords 2014
Boaz Leskes on Two Use Cases for Scaling Data with Elasticsearch at Berlin Buzzwords 2014
And we'll have even more Buzzwords videos featured next week!
Where to find Us
We'd love to feature all the great Elasticsearch, Logstash, and Kibana presentations and meetups happening worldwide in this section. If you're speaking or hosting a meetup, let our Community Manager, Leslie Hawthorn, know!
RT @DataBe "“@DataScienceBe: Impressive presentations tonight about #elasticsearch by @wavyx @bigdatabe pic.twitter.com/gerhLX9L74""
— Brussels DataScience (@DataScienceBe) May 31, 2014
The organizers of the Benelux Elasticsearch User Group share their successes
- The Breizh JUG will welcome our very own David Pilato, to talk about all things Elasticsearch this Thursday. Amongst other topics, David will cover how Elasticsearch can make your life far easier than using plain old SQL queries. Doors open at 18:00.
- The Seventh Elasticsearch France User Group meeting is coming up on June 10th in Paris. Doors open at 7:30 PM.
The IGT Cloud Meetup Group is hosting a workshop on Applied Cloud Computing with Google Cloud Platform at Google Tel Aviv. Topics will include BigQuery with Logstash as Application Log Analysis Platform. The workshop runs from 9:30-13:30 on July 7th.
We've got not one but two awesome conferences going on in Amsterdam the week of June 16th:
- DevOps Days Amsterdam: Join us June 19-21st for an Elasticsearch workshop, a little on DevOps for Happier Humans and all the news you can use at the Elasticsearch table. Swing by to say hello to Leslie Hawthorn and Kurt Hurtado! Lee Hinman will also join us for the workshop. Yay Lee!
- GOTO Amsterdam: Our CTO, Shay Banon, will treat the crowd to a deep dive on Elasticsearch's APIs, plus we'll have lots of knowledge and swag on offer at our booth. See you there!
- Martijn Laarman will be speaking on from text to full-text search at the NDC Oslo 2014 conference. The show runs from June 2-6th.
- Mark Harwood will cover Big Data on Elasticsearch at the ComperioFrokost on June 11th.
The Code Lounge user group in Auckland will get together on June 7th to talk Getting Control of Your Logs. Lots of Logstash love on offer.
Alexander Reelsen and Britta Weber will be speaking at the Zurich Elasticsearch Meetup on June 7th. Alex will discuss What's new in Elasticsearch and Britta will cover the Significant Terms Aggregation. Doors open at 7 PM.
- David Pilato will take the stage at DevoxxUK to discuss Advanced Search for Your Legacy Application DevoxxUK takes place June 12-13th in London, and David will speak on June 12th at 4 PM in Room 1.
- Elasticsearch will be out in force at London Technology Week. Join us June 17-19th to celebrate all that is fashionable in the UK tech scene, especially at the Networking Bar. Even cooler, on June 17th, Mark Harwood will be telling you all about how to Make sense of your (big) data and analyze in real-time like you have never done before! Mark's talk is at 4 PM.
- There's still space for folks to attend tonight's Elasticsearch New York City Meetup! Doors open at 6:30 PM. Sports195 will present their use case, followed by Q&A with Elasticsearchers Uri Boness and Binh Ly. Many thanks to ThoughtWorks for hosting us!
- And another great meetup on tonight in Austin! The Austin Linux meetup group will be hosting an introduction to various technologies, including Logstash, Docker, OpenStack, and more. Doors open at 7:00 PM on Wednesday, and this meetup promises to be a great teaser for all the great content on offer at the upcoming Texas Linux Fest. (Psst: you can get an Introduction to Logstash from Aaron Mildenstein, Logstash Core Developer, at TLF!)
- The San Francisco Elasticsearch Meetup will focus on all things logging and DevOps goodness. Join Kurt Hurtado and Gaurav Gupta to learn all about how to get more love from your logs.
- The Philadelphia Elasticsearch Meetup group will be getting together for their second meetup on June 26th. Details are still in the works, but sign up now so you know where to go for ELK treats on the 26th.
- The SYN/(st)ACK NYC User Group will convene on June 30th at 6:30 PM. Squarespace is hosting the festivities, and Franklin Angulo, one of their Software Engineers, will be presenting. Franklin will cover building a scalable monitoring and alerting system using Graphite, Grafana, Collectd, Nagios, Logstash, Elasticsearch, and Kibana, among others.
Where to Find You
Our Community Manager, Leslie Hawthorn, is hard at work to help folks create more Elasticsearch meetup groups and to help meetup organizers find more speakers. If you are interested in either effort, take a moment to let her know.
Oh yeah, we're also hiring. If you'd like us to find you for employment purposes, just drop us a note. We care more about your skill set and passion for Elasticearch, Kibana, and Logstash than where you rest your head.
Attending #elasticsearch training, and learning a lot of new things from the experts. Great product and training. http://t.co/2ggBDq4G8S
— Trent Swanson (@trentmswanson) June 3, 2014
Thanks, Trent! Glad it was useful for you. :)
If you are interested in Elasticsearch training we have courses taught by our core developers coming up in:
- Amsterdam - June 27, 2014 (ELK workshop)
- Paris - July 08, 2014 (core Elasticsearch training)
- Cologne - July 09, 2014 (core Elasticsearch training)
- Seoul - July 10, 2014 (core Elasticsearch training)
- London - July 14, 2014 (core Elasticsearch training)
- Tokyo - July 14, 2014 (core Elasticsearch training)
- Berlin - July 15, 2014 (ELK workshop)
- Milan - July 16, 2014 (core Elasticsearch training)