This Week in Elasticsearch - July 30, 2014

Welcome to This Week in Elasticsearch. In this roundup, we try to inform you about the latest and greatest changes in Elasticsearch. We cover what happened in the GitHub repositories, as well as many Elasticsearch events happening worldwide, and give you a small peek into the future of the project.

EC2! DDoS! Cloud! Elasticsearch!

If you've been reading the news, you've been seeing those words together quite a bit for the past few days. Don't panic! (And bring your towel.) We've got you covered.

tl;dr:

  • Block ports 9200 and 9300 from all machines not part of your development environment
  • Don't run Elasticsearch as root
  • Disable dynamic scripting for non-sandboxed environments, which is disabled by default in our 1.2.x and later branches. If you would like to use dynamic scripting, make sure you set up your environment securely by following these guidelines.
  • Keep current on the latest version of Elasticsearch as we're constantly making improvements to our software, including updating our security defaults. You can download the latest version here, and there are instructions on how to upgrade here.

Elasticsearch core

  • Fielddata: Fix the ordinals impl for sparse fields (#6908, master and 1.4)
  • Fielddata: remove comparators (#5980, master and 1.4)
  • DocValues: Change numeric data types to use SORTED_NUMERIC docvalues type instead of a custom encoding in BINARY (#6967, master and 1.4)
  • Search & Count API: Add option to early terminate doc collection (#6876, master and 1.4)
  • Aggregations: change to default shard_size in terms aggregation (#6857, master and 1.4)
  • Internal: streamline use of IndexClosedException when executing operation on closed indices (#6988, master and 1.4)
  • Aggregations: Make _source parsing in top_hits aggregation and search api consistent (#6997, master, 1.4 and 1.3)
  • Sorting: Speed up string sort with custom missing value (#7005, master and 1.4)
  • Aggregations: Improve JSON output scoping (#7004, master, 1.4 and 1.3)
  • Internal: expose the indices names every action relates to if applicable (#6933, master and 1.4)
  • Filter cache: remove use of recycled set in filters eviction (#7012, master and 1.4)
  • Aggregations: Added pre and post offset to histogram aggregation (#6605, master and 1.4)
  • Flush API: Add wait_if_ongoing option to _flush requests (#6996, master and 1.4)
  • Geo: Fix parse error with complex shapes (#5773, master and 1.4)
  • Core: Use the provided cluster state instead of fetching a new cluster state from cluster service (#7013, master and 1.4)
  • Caching: Add a periodic cleanup thread for IndexFieldCache caches (#7010, master and 1.4)
  • CORS: Support regular expressions for origin to match against (#6923, master and 1.4)
  • Aggregations: Added an option to show the upper bound of the error for the terms aggregation (#6696, master and 1.4)
  • Mapping: Add multi_field support for Mapper externalValue (#6867, master and 1.4)
  • Aggregations: The nested aggregator should also resolve and use the parentFilter of the closest reverse_nested aggregator (#7048, master, 1.4 and 1.3)
  • Aggregations: fixed value count so it can be used in terms order (#7050, master and 1.4)
  • Circuit breaker: Add HierarchyCircuitBreakerService (#6739, master and 1.4)
  • Internal: Support parsing lucene minor version strings (#7055, master, 1.4 and 1.3)
  • Internal: use AtomicInteger instead of volatile int for the current action filter position (#7021, master and 1.4)
  • Rest: fixed filters execution order to be from lowest to highest rather than the other way around (#7019, master and 1.4)
  • Transport Client: fixed the node retry mechanism which could fail without trying all the connected nodes (#6829, master and 1.4)
  • Transport Client: Don't add listed nodes to connected nodes list in sniff mode (#7067, master and 1.4)

Elasticsearch Ecosystem

Here's some more information about what is happening in the ecosystem we are maintaining around the ELK stack - that's Elasticsearch plus Logstash and Kibana - including plugin and driver releases.

Slides & Videos

Learn how Elasticsearch powers the Building Performance Database, helping folks understand their building's energy footprint

All about AOL's Network Forensics tool Moloch, powered by Elasticsearch

Florian Hopf shared his slides on Elasticsearch Use Cases, as presented at Java Forum Stuttgart 2014.

Rotem Hermon, the organizer of the Elasticsearch Tel Aviv meetup, on serendip.me How Clairvoyant uses the ELK stack for log analysis

Where to find Us

We'd love to feature all the great Elasticsearch, Logstash, and Kibana presentations and meetups happening worldwide in this section. If you're speaking or hosting a meetup, let our Community Manager, Leslie Hawthorn, know!

Germany

Israel

The OpsTalk Meetup Group will get together on August 4th for a full day workshop on the ELK stack. Register now to save your place!

Poland

The Warsaw Java Users Group is holding a Microservices Hackathon on August 9th from 10 AM to 10 PM. You can spend those twelve hours hacking on the ELK stack and a variety of other tools. Register now to attend.

We hope to see you on Thursday!

United States

  • If you haven't gotten enough of the Pacific Northwest with all things OSCON, then head on over to CascadiaJS on July 31st - August 1st. Joe Fleming from the Kibana engineering team will be attending, so say hello to him in the hallways!
  • Heading to Chicago for LinuxCon and Cloud Open North America? Say hello to Leslie Hawthorn, and check out her panel Empowering Your Corporate Open Source Software Developers.

Where to Find You

Our Community Manager, Leslie Hawthorn, is hard at work to help folks create more Elasticsearch meetup groups and to help meetup organizers find more speakers. If you are interested in either effort, take a moment to let her know.

Oh yeah, we're also hiring. If you'd like us to find you for employment purposes, just drop us a note.  We care more about your skill set and passion for Elasticsearch, Kibana, and Logstash than where you rest your head.

Training

If you are interested in Elasticsearch training we have courses taught by our core developers coming up in:

  • New York - August 12, 2014 (core Elasticsearch training)
  • Sao Paulo - August 14, 2014 (core Elasticsearch training)
  • Rio de Janeiro - August 18, 2014 (core Elasticsearch training)
  • Amsterdam - August 20, 2014 (core Elasticsearch training)
  • Boston - August 27, 2014 (core Elasticsearch training)
  • New York - August 27, 2014 (ELK workshop)
  • San Francisco - August 28, 2014 (ELK workshop)