the future is elastic

When I first discovered Elasticsearch, in late 2010, I immediately considered it an elaborate hoax.

What else can you think when you discover a software project which comes sprinkled with so many buzzwords: a RESTful HTTP interface, working with schema-free documents, talking in JSON through a rich DSL, horizontally scalable and distributed by design, cloud-ready and cloud-friendly? And that software project is a search engine? A hoax, clearly.

But then I downloaded the source code, had a look a it... and it seemed quite legit. I became suspicious - it seemed like a little bit too much work for a prank. So I installed said project, and, to my astonishment, it did seem to work the way it was advertised on the tin.

At that time, I was working as a lead developer for a social media monitoring application, which was using CouchDB and CouchDB-Lucene as core technologies, so, as you can easily imagine, my neural paths had just exploded like fireworks. Working with schema-free JSON-based documents over HTTP and using full text queries and facets was something I was used to, only this time, it looked way, way better.

Being a Ruby shop, we naturally needed a Ruby library for talking to Elasticsearch, and not being satisfied with the offering, I set out to write a Ruby gem for Elasticsearch. In spite of working hard to make it usable outside of our pretty specific use case, I never imagined it will get nearly nine hundreds of GitHub watchers and be featured on Railscasts in the process. Or that I will get to solve more then three hundred issues and pull requests, curating other people's code and making sure the library does not crumble under the weight of different styles, approaches and opinions.

In no time, I was writing articles for the Elasticsearch site about creating data visualizations from search facets, writing self-hosted applications in JavaScript, installing Elasticsearch with Chef, and in the process got to write a Chef cookbook, Elasticsearch data adapter for Ember.js, a monitoring tool to check for health problems of the cluster. I presented about Elasticsearch at three conferences. In short, my life became fully immersed in this crazy project. During all that time, Shay was an immensely helpful source of information, advice, and inspiration for cranking out features with incredible speed.

Today marks a special day on this journey. As a member of the Elasticsearch.com team, I will be able to put more of my time and energy into Elasticsearch and the surrounding ecosystem — the Ruby library, provisioning and monitoring tools, research and documentation. My specific goal is to make it even more easy for Rubyists to work with full text search in their applications, to understand it better, and to make most of it.

After all, search is the primary means for keeping up with the vast amounts of data; every day, from Google to Mac OS X Spotlight, we rely on search to find information. In fact, the future of our civilization depends on how well we understand the data about ourselves, be it the global Twitter chatter, air pollution metrics, DNA sequences or evidence gathered by the Curiosity rover on Mars. I believe that full text search, and Elasticsearch in particular, will be an important part of this future.