There are so many great examples of community members using the powers of Elasticsearch, Logstash, and Kibana (the ELK stack) to get meaningful insights from data. From companies:
- Providing their users with search as navigation with Elasticsearch
- Logging analytics with the addition of Logstash and Kibana
- Delivering dynamic advertising on mobile devices
- Taking Hadoop that last mile
The list goes on and on. At Elasticsearch, we're continually inspired and amazed at the creative ways companies are using our software stack to extract meaningful value from their data. It should come as no surprise that we also work hard on finding insights in many publicly available data sets in order to put our products to the test. Unfortunately, as of yet, we haven’t had a great collaborative way to share this work with the community and for you to share with each other. That changes now!
Start Your Engines
We are delighted to announce a new repository in GitHub under elasticsearch/demo where we will begin sharing easy-to-use demos of the ELK stack for everyone to enjoy, enhance, and contribute back with the goal of greater education. The first demo will launch in less than 48 hours, so get your git clone commands ready!
These demos will include all Logstash configs, plus Kibana dashboards and custom settings in Elasticsearch to make them a great, real-world starting point for those of you looking to kickstart an internal project. We aren’t going to limit this repo to just us, though. If you have come across a great public dataset and want to contribute a demo to the repo, please do. You can also post an issue in the repo with a link and description of a dataset you would like to see turned into a demo eventually. While we probably won’t be able to get to all of them, we promise to do our best!
Under the Hood
With the goal of making your demo-playing experience as pain-free as possible, we have come up with a simple process and structure. We will adhere to this structure for all the demos we create and ask that you do the same. That said, if you have suggestions for improvement, we're ready and eager to listen via email at firstname.lastname@example.org or Twitter.
First, we wanted the 'getting started process' to be as simple and quick as possible. Second, we wanted the demos to run in a self-contained environment. With this in mind, we’ve decided to combine Vagrant, Virtual Box, and Puppet, with Snapshots (a feature we introduced in Elasticsearch 1.0) to help you download, install, configure, and run the full ELK stack. Every time we release a demo, we will accompany it with a blog post with instructions and details about the insights we gained from it. The instructions will also be part of the
NOTE: If you'd rather not setup a VM and understand how to restore a snapshot using our REST API, by all means, clone the repo and just restore the snapshot. In that case, there's no need for Vagrant or Virtual Box. If you decide to go this route, please do it in a fresh install of Elasticsearch rather than an existing one to keep things simple. We don't want to overwrite anything you've already been working on.
As of now, we’ve tested this on a lot of Mac/Linux laptops and a bit on Windows, so if you have issues with your particular OS, please let us know and we will get things fixed or, if need be, make recommendations on alternatives. For Windows in particular, the first demo we publish will have specific instructions so keep an eye out for those.
Feel Free to Take the wheel
We hope you enjoy this new project and please don't hesitate to contribute your own hard work for everyone else to benefit from and build upon.