Today is a defining day in the history of our company: We are proud to announce that Jordan Sissel, the creator of Logstash and a good friend, is joining Elasticsearch! This means that Elasticsearch, the company, now provides a fully open source product stack for logging and events management: Logstash for log processing, Elasticsearch as the real time analytics and search engine, and Kibana (created by Rashid Khan) as the visual front end.
Neither Jordan nor Logstash really need an introduction, but I'd like to give you an idea about why this is amazing news for so many Elasticsearch and Logstash users.
Logstash, which just released version 1.2.0, is one of the most popular open source logs and events shipper/processor out there. It consumes logs (eg by tailing log files), processes and enriches the data and stores it in Elasticsearch. This means that your logging data can now be analyzed in real time. Kibana is a visual web front end which allows you to explore and monitor the analytics that matter to you.
The trio of Logstash, Kibana and Elasticsearch is already the most popular open source solution to logs management. The three products work together beautifully, which is not surprising, given that Jordan, Rashid and I have known each other for a long time and have worked closely on ensuring that our products work well together. Now with Jordan joining us full time, we will build a team dedicated to Logstash development as well, and will have more time to work together on developing the many new ideas we have for new features and smoother integration.
Logstash / Kibana support
Ever since the company started offering SLA based support subscriptions, we have received requests to offer commercial support for Logstash. We’re happy to report that, as of today, our support customers will receive Logstash and Kibana support as part of their existing subscriptions – at no additional cost. The same applies to future customers: our company will support Logstash and Kibana in addition to Elasticsearch, as part of our standard support subscriptions without any change to the pricing.
Even though Elasticsearch itself was not designed specifically to be a logging product, the logging use case has contributed heavily to its popularity. There are a few reasons for this:
What is a log?
Logs used to be an (often indecipherable) line of text intended for offline human analysis of what went wrong. Today, a log can be any piece of structured or unstructured data, usually associated with a timestamp, that may come from access logs, application logs, or even tweets, financial transactions, audit events, etc.
Elasticsearch was built from the ground up to handle any type of data and, over the years, the time based data model has proved to be a very good fit for Elasticsearch. Moreover, the ability to slice, dice and aggregate data on the fly, based on any field in the logs has freed users from worrying about how to turn their raw logs into valuable insights.
When Rashid joined our company, Kibana was tied to data generated by Logstash. Since then, Rashid and our team have been heavily at work building Kibana 3. The new version of Kibana today allows users to explore any time based data stored in Elasticsearch including, obviously, our vision of what constitutes a log.
Logstash shares the same vision. Effectively, Logstash is a generic system to process events. It provides a pluggable pipeline to combine different ways of inputting data, enriching it, and outputting the results. The plethora of inputs, filters, and outputs, and the amazing community that have developed around them, makes Logstash a Swiss Army knife suitable for almost any type of data munging.
This same thread runs through all of our products, and our vision for redefining the logging space is not just a happy coincidence. In the same way that we have redefined search with Elasticsearch, we want to redefine the log with Logstash.
Users have many different types of data in their applications or organizations, such as logs, documents and database records. Usually each dataset exists in a separate silo, but users want to understand all of this information within a single context. We see users today pushing all of these datasets into Elasticsearch, and using it to join the dots, often resulting in serendipitous discoveries. This is something that would not have been possible before.
One of my favourite examples is that of a Fortune 100 company that stored all of their documents and their access logs recording views and modifications in Elasticsearch. They wanted to ask questions about how people were using the confidential documents. They used “more like this” queries to find places where people had copied and pasted information across documents, which was against policy, and then used the access logs to find who had accessed those documents. This was only possible by correlating the documents and access logs, putting them into the same context.
Another example is a company which uses Logstash and Elasticsearch not only for all their application logs, but also for all of their application metrics. The ability to tie metrics indicating high CPU usage to a log message of “mmm, we shouldn’t really get here” has proven to be invaluable more than once.
This ability to view information in a single context has lead to Elasticsearch becoming a favorite tool of many DevOps. Thousands of companies, Fortune 100 enterprises and hyper-growth startups alike, from all over the world now run Elasticsearch, Logstash and Kibana to analyze logging data.
Our company roots lie deep in open source soil and we believe that our user community will benefit most from our products if they are open source. Needless to say, Jordan will continue to head up Logstash development, and Elasticsearch, Kibana and Logstash will continue to be available under the Apache 2 open source license. Also the option to use either one of the three products without the others will continue to be possible.
I don’t have a crystal ball to predict the future, but I suspect you don’t need one to predict that today’s news will bring a lot value to whoever uses our products. I’m incredibly excited, as we just made a big step forwards in realizing our vision.