Our story

From the early days of Elasticsearch to how the ELK Stack came to be, a period of awesome (but chaotic) development, the introduction of the Elastic Stack, and a new era of openness — there’s a lot of goodness to unpack in our narrative.

It started with a recipe app

In a London apartment, Shay Banon was looking for a job while his wife attended cooking school at Le Cordon Bleu. In his spare time, he started building a search engine for her growing list of recipes.

His first iteration was called Compass. The second was Elasticsearch (with Apache Lucene under the hood). He open sourced Elasticsearch, created the #elasticsearch IRC channel, and waited for users to appear.

The response was impressive. Users took to it naturally and easily. Adoption went through the roof, a community started to form, and people noticed — namely Steven Schuurman, Uri Boness, and Simon Willnauer. Together, they founded a search company.

You know, for Search Inc.

Around the time Elasticsearch Inc. was founded two other open source projects were taking flight.

Jordan Sissel was working on Logstash, an open source, pluggable ingestion tool for sending logs to the "stash" of the user’s choosing, one of which was Elasticsearch. He was also developing a UI on top of it to visualize log data — and it was shaky at best.

Fortunately, someone else was tinkering with the visualization challenge. Enter Rashid Khan, who was working on an open source UI called Kibana.

Shay, Jordan, and Rashid had known each other and their projects for some time and decided to team up, resulting in the ELK Stack — Elasticsearch, Logstash, and Kibana Stack.

A while later, we released two commercial plugins: Marvel for monitoring and Shield for security.

Heya, Elastic. Welcome, Found.

At Elastic{ON} 2015 in San Francisco, we made two big announcements. The first: we rebranded the company to be called Elastic. The new name better represented our growing product ecosystem and suite of use cases. The second: we joined forces with Found, a company delivering hosted and managed Elasticsearch on AWS. By teaming up, we could provide the simplest, most complete offering on the market.

Emerging from our primordial ooze

In the early days, building and releasing software at Elastic took an every-engineer-for-themselves approach: ship whatever version you want, whenever you want — just make it awesome. Kibana had betas, Logstash had milestones, Elasticsearch had numbers. Plugins happened as they pleased. It was chaotic, but it worked...until it didn't.

As users were doing more with the product, we needed to build a product that did more for the users. We added more capability, submitted more pull requests, built new plugins and extensions. The awesomeness increased, complexity emerged, and things got messy for our technology stack.

For instance, if you were running version 1.7 of Elasticsearch and version 2.3 of some plugin, there wasn’t an automatic way to know if they were compatible or if the plugin was silently failing. This was a bug.

We also started to hear ourselves say things like, “If you want to use Shield, you need Elasticsearch 1.4.2….unless you’re using Watcher. In that case, you’ll need Elasticsearch 1.5.2. And if you’re using Elasticsearch 1.5.2, that’s only compatible with Kibana 4.0.x, Logstash 1.4.x, Shield 1.2.x, and Watcher 1.0.x.”

We had arrived at a special sort of versioning hell — and the support matrix didn’t look any better. It was time for a change.

Pausing to take a beat(s)

While the product teams wrestled with version numbers, another product story was unfolding. In 2015, we welcomed Packetbeat, a Berlin-based, husband-and-wife team engineering a lightweight way to send network data to Elasticsearch, to the Elastic family.

That got us thinking: what if we had a family of single-purpose, lightweight data shippers to send network data, logs, metrics, audit data, and more from edge machines to Logstash and Elasticsearch? And so Beats was born.

The bonanza begins

October 2015 marked a turning point for addressing our product versioning and compatibility complexities.

Dubbed the “release bonanza,” it was the first time all of our products — Elasticsearch 2.0, Logstash 2.0, Watcher 2.0, Shield 2.0, and Kibana 4.2 — shipped together on the same day. (Beats 1.0 had another month to bake.)

Coordinating this effort wasn’t easy. The engineering teams had to change the way they worked together to build and test the products. It was worthwhile, though. This shift made it easier for users to get started with our products and made our products more reliable to do amazing things with.

Enter, Elastic Cloud

A few months later, the release bonanza would no longer only be a downloadable experience. Elasticsearch and Kibana were now available as a service on AWS through Elastic Cloud, the offering formerly known as Found.

BELK 5.0 Elastic Stack 5.0

Aligning release cadence with Elasticsearch 2.0 was the first step toward a more mature product offering. The 5.0 launch was the second step. It introduced a more integrated, better tested, and easier getting started experience than ever before.

The 5.0 release also bundled all of our commercial plugins (which we called Shield, Marvel, and Watcher at the time) into a single extension called X-Pack. It consisted of features like security, monitoring, and alerting for our core products, and grew to include machine learning when we brought a London-based company called Prelert into the Elastic family.

So modules, much simplified

In version 5.3 (released in March 2017), Filebeat formally introduced the concept of "modules," or a set of safe configurations to ship, parse, store, analyze, and visualize common log formats (e.g., Apache, Nginx, MySQL, etc.) in the Elastic Stack. Modules simplified the getting started experience of going from dataset to dashboard.

Metricbeat and Packetbeat had their own flavors of modules, and months later, Logstash would introduce modules of its own for ArcSight and NetFlow data.

A new frontier: introducing ECE

From the beginning, we had a vision for simplifying how users deploy Elastic in their organizations. We took the technology we use to manage our own Elastic Cloud service and released Elastic Cloud Enterprise (or ECE) to allow businesses big and small to download all the goodness of our hosted offering and run it themselves. ECE made managing one cluster or thousands seamless, streamlining the management and orchestration of Elastic products and solutions in any environment.

Elastic solutions precipitate

As modules started to multiply, getting started with the Elastic Stack to address a particular use case like logging or metrics got easier and easier. And momentum continued to build when we joined forces with Opbeat, a Copenhagen-based application performance monitoring (APM) company, and Swiftype, a San Francisco-based site and enterprise search company, a few months later. Both companies became part of Elastic.

By this time, our company had matured to a place where we could offer streamlined ways for solving common problems leading us to formally introduce our solutions. While our solutions range from a DIY experience to something more turnkey, each has real product behind it and can be deployed in just a few minutes.

Opening our X-Pack code

From open source to open communication, being open is at the heart of all we do. This is why we made the decision to open the code to our commercial X-Pack features in order to speed up development time and increase community engagement, allowing everyone to contribute to, comment on, and inspect our code.

As a result, getting started with the Elastic Stack became even easier, with all of the X-Pack features now shipping with the default distributions of Elasticsearch, Kibana, Beats, and Logstash. This change didn’t take away any Apache 2.0 code. Instead, we doubled down on being open.

Ringing ze bell

At exactly 9:30 a.m. Eastern on October 5, the bell at the New York Stock Exchange rang out, officially making Elastic a public company. With a record-setting 230 Elasticians on the trading floor and hundreds more around the world, our distributed company celebrated reaching this remarkable milestone together. While it is just one day in our long journey, it was a pretty grand one.

There’s more to our story. Stay tuned for updates as our adventure continues to unfold.