Open source is how a project like Elasticsearch goes from a few downloads in 2010 to over 250 million in 2018. It lowers a user’s barrier to entry because the code is free and open, and facilitates exceptionally rapid adoption.
But open source is not just an effective way to distribute software. It’s an effective way to make the best product possible. Each download is an opportunity for the project to improve and evolve. Security gets hardened as users scrutinize the code. Reliability and resiliency become more established with rigorous testing across various architectures and environments. Its extensibility increases as users experiment with new use cases and build integrations, add-ons, plugins, and frameworks that expand the project’s reach.
An open source project like Elasticsearch doesn’t become what it is today by hiding it away until the code is perfect. It became what it is by dropping it into the incubator that is open source and letting it evolve naturally, download by download.
Building software is good, but building a community around your software is better. With open source, a community forms with the first download. That first download leads to the first direct contact between author and user in a forum or IRC, which leads to questions or observations, followed by the first open issue, then the first pull request, and finally the first improvement or enhancement. Together they may have fixed a bug or corrected a typo in the documentation. No matter how big or small, the software is better thanks to collaboration. Now, imagine that process after millions of downloads.
Since everyone works on the same code, our products grow through a sort of natural selection — a BDFL approach, as some might say. Sure, multiple versions of a feature might appear within the community, but only a select group makes the final decision on what gets incorporated. This means we don’t have arbitrary features, only ones that make the products stronger and better.
Now, maintaining such a large community can be as much work as maintaining the code. Over the years, we’ve learned a lot. We’ve learned it is important to create a welcoming community, where everyone feels welcome. We’ve learned we have to really hear everyone (even when there are requests of the polar-opposite variety). And we’re not done learning yet.
In the early days, it was easy to think of Elasticsearch as a really great engine for just enterprise search. But with open source, users can easily take a product’s use in a new direction.
For example, Jordan Sissel and Rashid Khan, two sysadmins separated by several U.S. states, wondered what would happen if you sent log files to Elasticsearch and tried to visualize them. Out popped Logstash, a data ingestion pipeline, and Kibana, a visualization engine, along with the beginnings of a new use case (logging) for what became software stack rather than a solo product.
An ocean away, Monica Sarbu and Tudor Golubenco thought about taking network packet data from edge machines and shipping them to Elasticsearch. So they developed Packetbeat to do just that. This later resulted in the creation of an entire family of lightweight shippers, called Beats, for many types of data.
And the pattern continued with community-created and inspired features, extensions, plugins, add-ons, and use cases. From machine learning to natural language processing, application performance metrics and color categorization, security events and blog KPIs.
We haven’t had to imagine what would happen if a thousand people innovated on top of one codebase — we’ve seen it.
Building an open source business can have its challenges, and we’ve learned a lot from observing others.
Some open source companies base their bottom line on a support-only business model. We believe this approach puts what’s best for the company and what’s best for the user in direct conflict with one another, where one can only succeed if the other struggles. This approach lacks the incentive to make the product easy to use or empower customers to be successful since the company’s revenue is based upon customers needing regular support.
Some open source companies fork a commercial (or “enterprise") offering from an original open source project. We believe this fractures the project code and community. It also undermines community trust, limits product testing, and dilutes product quality, ultimately competing with the business efficiency gained from open source software in the first place.
So we’ve built our business differently. Ours aims to strike a healthy balance between open source and commercial code in a single, open software stack, in addition to support and services. It is up to us to deliver users with enough value (and therefore reason) across all our offerings to invest in us. As a result, we can engineer products that are easy to use and reliable, empower our users to be skilled and knowledgeable, and still be a successful company.
In the way that our code evolves organically, our adoption within businesses also grows organically. It usually starts with a "developer zero" who tries our products and has a good experience. Their use then spreads to other colleagues, quickly transforming a small proof-of-concept into a large-scale deployment powering mission-critical systems.
Eventually, a decision-maker or executive takes notice and makes a call on whether or not to formally invest in Elastic. They want to buy into tools that deliver value, help the business run more efficiently, and also happen to be a technology developers across the organization want to use.
Value is the common denominator. While open source is what causes the initial adoption, it is value that drives investment. This is how Elastic can go from a lone application search project and expand to address a logging use case, or a threat hunting project, or an application performance monitoring project, and beyond.
We believe in open source, and our investment in it will continue unchanged. Many businesses become more closed as they grow. Not us. We’ve made a clear choice to be more open and keep our business incentives aligned with our open source community.
That’s why we opened up the code to our proprietary Elastic Stack features (previously bundled under X-Pack). Being this open speeds up development by eliminating excess overhead and complexity, and increases engagement across the entire community.
The open source path isn’t always easy. But we firmly believe that it’s the best way to develop software and ensure ongoing success for our products, company, and community.