Elasticsearch Consensus: The Past, the Present, and the Future
Take a journey through the evolution of consensus within the distributed system we call Elasticsearch. See the overarching strategy over the years and learn how (and why) the Elasticsearch team improved things iteratively while still shipping software.
We'll backtrack to the 0.90 days and explain the challenges we were facing. Then we'll discuss the big 1.4 improvements to ZenDiscovery and choosing our current strategy to be open to our users with the resiliency page. We'll also explain why we chose working on our high throughput data replication first (seq# ftw) and provide a glimpse into the future as well as the current ZenDiscovery rewrite. Learn about the effort that went into this area, get insights into our thought process, and see how we’re continuing to increase user confidence in our future as a primary datastore.
Boaz is a core Elasticsearch developer. When not working on consensus algorithms, cluster state changes, data replication, and sequence numbers, you can find him at the ping pong table, playing office DJ, collaborating with colleagues on Zoom, or, if it's Friday, eating hummus.
Yannick Welsch is an Elasticsearch engineer based in Luxembourg. He works on the distributed bits of Elasticsearch, applying his experience in running large clusters as well as making use of his background in formal specification languages.
Jason Tedor is a Software Engineer for Elastic. Jason is an Elasticsearch core developer with a love for all things distributed. In roles prior to joining Elastic, Jason was a backend engineer using the Hadoop ecosystem to handle one of the largest clinical datasets in the world, and built Monte Carlo simulations to model commercial loan portfolios.