Finding a home (and career) in the open source community
Open source software development can have a reputation for abrasive behavior. The search community is a clear counterexample for me, with a culture that emphasizes respect and acceptance. This culture played an important part in my own path to open source development.
A little over six years ago, I was a wide-eyed software engineer settling into my first full-time job. My company made data analysis software for governments and large organizations, and I had just joined the team that built and maintained the search infrastructure.
Until then, when I thought of ‘search’, what came to mind was a web search engine or online shopping site. It soon became clear that search was a far more general and fundamental problem — our users needed to find the relevant parts of their data, and find them quickly. Search formed a core component of each organization’s critical work, which included anti-fraud investigations, combating human trafficking, tracking epidemics, and more.
Search technology brings together work from often disparate fields: systems and databases, linguistics, and even mathematical fields like statistics and information theory. I found it so engaging to be able to work across these different areas — in a given week I might be designing a data schema, pondering the rules of plural nouns, or debugging why certain queries had high tail latencies. Needless to say, it didn’t take long for me to fall in love with search.
Our search infrastructure was built on the open source search library Apache Lucene, and we were working on writing a new search service using Elasticsearch (which is also based on Lucene under the hood). Using an open source library made it possible to deliver features to our users that would have taken years for our tiny search team to develop ourselves. In addition to being packed full of capabilities, Lucene was so fast and seemed to constantly be getting faster. During my studies, I took a deep interest in data structures and foundational algorithms like prefix trees, skip lists, and sorting techniques. In Lucene I saw these topics come to life in code and play a role in powering hundreds of critical applications.
A few months into my new job, I was scratching my head at why a new search highlighting feature we were building wasn’t working as expected. In debugging the problem, I discovered an issue in Elasticsearch where nested fields weren’t highlighted. I continued to dig deeper and traced the problem to Lucene. I had a sense for how to fix the issue, and it seemed like a good opportunity to contribute to a project that had benefited my team so much.
Although it was a small change, I was quite nervous once it came time to submit the patch. One of the "Mythical Lucene Developers" would be reviewing it, those search experts who built this wonderful package. And the code review would be open for the whole internet to see — in other open source projects I had seen examples of developers being publicly berated for a bad patch. From my experience in university and my first job, I had developed a working picture of an accomplished software engineer: they were ‘prickly’ and forceful with their opinions, and were always self-assured, sometimes to the point of overconfidence. I assumed these talented open source developers would fit this mold.
I closed my eyes and hit ‘submit’, and after a friendly and straightforward review, the patch was merged! Encouraged by this experience, I made some small contributions to Elasticsearch in my rare free time at work. A couple years after my first open source contributions, I started my own open source project and became the one giving friendly and straightforward reviews.
I now work for Elastic as a core developer on Elasticsearch, where I contribute full-time to these projects I once admired from afar. As I’ve spent more time in the open source search community, I’ve seen that these initial helpful reviews are part of a broader pattern. By and large, the developers I interact with treat each other with respect and kindness. Despite their considerable accomplishments, most contributors don’t have ‘big heads’ — the ratio of ego to talent remains surprisingly low. Of course there are arguments and heated discussions, but engineers seem motivated by what they think is technically ‘right’, by what they feel is best for the project.
In previous jobs, design discussions were always centered around a whiteboard. A group of engineers would huddle in a room, sometimes for hours, and debate the details of a new feature or architecture. Although this process was engaging, I preferred having the time to think deeply about the problem on my own, and organize my thoughts before discussing with others. As a stereotypically polite Canadian, I sometimes found it difficult to seize the floor in these lively discussions filled with interruptions.
In open source development, it’s much more common to rely on written, asynchronous communication. I appreciate having the space to carefully ponder a problem, then draft up an email or GitHub post laying out my thoughts. Even the most soft-spoken engineers are able to weigh in and bring their opinions to the discussion.
The open source search community spans many cultures and languages, which contributes to there being a diversity of communication styles. Contributors tend to be quite flexible when someone has a style that doesn’t match their own. I found that I could ‘speak Canadian’ and still be understood, like when I deliver important but gentle feedback in a code review, or apologize in the middle of a bug report.
Lucene turned 20 years old this year, which is a big accomplishment for such a widely-used open source project. In terms of both its technology and its people, the open source search community is one to admire. I am so happy to be part of it, and hope it continues to grow in the years to come.
Interested in joining a company with a Source Code to live by? We’re hiring. Check out our teams and find your fit! Want to read more about life at Elastic? Read more on our blog!