Fog Creek

The Challenge

Implement real-time search that could scale to terabytes of information

The Solution

Use Elasticsearch to provide instant search for 100,000 sourcecode repositories containing tens of billions of lines of code

Case Study Highlights

Create an amazing user experience

  • Provide instant search, enabling users to search as fast as they can type
  • Make search a key feature of your user experience
  • Provide real-time results to 50,000 users

Develop and scale easily

  • Scale to 3 terabytes of data and 300 million requests per month
  • Easily index across 100,000 source code repositories containing tens of billions of lines of code
  • Provision new nodes automatically without any downtime by expanding the cluster, then removing old servers

Intelligent Search Key to Enhancing Customer Experience

Fog Creek Software’s mission is "to help the world’s best developers create better software.” The company’s Kiln product was built from the ground up to support and simplify development workflow, helping more than 50,000 developers manage their development process.

Since its launch almost four years ago, the Kiln product has more than 50,000 users searching 100,000 sourcecode repositories. The team realized that the combination of SQL server and OpenGrok search clearly could no longer handle this level of data. Searches were taking 30 seconds, rendering Kiln unsearchable.

The Kiln team started looking for a better search solution. They evaluated Apache Lucene, but found that its search times were significantly slower than Elasticsearch. "We knew that Lucene wasn’t going to scale to search millions of change sets and tens of billions of lines of sourcecode,” explained Kevin Gessner, Kiln Team Lead at Fog Creek Software.

Featured Video

See how Fog Creek Software made their search function 1000x faster with Elasticsearch.

"We asked ourselves – how should we do search in today's world? How can we use search to make Kiln a better product? That's when we chose Elasticsearch."

Kevin Gessner, Kiln Team Lead

Elasticsearch Provides Instant Search

Then, Kevin's team put Elasticsearch to the test by turning all of their workstations into an Elasticsearch cluster. They had six machines searching 100 GB of data on Elasticsearch when they ran out of RAM. It was clear to them that Elasticsearch was going to perform well in production, so they began their migration.

The first step was to stop using SQL Server's full text search for their millions of commit messages. “Queries that were taking us 10 seconds before were taking 10 milliseconds with Elasticsearch. We were blown away," says Gessner.

The second step of the migration was to replace OpenGrok with Elasticsearch for code search. "Elasticsearch is holding up amazingly. We're still getting single millisecond searches with 330 GB of data and 150 million documents," explains Gessner.

Their previous search technology was so slow it had rendered Kiln's search unusable. With Elasticsearch, search has become a hallmark of the Kiln experience, with a search presented front at center at the top of each page.

"We've been able to create our own instant search for our users. Kiln searches as fast as our users can type ... or as fast as they can think. That's made search a hallmark of the Kiln experience."

Kevin Gessner, Kiln Team Lead

Fog Creek's benefits using Elasticsearch

Quickly prototype and test

Elasticsearch has greatly expedited the implementation and configuration of new features and functionality.

Instant search

Provide real-time search results across 100,000 source code repositories containing tens of billions lines of code.

Scale to terabytes of data

As Kiln users grow the volume of their Kiln data Elasticsearch will automatically spin up new nodes.