- Serve millions of job seekers
- Handle 1 billion documents
- Post hundreds of thousands of jobs in milliseconds
- Ensure website and app reliability
- Solve problems quickly
- Reduce feature development time from weeks to days
TheLadders (theladders.com) is the premier online job-matching service for career-driven professionals. The New York City-based company helps millions of job seekers search through hundreds of thousands of jobs to find the right fit for their career goals. Consequently, search has always been a key component of TheLadders service.
Prior to Elasticsearch, the company’s search function was based on Solr. After a decade in business, TheLadders needed to re-architect its website from the ground up to take advantage of the latest technology. At that time, TheLadders decided to switch all search functionality to Elasticsearch due to its ease-of-use, flexibility, visibility, reliability, and faster time to recovery.
At TheLadders, job seekers can be matched to jobs in several different ways, all powered by Elasticsearch. Job seekers can query the database of jobs. Recruiters can also find employees by querying the database of job seekers. In addition, both job seekers and recruiters are served with potential matches through an automated matching system, based on Elasticsearch. Even some of the logistic regression algorithms used to generate the matches run on top of Elasticsearch.
Elasticsearch also provides easy scalability so TheLadders can handle its growing user base numbering in the millions, while expanding its database containing a billion documents.
TheLadders developed various complex processes to pinpoint the perfect job for each user. For example, job matches can be calculated using a Hadoop batch process. Once the matches are identified, they are put into Elasticsearch and served through TheLadders web and mobile apps. Job seekers can search on top of the matches, using Elasticsearch, to dig in even deeper. However, the batch process takes time. For this reason, TheLadders runs an alternative process to rapidly match job seekers to jobs when necessary, also using Elasticsearch.
"It is important for the user to find the right job as soon as it becomes available, before other job seekers," Pathirana explains. "When the customer stays with TheLadders, we have time later to run Hadoop-based algorithms that are more involved. But to convince them to stay, we need to run real-time algorithms to deliver immediate value to our job seekers, and for that we use Elasticsearch. We can get matches to job seekers in real time – about 100 milliseconds – simply by running an Elasticsearch query."
"Elasticsearch is easy to work with, makes our developers more productive, and provides a great set of functionality," says Pathirana.
We have new junior developers who were able to ramp up and start working with Elasticsearch quickly, prototyping new ideas. Configuration can be managed through the Elasticsearch API, which is simple and easy to use in any language. We have developers using Scala, Java, Python, and nobody had any problems working with Elasticsearch in the language of their own choice – which is quite different compared to Solr.
Pathirana explains that decisions had to be made early on in Solr, before it was deployed, because it was too difficult to make changes in a production system. Elasticsearch, on the other hand, is much more flexible.
"Anybody can start prototyping with Elasticsearch in a day or two," he adds. "For example, a new developer at TheLadders built an autocomplete suggestion feature in a couple days using Elasticsearch. Similar functionality we had in Solr took an experienced engineer weeks to develop, and it was not as accurate."
"It is important for Elasticsearch to be reliable, because it is the core tool that serves our product," Pathirana confirms. "With millions of users coming to our site, we must stay up and running at all times. If something goes wrong, we must get back up and running quickly. We need a tool we can trust, that gives us visibility and the ability to move quickly to address issues. Elasticsearch gives us all that."
Firstly, Elasticsearch is more reliable than Solr. Since TheLadders deployed Elasticsearch, they have had minimal critical incidents. With Solr, performance issues often arose and recovery time was much longer.
Secondly, according to Pathirana, Elasticsearch gives TheLadders much more visibility and flexibility to solve problems, especially compared with Solr. If there is an issue in Elasticsearch, they can see which indexes are affected. They can even drill down to see which shards are affected.
With Solr, it was harder to track down where the actual problem was. We wasted a lot of time trying to figure out what was wrong. That slowed us down in responding to and recovering from the situation. With Elasticsearch, we have automated most of our monitoring. We know what is happening and what went wrong – all we have to do is take action. Being able to solve problems quickly and easily in Elasticsearch by using the REST API is amazing.