Matching Millions of People to the Right Jobs
TheLadders (theladders.com) is the premier online job-matching service for career-driven professionals. The New York City-based company helps millions of job seekers search through hundreds of thousands of jobs to find the right fit for their career goals. Consequently, search has always been a key component of TheLadders service.
Prior to Elasticsearch, the company’s search function was based on Solr. After a decade in business, TheLadders needed to re-architect its website from the ground up to take advantage of the latest technology. At that time, TheLadders decided to switch all search functionality to Elasticsearch due to its ease-of-use, flexibility, visibility, reliability, and faster time to recovery.
At TheLadders, job seekers can be matched to jobs in several different ways, all powered by Elasticsearch. Job seekers can query the database of jobs. Recruiters can also find employees by querying the database of job seekers. In addition, both job seekers and recruiters are served with potential matches through an automated matching system, based on Elasticsearch. Even some of the logistic regression algorithms used to generate the matches run on top of Elasticsearch.
Elasticsearch also provides easy scalability so TheLadders can handle its growing user base numbering in the millions, while expanding its database containing a billion documents.
Surfacing New, Relevant Jobs – Fast
TheLadders developed various complex processes to pinpoint the perfect job for each user. For example, job matches can be calculated using a Hadoop batch process. Once the matches are identified, they are put into Elasticsearch and served through TheLadders web and mobile apps. Job seekers can search on top of the matches, using Elasticsearch, to dig in even deeper. However, the batch process takes time. For this reason, TheLadders runs an alternative process to rapidly match job seekers to jobs when necessary, also using Elasticsearch.
"It is important for the user to find the right job as soon as it becomes available, before other job seekers," Pathirana explains. "When the customer stays with TheLadders, we have time later to run Hadoop-based algorithms that are more involved. But to convince them to stay, we need to run real-time algorithms to deliver immediate value to our job seekers, and for that we use Elasticsearch. We can get matches to job seekers in real time – about 100 milliseconds – simply by running an Elasticsearch query."
Making Development Simple
"Elasticsearch is easy to work with, makes our developers more productive, and provides a great set of functionality," says Pathirana.
Pathirana explains that decisions had to be made early on in Solr, before it was deployed, because it was too difficult to make changes in a production system. Elasticsearch, on the other hand, is much more flexible.
"Anybody can start prototyping with Elasticsearch in a day or two," he adds. "For example, a new developer at TheLadders built an autocomplete suggestion feature in a couple days using Elasticsearch. Similar functionality we had in Solr took an experienced engineer weeks to develop, and it was not as accurate."
Keeping the Site Up and Running
"It is important for Elasticsearch to be reliable, because it is the core tool that serves our product," Pathirana confirms. "With millions of users coming to our site, we must stay up and running at all times. If something goes wrong, we must get back up and running quickly. We need a tool we can trust, that gives us visibility and the ability to move quickly to address issues. Elasticsearch gives us all that."
Firstly, Elasticsearch is more reliable than Solr. Since TheLadders deployed Elasticsearch, they have had minimal critical incidents. With Solr, performance issues often arose and recovery time was much longer.
Secondly, according to Pathirana, Elasticsearch gives TheLadders much more visibility and flexibility to solve problems, especially compared with Solr. If there is an issue in Elasticsearch, they can see which indexes are affected. They can even drill down to see which shards are affected.