How do you connect millions of job seekers with hundreds of thousands of jobs?
By using Elasticsearch to make fast and accurate matches
Case study highlights
- Serve millions of job seekers
- Handle 1 billion documents
- Post hundreds of thousands of jobs in milliseconds
- Ensure website and app reliability
- Solve problems quickly
- Reduce feature development time from weeks to days
Matching millions of people to the right jobs
TheLadders (theladders.com) is the premier online job-matching service for career-driven professionals. The New York City-based company helps millions of job seekers search through hundreds of thousands of jobs to find the right fit for their career goals. Consequently, search has always been a key component of TheLadders service.
Prior to Elasticsearch, the company’s search function was based on Solr. After a decade in business, TheLadders needed to re-architect its website from the ground up to take advantage of the latest technology. At that time, TheLadders decided to switch all search functionality to Elasticsearch due to its ease-of-use, flexibility, visibility, reliability, and faster time to recovery.
"As a job site, we enable job seekers to find the right career opportunities," says Peter Pathirana, Lead Engineer, Recommendations Infrastructure, Platform Team at TheLadders. "Elasticsearch puts the ‘search’ in our job search. It powers TheLadders user experience."
At TheLadders, job seekers can be matched to jobs in several different ways, all powered by Elasticsearch. Job seekers can query the database of jobs. Recruiters can also find employees by querying the database of job seekers. In addition, both job seekers and recruiters are served with potential matches through an automated matching system, based on Elasticsearch. Even some of the logistic regression algorithms used to generate the matches run on top of Elasticsearch.
Elasticsearch also provides easy scalability so TheLadders can handle a growing user base of millions, and an expanding database of one billion documents.
Surfacing new, relevant jobs – fast
TheLadders developed a couple different complex processes to pinpoint the perfect job for each user. For example, job matches can be calculated using a Hadoop batch process. Once the matches are identified, they are put into Elasticsearch and served through TheLadders web and mobile apps. Job seekers can search on top of the matches, using Elasticsearch to dig in even deeper. However, the batch process takes time. For this reason, TheLadders runs an alternative process to rapidly match job seekers to jobs when necessary, also using Elasticsearch.
"It is important for the user to find the right job as soon as it becomes available, before other job seekers," Pathirana explains. "When the customer stays with TheLadders, we have time later to run Hadoop-based algorithms that are more involved. But to convince them to stay, we need to run real-time algorithms to deliver immediate value to our job seekers, and for that we use Elasticsearch. We can get matches to job seekers in real time – about 100 milliseconds – simply by running an Elasticsearch query."
Making development simple
"Elasticsearch is easy to work with, makes our developers more productive, and provides a great set of functionality," says Pathirana.
"We have new junior developers who were able to ramp up and start working with Elasticsearch quickly, prototyping new ideas," he continues. "Configuration can be managed through the Elasticsearch API, which is simple and easy to use in any language. We have developers using Scala, Java, Python, and nobody had any problems working with Elasticsearch in the language of their own choice – which is quite different compared to Solr."
Pathirana explains that decisions had to be made early on in Solr, before it was deployed, because it was too difficult to make changes in a production system. Elasticsearch, on the other hand, is much more flexible.
"Anybody can start prototyping with Elasticsearch in a day or two," he adds. "For example, a new developer at TheLadders built an autocomplete suggestion feature in a couple days using Elasticsearch. Similar functionality we had in Solr took an experienced engineer weeks to develop, and it was not as accurate."
Keeping the site up and running
"It is important for Elasticsearch to be reliable, because it is the core tool that serves our product," Pathirana confirms. "With millions of users coming to our site, we must stay up and running at all times. If something goes wrong, we must get back up and running quickly. We need a tool we can trust, that gives us visibility and the ability to move quickly to address issues. Elasticsearch gives us all that."
First, Elasticsearch is more reliable than Solr. Since TheLadders deployed Elasticsearch, they had minimal critical incidents. With Solr, performance issues arose often, and time to recover was much longer.
Second, Elasticsearch gives TheLadders much more visibility and flexibility to solve problems, especially compared with Solr, according to Pathirana. If there is an issue in Elasticsearch, they can see which indexes are affected. They can even drill down to see which shards are affected.
"With Solr, it was harder to track down where the actual problem was," explains Pathirana. "We wasted a lot of time trying to figure out what was wrong. That slowed us down in responding to and recovering from the situation. With Elasticsearch, we have automated most of our monitoring. We know what is happening and what went wrong – all we have to do is take action. Being able to solve problems quickly and easily in Elasticsearch by using the REST API is amazing."
Elasticsearch enables TheLadders to match job seekers with jobs much more quickly, a key competitive advantage in the industry.
TheLadders easily handles millions of users and one billion documents in Elasticsearch, and those numbers continue to grow.
Improved system performance
Elasticsearch delivers high reliability, and greater visibility and flexibility to solve issues as they arise.
Ease of use
Elasticsearch empowers new developers to ramp up quickly and innovate new features in days rather than weeks.