How do you create scalable, real-time search for a professional social network with 14 million members worldwide?
Use Elasticsearch to provide real-time indexing for your members' million updates per day
Case study highlights
Create a world-class user experience
- XING is Europe's largest professional social network with around 14 million members
- Search is a key component of the XING platform, helping members find jobs, colleagues, new assignments, and partners
- Search results must be immediately refreshed after member updates – more than one million per day
Focus on scalability and response time
- Real-time indexing makes any change in the platform immediately searchable
- Leverage the benefits of a distributed system
- Demand resilience against hardware failures
Connecting 14 Million Business People WorldwideWith 14 million members, XING is one of the leading social networks in Europe. Half of XING's users are based in Germany, with the other half located across Europe, Asia and Australia. The purpose of XING is to connect business people. With over 6 million queries per day, search is key functionality for the XING website and mobile applications, whether users are searching for other members, special interest groups, jobs, events or other business related content.
"According to the customer survey that we conduct every quarter, search is the most important feature on our platform."
Handling One Million Updates Per Day in Real Time
“According to the customer survey that we conduct every quarter, search is the most important feature on our platform," recalls Dr. Daniel Olmedilla, Vice President, Data Science at XING. “If we had a very high delay updating the data, that would be a very bad service. In the past, we were getting complaints from customers that would change their profile or post a new job ad, and they could not find the new content. They thought it was a system error rather than a technical limitation."
To make matters worse, search response time was slow, results would be incorrect or incomplete, and sometimes the search would simply time out, unable to provide any results.
“We had problems with real-time indexing" he continues. “Users would post changes to their profile, and it would take in the worst case up to 50 minutes before anyone could search for the new content." “We needed to find something that was faster and offered better search results," says Holger Buerger, Director of Site Operations for XING. “That was the reason we moved to Elasticsearch."
Leverage a Distributed Architecture
“Elasticsearch is a fundamental part of our infrastructure," Buerger confirms. “For XING, Elasticsearch is a crucial part of our platform because search is the most important functionality we offer to our users."
XING employs Elasticsearch as the base platform for any search-related service within the company. When XING customers search for members, jobs, groups, events or any type of content on the XING site, they are using Elasticsearch under the hood. In addition, XING developers utilize Elasticsearch internally to create advanced services and products, such as a new recommendation engine that better tailors content to users' needs.
“Automated and real-time indexing in Elasticsearch allows any change in the platform, any new data, to be available for search immediately," Olmedilla says.
Olmedilla also points out that because search is so important to XING, the need for a distributed, resilient search platform is vital. Elasticsearch's distributed architecture ensures that XING's platform doesn't suffer a major performance hit or go offline when any one machine fails.
“Resilience against failure is important," Olmedilla notes. “Whenever we have a problem with one machine, Elasticsearch allows our site to continue serving customers without any problem. Whenever we upgrade our technology, we can do it without any downtime."
Elasticsearch Empowers XING Developers to Add New Features
“With Elasticsearch, it is very easy to scale a cluster to meet new requirements," says Olmedilla.
XING continues to add more data streams to further augment their customer's experience. For example, the XING development team recently added messages as a new content type. This required a completely new index, but the feature was added quickly and easily. Elasticsearch scales horizontally to accommodate the growth in data.
These improvements are typical at XING, where the developers are continuously looking for new types of content that can be searched. Elasticsearch gives them the flexibility to both explore potential new functionality and to implement it quickly.
Autocompletion is just one example of a new feature that uses Elasticsearch under the hood, and it was easily added by XING's development team. The feature suggests words as the user is typing to help complete the search terms. According to Olmedilla, recommendations suggested to users are developed and computed much faster with Elasticsearch as one of the backend systems involved and the overall quality of the recommendations is greatly improved.
“Elasticsearch supports an agile development process when we add new features," Olmedilla continues. “The Elasticsearch APIs are very easy to use. The functionality and performance are excellent."
Previously, updates on the XING system would take a very long time before they were searchable. Now updates are done automatically and immediately in Elasticsearch.
With Elasticsearch, XING can easily and economically scale to add new users and content.
Reliability and Availability
Elasticsearch stability and resilience ensures the integrity of query responses, and protects the system against both planned and unplanned downtime.
> Elasticsearch delivers a competitive edge to XING by offering their developers a flexible platform that empowers them to envision new user functionality and allows them to quickly implement new features.