Searching through 4 Billion Terms
Quizlet has quickly grown to be one of the world's most popular online studying tool and, with more than 20 million unique visitors per month, they've also landed a place in the list of the top 100 US websites. Quizlet's users have already generated millions of online "study sets" – content in flashcard form that can be studied through various modes - covering more than 4 billion terms. The ability for users to search through these sets to find the right information is critical.
"Prior to Elasticsearch, we were using Sphinx for search," recounts Robert Deaton, Software Engineer at Quizlet. "We started to hit a wall in a few ways. First, we could not keep Sphinx live with the data, which is a big deal when a teacher creates a set that students need to find immediately."
When a user created a set, it was only available after Sphinx completed a full re-index, a process that could take up to 6 hours.
"The performance degraded to the point where it was not acceptable to us any longer," Deaton recalls. "Average search time per page was more than a second, and often 2 to 4 seconds, and that was just far too slow for what we wanted."
Finding and Creating Great Content
Elasticsearch replaced Sphinx as the main site search on the Quizlet site, and is also used to provide functionality – such as language detection and auto completion – to help users create new sets.
"Helping students and teachers get more use out of Quizlet requires that our users are able to find great user-generated content on our site quickly and efficiently," explains Thompson Paine, General Manager of Quizlet. "Elasticsearch is integral to this part of Quizlet."
"In addition, users need to be able to easily create high-quality content," he continues. "Elasticsearch is becoming integral to this aspect of Quizlet by enabling some exciting new features for creating content. I think this will be a key driver for our growth, and a key driver for educating kids in the US."
For example, when a user creates a set, Quizlet will be able to automatically recommend the next few terms, a whole new set of terms, or related existing sets, based on the user's input. For language sets, such as a French-to-English set, Quizlet will be able to automatically finish the French word and provide the English definition. This feature will be powered entirely by Elasticsearch's completion suggester.
"All this valuable functionality is powered by Elasticsearch," Paine notes. "For a site based on user-generated content, this functionality is so crucial for our growth. We are really relying on Elasticsearch."
"Elasticsearch is empowering education across the country with Quizlet," he adds. "We are the go-to utility for studying almost any subject in the US, and our service is made a lot better by Elasticsearch.
Elasticsearch is able to help us better support students and better impact learning across the country, and soon across the world. That is really exciting. Together with Elasticsearch, we are making a real impact in the world."
Reducing Page Load Times from Seconds to Milliseconds
"Speed is especially important when you are dealing with high schools from across the US that have substandard Internet access to begin with," says Paine.
Quizlet's baseline goal is to keep page load time under 100 milliseconds, with search pages under 500 milliseconds. Elasticsearch enables Quizlet to maintain these service levels, noticeably faster than the site's previous search tool.
"When we deployed Elasticsearch, we saw page load times significantly reduced across the site, and the load on our database dropped 30% ," says Deaton. "Now, changes to our database are updated immediately in Elasticsearch."
"The Elasticsearch cluster handled everything well and we did not have any problems scaling this past semester, which was amazing since traffic increased during December exams alone by 40%," Deaton confirms. "This was something that we were able to set aside and not worry about, when everything else needed attention."
Keeping Services Up and Running for 20 Million Users per Month
Quizlet also uses Elasticsearch, Logstash and Kibana internally for maintaining system performance, monitoring web servers, SQL servers, PHP and other services.
"Performance and uptime is a big deal for us," Paine explains. "Being able to constantly maintain a high level of performance on the site is crucial to providing our service to students and teachers."
"Elasticsearch lets us look back at any relevant events, to pinpoint when and why any performance problems started happening," says Deaton. "Elasticsearch has been an invaluable tool in watching the performance of the site, and diagnosing problems."
"Prior to using Logstash and Kibana, we didn't have any kind of centralized logging solution," Deaton adds. "So if we wanted to look something up on a log viewer, we had to write custom scripts or grep the logs manually. Now anyone can go search the logs. It is definitely helping our development team and made it easier for more people to be involved."