How do you manage, monitor and innovate a video content website that serves 500,000 visitors?
By using the ELK stack to provide powerful search and analytics to enhance and monitor user activity and system performance
Case study highlights
Simple, Scalable Search
- Fast query response time
- Feature rich and easy-to-implement
- Distributed to scale independent of application
Product and System Analytics
- Centralize log data for all developers to access
- Diagnose issues quickly with log infrastructure dashboard
- Track product usage to guide roadmap and development
It started with simple search …
Parleys is a Belgium-based website that calls itself "a Video Presentation and Course Sharing Platform." It hosts videos, mostly presentations from conferences, serving as a platform to combine slides and video in a very engaging manner.
"Search is an important part of our website," explains Carlo Waelens, sales and marketing director for Parleys. "If a visitor is looking for a particular subject, we need to make sure they can find it on Parleys."
"Before Elasticsearch, our search performance was a catastrophe," says Gert Leenders, product manager and developer,Parleys. "Query response time is very important. So we built our current infrastructure on MongoDB and Elasticsearch because both perform well and are very fast.
"Out of the box, Elasticsearch offers the most amazing search options," he continues. "The REST API is easy-to-understand and well-documented, and there is a vibrant community behind Elasticsearch to help us if needed.
"Elasticsearch is a lifesaver," Leenders adds. "It gives users the ability to search all the content on Parleys. They can find whatever they are looking for."
… that needed to scale to over half a million visitors
The Parleys website currently has 400,000+ unique visitors per year. By the end of 2015, Parleys expects the number of visitors to double, at least. New product offerings and contracts in the works could also cause Parleys' traffic to increase very quickly in the near term.
"Scalability is very important," Leenders notes. "If we need to scale up only for search, we can do that with Elasticsearch. We can just bring up a new instance and scale up our search, without having to scale up the entire application."
"If all of a sudden traffic to Parleys spikes and the site load increases significantly, we can scale horizontally as wide as we want by bringing in new instances," he continues. "As soon as I bring up a new Elasticsearch instance, the cluster is aware and will spread the load."
… and evolved into an all-in-one search and analytics stack
Parleys also uses Elasticsearch to power its product and systems analytics as part of the ELK stack, comprised of Elasticsearch, Logstash, and Kibana. Elasticsearch is the data store and query engine at the heart of the stack, while Logstash is the data pipeline and Kibana is the data visualization engine.
Parleys uses ELK to monitor logs from across the site to ensure system availability and performance, as well as track product usage to evaluate current and new features. Log messages are divided into technical and functional categories, each accessible via its own Kibana dashboard.
"I really like Kibana. It is easy to set up, and lets us create multiple dashboards for different needs, enabling total separation between the technical logs and functional logs, which is a very helpful advantage to our development team," says Leenders.
Monitoring site health
The technical dashboard displays logs from every component in the infrastructure, used to diagnose performance issues. Parleys has a large development team, and in the past they did not have adequate visibility into system health. The company has separate client-side and server-side development teams. If a developer from the client-side had to investigate a performance issue on the server-side, they had great difficulty accessing the necessary data. For example, they would need to become an admin on the server in order to access the logs, which would often result in security glitches.
"Before ELK, it was very difficult to check out logs from different devices and servers to investigate performance issues," Leenders recalls. "And some developers do not understand the whole infrastructure, so it became my responsibility to retrieve those messages for them, which took extra time. I always had to look at multiple instances because of the load-balancing infrastructure – and the error is always in the last instance that you look at.
"With Logstash and Elasticsearch in place, we have a central Kibana dashboard," he continues. "It is easy for any developer to simply view and access all the relevant log messages, providing us with significant productivity savings. Previously, if I was available, it would take 15 minutes to find the right logs. If I was not around, the developers had to wait even longer. But now it takes no time. Any developer can go into the Kibana dashboard and see the logs immediately."
"Elasticsearch, Logstash, and Kibana are helping everyone on the team keep our site healthy, support high performance and ensure a good user experience," he adds.
Making product and development decisions based on data
The functional dashboard uses logs to measure page traffic and how particular features are used, helping Parleys make more informed decisions on their product roadmap. For example, if the Parleys team sees that a new feature is not being used, they can eliminate it. If a feature is being used heavily, they can invest more development time on it.
"As developers, we get very excited about new innovations," Leenders explains. "We think everyone will love them and use them. But in real life that is not the case. We could spend three weeks fine-tuning a feature that is never used. With ELK, we can put a new feature in a beta phase and see how users react to it, before we spend more development time. If we see interest,only then will we develop the feature further. This can save Parleys a lot of money, in terms of development expenses."
One activity Parleys tracks in the functional dashboard is payments. For example, users can purchase a pay-per-view video by downloading it, or they can rent a video, which means watching it online during a limited timeframe. Parleys created a graph in Kibana to compare the activity on the two payment types, so they can track which option performs better. Similarly, if Parleys drops prices for content, they can track if this change has an impact on revenue.
"ELK gives us valuable insight to make the right decisions, and take advantage of marketing and sales opportunities," says Leenders. "As a site analytics solution, ELK is much easier to use than options such as Google Analytics, especially for backend developers. In addition, solutions such as Google Analytics only work with the main site, but not with our public API."
The functional dashboard also gives Parleys insight into the number of users and the number of logins, so they can easily see the user activity on with the website. This helps Parleys determine the best time for planned downtime to deploy a new feature, for example.
Simple, scalable search
Within a year, Parleys expects to more than double its annual visitors. Elasticsearch can be easily scaled to handle this dramatic increase in site traffic and the searches it will generate.
Increased development productivity
The ELK stack improves developer productivity at Parleys by simplifying access to all logs, regardless of the developer's department. Developers can find the information they need and solve performance issues much faster with ELK.
Improved site health
By monitoring logs via the ELK stack – Elasticsearch, Logstash and Kibana – Parleys is able to ensure site health, availability and performance. Ultimately ELK drives a better user experience.
Insight into product usage
Parleys uses ELK to monitor product usage so they can focus innovation efforts on features that resonate best with their users.