Improving User Intelligence with the ELK Stack at SCA
Understanding how your users interact with your search solution is key in knowing how to improve it. One way of going about this is to have your search solution record statistics on the user activity. The more the better.
Martin Johansson is a Senior Consultant at Findwise AB, Sweden. His interest lies in every spectra of the back-end of systems, from the interaction of modules and classes to how code is compiled. Every aspect is as important as the next to create a complete and effective solution. When he first came in contact with ELK one year ago he just knew that he, as Tech Lead, needed it on the SCA project. Today, the members of the SCA project find new usage areas with the platform on a regular basis. This is the reason that he has kind of become an ELK ambassador and show it to numerous clients every week.
SCA is a leading global hygiene and forest products company, employing around 44,000 people worldwide. The Group (all companies within SCA) develops and produces sustainable personal care, tissue and forest products. Sales are conducted in about 100 countries under many strong brands. Each brand each has its own website and its own search.
At SCA we use Elasticsearch, Logstash, and Kibana to record searches, clicks on result documents and user feedback, on both the intranet and external sites. We also collect qualitative metrics by asking our public users a question after showing search results: “Did you find what you were looking for?” The user has the option to give a thumbs up or down and also write a comment.
What is logged?
All search parameters and results information is recorded for each search event: the query string, paging, sorting, facets, the number of hits, search response time, the date and time of the search, etc. Clicking a result document also records a multitude of information: the position of the document in the result list, the time it took from search to click and various document metadata (such as URL, source, format, last modified, author, and more). A click event also gets connected with the search event that generated it. This is also the case for feedback events.
Each event is written to a log file that is being monitored by Logstash, which then creates a document from each event and pushes them to Elasticsearch where the data is visualized in Kibana.
Due to the extent of information that is indexed, we can answer questions from the very simple, such as “What are the ten most frequent queries during the past week?” and “Users who click on document X, what do they search for?” to the more complex like “What is the distribution of clicked documents’ last modified dates, coming from source S, on Wednesdays? The possibilities are almost endless!
The answers to these questions allow us to tune the search to meet the needs of the users to an even greater extent and deliver even greater value. Today, we use this analysis for everything from adjusting the relevance model, to adding new facets or removing old ones, or changing the layout of the search and result pages.
Experienced value – more than “just” logs
Recording search and click events are common practice, but at SCA we have extended this to include user feedback, as mentioned above. This increases the value of the statistics even more. It allows an administrator to follow up on negative feedback in detail, e.g. by recreating the scenario. It also enables implicitly evaluated trial periods for change requests. If a statistically significant increase in the share of positive feedbacks is observed, then that change made it easier for users to find what they were looking for. We can also find the answer to new questions, such as “What’s the feedback from the users who experience zero hits?” and “Are users more likely to find what they are looking for if they use facets?"
And server monitoring as well!
This setup is not only used to record information about user behavior, we also monitor the health of our servers. In that context Elasticsearch, Logstash and Kibana are being used as a "Time Series Database". Every few seconds we index information about each server’s CPU, memory and disk usage (time series data). The most obvious gain is the historic aspect. Not only can we see the resource usage at a specific point in time, we can also see trends that would not be noticeable if we only had access to data from right now. This can of course be correlated with the user statistics, e.g. if a rise in CPU usage can be correlated to an increase in query volume.
Benefits of the ELK Stack
What this means for SCA is that they get a search that is ever improving. We, the developers and administrators of the search system, are no longer in the dark regarding what changes actually change things for the better. The direct feedback loop between the users and administrators of the system creates a sense of community, especially when users see that their grievances are being tended to. Users find what they are looking for to a greater and greater extent, saving them time and frustration.
We rely heavily on Elasticsearch, Logstash and Kibana as the core of our search capability, and for the insight to continually improve. We're excited to see what the 2.0 versions bring. The challenge is to know what information you are after and create a model that will meet those needs. Getting the ELK platform up and running at SCA was the part of the project that took the least amount of our time, once the logs started streaming out of our systems.