Norway's Postal Service Moves to Elastic from FAST

A few years ago, we delivered a simple search solution called Tilbudssøk to one of our customers, Posten, Norway’s postal service. With Tilbudssøk, Posten’s commercial customers could upload their product catalogs and seasonal deal brochures to the Posten site and make their offers searchable to a broader audience.

At the time, Elasticsearch was still in its infancy. We deemed FAST ESP to be the appropriate search engine for the solution as we had a long relationship and a great deal of experience with it.

Fast forward a few years and Tilbudssøk became increasingly limited by what we could coax out of FAST, which was frustrating both for us, as developers, and the customer. For example, we wanted to improve the user experience by including geolocation data for stores such that a user could find the nearest one that stocked the product they wanted, but couldn’t because of FAST’s proprietary nature.

FAST is a closed-source, resource-hungry product that is no longer under development or officially supported. Due to its age, it also requires specific legacy OS versions and libraries, which makes it susceptible to security vulnerabilities. Elasticsearch, meanwhile, is a lightweight, feature rich platform under constant development. It is also built from the ground up to scale, making it especially ideal for us as we run Tilbudssøk on Amazon Web Services (AWS).

Additionally, Posten desired to increase Tilbudssøk’s user base. To achieve this, we needed to better understand how people actually used the solution, identify missing features, and ensure that it would scale easily.

We needed technology that gave us the freedom to realize our and Posten’s ambitions. Elasticsearch, Logstash, and Kibana were a natural fit for delivering the value we needed.  We embarked on a project that had three main goals:

1. Replace FAST ESP with Elasticsearch while maintaining as much of the existing architecture as possible
2. Add geolocation data to products so users could find the stores nearest to them
3. Take advantage of the full Elastic platform to create awesome dashboards in Kibana to better understand our users

Before our conversion to the Elastic platform, Tilbudssøk’s system architecture was very typical of most Comperio projects. Users interact with a Java-based, frontend web application, which passes queries to Comperio’s search abstraction layer, Front. This parses and enriches queries before sending them to the search engine. The results from the search engine are processed in a similar fashion before being returned to the frontend.

With Elastic, we saw an opportunity to reduce the system’s complexity. Elasticsearch has such a robust and complete Java API that we felt including Front would be overkill. And once we properly started using Elasticsearch’s Java API, we found it was a joy to work with! All of Elasticsearch’s functionality is exposed and easily accessible through various builders. We quickly built up a set of tools that we could combine and extend to handle Posten’s current and future needs.

With these tools, implementing geolocation was simple. Adding it to Tilbudssøk in the form of a store locator feature improved the user experience on the site significantly. Without it, a user might find a product and then be asked to exit the site to learn more. Now, users find a product on the site and, more and more, they are using the store locator to learn where they can purchase it. This trend is particularly significant as the store locator is the closest thing to a customer conversion funnel we have without Tilbudssøk being an actual ecommerce solution.

Setting up an initial cluster to handle Tilbudssøk’s current usage was also simple. We host everything on AWS so creating extra nodes is as simple as spinning up a new server, installing Elasticsearch, and pointing it at the existing cluster. A main client node that also acts as a load balancer for the rest of the cluster is the main point of communication between the index and the frontend web application.

Once our Elasticsearch-based system was ready to roll, the next step was to perform (and visualize) analytics on user behaviour by creating awesome dashboards in Kibana. Every request made on the site triggers a request to the search index via the Elasticsearch Java API. This allows us write as much information as we want about each request and response to a log. We can then parse this log with Logstash and visualize everything with Kibana.

We started by tracking classic search metrics such as query term, hit counts, and query time. As each user action triggers a search request, we realized we could expand this to use Kibana as a full web/user analytics dashboard. This allowed us to enrich our monthly reports for Posten and their customers with additional user insights based on data points such as most popular products per retailer, product category, and even how retailers measure up against each other. It also gives us a clear indication as to what features work, which need improving, and what might be lacking.

We have plans to further enrich our user behaviour tracking with tools such as Piwik. Logstash’s ease of use will allow us to store additional user data in Elasticsearch so we can get even more value out of Kibana.

To conclude, the project was a fun dive into the workings of Elasticsearch. We successfully added new features to Tilbudssøk that we could not have achieved without Elasticsearch. Perhaps even more importantly, we now understand Posten’s users so much more than we did previously — helping us deliver even more value back to our customers.