Compare's architecture comprises a combination of MySQL and MongoDB databases which are used to assemble data from a variety of sources including product offers from online and offline shops, product reviews, etc. Nearly eighty Dropwizard MicroServices running on Java – each with a datastore – run a range of different tasks on the data. With data fragmented across the different stores, another microservice collects all the data and publishes it in a MongoDB database where Elasticsearch is used to access it for maintenance purposes.
"With three thousand product categories, all of them different, a typical problem is matching the others provided by online shops with product categories in the database", said Maarten Janssen, Chief Technology Officer at Compare Group. "Thanks to Elasticsearch, we are able search for categories very quickly and directly map offers to products."
With hundreds of thousands of customers, speed is crucial to maintaining a responsive service that makes people come back time and time again. Plus, data needs to be kept continually up to date. As Mark Gaasbeek says, "When it comes to website performance, the faster it is and the better it performs, the more users engage with the site and stay longer."
Compare Group has a single index per label (vergelijk.nl, vergelijk.be, vertaa.fi etc), which is updated constantly during the day and night. They try to build as few indexes from scratch as possible and only do updates.
The full indexes containing around 100 GB of data and upwards 30 million records are currently swapped once a day.
It's not just at the front-end that things go faster. With Elasticsearch, the company is also saving time in the IT department. Not only are back-end processes at least twice as fast, but development times have come down significantly. Mark Gaasbeek explains: "Development hours are scarce, so that is an indirect time saver because we can spend the time that is left on other things." Chief Technology Officer, Maarten Janssen continues: "As its name suggests, Elasticsearch is flexible. I don't need to worry about structures or if I can integrate something. It's just there. I know I can talk to it and do a thing and it works. I also like the stability. It's fantastically stable and downtime is reduced to almost zero, which is a huge benefit for the business side."
To help Compare Group's marketing managers understand the data better, Kibana is being used to analyze data and identify trends to support marketing strategies. During an unexpectedly warm spell, for example, managers using Kibana noticed that there was a sudden run on air-conditioning units and were able to respond immediately, advising shops that they should feature air coolers.