Cogenta: Giving brands a competitive edge through real-time market analysis

Cogenta is a leading provider of retail market data and pricing intelligence to high-end retailers and brands throughout Europe, currently expanding operations to 20 countries. The company offers data and analysis to help retailers respond quickly to market conditions and customer needs.

Cogenta's primary business is based on a real-time pricing intelligence platform, developed in-house. The system gathers pricing and other product information from across the Web every day, enabling Cogenta to build a comprehensive and accurate picture of all major retail markets.

“We provide invaluable information to our customers to enable them to see what their competitors are doing, and react accordingly," explains Yann Cluchey, Chief Technology Officer at Cogenta. “It is all done through automated collection of data on a very large scale."

Cogenta's system collects 20 million price points per day, plus additional product attributes. This data is processed at a peak throughput of 3,000 messages per second, each containing an information update such as a cost change.

“We are constantly gathering data," says Cluchey. “Our challenge is to organize and make sense of that data."

Prior to Elasticsearch, Cogenta used a home grown search engine built on Apache Lucene, integrated with the company's SQL data store. To keep the data up-to-date, Cogenta's team had to constantly build and rebuild indexes in Lucene, which was very time consuming. The company's plans to expand operations to 18 additional countries made this even more challenging.

“We quickly hit a wall with pure Lucene," Cluchey recalls. “We needed a way to scale, in terms of volume of data and speed of updating data. Enter Elasticsearch."

Cogenta adopted Elasticsearch as a key part of its core platform. Elasticsearch easily integrates with SQL, MongoDB and RabbitMQ to accelerate the daily update process and make the data more accessible throughout the business. Cogenta uses SQL as the canonical data store; RabbitMQ for messaging and data transfer; MongoDB as a staging area for the data; and Elasticsearch as the platform for powering internal data processing and end-user applications, most notably the company's new flagship product, Retail Insight, which replaces their legacy search service.

“We are dealing with large volumes of constantly changing data that needs to be interrogated in lots of different ways," says Cluchey. “Elasticsearch is an ideal platform for this."

Cogenta's previous system required index rebuilding processes that would take five hours. Now, data coming into the system is immediately indexed by Elasticsearch and available in seconds.

“Historically, retailers looked at their competitors' websites and gathered information manually," Cluchey explains. “With the help of Elasticsearch, we deliver that competitive information directly to them, on a large scale, using information collected in real time." “Elasticsearch allows us to give our customers access to the data they need immediately without having to wait for us to go through complicated infrastructure updates," he continues. “Having access to updated information in real time, via Elasticsearch, is the key that enables our customers to take immediate action and react to the market quicker."

“Our business strategy has been to use Elasticsearch for most of our key systems," Cluchey says. “As a result of this strategy, we have been able to make considerable savings in our global expansion project. We are running our Elasticsearch cluster on inexpensive commodity hardware, and we scale by just adding more inexpensive nodes. It is seamless and quick."

Cogenta's cost for new hardware on which to deploy their new flagship product, Retail Insight, was only £9K . Cogenta estimates that using a purely SQL based approach would require an additional SQL server, at a total cost of about £16K per server. With each additional node, Cogenta would incur similar added expenses. With Elasticsearch, however, new nodes can be added for only £2K per annum, allowing Cogenta to cost-effectively scale to meet operational growth; using Elasticsearch has allowed Cogenta to cut their TCO by ~40% annually.

“To scale Elasticsearch to expand our operations to 20 countries, all we have to do is add more inexpensive nodes. In a SQL-only environment we would have to purchase more SQL servers, and more licenses," Cluchey says. “It would be very painful."