Logmatic.io

The Challenge

How do you scale from tens to hundreds of customers while improving the customer experience?

The Solution

By using Elasticsearch to deliver near real-time data updates and ultra-fast query response

Case study highlights

Deliver Top-of-the-Line Service

  • Update data from across the Internet within 10 seconds
  • Provide query responses in less than a second
  • Ensure continuous access to the system with no downtime

Scale to Hundreds of Customers

  • Deliver performance regardless of number of users
  • Add new nodes quickly and cost-effectively
  • Reduce cost of growth by 75 percent

Guaranteed data updates in 10 seconds

Logmatic.io listens to the Internet. The company collects data in real-time from micro-blogging sites, blogs, video sites, social networks, discussion forums, and news broadcast sites – anyone anywhere in the world that is talking about the topics of interest to their customers – and delivers the information back to these clients via an easy to use online platform. This data helps Logmatic.io customers answer critical business questions and gain essential insights into their markets and its consumers.

Using the Logmatic.io platform, customers can search for information on their own brand, customer feedback, competitors, their market verticals or any topics related their business. Logmatic.io indexes more than 100 million documents on their production platform, and the company guarantees customers the freshest data from the internet, making them available in the user interface within seconds.

Depending on fast response time

“Before Elasticsearch, providing data in real-time was getting harder and harder as volumes grew," Renaud Boutet, Co-Founder of Logmatic.io recalls. “We are aggregating tens of thousands of objects every minute. Our older NoSQL database solution was not able to keep the pace. All the systems were struggling."

On the previous system, Logmatic.io had to wait for all queries to finish before they could update it. With multiple users constantly performing queries, there was never a good time to update. This made it harder for Logmatic.io to deliver on their real-time promise.

“Before Elasticsearch, we were experiencing slow query response time as well," Boutet continues. “Sometimes customers were waiting 5 to 7 seconds, which is not good for a business intelligence tool. Queries would sometimes take more than a minute. You lose all the power of a business intelligence tool if you have slow responses."

Adding to the challenge, Logmatic.io replicated hundreds of gigabytes for data protection, but the synchronization process was slow and would often fail.

“We were very concerned at that time, because we only had ten customers, but we were already reaching the limit," Boutet adds. “We were afraid that we would never be able to scale up. We had to find a new solution because otherwise we would not be able to continue our business. That is why we moved to Elasticsearch."

"Before Elasticsearch, providing data in real-time was getting harder and harder as volumes grew. We are aggregating tens of thousands of objects every minute. Our older NoSQL database solution was not able to keep the pace. All the systems were struggling."

Renaud Boutet, Co-Founder

Elasticsearch delivers 10x search power

All customer data, now resides in the cloud on Microsoft Azure servers, goes through Elasticsearch to reach the customer.  One key element in making the decision to move to Elasticsearch was how well it behaved on Microsoft Azure servers.

“Some technologies do not translate well in the world of virtualized machines and network drives. But setting up Elasticsearch on Azure was a breeze, performance is excellent and the system very stable.", says Emmanuel Gueidan, Co-Founder and CTO of Logmatic.io. Coupled with Azure's easy-to-use Linux Virtual Machines, it is easy for Logmatic.io to fire up a new node to run tests or scale the platform.

According to Gueidan, Elasticsearch indexing is much more efficient than the previous system, enabling Logmatic.io to meet their real-time commitment.

“Elasticsearch is 10 times more powerful than the previous database," says Boutet. “We are extremely happy with Elasticsearch."

Replicating data in less than a minute

With Elasticsearch, files are as much as 10 times smaller than the previous database, and the protocol for communication between machines is better, so when Logmatic.io synchronizes machines it is now very fast to replicate the data to Microsoft Azure storage for backup.

“Before Elasticsearch, synchronization would take hours and then fail," Gueidan compares. “With Elasticsearch it takes less than a minute to synchronize the machines and get in green state where the clusters stabilized. Elasticsearch keeps the system running and the quality of service is maintained. The user has access to analytics during any maintenance time."

Scaling from tens to hundreds of customers

Logmatic.io is planning to sell 100 platforms in the next six months, and the 100 million documents will expand accordingly with each new customer, so scalability of the search technology is vital. Currently, Logmatic.io runs 3 Elasticsearch nodes, but in two years they expect to have 10.

“Scaling the previous database was prohibitively costly," he adds. “With Elasticsearch we can scale up very easily, decreasing the cost of administration, the cost of machines, and the cost for every customer."

Logmatic.io's benefits using Elasticsearch

Faster query processing

Elasticsearch has reduced query response time from as much as a minute down to less than a second, a critical capability for Logmatic.io's business intelligence tool.

Business continuity

The efficient indexing offered by Elasticsearch enables Logmatic.io to keep the system up and running during maintenance and backup, without impacting the customer experience.

Low latency updates

Elasticsearch enables Logmatic.io to meet its guarantee to customers that relevant data will be updated in the briefest delays, integrating new message within seconds of posting on the Internet.

Economic scalability

With Elasticsearch, Logmatic.io can scale from tens to hundreds of customers while delivering better performance and reducing costs by 75 percent.With Elasticsearch, Logmatic.io can scale from tens to hundreds of customers while delivering better performance and reducing costs by 75 percent.