The Challenge

How do you provide actionable insight into social media for the largest companies in the world?

The Solution

By using Elasticsearch to process hundreds of documents per second to deliver real-time data and analytics

Case study highlights

Demonstrate superior search power

  • Process up to 80,000 simultaneous queries
  • Handle hundreds of new documents per second after migrating from MongoDB to Elasticsearch
  • Deliver real-time data and analytics

Scale cost-effectively

  • Cut costs almost in half
  • Serve the world's largest brands
  • Respond immediately to massive queries

Processing hundreds of documents per second

MutualMind offers "Social Listening as a Service," enabling customers to monitor their brands on social networks, and gain actionable analytics and insights. MutualMind customers – including some of the best-known brands in the world such as AT&T, Kraft, Nestlé and Starbucks – use the MutualMind platform to support programs including product development, market research and consumer care.

Customers can use MutualMind to search for mentions of brands, products or any topic relating to their market or business. MutualMind treats every social media post – tweet, blog or YouTube video – as a document, and the platform consumes hundreds of documents per second to serve customer needs. The system can process between 30,000 to 80,000 different queries on these documents at any moment.

Being responsive to the speed of digital conversation

"Prior to Elasticsearch, we were using MongoDB as our primary content database here at MutualMind," recalls Nizam Sayeed, CTO and Co-Founder of MutualMind. "Six months into deploying it to production, we started having several performance and scaling problems."

"MongoDB kept crashing all the time, even though we hadn't even reached the scale we have today," adds Eric Swayne, Director, Product Management at Mutual Mind.

"MongoDB literally kept us awake many nights, which was unacceptable," Sayeed continues. "So we started looking for a better alternative. Elasticsearch's vision to provide a very fast, highly scalable and reliable database solution powered by Apache Lucene, a proven search engine technology, struck a chord with my team. We switched to Elasticsearch and have never looked back."

MutualMind also delivered social analytics from a Hadoop-based solution, and when they switched from MongoDB to Elasticsearch, the platform operated as a hybrid – Elasticsearch was used for consuming and delivering data, while Hadoop was used for statistics analysis.

"We realized we were quickly getting beyond the capability of Hadoop," Swayne remembers.  "We were running Hadoop batches every 15 minutes and sometimes longer depending on the volume of the data coming in. Our customers won't wait 15 minutes. They can barely wait 15 seconds."

"We decided to migrate all of our social analytics from a Hadoop-based solution to a real-time analytics cluster powered by ElasticSearch," Sayeed says.

"Now we use Elasticsearch for analysis and storage," Swayne confirms. "Elasticsearch makes content and analytics one and the same. As soon as documents come in, they are analyzed and those statistics are reflected in our platform. Elasticsearch allows us to be extremely responsive to the speed of digital conversation."

Real-time insight for a real-time world

"Our customers are living in a world quite literally moving at the speed of light," explains Swayne. "The digital conversations around them are often daunting, and wildly varying, and at a moment's notice can be transformed by a crisis, a product issue, or even by an irate celebrity. Having real-time support is absolutely essential. Elasticsearch helps us provide customers with the real-time insight they need to respond to a real-time world."

"MutualMind processes end-to-end data in seconds and milliseconds, from the moment someone tweets to the moment they provide that data to clients," he adds. "Elasticsearch allows us to provide analytics immediately, so the moment our customers initiate a query we are providing them with data. Very often we will be in a meeting with the client and we will type in a query, and we can see data coming in right away. When we can show that, it is a very powerful proof point."

"The advantage of Elasticsearch comes down to three things: time, efficiency and cost."

Eric Swayne, Director, Product Management

Delivering 24/7 reliability

Not only is MutualMind responsible to deliver high availability to customers but they are also required by data providers to maintain high uptime. If they do not meet these SLAs, MutualMind must pay a fine.

"Our business has to be up. Our platform has to be extremely robust," Swayne explains. "We have some command centers that are permanently installed in front of teams literally looking at them 24/7. We must provide as much uptime as possible, so we invest in Elasticsearch, which will let us do that with the best efficiency."

Scaling to serve very large customers

The real scalability challenge for MutualMind is the size of the queries. If Starbucks, Coke, Xerox, 3M and Cisco all simultaneously began using the system, MutualMind would have to scale exponentially and bring new clusters onboard. Spinning up all these new clusters comes at a cost, and scalability for MutualMind means controlling these costs. The more efficiently they can scale upward, the more easily they can accept the huge queries that their clients want to execute.

With Elasticsearch, Swayne says they have already seen a substantial drop in the number of machines to launch and manage as well as a reduction in the monthly price, cutting costs almost in half compared to MongoDB. In addition, they are processing and storing 10 to 15 times more data than when they used MongoDB.

"The cost-effectiveness of Elasticsearch is really powerful for us to get the big brands on board very quickly," Swayne concludes. "And big brands are where we are going to make the most money, because we charge by volume."

MutualMind's benefits using Elasticsearch

Real-time actionable insight

Elasticsearch enables MutualMind to deliver social media content to customers immediately, so they can react in real time.

Cost-effective scalability

Elasticsearch's low cost and efficiency empowers MutualMind to take on the world's largest brands as customers, while improving performance.

High availability

MutualMind depends on Elasticsearch reliability to meet the demanding uptime requirements of 24/7 command centers and strict SLAs from data providers.

Reduced operating costs

MutualMind cut storage and search costs almost in half by deploying Elasticsearch.