Software and Technology

Lexer: Powering the Search for Customer Insight


  • 250
    million records updated per day
  • 10
    TB total data size
  • 40
    minutes to restore complete data set

Accelerated Speed

The most frequently used features of Lexer’s product are up to 40% faster which translates to savings of hours a day for its clients.

Empowered growth

Lexer’s platform is now able to analyze and manage hundreds of millions of human behaviors per day to support their ever increasing customer base.

Rapid Recovery

While Lexer’s uptime is 99.999+%, with Elastic’s Snapshot & Restore feature, they can back up their entire data set every 15 minutes and, in the event of a complete platform corruption, they could recover in just 40 minutes as compared to an estimated six days in the past.

Reduced Maintenance

Migration to Elasticsearch 5.4.1 has reduced maintenance time and costs. Issues like failed nodes are now much simpler to solve and take 25% less time.

Company Overview

Melbourne-based Lexer combines human data and simple software to power genuine engagement between companies and their customers and prospects. It loads data from hundreds of millions of sources, turning it into actionable insights covering individual customer life-stages, behavior, attitudes, and demographics.

Lexer works with market leaders across all industries — including Qantas, EA, Billabong, Westpac, and Westfield — to embrace a data-first operating culture and extract maximum value from data using the latest machine learning, natural language processing, and data modeling techniques.

Powering the Search for Customer Insight

Lexer has built a powerful Customer Data Platform (CDP) which unifies data from a range of sources to provide clients with actionable insight into customers and prospects. These data sources enrich Lexer's client data with other data such as social media posts and consumer financial transaction data. This provides Lexer's clients with a powerful cross-channel view of each customer and prospect so they can better serve and win new customers.

Lexer's Customer Engagement Hub enables clients to take advantage of this data to engage with stakeholders in a more human way. For example, they can provide faster, more contextualized customer care. They can also personalize the customer experience of websites and apps, and target the right audiences with the right information.

As Lexer has evolved, the sources of data it brings together have multiplied, as have the tools it provides its clients. The Elastic Stack has helped power this evolution and is a core component in how Lexer and its clients search, analyze, and visualize data.

Lexer’s unified view of a customer

Lexer’s unified view of a customer

Lexer's Journey with Elastic

The What

Lexer was founded in 2010 and began using Elasticsearch that same year. Its Customer Data Platform is built on Elasticsearch, using the search and analytics engine to power actionable insights from all its data sources.

Initially, the Customer Data Platform managed around millions of pieces of social content per day, capturing mentions of its clients that were publicly posted to channels such as Facebook or Twitter. At that time, the Customer Data Platform was predominantly used for social listening. However, Lexer's clients soon wanted the ability to do more with their data such as engage customers on social media and understand their purchasing habits. Lexer responded by offering new tools to help clients with customer engagement and discovery.

Aaron Wallis of Lexer said that by connecting data from a number of 1st, 2nd and 3rd party sources the Customer Data Platform essentially created human profiles.

"Our data machine has grown bigger and bigger over the last couple of years. We are bringing in Wi-Fi, demographic, movement, and other unstructured datasets from a variety of partners and then layering it with our clients' data, including transaction information and customer identifiers like emails and customer IDs. So we can start to build unique profiles of each customer which can be used to personalize offers and communication," said Aaron.

The True View You Need

The True View You Need

These insights helps Lexer's clients engage with customers more personally and in real-time, enabling them to develop smarter strategies for activating new prospects. For example, Lexer's clients can better segment their audiences and understand the lifetime value of different customer groups. They can also identify those most likely to make a purchase and contact them at the right time with the right offer.

These capabilities have increased the value that Lexer delivers to clients and opened the door to new opportunities in Australia and the US. However, powering these capabilities and scaling the platform has come with some challenges. As Lexer ingested more and more data into its Customer Data Platform, the speed and robustness of the platform came into question.

Our platform was managing hundreds of millions digital human behaviours per day and we knew this would grow as we expanded into the US. Leveraging the latest Elasticsearch capabilities became a crucial next step.

– Aaron Wallis, Lexer
The Why

As its data handling needs grew, Lexer made the decision to upgrade from Elasticsearch 2.3.4 to Elasticsearch 5.4.1. With Elasticsearch already underpinning its successful Customer Data Platform, no other solution made sense, and upgrading to version 5.4.1 would provide the robustness and performance that Lexer needed as it continued to scale.

Elasticsearch has been evolutionary for our business. It's saved us hundreds of hours in development time and helped us to build a platform that can easily mine massive volumes of data in real time. There was no question that we still needed Elasticsearch, but it was time for us to upgrade.

– Aaron Wallis, Lexer

"Elasticsearch has been evolutionary for our business. It's saved us hundreds of hours in development time and helped us to build a platform that can easily mine massive volumes of data in real-time," said Aaron. "There was no question that we still needed Elasticsearch, but it was time for us to upgrade."

The How

Before Lexer could upgrade, it needed to move its data into smaller indexes. It had a single index of 2.8 billion social posts, comments, messages, articles and blogs which was resulting in a recognizable lag in query performance for their customers. To combat this, they transitioned to smaller indices tailored to the event type. This paved the way for a smooth migration and increased the speed and efficiency of client search requests. It also allowed for increased scalability.

Next, Lexer updated its client side libraries to take advantage of native Elasticsearch features such as geolocation searches.

Once testing was complete, it moved on to the upgrade using Elasticsearch's snapshot and restore facility. This facility allows users to copy index data and then restore that data very quickly into a different cluster. In Lexer's case, it copied the index data to Amazon's Simple Storage Service (S3) and completed a full disaster recovery scenario, restoring its entire 2.8 billion object data set within just 40 minutes.

"We not only upgraded our cluster, but created a robust system that backs up our entire data set every 15 minutes," said Aaron. "If something goes wrong, we can restore our entire cluster in 30 to 40 minutes whereas in the past, it could have taken us up to a week."

We not only upgraded our cluster, but created a robust system that backs up our entire data set every 15 minutes. If something goes wrong, we can restore our entire cluster in 30 to 40 minutes whereas in the past, it could have taken us up to a week.

– Aaron Wallis, Lexer

The Results

The benefits of the upgrade were immediate. General query performance improved 13% while performance of features in the Customer Engagement Hub improved up to 40%.

These performance enhancements translate to a time savings of 6 hours of waiting time per day across all of Lexer's clients.

The upgrade has also reduced the cost and burden of maintenance. Aaron said if a node were to fail in the middle of the night, the team can simply create a new one and get back to sleep. They don't worry about these types of issues like they did in the past. Also, the clusters have been configured in a way that future upgrades can be done with ease. This will lower the investment of time and costs and give Lexer more flexibility to take advantage of future versions.

The process of the upgrade was made easier with the support provided to Lexer as part of its Platinum subscription. During planning, Lexer was able to call on Elastic for support in building out its proof of concept for the upgrade and to confirm architecture requirements. This not only saved Lexer time but helped ensure a seamless upgrade.

Lexer continues to benefit from Platinum support, learning and adopting best practices that have improved the reliability and performance of its Customer Data Platform. Additionally, they've adopted more components of the Elastic Stack, including Logstash, Kibana, and Monitoring from the X-Pack. Logstash and Monitoring are particularly useful for monitoring the health and security of Lexer's cluster and infrastructure.

Lexer Listen

Lexer Listen

We came away from the project with faster speeds for clients, better processes for Elasticsearch cluster upgrades, and a robust backup system for rapid disaster recovery. It also gave us a lot of ideas on how we can continue to improve search performance and make our cluster more efficient and scalable for the future.

– Aaron Wallis, Lexer

Lexer’s Elastic Stack Clusters

  • Clusters
  • Indexes
  • Nodes
  • Query Rate
    ~25 per second
  • Hosting Environment
  • Replicas
  • Documents
    2.8 billion
  • Time-based Indices
    1 per day
  • Total Data Size
    10 TB (with replica)
  • Node Specifications
    AWS Instance r4.2xlarge with attached EBS GP2 (SSD) volumes
  • Daily Ingest Rate
    ~60GB (400 documents per second)