Observability
Software and Technology

Autopilot: Powering the Search for Insights

AT A GLANCE

  • 2.9
    billion documents
  • 2.19
    TB worth of data
  • 50
    million events per day

Fast and Flexible Product Innovation

Using the Elastic Stack, Autopilot has created a simple but powerful business analytics tool in just six months.

A New Unique Selling Point

Connecting campaigns with business results has made Autopilot’s marketing automation software much more effective in increasing conversions.

Improved Performance

A logging and monitoring solution powered by Elastic Stack helps Autopilot troubleshoot performance issues and address them right away.

Avoidance of Maintenance Costs and Resources

With Elastic Cloud, Autopilot can focus on its core mission and leave maintenance to Elastic.

Company Overview

Autopilot is on a mission to empower more companies to create remarkable marketing journeys. Its marketing automation software is as easy to use as a whiteboard and offers a visual approach to creating personalized customer journeys. Using the Elastic Stack, Autopilot has developed a powerful way to track and measure those journeys. Known as Insights, this powerful analytics tool has replaced static and fragmented reporting with real-time performance data. It lets customers visualize and analyze time series data and make smarter decisions to optimize campaigns. Insights was developed in just six months with the use of Elastic Cloud, fast tracking development and streamlining maintenance. It is now a core component of Autopilot’s software which is used globally by customers like Microsoft, Patreon, LiveChat, Instapage, and Crunchbase.

Powering the Search for Insights

Autopilot helps companies of all sizes and industries automate marketing, rekindle old customer relationships, reach new audiences, and grow their bottom line. Customers can create and send campaigns via email, in-app messaging, SMS, and even physical postcards. They can import all their contacts and organize them into smart segments that trigger customer journeys based on field values, behaviors, or UTM codes.

Integration with third party solutions like Salesforce, Zapier, and Slack extend Autopilot's capabilities even further. Five years after its launch, Autopilot's software is used to generate as many as 50 million events per day. Thanks to Insights, all of these events can now be tracked and measured, helping journey marketers improve the customer experience and convert more revenue.

Autopilot’s Journey with Elastic

The Who

Elastic was first introduced into Autopilot by Christopher Giffard, Senior Software Engineer at the marketing automation company. Giffard had used the Elastic Stack in a previous role, and within his first week at Autopilot, he brought it in for logging and monitoring.

It was the first time Autopilot had used a database driven solution for logging. The engineering team had been entirely focused on getting software out to market and to date had maintained the logs in a central server. Seeking to get more value from the logs, Giffard deployed Elasticsearch, Logstash, and Kibana.

This logging solution has made it easy for engineers and product managers to troubleshoot issues. "Kibana is an amazing exploratory tool and gives us a whole picture of what's happening within our system as well as individual errors. It means we have to do a lot less debugging and scripting to get the information we need, and can deal with issues more proactively," said Giffard. "It's our first port of call when there's a problem, helping us diagnose the root cause in a really integrated way."

It's also introduced the Elastic Stack into the technical foundations of Autopilot. Three months after implementing the logging solution, the engineering team was presented with a bigger challenge.

The What

Autopilot wanted to develop an easy to use analytics tool to help customers understand the status and success of their campaigns. At that time, reporting was fragmented with information coming from a number of different data stores.

"We wanted to aggregate all of the different events taking place, and make it possible for customers to track and visualize the performance of their campaigns over time," said Giffard. "The idea was to help them measure things like the success of email A vs email B but also identify the channels and messages needed to convert more revenue."

The first step in designing the tool was an analysis of the data behind it. Autopilot needed to understand where it was coming from and how it could be tracked. It was through this process that the Elastic Stack was identified as the ideal engine for the tool. "The data was all time series data and we needed a solution that could search through it rapidly and generate aggregations. Elasticsearch is bread and butter for that," said Giffard.

Autopilot’s Insights dashboards

The Why

The engineering team dove right into the project, setting up a trial of Elastic Cloud and spinning up a new cluster. At the time, Autopilot's logging solution was managed through a third party hosting solution. However, Giffard and his colleagues wanted the benefits of a direct subscription hosted through the Elastic Cloud. Chief amongst these were access to the latest versions of Elastic Stack as well as X-Pack.

Only weeks into the trial, Autopilot signed up for Elastic Cloud with a Platinum Subscription. Giffard said Elastic had exactly what it needed: analytical power, management convenience, and easy scalability. "When you are dealing with high volumes of data, you need to be attentive to how you are indexing and storing that data. There's a large amount of server maintenance involved," said Giffard. "By using Elastic Cloud, we can avoid doing that ourselves and stay focused on building features for our customers.

The How

Autopilot's software is built around the actor model, so for any event generated, there's an actor, action, object, context, and resource, allowing any kind of customer event to be recorded. Insights is also based on this model and uses Elasticsearch to capture these five elements plus a timestamp. The data can then be explored through the use of Kibana and Timelion at the back-end, and presented to customers via Insights.

In more practical terms, customers can define the success criteria for their campaigns based on the events they want to generate. These may include signing up for a newsletter or taking up a free trial. They can then track and measure how these goals are reached.

The time series analysis made possible with Elastic Stack means that customers can track conversions but also gain insight into what made customers convert. "Customers can see how many events are occurring and map them back to specific campaigns. With Insights and the analytical power afforded by Elasticsearch, they can also better understand how their campaigns are performing over time," said Giffard.

A Change in Direction

While the development of Insights took just six months, there was a big adjustment along the way. During beta testing, it was discovered the initial stats presented by Insights were not as useful as they could be and needed to be calculated in a completely different way. Originally, events were presented in graphs and aggregations at the times they were recorded, rather than taking into event attribution and cohort grouping.

The system was refactored to automatically attribute events and provide customers with deeper analysis on the performance of campaigns. For example, they can see how many contacts added to a campaign on a certain day went on to convert, regardless of the time in which they did so. Elastic helped Autopilot address the performance impact of the new queries, and adjust aspects of its system architecture to deliver a fast and stable system.

"Elastic was really helpful in ensuring we could redesign our queries and optimize performance," said Giffard. "Their support is in fact one of the most valuable aspects of our subscription. It is more of a dev consultancy arrangement where we can talk to them about any concerns or constraints we're experiencing and work together to address them."

The change could have thrown the project entirely off track but with support from Elastic, Autopilot was able to set up new queries in two weeks. And those queries were made read only with Autopilot using X-Pack Security to protect data integrity inside its cluster.

The Results

Autopilot's customers now have easy access to campaign performance data and can run complex queries in just a few clicks, and see results immediately. Giffard said that the team would not have been able to achieve the same level of performance for Insights without Elastic. "Our philosophy has always been to provide our customers with solutions that are really simple to use. Elastic has made it possible for us to deliver within that scope and create a business analytics tool that anyone can use," said Giffard.

Insights is now a key selling point for Autopilot's marketing automation software, helping customers connect the dots between campaigns and business results. It is also used internally to monitor the use of Autopilot's software and aggregate customer behaviour. "Our customers rave about Insights and, for us, it is a key step towards designing solutions for broader use cases uses. We'll be fuelling even more data into Elasticsearch to achieve this," said Giffard.

Autopilot’s Clusters

  • Clusters
    5+
  • Indexes
    270
  • Nodes
    19
  • Query Rate
    ~20/s
  • Hosting Environment
    Elastic Cloud (AWS)
  • Replicas
    1
  • Documents
    2.9 billion and growing by ∼100-1000 a second
  • Time-based Indices
    1 per day
  • Total Data Size
    2.13 TB
  • Node Specifications
    64 GB Elastic Cloud node size
  • Daily Ingest Rate
    14,759,427 docs on average