Cloud consultancy reduces client outages from four times a day to just once a quarter with Elastic-based cloud automation solution

Mentat recently put its network and automation telemetry solutions to good use for a major U.S. city government's IT organization. Its infrastructure includes multiple data centers, tens of thousands of endpoints, and millions of users.

Before the engagement with Mentat, the organization suffered data center outages about four times a day. There was no one with the time or expertise to uncover the underlying problem. "People were constantly on the phone, whether it was troubleshooting latency issues or packet drops, or servers being offline. They were constantly fighting fires," says Doughty.

"Most city agencies have horror stories where their services are offline or unavailable for periods of time," says Doughty. "With our solution, we can identify the source of the problem and deploy a long-term fix that reduces outages to an absolute minimum," Mentat gathered data from thousands of network devices to identify the problems causing the outages. Elastic Observability was at the heart of the solution, which also included Kafka to act as a buffer and Ansible as the data collector and data manipulator.

"Elastic configurations, like automated backups on a schedule for AWS S3 storage, were extremely helpful," says Doughty. "This gave us peace of mind that the data was always available."

Elastic also enabled Mentat to set up rigorous information lifecycle management (ILM) policies. "We move data out of our hot tier after two or three days. It resides in a warm tier for about 10 days, and then we just physically move it off our platform," says Doughty.

Once all the pieces were in place, the client asked Doughty and the Mentat team to look at cyclic redundancy check (CRC) metrics and pointed them towards the network switches they suspected to be sources of the outages.

An initial measurement of five switches showed that one had a physical defect. When Mentat scanned the entire network, it discovered that hundreds of devices were generating CRC errors. "Once the client worked through all their physical devices and replaced or updated their cabling, they went from four outages per day to one outage per quarter," says Doughty.

The error detection system was so successful that it has since been extensively deployed across the wider client organization. For this roll out, Mentat wrote a customized task for Azure DevOps that plugs into every pipeline across its automation platform and ships that data to its Elasticsearch cluster. The client can then visualize failure rates across different task types and segments of the pipelines.

The next step is to add event-driven, self-healing automation based on Elastic Observability-based triggers. "We set up an initial trigger to view certain cloud data and send an automated email to the cloud team who can address the issue. Now, we want to set up triggers on our platform to automatically perform a specific action and accelerate time to resolution," says Doughty.

"Elastic enables us to create simple, versatile, automated solutions that can easily adapt to many client use cases. It's good for our clients and good for the growth of our business," says Doughty. Mentat also makes extensive use of Kibana dashboards to visualize data over time and in multiple ways. "Whether it's a pie chart, line chart, or even to tag cloud entities, Kibana helps clients understand and fix problems more efficiently," says Doughty.

Mentat has identified about 10 different use cases for the self-healing, automated workflow that can benefit its government client in the future. For instance, scenarios can be baked into Elasticsearch triggers that can fire and execute resolution pipelines if a web server goes down or a database needs to be recovered.