19 March 2015 User Stories

Using ELK to Keep the Lights On

By John Boere

Keeping the lights on is literally what we mean. We're talking about the world's power grid or more generally, electricity.

Electricity, like air and water, is something that most of us take for granted. It should always be there, right? And we're OK when it is. Until it isn't, which somehow always happens at the worst time, like during a storm. How am I going to update my Facebook status now? Seriously though, as an electrical engineer it's my job to make sure we always have an abundance of reliable and high quality power. Without it the world would be a very different place. Most people don't think about it like this, but compare it to water. With shortages (droughts) or pollution, we and our food cannot prosper. It's the same with power, low quality will lead to outages and shorten the lifespan of devices. Interestingly enough without power most of us wouldn't even have access to water, as none of the pumps to deliver it would work. However, reliable power is easier said than done, and engineers have been working on improving this technology for over a 100 years now and still continue to do so. And so are we at Cliffhanger Solutions.

While we don't operate a power plant, we help our customers, utility companies, with preventative maintenance and reducing outage restoration times, among other things.

Today, the [power]grid is transitioning from being 'dumb' to getting smarter. This smart grid uses technology to detect and react to local changes in usage, using computer-based remote control and automation. A large number of smart devices and sensors capture data points from the grid. Data is being collected from a variety of systems such as Supervisory Control and Data Acquisition (SCADA), Outage Management Systems (OMS), Smart Meters (AMI) just to name a few. Capturing data from this vast amount of equipment results in a lot of data points. And a lot of this data is also recorded over time intervals, allowing for historical and trend analysis. So what we have is a high volume of data that is being captured at (somewhat) high velocity.

Ideally, this data would have to be analyzed in real time. But as it stands now, we have a long way to go before we get to that point. In one instance, data from smart meters doesn't work with the outage management system —from the same vendor!

The root of the problem is that all this data is in its own data silos, making collaboration much more difficult, both from a technical and organizational perspective. The end result is increased outage times and lower customer service. Instead, what we need is an effective mechanism to search and analyze across all these incompatible datasets, preferably in real time.

Hello ELK stack.

At Cliffhanger Solutions, we index data in real time from various sources using Elasticsearch and Logstash. Sources include GPS location data from maintenance trucks or from tablets running our app, readings from smart meters and facility data from GIS (geographical information systems).

To visualize, search and analyze this data we build a mapping application (called ATLAS web and mobile) in which we also embed Kibana dashboards. Think of it like Google Maps on steroids.

Operators can now quickly get answers to questions like: "Can I safely close this switch and restore power to these 1500 customers?" or "A storm is coming in from the South, how fast can I get my bucket trucks to the area where the storm will hit?". As for preventative maintenance, engineers can seek answers to questions like: "Transformers from vendor X have a higher than average MTBF (mean time between failures). Find all of them and sort them by installation date, then send them to the work order system for inspection or replacement". While it might not sound like a big deal, this is actually pretty incredible and this wasn't possible until now without heavy investments in consultants or getting locked in with the few "one stop shop" large vendors that offer a 'total solution'.

Anecdote: One company noticed that MTBF was higher for (wood) power poles in an area located near a certain park. After a lot of analysis it turned out that dogs like poles. Who would have thought? This resulted in a shortened lifespan for the pole. The easy fix was to coat some tar around the base of the poles.

Cliffhanger Solutions is a small company, but the flexibility of Elasticsearch allowed us to focus on creating value for our customers instead of getting stuck in maintaining different systems for different clients. And our clients are getting it as well. For example, we serve a tiny utility on a Caribbean island, with only 1 guy in the IT department. By using ATLAS (+Kibana) out of the box, we built them a dashboard to show them outages on a map, color coded by customer density. This would never have been possible even a few years ago. The ELK stack is pretty incredible at making data searchable even if the source data is not clearly defined. Unlike traditional databases you don't need to know your questions in advance, you can explore and find correlations you didn't even know existed. It reduces a lot of overhead (who needs a DBA?) and ultimately it keeps the light on!

Internally, at the Cliffhanger office, we use the ELK stack to monitor the status of our clients' applications. We use it to improve search relevance, performance, find errors and prevent hack attacks. We share this data with our clients. They like this level of transparency and it gives them confidence that their data is safe.

So next time you have a power outage, ask yourself why they are not our client yet. Send them our way, so you can keep updating your FB status ;)

About John
With over 20 years of software and electrical engineering experience, John is widely seen as an industry expert. He is CEO at Cliffhanger Solutions, a technology company aimed at helping utilities around the world. Over the years John has worked at dozens of clients in 8 countries.

Clients include Emera Maine, Salt River Project, Tucson Electric Power, German Telecom (parent of T-Mobile) and Elmar - a power utility in the Caribbean. John lives in Chandler, AZ with his wife and 3 sons (3, 6 and 10).