<img src="https://api.contentstack.io/v2/uploads/58789eed6c36611650aaa093/download?uid=blt23e25287504f727a" data-sys-asset-uid="blt23e25287504f727a" alt="mapp logo.jpg" width="305" height="88" style="float: right;width: 305;height: 88;margin-top: 0px;margin-right: 0px;margin-bottom: 0px;margin-left: 0px;text-align: right;"/>We are a small <a href="https://www.elastic.co/what-is/root-cause-analysis" target="_self">Root Cause Analysis (RCA)</a> team at <a href="https://mapp.com">MAPP</a>, and we are using Elasticsearch, Logstash and Kibana (Elastic stack) for logfile analysis. Logfiles have always been our friends, but at some point the standard Unix command line tools that we used before didn't suffice anymore, which is when&nbsp;we realized that we need a more powerful tool. We built a cluster and pumped lots of logs into it. Today, our biggest cluster holds 8.5 TB of data, giving visibility across applications and locations. The Elastic stack is changing the way we work, and it's spreading to ever more applications and teams.<h2>The vitality of the Elastic technology stack</h2>Different modules of our software produce their own logs, in different formats. Previously, we did all those dirty grep, sed, awk moves to extract certain patterns out of the raw logs and then do analysis on this. It was absolutely difficult, and it was impossible to see any trends based on different parameters. When we started using the power of the Elastic Stack, the whole log analysis became faster, more proactive, meaningful, and accurate. With Elasticsearch we can now do full text search in near real time. Using Logstash and its powerful plugins, we can deal with different types of logs from different software modules and extract valuable fields out of logs. Kibana gives us the power of creating and using dashboards as well as extracting the analysis results based on the logs. Also, passing the information from team to team became easy by&nbsp;sharing dashboards. Plugins such as Kopf, Curator and Marvel help us to manage Elasticsearch and its indices.<img src="https://api.contentstack.io/v2/assets/575e4d209985d589763782d5/download?uid=bltb489d6adac242038?uid=bltb489d6adac242038" dstyle="max-width: 100%;" style="max-width: 100%;"/>&nbsp;<h2>The Elastic technology stack from head to toe</h2>When we say that we use the Elastic Stack, this means that we are covering the full logfile analysis story, content and technology. Let's start with the application that produces the logs. We know it and the people behind it because of our <a href="https://www.elastic.co/what-is/root-cause-analysis" target="_self">Root Cause Analysis (RCA)</a> role. So we can give recommendations on the content of the logfile, and also on the structure. Next we make this visible in the Elastic stack, either by feeding it into an existing cluster or by creating a new stack. In the latter case, it's us who set up Elasticsearch, Logstash and Kibana. Then we go back to the users to ask how this works for them. Working with the Kibana dashboards, they will start to think differently and come up with new use cases. We'll keep developing the Elastic Stack, configure Logstash, do the sizing, optimize and upgrade all components. The Elastic Stack users are Third Level Support, DevOps, Developers and Product Management. And us. This is very important, there's no better way to understand your user than being one yourself.<h2>A powerful combination: Elastic Stack and RCA</h2>In this post, I'll argue that this is an amazingly powerful combination: RCA, the Elastic Stack and a full team.&nbsp;For one, as RCA, we've always been bridging teams and departments. We talk to everyone who is affected by incidents, and to everyone who knows about their causes and helps to prevent them - in short, to everyone.&nbsp;Second, we have a&nbsp;vital interest in the Elastic Stack, using it a lot, while at the same time the&nbsp;Elastic Stack&nbsp;has a&nbsp;great "Power to the User" built in philosophy that&nbsp;makes it possible to answer questions from many perspectives.&nbsp;Third - let me take one step back. Jez Humbel wrote in <a href="https://www.thoughtworks.com/de/insights/blog/there-no-such-thing-devops-team" target="_blank">this article</a>&nbsp;that "Bad behavior arises when you abstract people away from the consequences of their actions." In this spirit, we don't tell people "Don't worry, we take care of this", but aim at bringing people back in touch with the consequences of their actions. The Elastic stack is a great tool for this.&nbsp;Fourth, the Elastic Stack is flexible, so we are, too. Every component, Elasticsearch, Logstash and Kibana, adapts very well to many situations. It processes all kinds of input and lets you combine this with other data sources. For example, we extract information on human readable names from a database, build a dictionary, then use the Logstash translate plugin to&nbsp;add this information to our events.<h2>Attack of the Buzzwords</h2>Now, what happens when you have a team with the mission to identify and analyse incidents and the tool and the knowledge to spread this across the company? I'd like to show how this relates to Agile, Microservices, DevOps and Lean - sorry about this avalanche of buzzwords, but I promise that I have a reason to use them. The Elastic Stack is a natural microservice. It is small, it serves one purpose, and it is deployable independently. We are so lucky. Enjoy it, use it, and build a team that can handle it. Some members who are close to the content and interpretation, like experienced RCA people, and one or two DevOps. This team is a cross-functional group of people that have everything, and everyone, necessary to get your Logstash analysis flying - and is an agile team by this definition. There will be requests for more Elastic Stacks&nbsp;springing up all over the place. You have one in production, you need one for&nbsp;smoke tests. You have one for this&nbsp;applications, you need one for&nbsp;another one, too. You have one in EMEA, you need one in the U.S.. And so on.This in turn means that your team will be setting up Elastic Stacks in all kinds of environments. If server setups differ, they will be feeling it. If the configuration is not the same, this will show in the dashboards. If automation isn't simple, they'll be slowed down. They say that you can recognize the vanguard by the arrows sticking out of their breast - let's have a look, yes, there's a number. But this is what a vanguard is for. We push for smoother deployment processes and automation, and we cross department borders. We aren't the only ones, of course, but we have a good chance to succeed because our service is so small. When we do, this paves the road for a more agile culture.We've already dealt with most those buzzwords, except for "Lean". This comes in when you look at the Build-Measure-Learn Circle:<img src="https://api.contentstack.io/v2/assets/575e4d0512798d8e381fea0b/download?uid=blt9fe95f4ae864aa0c?uid=blt9fe95f4ae864aa0c" data-sys-asset-uid="bltbb6643e0c36f1940" alt="build_measure_learn_circle_teradata.png" style="max-width: 100%;"/>Full presentation of Eric Ries can be viewed <a href="http://www.slideshare.net/startuplessonslearned/2012-05-15-eric-ries-the-lean-startup-pwc-canada" target="_blank">here</a>. Focus on the lower left: From "Measure" to "Learn". There tend to be gaps. The Elastic Stack / RCA cannot fill all of them, but help some. The Elastic stack is good for "Measure Faster", and RCA is good for "Learn Faster”. And this is something worth working for.<h2>Outlook</h2>We are constantly working on our Elastic Stacks, presently implementing Shield and Watcher. Shield is important to enforce multitenancy, while retaining all the data in one cluster to facilitate analysis across customers. Watcher will help us to give special attention to issues, new workflows or jumpy customers. There are more adventures waiting for us with the Elastic stack.<hr/><img src="https://api.contentstack.io/v2/assets/575e4d1e58208ba076e298e9/download?uid=blte449d5ffb89ea94e?uid=blte449d5ffb89ea94e" data-sys-asset-uid="bltd7d31dd0108e9ed6" alt="Headshot_Suny.png" width="106" height="142" style="width: 106;height: 142;float: left;margin-top: 0px;margin-right: 0px;margin-bottom: 0px;margin-left: 0px;text-align: left;"/>Sun-Tsung Kim started working with Elasticsearch end of 2012 at Autoscout24 and has been working with it ever since. She has a background in statistics, programming, OS and database administration, various data stores like meetups or coursera. Additionally Sun-Tsung is interested in new ways to work and organise.

thumbnail-teradata-dashboard.png

Elastic stack for Root Cause Analysis at Mapp

Start free trial

Blog Footer CTA

Sign up for Elastic Cloud free trial

Blog Footer

Sun-Tsung Kim

Can't make it? Register and we'll send you the recording. You'll also receive an email with related content

Can't make it? Register and we'll send you the recording. You'll also receive an email with related content.

You'll also receive an email with related content.

You'll also receive an email with related content

Load More

By submitting you acknowledge that you've read and agree to our <a href="/legal/elastic-cloud-account-terms" target="_blank">Terms of Service</a>, and that Elastic may <a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">contact you</a> about our related products and services, using the details you provide above. See <a href="/legal/privacy-statement/" target="_blank">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.

Newsletter GDPR Text

Speakers

Close

See more insights

The content on this page is not available in the selected language. As Elastic grows globally, we continue to support content in multiple languages.

Elasticsearch Platform

ELK Stack

Elastic Cloud

Observability

Security

Search

By industry

By solution

Customer spotlight

Developers

Connect

Learn

Help

See what's happening at Elastic

Author

Articles by Sun-Tsung Kim

Elastic stack for Root Cause Analysis at Mapp

Follow us

About us

Join us

Press

Partners

Trust & Security

Investor relations

EXCELLENCE AWARDS