1&1 Internet SE explains how they made log file analysis faster, more accurate and without grep commands using the Elastic Stack.

We are an Operations team at <a href="http://www.1und1.de" target="_blank">1&1 Internet SE</a> working on executable and measurable BPMN processes (Business Process Model and Notation) that fulfils orders for all hosting products. An order&nbsp;process starts when the customer approves his order with a click on the “Confirmation”&nbsp;button at the 1&1 shop. As part of the process the customer details are being checked, the contract created, all technical features are provisioned and the customer communication is taken care of.
In the past, log file analysis was time consuming and only possible with single customer orders. With the importance of log files in mind we were searching for a solution to analyze our log files faster, more accurate and without grep commands. Using Elasticsearch, Logstash and Kibana&nbsp;was an experiment at the beginning but meanwhile it turned into a tool that determines and optimizes our everyday work very fast.
<h2>What operating drives</h2>So what do we operate? We are working on 10 BPMN processes with approx. 50,000 process instances a week. The logs generated by these processes are about 50GB a day. The log files contain technical information as well as business information. The Elastic Stack gives us the ability to analyze these logs in real-time and provides transparency on customer orders.
As an operations team we are driven by errors and incidents that attack customer orders. Our primary goal is customer satisfaction which we are trying to maintain by observing process SLAs like “in time provisioning” of customer orders. For being meaningful about the health of our processes we needed an overview as well as a categorization of errors in our processes. The questions we needed to answer were:
<ul>
	<li>What errors occur in our processes?</li>
	<li>Is an error process specific or is it a common problem?</li>
	<li>What is the total number of errors and how did it evolve?</li>
	<li>Are these errors known and is anybody working on it?</li>
	<li>Are these errors known and is there a known work-around?</li>
</ul><h2>Extending the Elastic Stack</h2>Not all of the information can be taken from our process logs. To completely answer our questions we needed to enrich our application logs with information from other sources like issue tracker, databases, etc.
The data sources in detail are:&nbsp;
<ul>
	<li>Error pattern configuration (knowledge database, manually added)&nbsp;</li>
	<li>Process information (productive process information from different databases)</li>
</ul>For conditioning this information we use the ETL tool Pentaho Kettle.
<img src="https://api.contentstack.io/v2/assets/575e4d360342dfd738264eb4/download?uid=blta5e0c5c910beaaae?uid=blta5e0c5c910beaaae" data-sys-asset-uid="blt17470d1b47b0a047" alt="conditioning-and-loading-elasticsearch-schema-new.png" style="max-width:100%;">
Image 1: Process for conditioning and loading data to Elasticsearch
<h2>Dashboard as a tool</h2>Our Kibana dashboard helps us working with our processes every day.
<img src="https://api.contentstack.io/v2/assets/575e4d360342dfd738264e9a/download?uid=blt678adcbebb6f705f?uid=blt678adcbebb6f705f" data-sys-asset-uid="bltdef55caa922df240" alt="kibana-dashboard-overview-11-internet.png" style="max-width:100%;">
Image 2: Kibana dashboard - Overview
(1) Number of errors by process (pie and table)
	 (2) Number of errors by issue tool ticket (pie and table)
	 (3) Number of errors by description text (pie and table)&nbsp; (4) Full list of errors
<img src="https://api.contentstack.io/v2/assets/575e4d3612798d8e381ff231/download?uid=blt9a12745a50241220?uid=blt9a12745a50241220" data-sys-asset-uid="blt2f8f314893c4109b" alt="kibana-dashboard-linking-11-internet.png" style="max-width:100%;">
Image 3: Kibana dashboard - Linking
In order to make operations more efficient, the dashboard integrates other tools for example linking to related bugs in our issue tracker.
In detail the dashboard provides:
(1) Link to issue tracker
	 (2) Link to central support tool showing detailed contract and customer information
	 (3) Link to Kibana dashboard showing all log information concerning that order (for drilldown) (4) Link to information tool showing detailed order and customer information
<h2>Outcome</h2>We are now using the Elastic&nbsp;Stack for 2 months. The outcome is significant.
More focus on what is better now e.g. increased error detection and addressing rate by 30 to 90 percent.&nbsp;
<ol>
	<li>Faster error detection&nbsp;</li>
	<li>More than 90 percent of our errors are addressed (coming from about 60 percent)</li>
	<li>Manual reporting is obsolete because dashboard is visible for everyone&nbsp;</li>
	<li>One central “tool/dashboard” that connects operating tooling with each other&nbsp;</li>
	<li>Visibility of errors across all processes (not just process specific)&nbsp;</li>
	<li>The Elastic Stack is also used by non-technical people designing dashboards</li>
</ol><h2>In the future…</h2>…we will concentrate on using the Elastic Stack for real-time monitoring purposes. For example a visualization of order content will help us to cluster our orders for estimation of campaigns (e.g. new product release). Therefore we need to deal with Apache SPARK for data processing and computations.&nbsp;Spark will calculate different KPIs (Key Performance Indicators) for us e.g. the duration of different process periods.<hr><img src="https://api.contentstack.io/v2/assets/575e4d3658208ba076e29cf7/download?uid=blt5c228c2bc29175c3?uid=blt5c228c2bc29175c3" data-sys-asset-uid="blt412e368a169984d2" alt="BenjaminSpeckmann.jpg" style="line-height: 1.6em; float: left; margin: 0px;">
Benjamin Speckmann is Business Process Manager at 1&1 Internet SE. Previously he was working as a consultant at NTT DATA Deutschland GmbH.
Since the beginning of his career Benjamin deals with executable BPMN processes (Business Process Model and Notation). Monitoring, Data-Analysis and Reporting of KPI’s was always a part of his work.
He holds a M.S. in Computer Science from Eastern Michigan University as well as the University of Applied Sciences in Karlsruhe.&nbsp;
 
<img src="https://api.contentstack.io/v2/assets/575e4d3612798d8e381ff23a/download?uid=blta4c46fc629c3c9cb?uid=blta4c46fc629c3c9cb" data-sys-asset-uid="blt312da0dd4b38d769" alt="ChristianHatz.jpg" width="137" height="137" style="line-height: 1.6em; width: 137px; height: 137px; float: left; margin: 0px;">
Christian Hatz is Advanced System Analyst at 1&1 Internet SE.
Previously Christian was working as a&nbsp;Requirements Engineer delivering business concepts for offshore developed business process monitoring solutions.
He is experienced in JBoss EAP and BPM dominated infrastructure. In 2015 Christian implemented the Elastic Stack for efficient monitoring and analysis of technical and business data in Order Management.

kibana-dashboard-linking-11-fuul-bleed copy.jpg

11-logo.png

Process transparency and error categorization at 1&1 by using the Elastic Stack and ETL

Start free trial

Blog Footer CTA

Sign up for Elastic Cloud free trial

Christian Hatz

By submitting you acknowledge that you've read and agree to our <a href="/legal/elastic-cloud-account-terms" target="_self">Terms of Service</a>, and you agree to receive the <a href="/legal/privacy-statement#how-we-use-the-information" target="_self">selected communications</a> at the contact details you provided above. See <a href="/legal/privacy-statement/" target="_self">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.

Communication Preferences - ONLY for /communication-preferences

提交即表示您确认您已阅读并同意我们的<a href="/cn/legal/elastic-cloud-account-terms" target="_self">服务条款</a>，并且您同意通过您提供的上述联系方式接收<a href="/cn/legal/privacy-statement#how-we-use-the-information" target="_self">所选的通信</a>。如需了解更多信息，或想随时取消订阅，请参阅 <a href="/cn/legal/privacy-statement/" target="_self">Elastic 的隐私声明</a>。

By submitting you acknowledge that you've read and agree to our <a href="/legal/serverless-preview-terms" target="_blank">Terms of Service</a>, and that Elastic may <a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">contact you</a> about our related products and services, using the details you provide above. See <a href="/legal/privacy-statement/" target="_blank">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.

Serverless GDPR Text

提交即表示您确认您已阅读并同意我们的<a href="/cn/legal/serverless-preview-terms" target="_self">服务条款</a>，并且 Elastic 可以使用您提供的上述详细信息与您联系，以向您介绍我们相关的<a href="/cn/legal/privacy-statement#how-we-use-the-information" target="_self">产品和服务</a>。如需了解更多信息，或想随时取消订阅，请参阅 <a href="/cn/legal/privacy-statement/" target="_self">Elastic 的隐私声明</a>。

By submitting you agree to <a href="/agreements/global/cloud-services-monthly" rel="nofollow">Elastic Cloud Terms of Service</a>. Your personal data will be processed in accordance with <a href="/legal/privacy-statement">Elastic's Privacy Statement</a>.

Cloud Trial GDPR Text

提交即表示您同意 <a href="/cn/agreements/global/cloud-services-monthly" rel="nofollow">Elastic Cloud 服务条款</a>。我们将会按照 <a href="/cn/legal/privacy-statement">Elastic 的隐私声明</a>处理您的个人数据。

By submitting you acknowledge that you've read and agree to our <a href="/legal/terms-of-use" target="_blank">Terms of Service</a>, and that Elastic may <a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">contact you</a> about our related products and services, using the details you provide above. See <a href="/legal/privacy-statement/" target="_blank">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.

Newsletter GDPR Text

提交即表示您确认您已阅读并同意我们的<a href="/cn/legal/terms-of-use" target="_self">服务条款</a>，并且 Elastic 可以使用您提供的上述详细信息与您联系，以向您介绍我们相关的<a href="/cn/legal/privacy-statement#how-we-use-the-information" target="_self">产品和服务</a>。如需了解更多信息，或想随时取消订阅，请参阅 <a href="/cn/legal/privacy-statement/" target="_self">Elastic 的隐私声明</a>。

Explore similar demos

Overview

Speakers

Close

See more insights

The content on this page is not available in the selected language. As Elastic grows globally, we continue to support content in multiple languages.

The content on this page is not available in the selected language.

本页内容尚不支持所选语言。Elastic 正在不断努力，以实现对多种语言内容的支持。感谢您在此期间给予的耐心与陪伴！

Author

Articles by

Learn more

Watch now (no PT)

Watch now

See all top stories

All (no PT translation)

Contact information

Press Release

Share on Reddit

Share this story

Share by Email

Thank you for your interest!

Contact Info

Follow us on Youtube

Load More

Share on LinkedIn

Follow us on LinkedIn

Search Integrations

All Solutions

Video for

Follow us on Twitter

See when this webinar starts in my time zone

查看本次网络研讨会在我的时区的开始时间

Register to Watch

Register now

See All Posts

Can't make it? Register and we'll send you the recording. You'll also receive an email with related content

无法实时参加？注册后我们会给您发送录像。您还将收到相关内容的电子邮件。

Author

Articles by Christian Hatz

Process transparency and error categorization at 1&1 by using the Elastic Stack and ETL

Sign up for Elastic Cloud free trial