1&1 Internet SE explains how they made log file analysis faster, more accurate and without grep commands using the Elastic Stack.

We are an Operations team at <a href="http://www.1und1.de" target="_blank">1&1 Internet SE</a> working on executable and measurable BPMN processes (Business Process Model and Notation) that fulfils orders for all hosting products. An order&nbsp;process starts when the customer approves his order with a click on the “Confirmation”&nbsp;button at the 1&1 shop. As part of the process the customer details are being checked, the contract created, all technical features are provisioned and the customer communication is taken care of.
In the past, log file analysis was time consuming and only possible with single customer orders. With the importance of log files in mind we were searching for a solution to analyze our log files faster, more accurate and without grep commands. Using Elasticsearch, Logstash and Kibana&nbsp;was an experiment at the beginning but meanwhile it turned into a tool that determines and optimizes our everyday work very fast.
<h2>What operating drives</h2>So what do we operate? We are working on 10 BPMN processes with approx. 50,000 process instances a week. The logs generated by these processes are about 50GB a day. The log files contain technical information as well as business information. The Elastic Stack gives us the ability to analyze these logs in real-time and provides transparency on customer orders.
As an operations team we are driven by errors and incidents that attack customer orders. Our primary goal is customer satisfaction which we are trying to maintain by observing process SLAs like “in time provisioning” of customer orders. For being meaningful about the health of our processes we needed an overview as well as a categorization of errors in our processes. The questions we needed to answer were:
<ul>
	<li>What errors occur in our processes?</li>
	<li>Is an error process specific or is it a common problem?</li>
	<li>What is the total number of errors and how did it evolve?</li>
	<li>Are these errors known and is anybody working on it?</li>
	<li>Are these errors known and is there a known work-around?</li>
</ul><h2>Extending the Elastic Stack</h2>Not all of the information can be taken from our process logs. To completely answer our questions we needed to enrich our application logs with information from other sources like issue tracker, databases, etc.
The data sources in detail are:&nbsp;
<ul>
	<li>Error pattern configuration (knowledge database, manually added)&nbsp;</li>
	<li>Process information (productive process information from different databases)</li>
</ul>For conditioning this information we use the ETL tool Pentaho Kettle.
<img src="https://api.contentstack.io/v2/assets/575e4d360342dfd738264eb4/download?uid=blta5e0c5c910beaaae?uid=blta5e0c5c910beaaae" data-sys-asset-uid="blt17470d1b47b0a047" alt="conditioning-and-loading-elasticsearch-schema-new.png" style="max-width:100%;">
Image 1: Process for conditioning and loading data to Elasticsearch
<h2>Dashboard as a tool</h2>Our Kibana dashboard helps us working with our processes every day.
<img src="https://api.contentstack.io/v2/assets/575e4d360342dfd738264e9a/download?uid=blt678adcbebb6f705f?uid=blt678adcbebb6f705f" data-sys-asset-uid="bltdef55caa922df240" alt="kibana-dashboard-overview-11-internet.png" style="max-width:100%;">
Image 2: Kibana dashboard - Overview
(1) Number of errors by process (pie and table)
	 (2) Number of errors by issue tool ticket (pie and table)
	 (3) Number of errors by description text (pie and table)&nbsp; (4) Full list of errors
<img src="https://api.contentstack.io/v2/assets/575e4d3612798d8e381ff231/download?uid=blt9a12745a50241220?uid=blt9a12745a50241220" data-sys-asset-uid="blt2f8f314893c4109b" alt="kibana-dashboard-linking-11-internet.png" style="max-width:100%;">
Image 3: Kibana dashboard - Linking
In order to make operations more efficient, the dashboard integrates other tools for example linking to related bugs in our issue tracker.
In detail the dashboard provides:
(1) Link to issue tracker
	 (2) Link to central support tool showing detailed contract and customer information
	 (3) Link to Kibana dashboard showing all log information concerning that order (for drilldown) (4) Link to information tool showing detailed order and customer information
<h2>Outcome</h2>We are now using the Elastic&nbsp;Stack for 2 months. The outcome is significant.
More focus on what is better now e.g. increased error detection and addressing rate by 30 to 90 percent.&nbsp;
<ol>
	<li>Faster error detection&nbsp;</li>
	<li>More than 90 percent of our errors are addressed (coming from about 60 percent)</li>
	<li>Manual reporting is obsolete because dashboard is visible for everyone&nbsp;</li>
	<li>One central “tool/dashboard” that connects operating tooling with each other&nbsp;</li>
	<li>Visibility of errors across all processes (not just process specific)&nbsp;</li>
	<li>The Elastic Stack is also used by non-technical people designing dashboards</li>
</ol><h2>In the future…</h2>…we will concentrate on using the Elastic Stack for real-time monitoring purposes. For example a visualization of order content will help us to cluster our orders for estimation of campaigns (e.g. new product release). Therefore we need to deal with Apache SPARK for data processing and computations.&nbsp;Spark will calculate different KPIs (Key Performance Indicators) for us e.g. the duration of different process periods.<hr><img src="https://api.contentstack.io/v2/assets/575e4d3658208ba076e29cf7/download?uid=blt5c228c2bc29175c3?uid=blt5c228c2bc29175c3" data-sys-asset-uid="blt412e368a169984d2" alt="BenjaminSpeckmann.jpg" style="line-height: 1.6em; float: left; margin: 0px;">
Benjamin Speckmann is Business Process Manager at 1&1 Internet SE. Previously he was working as a consultant at NTT DATA Deutschland GmbH.
Since the beginning of his career Benjamin deals with executable BPMN processes (Business Process Model and Notation). Monitoring, Data-Analysis and Reporting of KPI’s was always a part of his work.
He holds a M.S. in Computer Science from Eastern Michigan University as well as the University of Applied Sciences in Karlsruhe.&nbsp;
 
<img src="https://api.contentstack.io/v2/assets/575e4d3612798d8e381ff23a/download?uid=blta4c46fc629c3c9cb?uid=blta4c46fc629c3c9cb" data-sys-asset-uid="blt312da0dd4b38d769" alt="ChristianHatz.jpg" width="137" height="137" style="line-height: 1.6em; width: 137px; height: 137px; float: left; margin: 0px;">
Christian Hatz is Advanced System Analyst at 1&1 Internet SE.
Previously Christian was working as a&nbsp;Requirements Engineer delivering business concepts for offshore developed business process monitoring solutions.
He is experienced in JBoss EAP and BPM dominated infrastructure. In 2015 Christian implemented the Elastic Stack for efficient monitoring and analysis of technical and business data in Order Management.

kibana-dashboard-linking-11-fuul-bleed copy.jpg

11-logo.png

Process transparency and error categorization at 1&1 by using the Elastic Stack and ETL

Start free trial

Blog Footer CTA

Sign up for Elastic Cloud free trial

Christian Hatz

By submitting you acknowledge that you've read and agree to our <a href="/legal/elastic-cloud-account-terms" target="_self">Terms of Service</a>, and you agree to receive the <a href="/legal/privacy-statement#how-we-use-the-information" target="_self">selected communications</a> at the contact details you provided above. See <a href="/legal/privacy-statement/" target="_self">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.

Communication Preferences - ONLY for /communication-preferences

제출하면 귀하가 <a href="/kr/legal/elastic-cloud-account-terms" target="_self">Elastic 서비스 약관</a>을 읽고 이에 동의하며, 위에 제공된 연락처로 <a href="/kr/legal/privacy-statement#how-we-use-the-information" target="_self">선택한 커뮤니케이션</a>을 수신하는 데 동의한 것으로 간주됩니다. 언제든지 수신을 거부할 수 있으며 자세한 내용은 <a href="/kr/legal/privacy-statement/" target="_self">Elastic 개인정보 취급방침</a>을 참조하시기 바랍니다.

By submitting you acknowledge that you've read and agree to our <a href="/legal/serverless-preview-terms" target="_blank">Terms of Service</a>, and that Elastic may <a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">contact you</a> about our related products and services, using the details you provide above. See <a href="/legal/privacy-statement/" target="_blank">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.

Serverless GDPR Text

제출하면 귀하가 Elastic <a href="/kr/legal/serverless-preview-terms" target="_self">서비스 약관</a>을 읽고 이에 동의하며, 위에 제공된 세부 정보를 사용하여 Elastic이 관련 제품 및 서비스에 대해 귀하에게 <a href="/kr/legal/privacy-statement#how-we-use-the-information" target="_self">연락할 수 있음</a>을 이해한 것으로 간주됩니다. 언제든지 수신을 거부할 수 있으며 자세한 내용은 <a href="/kr/legal/privacy-statement/" target="_self">Elastic 개인정보 취급방침</a>을 참조하시기 바랍니다.

By submitting you agree to <a href="/agreements/global/cloud-services-monthly" rel="nofollow">Elastic Cloud Terms of Service</a>. Your personal data will be processed in accordance with <a href="/legal/privacy-statement">Elastic's Privacy Statement</a>.

Cloud Trial GDPR Text

제출하면 <a href="/kr/agreements/global/cloud-services-monthly" rel="nofollow">Elastic Cloud 서비스 약관</a>에 동의하시게 됩니다. 개인 데이터는 <a href="/kr/legal/privacy-statement">Elastic의 개인정보 취급방침</a>에 따라 처리됩니다.

By submitting you acknowledge that you've read and agree to our <a href="/legal/terms-of-use" target="_blank">Terms of Service</a>, and that Elastic may <a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">contact you</a> about our related products and services, using the details you provide above. See <a href="/legal/privacy-statement/" target="_blank">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.

Newsletter GDPR Text

제출하면 귀하가 Elastic <a href="/kr/legal/terms-of-use" target="_self">서비스 약관</a>을 읽고 이에 동의하며, 위에 제공된 세부 정보를 사용하여 Elastic이 관련 제품 및 서비스에 대해 귀하에게 <a href="/kr/legal/privacy-statement#how-we-use-the-information" target="_self">연락할 수 있음</a>을 이해한 것으로 간주됩니다. 언제든지 수신을 거부할 수 있으며 자세한 내용은 <a href="/kr/legal/privacy-statement/" target="_self">Elastic 개인정보 취급방침</a>을 참조하시기 바랍니다.

Explore similar demos

Overview

Speakers

Close

See more insights

The content on this page is not available in the selected language. As Elastic grows globally, we continue to support content in multiple languages.

The content on this page is not available in the selected language.

이 페이지의 콘텐츠는 선택하신 언어로 제공되지 않습니다. Elastic은 다양한 언어로 콘텐츠를 제공하기 위해 최선을 다하고 있습니다.조금만 더 기다려주세요!

Author

Articles by

Learn more

Watch now (no PT)

Watch now

See all top stories

All (no PT translation)

Contact information

Press Release

Share on Reddit

Share this story

Share by Email

Thank you for your interest!

Contact Info

Follow us on Youtube

Load More

Share on LinkedIn

Follow us on LinkedIn

Search Integrations

All Solutions

Video for

Follow us on Twitter

See when this webinar starts in my time zone

내 표준 시간대로 웨비나 시작 시간 확인하기

Register to Watch

Register now

See All Posts

Can't make it? Register and we'll send you the recording. You'll also receive an email with related content

참여가 어려우신가요? 등록하시면 녹화 자료를 보내드리겠습니다. 이메일로 관련 콘텐츠도 받아보실 수 있습니다.

Small image for

On-demand webinar

Featured webinar

Register to Attend

Share on Facebook

Reset all

Upcoming webinar

View more posts

Print

Hosted by

Sign In to Attend

View next

Follow us on Facebook

Share

Highlights

Read less

You'll also receive an email with related content

이메일을 통해 관련 내용을 보내드리겠습니다.

Thank you for registering. We will send you a confirmation email soon.

등록해 주셔서 감사합니다. 곧 확인 이메일을 보내드리겠습니다.

Author

Articles by Christian Hatz

Process transparency and error categorization at 1&1 by using the Elastic Stack and ETL

Sign up for Elastic Cloud free trial