1&1 Internet SE explains how they made log file analysis faster, more accurate and without grep commands using the Elastic Stack.

We are an Operations team at <a href="http://www.1und1.de" target="_blank">1&1 Internet SE</a> working on executable and measurable BPMN processes (Business Process Model and Notation) that fulfils orders for all hosting products. An order&nbsp;process starts when the customer approves his order with a click on the “Confirmation”&nbsp;button at the 1&1 shop. As part of the process the customer details are being checked, the contract created, all technical features are provisioned and the customer communication is taken care of.
In the past, log file analysis was time consuming and only possible with single customer orders. With the importance of log files in mind we were searching for a solution to analyze our log files faster, more accurate and without grep commands. Using Elasticsearch, Logstash and Kibana&nbsp;was an experiment at the beginning but meanwhile it turned into a tool that determines and optimizes our everyday work very fast.
<h2>What operating drives</h2>So what do we operate? We are working on 10 BPMN processes with approx. 50,000 process instances a week. The logs generated by these processes are about 50GB a day. The log files contain technical information as well as business information. The Elastic Stack gives us the ability to analyze these logs in real-time and provides transparency on customer orders.
As an operations team we are driven by errors and incidents that attack customer orders. Our primary goal is customer satisfaction which we are trying to maintain by observing process SLAs like “in time provisioning” of customer orders. For being meaningful about the health of our processes we needed an overview as well as a categorization of errors in our processes. The questions we needed to answer were:
<ul>
	<li>What errors occur in our processes?</li>
	<li>Is an error process specific or is it a common problem?</li>
	<li>What is the total number of errors and how did it evolve?</li>
	<li>Are these errors known and is anybody working on it?</li>
	<li>Are these errors known and is there a known work-around?</li>
</ul><h2>Extending the Elastic Stack</h2>Not all of the information can be taken from our process logs. To completely answer our questions we needed to enrich our application logs with information from other sources like issue tracker, databases, etc.
The data sources in detail are:&nbsp;
<ul>
	<li>Error pattern configuration (knowledge database, manually added)&nbsp;</li>
	<li>Process information (productive process information from different databases)</li>
</ul>For conditioning this information we use the ETL tool Pentaho Kettle.
<img src="https://api.contentstack.io/v2/assets/575e4d360342dfd738264eb4/download?uid=blta5e0c5c910beaaae?uid=blta5e0c5c910beaaae" data-sys-asset-uid="blt17470d1b47b0a047" alt="conditioning-and-loading-elasticsearch-schema-new.png" style="max-width:100%;">
Image 1: Process for conditioning and loading data to Elasticsearch
<h2>Dashboard as a tool</h2>Our Kibana dashboard helps us working with our processes every day.
<img src="https://api.contentstack.io/v2/assets/575e4d360342dfd738264e9a/download?uid=blt678adcbebb6f705f?uid=blt678adcbebb6f705f" data-sys-asset-uid="bltdef55caa922df240" alt="kibana-dashboard-overview-11-internet.png" style="max-width:100%;">
Image 2: Kibana dashboard - Overview
(1) Number of errors by process (pie and table)
	 (2) Number of errors by issue tool ticket (pie and table)
	 (3) Number of errors by description text (pie and table)&nbsp; (4) Full list of errors
<img src="https://api.contentstack.io/v2/assets/575e4d3612798d8e381ff231/download?uid=blt9a12745a50241220?uid=blt9a12745a50241220" data-sys-asset-uid="blt2f8f314893c4109b" alt="kibana-dashboard-linking-11-internet.png" style="max-width:100%;">
Image 3: Kibana dashboard - Linking
In order to make operations more efficient, the dashboard integrates other tools for example linking to related bugs in our issue tracker.
In detail the dashboard provides:
(1) Link to issue tracker
	 (2) Link to central support tool showing detailed contract and customer information
	 (3) Link to Kibana dashboard showing all log information concerning that order (for drilldown) (4) Link to information tool showing detailed order and customer information
<h2>Outcome</h2>We are now using the Elastic&nbsp;Stack for 2 months. The outcome is significant.
More focus on what is better now e.g. increased error detection and addressing rate by 30 to 90 percent.&nbsp;
<ol>
	<li>Faster error detection&nbsp;</li>
	<li>More than 90 percent of our errors are addressed (coming from about 60 percent)</li>
	<li>Manual reporting is obsolete because dashboard is visible for everyone&nbsp;</li>
	<li>One central “tool/dashboard” that connects operating tooling with each other&nbsp;</li>
	<li>Visibility of errors across all processes (not just process specific)&nbsp;</li>
	<li>The Elastic Stack is also used by non-technical people designing dashboards</li>
</ol><h2>In the future…</h2>…we will concentrate on using the Elastic Stack for real-time monitoring purposes. For example a visualization of order content will help us to cluster our orders for estimation of campaigns (e.g. new product release). Therefore we need to deal with Apache SPARK for data processing and computations.&nbsp;Spark will calculate different KPIs (Key Performance Indicators) for us e.g. the duration of different process periods.<hr><img src="https://api.contentstack.io/v2/assets/575e4d3658208ba076e29cf7/download?uid=blt5c228c2bc29175c3?uid=blt5c228c2bc29175c3" data-sys-asset-uid="blt412e368a169984d2" alt="BenjaminSpeckmann.jpg" style="line-height: 1.6em; float: left; margin: 0px;">
Benjamin Speckmann is Business Process Manager at 1&1 Internet SE. Previously he was working as a consultant at NTT DATA Deutschland GmbH.
Since the beginning of his career Benjamin deals with executable BPMN processes (Business Process Model and Notation). Monitoring, Data-Analysis and Reporting of KPI’s was always a part of his work.
He holds a M.S. in Computer Science from Eastern Michigan University as well as the University of Applied Sciences in Karlsruhe.&nbsp;
 
<img src="https://api.contentstack.io/v2/assets/575e4d3612798d8e381ff23a/download?uid=blta4c46fc629c3c9cb?uid=blta4c46fc629c3c9cb" data-sys-asset-uid="blt312da0dd4b38d769" alt="ChristianHatz.jpg" width="137" height="137" style="line-height: 1.6em; width: 137px; height: 137px; float: left; margin: 0px;">
Christian Hatz is Advanced System Analyst at 1&1 Internet SE.
Previously Christian was working as a&nbsp;Requirements Engineer delivering business concepts for offshore developed business process monitoring solutions.
He is experienced in JBoss EAP and BPM dominated infrastructure. In 2015 Christian implemented the Elastic Stack for efficient monitoring and analysis of technical and business data in Order Management.

kibana-dashboard-linking-11-fuul-bleed copy.jpg

11-logo.png

Process transparency and error categorization at 1&1 by using the Elastic Stack and ETL

Start free trial

Blog Footer CTA

Sign up for Elastic Cloud free trial

Christian Hatz

By submitting you acknowledge that you've read and agree to our <a href="/legal/elastic-cloud-account-terms" target="_self">Terms of Service</a>, and you agree to receive the <a href="/legal/privacy-statement#how-we-use-the-information" target="_self">selected communications</a> at the contact details you provided above. See <a href="/legal/privacy-statement/" target="_self">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.

Communication Preferences - ONLY for /communication-preferences

En confirmant votre envoi, vous reconnaissez que vous avez lu et que vous acceptez nos <a href="/fr/legal/elastic-cloud-account-terms" target="_self">Conditions générales d'utilisation</a>, et vous acceptez de recevoir les <a href="/fr/legal/privacy-statement#how-we-use-the-information" target="_self">types de notification sélectionnés</a> à l'adresse que vous avez fournie plus haut. Pour en savoir plus ou pour revenir sur votre décision (possible à tout moment), consultez la <a href="/fr/legal/privacy-statement/" target="_self">Déclaration de confidentialité d'Elastic</a>.

By submitting you acknowledge that you've read and agree to our <a href="/legal/elastic-cloud-account-terms" target="_blank">Terms of Service</a>, and that Elastic may <a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">contact you</a> about our related products and services, using the details you provide above. See <a href="/legal/privacy-statement/" target="_blank">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.By submitting this form, I agree to Elastic sharing my contact information with Google Cloud. I also agree to Elastic giving my information to the vendor for the purpose of shipping my food package.

GDPR Text + Google Cloud

By submitting you acknowledge that you've read and agree to our <a href="/legal/serverless-preview-terms" target="_blank">Terms of Service</a>, and that Elastic may <a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">contact you</a> about our related products and services, using the details you provide above. See <a href="/legal/privacy-statement/" target="_blank">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.

Serverless GDPR Text

En confirmant votre envoi, vous reconnaissez que vous avez lu et que vous acceptez nos <a href="/fr/legal/serverless-preview-terms" target="_self">Conditions générales d'utilisation</a>, et vous permettez à Elastic de <a href="/fr/legal/privacy-statement#how-we-use-the-information" target="_self">vous contacter</a> à propos de produits et services similaires en utilisant les informations que vous avez fournies plus haut. Pour en savoir plus ou pour revenir sur votre décision (possible à tout moment), consultez la <a href="/fr/legal/privacy-statement/" target="_self">Déclaration de confidentialité d'Elastic</a>.

By submitting you agree to <a href="/agreements/global/cloud-services-monthly" rel="nofollow">Elastic Cloud Terms of Service</a>. Your personal data will be processed in accordance with <a href="/legal/privacy-statement">Elastic's Privacy Statement</a>.

Cloud Trial GDPR Text

En envoyant votre demande d'inscription, vous acceptez les <a href="/fr/agreements/global/cloud-services-monthly" rel="nofollow">conditions générales d'utilisation d'Elastic Cloud</a>. Vos données personnelles seront traitées conformément à la <a href="/fr/legal/privacy-statement">déclaration de confidentialité d'Elastic</a>.

By submitting you acknowledge that you've read and agree to our <a href="/legal/terms-of-use" target="_blank">Terms of Service</a>, and that Elastic may <a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">contact you</a> about our related products and services, using the details you provide above. See <a href="/legal/privacy-statement/" target="_blank">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.

Newsletter GDPR Text

En confirmant votre envoi, vous reconnaissez que vous avez lu et que vous acceptez nos <a href="/fr/legal/terms-of-use" target="_self">Conditions générales d'utilisation</a>, et vous permettez à Elastic de <a href="/fr/legal/privacy-statement#how-we-use-the-information" target="_self">vous contacter</a> à propos de produits et services similaires en utilisant les informations que vous avez fournies plus haut. Pour en savoir plus ou pour revenir sur votre décision (possible à tout moment), consultez la <a href="/fr/legal/privacy-statement/" target="_self">Déclaration de confidentialité d'Elastic</a>.

Explore similar demos

Overview

You'll also receive an email with related content.

You'll also receive an email with related content

Nous vous enverrons plus de contenu susceptible de vous intéresser.

Speakers

Close

The content on this page is not available in the selected language. As Elastic grows globally, we continue to support content in multiple languages.

The content on this page is not available in the selected language.

Le contenu de cette page n'est pas disponible dans la langue sélectionnée. Chez Elastic, nous mettons tout en œuvre pour vous proposer du contenu dans différentes langues. En attendant, nous vous remercions de votre patience !

See more insights

Articles by

Author

Learn more

Watch now (no PT)

Watch now

See all top stories

All (no PT translation)

Contact information

Press Release

Share on Reddit

Share this story

Share by Email

Thank you for your interest!

Contact Info

Follow us on Youtube

Load More

Share on LinkedIn

Follow us on LinkedIn

Search Integrations

All Solutions

Video for

Follow us on Twitter

See when this webinar starts in my time zone

Voir à quelle heure débute le webinar dans mon fuseau horaire

Register to Watch

Register now

See All Posts

Can't make it? Register and we'll send you the recording. You'll also receive an email with related content

Si vous ne pouvez pas vous libérer le jour J, inscrivez-vous et nous vous enverrons l'enregistrement. Vous recevrez aussi un e-mail contenant des ressources connexes.

Small image for

On-demand webinar

Featured webinar

Register to Attend

Share on Facebook

Reset all

Upcoming webinar

View more posts

Print

Hosted by

Sign In to Attend

View next

Follow us on Facebook

Share

Highlights

Read less

Thank you for registering. We will send you a confirmation email soon.

Nous vous remercions de votre inscription. Vous allez recevoir un e-mail de confirmation.

Author

Articles by Christian Hatz

Process transparency and error categorization at 1&1 by using the Elastic Stack and ETL

Sign up for Elastic Cloud free trial