1&1 Internet SE explains how they made log file analysis faster, more accurate and without grep commands using the Elastic Stack.

We are an Operations team at <a href="http://www.1und1.de" target="_blank">1&1 Internet SE</a> working on executable and measurable BPMN processes (Business Process Model and Notation) that fulfils orders for all hosting products. An order&nbsp;process starts when the customer approves his order with a click on the “Confirmation”&nbsp;button at the 1&1 shop. As part of the process the customer details are being checked, the contract created, all technical features are provisioned and the customer communication is taken care of.
In the past, log file analysis was time consuming and only possible with single customer orders. With the importance of log files in mind we were searching for a solution to analyze our log files faster, more accurate and without grep commands. Using Elasticsearch, Logstash and Kibana&nbsp;was an experiment at the beginning but meanwhile it turned into a tool that determines and optimizes our everyday work very fast.
<h2>What operating drives</h2>So what do we operate? We are working on 10 BPMN processes with approx. 50,000 process instances a week. The logs generated by these processes are about 50GB a day. The log files contain technical information as well as business information. The Elastic Stack gives us the ability to analyze these logs in real-time and provides transparency on customer orders.
As an operations team we are driven by errors and incidents that attack customer orders. Our primary goal is customer satisfaction which we are trying to maintain by observing process SLAs like “in time provisioning” of customer orders. For being meaningful about the health of our processes we needed an overview as well as a categorization of errors in our processes. The questions we needed to answer were:
<ul>
	<li>What errors occur in our processes?</li>
	<li>Is an error process specific or is it a common problem?</li>
	<li>What is the total number of errors and how did it evolve?</li>
	<li>Are these errors known and is anybody working on it?</li>
	<li>Are these errors known and is there a known work-around?</li>
</ul><h2>Extending the Elastic Stack</h2>Not all of the information can be taken from our process logs. To completely answer our questions we needed to enrich our application logs with information from other sources like issue tracker, databases, etc.
The data sources in detail are:&nbsp;
<ul>
	<li>Error pattern configuration (knowledge database, manually added)&nbsp;</li>
	<li>Process information (productive process information from different databases)</li>
</ul>For conditioning this information we use the ETL tool Pentaho Kettle.
<img src="https://api.contentstack.io/v2/assets/575e4d360342dfd738264eb4/download?uid=blta5e0c5c910beaaae?uid=blta5e0c5c910beaaae" data-sys-asset-uid="blt17470d1b47b0a047" alt="conditioning-and-loading-elasticsearch-schema-new.png" style="max-width:100%;">
Image 1: Process for conditioning and loading data to Elasticsearch
<h2>Dashboard as a tool</h2>Our Kibana dashboard helps us working with our processes every day.
<img src="https://api.contentstack.io/v2/assets/575e4d360342dfd738264e9a/download?uid=blt678adcbebb6f705f?uid=blt678adcbebb6f705f" data-sys-asset-uid="bltdef55caa922df240" alt="kibana-dashboard-overview-11-internet.png" style="max-width:100%;">
Image 2: Kibana dashboard - Overview
(1) Number of errors by process (pie and table)
	 (2) Number of errors by issue tool ticket (pie and table)
	 (3) Number of errors by description text (pie and table)&nbsp; (4) Full list of errors
<img src="https://api.contentstack.io/v2/assets/575e4d3612798d8e381ff231/download?uid=blt9a12745a50241220?uid=blt9a12745a50241220" data-sys-asset-uid="blt2f8f314893c4109b" alt="kibana-dashboard-linking-11-internet.png" style="max-width:100%;">
Image 3: Kibana dashboard - Linking
In order to make operations more efficient, the dashboard integrates other tools for example linking to related bugs in our issue tracker.
In detail the dashboard provides:
(1) Link to issue tracker
	 (2) Link to central support tool showing detailed contract and customer information
	 (3) Link to Kibana dashboard showing all log information concerning that order (for drilldown) (4) Link to information tool showing detailed order and customer information
<h2>Outcome</h2>We are now using the Elastic&nbsp;Stack for 2 months. The outcome is significant.
More focus on what is better now e.g. increased error detection and addressing rate by 30 to 90 percent.&nbsp;
<ol>
	<li>Faster error detection&nbsp;</li>
	<li>More than 90 percent of our errors are addressed (coming from about 60 percent)</li>
	<li>Manual reporting is obsolete because dashboard is visible for everyone&nbsp;</li>
	<li>One central “tool/dashboard” that connects operating tooling with each other&nbsp;</li>
	<li>Visibility of errors across all processes (not just process specific)&nbsp;</li>
	<li>The Elastic Stack is also used by non-technical people designing dashboards</li>
</ol><h2>In the future…</h2>…we will concentrate on using the Elastic Stack for real-time monitoring purposes. For example a visualization of order content will help us to cluster our orders for estimation of campaigns (e.g. new product release). Therefore we need to deal with Apache SPARK for data processing and computations.&nbsp;Spark will calculate different KPIs (Key Performance Indicators) for us e.g. the duration of different process periods.<hr><img src="https://api.contentstack.io/v2/assets/575e4d3658208ba076e29cf7/download?uid=blt5c228c2bc29175c3?uid=blt5c228c2bc29175c3" data-sys-asset-uid="blt412e368a169984d2" alt="BenjaminSpeckmann.jpg" style="line-height: 1.6em; float: left; margin: 0px;">
Benjamin Speckmann is Business Process Manager at 1&1 Internet SE. Previously he was working as a consultant at NTT DATA Deutschland GmbH.
Since the beginning of his career Benjamin deals with executable BPMN processes (Business Process Model and Notation). Monitoring, Data-Analysis and Reporting of KPI’s was always a part of his work.
He holds a M.S. in Computer Science from Eastern Michigan University as well as the University of Applied Sciences in Karlsruhe.&nbsp;
 
<img src="https://api.contentstack.io/v2/assets/575e4d3612798d8e381ff23a/download?uid=blta4c46fc629c3c9cb?uid=blta4c46fc629c3c9cb" data-sys-asset-uid="blt312da0dd4b38d769" alt="ChristianHatz.jpg" width="137" height="137" style="line-height: 1.6em; width: 137px; height: 137px; float: left; margin: 0px;">
Christian Hatz is Advanced System Analyst at 1&1 Internet SE.
Previously Christian was working as a&nbsp;Requirements Engineer delivering business concepts for offshore developed business process monitoring solutions.
He is experienced in JBoss EAP and BPM dominated infrastructure. In 2015 Christian implemented the Elastic Stack for efficient monitoring and analysis of technical and business data in Order Management.

kibana-dashboard-linking-11-fuul-bleed copy.jpg

11-logo.png

Process transparency and error categorization at 1&1 by using the Elastic Stack and ETL

Start free trial

Blog Footer CTA

Sign up for Elastic Cloud free trial

Christian Hatz

The content on this page is not available in the selected language. As Elastic grows globally, we continue to support content in multiple languages.

The content on this page is not available in the selected language.

O conteúdo desta página não está disponível no idioma selecionado. A Elastic está trabalhando para garantir que o conteúdo esteja disponível em vários idiomas. Agradecemos a compreensão.

By submitting you acknowledge that you've read and agree to our <a href="/legal/elastic-cloud-account-terms" target="_self">Terms of Service</a>, and you agree to receive the <a href="/legal/privacy-statement#how-we-use-the-information" target="_self">selected communications</a> at the contact details you provided above. See <a href="/legal/privacy-statement/" target="_self">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.

Communication Preferences - ONLY for /communication-preferences

Ao enviar a confirmação, você reconhece que leu e concorda com nossos <a href="/pt/legal/elastic-cloud-account-terms" target="_self">Termos de Serviço</a> e também concorda em receber <a href="/pt/legal/privacy-statement#how-we-use-the-information" target="_self">comunicações especiais</a> por meio dos dados de contato que forneceu acima. Consulte a <a href="/pt/legal/privacy-statement/" target="_self">Declaração de Privacidade da Elastic</a> para saber mais detalhes ou para cancelar a qualquer momento.

By submitting you acknowledge that you've read and agree to our <a href="/legal/serverless-preview-terms" target="_blank">Terms of Service</a>, and that Elastic may <a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">contact you</a> about our related products and services, using the details you provide above. See <a href="/legal/privacy-statement/" target="_blank">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.

Serverless GDPR Text

Ao enviar a confirmação, você reconhece que leu e concorda com nossos <a href="/pt/legal/serverless-preview-terms" target="_self">Termos de Serviço</a> e que a Elastic pode <a href="/pt/legal/privacy-statement#how-we-use-the-information" target="_self">entrar em contato com você</a> para divulgar nossos produtos e serviços relacionados usando os dados que você forneceu acima. Consulte a <a href="/pt/legal/privacy-statement/" target="_self">Declaração de Privacidade da Elastic</a> para saber mais detalhes ou para cancelar a qualquer momento.

By submitting you agree to <a href="/agreements/global/cloud-services-monthly" rel="nofollow">Elastic Cloud Terms of Service</a>. Your personal data will be processed in accordance with <a href="/legal/privacy-statement">Elastic's Privacy Statement</a>.

Cloud Trial GDPR Text

Ao enviar, você concorda com os <a href="/pt/agreements/global/cloud-services-monthly" rel="nofollow">Termos de Serviço do Elastic Cloud</a>. Seus dados pessoais serão tratados de acordo com a <a href="/pt/legal/privacy-statement">Declaração de Privacidade da Elastic</a>.

By submitting you acknowledge that you've read and agree to our <a href="/legal/terms-of-use" target="_blank">Terms of Service</a>, and that Elastic may <a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">contact you</a> about our related products and services, using the details you provide above. See <a href="/legal/privacy-statement/" target="_blank">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.

Newsletter GDPR Text

Ao enviar a confirmação, você reconhece que leu e concorda com nossos <a href="/pt/legal/terms-of-use" target="_self">Termos de Serviço</a> e que a Elastic pode <a href="/pt/legal/privacy-statement#how-we-use-the-information" target="_self">entrar em contato com você</a> para divulgar nossos produtos e serviços relacionados usando os dados que você forneceu acima. Consulte a <a href="/pt/legal/privacy-statement/" target="_self">Declaração de Privacidade da Elastic</a> para saber mais detalhes ou para cancelar a qualquer momento.

Explore similar demos

Explore outras demonstrações semelhantes

Overview

Load More

Speakers

Close

See more insights

Author

Articles by

Learn more

Watch now (no PT)

Watch now

See all top stories

All (no PT translation)

Contact information

Press Release

Share on Reddit

Share this story

Share by Email

Thank you for your interest!

Contact Info

Follow us on Youtube

Share on LinkedIn

Follow us on LinkedIn

Search Integrations

All Solutions

Video for

Follow us on Twitter

See when this webinar starts in my time zone

Consultar a hora de início do webinar no meu fuso horário

Register to Watch

Register now

See All Posts

Can't make it? Register and we'll send you the recording. You'll also receive an email with related content

Não vai conseguir participar? Inscreva-se e nós lhe enviaremos a gravação. Você também receberá um email com conteúdo relacionado.

Author

Articles by Christian Hatz

Process transparency and error categorization at 1&1 by using the Elastic Stack and ETL

Sign up for Elastic Cloud free trial