Elastic Advent Calendar, 2019: the full recap!
Wow, it's finally here! After 25 fantastic articles we've reached the end of the 2019 Elastic Advent series.
We've covered Elasticsearch and Python, Auditbeat, ECS, data transform, jvm options, anomaly detector models, Maps, SSL configuration, Smart query cancellation, data transforms, SLM, the new enrich processor, App Search, and so much more. In the topics we've spoken in German, Greek, English, French, Finish, Spanish and Swedish.
We hope you have enjoyed the posts, topics and information shared in them from our engineers. Here's a recap of the final few days, as well as a summary from all 25 days.
The 25 days of Elastic
Dec 1 [Finnish] — Datatieteilijän työkalut: Elasticsearch ja Python (Elasticsearch and Python - tools for a data scientist) by Camilla Montonen
Yleensä suurin osa datatieteilijän työajasta saattaa kulua datan keräämiseen ja puhdistamiseen ennen kuin päästään itse koneoppimisalgoritmien käyttämiseen. Yksi osa tätä prosessia on datan varastoiminen sopivaan tietokantaan. Yhä useammat sovellukset käyttävät Elasticsearchia tietojen tallentamiseen, hakemiseen ja analysoimiseen. Tietojen analysoimiseen voidaan Elasticsearchin sisäänrakennettujen toimintojen lisäksi käyttää myös ulkoisia, asiaan soveltuvia ohjelmistoja ja kirjastoja. Yksi tällä hetkellä suosituimmista datatieteeseen soveltuvista ohjelmointikielistä on ehdottomasti avoimen lähdekoodin omaava Python, jolle on ohjelmoitu lukuisia datan analysoimiseen ja koneoppimiseen soveltuvia kirjastoja.
Dec 2 [English] — Monitoring Linux Command Execution with Auditbeat by Jason Bryan
The Auditbeat Auditd module can be used to capture all shell commands executed on system for all users. Monitoring shell commands is often desired on servers where end user shell activity is normally minimal.
Dec 3 [Spanish] — Como utilizar Elasticsearch para procesar la actividad de Mailgun by Gonzalo Servat
Mailgun es un proveedor que ofrece un servicio SaaS para enviar y recibir e-mails, procesar unsubscribes, y llevar un seguimiento de cada transacción que ocurre a través de su plataforma. Mailgun ofrece una interfaz que permite analizar los detalles de cada transacción (por ej. resultado de la entrega). Ésta información es sumamente útil no solo para generar reportes, sino también para responder preguntas acerca del resultado de los envíos. El hecho de que Mailgun tenga control de esta información genera varios problemas como por ejemplo que sea necesario tener una cuenta en Mailgun para acceder a esta información, y si bien la interfaz permite investigar transacciones, la misma es muy limitada. Asimismo, Mailgun solo retiene los registros por un corto plazo.
Dec 4 [French / English] — Explorer Elastic Common Schema (ECS) avec Elasticsearch et Kibana / Explore Elastic Common Schema (ECS) with Elasticsearch and Kibana by Mathieu Martin
Plusieurs ressources existent déjà, pour apprendre ECS. Bien sûr, il y la documentation officielle. Certains utilisateurs utilisent également l'export generated/csv/fields.csv. L'export CSV permet de visualiser rapidement tous les champs dans leur ensemble, ou encore d'importer le schéma dans une spreadsheet.
Saviez-vous qu'il est aussi possible d'importer ce CSV directement dans Elasticsearch?
Dec 5 [German] — Weihnachtswunschzettel: Wünsche zusammenfassen mit Transforms(Collapsing x-mas desires with transform) by Hendrik Muhs
Zu Weihnachten gibt es die schöne Tradition des Wunschzettels, Kinder können hier Ihre Wünsche aufschreiben, aber auch Sorgen und andere Dinge dem Weihnachtsmann mitteilen. In Deutschland gibt es mehrere Weihnachtspostämter, die jährlich hunderttausende Briefe beantworten, noch viel mehr Wunschzettel bleiben wahrscheinlich innerhalb der Familie und landen bei den Eltern oder Großeltern.
Dec 6 [English] — Explain jvm.options in Elasticsearch by Alexander Reelsen
When configuring Elasticsearch, you will find one page about configuring JVM options. In this post we will take a look how to configure this and how this is implemented internally.
Dec 7 [English] — Looking behind the scenes of anomaly detector models by Ed Savage
Ever wondered what's really going on behind the scenes with Elastic's unsupervised machine learning anomaly detection modelling? (and if not why not?!). Sure you may think you know what's going on, you've read our extensive and beautifully written documentation, maybe you've even enabled model plot in the job configuration and viewed the results in the single metric viewer (but would you like to know more?). Maybe you've downloaded the backend source code, compiled it, and ran the extensive tests. If you have, kudos to you! But do you want to understand what those tests are about? Read on!
Dec 8 [german]: Loggen in Elasticsearch und Elastic Cloud (logging in elasticsearch and elastic cloud), by Philipp Krenn
Loggen ist eines der Themen, das gerne ignoriert wird, bis man es benötigt. Und dann braucht man meistens eine schnelle Lösung. Glücklicherweise ist das Loggen in Elasticsearch sehr flexibel und auch relativ mächtig. Aber schauen wir uns das gleich konkret an.
Dec 9 [english]: Smart query cancellation in Kibana, by Lukas Olson
Since the beginning, Kibana has always been about taking huge volumes of data in Elasticsearch and making sense of it visually. Over time, Kibana has stretched the limits of Elasticsearch, and we’ve needed to make changes in Elasticsearch in order to more powerfully enable users of Kibana.
Dec 10 [greek]: Κατηγορίες SSL/TLS παραμετροποίησης στο Elasticsearch, by Ioannis Kakavas
Η παραμετροποίηση του SSL/TLS πρωτοκόλλου δεν είναι απαραίτητα το πιο απλό πράγμα που έχει να κάνει ένας διαχειριστής που στήνει για πρώτη φορά ένα elasticsearch cluster. Με την αλλαγή την οποία κάναμε πρόσφατα, όπου η κύρια λειτουργικότητα ασφάλειας (και κατά συνέπεια η υποστήριξη για SSL/TLS ) είναι διαθέσιμη δωρεάν με την βασική άδεια λειτουργίας (basic license), παρατηρούμε ότι όλο και περισσότεροι νέοι χρήστες σπεύδουν να χρησιμοποιήσουν τα νέα αυτά χαρακτηριστικά, θα ήταν χρήσιμο να συζητήσουμε μερικά πράγματα στα πλαίσια του advent calendar.
Dec 11 [english]: Maps getting even easier with 7.5: A practical example, by Radovan Ondas
Kibana Maps were already introduced in version 6.7. Since then each release brought fixes and also many new features and improvements to existing ones.
It was already possible to add color to a location and also custom color map based on chosen document value. This approach is simple and effective for many use cases. Later on we added possibility to use Icons instead of a circle and also coloring based on their value.
With the release of version 7.5 there is a new possibility for how to style your location points you are placing on the layer. As we add many new features to every Kibana release, this little improvement was not even mentioned in the blog post.
Dec 12 [english]: Data Transforms: More than Meets the Eye, by Ken MacInnis
You're running at scale, with petabytes of proxy logs, desktop event streams, and endpoint security alerts at your fingertips. The thing is, human beings don't think about log lines and event fields - we think in terms of users, sessions, and vulnerabilities. But repeatedly querying many indexes across multiple petabytes seems expensive.
There's another option: data transforms, introduced in version 7.3 of the Elastic stack. Data transforms are a way to create summary indexes from existing data, either one time or on an ongoing basis.
Dec 13 [english]: How to diagnose and cure web app flu, by Emanuil Tolev
'Tis the season for festivities, overindulgence and occasional illness. We could suggest season-appropriate decoration 1 (at least if your app is hosted in the Northern hemisphere). However, today we will focus on apps which are feeling unwell and have become sluggish - and how to use APM to diagnose them.
Dec 14 [swedish]: Så här börjar du med Elastic Maskininlärning (Getting started with Elastic Machine Learning), by Camilla Montonen
På Elastic, har vi utvecklat olika maskininlärningsprogram sedan 2016. Då började vi med Elastic Anomaly Detection. Vårt Anomaly Detection program matas med tidsserier och försöker att hitta avvikande datapunkter.
Fördelen med Anomaly Detection är att den inte behöver tränas med märkt data. I stället, lär den sig över tiden vad som är normalt beteende i systemet och övervakar nyankomna data för att hitta avvikelser. Den här är en typ av oövervakad maskininlärning.
Det finns många intressanta problem som inte kan lösas med sådana oövervakade tekniker och i stället behöver övervakade maskininlärningsalgoritmer. Därför har vi i år satsat mycket tid för att utveckla övervakad maskininlärningsfunktioner. I den här korta artikeln bjuder vi på en inledning i övervakad maskininlärning i Elasticsearch.
Dec 15 [english] — SLM: setting up an S3 repository using Minio.io, by Luca Belluccini
Snapshot Lifecycle Management (SLM) is a way to schedule and define snapshot policies using the Elasticsearch API or the Kibana UI. It also allows to check the existing snapshots and restore them directly on Kibana.
Dec 16 [english] — Indexing Github Events for Fun and Profit, by Ron Toland
Like a lot of companies, we lean on Github for more than just git repo hosting. We use it to track work via tickets, asynchronously pair on code via PR reviews, and even use project boards for some Kanban goodness.
All well and good, but there are certain things Github just can't do. Like tell us whether the number of incoming critical bugs is trending up. Or give us hard numbers about who on the team is going above and beyond to review as many PRs as possible.
There are custom tools that can fetch data from Github via its API and assemble this kind of information for us. But we thought: What if we could just get this into Elasticsearch? Once there, could we build visualizations in Kibana to get the answers we need?
Dec 17 [english] — IoT: Christmas tree lights with Elasticsearch and Micro:bit, by Hendrik Muhs
Like a lot of companies, we lean on Github for more than just git repo hosting. We use it to track work via tickets, asynchronously pair on code via PR reviews, and even use project boards for some Kanban goodness.
All well and good, but there are certain things Github just can't do. Like tell us whether the number of incoming critical bugs is trending up. Or give us hard numbers about who on the team is going above and beyond to review as many PRs as possible.
There are custom tools that can fetch data from Github via its API and assemble this kind of information for us. But we thought: What if we could just get this into Elasticsearch? Once there, could we build visualizations in Kibana to get the answers we need?
Dec 18 [english] — Monitoring your home network with the Elastic Stack, by Paul Coghlan
This article describes how to monitor your home router with the Elastic Stack.
Dec 19 [english] — Simplifying Ingest Pipelines with the new Enrich Processor, by Imma Valls
When ingesting data through a regular Elasticsearch ingest pipeline (e.g. with dissect, rename, or remove processors) we can now add an Enrich Processor.
This allows us to do lookups on other Elasticsearch’s indices and enrich the incoming document before sending it to its own index.
Dec 20 [english] — Getting Started with App Search, by Aravind Putrevu
App Search is a search solution that is built on Elasticsearch. Using App Search, one can build search experience faster. One can tune relevance, create synonyms or even change the index schema using the App Search Dashboard. All one needs is a dataset they want to search upon.
Dec 21 [english] — Gain insights from large datasets with choropleth maps and Elastic maps, by Nathan Reese
"Get It Done" is a San Diego initiative to engage citizens in reporting non-emergency problems like potholes, missed garbage collection, and street light outages. Smart phone applications make reporting easy. The "Get It Done" data set is publicly available and can be downloaded as CSV 1. Follow these instructions 1 to ingest the "Get It Done" data set into Elasticsearch.
The 2019 CSV contains over 250,000 reports. Each report contains a GPS location. Lets start by viewing the reports on a map.
Dec 22 [english] — Outlier Detection by Example, by Michael Hirsch
Outlier detection has been available in machine learning since 7.2 - the blog post is a demonstration about how to create outlier detection analyses and how to analyze the results.
Dec 23 [french] — 5 choses que j'aurais aimé apprendre du Père Noël avant de passer la certification Elastic Engineer, by Melvyn Peignon
Bien que j’adore écrire des articles techniques, aujourd’hui je vais plutôt essayer de démystifier un sujet : la certification "Elastic Engineer Certified".
Comment réussir la certification ?
C’est probablement la question qui m’est le plus demandée, à part bien sûr l’indémodable :
Combien de noeuds il me faut ?
Mais contrairement à la question relatif au nombre de noeuds, cette fois je ne vais pas vous répondre "Ça dépend !".
Donc voici 5 les choses à faire avant de passer la certification.
Dec 24 [english] — Canada open data in Kibana maps, by Bhavya Raju Mandya
I love maps, I love geo-spatial data. And last year I plugged in Canada's population data into our region maps for advent calendar. But what I really wanted to do back then was to see what happens when I plug in the data in our amazing new maps app. But at that time our maps application was in development mode. But now its here and it's awesome. So, let's see what we can do with it.
Dec 25 [english] — A santa claus story, by Tamara Rosini
Here I was at home and out of nowhere I heard a voice, when I look back all I could see a multicolor smoke and when it starts to fade out a figure start to shape. I was mesmerized and it was Santa, I froze and slowly hold my mobile as I was ready to call the police. He walks at my direction and says with a soft but firm voice: “Hi Tamara, I was wondering if you could help me.”, I was so shocked that I stammering “me? How could I ever help Santa?”, I could barely believe that I said “Santa”. He looks straight into my eyes, with kind eyes and says “Can you help me to monitor my toys production for Christmas?”. I pinched myself myself and said “Yes”.
Thank you!
All the of the topics will be kept on the Elastic Discuss Forums so you can refer back to them at any time. And, as these are Discuss topics, you can also continue the conversation with the authors and other community members.
Thanks for following this year's series, we hope it’s provided some useful inspiration for your use of the Elastic Stack. If you’d like us to repeat this, if you have ideas for next year or any other feedback, please let us know via Twitter (@elastic) or feel free to create a topic in our Meta category with your comments.
We hope 2019 has been an amazing year for you and we look forward to building upon it for 2020!