26 December 2018 Engineering

The Elastic Advent Calendar 2018, Week 4

By Mark Walkom

Wow, it's finally here! After 25 fantastic articles we've reached the end of the 2018 Elastic Advent series.

We've covered Kibana - from spaces to maps, Elasticsearch - CCR and relevancy, Logstash enrichment, ingestion with multiple Beats, APM, App Search, Machine Learning and even into new areas around Observability and so much more. In the topics we've spoken in Chinese, Japanese, German, English, French and Portuguese.

We hope you have enjoyed the posts, topics and information shared in them from our engineers. Here's a recap of the final few days, as well as a summary from all 25 days.

The 25 days of Elastic

Dec 1 :[FR][Logstash] Enriching Postal Addresses by David Pilato

Découvrez dans cet article comment transformer des addresses postales en coordonnées géographiques en local à l'aide de la suite Elastic et des données publiques d'Open Street Map.

Dec 2: [JP][Kibana] Understanding Analyzer with Analyze UI plugin by Jun Ohtani

Analyzerの動きを理解することが、Elasticsearchの検索をよりよくする、検索にヒットしない理由を解析する近道だったりします。

素晴らしいことに、Elasticsearchには_analyze APIという便利なAPIが存在します。

Dec 3: [KR][Logstash] Crawl data from Facebook to Elastic through Logstash by Jongmin Kim

저는 Elastic 한국 사용자 그룹의 글을 가져오기 위해서 만들었는데 Facebook Group의 글을 가져오기 위해서는 추가 검수가 필요합니다.

Dec 4: [EN][Machine Learning] Rarity Analysis with Machine Learning by Rich Collier

Finding items that rarely occur is often very useful. Some example use cases are finding:

    • Rarely occurring log messages
    • Rare running processes for a server
    • Rare connection destinations

ML's rare function is only available in the Advanced Job Wizard, but the configuration is relatively simple. 

Dec 5: [EN][App Search] Quick and easy typeahead search implementation with Elastic App Search by Peter Kim

Elastic App Search is a Search Solution from Elastic that makes it easier for application developers to implement and manage core search functionality in their user-facing applications. There's no need to understand the nuances of text analysis, relevance scoring algorithms, spell correction, etc.

Dec 6: [JP][Elasticsearch] Dealing with search relevancy by Kosho Owa

ただ、検索機能をアプリケーションに実装する場合、検索順位を制御したい場合が多々あります。「1000円以上の商品を安い順」「今週入荷した商品を新しい順」「東京都庁から近い順」に並べるといった場合です。それらのうち、複数の条件を組み合わせたい場合もあるでしょう。Elasticsearchを使用して、どのように実装したら良いでしょうか。

Dec 7: [FR][Kibana] Kibana Canvas by Christelle Kerloc'h

Avec la sortie de la suite elastic 6.5, Kibana Canvas est là ! Vous pouvez désormais utiliser cette nouvelle application. Venez découvrir l'historie de Canvas et quelques cas d'utilisation pour vous inspirer.

Dec 8: [EN][Python/Elasticsearch] Getting started with Elasticsearch DSL and Python by Honza Krai

The purpose of elasticsearch-dsl is to provide an easier and more familiar way to work with Elasticsearch. It focuses on just the most common operations like search and generally working with data...

Dec 9: [EN][Elastic Stack] Ubiquiti metrics in the Elastic Stack by Mark Walkom

The Ubiquiti Unifi 15 kit is pretty amazing for its price point. I use it extensively at home to replace a bunch of Google Wifi, power line and wifi extenders and other devices that just never really worked for me. Other than the value for money they provide, it's super simple to setup and manage.

Dec 10: [EN][Elastic Stack] Correlate and Alert for Security Analytics by Sherry Ger

Security analytics is a common use case for the Elastic Stack. In this short topic, we will set up a rule based alert that triggers a notification when network or log events correlate with indicators of compromise. We would alert near real time so investigators can start their investigation and remediate as quickly as possible.

Dec 11: [PT-BR][Elasticsearch] Usando Field Aliases para criar uma visão unificada entre índices com mapeamento distinto by Thiago Souza

Field Aliases, que apareceu na versão 6.4, é uma poderosa funcionalidade que permite criar uma visão unificada entre índices com mapeamentos distintos.

Dec 12th: [EN][Elasticsearch] Automatically adding a timestamp to documents by Abdon Pijpelink

Back in the old days, prior to version 5 of Elasticsearch, documents had a metadata field called _timestamp. When enabled, this _timestamp was automatically added to every document. It would tell you the exact time a document had been indexed.

Dec 13th: [EN][Elasticsearch] Chaining Ingest Pipelines by Luca Wintergerst

In the following example we will explore the ability to reference ingest pipelines to remove some of the complexity and duplication. For this we will use the new pipeline processor 1 in 6.5. That allows us to call a pipeline from another pipeline...

Dec 14th: [EN][Kibana] Canvas: From 0 to dashboard before the morning is out by Aaron Aldrich

Canvas is probably my favorite new feature of Kibana. Since its first technology previews I was thrilled with the idea of live-data backed slide decks built inside of Kibana and beautiful free-form dashboards. It's only recently, though, that I had the time and inspiration to dig in myself. The Central Pennsylvania Open Source Conference gave me just the excuse (and dataset) I needed to get started.

Dec 15: [EN][APM] Overhead of the Java agent by Felix Barnsteiner

In this little blog post, I want to give insight into how much the [java] agent affects the performance of your applications and which configuration settings affect the performance.

Dec 16: [DE][Elasticsearch] Zusammengesetzte Worte in Elasticsearch by Philipp Krenn

German, Finnish, Korean and the Scandinavian languages have the characteristic impractical for full-text search that they form compounds, ie combine several words into one. This leads to problems, for example, if you have saved the word "full-text search" and would like to find it by just searching for "search". But how can this request still be implemented?

Dec 17: [FR][Elastic Stack] Indexer facilement les slow logs d’Elasticsearch avec Filebeat by Lionel Palacin

Pour détecter les requêtes les plus lentes, Elasticsearch fournit une fonctionnalité appelée Slow logs. Cette fonctionnalité permet d'écrire dans les fichiers de logs les requêtes de recherche ou d'indexation dont le temps d'execution est supérieur à un seuil défini par l'administrateur du cluster Elasticsearch.

Dec 18: [EN][Elastic Stack] Set up for success - don’t repeat yourself! by Ken Macinnis

Do you find yourself re-using the same index settings and mappings frequently? Do those mappings have lots of fields with repetitive naming conventions?

If so, fear not! You aren't alone, and there's relief at hand. Let's talk about Dynamic Templates and Index Templates.

Dec 19th: [EN][Heartbeat] Configure a TLS expiration notification using Heartbeat by Dov Hershkovitch

In this blog I would like to cover one of the new features we’ve added to Heartbeat, TLS information. Starting from 6.5, Heartbeat, will retrieve TLS information from the end point it is checking.

Dec 20th: [EN][Python/Elasticsearch] Python DSL for Elasticsearch by Honza Krai

In the previous post we learned about elasticsearch-dsl and created an index in Elasticsearch to store all of our git history, now it's time to load in the data and see what we can do with it.

Dec 21st: [EN][Elasticsearch] Using SQL to query data in Elasticsearch by Aravind Putrevu

Querying data using Structured Querying Language (SQL) has been introduced in Elasticsearch 6.3 as a plugin. One can write SQL queries on the data stored in any Elasticsearch Index.

Dec 22nd: [CN][Elasticsearch] 跨集群复制(CCR)的使用 by Medcl Zeng

CCR - Cross Cluster Replication - 跨集群复制是 Elasticsearch v6.5 发布的一个新的特性,这个特性可以让你将一个集群的索引数据同步复制到远程的另外一个集群上面去。

Dec 23rd: [EN][Observability] Querying metrics from Prometheus by Tanya Bragin

Elastic is increasingly used as a single operational data store for logs, metrics, and trace data. As a result, we get asked whether it is possible to ingest metrics from Prometheus exporters or integrate with Prometheus server.

Dec 24th: [EN][Kibana] Region maps in Kibana by Bhavya Mandya

Maps in Kibana are getting awesomer day by day. We have added a ton of features to our existing region and co-ordinate maps.

Dec 25th: [EN][Kibana] Awesome Kibana Spaces by Rashmi Kulkarni

Kibana Spaces allows you to organize your dashboards, visualizations and other saved objects into meaningful categories based on team, use case, individual or really any way you choose to use it. The beauty of this feature is each space created is independent, so objects that exist in one space don't clutter other spaces.

Thank You!

All the of the topics will be kept on the Elastic Discuss Forums so you can refer back to them at any time. And, as these are Discuss topics, you can also continue the conversation with the authors and other community members.

Thanks for following this year's series, we hope it’s provided some useful inspiration for your use of the Elastic Stack. If you’d like us to repeat this, if you have ideas for next year or any other feedback, please let us know via Twitter (@elastic) or feel free to create a topic in our Meta category with your comments.

We hope 2018 has been an amazing year for you and we look forward to building upon it for 2019!