Searching and Alerting for application logs with Elasticsearch at Naver (Japanese subtitles)
Naver Corporation is the biggest Internet content service provider in South Korea. The company is famous for developing and distributing such mobile and server applications like Naver search portal, Line and Band messengers. Our developers often need to check application logs to resolve customer issues. However finding the root reason of distributed or mobile application's fault is not easy. Logs are stored on hundreds of servers and/or only on customers' devices. NELO2 is an in-house logging system that provides log collection, storage, full-text search, alert and analytics features. Developers can easily send their logs to NELO2 collector servers using various SDKs from many platforms. Collectors filter logs, throttle, and store them to Apache Kafka which is a distributed messaging queue. Logs in Kafka are consumed and indexed into Elasticsearch. Through NELO2 Web interface developers can search their logs, aggregate them, register alerts with lucene query, subscribe for daily reports, etc. Elasticsearch is used in NELO2 as a main log storage, search and alert. NELO2 is indexing and percolating 1.5 billion logs daily with Elasticsearch. 126 TB of logs are stored in 7 Elasticsearch clusters. In this session, first, we will talk about the architecture of NELO2 and how we use and maintain Elasticsearch clusters. Then, we will explain the detailed implementation design of the new real time notification system in NELO2.
Jae Ik Lee
Jaeik is leading in-house log management platform development for Naver and has used Elasticsearch from version 0.19 for developing log search features. He is a big fan of Elasticsearch because of its flexibility. Previously, Jaeik worked at Samsung Electronics for mobile Linux platform development. Every morning, he enjoys watching Coursera lectures while he washes the dishes to make his wife happy. Playing with their two sons and mountain climbing with them are Jaeik's biggest pleasures.
Seung Jin Lee
Seungjin is a member of the in-house logging platform team at Naver, currently acting as backend data pipeline engineer. He is interested in various open-source projects including Elasticsearch and Storm, which the team used for developing the real-time notification server discussed in this presentation. Seungjin likes to travel and do yoga, which he thinks is really beneficial to software engineers.