I have been following Packetbeat for quite some time, being one of those projects I just love. Packetbeat is a lightweight network packet analyzer that parses different protocols (HTTP, MySQL, Postgresql, Redis, Thrift-RPC), extracts the relevant information from them, ships the data to Elasticsearch and uses Kibana to visualize it.
It has grown to become quite successful, with a strong following. These type of projects are what make open source such a great platform for innovation. Here at Elastic we are building products like Elasticsearch, Kibana, and Logstash, and a project is born using them in a wonderful new direction.
A project, though, is just part of the story. It’s also the people behind it that make the difference. I had a chance to meet Tudor and Monica, the couple behind Packetbeat, and I was blown away by their dedication to the project, their commitment to its success, and their vision to where it can be taken.
At the same time, we were chatting here at Elastic when it comes to lightweight, native shippers within our stack. Logstash is a great tool to centralize and process your event data, and Logstash-forwarder, the lightweight native file log shipper, is seeing a lot of usage in the field shipping data from upstream servers to Logstash using a native protocol called Lumberjack.
Native, lightweight shipping technology is a key part of our technology stack, and it is a role currently being filled by Logstash-forwarder, which is file-centric and in need of love. As we look to the future, we see the value of lightweight shippers for many data sources – network data, metrics, and various types of events. Packetbeat is an excellent example.
Back to Tudor and Monica, when I chatted to them about our vision, we quickly found common ground. It also helps that they are just wonderful people that fit right in into how we work at Elastic.
I am happy to announce that both have joined Elastic and will lead our newly formed Beats team, focusing on building lightweight data shippers, with the first Beat being Packetbeat.
Beats are the future of open source data shippers to both Elasticsearch and Logstash. Today, Beats are composed of libbeat, a core library holding common functionality shared by all the different beats, and Packetbeat, our first beat, focused on capturing network data and shipping it to Elasticsearch. In order to help with wider adoption of Beats/Packetbeat, we have relicensed the Packetbeat code base from GPL v2 to Apache 2.0.
This is just the beginning though, we have an aggressive vision to what Beats can do, and I would love to share it with you.
Today, libbeat is simply code extracted from the original Packetbeat project that would apply to all future Beats. For example, it includes the code used to ship data to Elasticsearch. In the future, we plan to add support for Lumberjack, the native Logstash protocol to make sure data can be shipped either to Elasticsearch directly or to Logstash for further enrichment.
Packetbeat will be successful based on its breadth of network protocols it can parse, making it applicable to a wide variety of use cases. Nothing beats open source when it comes to building such a repository and community, and we are excited to help make it a reality. Is there a product out there that you specialize in, and want to make it more accessible to its user base? Help us in writing a parser for it!
Filebeat is the Logstash-forwarder, the next generation. I mentioned before that we plan to take Lumberjack, the native Logstash protocol, and make it available to any Beat through libbeat. The next step is to transform Logstash-forwarder into a proper Beat, one that is focused in tailing log files and shipping them over to Logstash.
Tons of ideas here, our focus will initially be on the above, but we are excited with all the future Beats that can be developed, especially on top of libbeat as it matures – event logs, metrics, put your idea here.
Welcome Tudor & Monica!