Brewing in Beats: The Git repository merge

Welcome to Weekly Beats! With this series, we're keeping you up to date with all that's new in Beats, from the details of work in progress pull requests to releases and learning resources.

The past week we did a reorg of our Github repositories and worked on the design for a few key features we’re planning for the next future releases.

Repository merging

We merged libbeat and the four officially supported beats into a single repository: https://github.com/elastic/beats. We’ll have a blog post detailing the reasons, but we essentially found a single repo to result in a lot less work for us but also be more efficient in terms of communication with the rest of the community.

Filebeat filtering

One of the most often requested features for Filebeat is to be able to do basic line filtering before sending to Logstash. Think of it as doing a “egrep -v” to filter out, for example, debug messages before sending the logs out. If you were previously dropping those lines in Logstash, there are now two handy options in Filebeat that can save you network traffic and CPU load: include_lines and exclude_lines.

All the details are in the Github issue and the Pull Request.

Design for generic filtering

In addition to the simple filtering described above, we started thinking also about a more generic way to filter events or fields, from which all Beats will benefit. The discussion is happening in this issue and you are invited!

Design for multiline

Another Frequently Requested Feature for Filebeat is about being able to merge related lines (like the ones from an exception, for example) into a single event. This can be done in Logstash already with the multiline codec, but doing this closer to where the files are read might make it more reliable for some people and easier to configure and maintain for others. This is also in the design phase, see this GitHub issue. Once again, your input here would be valuable.

Design for Packetbeat’s HTTP body handling

Another interesting issue on which design is discussed is about making Packetbeat be able to understand the application specific HTTP payload. For example, if we know that the body is JSON created by an Elasticsearch client or server, we could extract some more interesting information, like the index name, the search query, the number of results, etc. The same can be done, and with relatively little effort, for any protocol that works over http, like CouchDB, Docker’s protocol, XML-RPC, JSON-RPC, etc.

Apachebeat

We are very happy to see another Beat created by the community: Apachebeat created by Radovan Ondas. Similar to the existing Nginxbeat, Apachebeat can be used to insert the key Apache metrics into Elasticsearch.

Connect to Elasticsearch over proxies

It’s now possible for all Beats to also use an HTTP proxy when connecting to Elasticsearch.

Packetbeat performance improvements

Steffen has been profiling and improving the performance of the Redis and Http parsers in Packetbeat. These are still in progress and being discussed.

Winlogbeat

Winlogbeat is slowly getting closer to being releasable, and now it also has his own docs.