20 October 2015

Brewing in Beats: Preparing the first Filebeat release

By Tudor Golubenco

Welcome to Weekly Beats! With this series, we're keeping you up to date with all that's new in Beats, from the details of work in progress pull requests to releases and learning resources.

Here is what we worked on over the past week.

Preparing for the first Filebeat release

We’re very close to releasing Filebeat (the Logstash-Forwarder successor) and we are excited to share it with the rest of the world.

We have a new Logstash plugin that speaks the Beats flavor of the Lumberjack protocol. The initial plan was to enhance the Lumberjack plugin for Filebeat  in a backwards compatible way but we changed course when we realised that there will be at least one backwards incompatible change (TLS off by default) and that having a separate plugin would make the LSF -> Filebeat migration path easier.

The new plugin name also makes our intentions of using it for all existing and future Beats more clear to our users. In a similar change, the Lumberjack output in Filebeat is renamed to “Logstash”.

The new plugin compatible with both Logstash 1.5.x and 2.x branches and it makes it possible to load the “beats” and the “lumberjack” plugins at the same time.

Documentation effort

We had a team effort on pushing forward the documentation for Filebeat as well as incrementally improving the docs for the other Beats as well, for example by documenting the CLI flags and the TLS config options. The developer guides for adding a new protocol to Packetbeat were also updated.

We’re also working on a migration guide from LSF.

Direction field in Packetbeat

Coming from a community contributor, this PR adds a “direction” field to the transactions captured by Packetbeat. If the request is received by the machine on which Packetbeat runs, the value will be “in”. If the request is sent by the machine on which Packetbeat runs, the value for it will be “out”. This is useful for differentiating between the client and server side of the same transactions, so one can measure the response time as seen by the client, the response time as seen by the server, and then calculate the network time by subtracting the two.

Platform dependent location for the registry db in Filebeat

Filebeat persists the read offsets in a json file on disk, so no log lines are lost in case of Fiebeat restart. The JSON file is by default named “.filebeat” and placed in the current working directory. This is not very convenient when installing from packages or running as a service on Windows, so we made the location and name depend on the way Filebeat is installed. It’s now `/var/lib/filebeat/registry` for deb/rpm and `C:\ProgramData\filebeat\registry` on Windows.

Gofuzz on the protocol parsers

Another community contributor, darxriggs, tried a cool coverage-guided fuzzing tool on the Packetbeat parsers and found two bugs in the mongodb parser. Great way of contributing! Small side note: a nice article about running go-fuzz on a DNS parser.

Configurable transaction timeout in Packetbeat

The time after we removed a transaction from memory if the response was not received was hardcoded to 10 seconds in Packetbeat. This essentially meant that we couldn’t correlate transactions that take longer than that. The timeout is now configurable on a per-protocol basis.

Keep a change log

We started to to keep change logs in all our projects by using the keepachangelog.com standard. Here is the current one for Packetbeat. This will save time for us at the release time and because it’s part of the review process, it’s less likely to be incomplete, so it will also save time for the users.

One gotcha with keeping the change log in a committed file is that it tends to result in lots of merge conflicts. We had this problem a lot until Steffen found a way to tell Git how to always union-merge the file.

Better user experience with -devices

The `-devices` CLI flag of Packetbeat prints all the network devices on which it can sniff. If you run it without `sudo`, you cannot sniff so it used to print absolutely nothing. It now prints a nice message and on non-windows platforms it suggest using sudo.