Brewing in Beats: First Filebeat release

Welcome to Weekly Beats! With this series, we're keeping you up to date with all that's new in Beats, from the details of work in progress pull requests to releases and learning resources.

ICYMI, we released 1.0.0-beta4 on Thursday.  The clear highlight of the release is the first version of FIlebeat, the lightweight log shipper successor of the Logstash Forwarder. Thank you everyone that helped testing the internal candidate, we caught at least one pretty bad issue and we got a ton of input on how we can improve the docs.

Most of the work this week went into preparing and testing the release.

Other noteworthy things:

Configurable file encoding in Filebeat

Filebeat can now be configured on a per-glob basis to expect a particular encoding. This wasn’t a problem for the Logstash Forwarder because its serialization protocol was encoding agnostic and the user could configure a codec in Logstash. It was a problem for Filebeat because (1) it uses JSON for communicating with Logstash and (2) it can send the data directly to Elasticsearch. With Windows using UCS-2 by default, it was important for us to have a solution for the beta4.

Even if this means an extra configuration parameter for Filebeat, we expect that configuring the encoding close to where the files are is better for users, so it is likely a plus overall.

Redis output deprecated

With the beta4 release, which makes it possible for all Beats to send data to Logstash without any intermediaries, we deprecated the Redis output. See this issue, where the rationale behind this decision is further explained.

Lumberjack code copied in the Beats input (Logstash)

The Lumberjack code -- used by both the new Beats input plugin and the Lumberjack input plugins in Logstash -- used to live in its own library, even though fairly different code paths were taken depending on the plugin in use. This caused some dependency conflicts at install time. To fix this, and also to gives the beats-input plugin more freedom in changes to the protocol, the protocol code was moved in the plugin repo. 

Thanks to this change, installing the beats input is now as easy as `./bin/plugin install logstash-input-beats`. 

With the next Logstash release, this step won’t be required at all as the plugin will be bundled in the default distribution.

Documentation effort

We continued our push for improving the docs, with everyone in the team contributing. We now have a professional editor helping with this, so we expect the docs quality to be rapidly improved.

Related to the documentation effort, we have updated the sample configuration files to contain a lot more comments describing the available options.

Always use UTC

We found and fixed a case where the index names that we create (e.g. [packebeat-]YYYY.MM.DD) was not using the UTC timezone.

TLS integration tests

We now have automatic tests for the Filebeat -> Logstash communication also for the TLS enabled case, including server certificate validation. Yay for one less thing to manually check and worry about.

Nightlies and automatizing the packaging tests

The listing on the S3 bucket serving the beats nightlies is now easier to browse thanks to a new JS listing snippet. More importantly, the last built artifacts now use “latest” as their version and show up first on each page, so it is now easier to download the most recent build.

We have an ongoing effort to execute the packaging tests daily on a bare metal machine (has to be bare metal because it makes use of Virtualbox machines), so having predictable download URLs was needed.