02 November 2015

Brewing in Beats: On the way to RC1

Welcome to Weekly Beats! With this series, we're keeping you up to date with all that's new in Beats, from the details of work in progress pull requests to releases and learning resources.

Getting Feedback

The Beta4 release brought us positive feedback from the community, but also more work for our team. There were many questions and issues opened that kept us busy for the week, but also motivated us to improve the Beats on the way to GA.

Our current plan is to release an RC1 the middle of this week, and if things go well, release the GA a bit later.

Apt and yum repositories

It’s now easier to install Packetbeat, Filebeat, and Topbeat by using the apt and yum commands. The beta4 release packages, as well as previous releases, are now signed and uploaded to packages.elasticsearch.org where all the other Elastic products are also available. The steps to install the Elastic public key and then the packages from the repositories are in this ticket.

Renamed the ‘timestamp’ field to ‘@timestamp’

Everything started with an issue reported against Filebeat where the timestamp information is available twice in Elasticsearch when Logstash output is enabled. Filebeat sets the `timestamp` to the time when the event is read, and Logstash adds an additional `@timestamp` field set to the time when the event was received or to the value parsed from the log line. Having two timestamp fields, usually with similar but not identical values, can be quite confusing.

So we renamed `timestamp` to `@timestamp` in all Beats. Filebeat exports the `@timestamp` field, and if the Logstash date-filter is used, the field is overwritten with the real timestamp of when the event was produced. This keeps the field consistently named across Beats and also consistent with the file-input plugin from Logstash.

On the downside, the meaning of the `@timestamp` field is different for each Beat. For Packetbeat, it means the time of the request, for Topbeat it is the time when system metrics were read, and for Filebeat it is the time when the log line was read.

Filebeat Windows fixes

Windows is a challenging platform for log readers due to its strict file locking rules. With the help of the community, we discovered and fixed an important issue this week. Because Filebeat was requesting GENERIC_WRITE access to the files, they were being marked as read-only for new open operations.  Our automatic tests failed to detect this issue because the tests are written in Python, and Python simply ignores this flag.  Removing requesting the GENERIC_WRITE access fixed it.

Add stdout as file output

We added stdout as file output as the result of this thread on discuss. This output is useful for testing and potentially for chaining the Beats.

Getting to an awesome documentation

This week we worked closely with DeDe, our great technical writer, on improving the Beats documentation. You can already see the improvements online for the Beats Platform Guide, Packetbeat Guide, Topbeat Guide, and Filebeat Guide.

Closing other Beta4 Issues

A big Thank You to everyone who tried the Beta4 release and posted issues on GitHub or Discuss. We fixed most of them this week.