Brewing in Beats: Plugin support for Filebeat prospectors

Welcome to Brewing in Beats! With this weekly series, we're keeping you up to date with what's new in Beats, including the latest commits and releases.

Last week, we released the second beta release of the Elastic Stack 6.0. Read more details in the blog post about what’s new in the Beats 6.0.0-beta2 release.

Filebeat: Prospectors now have a plugin interface

The Filebeat prospectors are responsible for the actual data collection. The most used one is log, which can tail rotating files, but we also have stdin, udp, redis slowlogs, and we plan more. This refactoring PR makes the prospectors be pluggable internally, just like the libbeat outputs, processors, Metricbeat modules, a.s.o. This means that it is now easier to create a Beat that uses “Filebeat as a library” and that adds it’s custom prospector.

This change is currently present only in the master branch.

Metricbeat: use less memory in the Windows perfmon module

The Windows perfmon module was allocating more memory than it’s needed to receive the performance counters. The memory wasn’t leaked, but just remained unused until it’s cleaned up by GC later. 

The fix will be available in the next 6.0 release.

Other changes

Repository: elastic/beats

Affecting all Beats

Changes in master:

  • Combine fields.yml properties when they are defined in different sources #5077
  • Fix reloader error message to only print on actual error #5066
  • Fix accidental removal of --setup flag #5060
  • Modify events dropping on Close #5044
  • Fix make index-template command #5034

Changes in 6.0:

  • Combine fields.yml properties when they are defined in different sources #5077
  • Fix accidental removal of --setup flag #5060
Metricbeat

Changes in master:

  • Fix ceph cluster_status field mapping #5070
  • Reduce perfmon memory allocations #5036

Changes in 6.0:

  • Reduce perfmon memory allocations #5036
Packetbeat

Changes in master:

  • Fix invalid filter error in packetbeat #5041

Changes in 6.0:

  • Fix invalid filter error in packetbeat #5041
Filebeat

Changes in master:

  • Add prospector plugin && make imports #5026
Testing

Changes in master:

  • Run only the required containers when testing #4962

Changes in 6.0:

  • Update testing env to use 6.0-beta2 snapshots #5056
Packaging

Changes in master:

  • Use fpm --deb-systemd option for systemd unit file #5086
  • Fix /usr/bin/beatname script to accept -d "*" #5043

Changes in 6.0:

  • Fix /usr/bin/beatname script to accept -d "*" #5043
Documentation

Changes in master:

  • Added the beta2 changelog to the master branch #5033

Changes in 5.4:

  • [Docs] Fix broken link to repo install instructions #5062

Changes in 5.5:

  • [Docs] Fix broken link to repo install instructions #5063

Changes in 6.0:

  • Bump docs version to 6.0-beta2 #5057
  • Close changelog 6.0.0-beta2 #5031