12 April 2018

Brewing in Beats: Spooling to disk in Beats

By Monica Sarbu

Welcome to Brewing in Beats! With these weekly series, we're keeping you up to date with what's new in Beats, including the latest commits and releases.

Did you know that Beats 6.2 is already available? Try it and let us know what you think. If you are curious to see the Beats in action, check out the Getting Started with Beats webinar.

Spooling to disk in Beats

Spooling to disk adds an optional persistent queue to all Beats publisher pipeline. Any event to be published will be written to the spool first and only forwarded to Elasticsearch or Logstash afterwards. With spooling to disk, events not yet published are still available after Beats have been restarted or in case outputs have been stalled for a long time due to network issues.

Quite a lot of work has gone in this feature, and it is going to be valuable especially for Beats like Metricbeat or Auditbeat where there’s no natural queue on disk (like in the case of logging). We plan to combine this with End-to-End ACKs in Logstash, and ID deduplication to achieve better resilience and “exactly once” guarantees.

Initial support for spooling (PR #6581) has been merged end of last week and will be released as Beta feature with Beats 6.3.

Filebeat TCP Input

With #6700 a new input type for receiving events via TCP has been added to Filebeat. The new input will wait for TCP clients, each publishing a stream of events. By default, the events are split on newlines, but the event delimiter can be freely configured. This new feature will be available in 6.3.

Default metricsets for each Metricbeat module

All Metricbeat modules have now default metricsets. Instead of having to configure at least one metricset when running a module, the module name and host is enough and the module will automatically run its default metricsets. For autodiscovery, this means only the module has to be specified to be enabled instead of modules and metricsets. This feature will be available in 6.3.

Documentation

Changes in master:

  • Cleanup changelog #6795
  • Improve module overview list #6788
  • Docs: use a simple clone in the New Beat guide #6786
  • Add pubsubbeat to list of community beats #6748
  • Added GCSBeat to community beats list #6732
  • Fixes autodiscover's appender configuration example #6720
  • Set default metricsets for system module #6689
  • Add dashboard check to metricbeat module overview #6686
  • Set all couchbase metricsets as default in metricbeat module. #6683
  • Set all ceph metrics as default in metricbeat module #6682
  • Set reference config as config for docs #6671

All changes

Repository: elastic/beats

Metricbeat

Changes in master:

  • Added changelog for changes in default metricsets (#6668) #6781
  • Set `stubstatus` as default metricset for nginx module #6770
  • Set status as default in mysql module #6769
  • Set `pool` as default metricset for php_fpm module #6768
  • Set node as default metricset for munin #6767
  • Set default metricsets for mongodb module #6766
  • Set stats as default metricset for memcached module #6765
  • Set default metricsets for kubernetes module. #6764
  • Set all metricsets as default for Logstash module #6763
  • Set status as default metricset for Kibana module #6762
  • Set default metricsets for postgresql module #6761
  • Make all metricsets in Kafka module defaults. #6759
  • Set all metricsets as default for HAProxy module #6758
  • Set server metricset in graphite as default #6757
  • Set default metricsets for rabbitmq module #6743
  • Set default metricsets for redis module #6742
  • Add changelog entry for system.raid promotion to beta #6739
  • Set collector as default in dropwizard module #6719
  • Set default metricsets in Docker module #6718
  • Mark Metricbeat uswgi module and metricsets as beta #6717
  • Move Logstash node_stats under node.stats #6714
  • Mark metricbeat logstash module and metricsets as beta #6713
  • Mark http.server metricset as beta #6712
  • Mark Golang module and metricsets as beta #6711
  • Mark system.raid metricset as beta #6710
  • Fix differences in Jolokia config examples (#6643) #6657

Changes in 6.2:

  • Change kubernetes.node.cpu.allocatable.cores to float #6130
Packetbeat

Changes in master:

  • Fix minor panic in http parser #6750
  • Reduce memory usage of http parser #6680
Filebeat

Changes in master:

  • Fix panic in log harvester error handling #6801
  • Add missing logger parameter to error message #6745
  • Feature: TCP Input #6700
  • Extracting the UDP Server logic from the UDP input #6439
Auditbeat

Changes in master:

Changes in 6.2:

Processors

Changes in master:

  • Add Condition to check if a list of fields exists #6653
Testing

Changes in master:

  • Metricbeat: Integration tests for haproxy 1.7 and 1.8 #6793
  • Introduce environment variable to skip compose starting environment #6716
  • Add test for error checking in ML #6666
Infrastructure

Changes in master:

  • replace smart quotes to improve license detection #6744
  • Workaround for bug in virtualenv tool on (at least) Amazon Linux. #6698

Repository: elastic/go-sysinfo

Changes in master:

  • Update file headers and license text #9
  • Fix package name #8
  • Use /proc/self on Linux for Self() #7

Repository: elastic/go-libaudit

Changes in master:

  • Add s390 and s390x to runtime arch list #23