05 April 2018

Brewing in Beats: Exclusive Filebeat stdin input

By Monica Sarbu

Did you know that Beats 6.2 is already available? Try it and let us know what you think. If you are curious to see the Beats in action, check out the Getting Started with Beats webinar.

Exclusive Filebeat stdin input

With PR #6463, Filebeat will not accept any other input if stdin is configured. If the stdin input is used, Filebeat will stop once the end of the input stream is reached.

Spooling to disk, in progress

Spooling to disk adds a persistent queue to all beats publisher pipeline. Any event to be published will be written to the spool first and only forwarded to Elasticsearch or Logstash afterwards. With spooling to disk, events not yet published are still available after beats have been restarted or in case outputs have been stalled for a long time due to network issues.

The PR #6581 is actively reviewed. Right now we’re running many manual tests, looking for potential bugs, failures and memory leaks. As this feature is quite big, we still hope to have it available in beta for 6.3, so to allow users to test early. Additional implementation details are available in the PR.

Community Beat: GCSBeat

@josephlewis42 from Google created GCSBeat to read logs/data from Google Cloud Storage (GCS) buckets. The Beat reads JSON objects or raw text from files in a bucket and forwards them to a Beats output. You can use it in use-cases like: 

  • Read Stackdriver logs from a GCS bucket into Elastic. 
  • Read gzipped logs from cold-storage into Elastic. 
  • Restore data from an Elastic dump. 
  • Watch files on a local path (possibly a mounted GCS bucket) and upload them. 
  • Parse JSON logs and upload them.

Note: While this project is partially maintained by Google, this is not an official Google product.

Documentation

Repository: elastic/beats

Changes in master:

  • Fixes autodiscover's appender configuration example #6720
  • Set all couchbase metricsets as default in metricbeat module. #6683
  • Set default metricsets in vsphere module #6676
  • Set default metricset in windows module to `service` #6675
  • Set default metricset in zookeeper module #6674
  • Set `status` as default metricset in Apache module #6673

Changes in 6.2:

  • Fix config reload example #6567

Other changes

Repository: elastic/beats

Metricbeat

Changes in master:

  • Mark Metricbeat kubernetes.event metricset as beta. #6715
  • Set default metricset for uwsgi module #6688
  • Set namespace as default metricset in Aerospike module #6669
  • Metricbeat: Use system-wide ticks to calculate docker cpu usage #6608

Changes in 6.2:

  • Fix system process metricset for kernel processes #6224
Filebeat

Changes in master:

  • Prevent to start filebeat when STDIN is used with other inputs #6463
Winlogbeat

Changes in master:

  • Use random sentence instead of random characters in test #6690

Changes in 6.2:

  • Use random sentence instead of random characters in test #6690
Auditbeat

Changes in master:

Changes in 6.2:

Repository: elastic/go-sysinfo

Changes in master:

  • Added LICENSE file #6
  • Add Memory and CPUTime to Host interface #5

Repository: elastic/gosigar

Changes in master:

  • Add missing runtime import for FreeBSD #104
  • Recover as workaround for panic when reading proc files #103

Repository: elastic/go-libaudit

Changes in master:

  • Prepare release v0.1.0 #22
  • Hex decode name value in PATH record #21