Brewing in Beats: Load Auditbeat rules from a configuration directory

Welcome to Brewing in Beats! With these weekly series, we're keeping you up to date with what's new in Beats, including the latest commits and releases.

Did you know that Beats 6.3 is already available? Try it and let us know what you think. 

Auditbeat: load audit rules

Auditbeat is getting support for loading audit rules from /etc/auditbeat/audit.rules.d/*.conf. Users can define their rules separately from the config file so that they don’t have to worry about how strings are formatted in YAML, indentation, or escaping.

Automatically enrich Kubernetes module events

This has been a popular request among our users, starting with 6.4, Metricbeat Kubernetes module will automatically enrich all metrics coming out of it with metadata (labels and annotations) from the resource being monitored.

All changes

Repository: elastic/beats

Metricbeat

Changes in 6.3:

  • Add bearer_token_file parameter to HTTP helper #7527

Changes in master:

  • XPack helper for naming monitoring indices #7586
  • Add bearer_token_file parameter to HTTP helper #7527
  • Automatically enrich Kubernetes module events #7470
  • Add basic index recovery metricset #7225
  • Add ml_job metricset to Elasticsearch module #7196
Packetbeat

Changes in master:

  • Add UDP support to packetbeat's process monitor #7571
  • nfs: add support for v4.2 operations and error codes #7397
Filebeat

Changes in master:

  • Add missing changlelog entry about missing logs #7597
  • Filebeat: Add option to convert kafka module timezones to UTC #7578
  • Fix Grok pattern of MongoDB module #7568
  • Update field naming for Elasticsearch slowlog fileset #7556
  • Update field names for Elasticsearch audit fileset #7555
  • Update field naming for Elasticsearch server fileset #7554
  • [Filebeat, ES module] Follow up to improve Server fileset #7549
Auditbeat

Changes in master:

  • Use a separate audit client for lost event monitoring #7561
  • Allow to specify auditd rules in separate files #7331
Testing

Changes in master:

  • Disable headers check in generator tests #7580
  • Add tests to heartbeat's look.go #7544
Documentation

Changes in master:

  • Add conditional coding to security topic #7602
  • Note that python 2 is required for generate.py #7588
  • Update Filebeat module dev guide #7585
  • add conditionals to support apm-server docs update #7572
  • Clarify docs to indicate where processors are valid in the config #7085

Repository: elastic/logstash

  • fix broken classpath when whitespaces are in the path #9832
  • clean backport of #9622 that missed the 6.3 branch
Documentation

Changes in master:

  • [DOCS] Removes alternative docker pull example #9831
  • [DOCS] Clarify methods for stopping Logstash #9828

Repositories under elastic/logstash-plugins

logstash-plugins/logstash-input-file - 4.1.4

  • Fixed a regression where files discovered after first discovery were not always read from the beginning. Applies to tail mode only. #198
  • Added much better support for file rotation schemes of copy/truncate and rename cascading. Applies to tail mode only.
  • Added support for processing files over remote mounts e.g. NFS. Before, it was possible to read into memory allocated but not filled with data resulting in ASCII NUL (0) bytes in the message field. Now, files are read up to the size as given by the remote filesystem client. Applies to tail and read modes.

logstash-plugins/logstash-output-graphite - 3.1.6

  • Fixed exception handling during socket writing to prevent logstash termination #33

logstash-plugins/logstash-input-beats - 5.1.2

  • Upgrade to Gradle 4.8.1 #334
  • Explicitly set Java compiler encoding to UTF-8
  • Fix sending of acks when sequence number of batch does not start with 1 #342

Repository: elastic/logstash-docs

Changes in versioned_plugin_docs:

  • auto generated update of versioned plugin documentation #586

Changes in master:

  • updated master docs #599
  • Add azure_event_hubs input to the plug-in docs #576

Repository: elastic/go-ucfg

Changes in master:

  • Prepare 0.6.1 #114
  • README syntax highlighting #113