Brewing in Beats: Password Keystore

Did you know that Beats 6.1 is already available? Try it and let us know what you think. If you are curious to see the Beats in action, we just published the Getting Started with Beats webinar.

This update includes the changes over the last two weeks.

Password keystore

We have merged the PR which allow users to define sensitive information into an obfuscated data store on disk instead of having them defined in plaintext in the yaml configuration.

# create new keystore to disk 
./metricbeat keystore create
# add a new key to the store.
./metricbeat keystore add elasticsearch_password
# remove a key from the store
./metricbeat keystore remove elasticsearch_password
# list the configured keys without the sensitive information
./metricbeat keystore list

You can then reference the keys from the keystore using the same syntax that we use for the environment variables:

password: "${elasticsearch_password}"

In the current implementation, the passwords are not encrypted into the keystore, only obfuscated. This new feature is planned to be released with the 6.2 release.

Structured logging in libbeat

This PR refactors the logging of libbeat and adds support for structured logging. The new logging implementation is based on zap, which is one of the most efficient structured logging libraries for Golang.

To switch to the JSON format, simply add logging.json: true to the configuration file.

Another enhancement is that the Beats can also log directly to the Windows Event Log. By setting logging.to_eventlog: true, all logs will be written to the Application log. The source name will be the name of the Beat.

Besides this, there are no changes to the user facing logging configuration. The non-JSON logger output has some format differences, but, in general, it will have a more consistent format across outputs.

These changes are only in the master branch at the moment, but we will likely include it in 6.2.

Metricbeat: Read HAProxy metrics over HTTP

Thanks to @jsoriano, the HAProxy module can read the stats over HTTP in addition to the TCP socket. This means HTTP authentication is also supported when reading the stats. The improvement will be available in the 6.2 release.

Other changes:

Repository: elastic/beats

Affecting all Beats

Changes in master:

  • Use docker prospector in examples, fixes #5934 and #5920 #5963
  • Implement zapcore.ObjectMarshaler for common.MapStr #5929
  • Add log output to Windows Event Log #5913
Metricbeat

Changes in 6.1:

  • metricbeat: fix zookeeper fetching file descriptor info incorrectly #5902
Packetbeat

Changes in master:

  • Fix TLS version parsing #5952
Auditbeat

Changes in master:

  • Add support for kMDItemWhereFroms in Auditbeat (#5336) #5951
  • Add back lost audit.file.path.raw field #5943
  • Auditbeat: Add setuid/setgid fields to file_integrity #5933
  • Add blake2b hashing algorithm #5926
Testing

Changes in master:

  • Update Kibana docker image #5928

Changes in 6.1:

  • Filter packages in Jenkins Powershell #5894

Changes in 6.0:

  • Filter packages in Jenkins Powershell #5894
Infrastructure

Changes in master:

  • Fix typo in metricbeat k8s deploy script #5940
Documentation

Changes in 6.1:

  • Fix docs branch in the 6.1 branch #5937