12 July 2016

Brewing in Beats: Set configuration options from the CLI

By Tudor Golubenco

Welcome to Brewing in Beats! With this series, we're keeping you up to date with all that's new in Beats, from the details of work in progress pull requests to releases and learning resources.

New community Beat: Saltbeat

Salt is an open source automation framework, similar to Ansible for example. Saltbeat is a forwarder for messages from the salt master event bus to Logstash and Elasticsearch.

Set configuration options via the CLI

The Beats are getting a -E flag, similar to the Elaticsearch -E option, that allows overwriting configuration options from the command line. For example, you can now quickly enable the console output by adding -E output.console.pretty=true.

Combine processor conditions with and/or/not

The Beats processors (a.k.a generic filters), which are new in 5.0, now support combining conditions with logical operators. This means you can express complex conditions for filtering out the interesting events. For example, to drop all events that have codes 200 or 404 on a given URL:

    - equals.request_url: “/test”
    - or:
      - equals.http.code: 404
      - equals.http.code: 200

Logging verbosity cleanup + more internal metrics

We started with a bug report (thanks Lee) that highlighted how sometimes we fail to provide the operator with important information and started doing a largish effort to rethink what we log and with what verbosity level. Part of this, we reduced the numbers of WARN and INFO messages that we produce, so we can switch the default log level from ERROR to INFO. To compensate, we replaced the most verbose warnings/infos with internal metrics. These internal metrics are dumped periodically in logs and will provide data for the future central monitoring system.

Environment variable expansion based on go-ucfg

The previous implementation for environment variable expansion in our configuration files was done via textual replacement before parsing. This was easy and elegant to implement, but could lead issues when the ${var} sequence shows up accidentally (e.g. in a password field). By passing the expansion responsibility to our new go-ucfg library, we can be more selective on where we accept variable expansions.

This also comes with a breaking change: we used to replace unresolved variables with an empty string, the shell way. Now unresolved variables result in a configuration error, which should help catching configuration bugs earlier.

Specify multiple configuration files

Another improvement that we owe to go-ucfg is that you can now specify multiple configuration files by repeating the -c flag. You can use this, for example, for setting defaults in a base configuration file, and overwrite settings via local configs.

Filebeat fix for very quick file rotation

In case a file was renamed after the state was read but before the file was opened by the harvester it could happen that the wrong file was opened. This lead to the issue, that the wrong file was read including reporting the wrong state for a file. In general this was rather unlikely to happen, but can happen when scan_frequency is very low and number of files rotated is very high.