New community Beat: Retsbeat
By creating Retsbeat, Valentin Crettaz shows that Beats can be a solution for other use cases, such as real estate. Retsbeat is gathering metrics and insights from Multiple Listing Service (MLS) servers supporting the RETS specification. Real Estate Transaction Standard (RETS) is used in the real estate industry to facilitate the exchange of data among real estate actors such as agents, brokers, owners and many more, over the HTTP protocol. Give it a try and let us know what you think!
Filebeat: Limit the number of harvesters that are started
The config option harvester_limit was added. It can be used in case a large amount of files have to be processed at the same time causing Filebeat to hit max-open-files. This is especially useful when Filebeat is started and has to go through old log files and not only the most recent once.
Filebeat: End Harvester after a predefined time
In Filebeat the config option close_timeout was added. The effect of close_timeout is that a harvester will be stopped after a certain time instead of, for example, at the end of the file with
close_eof. This option can be used in case the network is not fast enough during peak times to ship all the events and data loss is acceptable. Combining
ignore_older will not process the files again.
Filebeat: Ignore BOM from UTF encoded files
On UTF-8/16/32 files it is possible to have a marker at the beginning of the file that decides the byte order (little endian vs big endian). Filebeat used to send this marker along with the first line in the file. This pull request skips over the BOM from the beginning of the files.
libbeat: Accept arrays, slices and pointers as fields in the event
Our normalization function, which is applied to all events when they enter libbeat, is now able to deal with pointers, arrays and slices. In case of pointers, the value is dereferenced, and in case of nil the field is dropped from the event. If a field has an unexpected type, the field is dropped from the event.
libbeat: Rename TLS to SSL in option names
This is done as part of a larger effort to align the security settings across all the Elastic products. Changing TLS to SSL in all output types breaks the compatibility with the older versions. To make it easier for the users to upgrade their configuration file, we updated the migration script as well.
Metricbeat: Optimize float metrics
All the Metricbeat fields that have the float type are reviewed, and their type is adjusted to scaled_float in order to be more efficient represented in Elasticsearch.
Metricbeat: Rename the exported Redis fields
Recently, we came up with a naming convention for the exported event fields, and we are working on changing the fields exported by the official Beats to respect the convention. As renaming the fields is breaking the compatibility with the previous version, we took the chance to adjust the Redis fields before the beta1 release. Here is the pull request with the changes.