05 September 2016

Brewing in Beats: New community Beat for RETS

Von Monica Sarbu

New community Beat: Retsbeat

By creating Retsbeat, Valentin Crettaz shows that Beats can be a solution for other use cases, such as real estate. Retsbeat is gathering metrics and insights from Multiple Listing Service (MLS) servers supporting the RETS specification. Real Estate Transaction Standard (RETS) is used in the real estate industry to facilitate the exchange of data among real estate actors such as agents, brokers, owners and many more, over the HTTP protocol. Give it a try and let us know what you think!

Filebeat: Limit the number of harvesters that are started

The config option harvester_limit was added. It can be used in case a large amount of files have to be processed at the same time causing Filebeat to hit max-open-files. This is especially useful when Filebeat is started and has to go through old log files and not only the most recent once.

Filebeat: End Harvester after a predefined time

In Filebeat the config option close_timeout was added. The effect of close_timeout is that a harvester will be stopped after a certain time instead of, for example, at the end of the file with close_eof. This option can be used in case the network is not fast enough during peak times to ship all the events and data loss is acceptable. Combining close_timeout with ignore_older will not process the files again.

Filebeat: Ignore BOM from UTF encoded files

On UTF-8/16/32 files it is possible to have a marker at the beginning of the file that decides the byte order (little endian vs big endian). Filebeat used to send this marker along with the first line in the file. This pull request skips over the BOM from the beginning of the files.

libbeat: Accept arrays, slices and pointers as fields in the event

Our normalization function, which is applied to all events when they enter libbeat, is now able to deal with pointers, arrays and slices. In case of pointers, the value is dereferenced, and in case of nil the field is dropped from the event. If a field has an unexpected type, the field is dropped from the event.

libbeat: Rename TLS to SSL in option names

This is done as part of a larger effort to align the security settings across all the Elastic products. Changing TLS to SSL in all output types breaks the compatibility with the older versions. To make it easier for the users to upgrade their configuration file, we updated the migration script as well.

Metricbeat: Optimize float metrics

All the Metricbeat fields that have the float type are reviewed, and their type is adjusted to scaled_float in order to be more efficient represented in Elasticsearch.

Metricbeat: Rename the exported Redis fields

Recently, we came up with a naming convention for the exported event fields, and we are working on changing the fields exported by the official Beats to respect the convention. As renaming the fields is breaking the compatibility with the previous version, we took the chance to adjust the Redis fields before the beta1 release. Here is the pull request with the changes.

Packetbeat: Deprecate Topology Mapping

We’re deprecating the Packetbeat functionality that maps IP address to host names using Elasticsearch or Redis as a backend. While this can be useful functionality, the current implementation is limited (only works with the Elasticsearch and Redis outputs) and it's done in the wrong place (outputs). We do hope to have a better implementation before or soon after removing the current one.