Brewing in Beats: Track network connections

Happy New Year everyone! 🍾

New community Beat: Cloudflarebeat

Al Lefebvre created Cloudflarebeat to fetch Cloudflare logs via the Enterprise Log Share API. It works by requesting the logs within a time range, ending 30 minutes ago latest, and then saving the gzip content in a local file. The JSON log entries are read from the file, processed and indexed in Elasticsearch.

Metricbeat: Track network connections

The new system.socket metricset reports an event for each TCP socket that it sees on Linux systems. For each TCP socket, it reports the process that opened the socket, the local and remote IPs involved in the communication, and the direction (incoming, outgoing or listening). It can also perform a reverse lookup on the remote IP.

It works by polling the Linux kernel to get the sockets, so a short polling interval is recommended if catching short lived connections is required.

Once the data is in Elasticsearch, you can visualize the network connections using Graph:


Filebeat: Per module Kibana dashboards

With the addition of Filebeat modules, the Kibana dashboards are spread between the modules, and they can be gathered by performing the make collect command. As a consequence, the _meta/kibana directory is no longer checked in as it can be easily generated. The same approach was applied to Metricbeat as well.

Filebeat: Add integration tests for modules

The integration tests consist of loading the template and the pipeline in Elasticsearch, reading the log files from the module test directory and then check if the expected data is available in Elasticsearch.