07 December 2017

Brewing in Beats: Recursive file watching on macOS with Auditbeat

By Monica Sarbu

Did you know that Beats 6.0 is already available? Try it and let us know what you think. If you are curious to see the Beats 6.0 in action, we just published the Getting Started with Beats webinar.

This update covers the last two weeks

Auditbeat: recursive file watching on macOS

Auditbeat now supports watching files recursively on macOS. This functionality is based on the FSEvents library. One drawback of FSEvents is that in the case of multiple events on the same file, they have coalesced in a single notification. The PR orders the set of actions in a single event to be meaningful depending if the file existed in the beat database and if it doesn’t exist anymore at the moment of processing the event.

Packetbeat: add_kubernetes_metadata

After Filebeat and Metricbeat, Packetbeat is the next in line to get Kubernetes support. The `add_kubernetes_metada` processor is now able to match the IP/ports from the network traffic with the pods and enhance the events with Kubernetes metadata. This feature was merged into master and is scheduled to be released in 6.2.

Packetbeat: Several TLS support enhancements

Packetbeat now includes a dashboard for the TLS data. It can also report the handshake latency, which is defined as the time spent between first packet and completion of the handshake. Finally, it can now calculate JA3 fingerprints for the client TLS sessions. The JA3 fingerprints are efficient for detecting malware or unauthorized applications.

These features are merged into master and are scheduled to be released with 6.2.

tls-dashboard.png

Filebeat: use the local timezone in the system module

An issue that we had in Filebeat modules was that the Ingest Node pipelines assume the incoming logs have timestamps in UTC. In 6.1, Elasticsearch is getting the ability to parse timestamp in the timezone specified by another field in the message. We now make use of this feature in the Filebeat system module, so the local timezone can be correctly used when decoding the timestamp. This feature will be present in 6.1 but disabled by default.

Filebeat and Metricbeat modules for Logstash monitoring

The Filebeat module for Logstash was merged in time for 6.1. The Metricbeat module got a node_stats metricset with basic event stats, also in time for 6.1.

Other changes:

Repository: elastic/beats

Affecting all Beats

Changes in master:

  • Fix default host to localhost for Kibana dashboard loading #5769
  • Dashboard loading improvements and fixes #5686
  • Fix template loading bug with empty fields #5679
  • Support dashboard loading without Elasticsearch #5653

Changes in 6.1:

  • Fix default host to localhost for Kibana dashboard loading #5769
  • Backport: Dashboard loading improvements and fixes (#5686) #5725
  • Fix template loading bug with empty fields #5681

Changes in 6.0:

  • Ignore docker kill events in `add_docker_metadata` #5788
  • Adjust generated CSV dependency file #5618
Metricbeat

Changes in master:

  • Make logstash node system test more stable #5797
  • Update modules list for Metricbeat #5794
  • Fix Metricbeat System Dashboards #5768
  • Add field network_names of hosts and virtual machines to vsphere module #5732
  • Fix the include top N processes feature for cases where there are fewer processes than N #5729
  • Windows service metricset - write only once into log #5718
  • metricbeat: fix a typo (optained -> obtained) in golang heap module #5716
  • Fix flaky node_stats Logstash tests #5714
  • Skip flaky LS node_stats tests #5713
  • fix connection leak in the mongodb module #5711
  • Allow specifying default metricsets #5675
  • Fix autodiscovery config hashing #5660
  • Add trigger and delay info for windows service metricset #5627
  • Add ceph osd_df to metricbeat #5606
  • Fixed docker diskio bug due to reseting of map. #5582
  • Fix Metricbeat/vsphere - Error "datastore '*' not found" #4879 #4883

Changes in 5.6:

  • Fixed docker diskio bug due to reseting of map. (#5582) #5705

Changes in 6.1:

  • Fix flaky node_stats Logstash tests #5714
  • Fix the include top N processes feature for cases where there are fewer processes than N #5729
  • fix connection leak in the mongodb module #5711

Changes in 6.0:

  • Fix the include top N processes feature for cases where there are fewer processes than N #5729
  • fix connection leak in the mongodb module #5711
Packetbeat

Changes in master:

  • TLS: Remove timestamp field #5670
  • TLS: Populate server field to SNI #5669
  • TLS: Fix missing IP addresses and better skip of encrypted messages #5668
  • TLS: Config flag to include raw certificates #5655
Filebeat

Changes in master:

  • Fixes #5739 - Removing index patterns with '*' from TSVB based visualizations #5741
  • Change pipeline delimiter to {< and >} #5702
Heartbeat

Changes in master:

  • Make dialer based job creation of TCP monitor reusable #5748
Processors

Changes in master:

  • Index pod IP too in `ip_port` kubernetes indexer #5721
Testing

Changes in master:

  • Adds system test to Metricbeat to load dashboards #5770
  • Fix WatcherDie test timings #5759
  • Test all beats can setup their template #5682

Changes in 6.1:

  • Update testing version for 6.1 branch #5777
  • Fix WatcherDie test timings #5759

Changes in 6.0:

  • Make python tests not executables #5449
Infrastructure

Changes in master:

  • Remove leftover glide file in docker module #5771
  • Use SPDX licenses identifiers in CSV dependency file #5733
  • Remove full kibana directory when collecting kibana files #5695
  • Enable specifying pip install options #5693

Changes in 6.1:

  • Use SPDX licenses identifiers in CSV dependency file #5733
Packaging

Changes in master:

  • Fix `:doc_branch:` editing #5756
  • Fix README.md link in packages #5715
  • Backout deb fpm changes #5631

Changes in 6.0:

  • Fix README.md link in packages #5715
Documentation

Changes in master:

  • Add docs for docker prospector #5752
  • Forward port to master: Misc doc changes #5356 and #5678 #5740
  • Fix typos in Heartbeat fields.yml #5724
  • Create module overview page for Metricbeat #5717
  • Add link to relnotes that points to breaking changes #5677
  • Show config for passing credentials rather than command line #5673
  • Add prerelease conditional coding to apt/yum install instructions #5654
  • Clarify docs about manually loading the template #5635
  • Document change to logstash index setting under breaking changes #5634
  • Deprecate the LS document_type config option #5633
  • Added a FAQ topic on disk full conditions #5577

Changes in 6.1:

  • Add docs for docker prospector #5752
  • Backport multiple doc changes to 6.1: #5356 #5633 #5634 #5635 #5673 #5677 #5738

Changes in 6.0:

  • Update port for SSL/TLS on http monitor #5773
  • Backport multiple doc changes to 6.0: #5356 #5577 #5619 #5620 #5633 #5634 #5635 #5654 #5673 #5677 #5736
  • Update filters to processors #5616
  • Add auth fileset options #5678

Repository: elastic/gosigar

Changes in master:

  • Fix ProcState parsing for process names with parentheses #86
  • Prepare 0.6.0 release #85
  • Fix incorrect Mem.Used calculation #82