Brewing in Beats: Recursive file watching on macOS with Auditbeat
Did you know that Beats 6.0 is already available? Try it and let us know what you think. If you are curious to see the Beats 6.0 in action, we just published the Getting Started with Beats webinar.
This update covers the last two weeks
Auditbeat: recursive file watching on macOS
Auditbeat now supports watching files recursively on macOS. This functionality is based on the FSEvents library. One drawback of FSEvents is that in the case of multiple events on the same file, they have coalesced in a single notification. The PR orders the set of actions in a single event to be meaningful depending if the file existed in the beat database and if it doesn’t exist anymore at the moment of processing the event.
Packetbeat: add_kubernetes_metadata
After Filebeat and Metricbeat, Packetbeat is the next in line to get Kubernetes support. The `add_kubernetes_metada` processor is now able to match the IP/ports from the network traffic with the pods and enhance the events with Kubernetes metadata. This feature was merged into master and is scheduled to be released in 6.2.
Packetbeat: Several TLS support enhancements
Packetbeat now includes a dashboard for the TLS data. It can also report the handshake latency, which is defined as the time spent between first packet and completion of the handshake. Finally, it can now calculate JA3 fingerprints for the client TLS sessions. The JA3 fingerprints are efficient for detecting malware or unauthorized applications.
These features are merged into master and are scheduled to be released with 6.2.
Filebeat: use the local timezone in the system module
An issue that we had in Filebeat modules was that the Ingest Node pipelines assume the incoming logs have timestamps in UTC. In 6.1, Elasticsearch is getting the ability to parse timestamp in the timezone specified by another field in the message. We now make use of this feature in the Filebeat system module, so the local timezone can be correctly used when decoding the timestamp. This feature will be present in 6.1 but disabled by default.
Filebeat and Metricbeat modules for Logstash monitoring
The Filebeat module for Logstash was merged in time for 6.1. The Metricbeat module got a node_stats metricset with basic event stats, also in time for 6.1.
Other changes:
Repository: elastic/beats
Affecting all Beats
Changes in master:
- Fix default host to localhost for Kibana dashboard loading #5769
- Dashboard loading improvements and fixes #5686
- Fix template loading bug with empty fields #5679
- Support dashboard loading without Elasticsearch #5653
Changes in 6.1:
- Fix default host to localhost for Kibana dashboard loading #5769
- Backport: Dashboard loading improvements and fixes (#5686) #5725
- Fix template loading bug with empty fields #5681
Changes in 6.0:
Metricbeat
Changes in master:
- Make logstash node system test more stable #5797
- Update modules list for Metricbeat #5794
- Fix Metricbeat System Dashboards #5768
- Add field network_names of hosts and virtual machines to vsphere module #5732
- Fix the include top N processes feature for cases where there are fewer processes than N #5729
- Windows service metricset - write only once into log #5718
- metricbeat: fix a typo (optained -> obtained) in golang heap module #5716
- Fix flaky node_stats Logstash tests #5714
- Skip flaky LS node_stats tests #5713
- fix connection leak in the mongodb module #5711
- Allow specifying default metricsets #5675
- Fix autodiscovery config hashing #5660
- Add trigger and delay info for windows service metricset #5627
- Add ceph osd_df to metricbeat #5606
- Fixed docker diskio bug due to reseting of map. #5582
- Fix Metricbeat/vsphere - Error "datastore '*' not found" #4879 #4883
Changes in 5.6:
- Fixed docker diskio bug due to reseting of map. (#5582) #5705
Changes in 6.1:
- Fix flaky node_stats Logstash tests #5714
- Fix the include top N processes feature for cases where there are fewer processes than N #5729
- fix connection leak in the mongodb module #5711
Changes in 6.0:
- Fix the include top N processes feature for cases where there are fewer processes than N #5729
- fix connection leak in the mongodb module #5711
Packetbeat
Changes in master:
- TLS: Remove timestamp field #5670
- TLS: Populate server field to SNI #5669
- TLS: Fix missing IP addresses and better skip of encrypted messages #5668
- TLS: Config flag to include raw certificates #5655
Filebeat
Changes in master:
- Fixes #5739 - Removing index patterns with '*' from TSVB based visualizations #5741
- Change pipeline delimiter to {< and >} #5702
Heartbeat
Changes in master:
- Make dialer based job creation of TCP monitor reusable #5748
Processors
Changes in master:
- Index pod IP too in `ip_port` kubernetes indexer #5721
Testing
Changes in master:
- Adds system test to Metricbeat to load dashboards #5770
- Fix WatcherDie test timings #5759
- Test all beats can setup their template #5682
Changes in 6.1:
Changes in 6.0:
- Make python tests not executables #5449
Infrastructure
Changes in master:
- Remove leftover glide file in docker module #5771
- Use SPDX licenses identifiers in CSV dependency file #5733
- Remove full kibana directory when collecting kibana files #5695
- Enable specifying pip install options #5693
Changes in 6.1:
- Use SPDX licenses identifiers in CSV dependency file #5733
Packaging
Changes in master:
Changes in 6.0:
- Fix README.md link in packages #5715
Documentation
Changes in master:
- Add docs for docker prospector #5752
- Forward port to master: Misc doc changes #5356 and #5678 #5740
- Fix typos in Heartbeat fields.yml #5724
- Create module overview page for Metricbeat #5717
- Add link to relnotes that points to breaking changes #5677
- Show config for passing credentials rather than command line #5673
- Add prerelease conditional coding to apt/yum install instructions #5654
- Clarify docs about manually loading the template #5635
- Document change to logstash index setting under breaking changes #5634
- Deprecate the LS document_type config option #5633
- Added a FAQ topic on disk full conditions #5577
Changes in 6.1:
- Add docs for docker prospector #5752
- Backport multiple doc changes to 6.1: #5356 #5633 #5634 #5635 #5673 #5677 #5738
Changes in 6.0:
- Update port for SSL/TLS on http monitor #5773
- Backport multiple doc changes to 6.0: #5356 #5577 #5619 #5620 #5633 #5634 #5635 #5654 #5673 #5677 #5736
- Update filters to processors #5616
- Add auth fileset options #5678
Repository: elastic/gosigar
Changes in master: