Beats version 7.4.0

View commits

Breaking changes

Affecting all Beats

  • Update to Golang 1.12.7. 12931
  • Remove in_cluster configuration parameter for Kuberentes, now in-cluster configuration is used only if no other kubeconfig is specified 13051

Auditbeat

  • Socket dataset: New implementation using Kprobes for finer-grained monitoring and UDP support. 13058

Filebeat

  • Fix a race condition in the TCP input when close the client socket. 13038
  • cisco/asa fileset: Renamed log.original to event.original and cisco.asa.list_id to cisco.asa.rule_name. 13286
  • cisco/asa fileset: Fix parsing of 302021 message code. 13476

Metricbeat

  • Add new Dashboard for PostgreSQL database stats 13187
  • Add new dashboard for CouchDB database 13198
  • Add new dashboard for Ceph cluster stats 13216
  • Add new dashboard for Aerospike database stats 13217
  • Add new dashboard for Couchbase cluster stats 13212
  • Add new dashboard for Prometheus server stats 13126
  • Add statistic option into cloudwatch metricset. If there is no statistic method specified, default is to collect Average, Sum, Maximum, Minimum and SampleCount. 12370 12840
  • Fix rds metricset dashboard. 13721

Functionbeat

  • Separate management and functions in Functionbeat. 12939

Bugfixes

Affecting all Beats

  • ILM: Use GET instead of HEAD when checking for alias to expose detailed error message. 12886
  • Fix unexpected stops on docker autodiscover when a container is restarted before cleanup_timeout. 12962 13127
  • Fix some incorrect types and formats in field.yml files. 13188
  • Load DLLs only from Windows system directory. 13234 13384
  • Fix mapping for kubernetes.labels and kubernetes.annotations in add_kubernetes_metadata. 12638 13226
  • Fix case insensitive regular expressions not working correctly. 13250

Auditbeat

  • Host dataset: Export Host fields to gob encoder. 12940

Filebeat

  • Fix filebeat autodiscover fileset hint for container input. 13296
  • Fix incorrect references to index patterns in AWS and CoreDNS dashboards. 13303
  • Fix timezone parsing of system module ingest pipelines. 13308
  • Fix timezone parsing of elasticsearch module ingest pipelines. 13367
  • Change iis url path grok pattern from URIPATH to NOTSPACE. 12710 13225 7951 13378
  • Add timezone information to apache error fileset. 12772 13304
  • Fix timezone parsing of nginx module ingest pipelines. 13369
  • Allow path variables to be used in files loaded from modules.d. 13184
  • Fix incorrect field references in envoyproxy dashboard 13420 13421

Heartbeat

  • Fix integer comparison on JSON responses. 13348

Metricbeat

  • Ramdisk is not filtered out when collecting disk performance counters in diskio metricset 12814 12829
  • Fix redis key metricset dashboard references to index pattern. 13303
  • Check if fields in DBInstance is nil in rds metricset. 13294 13037
  • Fix silent failures in kafka and prometheus module. 13353 13252
  • Fix module-level fields in Kubernetes metricsets. 13433 13544
  • Fix panic in Redis Key metricset when collecting information from a removed key. 13426
  • In the elasticsearch/node_stats metricset, if xpack is enabled, make parsing of ES node load average optional as ES on Windows doesn’t report load average. 12866
  • Print errors that were being omitted in vSphere metricsets. 12816
  • Fix issue with aws cloudwatch module where dimensions and/or namespaces that contain space are not being parsed correctly 13389
  • Fix reporting empty events in cloudwatch metricset. 13458
  • Fix data race affecting config validation at startup. 13005

Packetbeat

  • Fix parsing the extended RCODE in the DNS parser. 12805

Functionbeat

  • Fix Cloudwatch logs timestamp to use timestamp of the log record instead of when the record was processed 13291
  • Look for the keystore under the correct path. 13332

Added

Affecting all Beats

  • Add support for reading the network.iana_number field by default to the community_id processor. 12701
  • Add a check so alias creation explicitely fails if there is an index with the same name. 13070
  • Update kubernetes watcher to use official client-go libraries. 13051
  • Add support for unix epoch time values in the timestamp processor. 13319
  • add_host_metadata is now GA. 13148
  • Add an ignore_missing configuration option the drop_fields processor. 13318
  • Add registered_domain processor for deriving the registered domain from a given FQDN. 13326
  • Add support for RFC3339 time zone offsets in JSON output. 13227
  • Added monitoring.cluster_uuid setting to associate Beat data with specified ES cluster in Stack Monitoring UI. 13182

Filebeat

  • Add netflow dashboards based on Logstash netflow. 12857
  • Parse more fields from Elasticsearch slowlogs. 11939
  • Update module pipelines to enrich events with autonomous system fields. 13036
  • Add module for ingesting IBM MQ logs. 8782
  • Add S3 input to retrieve logs from AWS S3 buckets. 12640 12582
  • Add aws module s3access metricset. 13170 12880
  • Update Suricata module to populate ECS DNS fields and handle EVE DNS version 2. 13320 13329
  • Update PAN-OS fileset to use the ECS NAT fields. 13320 13330
  • Add fields to the Zeek DNS fileset for ECS DNS. 13320 13324
  • Add container image in Kubernetes metadata 13356 12688
  • Add module for ingesting Cisco FTD logs over syslog. 13286

Heartbeat

  • Record HTTP body metadata and optionally contents in http.response.body.* fields. 13022

Metricbeat

  • Add Kubernetes proxy dashboard to Kubernetes module 12734
  • Add Kubernetes controller manager dashboard to Kubernetes module 12744
  • Add metrics to kubernetes apiserver metricset. 12922
  • Add Kubernetes scheduler dashboard to Kubernetes module 12749
  • Collect client provided name for rabbitmq connection. 12851 12852
  • Add support to load default aws config file to get credentials. 12727 12708
  • Add statistic option into cloudwatch metricset. 12370 12840
  • Add support for kubernetes cronjobs 13001
  • Add cgroup memory stats to docker/memory metricset 12916
  • Add AWS elb metricset. 12952 11701
  • Add AWS ebs metricset. 13167 11699
  • Add metricset.period field with the configured fetching period. 13242 12616
  • Add rate metrics for ec2 metricset. 13203
  • Add Performance metricset to Oracle module 12547
  • Use DefaultMetaGeneratorConfig in MetadataEnrichers to initialize configurations 13414
  • Add module for statsd. 13109

Packetbeat

  • Update DNS protocol plugin to produce events with ECS fields for DNS. 13320 13354

Functionbeat

  • Add timeout option to reference configuration. 13351
  • Configurable tags for Lambda functions. 13352
  • Add input for Cloudwatch logs through Kinesis. 13317
  • Enable Logstash output. 13345

Winlogbeat

  • Add support for event ID 4634 and 4647 to the Security module. 12906
  • Add network.community_id to Sysmon network events (event ID 3). 13034
  • Add event.module to Winlogbeat modules. 13047
  • Add event.category: process and event.type: process_start/process_end to Sysmon process events (event ID 1 and 5). 13047
  • Add support for event ID 4672 to the Security module. 12975
  • Add support for event ID 22 (DNS query) to the Sysmon module. 12960
  • Add support for event ID 4634 and 4647 to the Security module. 12906
  • Add network.community_id to Sysmon network events (event ID 3). 13034
  • Add event.module to Winlogbeat modules. 13047
  • Add event.category: process and event.type: process_start/process_end to Sysmon process events (event ID 1 and 5). 13047
  • Add support for event ID 4672 to the Security module. 12975
  • Add support for event ID 22 (DNS query) to the Sysmon module. 12960
  • Add certain winlog.event_data.* fields to the index template. 13700 13704