Beats version 6.4.0

View commits

Known issue

Due to a packaging mistake, the modules.d configuration directory is installed in the wrong path in the Metricbeat DEB and RPM packages. This issue results in an empty list when you run metricbeat modules list and failures when you try to enable or disable modules. To work around this issue, run the following command:

sudo cp -r /usr/share/metricbeat/modules.d /etc/metricbeat/

This issue affects all new installations on DEB and RPM. Upgrades will run, but use old configurations defined in the modules.d directory from the previous installation.

The issue will be fixed in the 6.4.1 release.

Breaking changes

Affecting all Beats

  • Set default kafka version to 1.0.0 in kafka output. Older versions are still supported by configuring the version setting. Minimally supported version is 0.11 (older versions might work, but are untested). 7025

Heartbeat

  • Rename http.response.status to http.response.status_code to align with ECS. 7274
  • Remove type field as not needed. 7307

Metricbeat

  • Fixed typo in values for state_container status.phase, from terminate to terminated. 6916
  • RabbitMQ management plugin path is now configured at the module level instead of having to do it in each of the metricsets. New management_path_prefix option should be used now 7074
  • RabbitMQ node metricset only collects metrics of the instance it connects to, node.collect: cluster can be used to collect all nodes as before. 6556 6971
  • Change http/server metricset to put events by default under http.server and prefix config options with server.. 7100
  • Disable dedotting in docker module configuration. This will change the out-of-the-box behaviour, but not the one of already configured instances. 7485
  • Fix typo in etcd/self metricset fields from *.bandwithrate to *.bandwidthrate. 7456
  • Changed the definition of the system.cpu.total.pct and system.cpu.total.norm.cou fields to exclude the IOWait time. 7691

Bugfixes

Affecting all Beats

  • Error out on invalid Autodiscover template conditions settings. 7200
  • Allow to override the ignore_above option when defining new field with the type keyword. 7238
  • Fix a panic on the Dissect processor when we have data remaining after the last delimiter. 7449
  • When we fail to build a Kubernetes' indexer or matcher we produce a warning but we don’t add them to the execution. 7466
  • Fix default value for logging.files.keepfiles. It was being set to 0 and now it’s set to the documented value of 7. 7494
  • Retain compatibility with older Docker server versions. 7542
  • Fix errors unpacking configs modified via CLI by ignoring -E key=value pairs with missing value. 7599

Auditbeat

  • Allow auditbeat setup to run without requiring elevated privileges for the audit client. 7111
  • Fix goroutine leak that occurred when the auditd module was stopped. 7163

Filebeat

  • Fix a data race between stopping and starting of the harvesters. 6879
  • Fix an issue when parsing ISO8601 dates with timezone definition 7367
  • Fix Grok pattern of MongoDB module. 7568
  • Fix registry duplicates and log resending on upgrade. 7634

Metricbeat

  • Fix Windows service metricset when using a 32-bit binary on a 64-bit OS. 7294
  • Do not report Metricbeat container host as hostname in Kubernetes deployment. 7199
  • Ensure metadata updates don’t replace existing pod metrics. 7573
  • Fix kubernetes pct fields reporting. 7677
  • Add support for new kube_node_status_condition in Kubernetes state_node. 7699

Added

Affecting all Beats

  • Add dissect processor. 6925
  • Add IP-addresses and MAC-addresses to add_host_metadata. 6878
  • Added a seccomp (secure computing) filter on Linux that whitelists the necessary system calls used by each Beat. 5213
  • Ship fields.yml as part of the binary 4834
  • Added options to dev-tools/cmd/dashboards/export_dashboard.go: -indexPattern to include index-pattern in output, -quiet to be quiet. 7101
  • Add Indexer indexing by pod uid. Enable pod uid metadata gathering in add_kubernetes_metadata. Extended Matcher log_path matching to support volume mounts 7072
  • Add default_fields to Elasticsearch template when connecting to Elasticsearch >= 7.0. 7015
  • Add support for loading a template.json file directly instead of using fields.yml. 7039
  • Add support for keyword multifields in field.yml. 7131
  • Add experimental Jolokia Discovery autodiscover provider. 7141
  • Add owner object info to Kubernetes metadata. 7231
  • Add Beat export dashboard command. 7239
  • Add support for docker autodiscover to monitor containers on host network 6708
  • Add ability to define input configuration as stringified JSON for autodiscover. 7372
  • Add processor definition support for hints builder 7386
  • Add support to disable html escaping in outputs. 7445
  • Refactor error handing in schema.Apply(). 7335
  • Add additional types to Kubernetes metadata 7457
  • Add module state reporting for Beats Monitoring. 7075
  • Release the rename processor as GA. 7656
  • Add support for Openstack Nova in add_cloud_metadata processor. 7663
  • Add support to set Beats services to automatic-delayed start on Windows. 8711

Auditbeat

  • Added XXH64 hash option for file integrity checks. 7311
  • Added the show auditd-rules and show auditd-status commands to show kernel rules and status. 7114
  • Add Kubernetes specs for auditbeat file integrity monitoring 7642

Filebeat

  • Add Kibana module with log fileset. 7052
  • Support MySQL 5.7.19 by mysql/slowlog 6969
  • Correctly join partial log lines when using docker input. 6967
  • Add support for TLS with client authentication to the TCP input 7056
  • Converted part of pipeline from treafik/access metricSet to dissect to improve efficiency. 7209
  • Add GC fileset to the Elasticsearch module. 7305
  • Add Audit log fileset to the Elasticsearch module. 7365
  • Add Slow log fileset to the Elasticsearch module. 7473
  • Add deprecation fileset to the Elasticsearch module. 7474
  • Add convert_timezone option to Kafka module to convert dates to UTC. 7546 7578
  • Add patterns for kafka 1.1 logs. 7608
  • Move debug messages in tcp input source 7712

Metricbeat

  • Add experimental Elasticsearch index metricset. 6881
  • Add dashboards and visualizations for haproxy metrics. 6934
  • Add Jolokia agent in proxy mode. 6475
  • Add message rates to the RabbitMQ queue metricset 6442 6606
  • Add exchanges metricset to the RabbitMQ module 6442 6607
  • Add Elasticsearch index_summary metricset. 6918
  • Add shard metricset to Elasticsearch module. 7006
  • Add apiserver metricset to Kubernetes module. 7059
  • Add maxmemory to redis info metricset. 7127
  • Set guest as default user in RabbitMQ module. 7107
  • Add postgresql statement metricset. 7048 7060
  • Update state_container metricset to support latest kube-state-metrics version. 7216
  • Add TLS support to MongoDB module. 7401
  • Added Traefik module with health metricset. 7413
  • Add Elasticsearch ml_job metricsets. 7196
  • Add support for bearer token files to HTTP helper. 7527
  • Add Elasticsearch index recovery metricset. 7225
  • Add locks, global_locks, oplatencies and process fields to status metricset of MongoDB module. 7613
  • Run Kafka integration tests on version 1.1.0 7616
  • Release raid and socket metricset from system module as GA. 7658
  • Release elasticsearch module and all its metricsets as beta. 7662
  • Release munin and traefik module as beta. 7660
  • Add envoyproxy module. 7569
  • Release prometheus collector metricset as GA. 7660
  • Add Elasticsearch cluster_stats metricset. 7638
  • Added basepath setting for HTTP-based metricsets 7700
  • Add couchdb module. 9406

Packetbeat

  • The process monitor now reports the command-line for all processes, under Linux and Windows. 7135
  • Updated the TLS protocol parser with new cipher suites added to TLS 1.3. 7455
  • Flows are enriched with process information using the process monitor. 7507
  • Added UDP support to process monitor. 7571

Deprecated

Metricbeat

  • Kubernetes state_container cpu.limit.nanocores and cpu.request.nanocores have been deprecated in favor of cpu.*.cores. 6916