Beats version 7.5.0edit

View commits

Breaking changesedit

Affecting all Beats

  • By default, all Beats-created files and folders will have a umask of 0027 (on POSIX systems). 14119

Filebeat

Heartbeat

  • JSON/Regex checks against HTTP bodies will only consider the first 100MiB of the HTTP body to prevent excessive memory usage. 14223

Metricbeat

Bugfixesedit

Affecting all Beats

  • Disable add_kubernetes_metadata if no matchers found. 13709
  • Better wording for xpack beats when the _xpack endpoint is not reachable. 13771
  • Kubernetes watcher at add_kubernetes_metadata fails with StatefulSets 13905
  • Fix panics that could result from invalid TLS certificates. This can affect Beats that connect over TLS or Beats that accept connections over TLS and validate client certificates. 14146
  • Fix memory leak in kubernetes autodiscover provider and add_kubernetes_metadata processor happening when pods are terminated without sending a delete event. 14259
  • Fix kubernetes metaGenerator.ResourceMetadata when parent reference controller is nil 14320 14329

Auditbeat

  • Socket dataset: Fix start errors when IPv6 is disabled on the kernel. 13953 13966

Filebeat

  • Fix a denial of service flaw when parsing malformed DSA public keys in Go. If Filebeat is configured to accept incoming TLS connections with client authentication enabled, a remote attacker could cause the Beat to stop processing events. (CVE-2019-17596) See https://www.elastic.co/community/security/
  • Fix timezone parsing of rabbitmq module ingest pipelines. 13879
  • Fix conditions and error checking of date processors in ingest pipelines that use event.timezone to parse dates. 13883
  • Fix timezone parsing of Cisco module ingest pipelines. 13893
  • Fix timezone parsing of logstash module ingest pipelines. 13890
  • Fix timezone parsing of iptables, mssql and panw module ingest pipelines. 13926
  • Fixed increased memory usage with large files when multiline pattern does not match. 14068
  • Fix azure fields names. 14098 14132
  • Fix calculation of network.bytes and network.packets for bi-directional netflow events. 14111
  • Accept - as http.response.body.bytes in apache module. 14137
  • Fix timezone parsing of MySQL module ingest pipelines. 14130
  • Improve error message in s3 input when handleSQSMessage failed. 14113
  • Fix race condition in S3 input plugin. 14359

Heartbeat

  • Fix storage of HTTP bodies to work when JSON/Regex body checks are enabled. 14223

Metricbeat

  • Fix a denial of service flaw when parsing malformed DSA public keys in Go. If Metricbeat is configured to accept incoming TLS connections with client authentication enabled, a remote attacker could cause the Beat to stop processing events. (CVE-2019-17596) See https://www.elastic.co/community/security/
  • PdhExpandWildCardPathW will not expand counter paths in 32 bit windows systems, workaround will use a different function. 12590 12622
  • Fix docker.cpu.system.pct calculation by using the reported number online cpus instead of the number of metrics per cpu. 13691
  • Change kubernetes.event.message to text 13964
  • Fix performance counter values for windows/perfmon metricset.https://github.com/elastic/beats/issues/14036[14036] 14039 14108
  • Add FailOnRequired when applying schema and fix metric names in mongodb metrics metricset. 14143
  • Convert indexed ms-since-epoch timestamp fields in elasticsearch/ml_job metricset to ints from float64s. 14220 14222
  • Fix ARN parsing function to work for ELB ARNs. 14316
  • Update azure configuration example. 14224
  • Limit some of the error messages to the logs only 14317 14327
  • Fix cloudwatch metricset with names and dimensions in config. 14376 14391
  • Fix marshaling of ms-since-epoch values in elasticsearch/cluster_stats metricset. 14378

Packetbeat

  • Fix parsing of the HTTP host header when it contains a port or an IPv6 address. 14215

Addededit

Affecting all Beats

  • Fail with error when autodiscover providers have no defined configs. 13078
  • Add autodetection mode for add_docker_metadata and enable it by default in included configuration fileshttps://github.com/elastic/beats/pull/13374[13374]
  • Add autodetection mode for add_kubernetes_metadata and enable it by default in included configuration files. 13473
  • Use less restrictive API to check if template exists. 13847
  • Do not check for alias when setup.ilm.check_exists is false. 13848
  • Add support for numeric time zone offsets in timestamp processor. 13902
  • Add condition to the config file template for add_kubernetes_metadata 14056
  • Marking Central Management deprecated. 14018
  • Add keep_null setting to allow Beats to publish null values in events. 5522 13928
  • Add shared_credential_file option in aws related config for specifying credential file directory. 14157 14178
  • Ensure that init containers are no longer tailed after they stop. 14394
  • Libbeat HTTP’s Server can listen to a unix socket using the unix:///tmp/hello.sock syntax. 13655
  • Libbeat HTTP’s Server can listen to a Windows named pipe using the npipe:///hello syntax. 13655
  • Adding new Enterprise license type to the licenser. 14246
  • Add endpoint config in AWS config to support using custom endpoint accessing AWS APIs. 16245 16263

Auditbeat

  • Socket: Add DNS enrichment. 14004

Filebeat

  • Add support for virtual host in Apache access logs 12778
  • Update CoreDNS module to populate ECS DNS fields. 13320 13505
  • Parse query steps in PostgreSQL slowlogs. 13496 13701
  • Add filebeat azure module with activitylogs, auditlogs, signinlogs filesets. 13776
  • Add support to set the document id in the json reader. 5844
  • Add input httpjson. 13545 13546
  • Filebeat Netflow input: Remove beta label. 13858
  • Remove event.timezone from events that don’t need it in some modules that support log formats with and without timezones. 13918
  • Add ExpandEventListFromField config option in the kafka input. 13965
  • Add ELB fileset to AWS module. 14020
  • Add module for MISP (Malware Information Sharing Platform). 13805
  • Add filebeat azure module with activitylogs, auditlogs, signinlogs filesets. 13776 14033 14107
  • Add support for all the ObjectCreated events in S3 input. 14077
  • Add source.bytes and source.packets for uni-directional netflow events. 14111
  • Add Kibana Dashboard for MISP module. 14147
  • Add support for gzipped files in S3 input 13980
  • Add Filebeat Azure Dashboards 14127
  • Add support for thread ID in Filebeat Kafka module. 19463

Heartbeat - Add non-privileged icmp on linux and darwin(mac). 13795 11498 - Allow hosts to be used to configure http monitors 13703

Metricbeat

  • Add refresh list of perf counters at every fetch 13091
  • Add proc/vmstat data to the system/memory metricset on linux 13322
  • Add support for NATS version 2. 13601
  • Add docker.cpu.*.norm.pct metrics for cpu metricset of Docker Metricbeat module. 13695
  • Add instance label by default when using Prometheus collector. 13737
  • Add azure module. 13196 13859 13988
  • Add Apache Tomcat module 13491
  • Add ECS container.id and container.runtime to kubernetes state_container metricset. 13884
  • Add job label by default when using Prometheus collector. 13878
  • Add state_resourcequota metricset for Kubernetes module. 13693
  • Add tags filter in ec2 metricset. 13872 13145
  • Add cloud.account.id and cloud.account.name into events from aws module. 13551 13558
  • Add metrics_path as known hint for autodiscovery 13996
  • Leverage KUBECONFIG when creating k8s client. 13916
  • Add ability to filter by tags for cloudwatch metricset. 13758 13145
  • Release cloudwatch, s3_daily_storage, s3_request, sqs and rds metricset as GA. 14114 14059
  • Add elasticsearch/enrich metricset. 14243 14221
  • Add new dashboards for Azure vms, vm guest metrics, vm scale sets 14000
  • Add vpc metricset for aws module. 16111 14854

Functionbeat

  • Make bulk_max_size configurable in outputs. 13493

Winlogbeat

  • Fill event.provider. 13937
  • Add support for user management events to the Security module. 13530
  • Made the event parser more lenient w.r.t. invalid event log definition version numbers. 15838

Deprecatededit

Metricbeat

  • kubernetes.container.id field for state_container is deprecated in favour of ECS container.id and container.runtime. 13884