Beats version 7.2.0

View commits

Breaking changes

Affecting all Beats

  • Update to Golang 1.12.4. 11782

Auditbeat

  • Auditd module: Normalized value of event.category field from user-login to authentication. 11432
  • Auditd module: Unset auditd.session and user.audit.id fields are removed from audit events. 11431 11815
  • Socket dataset: Exclude localhost by default 11993

Filebeat

  • Add read_buffer configuration option. 11739

Heartbeat

  • Removed the add_host_metadata and add_cloud_metadata processors from the default config. These don’t fit well with ECS for Heartbeat and were rarely used.

Journalbeat

Metricbeat

  • Add new option OpMultiplyBuckets to scale histogram buckets to avoid decimal points in final events 10994
  • system/raid metricset now uses /sys/block instead of /proc/mdstat for data. 11613

Packetbeat

  • Add support for mongodb opcode 2013 (OP_MSG). 6191 8594
  • NFSv4: Always use opname ILLEGAL when failed to match request to a valid nfs operation. 11503

Winlogbeat

Functionbeat

Bugfixes

Affecting all Beats

  • Ensure all beat commands respect configured settings. 10721
  • Add missing fields and test cases for libbeat add_kubernetes_metadata processor. 11133, 11134
  • decode_json_field: process objects and arrays only 11312
  • decode_json_field: do not process arrays when flag not set. 11318
  • Report faulting file when config reload fails. 11304
  • Fix a typo in libbeat/outputs/transport/client.go by updating c.conn.LocalAddr() to c.conn.RemoteAddr(). 11242
  • Management configuration backup file will now have a timestamps in their name. 11034
  • [CM] Parse enrollment_token response correctly 11648
  • Not hiding error in case of http failure using elastic fetcher 11604
  • Escape BOM on JsonReader before trying to decode line 11661
  • Fix matching of string arrays in contains condition. 11691
  • Replace wmi queries with win32 api calls as they were consuming CPU resources 3249 and 11840
  • Fix queue.spool.write.flush.events config type. 12080
  • Fixed a memory leak when using the add_process_metadata processor under Windows. 12100
  • Fix of docker json parser for missing "log" jsonkey in docker container’s log 11464
  • Fixed Beat ID being reported by GET / API. 12180
  • Add host.os.codename to fields.yml. 12261
  • Fix @timestamp being duplicated in events if @timestamp is set in a processor (or by any code utilizing PutValue() on a beat.Event).
  • Fix leak in script processor when using Javascript functions in a processor chain. 12600

Auditbeat

  • Process dataset: Fixed a memory leak under Windows. 12100
  • Login dataset: Fix re-read of utmp files. 12028
  • Package dataset: Fixed a crash inside librpm after Auditbeat has been running for a while. 12147 12168
  • Fix formatting of config files on macOS and Windows. 12148
  • Fix direction of incoming IPv6 sockets. 12248
  • Package dataset: Auto-detect package directories. 12289
  • System module: Start system module without host ID. 12373

Filebeat

  • Add support for Cisco syslog format used by their switch. 10760
  • Cover empty request data, url and version in Apache2 modulehttps://github.com/elastic/beats/pull/10730[10730]
  • Fix registry entries not being cleaned due to race conditions. 10747
  • Improve detection of file deletion on Windows. 10747
  • Add missing Kubernetes metadata fields to Filebeat CoreDNS module, and fix a documentation error. 11591
  • Reduce memory usage if long lines are truncated to fit max_bytes limit. The line buffer is copied into a smaller buffer now. This allows the runtime to release unused memory earlier. 11524
  • Fix memory leak in Filebeat pipeline acker. 12063
  • Fix goroutine leak caused on initialization failures of log input. 12125
  • Fix goroutine leak on non-explicit finalization of log input. 12164
  • Require client_auth by default when ssl is enabled for tcp input 12333
  • Fix timezone offset parsing in system/syslog. 12529

Heartbeat

  • Fix NPEs / resource leaks when executing config checks. 11165
  • Fix duplicated IPs on mode: all monitors. 12458

Journalbeat

  • Use backoff when no new events are found. 11861

Metricbeat

  • Change diskio metrics retrieval method (only for Windows) from wmi query to DeviceIOControl function using the IOCTL_DISK_PERFORMANCE control code 11635
  • Call GetMetricData api per region instead of per instance. 11820 11882
  • Update documentation with cloudwatch:ListMetrics permission. 11987
  • Check permissions in system socket metricset based on capabilities. 12039
  • Get process information from sockets owned by current user when system socket metricset is run without privileges. 12039
  • Avoid generating hints-based configuration with empty hosts when no exposed port is suitable for the hosts hint. 8264 12086
  • Fixed a socket leak in the postgresql module under Windows when SSL is disabled on the server. 11393
  • Change some field type from scaled_float to long in aws module. 11982
  • Fixed RabbitMQ queue metricset gathering when consumer_utilisation is set empty at the metrics source 12089
  • Fix direction of incoming IPv6 sockets. 12248
  • Ignore prometheus metrics when their values are NaN or Inf. 12084 10849
  • Require client_auth by default when ssl is enabled for module http metricset serverhttps://github.com/elastic/beats/pull/12333[12333]
  • The elasticsearch/index_summary metricset gracefully handles an empty Elasticsearch cluster when xpack.enabled: true is set. 12489 12487

Packetbeat

  • Prevent duplicate packet loss error messages in HTTP events. 10709
  • Fixed a memory leak when using process monitoring under Windows. 12100
  • Improved debug logging efficiency in PGQSL module. 12150

Winlogbeat

Functionbeat

  • Fix function name reference for Kinesis streams in CloudFormation templates 11646

Added

Affecting all Beats

  • Add an option to append to existing logs rather than always rotate on start. 11953
  • Add network condition to processors for matching IP addresses against CIDRs. 10743
  • Add if/then/else support to processors. 10744
  • Add community_id processor for computing network flow hashes. 10745
  • Add output test to kafka output 10834
  • Gracefully shut down on SIGHUP 10704
  • New processor: copy_fields. 11303
  • Add error.message to events when fail_on_error is set in rename and copy_fields processors. 11303
  • New processor: truncate_fields. 11297
  • Allow a beat to ship monitoring data directly to an Elasticsearch monitoring clsuter. 9260
  • Updated go-seccomp-bpf library to v1.1.0 which updates syscall lists for Linux v5.0. NNNN
  • Add add_observer_metadata processor. 11394
  • Add decode_csv_fields processor. 11753
  • Add convert processor for converting data types of fields. 8124 11686
  • New extract_array processor. 11761
  • Add number of goroutines to reported metrics. 12135

Auditbeat

  • Auditd module: Add event.outcome and event.type for ECS. 11432
  • Process: Add file hash of process executable. 11722
  • Socket: Add network.transport and network.community_id. 12231
  • Host: Fill top-level host fields. 12259

Filebeat

  • Add more info to message logged when a duplicated symlink file is found 10845
  • Add option to configure docker input with paths 10687
  • Add Netflow module to enrich flow events with geoip data. 10877
  • Set event.category: network_traffic for Suricata. 10882
  • Allow custom default settings with autodiscover (for example, use of CRI paths for logs). 12193
  • Allow to disable hints based autodiscover default behavior (fetching all logs). 12193
  • Change Suricata module pipeline to handle destination.domain being set if a reverse DNS processor is used. 10510
  • Add the network.community_id flow identifier to field to the IPTables, Suricata, and Zeek modules. 11005
  • New Filebeat coredns module to ingest coredns logs. It supports both native coredns deployment and coredns deployment in kubernetes. 11200
  • New module for Cisco ASA logs. 9200 11171
  • Added support for Cisco ASA fields to the netflow input. 11201
  • Configurable line terminator. 11015
  • Add Filebeat envoyproxy module. 11700
  • Add apache2(httpd) log path (/var/log/httpd) to make apache2 module work out of the box on Redhat-family OSes. 11887 11888
  • Add support to new MongoDB additional diagnostic information 11952
  • New module panw for Palo Alto Networks PAN-OS logs. 11999
  • Add RabbitMQ module. 12032
  • Add new container input. 12162

Heartbeat

  • Enable add_observer_metadata processor in default config. 11394

Journalbeat

Metricbeat

  • Add AWS SQS metricset. 10684 10053
  • Add AWS s3_request metricset. 10949 10055
  • Add s3_daily_storage metricset. 10940 10055
  • Add coredns metricbeat module. 10585
  • Add SSL support for Metricbeat HTTP server. 11482 11457
  • The elasticsearch.index metricset (with xpack.enabled: true) now collects refresh.external_total_time_in_millis fields from Elasticsearch. 11616
  • Allow module configurations to have variants 9118
  • Add timeseries.instance field calculation. 10293
  • Added new disk states and raid level to the system/raid metricset. 11613
  • Added path_name and start_name to service metricset on windows module 8364 11877
  • Add check on object name in the counter path if the instance name is missing 6528 11878
  • Add AWS cloudwatch metricset. 11798 11734
  • Add regions in aws module config to specify target regions for querying cloudwatch metrics. 11932 11956
  • Keep etcd followers members from reporting leader metricset events 12004
  • Add validation for elasticsearch and kibana modules' metricsets when xpack.enabled is set to true. 12386

Packetbeat

Functionbeat

  • New options to configure roles and VPC. 11779

Winlogbeat

  • Add support for reading from .evtx files. 4450

Deprecated

Affecting all Beats

Filebeat

  • docker input is deprecated in favour container. 12162

Heartbeat

Journalbeat

Metricbeat

Packetbeat

Winlogbeat

Functionbeat

Known Issue

Journalbeat