Brewing in Beats: Vsphere module in Metricbeat

Von
Monica Sarbu

Welcome to Brewing in Beats! With this weekly series, we're keeping you up to date with what's new in Beats, including the latest commits and releases.

Vsphere module in Metricbeat

Thanks to Amanda, Metricbeat has a new vsphere module for monitoring VMware vSphere. It uses the library Govmomi to collect metrics from any Vmware SDK URL (ESXi/VCenter). This library is built for and tested against ESXi and vCenter 5.5, 6.0 and 6.5.

Make kubernetes indexers pluggable

The libbeat processor that adds metadata by calling the Kubernetes APIs needs a way to extract the container ID from the events. The way to do that depends on the Beat. For Filebeat, for example, it’s extracted from the file path. For Metricbeat, it’s obtained based on the IP and port of the endpoint. This PR makes this step pluggable, so each Beat can provide it’s own custom implementation. This improvement is based on a community contribution from the same Vijay Samuel.

Move template loading configuration under setup

Each Beat loads the Elasticsearch mapping template at startup. The template is autogenerated from the fields.yml, that defines the mapping of each exported field. The path to the fields.yml, the name of the template and other options are defined in the Elasticsearch output configuration under outputs.elasticsearch.template.*. This pull request moves the template loading logic from the Elasticsearch output into a separate package that is configured under the setup.template.*

 
 # A template is used to set the mapping in Elasticsearch
 # By default template loading is enabled and the template is loaded.
 # These settings can be adjusted to load your own template or overwrite existing ones.
 
 # Set to false to disable template loading.
 setup.template.enabled: true
 
 # Template name. By default the template name is filebeat.
 # The version of the beat will always be appended to the given name
 # so the final name is filebeat-%{[beat.version]}.
 setup.template.name: "filebeat"
 
 # Path to fields.yml file to generate the template
 setup.template.fields: "${path.config}/fields.yml"
 
 # Overwrite existing template
 setup.template.overwrite: false

Kubernetes dashboard

With this change, the Kubelet module is renamed to kubernetes. A new sample dashboard is created and added to the list of sample Metricbeat dashboards.

kubernetes-dashboard.png

Other changes

Repository: elastic/beats

Affecting all Beats

Changes in 5.x:

  • elasticsearch: set _type=doc #3757
Metricbeat

Changes in master:

  • Update docker dependency and modify docker client #4245
  • Implement diskio metricset on macOS #4237
  • Fix non-deterministic vsphere datastore tests #4220
  • Follow up addressing comments in #4127 #4173
  • Http module improvements #4170
  • Abstracting pod interface in kubernetes plugin to enable easier vendoring #4152
  • Kubelet dashboards #4138
  • Add Vsphere module with metrics. WIP #4028
  • Perfmon metricset output format option #3976
Heartbeat

Changes in master:

  • Move fields to each monitor in heartbeat #4168
Filebeat

Changes in master:

  • Fix grok pattern in filebeat module system/auth without hostname #4224
  • Fix problem on shutdown with offset + 1 event #4215
  • Fix 2.x build by excluding filebeat module tests #4207
  • Assert on errors in the filebeat modules #4186
  • Add type to state info #4185
  • Remove duplicate error key in fields.yml #4184
  • Fix mysql slowlog parser #4183
  • Add additional mysql slow log to system tests #4181
  • Unify event, data and state handling in filebeat #4171
  • Cleanup prospector shutdown #4157
  • Fix issue that new prospector was not reloaded on conflict #4128
Infrastructure

Changes in master:

  • Fix freebsd tests by setting hostname #4222
  • Make fix-permission less verbose in CI #4208
  • Make nosetests more verbose #4172
Documentation

Changes in master:

  • Add info about official Beats docker images #4179
  • Forward port: Cherrypick #4095 into master #4162
  • Move content about processors to filtering section in the doc #4149
  • Remove docker statement #4147
  • Fix Jolokia docs #4140
  • Add note about editorconfig to CONTRIBUTING.md #4136
  • Clarify doc for Windows commands to indicate that Powershell is required #4135

Changes in 5.3:

  • Fix 5.3 check as docs were not up-to-date #4201
  • Update how-filebeat-works.asciidoc #4199 #4196
  • Update overview.asciidoc #4195
  • Replace wrong command (chown) with chmod #4194
  • Backport: #4147 to 5.3 #4177
  • Backport: #4108 and #4148 into 5.3 #4160
  • Fix typo in port number for metrics.bindaddress #4139

Changes in 5.4:

  • Change release state to released #4198
  • Backport: Cherrypick #4148, #4108, #4095 into 5.4 #4161
Dashboards

Changes in master:

  • Workaround the dashboards' reliance on types #4218
  • Re-export Packetbeat dashboards with a newer version of Kibana #4188
  • Fix Total Transferred in Network Metric visualization #4175
  • Update host details dashboard to use Series Agg #4146
Packaging

Changes in master:

  • Compile libpcap from source while packaging #4217