02 May 2017

Brewing in Beats: Http module in Metricbeat

By Monica Sarbu

Welcome to Brewing in Beats! With this weekly series, we're keeping you up to date with what's new in Beats, including the latest commits and releases.

New community Beat: Rsbeat

Rsbeat is built on top of libbeat infrastructure to collect the Redis slow logs and ship them to Elasticsearch directly or via Logstash. The slow logs are accumulated by Redis in memory, so no files are written with information about the slow command executions. Rsbeat interrogates Redis to retrieve the slow logs.

Metricbeat: new http module

Thanks to Christian Galsterer, the creator of the successful Httpbeat, Metricbeat has a generic module for collecting data periodically from multiple HTTP endpoints, and indexing the result in Elasticsearch. The http module is especially useful when no dedicated Metricbeat module is available, but there is a HTTP based monitoring endpoint.
In case the HTTP endpoint returns the result in a json format, you can configure the json metricset to export all the JSON fields under a configured namespace field as shown in the following example:

{ 
"@timestamp": "2017-05-01T13:00:24.745Z", 
"http": { 
    "http_json_namespace": { 
        "date": "05-01-2017", 
        "milliseconds_since_epoch": 1493643625474.000000,
        "time": "01:00:25 PM" 
    } 
}, 
...
}

Dashboards using the Time Series Visual Builder

The system module of Metricbeat gets two new Kibana dashboards, built using the new Time Series Visual Builder that will be added with the new Kibana 5.4 release. You can watch a demo of the time series visual builder during the Kibana visualizations deep dive.

host details.png

Metricbeat: report the topN processes

This adds the option to only report on the top N processes by CPU and/or memory. It is useful because storing metrics about each and every process from every host can be quite expensive from a storage point of view. Previously it was possible to filter processes by name, which was useful if one knew in advance which are the most interesting processes. This adds a new option which should be quite convenient in practice, because the number of per-process documents gets limited while still allowing to display the top processes.

Add Alibaba Cloud metadata

Thanks to the community contribution, the add_cloud_metadata processor enhances each event with Alibaba Cloud metadata.  This metadata is only available when VPC is selected as the network type of the instance.

Other changes

Repository: elastic/beats

Affecting all Beats

Changes in 5.4:

  • @timestamp doesn't get printed when specified in message codec #3721

Changes in master:

  • Ensure correct permissions for config files when running make update #4137
  • Remove duplicate code in glob_watcher_test.go #4117

Processors

Changes in master:

  • Make Kubernetes indexers/matchers pluggable #4151

Filebeat

Changes in master:

  • Move handling of JSON fields to harvester #4159
  • Extract stdin prospector #4158
  • Cleanup prospector shutdown #4157
  • Disable default prospector and adjust short configs #4105
  • Refactor harvester to send events directly to the spooler #4070
  • Filebeat: expand double wildcards in prospector #3980

Winlogbeat

Changes in 5.x:

  • Deprecate the metrics endpoint in Winlogbeat #4145

Metricbeat

Changes in master:

  • New http module #4156
  • Fixing multiEventFetch error reporting when no events are returned #4153
  • Fixing nil pointer on Prometheus collector when HTTP response is nil #4119

Heartbeat

Changes in master:

  • Move fields to each monitor in Heartbeat #4168
  • Heartbeat event format #4091

Infrastructure

Changes in master:

  • Adding goimports support to make check and fmt #4114

Changes in 5.3:

  • Fix link to the MacOSX SDK tarball #4120

Changes in 5.4:

  • Fix link to the MacOSX SDK tarball #4120

Documentation

Changes in 5.4:

  • Backport: Cherrypick #4148, #4108, #4095 into 5.4 #4161

Changes in 5.3:

  • Backport: #4108 and #4148 into 5.3 #4160
  • Fix typo in port number for metrics.bindaddress #4139

Changes in master:

  • Forward port: Cherrypick #4095 into master #4162
  • Wrong "make collect" statement documented instead of "make update" when documentation should be regenerated #4155
  • Fix config example #4148
  • Remove docker statement #4147
  • Fix Jolokia docs #4140
  • Add note about editor config to CONTRIBUTING.md #4136
  • Add community beats topic to dev guide #4134

Dashboards

Changes in master:

  • Update host details dashboard to use Series Agg #4146