Creating a Metricset
editCreating a Metricset
editA metricset is the part of a Metricbeat module that fetches and structures the data from the remote service. Each module can have multiple metricsets. In this guide, you learn how to create your own metricset. If you want to create your own Beat that uses Metricbeat as a library, see Creating a Beat based on Metricbeat.
When creating a metricset for the first time, it generally helps to look at the implementation of existing metricsets for inspiration.
To create a new metricset:
-
Run the following command inside your beat directory:
make create-metricset
You’ll be prompted to enter a module and metricset name. Only use characters
[a-z]
and, if required, underscores (_
). No other characters are allowed.When you run
make create-metricset
, it creates all the basic files for your metricset, along with the required module files if the module does not already exist. See Creating a Metricbeat Module for more details about the module files.We use
{metricset}
,{module}
, and{beat}
in this guide as placeholders. You need to replace these with the actual names of your metricset, module, and beat.The metricset that you created is already a functioning metricset and can be compiled.
-
Compile your new metricset by running the following command:
make collect make
The first command,
make collect
, updates all generated files with the most recent files, data, and meta information from the metricset. The second command,make
, compiles your source code and provides you with a binary called{beat}
in the beat folder. You can run the binary in debug mode with the following command:./{beat} -e -d "*"
After running the make commands, you’ll find the metricset, along with its generated files, under module/{module}/{metricset}
. This directory
contains the following files:
-
\{metricset}.go
-
_meta/docs.asciidoc
-
_meta/data.json
-
_meta/fields.yml
Let’s look at the files in more detail next.
{metricset}.go File
editThe first file is {metricset}.go
. It contains the logic on how to fetch data from the service and convert it for sending to the output.
The generated file looks like this:
https://github.com/elastic/beats/blob/master/metricbeat/scripts/module/metricset/metricset.go.tmpl
package {metricset} import ( "github.com/elastic/beats/libbeat/common" "github.com/elastic/beats/libbeat/common/cfgwarn" "github.com/elastic/beats/metricbeat/mb" ) // init registers the MetricSet with the central registry as soon as the program // starts. The New function will be called later to instantiate an instance of // the MetricSet for each host defined in the module's configuration. After the // MetricSet has been created then Fetch will begin to be called periodically. func init() { mb.Registry.MustAddMetricSet("{module}", "{metricset}", New) } // MetricSet holds any configuration or state information. It must implement // the mb.MetricSet interface. And this is best achieved by embedding // mb.BaseMetricSet because it implements all of the required mb.MetricSet // interface methods except for Fetch. type MetricSet struct { mb.BaseMetricSet counter int } // New creates a new instance of the MetricSet. New is responsible for unpacking // any MetricSet specific configuration options if there are any. func New(base mb.BaseMetricSet) (mb.MetricSet, error) { cfgwarn.Experimental("The {module} {metricset} metricset is experimental.") config := struct{}{} if err := base.Module().UnpackConfig(&config); err != nil { return nil, err } return &MetricSet{ BaseMetricSet: base, counter: 1, }, nil } // Fetch methods implements the data gathering and data conversion to the right // format. It publishes the event which is then forwarded to the output. In case // of an error set the Error field of mb.Event or simply call report.Error(). func (m *MetricSet) Fetch(report mb.ReporterV2) { report.Event(mb.Event{ MetricSetFields: common.MapStr{ "counter": m.counter, }, }) m.counter++ }
The package
clause and import
declaration are part of the base structure of each Go file. You should only
modify this part of the file if your implementation requires more imports.
Initialisation
editThe init method registers the metricset with the central registry. In Go the init()
function is called
before the execution of all other code. This means the module will be automatically registered with the global registry.
The New
method, which is passed to AddMetricSet
, will be called after the setup of the module and before starting to fetch data. You normally don’t need to change this part of the file.
func init() { if err := mb.Registry.AddMetricSet("{module}", "{metricset}", New); err != nil { panic(err) } }
Definition
editThe MetricSet type defines all fields of the metricset. As a minimum it must inherit the mb.BaseMetricSet
fields,
but can be extended with additional entries. These variables can be used to persist data or configuration between
multiple fetch calls.
You can add more fields to the MetricSet type, as you can see in the following example where the username
and password
string fields are added:
type MetricSet struct { mb.BaseMetricSet username string password string }
Creation
editThe New
function creates a new instance of the MetricSet. The setup process
of the MetricSet is also part of New
. This method will be called before Fetch
is called the first time.
The New
function also sets up the configuration by processing additional
configuration entries, if needed.
func New(base mb.BaseMetricSet) (mb.MetricSet, error) { config := struct{}{} if err := base.Module().UnpackConfig(&config); err != nil { return nil, err } return &MetricSet{ BaseMetricSet: base, }, nil }
Fetching
editThe Fetch
method is the central part of the metricset. Fetch
is called every
time new data is retrieved. If more than one host is defined, Fetch
is
called once for each host. The frequency of calling Fetch
is based on the period
defined in the configuration file.
Fetch
must publish the event using the mb.ReporterV2.Event
method. If an error
happens, the error must be published using the mb.ReporterV2.Error
method. This means
that Metricbeat always sends an event, even on failure. You must make sure that the
error message helps to identify the actual error.
The following example shows a metricset Fetch
method with a counter that is
incremented for each Fetch
call:
func (m *MetricSet) Fetch(report mb.ReporterV2) { report.Event(mb.Event{ MetricSetFields: common.MapStr{ "counter": m.counter, } }) m.counter++ }
The JSON output derived from the reported event will be identical to the naming and
structure you use in common.MapStr
. For more details about MapStr
and its functions, see the
MapStr API docs.
Multi Fetching
editMetricbeat has two different Fetch
interfaces. One of the interfaces, which you saw in the
previous example, fetches a single event. In some cases, every fetch returns a list of events.
Instead of using an array inside a JSON document, we recommend that you create a list of events.
For this kind of data, you can use the following Fetch
interface:
(m *MetricSet) Fetch() ([]common.MapStr, error)
The only difference between this and the previous example is that the second example returns []common.MapStr
.
Metricbeat will add the same timestamp to all the events in the list to make it possible to correlate the events.
Parsing and Normalizing Fields
editIn Metricbeat we aim to normalize the metric names from all metricsets to respect a common set of conventions. This makes it easy for users to find and interpret metrics. To simplify parsing, converting, renaming, and restructuring of the object read from the monitored system to the Metricbeat format, we have created the schema package that allows you to declaratively define transformations.
For example, assuming this input object:
input := map[string]interface{}{ "testString": "hello", "testInt": "42", "testBool": "true", "testFloat": "42.1", "testObjString": "hello, object", }
And the requirement to transform it into this one:
common.MapStr{ "test_string": "hello", "test_int": int64(42), "test_bool": true, "test_float": 42.1, "test_obj": common.MapStr{ "test_obj_string": "hello, object", }, }
You can use the following code to make the transformations:
import ( s "github.com/elastic/beats/libbeat/common/schema" c "github.com/elastic/beats/libbeat/common/schema/mapstrstr" ) var ( schema = s.Schema{ "test_string": c.Str("testString"), "test_int": c.Int("testInt"), "test_bool": c.Bool("testBool"), "test_float": c.Float("testFloat"), "test_obj": s.Object{ "test_obj_string": c.Str("testObjString"), }, } ) func eventMapping(input map[string]interface{}) common.MapStr { return schema.Apply(input) }
In the above example, note that it is possible to create the schema object once and apply it to all events.
Configuration File
editThe configuration file for a metricset is handled by the module. If there are multiple metricsets in one module, make sure you add all metricsets to the configuration. For example:
metricbeat: modules: - module: {module-name} metricsets: ["{metricset1}", "{metricset2}"]
Make sure that you run make collect
after updating the config file
so that your changes are also applied to the global configuration file and the docs.
For more details about the Metricbeat configuration file, see the topic about Modules in the Metricbeat documentation.
What to Do Next
editThis topic provides basic steps for creating a metricset. For more details about metricsets and how to extend your metricset further, see Metricset Details.