Metricset Details
editMetricset Details
editThis topic provides additional details about creating metricsets.
Adding Special Configuration Options
editEach metricset can have its own configuration variables defined. To make use of
these variables, you must extend the New
method. For example, let’s assume that
you want to add a password
config option to the metricset. You would extend
beat.yml
in the following way:
metricbeat.modules: - module: {module} metricsets: ["{metricset}"] password: "test1234"
To read in the new password
config option, you need to modify the New
method. First you define a config
struct that contains the value types to be read. You can set default values, as needed. Then you pass the config to
the UnpackConfig
method for loading the configuration.
Your implementation should look something like this:
type MetricSet struct { mb.BaseMetricSet password string } func New(base mb.BaseMetricSet) (mb.MetricSet, error) { // Unpack additional configuration options. config := struct { Password string `config:"password"` }{ Password: "", } err := base.Module().UnpackConfig(&config) if err != nil { return nil, err } return &MetricSet{ BaseMetricSet: base, password: config.Password, }, nil }
Timeout Connections to Services
editEach time the Fetch
method is called, it makes a request to the service, so it’s
important to handle the connections correctly. We recommended that you set up the
connections in the New
method and persist them in the MetricSet
object. This allows
connections to be reused.
One very important point is that connections must respect the timeout variable:
base.Module().Config().Timeout
. If the timeout elapses before the request completes,
the request must be ended and an error must be returned to make sure the next request
can be started on time. By default the Timeout is set to Period, so one request gets
ended before a new request is made.
If a request must be ended or has an error, make sure that you return a useful error message. This error message is also sent to Elasticsearch, making it possible to not only fetch metrics from the service, but also report potential problems or errors with the metricset.
Data Transformation
editIf the data transformation that has to happen in the Fetch
method is
extensive, we recommend that you create a second file called data.go
in the same package
as the metricset. The data.go
file should contain a function called eventMapping(...)
.
A separate file is not required, but is currently a best practice because it isolates the
functionality of the metricset and Fetch
method from the data mapping.
fields.yml
editThe fields.yml
file is used for different purposes:
- Creates the Elasticsearch template
- Creates the Kibana index pattern configuration
- Creates the Exported Fields documentation for the metricset
To make sure the Elasticsearch template is correct, it’s important to keep this file up-to-date
with all the changes. There is a fields.yml
file under module/{module}/_meta/fields.yml
that contains
the general top level structure for all metricsets. Normally you only need to modify the description in this file.
Here an example for the fields.yml
file from the MySQL module.
- key: mysql title: "MySQL" description: > MySQL server status metrics collected from MySQL. short_config: false release: ga fields: - name: mysql type: group description: > `mysql` contains the metrics that were obtained from MySQL query. fields:
There is another fields.yml
file under module/{module}/{metricset}/_meta/fields.yml
that contains all fields retrieved
by the metricset. As field types, each field must have a core data type
supported by elasticsearch. Here’s a very basic example that shows one group from the MySQL status
metricset:
- name: status type: group description: > `status` contains the metrics that were obtained by the status SQL query. fields: - name: aborted type: group description: > Aborted status fields. fields: - name: clients type: integer description: > The number of connections that were aborted because the client died without closing the connection properly. - name: connects type: integer description: > The number of failed attempts to connect to the MySQL server.
As you can see, if there are nested fields, you must use the type group
.
Testing
editIt’s important to also add tests for your metricset. There are three different types of tests that you need for testing a Beat:
- unit tests
- integration tests
- system tests
We recommend that you use all three when you create a metricset. Unit tests are
written in Go and have no dependencies. Integration tests are also written
in Go but require the service from which the module collects metrics to also be running.
System tests for Metricbeat also require the service to be running in most cases and are
written in Python based on our small Python test framework.
We use virtualenv
to deal with Python dependencies.
You can simply run the command make python-env
and then . build/python-env/bin/activate
.
You should use a combination of the three test types to test your metricsets because
each method has advantages and disadvantages. To get started with your own tests, it’s best
to look at the existing tests. You’ll find the unit and integration tests
in the _test.go
files under existing modules and metricsets. The system
tests are under tests/systems
.
Adding a Test Environment
editIntegration and system tests need an environment that’s running the service. You can create this environment by using Docker and a docker-compose file. If you add a module that requires a service, you must add the service to the virtual environment. To do this, you:
-
Update the
docker-compose.yml
file with your environment -
Update the
docker-entrypoint.sh
script
The docker-compose.yml
file is at the root of Metricbeat. Most services have
existing Docker modules and can be added as simply as Redis:
redis: image: redis:3.2.3
To allow the Beat to access your service, make sure that you define the environment variables in the docker-compose file and add the link to the container:
beat: links: - redis environment: - REDIS_HOST=redis - REDIS_PORT=6379
To make sure the service is running before the tests are started, modify the
docker-entrypoint.sh
script to add a check that verifies your service is
running. For example, the check for Redis looks like this:
waitFor ${REDIS_HOST} ${REDIS_PORT} Redis
The environment expects your service to be available as soon as it receives a response from the given address and port.
Running the Tests
editTo run all the tests, run make testsuite
. To only run unit tests, run
make unit-tests
or for integration tests make integration-tests-environment
. Be aware that
a running Docker environment is needed for integration and system tests.
Sometimes you may want to run a single integration test, for example, to test a
module such as the apache
module. To do this, you can:
-
Start the Docker service by running
docker-compose run -p port:port apache
. You can skip this step if, like thegolang
module, your module doesn’t need a Docker service. -
Run
cd tests/system
to change to the folder that contains the integration tests. -
Run
INTEGRATION_TESTS=true nosetests test_apache.py
, remembering to replacetest_apache.py
with your own test file.
Documentation
editEach module must be documented. The documentation is based on asciidoc and is in
the file module/{module}/_meta/docs.asciidoc
for the module and in module/{module}/{metricset}/_meta/docs.asciidoc
for the metricset. Basic documentation with the config file and an example output is automatically
generated. Use these files to document specific configuration options or usage examples.