Light modules are a new way to extend Metricbeat functionality without code, by reusing existing modules and configurations.

Metricbeat has modules that support the collection of metrics using specific protocols in a generic way, allowing you to leverage the Elastic Stack for fetching, storing, and analyzing information coming from other existing solutions. These generic modules also provide a way of extending Metricbeat functionality by monitoring services that don’t have their own native modules. Some examples are the modules for<a href="/guide/en/beats/metricbeat/7.3/metricbeat-module-prometheus.html"> Prometheus</a>, <a href="/guide/en/beats/metricbeat/7.3/metricbeat-module-jolokia.html">Jolokia</a>, and <a href="/guide/en/beats/metricbeat/7.3/metricbeat-metricset-aws-cloudwatch.html">AWS cloudwatch</a>.
Internally, we sometimes call these modules “input” 
modules, because of the parallelism we find between them and Filebeat 
inputs. 
	<a href="/guide/en/beats/filebeat/7.3/configuration-filebeat-options.html">Filebeat inputs</a> can be used on their own, but they are also the foundations of <a href="/guide/en/beats/filebeat/7.3/filebeat-modules-overview.html">Filebeat modules</a>. Filebeat modules can be seen as packaged Filebeat configurations that cover specific services or use cases. They work by collecting 
information using inputs, but also include documentation, field mappings, and dashboards, extending filebeat functionality without Go code. We wondered if we could have something similar in Metricbeat, and thus, we created the concept of Metricbeat light modules.
Light modules are a new way of defining Metricbeat 
modules based on existing implementations. They don't contain any Go 
code — all functionality is provided by existing metricsets. They can be
 seen as predefined configurations for generic metricsets. They speed up
 the development of new modules for services whose monitoring is based 
on protocols we already support, and they provide a way to share 
existing Metricbeat configurations and dashboards.
<h2>Implementing Metricbeat light modules</h2>Metricbeat light modules keep the known structure of modules and metricsets, but 
instead of using Go code to define the modules and metricsets, files containing definitions as metadata are used.
Each light module includes a <code>module.yml</code> file with the list of light metricsets, and for each metricset, there is a directory with a <code>manifest.yml</code> with its definition. Light modules included in the <a href="https://www.github.com/elastic/beats">beats</a> repository can also provide documentation files, dashboards, field mappings, or automated tests, just like any other module.
A basic light module with two metricsets would have a structure like this one:
<pre class="prettyprint">├── module.yml
├── _meta
│ ├── config.yml
│ └── docs.asciidoc
├── metricset1
│ ├── manifest.yml
│ └── _meta
│ └── docs.asciidoc
└── metricset2
 ├── manifest.yml
 └── _meta
 └── docs.asciidoc
</pre>A manifest file includes a reference to an existing 
metricset, allowing to override or define new defaults. For example the 
following file would define a module based on the metricset 
	<code>somemetricset</code> of the module <code>somemodule</code>, and would set a couple of default settings:
<pre class="prettyprint">input:
 module: somemodule
 metricset: somemetricset
 defaults:
 some_option: foo
 other_option: bar
 default: true
</pre>A Metricbeat user could then use it with a configuration similar to the one of any other module:
<pre class="prettyprint">- module: module
 metricsets: ['metricset1', 'metricset2']
 ...
</pre>Light modules are designed as a fallback mechanism to current modules — if a module is included in Metricbeat configuration and is not found along the registered ones, then Metricbeat looks for an existing light module and metricset with the given name in its home path. If it is found, it is loaded and instantiated.
As we have seen, a light module manifest references a metricset as its input implementation. The reference metricset must be an existing native metricset — it cannot reference another light module. The final module would be instantiated and customized with the default settings defined in the manifest.
There can be modules that mix native metricsets with light metricsets, for example we plan to implement some AWS metricsets, like <code>elb</code>, as light modules based on the AWS <code>cloudwatch</code> metricset.
The first module implemented as a light module is for CockroachDB; its beta version is included in Metricbeat 7.3.
<h2>Example: CockroachDB light module</h2><a href="https://github.com/cockroachdb/cockroach">CockroachDB</a> is a good service to use as an initial example because it can be monitored using an existing metricset — the <a href="/guide/en/beats/metricbeat/7.3/metricbeat-metricset-prometheus-collector.html">Prometheus collector</a> — but it requires a small customization to set the proper metrics endpoint (it uses <code>/_status/vars</code> as the metrics path instead of the default <code>/metrics</code>). If we wanted to monitor CockroachDB with the Prometheus collector metricset, we could do it using a configuration like the following one:
<pre class="prettyprint">- module: prometheus
 metricsets: ['collector']
 hosts: ['localhost:8080']
 metrics_path: '/_status/vars'
</pre>If we want to migrate this configuration to a light module, we can do it with the following files:
<code>x-pack/metricbeat/module/cockroachdb/module.yml</code>
<pre class="prettyprint">name: cockroachdb
metricsets:
- status
</pre><code>x-pack/metricbeat/module/cockroachdb/status/manifest.yml</code>
<pre class="prettyprint">input:
 module: prometheus
 metricset: collector
 defaults:
 metrics_path: /_status/vars
default: true
</pre>With this module definition we are telling Metricbeat to use the <code>collector</code> metricset of the <code>prometheus</code> module, but replacing the default <code>metrics_path</code> with <code>/_status/vars</code>. We are also indicating that this is a default metricset of this module, so it is instantiated if no other metricset is specified in the configuration file.
Despite not having code, testing of this module is done in a similar way to other modules. But because its definition won't be included in the test binary files, we have to take a few things into account.
First, base module and metricset may not be registered in the modules registry of the testing binary, so we have to explicitly import it so it gets registered:
<pre class="prettyprint">import (
 _ "github.com/elastic/beats/metricbeat/module/prometheus"
 _ "github.com/elastic/beats/metricbeat/module/prometheus/collector"
)
</pre>And the modules registry doesn’t know anything about light modules in 
test binaries, so we have to explicitly indicate where they are located. 
They will be loaded from this location as configuration files, so we 
also remove the permission checks that are applied to these files. Both 
things can be done in an 
	<code>init()</code> function in the test files of the module:
<pre class="prettyprint">func init() {
 os.Setenv("BEAT_STRICT_PERMS", "false")
 mb.Registry.SetSecondarySource(
 xpackmb.NewLightModulesSource("../../../module"))
}
</pre>Once the module is ready, it can be used in a way that feels familiar to Metricbeat users:
<pre class="prettyprint">- module: cockroachdb
 metricsets: ['status']
 hosts: ['localhost:8080']
</pre>Events generated look like these:
<pre class="prettyprint">{
 "@timestamp": "2019-05-22T10:00:30.818Z",
 "event": {
 "dataset": "cockroachdb.status",
 "duration": 29065910,
 "module": "cockroachdb"
 },
 "metricset": {
 "name": "status"
 },
 "prometheus": {
 "labels": {
 "le": "1638399",
 "store": "1"
 },
 "metrics": {
 "raft_process_applycommitted_latency_bucket": 106946,
 "raft_process_commandcommit_latency_bucket": 111112,
 "raft_process_handleready_latency_bucket": 21243,
 "raft_process_logcommit_latency_bucket": 53080
 }
 },
 "service": {
 "address": "172.17.0.2:8080",
 "type": "cockroachdb"
 }
}
</pre>They are events like the ones that would be generated by the Prometheus 
collector metricset if used on its own, but with the event and service 
information replaced to indicate the module and dataset used.
Prometheus namespacing is intentionally kept so it reuses the fields mappings of the original metricset.
Finally, this module also includes a dashboard that makes use of some of the available metrics, like:
<ul>
	<li><code>prometheus.metrics.sql_conns</code> for the number of active SQL connections</li>
	<li><code>prometheus.metrics.replicas</code> for the number of replicas</li>
	<li><code>prometheus.metrics.ranges_unavailable</code> for the number of unavailable ranges</li>
</ul><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt65c9430e7da6e918/5d487981c1c3087f2e8337d4/light-modules-dashboard-blog.jpg" data-sys-asset-uid="blt65c9430e7da6e918" alt="Dashboard showing active SQL connections, replicas, and unavailable ranges " style="display: block; margin: auto;">
<h2>What's next?</h2>There are many use cases where this new framework for developing Metricbeat 
modules can be useful. Besides the example based on Prometheus seen in this article, we are already working on the 
	<a href="https://github.com/elastic/beats/pull/12952">AWS ELB metricset</a> based on the <a href="/guide/en/beats/metricbeat/7.3/metricbeat-metricset-aws-cloudwatch.html">cloudwatch metricset</a>, and we also plan to start working on supporting more Java services using the <a href="/guide/en/beats/metricbeat/7.3/metricbeat-module-jolokia.html">Jolokia module</a>.
On the tooling side, we think that <code>metricbeat</code> subcommands give users a complete toolset for contributing new modules without more requirements than the Elastic Stack itself. In this sense we are thinking about ways to extend 
	<code>metricbeat test</code>, or introduce new subcommands that can be used for light modules validation.
Finally, we invite you to create your own light modules. If you're already using one of the mentioned generic modules to monitor some services, consider migrating this configuration to a module others can use.
If you want to learn more, take a look at the <a href="https://github.com/elastic/beats/issues/12270">design issue</a>, the <a href="https://github.com/elastic/beats/pull/12465">initial implementation</a>, and the <a href="https://github.com/elastic/beats/pull/12467">pull request for the CockroachDB module</a>. And if you have any questions, remember that we are always happy to help on the <a href="https://discuss.elastic.co/c/beats">Discuss forums</a>.

lightbeat-modules-blog-banner.jpg

light-modules-blog-thumb.jpg

Writing Metricbeat modules based on other modules: Introducing light modules

Learn to use Filebeat and Metricbeat - and their preconfigured Kibana dashboards -  to monitor your MySQL, Percona, and MariaDB databases.

Databases are a fundamental keystone of most applications, and there are many options for many different use cases. Even a single database can be used in a variety of ways, but regardless of the database type, implementation, or use case, the need for a good solution for monitoring them remains a common denominator. Databases keep the most precious information of any company, and any hiccup can be a reason for alarm.
The majority of <a href="https://db-engines.com/en/ranking">the most popular databases</a> have a place in our solutions for <a href="https://www.elastic.co/solutions/logging">logs</a> and <a href="https://www.elastic.co/solutions/metrics">metrics</a>, and we are continuously increasing our coverage by supporting more services and more versions. The <a href="https://github.com/elastic/beats/commit/168e63f374c8a8d1cfe64b059778247d2bbe34ed">first implementation of Metricbeat</a> already included support for MySQL and Redis, and the recent 7.0 release came with a new <a href="https://www.elastic.co/guide/en/beats/metricbeat/7.0/metricbeat-module-mssql.html">module for Microsoft SQL Server</a>.
MySQL has a long history of success. Being included in the popular LAMP stack made it incredibly popular, first between enthusiasts of new technologies wanting to use it for their websites and blogs, and then in the biggest companies that saw in it as a free solution that offered a new opportunity to scale with any kind of hardware.
<h2>MySQL modules for Metricbeat and Filebeat</h2>Metricbeat and Filebeat have MySQL well covered from the metrics and logs monitoring perspective. Let’s look at Filebeat first.
<h3>Filebeat for MySQL</h3>The <a href="https://www.elastic.co/guide/en/beats/filebeat/7.0/filebeat-module-mysql.html">Filebeat module for MySQL</a> includes two filesets, one for the database error log and another one for the slow query log.
The MySQL error log contains a record of messages about the server startup and stop, and about some events that may happen while the service is running. This log is useful to diagnostic problems, it describes for example the reasons why some options cannot be applied, the causes of a replication failure, and it also informs when a table needs repair and if it could be automatically repaired.
The <code>error</code> fileset parses messages from the MySQL error log to extract the severity of the message and the thread that caused the problem. This information is helpful when monitoring several MySQL nodes, so you can focus on the more problematic messages or you can check how often certain problems occur. 
<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt5cfca8a19af31f60/5cbdbe00a04c2c9319d4f541/monitoring-mysql-1.png" data-sys-asset-uid="blt5cfca8a19af31f60" alt="MySQL error logs ingested by Filebeat" "="">The <code>slowlog</code> fileset is very useful for identifying activities that are consuming a high amount of database resources. The slow query log is a very well known feature for MySQL DBAs, and it includes a lot of information about the most expensive queries, like the query itself, its duration, some basic metrics about rows affected and data transferred, and also detailed information about the optimizations that were applied. This feature can be configured to be more or less detailed and to include different sets of fields. <a href="https://www.elastic.co/guide/en/beats/filebeat/7.0/exported-fields-mysql.html">Most of these fields</a> are parsed by the <code>slowlog</code> fileset, giving the possibility of performing advanced analytics on this information, allowing you to detect and take early decisions on specific queries that can be problematic for your applications and your databases.
<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltc45a6bf0a20da5cf/5cbdbe2bd6780d74198d62d6/monitoring-mysql-2.jpg" data-sys-asset-uid="bltc45a6bf0a20da5cf" alt="MySQL slowlogs ingested by Filebeat" "="">In the preconfigured dashboard, both filesets are combined to give an overview of the messages being reported by all the MySQL instances:
<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt8a49c5930f07f66c/5cbdbe65d6780d74198d62dc/monitoring-mysql-3.png" data-sys-asset-uid="blt8a49c5930f07f66c" alt="Preconfigured dashboard for MySQL Filebeat modules" "=""><h3>Metricbeat for MySQL</h3>The <a href="https://www.elastic.co/guide/en/beats/metricbeat/7.0/metricbeat-module-mysql.html">Metricbeat module for MySQL</a> collects information from the <code>SHOW GLOBAL STATUS;</code> query. This query is the most important provider of information about the status of a server, it includes many metrics about connections, threads, network, replication binlog, types of queries executed, etc. The most important of these metrics are collected by Metricbeat, so they can be used to monitor the status of your servers or take capacity decisions.
It also includes a dashboard that offers a view on time of the most important fields:
<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt806b7df670b87597/5cbdbe8bfcad0dd81e21b2ce/monitoring-mysql-4.png" data-sys-asset-uid="blt806b7df670b87597" alt="Preconfigured dashboard for MySQL Metricbeat modules" "=""><h2>MySQL friends: Percona and MariaDB</h2>As mentioned before, MySQL has a story of success. From the original open source project, a whole community and ecosystem of several companies has grown up.
<a href="https://www.percona.com/">Percona</a> is one of the best known companies in the MySQL ecosystem. It is the developer of many solutions based on MySQL, including their own distribution of MySQL, curated for performance and security. Percona Server for MySQL includes backports of new features and bug fixes, as well as specific features to improve instrumentation and many other things. It is intended to be a drop-in replacement of MySQL server.
MySQL AB, the company behind MySQL was acquired in 2008 by Sun Microsystems, which was later acquired by Oracle Corporation. These movements created some concerns in the community, prompting the creation of <a href="https://mariadb.com/">MariaDB</a> as a fork of the original project. MariaDB is intended to remain free and open source under the GPL license, and is currently maintained by the MariaDB Foundation. Even if it is still in part compatible with MySQL server, it has completely different implementations of some features and it has its own release cycles. At the moment it is the default MySQL distribution in some Linux distributions and is offered by many cloud providers.
In Metricbeat and Filebeat 6.7, we started to run our tests also on Percona and MariaDB, which allowed us to identify new sources of information. Since then we have increased the number of fields collected by the slowlog fileset, allowing even more insightful analysis of the slow query log.
Metricbeat and Filebeat 7.0 are compatible with MySQL, Percona Server, and MariaDB.
And last but not least, Metricbeat also has experimental support for the collection of <a href="http://galeracluster.com/products/">Galera</a>-specific metrics. Galera is a solution for multi-master clustering of MySQL servers. Our current implementation, the <code>galera_status</code> metricset, is heavily based on the <code>status</code> metricset but is not very well tested at the moment. Contributions and feedback are very welcome.
<h2>MySQL 8 and beyond</h2>Before 6.7 we were only testing the MySQL modules with MySQL 5.7 and with some log cases of MySQL 5.5, since then we have increased the coverage to support also MySQL 8.0, and the same versions of Percona Server. We now include tests on MariaDB 10.2 and 10.3.
Recent versions of MySQL and Percona include stronger authentication methods, so we have updated our MySQL clients to support them.
In the future we have plans to continue updating our modules to follow the latest developments of all these products.
If you want to get more visibility into your MySQL servers with the Elastic Stack also take a look at:
<ul>
	<li><a href="https://www.elastic.co/es/solutions/apm">Elastic APM</a>&nbsp; - Allows you to instrument your applications to know, for example, how much time they spend querying&nbsp;your database servers</li>
	<li><a href="https://www.elastic.co/es/products/beats/packetbeat">Packetbeat</a>&nbsp;- Ingests&nbsp;<a href="https://www.elastic.co/guide/en/beats/packetbeat/7.0/exported-fields-mysql.html">MySQL information</a> from network connections</li>
</ul>Getting started is easy, simply download an install <a href="https://www.elastic.co/downloads/beats/metricbeat">Metricbeat</a> and <a href="https://www.elastic.co/downloads/beats/filebeat">Filebeat</a>, enable the appropriate <a href="https://www.elastic.co/guide/en/beats/metricbeat/7.0/metricbeat-module-mysql.html">module</a>, and start sending your data to your <a href="https://www.elastic.co/downloads?elektra=products&storm=main">local cluster</a>, or to your <a href="https://www.elastic.co/cloud/elasticsearch-service/signup">14-day free trial of Elasticsearch Service</a> on <a href="https://www.elastic.co/cloud/">Elastic Cloud</a>. If you have any questions or ideas about how we can improve, reach out on the <a href="https://discuss.elastic.co/c/beats">Discuss</a> forums. Enjoy!

monitoring-mysql-banner.jpg

monitoring-mysql-thumb.jpg

Monitoring MySQL, Percona Server, and MariaDB with the Elastic Stack

Overview of current features and future plans for Java application monitoring with Metricbeat and Jolokia.

AUTHOR

Articles by Jaime Soriano

Writing Metricbeat modules based on other modules: Introducing light modules

Monitoring MySQL, Percona Server, and MariaDB with the Elastic Stack

Monitoring Java Applications with Metricbeat and Jolokia

Sign up for Elastic Cloud free trial

Products & Solutions

Company

We're hiring

Resources

Solutions

Elastic (ELK) Stack

New

Featured topics

News

AUTHOR

Articles by Jaime Soriano

Writing Metricbeat modules based on other modules: Introducing light modules

Monitoring MySQL, Percona Server, and MariaDB with the Elastic Stack

Monitoring Java Applications with Metricbeat and Jolokia

Sign up for Elastic Cloud free trial

Company

We're hiring

Resources