Kafka module
editKafka module
editThis is the Kafka module.
The default metricsets are consumergroup
and partition
.
If authorization is configured in the Kafka cluster, the following ACLs are required for the Metricbeat user:
- READ Topic, for the topics to be monitored
- DESCRIBE Group, for the groups to be monitored
For example, if the stats
user is being used for Metricbeat, to monitor all
topics and all consumer groups, ACLS can be granted with the following commands:
kafka-acls --authorizer-properties zookeeper.connect=localhost:2181 --add --allow-principal User:stats --operation Read --topic '*' kafka-acls --authorizer-properties zookeeper.connect=localhost:2181 --add --allow-principal User:stats --operation Describe --group '*'
Compatibility
editThis module is tested with Kafka 0.10.2.1, 1.1.0, 2.1.1, and 2.2.2.
The Broker, Producer, Consumer metricsets require Jolokia to fetch JMX metrics. Refer to the link for Jolokia’s compatibility notes.
Usage
editThe Broker, Producer, Consumer metricsets require Jolokia to fetch JMX metrics. Refer to those Metricsets' documentation about how to use Jolokia.
Dashboard
editThe Kafka module comes with a predefined dashboard. For example:
Example configuration
editThe Kafka module supports the standard configuration options that are described in Modules. Here is an example configuration:
metricbeat.modules: # Kafka metrics collected using the Kafka protocol - module: kafka #metricsets: # - partition # - consumergroup period: 10s hosts: ["localhost:9092"] #client_id: metricbeat #retries: 3 #backoff: 250ms # List of Topics to query metadata for. If empty, all topics will be queried. #topics: [] # Optional SSL. By default is off. # List of root certificates for HTTPS server verifications #ssl.certificate_authorities: ["/etc/pki/root/ca.pem"] # Certificate for SSL client authentication #ssl.certificate: "/etc/pki/client/cert.pem" # Client Certificate Key #ssl.key: "/etc/pki/client/cert.key" # Client Certificate Passphrase (in case your Client Certificate Key is encrypted) #ssl.key_passphrase: "yourKeyPassphrase" # SASL authentication #username: "" #password: "" # Metrics collected from a Kafka broker using Jolokia #- module: kafka # metricsets: # - broker # period: 10s # hosts: ["localhost:8779"] # Metrics collected from a Java Kafka consumer using Jolokia #- module: kafka # metricsets: # - consumer # period: 10s # hosts: ["localhost:8774"] # Metrics collected from a Java Kafka producer using Jolokia #- module: kafka # metricsets: # - producer # period: 10s # hosts: ["localhost:8775"]
Metricsets
editThe following metricsets are available: