ZooKeeper Metrics

Collect metrics from ZooKeeper service with Elastic Agent.

What is an Elastic integration?
Version
1.13.0 (View all)
Compatible Kibana version(s)
8.13.0 or higher
Supported Serverless project types

What's this?
Security
Observability
Subscription level
What's this?
Basic
Level of support
What's this?
Elastic

This integration periodically fetches metrics from the ZooKeeper service.

Compatibility

The ZooKeeper integration is tested with ZooKeeper 3.4.8 and is expected to work with all version >= 3.4.0. Versions prior to 3.4 do not support the mntr command.

Metrics

connection

The connection dataset fetches the data returned by the cons admin keyword.

An example event for connection looks as following:

{
    "@timestamp": "2020-07-06T16:12:07.612Z",
    "agent": {
        "ephemeral_id": "4d221f8f-7147-4855-8ea3-b4d2a5b80ae0",
        "id": "2ff8a09c-c7f0-42f2-9fe1-65f7fd460651",
        "name": "zookeeper-01",
        "type": "metricbeat",
        "version": "8.0.0"
    },
    "client": {
        "ip": "172.28.0.1",
        "port": 44338
    },
    "ecs": {
        "version": "8.11.0"
    },
    "event": {
        "dataset": "zookeeper.connection",
        "duration": 3093417,
        "module": "zookeeper"
    },
    "host": {
        "name": "zookeeper-01"
    },
    "metricset": {
        "name": "connection",
        "period": 10000
    },
    "service": {
        "address": "localhost:2181",
        "type": "zookeeper"
    },
    "zookeeper": {
        "connection": {
            "interest_ops": 0,
            "queued": 0,
            "received": 1,
            "sent": 0
        }
    }
}

ECS Field Reference

Please refer to the following document for detailed information on ECS fields.

Exported fields

FieldDescriptionTypeMetric Type
@timestamp
Event timestamp.
date
agent.id
keyword
client.ip
IP address of the client (IPv4 or IPv6).
ip
client.port
Port of the client.
long
cloud.account.id
The cloud account or organization id used to identify different entities in a multi-tenant environment. Examples: AWS account id, Google Cloud ORG Id, or other unique identifier.
keyword
cloud.availability_zone
Availability zone in which this host is running.
keyword
cloud.image.id
Image ID for the cloud instance.
keyword
cloud.instance.id
Instance ID of the host machine.
keyword
cloud.provider
Name of the cloud provider. Example values are aws, azure, gcp, or digitalocean.
keyword
cloud.region
Region in which this host is running.
keyword
container.id
Unique container id.
keyword
data_stream.dataset
Data stream dataset.
constant_keyword
data_stream.namespace
Data stream namespace.
constant_keyword
data_stream.type
Data stream type.
constant_keyword
event.dataset
Event dataset
constant_keyword
event.module
Event module
constant_keyword
host.containerized
If the host is a container.
boolean
host.name
Name of the host. It can contain what hostname returns on Unix systems, the fully qualified domain name, or a name specified by the user. The sender decides which value to use.
keyword
host.os.build
OS build information.
keyword
host.os.codename
OS codename, if any.
keyword
service.address
Address where data about this service was collected from. This should be a URI, network address (ipv4:port or [ipv6]:port) or a resource path (sockets).
keyword
zookeeper.connection.interest_ops
Interest ops
long
zookeeper.connection.queued
Queued connections
long
gauge
zookeeper.connection.received
Received connections
long
counter
zookeeper.connection.sent
Connections sent
long
counter

mntr

The mntr Metricset fetches the data returned by the mntr admin keyword.

An example event for mntr looks as following:

{
    "@timestamp": "2020-07-06T16:12:08.494Z",
    "agent": {
        "ephemeral_id": "4d221f8f-7147-4855-8ea3-b4d2a5b80ae0",
        "id": "2ff8a09c-c7f0-42f2-9fe1-65f7fd460651",
        "name": "zookeeper-01",
        "type": "metricbeat",
        "version": "8.0.0"
    },
    "ecs": {
        "version": "8.11.0"
    },
    "event": {
        "dataset": "zookeeper.mntr",
        "duration": 15795652,
        "module": "zookeeper"
    },
    "host": {
        "name": "zookeeper-01"
    },
    "metricset": {
        "name": "mntr",
        "period": 10000
    },
    "service": {
        "address": "localhost:2181",
        "type": "zookeeper",
        "version": "3.5.5-390fe37ea45dee01bf87dc1c042b5e3dcce88653, built on 05/03/2019 12:07 GMT"
    },
    "zookeeper": {
        "mntr": {
            "approximate_data_size": 44,
            "ephemerals_count": 0,
            "latency": {
                "avg": 0,
                "max": 0,
                "min": 0
            },
            "max_file_descriptor_count": 1048576,
            "num_alive_connections": 1,
            "open_file_descriptor_count": 49,
            "outstanding_requests": 0,
            "packets": {
                "received": 152,
                "sent": 151
            },
            "server_state": "standalone",
            "watch_count": 0,
            "znode_count": 5
        }
    }
}

ECS Field Reference

Please refer to the following document for detailed information on ECS fields.

Exported fields

FieldDescriptionTypeMetric Type
@timestamp
Event timestamp.
date
agent.id
keyword
cloud.account.id
The cloud account or organization id used to identify different entities in a multi-tenant environment. Examples: AWS account id, Google Cloud ORG Id, or other unique identifier.
keyword
cloud.availability_zone
Availability zone in which this host is running.
keyword
cloud.image.id
Image ID for the cloud instance.
keyword
cloud.instance.id
Instance ID of the host machine.
keyword
cloud.provider
Name of the cloud provider. Example values are aws, azure, gcp, or digitalocean.
keyword
cloud.region
Region in which this host is running.
keyword
container.id
Unique container id.
keyword
data_stream.dataset
Data stream dataset.
constant_keyword
data_stream.namespace
Data stream namespace.
constant_keyword
data_stream.type
Data stream type.
constant_keyword
event.dataset
Event dataset
constant_keyword
event.module
Event module
constant_keyword
host.containerized
If the host is a container.
boolean
host.name
Name of the host. It can contain what hostname returns on Unix systems, the fully qualified domain name, or a name specified by the user. The sender decides which value to use.
keyword
host.os.build
OS build information.
keyword
host.os.codename
OS codename, if any.
keyword
service.address
Address where data about this service was collected from. This should be a URI, network address (ipv4:port or [ipv6]:port) or a resource path (sockets).
keyword
zookeeper.mntr.approximate_data_size
Approximate size of ZooKeeper data.
long
gauge
zookeeper.mntr.ephemerals_count
Number of ephemeral znodes.
long
gauge
zookeeper.mntr.followers
Number of followers seen by the current host.
long
gauge
zookeeper.mntr.hostname
ZooKeeper hostname.
keyword
zookeeper.mntr.latency.avg
Average latency between ensemble hosts in milliseconds.
long
gauge
zookeeper.mntr.latency.max
Maximum latency in milliseconds.
long
gauge
zookeeper.mntr.latency.min
Minimum latency in milliseconds.
long
gauge
zookeeper.mntr.max_file_descriptor_count
Maximum number of file descriptors allowed for the ZooKeeper process.
long
gauge
zookeeper.mntr.num_alive_connections
Number of connections to ZooKeeper that are currently alive.
long
gauge
zookeeper.mntr.open_file_descriptor_count
Number of file descriptors open by the ZooKeeper process.
long
gauge
zookeeper.mntr.outstanding_requests
Number of outstanding requests that need to be processed by the cluster.
long
gauge
zookeeper.mntr.packets.received
Number of ZooKeeper network packets received.
long
gauge
zookeeper.mntr.packets.sent
Number of ZooKeeper network packets sent.
long
gauge
zookeeper.mntr.pending_syncs
Number of pending syncs to carry out to ZooKeeper ensemble followers.
long
gauge
zookeeper.mntr.server_state
Role in the ZooKeeper ensemble.
keyword
zookeeper.mntr.synced_followers
Number of synced followers reported when a node server_state is leader.
long
gauge
zookeeper.mntr.watch_count
Number of watches currently set on the local ZooKeeper process.
long
gauge
zookeeper.mntr.znode_count
Number of znodes reported by the local ZooKeeper process.
long
gauge

server

The server Metricset fetches the data returned by the srvr admin keyword.

An example event for server looks as following:

{
    "@timestamp": "2020-07-06T16:12:12.409Z",
    "agent": {
        "ephemeral_id": "4d221f8f-7147-4855-8ea3-b4d2a5b80ae0",
        "id": "2ff8a09c-c7f0-42f2-9fe1-65f7fd460651",
        "name": "zookeeper-01",
        "type": "metricbeat",
        "version": "8.0.0"
    },
    "ecs": {
        "version": "8.11.0"
    },
    "event": {
        "dataset": "zookeeper.server",
        "duration": 3001938,
        "module": "zookeeper"
    },
    "host": {
        "name": "zookeeper-01"
    },
    "metricset": {
        "name": "server",
        "period": 10000
    },
    "service": {
        "address": "localhost:2181",
        "type": "zookeeper",
        "version": "3.5.5-390fe37ea45dee01bf87dc1c042b5e3dcce88653"
    },
    "zookeeper": {
        "server": {
            "connections": 1,
            "count": 0,
            "epoch": 0,
            "latency": {
                "avg": 0,
                "max": 0,
                "min": 0
            },
            "mode": "standalone",
            "node_count": 5,
            "outstanding": 0,
            "received": 156,
            "sent": 155,
            "version_date": "2019-05-03T12:07:00Z",
            "zxid": "0x0"
        }
    }
}

ECS Field Reference

Please refer to the following document for detailed information on ECS fields.

Exported fields

FieldDescriptionTypeMetric Type
@timestamp
Event timestamp.
date
agent.id
keyword
cloud.account.id
The cloud account or organization id used to identify different entities in a multi-tenant environment. Examples: AWS account id, Google Cloud ORG Id, or other unique identifier.
keyword
cloud.availability_zone
Availability zone in which this host is running.
keyword
cloud.image.id
Image ID for the cloud instance.
keyword
cloud.instance.id
Instance ID of the host machine.
keyword
cloud.provider
Name of the cloud provider. Example values are aws, azure, gcp, or digitalocean.
keyword
cloud.region
Region in which this host is running.
keyword
container.id
Unique container id.
keyword
data_stream.dataset
Data stream dataset.
constant_keyword
data_stream.namespace
Data stream namespace.
constant_keyword
data_stream.type
Data stream type.
constant_keyword
event.dataset
Event dataset
constant_keyword
event.module
Event module
constant_keyword
host.containerized
If the host is a container.
boolean
host.name
Name of the host. It can contain what hostname returns on Unix systems, the fully qualified domain name, or a name specified by the user. The sender decides which value to use.
keyword
host.os.build
OS build information.
keyword
host.os.codename
OS codename, if any.
keyword
service.address
Address where data about this service was collected from. This should be a URI, network address (ipv4:port or [ipv6]:port) or a resource path (sockets).
keyword
zookeeper.server.connections
Number of clients currently connected to the server
long
gauge
zookeeper.server.count
Total transactions of the leader in epoch
long
counter
zookeeper.server.epoch
Epoch value of the Zookeeper transaction ID. An epoch signifies the period in which a server is a leader
long
zookeeper.server.latency.avg
Average amount of time taken for the server to respond to a client request
long
gauge
zookeeper.server.latency.max
Maximum amount of time taken for the server to respond to a client request
long
gauge
zookeeper.server.latency.min
Minimum amount of time taken for the server to respond to a client request
long
gauge
zookeeper.server.mode
Mode of the server. In an ensemble, this may either be leader or follower. Otherwise, it is standalone
keyword
zookeeper.server.node_count
Total number of nodes
long
gauge
zookeeper.server.outstanding
Number of requests queued at the server. This exceeds zero when the server receives more requests than it is able to process
long
gauge
zookeeper.server.received
Number of requests received by the server
long
counter
zookeeper.server.sent
Number of requests sent by the server
long
counter
zookeeper.server.version_date
Date of the Zookeeper release currently in use
date
zookeeper.server.zxid
Unique value of the Zookeeper transaction ID. The zxid consists of an epoch and a counter. It is established by the leader and is used to determine the temporal ordering of changes
keyword

Changelog

VersionDetailsKibana version(s)

1.13.0

Enhancement View pull request
Add processor support for connection, mntr and server data streams.

8.13.0 or higher

1.12.0

Enhancement View pull request
ECS version updated to 8.11.0. Update the kibana constraint to ^8.13.0. Modified the field definitions to remove ECS fields made redundant by the ecs@mappings component template.

8.13.0 or higher

1.11.0

Enhancement View pull request
Add global filter on data_stream.dataset to improve performance.

8.8.0 or higher

1.10.0

Enhancement View pull request
Update the package format_version to 3.0.0.

8.8.0 or higher

1.9.0

Enhancement View pull request
Enable time series data streams for the metrics datasets. This improves storage usage and query performance. For more details, see https://www.elastic.co/guide/en/elasticsearch/reference/current/tsds.html

8.8.0 or higher

1.8.5

Enhancement View pull request
Added dimension mapping for server datastream.

8.2.0 or higher

1.8.4

Enhancement View pull request
Added dimension mapping for mntr datastream.

8.2.0 or higher

1.8.3

Enhancement View pull request
Added dimension mapping for connection datastream.

8.2.0 or higher

1.8.2

Enhancement View pull request
Added metric_type mapping for server datastream.

8.2.0 or higher

1.8.1

Enhancement View pull request
Added metric_type mapping for mntr datastream.

8.2.0 or higher

1.8.0

Enhancement View pull request
Added metric_type mapping for connection datastream

8.2.0 or higher

1.7.0

Enhancement View pull request
Rename ownership from obs-service-integrations to obs-infraobs-integrations

8.2.0 or higher

1.6.0

Enhancement View pull request
Migrate visualizations to lens.

8.2.0 or higher

1.5.1

Enhancement View pull request
Added categories and/or subcategories.

7.14.0 or higher
8.0.0 or higher

1.5.0

Enhancement View pull request
Update ECS version to 8.5.1

7.14.0 or higher
8.0.0 or higher

1.4.0

Enhancement View pull request
Added infrastructure category.

7.14.0 or higher
8.0.0 or higher

1.3.1

Enhancement View pull request
Add documentation for multi-fields

7.14.0 or higher
8.0.0 or higher

1.3.0

Enhancement View pull request
Update to ECS 8.0

—

1.2.0

Enhancement View pull request
Support Kibana 8.0

7.14.0 or higher
8.0.0 or higher

1.1.1

Enhancement View pull request
Uniform with guidelines

—

1.1.0

Enhancement View pull request
Update to ECS 1.12.0

7.14.0 or higher

1.0.0

Enhancement View pull request
Release Kafka as GA

—

0.4.0

Enhancement View pull request
Update integration description

—

0.3.1

Bug fix View pull request
Add missing data_stream fields

—

0.3.0

Enhancement View pull request
Set "event.module" and "event.dataset"

—

0.2.7

Enhancement View pull request
Updating package owner

—

0.2.6

Bug fix View pull request
Correct sample event file.

—

0.1.0

Enhancement View pull request
initial release

—

On this page