Set up Heartbeat monitors

edit

To configure Heartbeat define a set of monitors to check your remote hosts. Specify monitors either directly inside the heartbeat.yml config file, or in external dynamically loaded files located in the directory referenced by heartbeat.config.monitors.path. One advantage of using external files is that these can be automatically reloaded without stopping the Heartbeat process.

Each monitor item is an entry in a yaml list, and so begins with a dash (-). You can define the type of monitor to use, the hosts to check, and other optional settings that control Heartbeat behavior.

The following example configures three monitors checking via the icmp, tcp, and http protocols directly inside the heartbeat.yml file, and demonstrates how to use TCP Echo and HTTP response verification:

# heartbeat.yml
heartbeat.monitors:
- type: icmp
  schedule: '*/5 * * * * * *'
  hosts: ["myhost"]
- type: tcp
  schedule: '@every 5s'
  hosts: ["myhost:7"]  # default TCP Echo Protocol
  check.send: "Check"
  check.receive: "Check"
- type: http
  schedule: '@every 5s'
  urls: ["http://localhost:80/service/status"]
  check.response.status: 200
heartbeat.scheduler:
  limit: 10

Using the heartbeat.yml configuration file is convenient, but has two drawbacks: it can become hard to manage with large numbers of monitors, and it will not reload heartbeat automatically when its contents changes.

Define monitors via the heartbeat.config.monitors to prevent those issues from happening to you. To do so you would instead have your heartbeat.yml file contain the following:

# heartbeat.yml
heartbeat.config.monitors:
  # Directory + glob pattern to search for configuration files
  path: /path/to/my/monitors.d/*.yml
  # If enabled, heartbeat will periodically check the config.monitors path for changes
  reload.enabled: true
  # How often to check for changes
  reload.period: 1s

Then, define one or more files in the directory pointed to by heartbeat.config.monitors.path. You may specify multiple monitors in a given file if you like. The contents of these files is monitor definitions only, e.g. what is normally under the heartbeat.monitors section of heartbeat.yml. See below for an example

# /path/to/my/monitors.d/localhost_service_check.yml
- type: http
  schedule: '@every 5s'
  urls: ["http://localhost:80/service/status"]
  check.response.status: 200

Monitor options

edit

You can specify the following options when defining a Heartbeat monitor in any location. These options are the same for all monitors. Each monitor type has additional configuration options that are specific to that monitor type.

type

edit

The type of monitor to run. One of:

  • icmp: Uses an ICMP (v4 and v6) Echo Request to ping the configured hosts. Requires root access. See ICMP options.
  • tcp: Connects via TCP and optionally verifies the endpoint by sending and/or receiving a custom payload. See TCP options.
  • http: Connects via HTTP and optionally verifies that the host returns the expected response. See HTTP options.

The tcp and http monitor types both support SSL/TLS and some proxy settings.

name

edit

The monitor name. This value appears in the exported fields under the monitor field as the job name and the type field as the job type.

enabled

edit

A Boolean value that specifies whether the module is enabled. If the enabled option is missing from the configuration block, the module is enabled by default.

schedule

edit

A cron-like expression that specifies the task schedule. For example:

  • */5 * * * * * * runs the task every 5 seconds (for example, at 10:00:00, 10:00:05, and so on).
  • @every 5s runs the task every 5 seconds from the time when Heartbeat was started.

The schedule option uses a cron-like syntax based on this cronexpr implementation, but adds the @every keyword.

ipv4

edit

A Boolean value that specifies whether to ping using the ipv4 protocol if hostnames are configured. The default is true.

ipv6

edit

A Boolean value that specifies whether to ping using the ipv6 protocol if hostnames are configured. The default is true.

mode

edit

If mode is any, the monitor pings only one IP address for a hostname. If mode is all, the monitor pings all resolvable IPs for a hostname. The mode: all setting is useful if you are using a DNS-load balancer and want to ping every IP address for the specified hostname. The default is any.

timeout

edit

The total running time for each ping test. This is the total time allowed for testing the connection and exchanging data. The default is 16 seconds (16s).

If the timeout is exceeded, Heartbeat publishes a service-down event. If the value specified for timeout is greater than schedule, intermediate checks will not be executed by the scheduler.

fields

edit

Optional fields that you can specify to add additional information to the output. For example, you might add fields that you can use for filtering log data. Fields can be scalar values, arrays, dictionaries, or any nested combination of these. By default, the fields that you specify here will be grouped under a fields sub-dictionary in the output document. To store the custom fields as top-level fields, set the fields_under_root option to true. If a duplicate field is declared in the general configuration, then its value will be overwritten by the value declared here.

fields_under_root

edit

If this option is set to true, the custom fields are stored as top-level fields in the output document instead of being grouped under a fields sub-dictionary. If the custom field names conflict with other field names added by Heartbeat, then the custom fields overwrite the other fields.

tags

edit

A list of tags that will be sent with the monitor event. This setting is optional.

processors

edit

A list of processors to apply to the data generated by the monitor.

See Filter and Enhance the exported data for information about specifying processors in your config.

ICMP options

edit

These options configure Heartbeat to use ICMP (v4 and v6) Echo Requests to check the configured hosts. These options are valid when the type is icmp.

hosts

edit

A list of hosts to ping.

wait

edit

The duration to wait before emitting another ICMP Echo Request. The default is 1 second (1s).

TCP options

edit

These options configure Heartbeat to connect via TCP and optionally verify the endpoint by sending and/or receiving a custom payload. These options are valid when the type is tcp.

hosts

edit

A list of hosts to ping. The entries in the list can be:

  • A plain host name, such as localhost, or an IP address. If you specify this option, you must also specify a value for ports. If the monitor is configured to use SSL, Heartbeat establishes an SSL/TLS-based connection. Otherwise, it establishes a plain TCP connection.
  • A hostname and port, such as localhost:12345. Heartbeat connects to the port on the specified host. If the monitor is configured to use SSL, Heartbeat establishes an SSL/TLS-based connection. Otherwise, it establishes a TCP connection.
  • A full URL using the syntax scheme://<host>:[port], where:

    • scheme is one of tcp, plain, ssl or tls. If tcp or plain is specified, Heartbeat establishes a TCP connection even if the monitor is configured to use SSL. If tls or ssl is specified, Heartbeat establishes an SSL connection. However, if the monitor is not configured to use SSL, the system defaults are used (currently not supported on Windows).
    • host is the hostname.
    • port is the port number. If port is missing in the URL, the ports setting is required.

ports

edit

A list of ports to ping if the host specified in hosts does not contain a port number.

Example configuration:

- type: tcp
  schedule: '@every 5s'
  hosts: ["myhost"]
  ports: [80, 9200, 5044]

check

edit

An optional payload string to send to the remote host and the expected answer. If no payload is specified, the endpoint is assumed to be available if the connection attempt was successful. If send is specified without receive, any response is accepted as OK. If receive is specified without send, no payload is sent, but the client expects to receive a payload in the form of a "hello message" or "banner" on connect.

Example configuration:

- type: tcp
  schedule: '@every 5s'
  hosts: ["myhost"]
  ports: [7]
  check.send: 'Hello World'
  check.receive: 'Hello World'

proxy_url

edit

The URL of the SOCKS5 proxy to use when connecting to the server. The value must be a URL with a scheme of socks5://.

If the SOCKS5 proxy server requires client authentication, then a username and password can be embedded in the URL as shown in the example.

  proxy_url: socks5://user:password@socks5-proxy:2233

When using a proxy, hostnames are resolved on the proxy server instead of on the client. You can change this behavior by setting the proxy_use_local_resolver option.

proxy_use_local_resolver

edit

A Boolean value that determines whether hostnames are resolved locally instead of being resolved on the proxy server. The default value is false, which means that name resolution occurs on the proxy server.

ssl

edit

The TLS/SSL connection settings. If the monitor is configured to use SSL, it will attempt an SSL handshake. If check is not configured, the monitor will only check to see if it can establish an SSL/TLS connection. This check can fail either at TCP level or during certificate validation.

Example configuration:

- type: tcp
  schedule: '@every 5s'
  hosts: ["myhost"]
  ports: [80, 9200, 5044]
  ssl:
    certificate_authorities: ['/etc/ca.crt']
    supported_protocols: ["TLSv1.0", "TLSv1.1", "TLSv1.2"]

Also see Specify SSL settings for a full description of the ssl options.

HTTP options

edit

These options configure Heartbeat to connect via HTTP and optionally verify that the host returns the expected response. These options are valid when the type is http.

urls

edit

A list of URLs to ping.

Example configuration:

- type: http
  schedule: '@every 5s'
  urls: ["http://myhost:80"]

proxy_url

edit

The HTTP proxy URL. This setting is optional.

username

edit

The username for authenticating with the server. The credentials are passed with the request. This setting is optional.

You need to specify credentials when your check.response settings require it. For example, you can check for a 403 response (check.response.status: 403) without setting credentials.

password

edit

The password for authenticating with the server. This setting is optional.

ssl

edit

The TLS/SSL connection settings for use with the HTTPS endpoint. If you don’t specify settings, the system defaults are used.

Example configuration:

- type: http
  schedule: '@every 5s'
  urls: ["https://myhost:443"]
  ssl:
    certificate_authorities: ['/etc/ca.crt']
    supported_protocols: ["TLSv1.0", "TLSv1.1", "TLSv1.2"]

Also see Specify SSL settings for a full description of the ssl options.

check

edit

An optional request to send to the remote host and the expected response.

Example configuration:

- type: http
  schedule: '@every 5s'
  urls: ["http://myhost:80"]
  check.request.method: HEAD
  check.response.status: 200

Under check.request, specify these options:

method
The HTTP method to use. Valid values are "HEAD", "GET" and "POST".
headers
A dictionary of additional HTTP headers to send.
body
Optional request body content.

Under check.response, specify these options:

status
The expected status code. If this setting is not configured or it’s set to 0, any status code other than 404 is accepted.
headers
The required response headers.
body
A list of regular expressions to match the the body output. Only a single expression needs to match.

The following configuration shows how to check the response when the body contains JSON:

- type: http
  schedule: '@every 5s'
  urls: ["https://myhost:80"]
  check.request:
    method: GET
    headers:
      'X-API-Key': '12345-mykey-67890'
  check.response:
    status: 200
    body: '{"status": "ok"}'

The following configuration shows how to check the response for multiple regex patterns:

- type: http
  schedule: '@every 5s'
  urls: ["https://myhost:80"]
  check.request:
    method: GET
    headers:
      'X-API-Key': '12345-mykey-67890'
  check.response:
    status: 200
    body:
      - hello
      - world

The following configuration shows how to check the response with a multiline regex:

- type: http
  schedule: '@every 5s'
  urls: ["https://myhost:80"]
  check.request:
    method: GET
    headers:
      'X-API-Key': '12345-mykey-67890'
  check.response:
    status: 200
    body: '(?s)first.*second.*third'

Scheduler options

edit

You specify options under heartbeat.scheduler to control the behavior of the task scheduler.

Example configuration:

heartbeat.scheduler:
  limit: 10
  location: 'UTC-08:00'

In the example, setting limit to 10 guarantees that only 10 concurrent I/O tasks will be active. An I/O task can be the actual check or resolving an address via DNS.

limit

edit

The number of concurrent I/O tasks that Heartbeat is allowed to execute. If set to 0, there is no limit. The default is 0.

Most operating systems set a file descriptor limit of 1024. For Heartbeat to operate correctly and not accidentally block libbeat output, the value that you specify for limit should be below the configured ulimit.

location

edit

The timezone for the scheduler. By default the scheduler uses localtime.

watch.poll_file

edit

[6.5.0] Deprecated in 6.5.0. Replaced by using dynamic reloading via the heartbeat.config.monitors option.

The JSON file to watch for additional monitor configurations. The JSON file can contain multiple objects, each of which specifies a different monitor config. Heartbeat checks this file periodically and starts a new monitor instance for each new JSON object added to the file. For example, imagine that you add 10 new entries to the JSON file, each for a different hostname. When Heartbeat picks up the changes in the file, it merges the original config (heartbeat.yml) plus the JSON objects, and starts a monitor for each new host that you’ve configured. If you delete an object from the JSON file and it doesn’t exist in the main config, Heartbeat stops the monitor instance running for that object.

Each monitor has a unique ID that’s based on parameters like protocol, host, and port. If two monitors have the same ID, Heartbeat uses the settings that are defined in the last JSON object of the merged config. This means that you can specify settings in the JSON file that overwrite the settings in the main config. In this way, the configuration that you specify for the monitor in the main Heartbeat config file acts like a default config that you can live-reconfigure by specifying additional configurations in the external JSON file.

Example configuration:

heartbeat.monitors:
- type: tcp
  schedule: '*/5 * * * * * *'
  hosts: ["myhost"]
  watch.poll_file:
    path: {path.config}/monitors/dynamic.json
    interval: 5s
path
Specifies the path to the JSON file to check for updates.
interval
Specifies how often Heartbeat checks the file for changes.

To reconfigure the settings specified in the example config, you could define the following JSON objects in dynamic.json:

{"hosts": ["myhost:1234"], "schedule": "*/15 * * * * * *"} 
{"hosts": ["tls://otherhost:479"], "ssl.certificate_authorities": ["path/to/ca/file.pem"]} 

Upon detecting the changes, Heartbeat stops the old monitor and then restarts it with a schedule of 15 seconds between checks.

Heartbeat starts a new monitor that uses a TLS-based connection with a custom CA certificate.