Alerting and action settings in Kibana

edit

Alerting and action settings in Kibana

edit

Alerting and actions are enabled by default in Kibana, but require you to configure the following:

You can configure the following settings in the kibana.yml file.

General settings

edit
xpack.encryptedSavedObjects.encryptionKey

A string of 32 or more characters used to encrypt sensitive properties on alerting rules and actions before they’re stored in Elasticsearch. Third party credentials — such as the username and password used to connect to an SMTP service — are an example of encrypted properties.

Kibana offers a CLI tool to help generate this encryption key.

If not set, Kibana will generate a random key on startup, but all alerting and action functions will be blocked. Generated keys are not allowed for alerting and actions because when a new key is generated on restart, existing encrypted data becomes inaccessible. For the same reason, alerting and actions in high-availability deployments of Kibana will behave unexpectedly if the key isn’t the same on all instances of Kibana.

Although the key can be specified in clear text in kibana.yml, it’s recommended to store this key securely in the Kibana Keystore. Be sure to back up the encryption key value somewhere safe, as your alerting rules and actions will cease to function due to decryption failures should you lose it. If you want to rotate the encryption key, be sure to follow the instructions on encryption key rotation.

Action settings

edit
xpack.actions.allowedHosts logo cloud

A list of hostnames that Kibana is allowed to connect to when built-in actions are triggered. It defaults to ["*"], allowing any host, but keep in mind the potential for SSRF attacks when hosts are not explicitly added to the allowed hosts. An empty list [] can be used to block built-in actions from making any external connections.

Note that hosts associated with built-in actions, such as Slack and PagerDuty, are not automatically added to allowed hosts. If you are not using the default ["*"] setting, you must ensure that the corresponding endpoints are added to the allowed hosts as well.

xpack.actions.customHostSettings logo cloud

A list of custom host settings to override existing global settings. Default: an empty list.

Each entry in the list must have a url property, to associate a connection type (mail or https), hostname and port with the remaining options in the entry.

In the following example, two custom host settings are defined. The first provides a custom host setting for mail server mail.example.com using port 465 that supplies server certificate authentication data from both a file and inline, and requires TLS for the connection. The second provides a custom host setting for https server webhook.example.com which turns off server certificate authentication, that will allow Kibana to connect to the server if it’s using a self-signed certificate. The individual properties that can be used in the settings are documented below.

xpack.actions.customHostSettings:
  - url: smtp://mail.example.com:465
    ssl:
      verificationMode: 'full'
      certificateAuthoritiesFiles: [ 'one.crt' ]
      certificateAuthoritiesData: |
          -----BEGIN CERTIFICATE-----
          ... multiple lines of certificate data here ...
          -----END CERTIFICATE-----
    smtp:
      requireTLS: true
  - url: https://webhook.example.com
    ssl:
      verificationMode: 'none'

The settings in xpack.actions.customHostSettings can be used to override the global option xpack.actions.ssl.verificationMode and provide customized TLS settings on a per-server basis. Set xpack.actions.ssl.verificationMode to the value to be used by default for all servers, then add an entry in xpack.actions.customHostSettings for every server that requires customized settings.

xpack.actions.customHostSettings[n].url logo cloud

A URL associated with this custom host setting. Should be in the form of protocol://hostname:port, where protocol is https or smtp. If the port is not provided, 443 is used for https and 25 is used for smtp. The smtp URLs are used for the Email actions that use this server, and the https URLs are used for actions which use https to connect to services.

Entries with https URLs can use the ssl options, and entries with smtp URLs can use both the ssl and smtp options.

No other URL values should be part of this URL, including paths, query strings, and authentication information. When an http or smtp request is made as part of running an action, only the protocol, hostname, and port of the URL for that request are used to look up these configuration values.

xpack.actions.customHostSettings[n].smtp.ignoreTLS logo cloud
A boolean value indicating that TLS must not be used for this connection. The options smtp.ignoreTLS and smtp.requireTLS can not both be set to true. Default: false.
xpack.actions.customHostSettings[n].smtp.requireTLS logo cloud
A boolean value indicating that TLS must be used for this connection. The options smtp.ignoreTLS and smtp.requireTLS can not both be set to true. Default: false.
xpack.actions.customHostSettings[n].ssl.rejectUnauthorized
[8.0.0] Deprecated in 8.0.0. Use xpack.actions.customHostSettings.ssl.verificationMode instead. A boolean value indicating whether to bypass server certificate validation. Overrides the general xpack.actions.rejectUnauthorized configuration for requests made for this hostname/port.
xpack.actions.customHostSettings[n].ssl.verificationMode logo cloud
Controls the verification of the server certificate that Kibana receives when making an outbound SSL/TLS connection to the host server. Valid values are full, certificate, and none. Use full to perform hostname verification, certificate to skip hostname verification, and none to skip verification. Default: full. Equivalent Kibana setting. Overrides the general xpack.actions.ssl.verificationMode configuration for requests made for this hostname/port.
xpack.actions.customHostSettings[n].ssl.certificateAuthoritiesFiles
A file name or list of file names of PEM-encoded certificate files to use to validate the server.
xpack.actions.customHostSettings[n].ssl.certificateAuthoritiesData logo cloud
The contents of a PEM-encoded certificate file, or multiple files appended into a single string. This configuration can be used for environments where the files cannot be made available.
xpack.actions.email.domain_allowlist logo cloud

A list of allowed email domains which can be used with the email connector. When this setting is not used, all email domains are allowed. When this setting is used, if any email is attempted to be sent that (a) includes an addressee with an email domain that is not in the allowlist, or (b) includes a from address domain that is not in the allowlist, it will fail with a message indicating the email is not allowed.

This feature is available in Kibana 7.17.4 and 8.3.0 onwards but is not supported in Kibana 8.0, 8.1 or 8.2. As such, this setting should be removed before upgrading from 7.17 to 8.0, 8.1 or 8.2. It is possible to configure the settings in 7.17.4 and then upgrade to 8.3.0 directly.

xpack.actions.enableFooterInEmail logo cloud
A boolean value indicating that a footer with a relevant link should be added to emails sent as alerting actions. Default: true.
xpack.actions.enabledActionTypes logo cloud

A list of action types that are enabled. It defaults to ["*"], enabling all types. The names for built-in Kibana action types are prefixed with a . and include: .email, .index, .jira, .opsgenie, .pagerduty, .resilient, .server-log, .servicenow, .servicenow-itom, .servicenow-sir, .slack, .swimlane, .teams, .tines, .torq, .xmatters, .gen-ai, .d3security, and .webhook. An empty list [] will disable all action types.

Disabled action types will not appear as an option when creating new connectors, but existing connectors and actions of that type will remain in Kibana and will not function.

xpack.actions.preconfiguredAlertHistoryEsIndex logo cloud
Enables a preconfigured alert history Elasticsearch Index connector. Default: false.
xpack.actions.preconfigured
Specifies preconfigured connector IDs and configs. Default: {}.
xpack.actions.proxyUrl logo cloud

Specifies the proxy URL to use, if using a proxy for actions. By default, no proxy is used.

Proxies may be used to proxy http or https requests through a proxy using the http or https protocol. Kibana only uses proxies in "CONNECT" mode (sometimes referred to as "tunneling" TCP mode, compared to HTTP mode). That is, Kibana will always make requests through a proxy using the HTTP CONNECT method.

If your proxy is using the https protocol (vs the http protocol), the setting xpack.actions.ssl.proxyVerificationMode: none will likely be needed, unless your proxy’s certificates are signed using a publicly available certificate authority.

There is currently no support for using basic authentication with a proxy (authentication for the proxy itself, not the URL being requested through the proxy).

To help diagnose problems using a proxy, you can use the curl command with options to use your proxy, and log debug information, with the following command, replacing the proxy and target URLs as appropriate. This will force the request to be made to the proxy in tunneling mode, and display some of the interaction between the client and the proxy.

curl --verbose --proxytunnel --proxy http://localhost:8080 http://example.com
xpack.actions.proxyBypassHosts logo cloud

Specifies hostnames which should not use the proxy, if using a proxy for actions. The value is an array of hostnames as strings. Example:

# If applicable, include the subdomain in the hostname
xpack.actions.proxyBypassHosts: [ "events.pagerduty.com" ]

By default, all hosts will use the proxy, but if an action’s hostname is in this list, the proxy will not be used. The settings xpack.actions.proxyBypassHosts and xpack.actions.proxyOnlyHosts cannot be used at the same time.

xpack.actions.proxyOnlyHosts logo cloud

Specifies hostnames which should only use the proxy, if using a proxy for actions. The value is an array of hostnames as strings. Example:

# If applicable, include the subdomain in the hostname
xpack.actions.proxyOnlyHosts: [ "events.pagerduty.com" ]

By default, no hosts will use the proxy, but if an action’s hostname is in this list, the proxy will be used. The settings xpack.actions.proxyBypassHosts and xpack.actions.proxyOnlyHosts cannot be used at the same time.

xpack.actions.proxyHeaders logo cloud
Specifies HTTP headers for the proxy, if using a proxy for actions. Default: {}.
xpack.actions.proxyRejectUnauthorizedCertificates logo cloud
[8.0.0] Deprecated in 8.0.0. Use xpack.actions.ssl.proxyVerificationMode instead. Set to false to bypass certificate validation for the proxy, if using a proxy for actions. Default: true.
xpack.actions.ssl.proxyVerificationMode logo cloud
Controls the verification for the proxy server certificate that Kibana receives when making an outbound SSL/TLS connection to the proxy server. Valid values are full, certificate, and none. Use full to perform hostname verification, certificate to skip hostname verification, and none to skip verification. Default: full. Equivalent Kibana setting.
xpack.actions.rejectUnauthorized logo cloud

[8.0.0] Deprecated in 8.0.0. Use xpack.actions.ssl.verificationMode instead. Set to false to bypass certificate validation for actions. Default: true.

As an alternative to setting xpack.actions.rejectUnauthorized, you can use the setting xpack.actions.customHostSettings to set SSL options for specific servers.

xpack.actions.ssl.verificationMode logo cloud

Controls the verification for the server certificate that Elastic Maps Server receives when making an outbound SSL/TLS connection for actions. Valid values are full, certificate, and none. Use full to perform hostname verification, certificate to skip hostname verification, and none to skip verification. Default: full. Equivalent Kibana setting.

This setting can be overridden for specific URLs by using the setting xpack.actions.customHostSettings[n].ssl.verificationMode (described above) to a different value.

xpack.actions.maxResponseContentLength logo cloud
Specifies the max number of bytes of the http response for requests to external resources. Default: 1000000 (1MB).
xpack.actions.responseTimeout logo cloud
Specifies the time allowed for requests to external resources. Requests that take longer are canceled. The time is formatted as a number and a time unit (ms, s, m, h, d, w, M, or Y). For example, 20m, 24h, 7d, 1w. Default: 60s.
xpack.actions.run.maxAttempts logo cloud
Specifies the maximum number of times an action can be attempted to run. Can be minimum 1 and maximum 10.
xpack.actions.run.connectorTypeOverrides logo cloud

Overrides the configs under xpack.actions.run for the connector type with the given ID. List the connector type identifier and its settings in an array of objects. For example:

xpack.actions.run:
    maxAttempts: 1
    connectorTypeOverrides:
        - id: '.server-log'
          maxAttempts: 5

Alerting settings

edit
xpack.alerting.maxEphemeralActionsPerAlert logo cloud
[8.8.0] Deprecated in 8.8.0. Sets the number of actions that will run ephemerally. To use this, enable ephemeral tasks in task manager first with xpack.task_manager.ephemeral_tasks.enabled
xpack.alerting.cancelAlertsOnRuleTimeout logo cloud
Specifies whether to skip writing alerts and scheduling actions if rule processing was cancelled due to a timeout. Default: true. This setting can be overridden by individual rule types.
xpack.alerting.rules.minimumScheduleInterval.value logo cloud
Specifies the minimum schedule interval for rules. This minimum is applied to all rules created or updated after you set this value. The time is formatted as a number and a time unit (s, m, h, or d). For example, 20m, 24h, 7d. This duration cannot exceed 1d. Default: 1m.
xpack.alerting.rules.minimumScheduleInterval.enforce logo cloud
Specifies the behavior when a new or changed rule has a schedule interval less than the value defined in xpack.alerting.rules.minimumScheduleInterval.value. If false, rules with schedules less than the interval will be created but warnings will be logged. If true, rules with schedules less than the interval cannot be created. Default: false.
xpack.alerting.rules.run.actions.max logo cloud
Specifies the maximum number of actions that a rule can generate each time detection checks run.
xpack.alerting.rules.run.alerts.max logo cloud

Specifies the maximum number of alerts that a rule can generate each time detection checks run. Default: 1000.

The exact number of alerts your cluster can safely handle depends on your cluster configuration and workload, however setting a value higher than the default (1000) is not recommended or supported. Doing so could strain system resources and lead to performance issues, delays in alert processing, and potential disruptions during high alert activity periods.

xpack.alerting.rules.run.timeout logo cloud
Specifies the default timeout for tasks associated with all types of rules. The time is formatted as a number and a time unit (ms, s, m, h, d, w, M, or Y). For example, 20m, 24h, 7d, 1w. Default: 5m.
xpack.alerting.rules.run.ruleTypeOverrides logo cloud

Overrides the configs under xpack.alerting.rules.run for the rule type with the given ID. List the rule identifier and its settings in an array of objects. For example:

xpack.alerting.rules.run:
    timeout: '5m'
    ruleTypeOverrides:
        - id: '.index-threshold'
          timeout: '15m'
xpack.alerting.rules.run.actions.connectorTypeOverrides logo cloud

Overrides the configs under xpack.alerting.rules.run.actions for the connector type with the given ID. List the connector type identifier and its settings in an array of objects. For example:

xpack.alerting.rules.run:
    actions:
        max: 10
        connectorTypeOverrides:
            - id: '.server-log'
              max: 5