Monitoring in a production environmentedit

By default, the X-Pack monitoring agents on Elasticsearch index data into the cluster where they’re running. In production, you should send data to a separate monitoring cluster so that historical monitoring data is available even if the nodes you are monitoring are not.

[beta] This functionality is in beta and is subject to change. The design and code is less mature than official GA features and is being provided as-is with no warranties. Beta features are not subject to the support SLA of official GA features. In 6.4 and later, you can use Metricbeat to ship monitoring data about Kibana to a separate monitoring cluster. In 6.5 and later, you can do the same for Elasticsearch.

If you have at least a gold license, using a dedicated monitoring cluster also enables you to monitor multiple clusters from a central location.

To store monitoring data in a separate cluster:

  1. Set up the Elasticsearch cluster you want to use as the monitoring cluster. For example, you might set up a two host cluster with the nodes es-mon-1 and es-mon-2.

    Note

    To monitor an Elasticsearch 6.x cluster, you must run Elasticsearch 6.x on the monitoring cluster.

    1. (Optional) Verify that the collection of monitoring data is disabled on the monitoring cluster. By default, the xpack.monitoring.collection.enabled setting is false.

      For example, you can use the following APIs to review and change this setting:

      GET _cluster/settings
      
      PUT _cluster/settings
      {
        "persistent": {
          "xpack.monitoring.collection.enabled": false
        }
      }
    2. If the Elasticsearch security features are enabled on the monitoring cluster, create users that can send and retrieve monitoring data.

      Note

      If you plan to use Kibana to view monitoring data, username and password credentials must be valid on both the Kibana server and the monitoring cluster.

      • [beta] This functionality is in beta and is subject to change. The design and code is less mature than official GA features and is being provided as-is with no warranties. Beta features are not subject to the support SLA of official GA features. If you plan to use Metricbeat to collect data about Elasticsearch or Kibana, create a user that has the remote_monitoring_collector built-in role and a user that has the remote_monitoring_agent built-in role. Alternatively, use the remote_monitoring_user built-in user.
      • If you plan to use HTTP exporters to route data through your production cluster, create a user that has the remote_monitoring_agent built-in role.

        For example, the following request creates a remote_monitor user that has the remote_monitoring_agent role:

        POST /_xpack/security/user/remote_monitor
        {
          "password" : "changeme",
          "roles" : [ "remote_monitoring_agent"],
          "full_name" : "Internal Agent For Remote Monitoring"
        }

        Alternatively, use the remote_monitoring_user built-in user.

  2. Configure your production cluster to collect data and send it to the monitoring cluster.

    • [beta] This functionality is in beta and is subject to change. The design and code is less mature than official GA features and is being provided as-is with no warranties. Beta features are not subject to the support SLA of official GA features. Use Metricbeat. This option is available in 6.5 and later versions.
    • Use HTTP exporters.
  3. (Optional) Configure Logstash to collect data and send it to the monitoring cluster.

    Note

    You must configure HTTP exporters in the production cluster to route this data to the monitoring cluster. It cannot be accomplished by using Metricbeat.

  4. (Optional) Configure Kibana to collect data and send it to the monitoring cluster:

    • [beta] This functionality is in beta and is subject to change. The design and code is less mature than official GA features and is being provided as-is with no warranties. Beta features are not subject to the support SLA of official GA features. Use Metricbeat. This option is available in 6.4 and later versions.
    • Use HTTP exporters.
  5. (Optional) Create a dedicated Kibana instance for monitoring, rather than using a single Kibana instance to access both your production cluster and monitoring cluster.

    1. (Optional) Disable the collection of monitoring data in this Kibana instance. Set the xpack.monitoring.kibana.collection.enabled setting to false in the kibana.yml file. For more information about this setting, see Monitoring settings in Kibana.
  6. Configure Kibana to retrieve and display the monitoring data.