Monitoring your deployments

After you set up your deployment of Enterprise Search, you may want to integrate it with your favorite monitoring system to better understand the health and the performance of the product. To help you with that, we have added the following of solution-level management APIs:

  • Health API – an API endpoint for quickly getting information on the health of a deployment and basic statistics around resource usage.
  • Stats API – a dedicated API for getting information about the resource usage of the application, the state of different internal queues, etc.

Authentication and Authorization

To make a call to one of the monitoring APIs, you need to have a set of Elasticsearch credentials (username/password) for a user that has a monitor Elasticsearch cluster privilege.

Health API

This is the primary API endpoint you can use for gaining insight into the status of your Enterprise Search deployment.

All of the metrics in the API response are point-in-time values from the instance that responds to the request. In clusters with multiple API instances you will need to perform a health API call to each instance to get a view of the whole deployment.

Health API Response Structure

Here are the key components of the response from the Health API:

  • name – the hostname serving the API request
  • version – detailed product version information
  • jvm – detailed Java Virtual Machine (JVM) information, including version and resource usage
  • filebeat – health information for an embedded filebeat instance responsible for shipping application and analytics logs to Elasticsearch
  • system – basic platform information (OS version, Java version, etc)

Health API Example

Here is an example Health API request using a curl command:

curl -X GET http://localhost:3002/api/ent/v1/internal/health \
     -H "Content-Type: application/json" \
     -u username:password

An example response:

{
  "name": "oleksiy-5k.local",
  "version": {
    "number": "7.7.0",
    "build_hash": "7e583e4a8b11471b05ebf3db261e9a8e6558ee4c",
    "build_date": "2020-04-02T11:21:24-04:00"
  },
  "jvm": {
    "pid": 88213,
    "uptime": 234583,
    "memory_usage": {
      "heap_init": 2147483648,
      "heap_used": 811898344,
      "heap_committed": 2010120192,
      "heap_max": 2010120192,
      "non_heap_init": 2555904,
      "non_heap_committed": 344653824
    },
    "memory_pools": [
      "Code Cache",
      "Metaspace",
      "Compressed Class Space",
      "PS Eden Space",
      "PS Survivor Space",
      "PS Old Gen"
    ],
    "threads": {
      "thread_count": 35,
      "peak_thread_count": 37,
      "total_started_thread_count": 39,
      "daemon_thread_count": 13
    },
    "vm_version": "25.202-b08",
    "vm_vendor": "Oracle Corporation",
    "vm_name": "Java HotSpot(TM) 64-Bit Server VM"
  },
  "filebeat": {
    "pid": 88458,
    "alive": true,
    "restart_count": 0,
    "seconds_since_last_restart": null
  },
  "system": {
    "java_version": "1.8.0_202",
    "jruby_version": "9.2.9.0",
    "os_name": "Mac OS X",
    "os_version": "10.14.6"
  }
}

Stats API

For more sophisticated monitoring configurations of Enterprise Search, you can use the Stats API endpoint, which provides deeper information on internal processes and data structures used within your deployment.

Stats API Response Structure

Here are the key components of the response from the Stats API:

  • app – a set of global metrics global for the application instance (including process id, start and end time for the stats collection interval, etc)
  • queues – a detailed set of metrics for all internal queues used for asynchronous work processing
  • connectors – a detailed set of metrics for Workplace Search connectors and job queues

Stats API Example

Here is an example Stats API request using a curl command:

curl -X GET http://localhost:3002/api/ent/v1/internal/stats \
     -H "Content-Type: application/json" \
     -u username:password

An example response:

{
  "app": {
    "pid": 88213,
    "start": "2020-04-21T21:33:25+00:00",
    "end": "2020-04-21T21:34:25+00:00",
    "metrics": {
      "timers.actastic.relation.search": {
        "sum": 184.0155036188662,
        "max": 13.439683942124248,
        "mean": 7.667312650786092
      }
    }
  },
  "queues": {
    "connectors": {
      "pending": 0
    },
    "document_destroyer": {
      "pending": 0
    },
    "engine_destroyer": {
      "pending": 0
    },
    "index_adder": {
      "pending": 0
    },
    "indexed_doc_remover": {
      "pending": 0
    },
    "mailer": {
      "pending": 0
    },
    "refresh_document_counts": {
      "pending": 0
    },
    "reindexer": {
      "pending": 0
    },
    "schema_updater": {
      "pending": 0
    },
    "workplace_search": {
      "pending": 0
    },
    "failed": []
  },
  "connectors": {
    "alive": true,
    "pool": {
      "extract_worker_pool": {
        "running": true,
        "queue_depth": 0,
        "size": 8,
        "busy": 1,
        "idle": 7,
        "total_scheduled": 329,
        "total_completed": 328
      },
      "subextract_worker_pool": {
        "running": true,
        "queue_depth": 0,
        "size": 0,
        "busy": 0,
        "idle": 0,
        "total_scheduled": 0,
        "total_completed": 0
      },
      "publish_worker_pool": {
        "running": true,
        "queue_depth": 0,
        "size": 0,
        "busy": 0,
        "idle": 0,
        "total_scheduled": 0,
        "total_completed": 0
      },
      "status_update_worker_pool": {
        "running": true,
        "queue_depth": 0,
        "size": 0,
        "busy": 0,
        "idle": 0,
        "total_scheduled": 0,
        "total_completed": 0
      }
    },
    "job_store": {
      "waiting": 0,
      "working": 0,
      "job_types": {
        "full": 0,
        "incremental": 0,
        "delete": 0,
        "summary": 0,
        "permissions": 0
      }
    }
  }
}