Switch from APM to Integrations Server payloadedit
This example shows how to use the Elasticsearch Service RESTful API to switch from using APM & Fleet Server to Integrations Server.
Requirementsedit
Given a deployment that is using an APM & Fleet Server with Elastic Stack version 8.0 or later, it is possible to start using Integrations Server instead by updating the deployment with an IntegrationsServerPayload.
Switching from APM & Fleet Server to Integrations Server in this way ensures that the endpoints and credentials currently used by APM Server and Fleet Server remain the same after the switch.
In order to start using the IntegrationsServerPayload, you first need to enable the APM integration for Elastic Agent by following the steps in Switch to the Elastic APM integration.
API request exampleedit
The example shows how to use the API to create a deployment with APM with version 8.0 and update the deployment to switch to Integrations Server.
Create a deployment with APMedit
When creating a deployment with version 8.0 using the ApmPayload, the APM integration for Elastic Agent is enabled by default.
The following creates a deployment that uses the gcp-storage-optimized deployment template in the gcp-us-east4 region
curl -XPOST \
-H 'Content-Type: application/json' \
-H "Authorization: ApiKey $EC_API_KEY" \
"https://api.elastic-cloud.com/api/v1/deployments" \
-d '
{
"resources": {
"apm": [
{
"elasticsearch_cluster_ref_id": "main-elasticsearch",
"region": "gcp-us-east4",
"plan": {
"cluster_topology": [
{
"instance_configuration_id": "gcp.apm.n2.68x32x45",
"zone_count": 1,
"size": {
"resource": "memory",
"value": 1024
}
}
],
"apm": {
"version": "8.0.0"
}
},
"ref_id": "main-apm"
}
],
"elasticsearch": [
{
"region": "gcp-us-east4",
"settings": {
"dedicated_masters_threshold": 6
},
"plan": {
"autoscaling_enabled": false,
"cluster_topology": [
{
"zone_count": 2,
"elasticsearch": {
"node_attributes": {
"data": "hot"
},
"enabled_built_in_plugins": []
},
"instance_configuration_id": "gcp.es.datahot.n2.68x10x45",
"node_roles": [
"master",
"ingest",
"transform",
"data_hot",
"remote_cluster_client",
"data_content"
],
"id": "hot_content",
"size": {
"resource": "memory",
"value": 8192
}
},
{
"zone_count": 2,
"elasticsearch": {
"node_attributes": {
"data": "warm"
},
"enabled_built_in_plugins": []
},
"instance_configuration_id": "gcp.es.datawarm.n2.68x10x190",
"node_roles": [
"data_warm",
"remote_cluster_client"
],
"id": "warm",
"size": {
"resource": "memory",
"value": 0
}
},
{
"zone_count": 1,
"elasticsearch": {
"node_attributes": {
"data": "cold"
},
"enabled_built_in_plugins": []
},
"instance_configuration_id": "gcp.es.datacold.n2.68x10x190",
"node_roles": [
"data_cold",
"remote_cluster_client"
],
"id": "cold",
"size": {
"resource": "memory",
"value": 0
}
},
{
"zone_count": 1,
"elasticsearch": {
"node_attributes": {
"data": "frozen"
},
"enabled_built_in_plugins": []
},
"instance_configuration_id": "gcp.es.datafrozen.n2.68x10x95",
"node_roles": [
"data_frozen"
],
"id": "frozen",
"size": {
"resource": "memory",
"value": 0
}
},
{
"zone_count": 3,
"instance_configuration_id": "gcp.es.master.n2.68x32x45",
"node_roles": [
"master",
"remote_cluster_client"
],
"id": "master",
"size": {
"resource": "memory",
"value": 0
},
"elasticsearch": {
"enabled_built_in_plugins": []
}
},
{
"zone_count": 2,
"instance_configuration_id": "gcp.es.coordinating.n2.68x16x45",
"node_roles": [
"ingest",
"remote_cluster_client"
],
"id": "coordinating",
"size": {
"resource": "memory",
"value": 0
},
"elasticsearch": {
"enabled_built_in_plugins": []
}
},
{
"zone_count": 1,
"instance_configuration_id": "gcp.es.ml.n2.68x16x45",
"node_roles": [
"ml",
"remote_cluster_client"
],
"id": "ml",
"size": {
"resource": "memory",
"value": 0
},
"elasticsearch": {
"enabled_built_in_plugins": []
}
}
],
"elasticsearch": {
"version": "8.0.0"
},
"deployment_template": {
"id": "gcp-storage-optimized"
}
},
"ref_id": "main-elasticsearch"
}
],
"enterprise_search": [],
"kibana": [
{
"elasticsearch_cluster_ref_id": "main-elasticsearch",
"region": "gcp-us-east4",
"plan": {
"cluster_topology": [
{
"instance_configuration_id": "gcp.kibana.n2.68x32x45",
"zone_count": 1,
"size": {
"resource": "memory",
"value": 1024
}
}
],
"kibana": {
"version": "8.0.0"
}
},
"ref_id": "main-kibana"
}
]
},
"name": "switch-to-integrations-server",
"metadata": {
"system_owned": false
}
}
'
Identify the instance configuration to use for Integrations Serveredit
Once the deployment is created, find the instance_configuration_id for IntegrationsServerPayload supported by the deployment template used by the deployment created in the previous step.
In the example above, the deployment was created using the gcp-storage-optimized deployment template in the gcp-us-east4 region.
To find the instance_configuration_id, fetch the deployment template using the template ID, the region, and the version used by the deployment (Integrations Server is supported on version 8.0 and higher).
curl -XGET \ -H 'Content-Type: application/json' \ -H "Authorization: ApiKey $EC_API_KEY" \ "https://api.elastic-cloud.com/api/v1/deployments/templates/gcp-storage-optimized?region=gcp-us-east4&show_instance_configurations=false&stack_version=8.0.0"
This returns a deployment template like the following:
{
"description": "Good for most ingestion use cases with 7-10 days of data available for fast access. Also good for light search use cases without heavy indexing or CPU needs.",
"name": "Storage optimized",
"template_category_id": "storage-optimized",
"id": "gcp-storage-optimized",
"deployment_template": {
"resources": {
"integrations_server": [
{
"elasticsearch_cluster_ref_id": "es-ref-id",
"region": "gcp-us-east4",
"plan": {
"cluster_topology": [
{
"instance_configuration_id": "gcp.integrationsserver.n2.68x32x45",
"zone_count": 1,
"size": {
"resource": "memory",
"value": 1024
}
}
],
"integrations_server": {}
},
"ref_id": "integrations_server-ref-id"
}
],
"elasticsearch": [
...
],
"enterprise_search": [
...
],
"kibana": [
...
],
"apm": [
...
]
}
},
"order": 1,
"system_owned": true,
"metadata": [
...
]
}
|
The instance configuration ID to use in |
Update deployment using IntegrationsServerPayloadedit
Finally, to switch to Integrations Server, update the deployment using the IntegrationsServerPayload, setting
instance_configuration_id to the value identified in the previous step.
curl -XPUT \
-H 'Content-Type: application/json' \
-H "Authorization: ApiKey $EC_API_KEY" \
"https://api.elastic-cloud.com/api/v1/deployments/<deployment-id>" \
-d '
{
"name": "switch-to-integrations-server",
"alias": "switch-to-integrations-server",
"prune_orphans": false,
"metadata": {
"system_owned": false,
"hidden": false
},
"resources": {
"integrations_server": [
{
"region": "gcp-us-east4",
"ref_id": "main-integrations_server",
"elasticsearch_cluster_ref_id": "main-elasticsearch",
"plan": {
"cluster_topology": [
{
"instance_configuration_id": "gcp.integrationsserver.n2.68x32x45",
"zone_count": 1,
"size": {
"resource": "memory",
"value": 1024
}
}
],
"integrations_server": {
"version": "8.0.0"
},
"transient": {
"strategy": {
"autodetect": {}
}
}
}
}
]
}
}
'
|
Make sure to use the |
|
|
Make sure |
Manage your APM & Fleet Server (7.13+)edit
Beginning with Elastic Stack version 8.0, Integrations Server is replacing APM & Fleet Server. New deployments with version 8.0 will use Integrations Server automatically. Existing deployments using APM & Fleet Server will continue to use APM & Fleet Server after upgrading to version 8.0.
Beginning with Elastic Stack version 7.16, the APM integration for Elastic Agent is generally available! This new APM ecosystem brings a number of new benefits to Elastic APM. Check Switch to the Elastic APM integration to learn more.
For deployments that are version 7.13 and later, you have the option to add a combined Application Performance Monitoring (APM) Server and Fleet Server to your deployment. APM allows you to monitor software services and applications in real time, turning that data into documents stored in the Elasticsearch cluster. Fleet allows you to centrally manage Elastic Agents on many hosts.
As part of provisioning, the APM Server and Fleet Server are already configured to work with Elasticsearch and Kibana. At the end of provisioning, you are shown the secret token to configure communication between the APM Server and the backend APM Agents. The APM Agents get deployed within your services and applications.
From the deployment APM & Fleet page you can also:
- Get the URL to complete the APM agent configuration.
-
Use the
elasticcredentials to go to the APM area of Kibana. Step by step instructions to configure a variety of agents are available right in Kibana. After that, you can use the pre-built, dedicated dashboards and the APM tab to visualize the data that is sent back from the APM Agents. -
Use the
elasticcredentials to go to the Fleet area of Kibana. Step by step instructions to download and install Elastic Agent on your hosts are available right in Kibana. After that, you can manage enrolled Elastic Agents on the Agents tab, and the data shipped back from those Elastic Agents on the Data streams tab. -
Reset the APM secret token.
Resetting the token disrupts your APM service and restarts the server. When the server restarts, you’ll need to update all of your agents with the new token.
- Access the APM & Fleet logs and metrics.
- Stop and restart your APM & Fleet Server.
- Upgrade your APM & Fleet Server version if it is out of sync with your Elasticsearch cluster.
- Fully remove the APM & Fleet Server, delete it from the disk, and stop the charges.
Upgrading to Elastic Stack 8.0edit
The following APM settings have been removed in Elastic Stack version 8.0. This change is only relevant to users upgrading a standalone (legacy) deployment of APM Server to Elastic Stack version 8.0. Check Add APM user settings for more details.
apm-server.api_key.enabled apm-server.api_key.limit apm-server.ilm.* apm-server.frontend.* apm-server.register.* apm-server.rum.allow_service_names apm-server.rum.event_rate.lru_size apm-server.rum.event_rate.limit apm-server.rum.rate_limit output.elasticsearch.bulk_max_size output.elasticsearch.index output.elasticsearch.indices output.elasticsearch.pipeline output.elasticsearch.pipelines output.elasticsearch.worker setup.* queue.*
Manage your APM Server (versions before 7.13)edit
For deployments that are version 6.3 and later, you have the option to add an Application Performance Monitoring (APM) Server to your deployment. APM allows you to monitor software services and applications in real time, turning that data into documents stored in the Elasticsearch cluster. The APM data is automatically available in Kibana for your searching and visualizing needs. For more information regarding Elastic APM, check www.elastic.co/solutions/apm.
As part of provisioning, the APM Server is already configured to work with Elasticsearch and Kibana. At the end of provisioning, you are shown the secret token to configure communication between the APM Server and the backend APM Agents. The secret token can be found and reset on the APM & Fleet page of the deployment menu (it is not managed in the user override settings). The APM Agents get deployed within your services and applications.
From the deployment APM page you can also:
- Get the URL to complete the agent configuration.
-
Use the
elasticcredentials to go to the APM area of Kibana. Step by step instructions to configure a variety of agents are available right in Kibana. After that, you can use the pre-built, dedicated dashboards and the APM tab to visualize the data that is sent back from the APM Agents. -
Reset the secret token.
Resetting the token disrupts your APM service and restarts the server. When the server restarts, you’ll need to update all of your agents with the new token.
- Access the APM logs and metrics.
- Stop and restart your APM Server.
- Upgrade your APM Server version if it is out of sync with your Elasticsearch cluster.
- Fully remove the APM Server, delete it from the disk, and stop the charges.
Starting with stack version 7.13, the APM Server is accompanied by a Fleet Server. Check Manage your APM and Fleet Server for details.